user-agent-string.info

About UA list Download Home

Project user-agent-string.info after nine years was terminated on November 30, 2014.

The new version can be found at udger.com

Participate in the project -> Collecting information on the robots

The user-agent-string.info project collects information about robots by detecting their access to the robots.txt file (http://www.example.com/robots.txt). The description of the syntax and use of robots.txt is in the document the robot exclusion standard .
The idea is simple. Before exploring a certain web site a correctly coded robot downloads the file robots.txt. If the access to the file is monitored, we have what we needed.

If you place a monitoring script on your server, you'll help with collecting the information. If you do this, don't forget to write so that you can be added to the list of cooperating websites.

What will be recorded into our DB?
- We record only the UA, the IP and time, nothing else.
Can the inaccessibility of the user-agent-string.info cause me any problems?
- No, only the information about the robot won't be recorded into our DB.

Solutions: APACHE + PHP + .htaccess

In the file .htaccess is written

The contents of a robots.txt file

<?PHP
//--[user-agent-string.info botdetect for APACHE + PHP + .htaccess]--
//------------------ CONF START ---------------------------------------

$filename = "robots.orig"; 	// new name for file robots.txt

//------------------ CONF END -----------------------------------------
$i .= "&ua=".urlencode($_SERVER["HTTP_USER_AGENT"]);
$i .= "&adr=".urlencode($_SERVER["REMOTE_ADDR"]);
$url = "http://user-agent-string.info/rpc/botdetect.php?$i";
$fp = @fopen($url, "rb");
@fclose ($fp);
if (!($file = @fopen ($filename, "r"))) {
	header ("HTTP/1.0 404 Not Found");
	echo "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n";
	echo "<html><head><title>404 Not Found</title></head><body><h1>Not Found</h1>\n";
	echo "<p>The requested URL /robots.txt (let us say: ".$filename.") was not found on this server.</p></body></html>";
}
else {
	$robots = fread ($file, filesize ($filename));
	fclose ($file);
	header("Last-Modified: ".gmdate("D, d M Y H:i:s")." GMT");
	header("Accept-Ranges: bytes");
	header("Content-Length: ".strlen($robots));
	header("Content-Type: text/plain");
	echo $robots;
}
?>

Example file robots.orig

- Recently Added UserAgent to DB - RSS - Contact