Fake yourself as a googlebot

For those who are unaware, a googlebot is a program that visits webpages on the internet, extract information to be stored in databases. And when you, normal users do a search using google, google will use these databases to give you links to appropriate websites.That’s a brief about the task of a googlebot.

So what’s the advantage of being a googlebot?

Well, the googlebot is allowed access to webpages that are normally restricted to registered users (members-only sites). This is mainly because the site owners want their websites to be searchable via google but at the same time preventing non-members from accessing. The most basic option they choose: check the User Agent

Basically, the User Agent contains info about the browser you are using(opera,firefox,internet explorer…), the operating system(winxp,vista,linux…)

So, all you have to do to fake yourself as a googlebot is simply to change the data in your User Agent.

How to do it

Firefox Users

Download and install the User Agent Switcher from here

After installation, you will find that a new menu item has been added in your firefox browser.

click on Tools–>User Agent Switcher–>options–>options (As shown in the shot below)

Now, select User Agents and click on “Add”

Fill in the fields as shown below.



Now, click on Tools–>User Agent Switcher–>googlebot

You are done, congrats, your browser will now be recognised as a googlebot!

To revert back, click on Tools–>User Agent Switcher–>Default

Internet Explorer Users

For Internet Explorer you need to change registry entries.

Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindowsCurrentVersionInternet Settings5.0User Agent] @=”Googlebot/2.1″ “Compatible”=”+http://www.googlebot.com/bot.html”

Save this as ua.reg and execute.

To revert the changes back, you need the following:

Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindowsCurrentVersionInternet Settings5.0User Agent] @=”Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)”

Save as oua.reg and execute.

Note: I copy pasted the part for Internet Explorer from here. When it ask you to save as oua.red, all you have to do is open notepad, copy paste the text there, and save as: oua.reg

Make sure you change the “save as type” from txt to All files!

+++++++++++++++++++++++++++++++++++++++++++++++++++++

TESTING

The washingtonpost is a news website but it restrict the news only to paying members Fortunately, it allows the googlebot to access the news So set your User Agent as googlebot and check out the news there…If you succeed, means your are correctly faking the googlebot, else, try again

To clear your doubts whether this faking is really working, try reading the news there as a normal user, (without being a googlebot).

Among the sites that will most likely let you in as googlebot is membership-required forums, some paying sites also.

here’s another forum which you can try:http://www.governmentsecurity.org/forum/index.php

Hmm, I wish to compile a list of such websites that restrict access to members and googlebots only. So help from all of you is welcomed, send the links through comments. (one link only per comment entry)

References:

http://quicksilverscreen.com/ipb/index.php?showtopic=29115

http://www.idealwebtools.com/blog/authenticate-googlebot/

http://classictutorials.blogspot.com/2006/11/become-googlebot-get-free-pass.html

Enter your email address:

Delivered by FeedBurner