Day 4 – Are you being Indexed ?



Are you being indexed ?

How do you know ?

If so how often are you being indexed ?

If you’ve managed to get your website listed in google search results, then well done! for getting that far ! Unfortunately that’s not the end of it, SEO can be a full time job ensuring that content is added/updated regularly and making connections online to gain popularity. The Google team is trying very hard to make websites “actually good” as opposed to being touted as such. So those who update their content regularly are given “more attention”. It might be time for you to check to see when you were last visited by Google Bot!

My post for today is to get you thinking about how often Google is looking at your site, is it daily/weekly/monthly/yearly ? Perhaps the static nature of your content is causing Googlebot to visit you only once or twice a year ? You wont know unless you check !

The Technical Method

In order to check the frequency of Google bot visits to your site you’ll need access to your website logs. Not all hosting options will give you access to them especially if your hosting is “shared-hosting”. This is a consideration for people who what that level of control, you may need to pay extra for a hosting service that is dedicated and gives you full access to the web server logs.

If you have linux shell access to your website you can check for Google bot activity on your site using this command.

cd /var/log/httpd/
grep -i "googlebot" access_log | more

You should see some results similar to this below. Line #1, tells me that Googlebot downloaded the robots.txt file on my server with IP on the 6th of January. - - [06/Jan/2013:04:05:22 +0000] "GET /robots.txt HTTP/1.1" 302 - "-" "Mozilla/5.0 compatible; Googlebot/2.1; +" - - [06/Jan/2013:04:05:23 +0000] "GET /robots.txt HTTP/1.1" 200 95 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +" - - [06/Jan/2013:04:13:31 +0000] "GET /robots.txt HTTP/1.1" 302 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +" - - [06/Jan/2013:04:13:31 +0000] "GET /robots.txt HTTP/1.1" 200 101 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +" - - [06/Jan/2013:04:44:46 +0000] "GET /2012/02/630/ HTTP/1.1" 200 16689 "-" "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +" - - [06/Jan/2013:04:50:47 +0000] "GET /files/2012/10/merrigold-red-buggy.jpg HTTP/1.1" 304 - "-" "Googlebot-Image/1.0" - - [06/Jan/2013:05:40:05 +0000] "GET /robots.txt HTTP/1.1" 302 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +" - - [06/Jan/2013:05:40:06 +0000] "GET /robots.txt HTTP/1.1" 200 89 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +" - - [06/Jan/2013:05:40:06 +0000] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +" - - [06/Jan/2013:05:41:06 +0000] "GET /2012/07/12/hello-world/ HTTP/1.1" 200 14742 "-" "SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/ (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +"

The Non-Technical Method

Google Webmaster Tools has a place to check on “crawl stats”. Once you sign up for an account, add and verify ownership of your website you can select the “Crawl Stats” menu item from the Health Menu. See this screenshot.

Google Webmaster Tools


How do you get added to the index ?

You can request Google to index your website by visiting this Google Webmaster Tools page.   There is no guarantee that it will happen quickly or at all, but it is certainly one way to notify Google that your new site exists.   Generally speaking if you create a new website with great content that people are visiting through your own advertising (either paid or social networks), then Google will find you and index you.   You are more likely to be indexed quickly if you have “backlinks” from prominent websites or blogs.

Matthew Aldous - Adelaide SEO
0420 833 220
9-5 weekdays (CST)
Wordpress Template Developed by Matthew Aldous