How Often Does Google Crawl Your Site?
January 31st 2008 00:12
Google Crawl Cycle: An Important Metric
Would you like to know how often Google crawls your site? If not, why not? According to the folks at SEOmeter.com, "How often search engine [sic] visits and crawls website content is an often neglected, but important metric for search engine optimization." They explain that a site's Crawl Cycle, or CC, is an important indicator of how much the search engines "trust" a particular website. The shorter the website's CC, the more trusted the site.
Why Is Crawl Cycle Important?
The main reason you might want to pay attention to this metric is that the degree of trust your website enjoys with the search engines is directly reflected in your site's search engine ranking. SEOmeter offers a free tool (free at least for your top-level domain and/or sub-domain) which will help you track your own site's Google Crawl Cycle. (If you'd like to track internal URLs using this tool, however, there is an annual payment.) The SEOmeter site has been in operation for about a month, so far. To read more about the site and the SEOmeter tool, visit the site's blog.
SEO Meter's Widgets
Here are examples of some of the nifty widget styles you may choose when you use SEOmeter on your website:
80 x 15 pixel button:
120 x 60 pixel button:
120 x 90 pixel button:
125 x 125 pixel button:
Other SEO Meter Features
Another neat thing about this site is that you can compare the Crawl Cycle of different websites, which are broken down into categories, with the top 20 in each category conveniently listed on the Top-20 Most Crawled Sites on the Web page.
You can also display the Crawl Cycle of your site and two others on a line graph, if you'd like to see a visual of the comparative CCs of the three sites to find out where your site ranks in relation to others in your niche.
Ease of Use/Benefits
The SEOmeter and other handy features of this website are easy--and even fun--to use and can help you keep track of your site's popularity with the search engines, since any attention your website or blog receives from Google is likely to be reflected in Yahoo! and other search engines, as well.
What could be easier than placing the SEOmeter tool on your site and simply watching it continuously recalculate your website's Crawl Cycle? Though I've only had mine for a few days, I have a feeling that, as I continually monitor it, I can expect to learn a lot.
Want to do the same?
Jeanne
NOTE: The only thing I had trouble figuring out was why all four of the widgets used as examples above weren't displaying the same Crawl Cycle. I'd wondered whether this indicated inaccuracy in the tool or something else. After e-mailing SEOmeter, I received the following explanation from Peter:
To reduce our server load, we do not calculate the statistics on the widget for each page refresh on your blog, but store (cache) the previously calculated stats somewhere in our server, and show it on the widget. And this caching thing happens for each size of widgets independently. So in this case, 1.9 was an outdated number, which was previously cached.
I just tried clearing the stat cache on your widgets, and they are now corrected. The stats on the widget get updated once every 24 hours, so even if you see an outdated/mismatching number, dont worry about it. It will be self corrected within the next 24 hours.
I just tried clearing the stat cache on your widgets, and they are now corrected. The stats on the widget get updated once every 24 hours, so even if you see an outdated/mismatching number, dont worry about it. It will be self corrected within the next 24 hours.
I certainly appreciated that explanation! Thanks, Peter!
Did you enjoy this post? Have any thoughts or any other handy SEO tools of your own to share? We'd love to hear from you!
| 91 |
| Vote |
Add To: 
subscribe to this blog
Comment by AmyHuang
Project Job Search
Travel Debate
Travel String
Love Adventures
Comment by Jeanne Dininni
Writer's Notes
While I'm sure the calculations, algorithms, and all that are complicated, the bottom line, for us bloggers is that the number on the widget represents the time period, in days, between the Googlebot's visits to our sites--and the smaller the number, the better. The more often Google crawls your site, the more important the search engines consider your site to be.
At the moment, my widget is reporting that Google crawls my site once every 2 days or so looking for content. That's not too bad, though it's by no means the best either, if you consider that ProBlogger's site is crawled by Google about twice a day. So, whereas my widget currently says "2.2," ProBlogger's currently reads "0.6." (Yesterday, my widget read "1.9" and I believe Problogger's read "0.5")
To get a comparison of how you rank compared to other sites in your niche, you can visit the Top 20 Most Crawled Sites on the Web page (link in my post) and compare your stats with those of other similar blogs. The SEO Meter site is only a month old, though, so the number of websites listed isn't that high yet (only about 800, according to an e-mail I received from them), though it's expected to grow as more people find out about the site.
Why not visit the site and take a look around? You can learn a lot that way. You can also check your own blog's crawl cycle right on the website, without downloading the widget to your blog, if you just want to see what it's all about before deciding whether to download it.
I find it really interesting, myself--even if my blog only ranks in the "top" 57% right now. (I was told that they have a lot of high profile websites listed--like CNN and MIT--which were added for testing purposes during their launch; but that, as other regular sites join, the ratio of high-profile sites to regular sites will go down, giving us normal people more of a chance for a better ranking.
Thanks for your comment!
Jeanne