30 January, 2015 02:00 PM
An example could throw more light on this new feature. A search on “modern paintings” in Google will show the search results, to be precise the full snippet. But a mobile search with a blocked snippet will display “A description for this result is not available because of this site’s robots.txt – learn more.”
The updated webmaster guideline specifies that the crawlers should be allowed access to critical files that your page uses for “optimal rendering and indexing.” Pierre Far, Google’s Webmaster Trends Analyst was quoted as saying “Blocking of the files will “directly harms how well our [Google] algorithms render and index your content and can result in suboptimal rankings.”
To ascertain whether GoogleBot can render your web pages properly, Google created the fetch and render tool within the webmaster tools. Many webmasters block GoogleBot from crawling files like CSS for better ranking. But Google has strictly recommended webmasters to refrain from this practice as it can hurt a website’s ranking.