News Corp’s Rupert Murdoch has been making a stink about charging for news and getting control of his content away from search engines (read: Google).
Google News is throwing Murdoch a bone with a change to the way Google honors the robots.txt file. Robots.txt is a standard way for sites to control what files search engines see and index. For example, you could tell search engines to ignore certain directories, or allow them to index Web pages but not images.
Sites can now exercise control specifically over the Google News spider. So for instance you could let the Google search engine spider index your content, but not allow that content to appear on Google News.
To me, that’s silly. If you’re a news site it’s to your benefit to appear in Google News. It can’t help but send traffic to your site. But if you’re Rupert Murdoch and you don’t want that Google now accommodates your wishes.
See also Google 5-click.
RSS Posts Feed
Murdoch is stupid about this for the reasons you mention, but I think Google is also doing the right thing here. Different user agents for different pulls makes intuitive sense.