So, you’re not happy with search engines (Google in particular) for exposing your content to an audience who aren’t paying for it. You don’t like the way they include snippets in their search results of your stories and you definitely don’t like them referring visitors to your websites (unless they are paying customers). You seem aggravated with aggregation and less than happy about linking. So here’s a suggestion for you.

Google themselves (yep those folks you keep moaning about) have kindly provided a really easy way for you to get your content out of their search index, and you can just block the news search crawler if you want now too! It couldn’t be easier. Let’s take a fictional website In the very first directory where the website files are stored (the root) you’ll find a text file called robots.txt (so that file lives at All you need to do is find that file, open it up in Notepad (or a text editor of your choosing), and add the two lines of text below to it. Save it and your Google problems are over (if this all proves a bit tricky, get one of your lovely web developers to help you (while you’re at it why not ask them if they think it’s a good idea too?).

User-agent: Googlebot

Disallow: /

Job done, give it a couple of weeks and none of your pages will be in Googles index anymore. That troublesome traffic will be no more. Satisfied?

Now come on, thats not the answer is it? Here’s an idea for you. Why not devote your time, energies, finances and skilled personnel into coming up with a new model to make all this free traffic and advertising work for you? Rather than moan about it, find a way to make it work for you. It’s about time someone made some advances in the world of online display advertising and I’d have thought that with all your web properties you’d be just the man/organisation to do so. The possibilities are endless, start to tap into the rich data you can glean from the tracks your web visitors leave each time they visit. Learn from it, find ways to encourage repeat visitors and new channels to monetise them through. Of course you may want to reconsider my earlier recommendation first, otherwise you won’t have enough traffic to benefit from any improvement to your advertising and other revenue streams.

If you can design experiences that encourage visitors to become loyal users of your content maybe you could even sell them something? Maybe (just maybe) if you make your sites engaging enough some might even subscribe! There are so many ways you could make more revenue from so much traffic. I’ll be available in July 2010 to help if you haven’t worked it out by that time…

What are you going to do? Just block the traffic, alienate potentially loyal users and try and get people to pay for your content? Or move forwards proactively, embrace the fact you get so much traffic to your sites (it’s a good thing, honest) and work out a really viable model to monetise it properly.

Personally, I’d go with the latter (with so many pages, so much content and so much traffic you have endless possibilities). The former just strikes me as the reactionary moves of an industry with so much promise in the digital world that gives off an impression of being on it’s last legs.


This post has been moved to my new home on the web The post itself can be found here. 23Musings is going to remain dormant for now and some of its more popular posts will be transferred to the blog at my new site.

My reason for moving and not taking the years of posts with me is the need for a fresh start, 23Musings had been dormant for long enough for me to feel it was time to start my blog from scratch.

Don’t forget to check out my new blog!

There’s so much talk about news at the moment with the moves to bring back paywalls (as I’ve written about previously here) and the general nervousness among old media houses that the likes of Google are stealing their thunder (and their ad dollars). Everyone is musing about how this will play out, who will be the winners and what is actually the best way to deliver news to web users. A couple of things have struck me in the last couple of days that deserved a blog post.

Firstly, Google have launched a new product from its labs department. Google Fast Flip is being touted as a new way to read the news, that brings the web experience more in line with newspapers. Basically Google has taken screenshots of web pages containing news stories and put them into a neat user interface that allows you to flip between stories and zoom in on them before deciding whether to click through to the original article.

Google Fast Flip

Google Fast Flip

It’s a really nice way to browse and a great UX to find serendipitous content as you might stumble across something you’re interested in. Of course there’s also Google’s search power under the hood so you can narrow down the content available, isn’t there?

No, it seems that Google hasn’t put it’s search power to good use by letting you search the full content of the articles available (I’ve tested it by searching for specific content within the stories). It is actually just a nice UI to flip and browse through news stories, it doesn’t put that UI on top of the power of Google News though.

Secondly, Microsoft recently unveiled a vision of a ‘Next-Generation Newspaper’ as part of a response to a request from the Newspaper Association of America asking for ideas about ‘monetizing digital content’. The next-generation newspaper delivered as part of their response is remarkably similar to Tweetdeck, as noted by Nieman Journalism Lab today.

Microsofts idea for a 'next-generation newspaper'

Microsofts idea for a 'next-generation newspaper'

The news-deck delivers content in an RSS reader kind of way using a stack of Microsoft technology, content and advertising products. It looks good, and the promise of semantic search, personalisation and contextual awareness is promising.

So, are either of these a ground breaking new way to consume news and do they hold enough promise to herald the future of web based news reading? I don’t think so. They both offer nice solutions to different types of user scenario in my opinion.

Google Fast Flip is a great UI that allows for casual browsing of the news in a similar kind of way to which I read the Sunday newspapers. It’s a coffee and bacon sandwich type situation, but not a serious trawl for information experience for me. It’s too much detail upfront with not enough ability to refine for anyone serious (or used to) discovering content online or even using RSS readers. At least with an RSS reader you can make your choices of who to subscribe to and read articles based on headlines or quick glances at the content. Yes, this is a quick way to thumb the pages but it’s a little gimicky for me and doesn’t fulfill my needs for consuming information. Google partnered with top newspapers to create this but I don’t think it helps them. Digital news content is consumed in a different way to paper content and I think this needs more work before it could ever become the norm for online news reading. Must admit it’s nice on an iPhone though!

Microsoft’s  news-deck aproach suits my needs much better. It has the search and refine type features I’d demand and a familiar UI that works for dashboard type experiences. Is it the future? I don’t think so. I actually think it’s a little lame of Microsoft to propose a solution like this to the challenge from the NAA, it’s too similar to RSS readers and looks too like a cross between Netvibes and Tweetdeck. In fact if you read the details of their response (available from the Nieman Labs post) it really does just sound like next-gen Netvibes to me. It’s not really the future, more what we have now mk2.

Now I’m not sure what the future of news consumption is but I’m pretty sure this isn’t it. If you could combine the two you might be getting somewhere, if Fast Flip allowed me to search and read all the content and not click off to the news source it might be better and if Microsofts dashboard wasn’t just a rehashed Netdeck/Tweetvibe it might get me more excited. Personally I’m seeking something more intelligent and with better data mining possibilities to satisfy my search for news (and I wouldn’t limit it to news, any solution should be my medium for information consumption as a whole).

One thing this does make apparent though is that as much as the newspapers and old media want to pull their content in and keep it close, they are going to have to relinquish their hold and let their content free as whatever the user experience of the future is it will demand that.

Paid content, pay walls and the link economy; topics receiving massive attention in the last week thanks to the document (via the Nieman Journalism Lab) distributed by the Associated Press which details their plans to reclaim their content and seek to make it work harder for them. It’s a troubling time for news companies as they struggle to find a way to make their websites profitable and look on in distrust as the aggregators increase their audience and in turn their ad dollars. Is locking up content behind subscriptions and selling online entities the rights to reuse it actually going to help them in the long run?

An interesting debate into this has been sparked by this article from Arnon Mishkin which elicited this response from Jeff Jarvis. They take opposing sides but at the same time both make valid points. Two posts worth reading!

In my opinion the answer isn’t charging for everything. Content creators have the right to charge for something if it’s valuable to someone else, the problem for news is that everyone writes slightly different opinions of the same thing (this post being one example of that). Newspapers and publishers need to have something unique to offer if they’re going to charge users to read it let alone charge others to reuse or further distribute. Some will be able to do this (the FT being a good example), others really don’t offer enough value to get away with it and I’m afraid I think their dreams of making the web their most profitable distribution channel are over.

One problem is that there’s only so many news stories out there and if everyone is covering the same thing then only the best coverage will survive (users will very quickly stop paying if the content isn’t good enough). The other problem I foresee for news is that if you lock your content away and prevent linking then you’ll kill the buzz that surrounds a good story you do publish (how many retweets will a pay walled article get?).

The web is built on links and content curation (not just creation); that’s how buzz explodes, memes spread and serendipity occurs. Cutting this off could just kill a lot of news sources as the more forward thinking (and innovative online) take advantage and continue to push the free and distributed model. Even newspapers get shared, picked up, discussed in the pub and curated. Will web pages behind pay walls have a lifespan or will they end up forgotten in some archive that Google can’t access?

I think linking’s vital to the webs survival and continued growth, it’s how it all started (my first job in web involved building link portals in 1995), and without it we’d be a much poorer community. Content creators need to find ways to monetize their assets while not hindering access or distribution, a new model is required.

Amazed to hear today about Reuters new deal with the International Herald Tribune. Reuters have agreed to allow the IHT to carry their content in return for a share of the advertising revenue generated. This is a major change in their business model (a change for the better as far as I am concerned).

News wires traditionally charge a subscription fee for content sites to carry their story and distribute it. This is a great step towards becoming a 21st century company for Reuters. They have the potential to make much more money from advertising than from subscriptions.

Yes, subscriptions are a safe flow of revenue, but by being mature and allowing people to use their content for no upfront charge they actually stand to make far more money.

Well done Reuters for wising up to the web! Next step, make it truly free (although this is a good first step)! And well done to the web for continuing to shake old media at it’s core and forcing them to grow up to keep up!

Google News goes visual

June 27, 2007

Google news have now launched a visual way to read the news through a new interface which displays images from the news stories of the day along with the headlines and links to the stories.

It’s a really nice interface and as you hover over an image the list of headlines on the right scrolls and highlights those that are associated (nice AJAX work).

I like this way of browsing the news, now Google, please could you reverse engineer it so that when I hover over a headline and intro on the text version I get to see an associated image?

Yahoo have teamed up with Reuters to begin a push into news related user generated content. They’ll be inviting the public to submit eyewitness photos and videos of news events.

Yahoo will allow users to submit related photo’s on their site, they will go through some kind of vetting process one would assume and then end up published on Yahoo meta-tagged as related to a particular story. The images wil be run as part of topical packages on the site, I guess meaning that as big news stories break Yahoo will build the content around those stories.

The You Witness system will be expanded to take content for sports, entertainment etc in the future, but news will be the initial push. The content will be shared with Reuters after a few months of testing on Yahoo only.

Yahoo and Reuters are working on a way to compensate contributors when their media is used in a commercial manner, which should ensure a good level of submission as budding photographers and journalists everywhere try to get in on the act.

Yahoo’s had a few good ideas lately, making use of their purchases such as Flickr in this case in some innovative ways is always good for their PR factor.