free html hit counter An Apology To My RSS Readers - But I Had To Do It. (Updated) | John Battelle's Search Blog

An Apology To My RSS Readers – But I Had To Do It. (Updated)

By - February 22, 2013

Some random site running my last post without asking.

If you’re a fan of this site, you’re also probably a fan of RSS – a once-ascendant technology that has been on most everyone’s deathwatch for five or so years. According to Google’s (almost totally outdated) Feedburner service, nearly 450,000 people subscribe to this blog via RSS – although the number of you who actually read my posts is far smaller (according to Feedburner statistics, which I’ve never fully understood).

In any case, from time to time I’ve poked at you poor RSS readers, just to find out if you’re alive. Remember this piece – Is RSS Really Dead? Or this one - Once Again, RSS Is Dead. But ONLY YOU Can Save It!?

In those posts, I asked if my beloved RSS readers were really out there. Turns out, I got tons of comments back – a very high number given the work involved in declaring fealty to the creaky old standard. (It kind of felt like a reshoot of that wonderful final scene in Horton Hears a Who – “Everybody yell real loud, and maybe Google will hear, and not deprecate Feedburner…”  But I digress.)

I’ve always kept my RSS feed “full text” – which means the entire post, pictures, words and all, goes out over RSS, and can be picked up by any RSS reader anywhere on the planet. I always have held the belief that it’s more important that my work get distributed than monetized. But not everyone can afford such high minded principles. Many publishers cut their feed short, teasing folks with headlines and a snippet of the story in the hopes that people will click through to the site, where their visit can be properly “monetized” via advertising.

My new feed (sadface).

After much thought, I’m going to do the same. But not for the extra clicks and ads. It’s due to the fraud that’s taken over the content space in the Indpendent Web. Untold legions of bad actors use RSS to scrape “real” sites like this one, then wrap them with ads from exchanges to make a quick buck. The rise of programmatic fraud has made this even worse (see It’s Time To Call Out Fraud In The Adtech Ecosystem for more on this). And no, I’m not going to link to examples – but you can Google “Content Scraping” if you want to learn more.

So, consider this an apology. I am very sorry that you have to click a link to get to the content I make here every day. But also consider this a plea – as in, please do click that link at the top. I very much want you to be part of this conversation.

(And if enough of you complain, you know I’ll listen, and figure out some way around this).

 

UPDATE: I turned full feed back on. Thanks for all the input.

Related Posts Plugin for WordPress, Blogger...
  • http://matthewlmcclure.com/ Matt McClure

    Curious: how much scraping fraud activity were you seeing of your posts?

    • johnbattelle

      I have no idea the full extent, but I see my own stuff about five to ten times a month somewhere else.

  • Claudio Ibarra

    I’ve been an RSS reader for a long time. I’ll keep the truncated feed, but I apologize in advance if I don’t click through much of the time. Thanks for being cognizant of RSS readers. Whenever I talk about RSS to other people, I have to start with a quick primer on what it is… and I’ve been with Google Reader for years and years now.

    • johnbattelle

      Yeah, RSS is so clearly representative of the strengths and weaknesses of the open web

  • Subramanian Rao

    Truncating RSS does not solve the issue you have. Fraudsters can still scrape your site and republish them. Screen scraping is simple script to write. If that was your only concern I don’t see any reason to not publish full text in the feed.

    • johnbattelle

      It’s a tiny bit more work to run a screen scaper, and I have found fraudsters are lazy as hell. Almost all the stuff I’ve found has been directly off my RSS feed.

      • http://josephsmarr.com/ Joseph Smarr

        Sorry John, I agree with Subramanian, this sucks for your loyal readers (like me) and is at best a temporary treatment of the current symptom, but certainly not a meaningful deterrent in the larger scheme of things (as long as there’s value to be had by copying public content, it’ll get copied). I think you’re dong more to hurt the web by turning off your RSS feed than by helping these guys make some extra money by scraping your content. I do hope you’ll reconsider.

        You’re one of the few tech pundits who genuinely (and compellingly) argues for the Open Web and data portability, so if supporting long-established open standards for your own personal public blog is too much trouble to be worth it, what chance do the rest of us have, and what weight will your words carry?

        • johnbattelle

          Your Open rationale makes sense to me. The open nature of the internet is its greatest strength and it’s biggest curse.

          • http://www.ouvre-boite.com Julien

            Also, of you’re worried about duplicate content, Matt Cutts published a video indicating that they use PubSubHubbub as means to detect the freshest content: http://www.youtube.com/watch?v=4LsB19wTt0Q

  • John Leahy

    Apology accepted. Enough websites use truncated feeds that I’m used to it.

    (Besides, the day will come when Google+ and/or Facebook will finally do the Embrace, Extend, Extinguish act on RSS… why it hasn’t happened yet I have no idea.)

  • long time reader

    Unsubscribed. Sorry but this is nonsense.

    • johnbattelle

      Nonsense? Can you at least tell me why you think that?

  • Ally

    My question is does truncating actually work for its intended purpose in a day with websites/services such as http://fulltextrssfeed.com/ ??? I hope for your sake that it does.

    • johnbattelle

      Sigh, probably not. But then again, not sure that many folks would use that service. Again, if I hear that you all hate this, I’ll do something else.

  • Cynthia Peterson

    Just to let you know that I am out here and am reading and don’t mind another click to get your posts!

    • johnbattelle

      Very kind of you thanks!

  • http://dannyman.toldme.com/ Daniel Howard

    So, punish 450,000 subscribers to discourage a few criminals, eh?

    Thank you for the warning.

    -danny

    • johnbattelle

      I’ll count that as a “bring it back to full text” vote…thanks!

  • http://twitter.com/mickmel mickmel

    I understand the reasoning, but why does it matter? How does this help you? Unless Google was letting those scrapers outrank your original posts, then don’t worry about it. If it’s a problem, add RSS-only footer links using WP SEO or similar tools to help Google understand that you’re the authority (most scrapers will leave them in place).

    Assuming that’s you’re outranking the scarper sites, which you clearly are, then:

    - Doing this doesn’t help you.
    - Doing this hurts your readers

    I understand that you’re not happy about it, but I’m not clear why it really matters. One of my blogs gets reposted around 100 times for each post, but I don’t care. It sucks, but my energy goes into producing additional solid content. Google knows I’m the authority, so I get the lion’s share of the traffic, and I can still offer full RSS feeds to my readers.

    • johnbattelle

      You make fine points (I added an RSS only header a while ago, but found many scrapers deprecate the links), but there’s one additional point that bears considering: I don’t want to feed the beast that is driving fraudulent content into the programmatic advertising ecosystem. It’s less about SEO to my site, and more about those 100 posts of yours – those are probably all being monetized and that just sucks – it destroys the value of the independent web in other ways.

      • http://twitter.com/mickmel mickmel

        I certainly agree that it sucks, but I still don’t quite get it.

        We both know that while those sites are being “monetized”, that:

        A – They’re making pennies per post, at best.
        B – You’re not losing any money as a result of them.

        Heck, most will probably continue to republish your new-truncated RSS feed and continue to “destroy the value of the independent web” to roughly the same degree. I agree that it’s awful and should be stopped, but I’m not fully convinced that this is the way to do it.

        If you’re simply fighting for the bigger cause, you need to rally bloggers everywhere to truncate their RSS feeds and make a difference. In the meantime, it seems more like a political statement at our expense, as it really won’t make a blip in the global RSS-republishing-garbage-sites volume.

        • johnbattelle

          It’s exactly these economics that make it so insidious. Processing is dirt cheap, so why not create millions of spam blogs, populate them with scraped content, and make pennies per post? The money just piles up. It sucks.

  • Autumn

    I’ll register my vote to keep the full feed. I agree, as long as you get most of the traffic, does it really matter? I’ll certainly stay subscribed, just probably read less content. Please make an effort to give us great headlines and descriptions.

    • johnbattelle

      I am so torn reading through all of this. Really appreciate all your feedback.

  • Michale Froomkin

    I would really miss this ‘voice in my head’ so please count me as a voice for the full feed. (FWIW, I provide a full feed on my blog, not that anyone is likely to want to steal it.)

  • http://www.facebook.com/nick.haler Nick Haler

    I’d just like to add my vote for the full feed. I read your blog (among others) offline on my phone on the tube. I’d only be able to click and get the full content if I had reception. Yep there’s plenty of time during the day when I have reception, or a proper computer, but I like being able to keep up with blogs on the daily commute.

    • johnbattelle

      I understand. I wonder if I can perhaps make the email newsletter full feed. I’ll look into that.

      • http://www.ili.as ilias

        Instapaper or Pocket? I tend to use them quite often and they blend pretty well with Twitter and other apps. What’s the added benefits of an RSS reader over these two services?

        • Ally

          but you have to go find the stuff you want to read and save it to Instapaper or Pocket – where as with the RSS reader, it’s automatically already there waiting for you…

  • Rayna

    I’m in, and will click through. If only to find out if you end up converting back to an iPhone:)

    • johnbattelle

      Ha no chance of that. I love my Nexus 4.

  • rss_junkie

    I’ll click through. It’s not ideal, but the reasons make sense. Take this as a big compliment to the quality of your writing; I’ve dropped a lot of feeds once they required clicking-through to read.

    • johnbattelle

      Honored thanks.

  • Andy Price

    This rocks, JBat. Amazing to me that customizing content for one’s tastes is such a clunky process fraught with so much peril in 2013. This is why I think Yahoo has a shot.

    • johnbattelle

      Yeah it’s time for some better tools for the “Rest of the Web”

  • http://feedspot.com/ Anuj Agarwal

    I was truly disheartened when i read the article title. First I thought you are no longer going to support rss feed for your blog. Anyways, this change does not affect me as i always visit the original article page to read comments and sometimes post a comment as well.

    • johnbattelle

      And I appreciate it!

  • http://twitter.com/bradbarrish Brad Barrish

    Hi John. Thanks for this. As someone who reads you via RSS from time to time, this bums me out a lot. Getting your voice and message out should trump your desire to not feed the machine. Don’t make it harder for people to get at your content. I’m not sure I believe that truncating your RSS is going to have the desired effect (and it doesn’t sound like you do either). Consider this a vote for the full feed, though a truncated feed just means I will likley read less of you, but won’t abandon you altogether.

    • johnbattelle

      The open web is a pain in the ass. I hear you.

  • Noah Brier

    Very far from ideal but you could publish full text rss under a secret URL that you share with commenters (or requesters who somehow prove they’re human/not scammers). Again, far from perfect but it might work ….

  • Da Wiz

    Atleast to read the comments of the article from the rip-off, one gets re-directed to your original article JB. I for one enjoy the discussions/comments quite a bit.

    Goodluck with the RSS feed, I use it for sites where I am worried about their overbearing design or flashy ads which will catch everyone’s attention at the work place (remember the belly reducing ads?). Your site however I access directly via your links on Twitter, I can trust it – so far.

    • Da Wiz

      Thinking of options, perhaps a captcha prior to sharing the RSS feed link might reduce some of that – if this practice is rampant based on your RSS feed alone. In any case I am not sure what is to prevent someone from scraping your site directly and updating content (assuming there is a diff), any small script could do that.

    • johnbattelle

      Ha – so far!

  • http://twitter.com/CoreyHyllested Corey Hyllested

    My preference is the full feed.
    I look forward to your posts and would probably click through a fair bit.

  • Aditya

    I’m surprised that you held out for so long. I’m so used to clicking on other feeds in my reader that it doesn’t really matter.

    That said, maybe a couple more paragraphs before truncating? Just so I can decide whether to click through now or later depending on when /where I’m reading.

    • johnbattelle

      For sure I can do that. Maybe even like 500 words.

  • vetinary

    I do understand the problem, and IF John says that he is not doing this for the extra page views, i’ll believe him. One more click for me is not such a big problem. However, the “if” of the first sentence, is still a big, capital, IF..

    • johnbattelle

      I swear it’s not for the Ads. Compared to real ad drive sites, my traffic is very low and even if it doubled due to this it’d not matter in the larger scheme of things.

  • Benedict Leigh

    This is another vote for the full feed. I read exclusively via rss, and on my phone (ios/byline/google reader) whilst commuting. I have a restricted data plan and limited reception. Full text feeds allow me to download everything via wifi before I leave in the morning.

    I enjoy the site and will probably switch to one of the multiple services that convert a truncated feed to full text. I suspect that, whilst a minor inconvenience to spammers, the only other thing this will do is make your stats more unreliable if others do the same.

    It’s a real shame that you are doing this, although it is your content and your decision.

    • johnbattelle

      I hear you, another vote for the full text

  • http://colinwalker.me.uk colinwalker

    I always prefer full feed but, having been the victim of RSS scraping myself a number of times (and it seems to be increasing), I can certainly understand the frustration.

    I’ve been lucky that I’ve managed to get a number of sites to stop publishing my stuff but the truly automated sites aren’t going to care about takedown notices etc. – they’ll just throw up another site if the current one gets removed.

  • Henrik Lindhe

    One vote for full feed

  • A.

    RSS full feed vote, I never click through. Sorry

  • http://arkanosis.net/ Jérémie Roquet

    I’m sorry I’ve no solution to propose for your problem, but I’ll complain anyway (though, you know, you don’t owe me anything, so at the end, you do what you want to).

    I’m suscribed to hundreds of RSS feeds ; I don’t read every single post that is written : I read one paragraph, maybe two, then I decide if the post is worth reading or not. With such an amoung of posts to “read”, I’m definitely not happy with having to click, wait for a full website to load, and so on. Then, unless the title itself suffices for me to know I really want to read the post (and this doesn’t happen very often), I’ll probably no try to read the full article.

    Best regards

    • http://arkanosis.net/ Jérémie Roquet

      To clarify: by “and this doesn’t happen very often”, I’m talking about the title being enough, not about me wanting to read the post.

  • http://friendlybit.com Emil Stenström

    Very few of my feeds are truncated, and I tend to skip cilcking through on them quite often (it sets a higher quality bar). This is even true on mobile, where the excellent “Reader” app caches and formats the text excellently for full feeds compared to often bad mobile experiences that requires click-through.

    In summary: I tend to lean to truncated feeds only when there’s advertising revenue that I find more important that actually getting people to read my content.

  • http://twitter.com/lukeglassford Luke Glassford

    Hi John, I’m very much a vote for keeping it as full text. I feel your pain regarding scrapers (we all have the same problem), but changing your publishing practice to the detriment of your readers is not the way to beat them – in my humble opinion.

  • http://twitter.com/brucewarila Bruce Warila

    Full feed or nothing. I rarely have the time to click into blogs that truncate. I hear your concern about ‘feeding the beast’, but you are trying to cure cancer with a toothpick. Your knowledge is more important to solving the problem than your proposed solution :)

    • johnbattelle

      Thanks

  • Sicaine

    Most of my feeds don’t have full content in it anyway. I don’t care, if the first paragraph or the title is interesting, i will click on it

  • Tirpstil

    Reading blogs on my tablet is just so much better with RSS. The text is always the same size! Your feed had the paragraph breaks zapped but I read it anyway. RSS really is a treasure that should be nurtured. People don’t know what they are missing.

    • johnbattelle

      Amen to that.

  • http://josephratliff.com/ JosephRatliff

    John,

    You might consider ensuring your site is “mobile friendly” first, then truncate away. :)

    (I don’t read your stuff in mobile personally, but if someone clicks to read in mobile, they need your site to be mobile friendly)

    Your free content, which I appreciate btw, is good enough to click on the site and read it. From the user’s side, we have Instapaper and Readability to convert the articles and read them cleanly if necessary.

    I can’t see how it’s going to solve the problem you’re having though, if “scrapers” want your content, they’ll get it, it’s a very difficult problem to solve.

  • muhamad fawad

    Hi,I am new one.I found your blog really informative and helpful.Thank you.
    http://superioreducationz.com
    Education Information

  • anon

    Voting for full feed as well!

  • http://www.dominicsayers.com/ Dominic Sayers

    Other commenters have made all the points I was going to make. Thanks for sticking with full text RSS for so long. Sorry to see it go. Maybe there’s a way I can get your full posts to read on the train, sure wish I didn’t have to find it though.

    • johnbattelle

      it’s really interesting to find out how many people read what I write offline. I had no idea.

  • Jason Cumberland

    I’ll throw a vote for “bring it back to full text” as well. If you can measure the amount of fraud before and after the truncation it would be interesting to see if it improves the issue.

    • johnbattelle

      I wish it was measurable. It’s like Whack A Mole.

  • http://www.google.com/profiles/jeremyarnold Jeremy Arnold

    So always read via the feed, and I will admit now that we’ve got to click through to read the story we will be wanting to see something that tells us why. Headlines work..

    However would you consider writing a summary paragraph on your article via RSS so we can see if want to click through ??

    • johnbattelle

      I’m not sure I’d write a summary, though it’s a great idea. I barely have time to write anything!

  • Nick R

    I’m a huge RSS reader. (at least 75% of my web reading is through RSS.) I used to hate truncated feeds. But, like others, I’ve gotten used to them. In fact, when done right, it helps me get through my feeds faster. I have no problem clicking through for a story I’m interested in.

    Here’s my advice, though. Title becomes much, much more important. Also, I think it’s important to have a strong first paragraph that does show up in the feed. Enough for me to get a gist of what the article is about and care enough to read the rest.

    • johnbattelle

      Yes, the idea that this might change my writing and headline style is kind of a negative for me. Hmmm /

  • y

    I read almost all posts via RSS, but with +1000 different posts a day, I haven’t the time or energy to click on a link to a full article. Except when the article/post seems to very interesting… So it is even more important that the first sentence and headline is extremely telling and unique. The bots will have the time to follow any link from any good blog anyway, so I don’t think you’ll beat them. I very much hope you’ll start publishing the full posts again!

  • ivoszz

    It is not problem for me as I always click through to your site when I am interested.

  • http://profiles.google.com/daniel.oconnor Daniel O’Connor

    Have you considered embedding advertising or similar in the RSS feed? If they are going to syndicate it, they can have your adwords on their blog too.

    Alternatively, use google’s verification features to assert ownership over your content. If it’s syndicated, and they publish your verification token… well, google will see you control said content.

    For example, with google’s webmaster tools, you can set up URL removal requests.
    Removing / from the index of google’s site… would likely be a good way to discourage content theft :)
    I believe with that you need to set up a meta tag or upload a HTML file; but surely there must be other similar ways you can use your control over content…

    • johnbattelle

      Yes I have considered it. And the problem is, we have no idea who you guys are, so we can’t market who you are to advertisers.

  • http://gabeprime.tumblr.com/ Gabriel Fua

    I vote for full RSS text. I tend to read your work through Google reader and Newsify on iOS. I basically agree with the points raised by the people who are for full RSS text so I don’t feel the need to reiterate them. I hope you reconsider.

  • obo
    • http://twitter.com/HiveMindMap HiveMindMap

      And just like that, the scrapers can follow suit.

    • http://dannyman.toldme.com/ Daniel Howard

      John, maybe you can put that link in your headers? It would be a pity for the scrapers to know where to get a full feed but not your readers . . .

  • http://www.facebook.com/LeninNair Lenin VJ Nair

    That is a good thing to do. Even I have my blog’s RSS cut short. One thing you can do is tynt to enable backlinking from any content scraped websites. There is however a chance to accumulate some bad neighborhood links which you can disavow.

    • johnbattelle

      so much work to just write a site.

      • http://www.facebook.com/LeninNair Lenin VJ Nair

        Indeed.

  • http://twitter.com/dosinga Douwe Osinga

    I’ll add my vote to keep the full feed. Most of the arguments seem to have been made already. I like what Brad says – don’t punish your actual fans for what fraudsters do.

  • Jean Jordaan

    I’m usually subscribed to between 100 and 200 feeds via Google Reader. I read them offline on long bus rides, in a cool little newsreader on my phone that gives me plain black on white fullscreen text in the font of my choice. There’s always more to read than I can finish, so incomplete feeds tend to get skipped.

  • dylanbiles

    This is a real shame because I like reading your stuff but I know myself well enough to know that I won’t be clicking through regularly. There have been entire blogs that I’ve stopped reading because I couldn’t read them natively in my Google Reader.

    Sorry it’s come to this, John.

    • johnbattelle

      I may change it back.

  • http://twitter.com/kwegner Kyle Wegner

    John – I realize you’ve gotten a lot of comments already, but here’s another vote for full feed. Not only is this the only way that I keep up with my favorite content from all of my top sources, but I also judge the type of content by full feed as well. Sometimes I don’t have the time to read through some of your long form stuff, so I can “keep unread” for as long as I need until I get through it. And I DO get through it. I’ve probably never quoted a single author more than you. Short form stuff I can get through right away. Truncated means I need to click through on every item to judge whether or not I have the time to read through the article. While the extra step seems trivial, it isn’t. RSS is about speedy browsing, and this breaks that cycle.

  • http://twitter.com/SeanMcGee Sean McGee

    ANNNNNNND Unsubscribe.

    I like your writing, but honestly, it’s not good enough to go through the trouble of clicking through. There are other blogs out there that have comparable content. Sorry!

  • Richard Posey

    I can’t believe how petty (and pompous) the unsubscribers sound. I am not the most regular reader of your blog, but I know you put in a great deal of time to not only write, but to cultivate the kind of contacts required to write knowledgeably. They can’t be bothered with another click? Good Heavens..How precious their time must be.

    As far as the Open Web is concerned, I guess I’m for not truncating. Still, I get your posts on iGoogle (the sentenced-to-death iGoogle) and they’re just headlines to me until I click..And, when I read your posts, they have value far in excess of the click that I “pay” to read it.

    • johnbattelle

      yes readers are fickle but i’m also a reader….

  • http://www.facebook.com/merrilee Merrilee Proffitt

    Please don’t make me click. I’m lazy and won’t. Especially when I’m on my phone, which is when I do a lot of my feed reading. I’ll probably wind up deleting and missing out on good content. Thank you!

  • http://fearmyblog.com/ tacanderson

    Here’s an idea: I know there are ways to do this, and maybe it’s not worth the effort (I’ll keep reading and clicking either way) but what if there was a way for the RSS subscriptions “relationship” to be more like Facebook and less like Twitter? What is the author had to approve someone to get the full feed? I know you can publish “private” full feeds, sites like NSFWCorp do this but it just seems like there’s a better way to give full access to people you trust vs those who don’t.

    Don’t know, just thinking out loud.

    • johnbattelle

      I think innovation in RSS stopped in like 2008. Alas.
      I think I’ll probably go back to full feed.

  • Full text for the win

    I have to say that I’m an occasional reader. I usually skim the full text and decide if I want to commit to the full article. I greatly dislike clicking through because I only get one paragraph to decide whether or not an article is something I want to spend time reading. I have no problem clicking through for content I find interesting, but the decision process is severely limited for truncated posts.

    I think the knowledge that one must front-load one’s writing to attract readership also cheapens the writing. Also as other readers have noted, as an advocate of the open web, you have a duty to ‘live by your principles’ in my opinion. I therefore vote for full-text.

    Finally I would say that If you do return to full text I will likely click through more often anyway given the level of care I’ve seen you put forth to respond to your readers.
    Best wishes

  • YK

    I’ll put in another vote for full feed – I mainly read through Google Reader, and over the years I’ve found the friction of clicking through for a full article is rather underestimated. It is almost always easier & more satisfying to just skip to another full post, since I have so many blogs to see, so my clickthroughs for abridged posts are in genera abysmally low. It’s far easier to skip to someone else’s full posts, sadly

    I’ve been reading your blog for a long time, before I started at Google, during my 6 years there, and still reading now after I left. Was one of the best sites I’ve come across when I was getting into the industry. Thanks for putting out such great content

    • johnbattelle

      Thanks. When I get a moment I’ll go back to full feed.

  • Jon

    Simple. On your RSS button, have it redirect to a form. Once people put their email in, they get sent a new RSS feed link. Only people who take the effort to sign up can get the feed. It might help…

  • Timothy

    Please bring back full text

    • johnbattelle

      I will this weekend!

  • Ayesha

    Wow! Very much appreciated work and increse .
    All Grand Theft Auto Games