If you’re a fan of this site, you’re also probably a fan of RSS – a once-ascendant technology that has been on most everyone’s deathwatch for five or so years. According to Google’s (almost totally outdated) Feedburner service, nearly 450,000 people subscribe to this blog via RSS – although the number of you who actually read my posts is far smaller (according to Feedburner statistics, which I’ve never fully understood).
In any case, from time to time I’ve poked at you poor RSS readers, just to find out if you’re alive. Remember this piece – Is RSS Really Dead? Or this one – Once Again, RSS Is Dead. But ONLY YOU Can Save It!?
In those posts, I asked if my beloved RSS readers were really out there. Turns out, I got tons of comments back – a very high number given the work involved in declaring fealty to the creaky old standard. (It kind of felt like a reshoot of that wonderful final scene in Horton Hears a Who – “Everybody yell real loud, and maybe Google will hear, and not deprecate Feedburner…” But I digress.)
I’ve always kept my RSS feed “full text” – which means the entire post, pictures, words and all, goes out over RSS, and can be picked up by any RSS reader anywhere on the planet. I always have held the belief that it’s more important that my work get distributed than monetized. But not everyone can afford such high minded principles. Many publishers cut their feed short, teasing folks with headlines and a snippet of the story in the hopes that people will click through to the site, where their visit can be properly “monetized” via advertising.
After much thought, I’m going to do the same. But not for the extra clicks and ads. It’s due to the fraud that’s taken over the content space in the Indpendent Web. Untold legions of bad actors use RSS to scrape “real” sites like this one, then wrap them with ads from exchanges to make a quick buck. The rise of programmatic fraud has made this even worse (see It’s Time To Call Out Fraud In The Adtech Ecosystem for more on this). And no, I’m not going to link to examples – but you can Google “Content Scraping” if you want to learn more.
So, consider this an apology. I am very sorry that you have to click a link to get to the content I make here every day. But also consider this a plea – as in, please do click that link at the top. I very much want you to be part of this conversation.
(And if enough of you complain, you know I’ll listen, and figure out some way around this).
UPDATE: I turned full feed back on. Thanks for all the input.
Curious: how much scraping fraud activity were you seeing of your posts?
I have no idea the full extent, but I see my own stuff about five to ten times a month somewhere else.
I’ve been an RSS reader for a long time. I’ll keep the truncated feed, but I apologize in advance if I don’t click through much of the time. Thanks for being cognizant of RSS readers. Whenever I talk about RSS to other people, I have to start with a quick primer on what it is… and I’ve been with Google Reader for years and years now.
Yeah, RSS is so clearly representative of the strengths and weaknesses of the open web
Truncating RSS does not solve the issue you have. Fraudsters can still scrape your site and republish them. Screen scraping is simple script to write. If that was your only concern I don’t see any reason to not publish full text in the feed.
It’s a tiny bit more work to run a screen scaper, and I have found fraudsters are lazy as hell. Almost all the stuff I’ve found has been directly off my RSS feed.
Sorry John, I agree with Subramanian, this sucks for your loyal readers (like me) and is at best a temporary treatment of the current symptom, but certainly not a meaningful deterrent in the larger scheme of things (as long as there’s value to be had by copying public content, it’ll get copied). I think you’re dong more to hurt the web by turning off your RSS feed than by helping these guys make some extra money by scraping your content. I do hope you’ll reconsider.
You’re one of the few tech pundits who genuinely (and compellingly) argues for the Open Web and data portability, so if supporting long-established open standards for your own personal public blog is too much trouble to be worth it, what chance do the rest of us have, and what weight will your words carry?
Your Open rationale makes sense to me. The open nature of the internet is its greatest strength and it’s biggest curse.
Also, of you’re worried about duplicate content, Matt Cutts published a video indicating that they use PubSubHubbub as means to detect the freshest content: http://www.youtube.com/watch?v=4LsB19wTt0Q
Apology accepted. Enough websites use truncated feeds that I’m used to it.
(Besides, the day will come when Google+ and/or Facebook will finally do the Embrace, Extend, Extinguish act on RSS… why it hasn’t happened yet I have no idea.)
Unsubscribed. Sorry but this is nonsense.
Nonsense? Can you at least tell me why you think that?
My question is does truncating actually work for its intended purpose in a day with websites/services such as http://fulltextrssfeed.com/ ??? I hope for your sake that it does.
Sigh, probably not. But then again, not sure that many folks would use that service. Again, if I hear that you all hate this, I’ll do something else.
Just to let you know that I am out here and am reading and don’t mind another click to get your posts!
Very kind of you thanks!
So, punish 450,000 subscribers to discourage a few criminals, eh?
Thank you for the warning.
-danny
I’ll count that as a “bring it back to full text” vote…thanks!
I understand the reasoning, but why does it matter? How does this help you? Unless Google was letting those scrapers outrank your original posts, then don’t worry about it. If it’s a problem, add RSS-only footer links using WP SEO or similar tools to help Google understand that you’re the authority (most scrapers will leave them in place).
Assuming that’s you’re outranking the scarper sites, which you clearly are, then:
– Doing this doesn’t help you.
– Doing this hurts your readers
I understand that you’re not happy about it, but I’m not clear why it really matters. One of my blogs gets reposted around 100 times for each post, but I don’t care. It sucks, but my energy goes into producing additional solid content. Google knows I’m the authority, so I get the lion’s share of the traffic, and I can still offer full RSS feeds to my readers.
You make fine points (I added an RSS only header a while ago, but found many scrapers deprecate the links), but there’s one additional point that bears considering: I don’t want to feed the beast that is driving fraudulent content into the programmatic advertising ecosystem. It’s less about SEO to my site, and more about those 100 posts of yours – those are probably all being monetized and that just sucks – it destroys the value of the independent web in other ways.
I certainly agree that it sucks, but I still don’t quite get it.
We both know that while those sites are being “monetized”, that:
A – They’re making pennies per post, at best.
B – You’re not losing any money as a result of them.
Heck, most will probably continue to republish your new-truncated RSS feed and continue to “destroy the value of the independent web” to roughly the same degree. I agree that it’s awful and should be stopped, but I’m not fully convinced that this is the way to do it.
If you’re simply fighting for the bigger cause, you need to rally bloggers everywhere to truncate their RSS feeds and make a difference. In the meantime, it seems more like a political statement at our expense, as it really won’t make a blip in the global RSS-republishing-garbage-sites volume.
It’s exactly these economics that make it so insidious. Processing is dirt cheap, so why not create millions of spam blogs, populate them with scraped content, and make pennies per post? The money just piles up. It sucks.
I’ll register my vote to keep the full feed. I agree, as long as you get most of the traffic, does it really matter? I’ll certainly stay subscribed, just probably read less content. Please make an effort to give us great headlines and descriptions.
I am so torn reading through all of this. Really appreciate all your feedback.
I would really miss this ‘voice in my head’ so please count me as a voice for the full feed. (FWIW, I provide a full feed on my blog, not that anyone is likely to want to steal it.)
I’d just like to add my vote for the full feed. I read your blog (among others) offline on my phone on the tube. I’d only be able to click and get the full content if I had reception. Yep there’s plenty of time during the day when I have reception, or a proper computer, but I like being able to keep up with blogs on the daily commute.
I understand. I wonder if I can perhaps make the email newsletter full feed. I’ll look into that.
Instapaper or Pocket? I tend to use them quite often and they blend pretty well with Twitter and other apps. What’s the added benefits of an RSS reader over these two services?
but you have to go find the stuff you want to read and save it to Instapaper or Pocket – where as with the RSS reader, it’s automatically already there waiting for you…
I’m in, and will click through. If only to find out if you end up converting back to an iPhone:)
Ha no chance of that. I love my Nexus 4.
I’ll click through. It’s not ideal, but the reasons make sense. Take this as a big compliment to the quality of your writing; I’ve dropped a lot of feeds once they required clicking-through to read.
Honored thanks.
This rocks, JBat. Amazing to me that customizing content for one’s tastes is such a clunky process fraught with so much peril in 2013. This is why I think Yahoo has a shot.
Yeah it’s time for some better tools for the “Rest of the Web”
I was truly disheartened when i read the article title. First I thought you are no longer going to support rss feed for your blog. Anyways, this change does not affect me as i always visit the original article page to read comments and sometimes post a comment as well.
And I appreciate it!
Hi John. Thanks for this. As someone who reads you via RSS from time to time, this bums me out a lot. Getting your voice and message out should trump your desire to not feed the machine. Don’t make it harder for people to get at your content. I’m not sure I believe that truncating your RSS is going to have the desired effect (and it doesn’t sound like you do either). Consider this a vote for the full feed, though a truncated feed just means I will likley read less of you, but won’t abandon you altogether.
The open web is a pain in the ass. I hear you.
Very far from ideal but you could publish full text rss under a secret URL that you share with commenters (or requesters who somehow prove they’re human/not scammers). Again, far from perfect but it might work ….
Atleast to read the comments of the article from the rip-off, one gets re-directed to your original article JB. I for one enjoy the discussions/comments quite a bit.
Goodluck with the RSS feed, I use it for sites where I am worried about their overbearing design or flashy ads which will catch everyone’s attention at the work place (remember the belly reducing ads?). Your site however I access directly via your links on Twitter, I can trust it – so far.
Thinking of options, perhaps a captcha prior to sharing the RSS feed link might reduce some of that – if this practice is rampant based on your RSS feed alone. In any case I am not sure what is to prevent someone from scraping your site directly and updating content (assuming there is a diff), any small script could do that.
Ha – so far!
My preference is the full feed.
I look forward to your posts and would probably click through a fair bit.
I’m surprised that you held out for so long. I’m so used to clicking on other feeds in my reader that it doesn’t really matter.
That said, maybe a couple more paragraphs before truncating? Just so I can decide whether to click through now or later depending on when /where I’m reading.
For sure I can do that. Maybe even like 500 words.
I do understand the problem, and IF John says that he is not doing this for the extra page views, i’ll believe him. One more click for me is not such a big problem. However, the “if” of the first sentence, is still a big, capital, IF..
I swear it’s not for the Ads. Compared to real ad drive sites, my traffic is very low and even if it doubled due to this it’d not matter in the larger scheme of things.
This is another vote for the full feed. I read exclusively via rss, and on my phone (ios/byline/google reader) whilst commuting. I have a restricted data plan and limited reception. Full text feeds allow me to download everything via wifi before I leave in the morning.
I enjoy the site and will probably switch to one of the multiple services that convert a truncated feed to full text. I suspect that, whilst a minor inconvenience to spammers, the only other thing this will do is make your stats more unreliable if others do the same.
It’s a real shame that you are doing this, although it is your content and your decision.
I hear you, another vote for the full text
I always prefer full feed but, having been the victim of RSS scraping myself a number of times (and it seems to be increasing), I can certainly understand the frustration.
I’ve been lucky that I’ve managed to get a number of sites to stop publishing my stuff but the truly automated sites aren’t going to care about takedown notices etc. – they’ll just throw up another site if the current one gets removed.
One vote for full feed
RSS full feed vote, I never click through. Sorry
I’m sorry I’ve no solution to propose for your problem, but I’ll complain anyway (though, you know, you don’t owe me anything, so at the end, you do what you want to).
I’m suscribed to hundreds of RSS feeds ; I don’t read every single post that is written : I read one paragraph, maybe two, then I decide if the post is worth reading or not. With such an amoung of posts to “read”, I’m definitely not happy with having to click, wait for a full website to load, and so on. Then, unless the title itself suffices for me to know I really want to read the post (and this doesn’t happen very often), I’ll probably no try to read the full article.
Best regards
To clarify: by “and this doesn’t happen very often”, I’m talking about the title being enough, not about me wanting to read the post.
Very few of my feeds are truncated, and I tend to skip cilcking through on them quite often (it sets a higher quality bar). This is even true on mobile, where the excellent “Reader” app caches and formats the text excellently for full feeds compared to often bad mobile experiences that requires click-through.
In summary: I tend to lean to truncated feeds only when there’s advertising revenue that I find more important that actually getting people to read my content.
Hi John, I’m very much a vote for keeping it as full text. I feel your pain regarding scrapers (we all have the same problem), but changing your publishing practice to the detriment of your readers is not the way to beat them – in my humble opinion.
Full feed or nothing. I rarely have the time to click into blogs that truncate. I hear your concern about ‘feeding the beast’, but you are trying to cure cancer with a toothpick. Your knowledge is more important to solving the problem than your proposed solution 🙂
Thanks
Most of my feeds don’t have full content in it anyway. I don’t care, if the first paragraph or the title is interesting, i will click on it
Reading blogs on my tablet is just so much better with RSS. The text is always the same size! Your feed had the paragraph breaks zapped but I read it anyway. RSS really is a treasure that should be nurtured. People don’t know what they are missing.
Amen to that.
John,
You might consider ensuring your site is “mobile friendly” first, then truncate away. 🙂
(I don’t read your stuff in mobile personally, but if someone clicks to read in mobile, they need your site to be mobile friendly)
Your free content, which I appreciate btw, is good enough to click on the site and read it. From the user’s side, we have Instapaper and Readability to convert the articles and read them cleanly if necessary.
I can’t see how it’s going to solve the problem you’re having though, if “scrapers” want your content, they’ll get it, it’s a very difficult problem to solve.
Hi,I am new one.I found your blog really informative and helpful.Thank you.
http://superioreducationz.com
Education Information
Voting for full feed as well!
Other commenters have made all the points I was going to make. Thanks for sticking with full text RSS for so long. Sorry to see it go. Maybe there’s a way I can get your full posts to read on the train, sure wish I didn’t have to find it though.
it’s really interesting to find out how many people read what I write offline. I had no idea.
I’ll throw a vote for “bring it back to full text” as well. If you can measure the amount of fraud before and after the truncation it would be interesting to see if it improves the issue.
I wish it was measurable. It’s like Whack A Mole.
So always read via the feed, and I will admit now that we’ve got to click through to read the story we will be wanting to see something that tells us why. Headlines work..
However would you consider writing a summary paragraph on your article via RSS so we can see if want to click through ??
I’m not sure I’d write a summary, though it’s a great idea. I barely have time to write anything!
I’m a huge RSS reader. (at least 75% of my web reading is through RSS.) I used to hate truncated feeds. But, like others, I’ve gotten used to them. In fact, when done right, it helps me get through my feeds faster. I have no problem clicking through for a story I’m interested in.
Here’s my advice, though. Title becomes much, much more important. Also, I think it’s important to have a strong first paragraph that does show up in the feed. Enough for me to get a gist of what the article is about and care enough to read the rest.
Yes, the idea that this might change my writing and headline style is kind of a negative for me. Hmmm /
I read almost all posts via RSS, but with +1000 different posts a day, I haven’t the time or energy to click on a link to a full article. Except when the article/post seems to very interesting… So it is even more important that the first sentence and headline is extremely telling and unique. The bots will have the time to follow any link from any good blog anyway, so I don’t think you’ll beat them. I very much hope you’ll start publishing the full posts again!
It is not problem for me as I always click through to your site when I am interested.
Have you considered embedding advertising or similar in the RSS feed? If they are going to syndicate it, they can have your adwords on their blog too.
Alternatively, use google’s verification features to assert ownership over your content. If it’s syndicated, and they publish your verification token… well, google will see you control said content.
For example, with google’s webmaster tools, you can set up URL removal requests.
Removing / from the index of google’s site… would likely be a good way to discourage content theft 🙂
I believe with that you need to set up a meta tag or upload a HTML file; but surely there must be other similar ways you can use your control over content…
Yes I have considered it. And the problem is, we have no idea who you guys are, so we can’t market who you are to advertisers.
I vote for full RSS text. I tend to read your work through Google reader and Newsify on iOS. I basically agree with the points raised by the people who are for full RSS text so I don’t feel the need to reiterate them. I hope you reconsider.
I just shrugged and switched to http://fulltextrssfeed.com/feeds.feedburner.com/JohnBattellesSearchblog
And just like that, the scrapers can follow suit.
John, maybe you can put that link in your headers? It would be a pity for the scrapers to know where to get a full feed but not your readers . . .
That is a good thing to do. Even I have my blog’s RSS cut short. One thing you can do is tynt to enable backlinking from any content scraped websites. There is however a chance to accumulate some bad neighborhood links which you can disavow.
so much work to just write a site.
Indeed.
I’ll add my vote to keep the full feed. Most of the arguments seem to have been made already. I like what Brad says – don’t punish your actual fans for what fraudsters do.
I’m usually subscribed to between 100 and 200 feeds via Google Reader. I read them offline on long bus rides, in a cool little newsreader on my phone that gives me plain black on white fullscreen text in the font of my choice. There’s always more to read than I can finish, so incomplete feeds tend to get skipped.
This is a real shame because I like reading your stuff but I know myself well enough to know that I won’t be clicking through regularly. There have been entire blogs that I’ve stopped reading because I couldn’t read them natively in my Google Reader.
Sorry it’s come to this, John.
I may change it back.
John – I realize you’ve gotten a lot of comments already, but here’s another vote for full feed. Not only is this the only way that I keep up with my favorite content from all of my top sources, but I also judge the type of content by full feed as well. Sometimes I don’t have the time to read through some of your long form stuff, so I can “keep unread” for as long as I need until I get through it. And I DO get through it. I’ve probably never quoted a single author more than you. Short form stuff I can get through right away. Truncated means I need to click through on every item to judge whether or not I have the time to read through the article. While the extra step seems trivial, it isn’t. RSS is about speedy browsing, and this breaks that cycle.
ANNNNNNND Unsubscribe.
I like your writing, but honestly, it’s not good enough to go through the trouble of clicking through. There are other blogs out there that have comparable content. Sorry!