Jamie Thingelstad's personal website

FriendFeed Duplication via Mashing

I’ve been working with financial news for years now and there is a really common problem in financial news, deduplication. The issue is when several news outlets will cover the same story or even redistribute another sources story, creating duplicates that readers find annoying. As the web becomes more mashable, I saw the first sign of this same problem on FriendFeed today.

One of the people I follow had both Twitter and Tumblr feeding into his FriendFeed. But, he also had Twitter feeding Tumblr. In flowed the dups!


He removed the Twitter to Tumblr feed to stop this, but it was a glimpse of a feature that FriendFeed is going to need to consider. In this instance, deduplication would be pretty easy since the content is exactly the same.

1 Comment

  1. yes, it jumped out to me fairly quickly after paying attention to my own feed(s) aggregation, that my tweets were duping because Tumblr was echoing them. And it only took that long because I pay much closer attention to the feeds of others than my own.

    For now, I think it is going to be on each individual contributor to self-monitor as to whether they are having the same thing happen.

    De-duplication is simple when your goal is to remove exact matches. But more challenging when you are doing it with fuzzy logic or “like” matching. De-duping in this case may introduce additional challenges.

    For example, will Tumblr or any other service come out with features to wrap around an imported Tweet where I need to have them fed in? And if so, do I not take advantage to avoid the dupe? Or worse yet, does FriendFeed make the decision to remove the echo at their application level thereby removing any chance of the new feature showing itself inside the FriendFeed environment?

Comments are closed.

© 2016 thingelstad

Theme by Anders NorenUp ↑