Thanks for the comments, everyone. Due to the traffic from Dave Winer, I got several interesting suggestions and we(my team) have had some discussions on how to proceed with this. These are some of the ideas we've been toying with for the last 2 days - none of them are final, of course.
Only Podcasts (enclosures)
Bittorrent doesn't work very well for small files which is what most RSS feeds are (< 200K). Couple this with the fact that only a few blogs get heavy traffic, there's not much use for creating torrent files for everyone. In fact, this would just slow things down as downloading small files over Bittorrent is slow. Right now, we're thinking of making a change to the way we handle feeds- instead of creating a torrent for the entire feed, we create a torrent only for the enclosures (the podcasts, videos,etc).
However, there is a problem with this approach. When some small site suddenly gets a heavy amount of traffic (from a Slashdotting or Scobelizing or whatever), we really can't help with their RSS bandwidth problem.
The update issue
Frankly, we have no idea right now as to how we'll handle blog updates. Whenever someone updates their blog by either posting a new entry or modifying an existing one, we need to detect the change and recreate a torrent file (with the hash for the new piece). We're considering a few ideas as to how to actually detect this but none of them seem elegant. We can either
(a) Ping the server periodically. Doesn't seem a very scalable solution when you have tons of blogs updating constantly
(b) Have the blog ping us (the Smoke server). This goes against our tenet of Smoke just working without any modification to current blogging platforms
(c) Keep track of pings to weblogs.com. This may work - but I have to do some more thinking on this.
Do note that if we go for the only podcast approach, updates to blogs won't cause a problem. The first time a request comes in for a file,we can create a torrent for it.
The big bad server
The question that most people have with our design is this - how do you scale such an app when only one big server in the middle is going to handle all those blogs? The initial idea was to have a bunch of 'known' servers (for e.g, smoke.weblogs.com,smoke.bloglines.com,etc) where the client can pick one randomly from the list. Since the servers don't need to stay in synchronize (that happens automatically), this could possibly work.
If we drop the idea of not making changes to blogging platforms, this could be a lot easier for us as each blog server could act as a smoke server and run a Bittorrent tracker. I suspect that this would happen eventually with blogging platforms anyway though it may not be our Smoke idea that actually causes it.
The Hash
In Bittorrent, a hash is made of the individual pieces by the guy uploading it. In our architecture, since there is no one authoritative uploader, who calculates the hash? If we have the first client do it, we face a problem where the first client could report an incorrect hash, thereby spoiling the download for everyone else. If the Smoke server does it, it would have to download the podcast itself first, which is something we didn't want.
None of these problems are show-stoppers..except the next one.
Not a new idea
Gary Lerhaupt, of Prodigem.com, was kind enough to contact me and offer me an account. The more I read about how his site works(http://www.torrentocracy.com/prodigem/about.php), the more doubts I get on whether I'm just doing something Gary has already figured out. In that case, there really is no point in re-inventing the wheel as Prodigem.com seems to be a pretty good service.
After seeing this Wired article (http://www.wired.com/news/technology/0,1282,66123,00.html), I find it hard to believe that this country is supposed to be in the midst of an information technology revolution.
... Bajaj's lawyers applied for bail equipped with a printout of the portal's terms and conditions, which included users of Baazee vouching that their items were legal. The engineering student had accepted the terms and conditions by pressing the Accept button. But the court rejected the bail application, according to an executive of the portal, "stating that since there was no ink-based signature, it is void." ...
And all this because two teenagers fooled around.
Frankly, don't all these people(the cops/the media/the people who write angry letters to newspapers about the degradation of society) have anything better to do? As an Indian, I find it hard to swallow that the number 1 priority for our policemen seems to be the task of hunting down of everyone who has seen a piece of porn. And in the process, they wind up arresting the CEO of one India's most successful e-commerce companies(www.bazee.com) .
This bothers me deeply.There is a huge 'digital divide' between the people who run the country and the people who run the computers and I don't see how the situation is going to improve anytime soon.
Make sure you see Vinod's hilarious post on 2004 which covers the CEO arrest as well.
Sorry for the non-techie posts, folks. From now on, its going to be rants about Bittorrent and .NET rather than cops and porn :-)
A comment on Slashdot from the Bill Gates/Mediacenter thread (http://games.slashdot.org/comments.pl?sid=135132&cid=11276035)
.. It's out current capitalist system that needs to be rethinked. Free market capitalism should only apply to small businesses. All corporations should be subject to heavy governmental regulations (that are actually enforced.) Personal wealth should also have limits.
Screw the rich. ..
Where's John Galt when you need him?