Big Fat Feeds!

This evening, while doing some technical review on a book draft, I happened to watch the tail end of the Syndic8 poll. The poll consists of one master process and 20 slave processes, all hooked together using a message queue.

18 of the 20 slaves had finished, and I was expecting the last two to wrap up at any second. However, the final 2 slaves ran, and, ran, and ran. Each slave appeared to be stuck on a single feed for a very long time.

After some investigation I f0und out that one feed was 9,576,634 bytes in length and the other was a whopping 15,199,291 bytes. Each of these feeds contained over 9,000 individual tags. Feeds of this size really tax my RSS parser and my headline analysis algorithms.

Is this incredible or what?