ASCII Art and Syndic8 Ping Processing

I was doing some file cleanup in the Syndic8 source tree and happened upon this lovely piece of ASCII Art:

Although this might look like it was tedious and time-consuming to draw, it wasn’t. I used Emacs Picture Mode and it probably took me 20 minutes or so.

If you look closely, you can see that this diagram documents Syndic8′s ping processing system. Right now that system processes 20 to 40 pings per second, 70 million pings in the month of October! The secret to doing this is to do as little work as possible in a synchronous fashion within the web server. Instead, the responder code simply writes the ping data to an in-memory message queue, where it is then processed in an asynchronous fashion by the ping whip process. The whip does some preparatory work to debounce the pings (so that a rapid fire series of pings are collapsed into one) and then defers further processing to an array of slave processes. The slave processes actually fetch the new XML and compare it to the version already on hand, detecting new items in the feed. The newest items are dumped into some in-memory FIFOs, where they show up on the News page. The individual pings show up on the Pings page.