Syndic8 2004 Retrospective

2004 has been a really good year for Syndic8, and I wanted to take the time to enumerate all that’s happened. I have spent a lot of time coding and answering email, and not enough time talking about what I’ve built and what I’ve learned. I will try to do better in 2005. Anyway, here’s a quick summary of what’s transpired.

Feeds – We had 42,699 feeds at the beginning of the year, and added 220,372 during the year, for a grand total of 263,043. Making room for all of these feeds, and processing them efficiently, was a challenge earlier in the year, but the site is running smoothly now. I know that we can do better with Reviewing, and I’ll get to that in a bit.

Users – We had 5,196 users at the beginning of the year, and added 4,948 during the year, for a grand total of 10,144.

Hardware – We continued on our four-processor BocaCom box through the year, adding a 73 GB disk in the spring and another 250GB in the fall, along with an additional gigabyte of RAM, bringing the total disk space to 400 GB and the total RAM to 3 GB.

System Reliability – We had a lot of problems with system reliability earlier in the year. Many mornings I would find that the system had locked up overnight, and that an Apache restart was required. I added some advanced monitoring and automated the restart process, which made things considerably better. There was an issue with a CPU heat sink, which caused a few lockups as well. At mid-year we had a hardware failure on the root drive, and we ended up with a newer (and less reliable) Linux kernel. It took a really long time to diagnose the problem; until I fixed it the system would crash every 12-24 hours. The problem has been fixed and the uptime now stands at 17 days.

Ranking and Ratings – Our Alexa rating moved from around 15,181 up to 7,394. Google records about 4,110 backlinks to the site. Additionally, the number of hits from a Google “inurl search” grew from 691 to 16,500.

New Features – We added a lot of new features and content in 2004:

• The Feed List allows filtering by language.
• Multibyte strings and multiple character sets are now supported throughout the site.
• The Atom syndication format is now supported. Jeremy Zawodny’s blog was the first Atom feed to be submitted.
• Published a document on the Syndic8 Personal List system.
• The Featured Feed system allows feed owners to publicize their feeds on Syndic8 at a low cost. This has worked out well, with an average of 10 feeds featured on any given week.
• DNS hosting moved from a 10 year old 486/66 box in my home, to professional hosting at zoneedit.com (highly recommended).
• Many more news aggregators are now supported through “Add” links on the Feed Info pages.
• The RSS 1.0 **** tag is now parsed and used to set the category of the feed, if present.
• It is now possible to generate an OPML or OCS list restricted by category, as seen on the category browser pages.
• Compressed data is requested when polling for feeds, and the results are tracked as statistics. It would be excellent if more feeds supported compression, given that each Syndic8 poll now pulls down over 2 GB of data, and that the polls run twice per day.
• Syndic8 now receives pings from a variety of sources, and it applies filtering and other checking to distinguish good pings from bad. The pings are displayed on the Recent Pinged Feeds page. Pinging and other real-time processing will be a big focus for Syndic8 in 2005.
• Newly discovered headlines are now published to the RSS Scroller at mod-pubsub.org.
• Syndic8 now tracks feed enclosures, and displays them on a new tab in the Feed Info pages. There are also some enclosure statistics, of course.
• The XML-RPC interface now has a GetFeedsInCategory function.
• The Feed Info pages now include a tab listing recent changes to the feed.
• It is possible to use the XML interface to generate a list of feeds that have enclosures of a given type. This isn’t documented yet; if you need it drop me some email.

You should know that there’s no faceless mega-corporation behind Syndic8. You’ve got me, working on this site in the wee hours of the morning (usually 5:20 to 7:20), and Bill Kearney, who has a lot of other projects going. We have one hosted server, DSL connections in our respective homes, and that’s it. No venture capital, no silent partners, no big pot of VC gold. Having been a part of several venture-backed startups, I’m still impressed with what we’ve managed to do with the resources available to us.

We have found that text link advertising is a great way to support the site; hosting and bandwidth costs are no longer a drain on my credit cards. I appreciate your clicks and your tolerance for these ads.

I should add a big “Thank You” to Bill Kearney, who helps out with the server, with tons of great ideas, and with lots of messages to the mailing list. Thanks, Bill!

Ok, so what didn’t go so well?

A couple of things. System reliability sucked for most of the year, and it took far too long to get that under control. The reviewing system is simply not scaling with load – a lot of people would like to be Reviewers, but the system is hard to use, there’s more manual work required than should be, and the “rules” for reviewing are not really clear. Fixing all of this is going to be a big priority for the first part of 2005.

The site’s user interface is definitely outdated, unattractive, and not all that easy to use. Again, this is going to be a focus for early 2005 — I am going to hire a part-time web developer within the next week or so. The first step will be to make the existing site look better; after that we will look into structural changes and additions.

It would be great if the Personal List feature had some more use. Bill Kearney designed this feature several years ago, and its really, really cool. However, it just doesn’t get enough use.

Other focus areas for 2005 include more real-time processing, better support for and use of enclosures, and a focus on international issues. I am also going to try to be a little bit less of the “behind the scenes” guy, and to participate in the mailing list on a more frequent and regular basis.

I would like to thank all of you for your participation in the site — for your contributions of time, energy, suggestions, and more. I think that we are doing something of real value here — helping the world to be more informed, giving them the ability to have multiple viewpoints on a topic or an event, and perhaps even expanding their mind a bit in the process. That’s a pretty cool thing to be able to do with some code and some XML!

On a personal note, I would like to say that this has definitely been the most exciting project that I’ve ever had the privilege to participate in, and that the fun is just beginning as far as I am concerned. I recently convinced my 19 year old son Stephen to become the first “second generation” Syndic8 member. He’s contributed a bunch of feeds, and he’ll do some reviewing next month.

Thanks to all of you for your great emails and your support, and here’s to a safe, successful, and syndicated 2005!