StackExchange Plugin for ThinkUp 2

The new Magento StackExchange went public this week, and so far the beta is coming along well. We’re a little short on some benchmarks, but with only a few days of public beta those numbers are sure to climb up. The only thing that has me slightly worried is the ask/answer ratio falling from a glut of questions, so be sure you’re checking regularly for things you can answer.

Stack Overflow, the parent of all StackExchange sites, came along at just the right time for me and my career, and the occasion of it meeting up with Magento, the other technology that has significantly shaped my work life, gave me the idea to dust off on old side-project.

I’m happy to announce the release of an alpha StackExchange plugin for the ThinkUp social analytics platform.

What’s ThinkUp

ThinkUp is an open source self-hosted web application, written in PHP, which downloads and archives content from your social networks and then tells you interesting facts about said content. There’s charts, graphs, and numbers. Very appealing to a certain type of mind.

I’ve mainly used ThinkUp to keep a local archive of tweets, (because someday my “Acting like an Enthused Knowledge Worker” schtick will be worth gold), and I’ve long wanted to do the same with my StackExchange content. While I found a few mentions of started projects, from what I’m able to tell there’s no StackExchange plugin available.

Since my whole racket is building things that should exist but don’t, this seemed like the perfect side project.

StackExchange and ThinkUp: A Difficult Match

After getting into the project, I quickly developed a few theories on why a plugin has never emerged.

ThinkUp’s data model is oriented towards the small bursts of text that we’ve — evolved? — into on social networks. StackExchange’s more complicated data model (with questions, comments, and answers, each linked) doesn’t fit neatly into ThinkUp’s “single table of posts” world view.

Also, ThinkUp is all about tracking your social interactions. StackExchange’s model is about stripping distracting social interactions away from a community to ensure the focus is on questions and answers which can be easily found later. Again, we have a situation where ThinkUp’s mechanics don’t line up neatly with StackExchange’s model.

Finally, there’s StackExchange itself, which is an overarching network with multiple sites. You have a StackExchange account, and then individual accounts with each site. While developing a plugin for a single StackExchange site is relatively straight forward, developing one that can account for multiple sites, sites a user may or may not use, turns out to have lots of tricky edge cases. Also, just like with posts, this doesn’t fit neatly into a single table of users model.

What’s in the Alpha

The alpha plugin addresses some of the issues above and lays groundwork for the others, but isn’t quite ready for non-technical users. Right now we have a crawler that works. It will go out and grab all your Stack Exchange content, and create ThinkUp user and post objects for the grabbed content. Additionally, SQL tables for each individual site are created that mirror the StackExchange API structure. These tables can be queried with ThinkUp DAO objects.

What’s needed is some testing and tuning of the crawler (I’ve only used my own data), as well as ideas for insight plugins. In other words, now that you have a local copy of your StackExchange data, what do you want to do with it?

I’ve posted a fork of ThinkUp to GitHub that includes the StackExchange plugin, and a few wiki articles with some background on the choices I’ve made.

If you’re a ThinkUp user (or want to be), the biggest thing needed right now are people testing the plugin out, and reporting back on bugs they run into. To help with this I’ve created a screencast that will give you an idea of what to expect on first run.

The second biggest thing is ideas for insight plugins. In other words, what do you want to know about your Stack Exchange content.

If you want to go above and beyond and help develop the plugin, then take a look at the issues lists and get in touch. I’ll also be posting to the ThinkUp developers list in the next few days, so you should join up there too.

My goal is to get this in good enough shape that the core ThinkUp team will accept it as the official StackExchange plugin — but even if that doesn’t happen having a tool to save our posts from eventual Internet Rot™ seems like a worthwhile thing in and of itself.

Finally, if you’re interested in ThinkUp’s PHP design patterns I’m planning a few articles/tutorials on its MVC architecture. After nearly half a decade of working with Magento it’s been interesting to drop into a more lightweight PHP framework. Comparing the tradeoffs involved in the two approaches seems like it might bear interesting fruit, so stay tuned for that!