My Side-Projects: Let Me Show You Them

I wrote about the value of side-projects back in February, in part because I had a total lack of them at the time. Shortly thereafter, that changed. Rather than picking up the side-projects I mentioned in that post, I've ended up spending a few spare hours on other things entirely

git-wiki

git-wiki is a web-based wiki whose data store is a Git repository of plain text files. I found the original implementation by Simon Rozet while browsing recent commits on GitHub and got inspired.

I kicked some patches around with Jesse Andrews, and pretty soon our version was a good ways beyond where Simon started. I've gotten permission from Simon to fork the project under a new (ideally, sexier) name, but nothing has come to me just yet. Others are creating their own forks of git-wiki and contributing improvements, expect it to be a pretty solid wiki for personal use or small teams fairly soon. If nothing else, it's got a simple, clean design courtesy of Timoni.

Down for everyone or just me?

The other day a friend Twittered "is LiveJournal down for everyone, or just me?" It was about the billionth time I've seen someone on a forum, IRC, IM, or Twitter ask if a site was down. See, sometimes bad things happen to good internet connections, and there's no telling if someone upstream from you biffed the DNS server or if your destination is actually unreachable. downforeveryoneorjustme.com is one trivial answer to this perennial question.

Here's how it works: you type in a domain, we transform what you typed in to something sane, and then we do a HEAD request against "/" for that domain. If the site responds in 4 seconds, it's up. If it doesn't, we report that it "looks down from here". This is, of course, totally cheesy. But it also tells you what you need to know 90% of the time.

Obviously, this is not a pro-grade monitoring solution. This is for quick, simple checks against popular domains. It gives you a quick answer and lets you go on your way, only encumbering you with some tasteful AdWords (not that I see them with my ad-blocker on). If it's not the level of detail you're looking for, there are a ton of tools that do geographically multi-homed monitoring, historical reports, etc.

I've done nothing to promote the site other than Twitter about it and put it on my del.icio.us since bringing it online this past Thursday evening. In that time it's had over 85,000 visits and over 232,000 pageviews according to Google Analytics, which is pretty much insane. It's running off of a tiny Gandi VPS and the code is a couple hundred lines (including templates) of Ruby, powered by the Sinatra micro web framework. I'm running four instances of the application behind Nginx, and save for the occasional slow request it's been standing up pretty well (low load on the box, response times are usually fast). I'll probably switch from serving via Mongrel to Thin fairly soon, once traffic has calmed down a bit.

Where Is Britt?

The future of geolocation is here.

What I've Learned

While downforeveryoneorjustme.com has quickly become popular, it's more responsibility than I really wanted out of a quick hack. If it gives someone a wrong answer, that sucks, but I don't have the time or resources or desire to build the ideal solution. I hope that some big ISP or networking outfit takes the simple design and puts it in front of a proper setup. In the meantime, it's novel to be making a few bucks off of AdWords, which I've never tried before.

Working on git-wiki is, like most developer-oriented projects I've spent time on, much more rewarding. Other developers are the best customers. But I'm actually spending more time hacking on it than, say, putting stuff in my personal wiki. I have the feeling that the quote-management application I've been dreaming of would be the same sort of affair: put hours into building it, use it for mere minutes.

Honestly, what I've learned is that I need a hobby that isn't coding. I haven't understood why so many programmers get involved with gaming when coding is, for me, really enjoyable. But it's hard to code something worthwhile and exciting that doesn't leave you beholden to supporting that code and its users. Really good personal projects quickly become jobs. Sometimes that's what you want, but one job is enough for me at this point in time.

Ultimately, I've ended up with a quandry: how do you keep side-projects manageable?

On Side-Projects

Side-projects are important to every programmer I admire. Google's much-publicized 20% time is a corporate codification of the importance of side-projects; even a company that's worth billions knows that you can't keep good people working on the same thing all the time.

Side-Projects and You

There are lots of good reasons to always have some side-projects going:

• Projects keep you learning. New programming languages, new technologies, new ideas.
• Projects are mentally refreshing. Taking a step away from the problems you normally deal with is relaxing, and can lend a new perspective on how you work.
• Projects can be fun. Fun is fun.
• Projects can be profitable. Little ideas can turn into products and services that people want to pay for (or at least click ads on). Unusual ideas can forge new markets.
• Projects make you friends. Getting involved with a community is rewarding personally and professionally.

All that may seem obvious if you've made a habit of having side-projects, but I'm always surprised by how many people don't bother. But then, you don't hear about those people because they don't blog, attend meetups and conferences, or generally do things that would make them visible. Side-projects are a sign that you care. They're something we ask about when interviewing at Twitter.

Side-Projects and Me

I've had two side-projects on my to-do list for ages. The first and oldest is Peeramour, which is more or less a dating site for bloggers that emphasizes one's existing online presence rather than requiring yet another half-baked profile. I've been wanting to build this for about the last three years. Peeramour was conceived to scratch a personal itch, but I think there's a business opportunity there too. It's also something that I think would make people happy, and I feel an obligation to give something back to the web community that's been so good to me. Peeramour isn't hard to build, but I want to build it right, both aesthetically and technically.

The second project I've wanted to work on is Quotidian, a Mac OS X (Cocoa) application with which you can store, tag, and organize your favorite quotations. I've also considered building a web compliment to Quotidian that would allow you to share your favorite quotes with friends and interested strangers, but Trsly pretty much gets this job done to my satisfaction. My goal for Quotidian is mostly educational: I use a Mac every day, but I have a relatively limited sense of how I'd build a native Mac tool for myself to use. I'm also concerned that too many of my eggs are in the web-programming basket. Web apps may be vogue, but desktop application programming isn't going to disappear any time soon. It's tough to be a skilled generalist, though, and while I've learned a bunch of theory about how to write Mac software, I haven't had time to get into the nitty-gritty with this project. Once again, the difference is between doing it and doing it right, and the latter requires a ton of knowledge about a development platform with a nearly 20-year heritage.

One of my old side-projects, acts_as_sanitized, has been forked and surpassed (with my hearty blessing) by xss_terminate, written by Luke Francl, who's blogged about it here. acts_as_sanitized was released just before I got swamped by work on Twitter, and I owe Luke for making it something useful again. It's a lesson in the value of open-sourcing, and it leads me to what follows.

Side-Projects and Twitter

Working at Twitter is more than a full-time job. As I mentioned in a previous post, we're still a very small technical team (presently five people writing code and two looking after servers). There's always something work-related I could/should be working on, which means that there's basically no room in my life for guilt-free side-projects. No surprise, right? We're a startup.

One of my goals is that Twitter gets big enough that we have room for side-projects. Right now it just doesn't make business sense. We barely have time to open-source projects like Starling that can benefit from the community's support, much less to code up our own off-the-wall ideas. Compared to our peers in the Bay Area Ruby community we open-source a pathetic amount of code, and I'm eager for that to change. Part of making that happen is approaching our internal goals with the idea that the solutions need to be generic enough that they can be readily opened-up to outside contribution.

The people I'm really excited about working with are all big open-source contributors, and I don't think I'm alone in that. As part of scouting for talent becomes evaluating open-source work, it's going to become a standard part of every good company's growth to standardize policies around open-source contribution and side-projects. After all, Twitter started out as a side-project, which pretty much says it all.

Where Did That Code Go?

If you were using acts_as_sanitized for (rudimentary) scrubbing of your Rails application's data, you'll now find it at http://actsassanitized.devjavu.com. If you previously had issues or patches with the plugin, please share them there.

If you're looking for the twitter_monitor.rb script that displays your Twitter updates via Growl you'll now find it at http://static.al3x.net/twitter_monitor.rb. Now that Twitterrific does Growl, though, I don't know why you'd use it.

I'm also sharing the code I used to migrate my blog from a custom Rails application to Blogger. You'll find that at http://static.al3x.net/blogger_import.py. It requires SQL Alchemy and some other libraries. Much of the code is ganked from the Blogger API Python developer's guide, but it's got a few tricks that someone might find useful.

Lastly, you'll find my old TextPattern export scripts at http://static.al3x.net/export_moveabletype.php and http://static.al3x.net/txp_update_comment_count.php. I hope you don't need them, though.

Finale

The last Serious DJs mix (more or less) is live. Sniff.

Because We Can, A New Serious DJs Mix

It’s right hurr. Insane, but no apologies.

Peeramour Is Live

The site I announced a while back, Peeramour, is now live and accepting testers.

I wanted to have the whole thing up and running on the 14th, but I also want to do the proper launch right. The testing phase won’t last forever, just long enough to get feedback from a group of motivated early adopters.

Happy Valentine’s day.

Updates to Acts As Sanitized Coming

It’s been nice to see that there’s some interest in Acts As Sanitized.

John Nunemaker referred me to the White List plugin by Rick Olsen, which seeks to solve a similar problem but for views, not models. Rick himself then mentioned that the sanitize method passes only a fraction of the test cases that he’s adapted from Rsnake’s XSS Cheat Sheet, something I’m well aware of.

Over the next couple days I’ll be expanding my test cases to encompass the XSS Cheat Sheet. Beyond that, I’ll be providing an enhanced filter along the lines of Rick’s solution. Rick has clearly done the difficult legwork here; the rest is just a matter of approach and implementation details.

Any other feature requests while I’m at it?

Announcing Acts As Sanitized

When I was doing my talk on Rails security at RailsConf Europe I joked about delivering a magical plugin, acts_as_impenetrable, that solved all of your security needs. There’s still no magic bullet for security, but I’d like to contribute a smidge of code that brings us a step closer.

Rails has the ability to mitigate cross-site scripting attacks in the form of its ActionView::Helpers::TextHelpers#sanitize method. This method won’t catch every clever XSS out there, but it sure helps. Sadly, sanitize is unavailable by default from your models and controllers; the expected usage pattern is that you’ll handle sanitization in your view, ex: <%= sanitize(@story.title) %> or <%=h @story.body %>.

I don’t think that’s especially, uh, agile (or whatever). Neither is importing those TextHelpers into your controller and reassigning your models’ various attributes to sanitized versions of themselves before saving. It’s tedious, it’s repetitive, and it opens the door to careless errors. Are you sure that you know every field that gets displayed in your views, even after all those revisions and migrations?

My solution to this is a plugin: acts_as_sanitized. You use it like so:

  class Comment < ActiveRecord::Base
    acts_as_sanitized
  end

That’s it. The plugin figures out which fields in your model are able to be sanitized at application runtime. If you’re not comfortable with that for some strange reason, you can also specify which fields you’d like sanitized. You can even tell it to strip all tags, not just the ones that the sanitize method in Rails handles (script and form). Plus no more monkeying around in your controllers, no more wasteful filtering in your views.

Install like so from the root directory of your Rails app:

  script/plugin install http://code.al3x.net/svn/acts_as_sanitized/

Documentation is included in the README file, and there’s a decent suite of tests included.

There isn’t much to this plugin as it stands, but as XSS attacks become ever-more complicated, I’m hoping that this plugin evolves to detect and combat them. If you see attacks that are sneaking by sanitize and strip_tags, let me know! Bugs and patches are more than welcome.

Teaser

Peeramour is coming next month.

In Case You Need My Code Scraps

Since rejiggering my domain recently, I’d been running code.al3x.net on Collaboa, a Rails-based Trac clone. However, the Collaboa project doesn’t seem to be going anywhere fast, despite an announcement of renewed activity. In response, a frustrated Collaboa user announced a fork called Retrospectiva, which I switched to a couple days ago.

Retrospectiva is quite full-featured for a beta release. It already goes a ways beyond Collaboa, and even Trac in some respects (the blog extension, for example). Retrospectiva isn’t production-ready, though, so for the time being code.al3x.net is simply serving up plain ol’ Apache-powered Subversion.

I’m so taken with Nginx that I decided to keep it as my public-facing HTTP server. I just run an Apache virtual host for Subversion on a high port and and tell Nginx to proxy to it like so:

server {
      listen 80;
      server_name code.al3x.net;

      location / {
        rewrite ^/ /svn/  permanent;
      }

      location /svn/ {
          proxy_pass http://127.0.0.1:8010;
      }
}

The rewrite bounce on / is thanks to httpd stupidness: you can’t serve up a repository index – that plain “Collection of repositories” page you’ve probably seen around – from Location / as best I can tell. Various attempts to Alias or Redirect in my Apache configuration didn’t prove fruitful, but Nginx works like a champ.

At some point I’ll move back to Retrospectiva or Trac or the like, but there’s simply not enough code there right now to worry about it.

The Great Typo Escape

I guess a lot of people are sick of Typo. I’ve had a number of emails in the last couple months asking me for the script I used to get my entries and comments out of Typo and into TextPattern. Poor souls.

Until recently most of my at-home geek time was devoted to Kenshoto, but now I’m freed up to work on lil’ side projects like this. So, without further ado, here’s how to get the heck outta Typo:

Part 1: Exporting to Moveable Type Format

1. Download mt_export.rb (ed: sorry, this file was lost ages ago!).


2. Copy it to the scripts directory of your Typo installation.


3. From a command line, change to the scripts directory and run my export script, piping the output to a new file: ruby mt_export.rb > import.txt


4. Set the new import.txt file aside somewhere safe.


5. Install TextPattern.

Part 2: Importing

1. Once you have TextPattern installed, bring up the admin section and open the import tab.


2. Read the help text that comes up when you click on the lil’ question marks.


3. Move import.txt to the directory specified in the help text: /textpattern/include/import


4. Do a “Moveable Type (File)” import. It went well, right?

Part 3: Cleaning Up

(TextPattern is awesome but it’s not so smart about importing. It’ll suck in new comments just fine but it won’t update its internal count of how many comments each post has recieved. When you look at your blog, it’ll look like there’s no comments. This fixes that.)

1. Download txp_update_comment_count.php to the server you’re working on.


2. Edit the commented variables to contain your database connection settings.


3. Run the damn thang: php txp_update_comment_count.php


4. If you see it figuring out how many comments each post has associated with it, you’re set.

Hope that puts you on the path to blog unsuckage. Lemme know if something breaks!