Scala lift-off: Martin Odersky Keynote

I'm at the Scala lift-off today in San Francisco, an unconference organized by Dave Pollak, author of the lift web framework and frequent hanger-out at the Twitter office (or Twoffice if you will).

Martin Odersky, author of the Scala programming language, is kicking off the day. Martin teaches at EPFL in Switzerland, and has a background in functional languages. Before Scala, he's had a hand in several functional languages on the JVM, as well as rewriting the Java compiler for performance and correctness. The Scala effort started in 2002, but the community has really been growing since 2006 when a redesigned implementation of the language was released. This week has seen quite a bit of Scala buzz at JavaOne, including an endorsement of Scala as the premier alternative JVM language by Java author James Gosling.

Odersky opens with an argument for a scalable language: one that's suitable for small and large programs, from parsing to scripting, from extension scripts to heavy lifting libraries, one that embraces Domain-Specific Languages (DSLs) rather than a reliance on external languages.

In one sense, Odersky suggests, Scala is a scripting language. It's possible to simply type code into an interpreter and have it evaluated. There can be very little boilerplate code in Scala, and most types can be inferred.

In another sense, Scala is the "Java of the future." Scala boasts most every language feature Java currently has, and many that are still pending or under debate in the Java community, such as closures, traits, and pattern matching. Scala complies down to .class files, and Odersky jokes that he should know, given that he's written two versions of the shipping Sun Java compiler. Scala's performance and operational profile is thus the same as Java's, give or take special cases and the use of alternative libraries.

In still another sense, Scala is a "composition language", one with a new approach to modular systems. In Scala, components are classes or traits that can be composed via mixins. Abstraction is accomplished through abstract members, parameters, or self types; essentially, both functional and OOP metaphors for abstraction are available.

For all that, Scala is trying to avoid becoming a "kitchen-sink language". Scala is comparable in feature count to Java, but has far fewer features than C++ or C#. Scala removes the following from the Java feature set: static members, primitive types, break, continue, special treatment of interfaces, wildcards, raw types, and enums. Scala "concentrates on the glue", not on features. Leveraging abstraction, users can implement the features they need in the language, rather than the language itself supporting every possible use case. The same set of constructs are intended to support both small and large programs.

Scala's lightweight syntax comes from a number of features: semicolon inference, type inference, lightweight classes, extensible APIs, and closures as control abstractions. On average, Odersky predicts a 2x reduction in lines of code between a Java program and a Scala port.

Odersky suggests that "scalability demands extensibility." As an example, he notes all the different data types traditionally used for numerics in programming languages: int, long, float, double, complex, rational, interval, polynomial, etc. While a language could support every possible numeric data type, Scala's extensibility strategy encourages abstractions and additions to existing types and methods to provide new numeric data types seamlessly. For example, it's possible to define + and - methods, essential to dropping in YourFancyNumeric class without pain.

As another example of Scala's extensibility, Oderksy demonstrates adding new control structures. A using method executes a block of operations on a resource, then closes that resource. It's a simple pattern with a short implementation, but the terse code is expressive and requires intimate understanding of Scala's type system to understand what's going on.

Scala makes its scalability claims on the tight integration of functional and object-oriented programming. One of the important concepts here is that functions themselves are objects in Scala. Additionally, Scala offers Erlang-style Actors for high-concurrency programs, contributing to "scalability" in the operational/performance sense.

Like Erlang, Scala provides Actors with mailboxes for messages passed to them. Conventionally, a programmer has her Actors process received messages with a pattern match. Odersky notes that the Actor features of Scala are libraries, not core language features, and argues that this is a testament to the strength of libraries and an extensible approach to language design. Actors in Scala are implemented atop the FJ Framework; the event-driven flavor of Actors (using react) scale up to hundreds of thousands of processes in tests.

The next problem to be solved for Scala is maintaining expressiveness while keeping programs safe and verifiable. A project on "pluggable type systems" is underway, but is unlikely to yield changes to Scala for several years.

Odersky notes a number of companies working in Scala: Google, Buy-a-Feature, Gump-It, Joberator, EDP Training, Reaktor Innovations, Sygneca. Scala tools are improving, from the language's own complier, background complier, and interactive shell to testing frameworks and plugins for all the major Java IDEs. Rounding out, the "official" Scala book will be coming out soon.

Question time!

Are Scala classes open? No, because that defeats static typing. But: mixins, etc.

Does calling into Scala code from Java necessitate reflection? [Answer not captured, sorry.]

How stable is the language and what's the path for future features? We've agreed to slow the development of the language. There will be a standard distribution and an experimental branch.

Is there a project to add Scala support to the Java compiler so you can mix the two languages in the same project/file? Yes! Eclipse has this working to some degree, other projects are underway.

Because of the unconference format, more questions will be asked of Martin throughout the day and the Q&A session was cut short.

al3x at Web 2.0

When rounding up my schedule for the next couple of months, I forgot to mention that I'll be speaking at O'Reilly's Web 2.0 Expo right here in San Francisco in late April.

On Wednesday, April 23rd, Michal Migurski (of Stamen) and I will be presenting a talk called Design Your API: Learnings from Twitter and Stamen. It's kind of a cool session, we think: Mike talks about consuming APIs from a developer/design perspective, and I talk about creating and maintaining APIs from a provider perspective. There's going to be war stories and pretty graphs. I'm excited.

Funnily enough, there's a panel at SXSW called Building Developer-Friendly Web Service APIs that's on the same tip. With APIs continuing to crop up left and right, it's no surprise that people want to nosh on some best practices. Michal and I intend to deliver, so join us in San Francisco!

al3x in Austin, al3x in NYC

...in that order!

From Friday, March 7th to Tuesday, March 11th I'll be in toasty Austin, Texas for SXSW Interactive 2008. I'm not speaking at SXSW itself, but my colleague Blaine Cook and I will be giving an informal presentation on where Twitter is going at BarCamp Austin III, probably on Saturday. Mostly, I'm going to SXSW to party and bullshit. Those seem to be the primary activities given my experience last year, so this year I'm coming prepared.

In April I'll be speaking at GoRuCo 2008, the Gotham Ruby Conference. It's a one-day conference on Saturday, April 26th, and I'm one of just six speakers (eep!). I'll be talking about Twitter from a code-first perspective. We've done our share of conference sessions in 2007, but I'm hoping to go a bit more in-depth with this one, particularly now that we're open-sourcing bits and pieces of our stack.

You can find information about my travels on Dopplr, which is now open to the public and a joy to use. I put my trips on Dopplr and on this blog because I'm eager to meet you. If you'd like to meet up in Austin or NYC to talk about Twitter, Ruby, social software, bitchin' records, whatever, drop a line!

RubyConf 2007, Day 3: Sploitin' with Ruby

Speaker Aaron Bedra, who works for startup gonowdo, is closing out the day with a talk about how Ruby has "infiltrated the security community."

The main thrust of the talk is introducing metasploit, a collection of information security tools for exploitation (attacking and owning systems). The tool has been ported from Perl to Ruby in its most recent version.

Metasploit has top-level concepts of exploits and their auxiliaries. Exploits depend on payloads, encoders, and nops (no operation instructions). Exploits are what trigger a vulnerability, and payloads determine what happens (for example, getting an administrative shell back from the target machine). Encoders transform payloads, avoiding restricted characters and applying intrusion detection evasion techniques.

A variety of payloads are available for different platforms. Bedra describes Metasploit's meterpreter as "the coolest and best use of Ruby in the application." It's an enhanced version of irb that exposes a DSL for exploitation. To delivery these payloads, Metasploit has three interfaces: a console, a Rails-powered web interface, and a GTK GUI that's still experimental.

Before Bedra can proceed with a demo, a couple of skeptical questions from audience interrupt him. One questioner was unsure of the benefit of being shown how to attack systems, and the other wanted more information on defending his web applications. A general discussion of security disclosure and the benefits of security research clears up some misconceptions, and the demo proceeds.

The target machine is a virtual machine running an old copy of Windows 2000. Bedra selects an exploit targeting a vulnerability in the WINS nameserver, and proceeds to demonstrate that the system is up and talking to the network. The meterpreter payload is selected, the host set, and the system is exploited. Everything is working flawlessly so far.

Bedra then demonstrates increasing privileges, listing processes, and retrieving password hashes. He then jumps to irb and manipulates various properties of the system from familiar Ruby syntax. It's noted that the meterpreter DLL is a mere 80k, and lives solely in memory unless otherwise specified. Spawning calc.exe and migrating to its privileged process is shown, as is uploading remote files.

Next, the web interface is demoed from a virtual instance of Backtrack, described as the evolution of the WACKS and Auditor LiveCDs. The process of exploiting from the web interface is quite literally point-and-click. The payload this time around contains VNC, allowing complete graphical control of the target system. Always impressive.

RubyConf 2007, Day 3: Solr-Ruby

Solr, the Apache foundation's open source search engine, has been good to me. Twitter's people search feature (the "find folks!" box you'll see around the site) is powered by Solr, and it's been a rock-solid piece of our infrastructure. Speaker Erik Hatcher is a committer on Solr and the search library that powers it, Lucene.

Solr boasts features like replication, faceting , highlighting, spell checking, an administrative interface, and more. Big names like CNET, the Internet Archive, Netflix, Digg, and AOL are making use of the project. The Lucene library is in use at Technorati, IBM, Monster, Wikipedia, and many more big installations. It's safe to say that Solr and Lucene have been battle-tested.

Lucene is an inverted index. You put documents into it, and those documents are composed of fields. Inside those fields, a tokenization process finds terms after applying transformations like downcasing, whitespace removal, word-stemming, and so forth. Those terms are then scored, and those scores are used to figure out which search results are the most relevant.

Solr is a Java web application, and can be deployed on any Java application server (Jetty, by default). Talking to Solr just requires speaking HTTP: POST to add or update a document, GET to search, and so forth. While default responses are in XML, it's possible to get other formats like JSON or Ruby hashes back from Solr.

solr-ruby is Hatcher's library for communicating with Solr from Ruby. He demos inserting a number of documents into a local Solr instance, and then retrieving some results in various formats directly from the browser.

Some brief sample ruby-solr code is then shown, with a note that "auto-committing" can be inefficient when loading many documents at one time. Hatcher also notes that Solr adds a primary key semantic where Lucene does not, by default. Lucene also does not have a concept of updating a document, only replacing or adding documents. A patch that provides updating behavior is in progress.

solr-ruby is behind the acts_as_solr Rails plugin (in use on Twitter with some small modifications), Flare, Blacklight (a project by Hatcher at the University of VA), Collex (a database of literature), and more. Hatcher suggests interacting directly with Solr as a means for sorting out bugs in acts_as_solr and other tools that hide away communication with the search engine. Also demonstrated is Luke, the Lucence Index Toolbox, which provides a host of interesting output and modification options for a Lucene index.

As a general solution for mapping Ruby objects to Solr results, Hatcher has developed Mapper. Unfortunately, his demonstration of its code was somewhat side-tracked by an explanation of Lucene's index structure. In a nutshell, it allows mapping Ruby symbols to attributes in Solr documents. The use of Procs is supported, allowing for more dynamic mappings.

In the future, Hatcher is interesting in bringing parts of acts_as_solr into solr-ruby, along with more introspection into Ruby objects and an improved response writer. He's looking for guidance on improving the library's API and DSL, as well as help with documentation. You'll find Solr-related recipes by Hatcher in the upcoming "Rails Recipes" book.

RubyConf 2007, Day 2: Mac OS X Loves Ruby

Apple's latest release of Mac OS X brings with it a number of improvements to and integrations with Ruby. Speaker Laurent Sansonetti - "Ruby Ninja" on his introductory slide - is the man behind much of this work. I'm amped for this talk.

Fundamentally, the distribution of Ruby in Leopard is much improved. It allows for multiple versions of Ruby to be present on the system, each with its own suite of gems. The version of Ruby shipped has support for DTrace, and Ruby has been integrated into Xcode and Interface Builder.

RubyCocoa, a bridge to Cocoa and C APIs, is included and supported by Apple in Leopard. Two RubyCocoa applications, PackRat and LimeChat, are offered as examples of robust applications taking advantage of the bridge. Leopard Server's own Podcast Producer makes use of Rails and RubyCocoa.

RubyCocoa works by creating proxies that forward messages from the Ruby runtime to its target Cocoa/C runtime. RubyCocoa can forward exceptions, convert types, and maintains Objective-C objects for the programmer. The bridge will also manage garbage collection. The messaging convention is to substitute every "." with an underscore in the method name (selector), while optionally omitting the final underscore.

Laurent then demos the creation of a simple RubyCocoa application. He creates a simple window in Interface Builder, containing a text input box and a button labeled "Speech!". Over in Xcode, he whips up some Ruby (complete with decent syntax highlighting). Interface Builder is aware of Ruby classes, outlets, and actions in the file he's editing back in Xcode. Predictably, the application reads aloud whatever in typed into the input box after hitting the button. It works! Applause.

Laurent then demos CocoaRepl, which allows for real-time evaluation of Ruby code. Window creation, Quartz Composer loading, and Ruby threads. The audience groans with geek lust as a window containing a Quartz composition is scaled while its opacity is modified, all code being executed from CocoaRepl.

A final demo: inject.rb, which allows a Ruby interpreter to be injected into the process context of a running Cocoa application. Laurent messes around with an instance of TextEdit from irb, grabbing properties about the window, changing the window's title and contents. More applause. Then, an instance of TextEdit whose input area is, itself, turned into an instance of irb. Fun.

Objective-C's APIs are described as "introspectible", while standard C APIs are a bit more difficult to interact with. Apple's new BridgeSupport provides XML metadata that covers static APIs, as well as a metadata generator so developers can allow bridges to take advantage of their own APIs. BridgeSupport is being used in both Ruby and PyObjC, as well as in Xcode, which is using it for auto-completion. Laurent demos BridgeSupport by generating metadata for libpcap and then using the resulting bridge output to write a basic sniffer in Ruby.

Running low on time, Laurent runs through an overview of the DTrace support in Leopard's Ruby. Probes in Ruby are available for function entry and return, exceptions, garbage collection, object creation and destruction, and more. A simple script is shown that will print function entries and returns, and said script is demoed. Leopard provides a graphical interface to DTrace's programmable probes called Instruments, and this is demoed tracing through an instance of LimeChat.

Wrapping up, Laurent talks about some of the limitations of bridging, including caching of instances, maintaining the object model twice, and converting objects and exceptions. Presently, RubyCocoa can't use Objective-C 2.0's garbage collection. Ruby 1.8 is also not thread safe, which presents some problems (this will be alleviated in Ruby 1.9). Laurent suggests that his team's future goal is to write the Ruby class model on top of the Objective-C runtime, which would solve many of the problems listed above.

Questions:

Can you distribute RubyCocoa apps on OS X versions earlier than Leopard? Yes, apparently there's a script for that.

Possible to see this on the iPhone? Next question. Big laughs.

Is the downloadable version the same as what's in Leopard? Not quite, but it's coming soon.

Building with a Rakefile rather than Xcode? You lose support, but you can try rucola as an alternative.

Do you lose Objective-C semantics in the conversion process? It's a work in progress.

Gem packaging? standaloneify is a work in progress.

Will the GC in Objective-C be committed to GCC? It's a different piece...

Keeping things up to date? You can do a gem update, but only big bugs and security problems will be addressed in Ruby itself. Not going to overwrite 1.8 with 1.9. Shooting for compatibility.

Why RubyCocoa over RubyObjC (my question)? RubyObjC was too small a project, wrong license, not the right features, etc.

RubyConf, Day 1: Summary

I've been advised by Dr. Nic not to apologize for my lack of more detailed entries covering the first day of RubyConf 2007. Suffice to say, the day went by in a blur. Here's a quick summary.

The conference began with a talk by Rails core member Marcel Molina, asking "what makes code beautiful?" Marcel has a background in literature and linguistics, and took an informal, unscientific approach to his inquiry. After some historical background on defintions of beauty, Marcel settled on three principles put forth by religious philosopher Thomas Aquinas: proportion, integrity, and clarity. These ideas were put to the test against some code designed to coerce strings of XML into their appropriate native Ruby objects; two implementations were compared, and neither found sufficiently beautiful. In the end, Molina's argument is that Ruby is optimized for beauty in a way that other languages are not. That said, he drew inspiration from Kent Beck's discussions of optimal Smalltalk patterns, alongside quotes from Niels Bohr and Rousseau. A good warm-up.

Jim Weirich then gave a talk on advanced Ruby class design. This boiled down to a series of patterns he's applied in Builder, Rake, and FlexMock. Techniques ranged from alternate strategies to class inheritance to the use of a BlankSlate object for metaprogramming without dangerous side-effects. Most ambitiously (no pun intended), Weirich attempted to build an ORM syntax in pure Ruby. While he ultimately doesn't feel that it's the best domain to apply his techniques to, he had interesting demonstrations of a node-based technique for "parsing" Ruby with Ruby itself. Engaging and informative.

The afternoon's sessions began for me with Nathaniel Talbott's talk on why the Camping web framework matters. Essentially, Talbott argued that it matters because it's more hackable than Rails. The bulk of his presentation was a general discussion of the Ruby community.

The Japanese authors of the AP4R library then presented on their asynchronous messaging solution. While their slides were extremely readable, the language barrier kept me from fully comprehending the entirety of the talk. Their solution seems like an interesting alternative to robust messaging systems like ActiveMQ, RabbitMQ, JMS, and so forth.

Nathan Sobo impressed the crowd with a talk on his Treetop library, with aims to improve syntactic analysis in Ruby. There are a number of problems for which regular expressions aren't a viable solution, but Treetop handles them nicely. Complex language grammars and more can be parsed, all from a comfortable Ruby DSL. A well-delivered presentation on a great-looking piece of code.

The day's sessions ended for me with Paul Brannan's talk on avoiding pitfalls when writing C extensions to Ruby. I can't say that a majority of the talk actually lodged itself in my brain, but I'm looking forward to a PDF of the presentation. I'm sure it'll be a valuable reference down the road.

After dinner, Ruby's creator got on stage to answer questions from the community. A huge variety of subjects were covered, but the general theme seemed to be that Matz is taking his time with Ruby 1.9, doesn't have any huge features in mind for Ruby 2.0, and isn't interested in optimizing Ruby heavily for performance at this point in time. Some frustrations with the core libraries were expressed, and there doesn't seem to be a clear plan for improving their quality. The community was grateful for Matz's time, but the session didn't impart the sort of excitement I was hoping for.

RubyConf 2007, Day 2: Maximizing Productivity

Eric Hodel, an illustrious Ruby hacker with over 40 projects under his belt, is presenting on how he organizes himself for maximum time spent coding.

Hodel starts by choosing projects carefully and not reinventing the wheel. He claims not be a "coding superhero," and breaks projects down into small, focused pieces. His acronym mantra: YAGNITAGAFI: You Ain't Gonna Need It, They Ain't Gonna Ask For It. It's important to keep to a clear purpose.

Testing is essential to his process, providing a groundwork for new features and ease of contribution. Hodel is a fan of autotest, and prefers to keep his entire development process visible. His desktop is four terminals: code, autotest, and a couple of extra testing windows. Later in the development process, Hodel uses the heckle mutation tester to ensure his test cases are robust. Automation is a general theme.

Hodel suggests knowing your editor inside and out, although he doesn't recommend a particular editor. Pair-programming is another suggested tool, including "ping-pong programming," in which two coders go back and forth writing tests and coding to satisfy those tests.

Documentation strategies include a README with a quick-start guide, a synopsis and feature overview, and a link to the bug tracker. The bug tracker itself should be set up with categories, groups, and email, and disallow anonymous feedback to allow for follow-up. RDoc documentation should include examples.

Hodel gets feedback before releasing. When he's ready to release, he uses the hoe tool to automate every step of the process, including uploading to RubyForge and making blog posts. Hodel suggests minor releases to make bug-hunting simpler, and then taking the time to relax after a release. "It's no fun to code 24/7." Amen.

Building a community around bug tracking is key. Beyond that, it's worth searching beyond the bug tracker to find reports on mailing lists and blogs. Responsible bug submissions are stressed: include the steps to reproduce the bug, your platform, etc. Consistent versioning is suggested. Hodel uses a major.minor.bugfix convention, and bumps the major version when API incompatibilities are introduced. Generally, while he claims to "write a lot of crappy software," he tries not to be concerned about complaints and to encourage contributions.

Hodel finds inspiration in hacking nights and solutions from other communities (ex: the Smalltalk environment's test feedback, which inspired autotest). He also suggests writing "pain-killers": software that solves nagging problems in your work and life.

RubyConf 2007, Day 2: Tightening the Feedback Loop

I missed the sessions after lunch due to a soul food stupor. I regret nothing. Currently up: Phil Hagelberg on "tightening the feedback loop," pragmatic strategies for integrating testing into the development cycle.

Hagelberg begins with a run-through of available Ruby testing tools, including Test::Unit, RSpec, and autotest. A quick poll of the room suggests that a vast majority are using some sort of test automation.

Still, Hagelberg suggests that autotest can be cumbersome and requires too much switching between the editor and terminal. He offers Flymake - which shows compiler warnings in-editor - as an intermediate improvement. A screenshot of a personal experiment with an improved visualization strategy is shown.

Then, a discussion of the "feedback principle": that feedback is the output of a system is being fed back into the system's input. This raises questions of what to measure, how to measure it, and when to feed the output back into the system.

Measuring the accuracy of feedback involves identifying a metric you care about, automating the measurement of that metric, and bringing notification of changes to the developer. Of course, one has to measure "meta-accuracy", ensuring your testing system itself is reliable and trustworthy. Tools like heckle and rcov can provide a baseline of sanity for your test coverage.

Testing the human-percieved maintainability of one's code is another challenge. While flog can offer some help, Hagelberg also suggests being judicious about a focused, clear commit strategy.

Measuring performance in an automated way is worth doing. While using a profiler is a very manual process, you can measure "black-box" response times in automated way, for example with a script that makes requests against your application. The results of these performance measurements should be visible and readily available. Keeping a history of feedback data is suggested.

Hagelberg has a tool called Augment that he then demos. Augment can ANSII-color files to show errors and failures in yellow and red respectively. Output is stored as JSON (for multi-language interchange) in an augment/ directory in your project. There's also an Emacs helper to bring the colorized output into the editor.

RubyConf 2007, Day 2: Rubinius 1.0

The second day of RubyConf began with a grip of talks on alternate Ruby VMs: IronPython (for the .NET platform), JRuby (for Java), and lastly Rubinius (for great justice). Sheepishly, admit that I slept through the first two; neither .NET nor Java are relevant to my interests.

Rubinius, however, is extremely relevant to my interets. Evan Phoenix — runner up to Havoc Pennington in the "best name in Open Source software" competition and the project's founder — let the science drop about the project's pending 1.0 release.

Phoenix has developed a signature presenting style: fast, direct, and punctuated by amusing, colorful slides. He opens with a brief history of Rubinius' founding and its origins at a previous RubyConf. Rubinius has ancestors in Smalltalk's implementation and borrows freely from "crazy compsci papers". At present, the project boasts generational garbage collection, stackless execution, JAR-style packaging, pre-compiled files, and a host of other exciting features.

The most critical part of Rubinius' implementation is that a majority of it is written in Ruby. Rather than being purely C, C#, or Java, Rubinius has only enough C to bootstrap the remainder of the language's core being written in Ruby. Phoenix claims that Rubinius is "Ruby for Ruby programmers."

Currently, Rubinius is beating the standard 1.8.x Ruby implementation in 24 of 31 of the YARV benchmarks. Beyond that, Phoenix isn't interested in talking about performance at this point in time, as the implementation's speed is highly variable at present.

Rubinius isn't done yet. By way of explanation, Phoenix mentions that the project is learning from its mistakes and "optimizing for fun". Community is stressed as an important part of the project's evolution. Rubinius now boasts 57 committers, of whom 17 have committed frequently. The project is eager to hand out commit rights to anyone who submits a single patch — an extremely liberal open source project management style.

Question time.

First up, a hardball: should Rubinius replace MRI (Matz's Ruby interpreter, the standard implementation of the language)? Phoenix suggests that's up to the ecosystem/community to decide what should be the standard.

Can you swap in another kernel? Yes. The project has been designed for modularity.

Are there more things you can override? By way of answer, Phoenix mentions that the entire compiler is written in Ruby, and that it's a "malleable system that can be changed over time." Compiler flags are available to change what decisions the compiler makes.

How small can the kernel be? The long-term goal is to have zero lines of C code by following some of the self-hosting strategies of Squeak's VM.

Concerns about incompatibilities? "We try to be as compatible as we can."

Hard numbers about the project's status and compatibility? "We're down to 500 failing specs. While MRI is the main compatibility target, "we're trying to be as agnostic as possible with the specs." The specs are written in a "mini-RSpec" implementation that's intended to be syntax-compatible with RSpec.

What about the backtrace format? Phoenix shows a trace that's smart about showing the developer where the program actually died, even though the failure involved a method call that bridged two classes. Rubinius will also be able to provide backtraces for segfaults in C extensions.

What about NextObject? Haven't figured out a good way to do this yet.

Continuations? Rubinius has first-class method contexts, and Phoenix demos its use. "A continuation is just the ability to save a point in time and go back to it". So, in Rubinius, you save a MethodContext and go back to it to implement continuations.

Will MethodContext be serializable? Yes, but nobody has written it yet.

Conference Burnout

C4[1] was a good time , albeit lacking last year's first-time jubilance. This year's conference suffered from a few rough edges: the wasted opportunity for a more productive (and less offensive) discussion on Saturday night, overlapping media- and marketing-centric talks, and Fake Steve's sick leave. Still, a good geeky time and way better than most conferences.

I've been doing the conference thing since I was a nervous little hacker kid going to HOPE and Summercon; about ten years. The last couple years have been particularly conference-heavy since the Rails community has taken off. I'm definitely starting to burn out.

Conferences are great when you don't frequently get an opportunity to hang out with your industry peers, but living in San Francisco is one big never-ending conference. It was decided that only one of us from Twitter will be speaking at RailsConf Europe in September, and while I'm disappointed to not be seeing Berlin, it'll be a nice reprieve.

As increasingly more high-caliber tech industry types turn to remote working, conferences are bound to change. One the one hand, people might feel a stronger need to meet up with their peers if they spend most of their time with family and neighbors. On the other hand, as online collaborative environments improve, a kind of cognitive dissonance could set in around having technical discussions offline. Still, people love to have a beer with folks they admire.

I'm hoping to maintain a conference moratorium until next year's South By Southwest Interactive. Again, that doesn't mean much living in the Bay Area, but I feel like I'm due for some purely recreational travel.

C4[1]: Cabel Sasser on Coda

Cabel works for Panic, makers of fine Mac software like Transmit, Unison, and most recently, Coda. Cabel met his business partner, Steven Frank, through the Amiga community when they were high school students. After heading off to college and getting "terrible" jobs, they decided to start a Mac software company on the side during "the worst time in Apple's history" (the Gil Amelio era). After a stab at a software updating app that they never released, the pair decided to write a competitor to the then-standard Mac FTP client, Fetch. That product was Transit, later renamed Transmit.

The company's next incarnation was "four guys in an apartment," working on Audion and a never-released interactive chat environment called Ripcord. They made the conscious decision to scrap much of their pre-OS X code at this juncture. The current "version" of the company is nine people in an office in Portland. They've branched into shirts as well, which makes up "5%, but a fun 5%" of their business; Cabel claims the shirts are all they wear in the office. In the next year, Panic is looking is looking to grow, and is grappling with questions of how much to expand their staff, their stable of apps, their office space, and so forth.

Coda

At the start of their application design process, Cabel mocks up the whole application in Photoshop; each feature is a layer group. This Photoshop document is then passed on to the engineers.

Cabel describes the decision process behind picking the feature set for Coda's features. He shows an example of a user-friendly grep feature, the sort of subtle improvement to the text editor concept that they'd decided on for Coda. More complex, Cabel had mocked-up an alternate toolbar look which ultimately necessitated a totally new toolbar implementation. Apparently, this new toolbar look may make it into Leopard, as Apple has taken notice. Cabel suggests a pragmatic and judicious application of this sort of UI revisionism. Some UI work, like nested windows and file browsing, proved challenging, and Cabel remains unsatisfied with some of the decisions they've made (ex: moving single- vs double-click file browser behavior to a preference pane).

The topic now changes to resolution independence. The simplest UI elements in Coda are drawn programmatically, which solves the issue of scaling to higher resolutions. PDFs are used for flat color and basic shapes in controls, and multi-resolution TIFFs are used elsewhere. Cabel zooms in tightly to Coda's interface to demonstrate the effect of through-and-through resolution independence, and it's stunning. For designing resolution-independent textures, Cabel recommends working with Photoshop's vector-based shapes and applying effects thereupon.

The process of deciding on a name and icon for Coda was difficult. The application was originally named "Transmit Studio," but this was abandoned in the icon selection process. The Icon Factory worked up an icon of a forklift, but this was abandoned when Panic realized that an application called Forklift with an eponymous icon was on the market. They then went in a more organic direction, embracing the concept of developers "growing" the applications they'd develop in this application. A leaf was settled on, and after several revisions they settled on a design that's a balance of hyper-realistic and a more idealized form. There's apparently a hint of fuzz around the stem of the leaf, testament to Cabel's and icon designer David Lanham's attention to detail. Two sets of toolbar icons were developed, and they settled on the more literal collection.

On to the top of partnering. Panic courted the developer of CSSEdit, as they had no interest in competing with "a great CSS editor." The deal didn't work out, so Panic wrote their own. Panic wanted to use O'Reilly for the books section of Coda, but the publisher wanted too large a cut of every sale. They partnered with No Starch Press instead.

On the subject of anti-piracy, Cabel claims it's a losing battle. They attempted to block the trading of serial numbers with a "depressing message" and network serial number checking, but haven't seen any wide trading of serial numbers yet. They have seen one attempt at scripted serial number generation.

Cabel concludes by mentioning that Panic is hiring, and that they're open to developers who work on shareware on the side.

Questions

"Comment on the shaded look of Coda?" Cabel says that "we've seen Aqua become less Aqua over time," and remarks on the dramatic change between 10.2 and the present OS X look. He likes the look of Leopard, and his hopeful about some of the best aspects of the iPhone making it in to the OS's look.

"Why hire a designer when you're clearly good at it?" Cabel claims he's not a good artist (and that Steven Frank is a great artist but a bad designer). He describes himself as the bottleneck in Panic's development process because all UI elements have to be approved by him.

"How did you make the decision to migrate to Cocoa?" Cabel says it simply felt like the right thing to do at the time.

"Do you really rapidly prototype in Photoshop?" Cabel claims he does, with occasional Interface Builder mockups to grab controls from.

"Any feedback on your blog post about resolution independent stuff from Apple?" Yup!

"What about the design philosophy a small, disparate Unix tools like grep?" Cabel describes the challenge as determining the scope of the "pieces of the web design puzzle" that they put together.

"How do you prototype interactivity beyond Photoshop?" Thinking about storyboarding.

"Opening up the books feature?" It's a very popular request. Not sure how it's going to look yet.

(I missed a couple questions at the end.)

C4[1]: Tim Burks on "Bridges and Beyond"

Tim Burks previously worked on very low-level chip software. In the process of delving deep into computer technology and learning the gritty details he claims to have "lost a lot of heros."

But Brad Cox, author of Objective-C, remains a hero of his, particularly for his paper "Planning the Software Industrial Revolution". Cox saw dynamic binding as essential for progress in the software industry, and Tim sees the need for a glue language beyond Objective-C. His requirements for a glue are reflection, interactivity, and expressiveness, all of which he found in Ruby.

To bridge ObjC and Ruby you can wedge libFFI between the two languages to bridge their respective method tables. Tim spent the first half of 2006 working with and documenting RubyCocoa, but ended up frustrated. He started working on RubyObjC, and overcame seven challenges in the process:

1. Message syntax is far different in ObjC than in most other languages.


2. Types have to be translated between the two languages.


3. Libraries have to be bridged (ex: NSDate to Ruby's Date).


4. Object storage: each language has its own way of constructing and storing objects.


5. Memory management: GC can kick in when you don't expect it.


6. Threading.


7. Namespace conflicts.

Tim goes on to demo a visual chip design application that was mocked up in Ruby with an interface in Cocoa. While parts of it have since migrated to ObjC, a Ruby console is available at all times. As he moves around a chip and traces paths between wires there are oohs and aahs. It's really impressive stuff.

At this point, the talk veers to Tim's latest project: Nu, an interpreted dialect of Lisp "written on, with, and for Objective-C." He demos a simple blog editing app with an embedded Nu console, including language functionality that mirrors Ruby's method_missing. More oohs and aahs. Colin Barret posted to the Twitter backchannel: "As a language nerd, this is getting kind of pornographic."

As a further demo, we find out that the blog server that the blog editor is talking to is all written in Nu with CoreData behind the scenes. A choice quote: "[the server code] is not as bureaucratic and dictatorial as Rails." Templates are written in an Erb-like syntax.

All told, probably the most brilliant talk so far this weekend.

C4[1]: Bob Ippolito on Exploring Erlang

Speaker works for Mochi Media, which builds analytics tools for Flash developers, ad targeting systems, etc. Their applications have high concurrency needs.

Introduction

Erlang is a functional language that revolves around a "process" abstraction. The language has no mutable data structures; which apparently necessitates lots of linked lists. State exists solely in the stack, so backtraces are comprehensive. A common pattern is to use accumulators that make use of tail calls (calls to another function at the end of a function). Another language feature is pattern matching, which binds variables when patterns are matched.

Types

Two collection types are available: lists and tuples. Basic types include atoms (effectively a constant string, apparently quite fast to work with), binary (a contiguous chunk of bytes), integer (unbounded), float (64-bit), and ref (a unique reference across a cluster of nodes). Other data types include fun (anonymous functions, which are serialized and close over variables), pid (reference to a process, which can be on another node), port (reference to a driver, socket, pipe, and so on, much like a pid). There are no boolean, character, or string types; the above types meet these needs in various configurations.

Syntax

Variable start with a capital letter, and can only be assigned once ("single assignment"). Equivalency (=) is a simple form of pattern matching. Case statements can take advantage of pattern matching in more complex ways; an example of matching particular dates in given. Pattern matching is also ideal for working with file formats and a bits-and-bytes level; an example of checking for Flash files is shown. Expressions end in commas (delimit expressions), semicolons (delimits clauses), and periods (which end expressions or trigger evaluation in the shell). Anything that starts with a question mark is a compiler macro. Exclamation points mean "send a message to the left".

Modules

Modules exist in a flat namespace, and explicitly export "public" functions. Modules are referenced by name (which is an atom) and compiled to a .beam file. Erlang boasts hot code reloading, and modules take advantage of this. When declaring functions, arity (the number of variables the function takes) is noted. To compile all the modules in your project just do make:all([load]).

Processes

Bob notes that Erlang processes are lightweight and fast, being neither native threads nor pthreads, but rather lightweight green threads. Processes communicate only with asynchronous messages, which they may selectively receive. Because there are no mutable variables, processes are where program state lives. Server processes are written in tail-recursive loops. Matching and logging unexpected messages is considered best practice; unhandled messages will pile up, which is effectively a memory leak.

Distributed

There's little security between nodes; a shared secret cookie is standard. Language semantics are such that communication between nodes works just like local communication. Nodes may connect transitively. It's fairly simple to start multiple nodes and add process monitoring. Erlang is designed around the idea that nodes will fail, and as such it's easy to trap errors and restart nodes.

OTP

Originally designed for high-availability telephony, OTP is now a way of designing fault-tolerant applications in Erlang. An OTP is a tree of supervisor processes which start, monitor, and restart child processes. Also available in OTP are behaviors, which are a kind of enforced inheritance. The example of gen_server and its conventions is shown.

Demo

The presentation was being given from a simple Erlang-based web server, which Bob then load tests to varying degrees, demonstrating excellent performance. Bob mentions that the networking core of Erlang is multiplexed and takes advantage of the OS-level polling features (kqueue, etc.). Performance for their app has been excellent: millions of hits per day per server.

Questions

The "Programming Erlang" book is "an extremely good resource," says Bob.

C4[1]: Alan Odgaard on TextMate

Alan Odgaard is the Danish developer behind TextMate, an elegant and hugely customizable editor favored by programmers on the Mac. I spend most of my day in TextMate, so it's great to take a peek behind the scenes.

TextMate was born out of Alan's dissatisfaction with Xcode; he had several other projects in the works, but none of them really grabbed his attention. Early versions of TextMate got spillover buzz from the original Rails screencast, which pushed him to get to 1.0 quickly. His priorities didn't necessarily match the community's; for example, print support was missing in 1.0. Features like snippets, tab triggers, folding, and columnar editing are some of TextMate's big sells. More generally, the editor's overall embrace of customization has been a huge sell.

The customization model starts with input (key strokes, dropped files, tab triggers) and context (file type, project type, location in file, SCM, editor state). Actions like inserting text and running commands can then be performed. The more context available, the more the available actions can be tailored to the task at hand.

Alan thinks his model can be generalized to other applications. Breaking out contexts into declarative semantic abstractions, in Alan's estimation, makes working easier and more foolproof for users. He claims to not work particularly hard on TextMate's community aspect; its bundles feature makes sharing customizations easy. In the near future, bundles will be moved to a distributed version control system (they currently live in a central Subversion repository). Bundles will be visible to the user as a set of features to be flipped on and off.

C4[1]: Shawn Morel on VMware Fusion

Rougher notes on this one, sorry. It's dense stuff!

Some things can't be virtualized: optical drives, graphics cards, etc. This talk mostly about CPU-level stuff.

Hypervisor idea originated at IBM, was formalized by Popek & Goldberg in 1974. Properties of a VMM (virtual machine monitor): equivalence (identical operation to host environment), resource control, efficiency. Formalized set of attributes for a virtualizable processor.

"We don't call it HijackOSX.kext because Apple wouldn't like that." Switching each direction [between host and VM] is 7 microseconds.

VT: Intel's set of instructions for virtualization; can tell it "trap this instruction for me." Works fine without it for 32-bit. VMware lives in high memory and protects the VM with segfault protection, which Intel clobbers in 64-bit, necessitating VT.

A lot of the code is cross-platform C and C++. The crux of the code is the Virtual Machine Monitor, which is hard to port to PPC. Porting doesn't buy you much, since you could only run PPC guest VMs.

3D: working on drivers that will get Aero Glass going. Some Direct X games working without shaders internally. GL coming after that.

"Host doesn't have to know anything about 64-bit."

OSx86 + VMware = virtual Mac on Mac right now. VMware would have to implement EFI and Apple's proprietary ROM.

Snapshot trees? "Want to make things simple for consumers ... We can do it, but ought we do it? Do we want Fusion and Workstation on the Mac?"

C4[1]: Daniel Jalkut on "Application Acquisition"

Daniel has recently taken over MarsEdit from Brent Simmons, developer of NetNewsWire. In that vein, he's speaking about how software acquisition pertains to indie developers. Meeting Brent at last year's C4 conference put him on the path to acquiring MarsEdit.

Daniel argues that developers already operate in a culture of acquisition, and that there's no shame in starting from someone else's code base; revision is easier than creation. An acquiring developer has to balance the PR gains in taking on an established product with the risks of being perceived as a "software flipping house."

Risks in acquiring include the up-front costs, supporting a big user base, and taking on a potential volatile set of technologies and requirements. Daniel suggests having a "minimum success plan", a balance between your investment of time and money in the acquired application and the potential returns. The total cost of acquisition is more than just the base price; factor in renovation, support, icons, hosting, bandwidth, design, and so forth.

Finding an asset is easiest when you ask for what you want; blogging about it worked for Daniel. Once you've found an application you want to acquire, evaluate the seller, the code, the cost and potential risks. Discern the seller's motivation for offloading their app. A principal point that Daniel emphasizes: are you passionate about the product you're acquiring?

Many factors go into to sealing the deal. You may consider getting a lawyer and/or an accountant. Determine exactly what's being sold in term of code and marketing assets, and what the acquisition schedule is. Figure out pricing and how to transition existing users into your customer relations setup. Definitely set up version control to stage and release. Make announcements and do PR.

Choice quotes

• "You think about acqusition and you think about people on golf courses, dressed like Rentzsch."
• "I'll take a drink of water now... [amplified sound of drinking] Huh, that's cool!"
• "If you refuse to start a blog or don't believe in blogs, you should give up."

C4[1]: Friday

I'm spending the weekend in sultry Chicago, cooped up in the City Center hotel with a couple hundred of the Mac development's community's best and brightest at the second year of Wolf Rentzsch's C4 conference.

Friday evening started with registration and dinner at the hotel, and a chance for attendees to catch up. Once dinner wrapped up, everyone migrated to the conference room next door for Wolf's introductory talk. After a few housekeeping notes, Wolf dived into an analysis of the nature of indie development as it relates to APIs and tools.

Wolf on indie development

Wolf believes that the barrier to entry for developers has been radically lowered in the last ten years. A new independent developer in the '90s had to shell out for development tools, a developer network membership, and book/magazines to keep up with the latest techniques. Today, developer tools are free and the best educational resources are peer-to-peer in the form of blog posts and forums. If you want to "go indie" on the Mac, all you need is a Mac.

Additionally, Wolf argues that the important APIs are moving to the web. For web services, a lack of an API can be a deal-killer when trying to build buzz and community. For developers, it means you're more in control of both providing and consuming market-making APIs. Big companies no longer hold the API keys.

The talk was kind of all over the map (in a good way), and included the assertion that JavaScript is the best newbie language out there, amongst other juicy tidbits. Bold assertions set the stage for Wil Shipley's uproarious talk.

Wil Shipley on hype

Wil Shipley is no stranger to hype. The developer of Delicious Library most recently and many of Omni's best apps previously, he's a huge personality in the Mac community and a notable figure in the broader tech community. A superb technical mind, he's outspoken on the topic of his own success. Who better to talk about hype?

Essentially, Wil's talk was a tutorial on the mechanics and particulars of homebrew PR for independent Mac developers. While this could have been a dry, business school affair, Wil managed to crank out a routine that suggests he's got a solid backup career in standup. To attempt to replicate the talk's humor in blog form would no doubt fall flat, which elucidates one of Wil's best points: everyone remembers a character.

In short, Wil suggested the following to get noticed: build something great, take advantage of online PR, be generous with licenses for beta testers and reviewers, talk to Apple and get your app in front of everyone you can there, and make sure you keep the public's appetite for your work whetted at all times. While most of this is solid and intuitive advice, it's clear that it helps to be Wil Shipley to have it all work out for you.

A standout: for all his success, Wil claims the most successful app he ever shipped reached a fraction of a percent of all worldwide Mac seats, which is in turn ~5% of the overall computer market. On the hand, it's motivational that one you can make a good living off such a small market. On the other, it's sad that such great work is reaching so few people.

The evening concluded with drinks on the hotel's rooftop pool. Classy!

Attending: Google Developer Day 2007

I'll be at Google Developer Day tomorrow in San Jose on behalf of Twitter. If there's talks worth blogging about, I'll blog about them. If you're gonna be there, say hi! I'll be wearing my Twitter shirt.

My RailsConf Schedule

I'm flying from San Francisco to Portland at 0640 in the bloody morning on Friday, with the aim of being checked in to my hotel and at the conference by sometime during the opening keynote.

Thenceforth, you'll find the schedule of talks I'm attending at MyConfPlan. Supposedly, my colleague from Twitter and I are giving a talk on Sunday at 10:45AM, but as of yet we're not on the schedule. There's no shortage of good sessions, though.

If you'd like to take the pulse of RailsConf via Twitter, check out the visualizer we put together. Most of the photos in the background were taken by Timoni on our trip to Portland earlier this year. It's pretty hypnotic, so use with caution.

We've got dinner plans on Friday night, but nothing set for Saturday. As much as I'd like to stay and hit the breweries, I'm flying back on Sunday evening.

See you in Portland tomorrow!