More Erlanging

One of my favorite reads from my infosec days was teenage mutant ninja hero coders, a blog by computer security expert Maximillian Dornseif. Dornseif usually says smart things and you should probably be reading him.

Lately, his subject matter has turned to more general programming topics and distributed systems. Dornseif wrote about Erlang's unsubstantiated reliability claims recently, and follows up on that today.

Dornseif's point:

"Erlang is an interesting language. OTP is great engineering. Erlang has considerable momentum compared to other languages with unusual concepts. There is no need use 99.9999999% which ring[s] so hollow."

That's pretty easy to agree with.

I've seen two conflicting schools towards the end of figuring out where Erlang fits in the current programming language landscape. One is the ever-popular "right tool for the right job" school, which posits that you build your distributed components in something like Erlang and other components in languages with more mature, easy-to-use libraries. While this is admirable and pragmatic, the downside is that you then have to couple your tools with language bridges or interoperable formats like XML, JSON, or a message queuing standard. The coupling components can be brittle or eat up processing time with expensive parsing and conversion.

The opposing school is well-represented by, again, Russ Beattie:

" [...] I just don't like working with multiple languages at the same time. I much prefer to have one language in my toolbox that I try to use for as much as possible, so that I can avoid time-sapping context changes, re-use code, and become more proficient with each line of code I write whether its on the desktop, in the browser or on the server."

I've known lots of developers for whom a solution isn't a solution if it isn't available in their language of choice. Context-switching can, indeed, be brutal.

At the moment, I'm not ready to take a side. I've always been a multi-linguist, and I see ups and downs to both sides. It's interesting, is all.

Russ Beattie on Erlang by Way of Java

Java has never been a particularly relevant language to me, despite being the first language I tried to teach myself. I've managed to avoid Java in my work, in my brief college eduction, and in my self-eductation. I've been aware that there's a whole giant word of Java out there in the IT industry but I consider myself privileged to have never been mired in it beyond editing the occasional XML configuration or fixing minor bugs.

I think Russ Beattie's claim that Java needs an overhaul is dead on, but I also liked this observation:

"The reason people are looking at Erlang is not because its beautiful syntax, great documentation, or up-to-date libraries. Trust me. It's because the Erlang VM can run for long periods of time, scaling linearly across cores or processors filling the same niche that Java does right now on the server."

If Ruby supported lightweight green threads, pattern matching, and some of the concurrency paradigms that Erlang offers, nobody would even be looking at the language. While Ruby may yet improve on this front, just about everybody I know who works on large-scale web applications has at least taken a hard look at Erlang, and some are diving right in.

C4[1]: Bob Ippolito on Exploring Erlang

Speaker works for Mochi Media, which builds analytics tools for Flash developers, ad targeting systems, etc. Their applications have high concurrency needs.

Introduction

Erlang is a functional language that revolves around a "process" abstraction. The language has no mutable data structures; which apparently necessitates lots of linked lists. State exists solely in the stack, so backtraces are comprehensive. A common pattern is to use accumulators that make use of tail calls (calls to another function at the end of a function). Another language feature is pattern matching, which binds variables when patterns are matched.

Types

Two collection types are available: lists and tuples. Basic types include atoms (effectively a constant string, apparently quite fast to work with), binary (a contiguous chunk of bytes), integer (unbounded), float (64-bit), and ref (a unique reference across a cluster of nodes). Other data types include fun (anonymous functions, which are serialized and close over variables), pid (reference to a process, which can be on another node), port (reference to a driver, socket, pipe, and so on, much like a pid). There are no boolean, character, or string types; the above types meet these needs in various configurations.

Syntax

Variable start with a capital letter, and can only be assigned once ("single assignment"). Equivalency (=) is a simple form of pattern matching. Case statements can take advantage of pattern matching in more complex ways; an example of matching particular dates in given. Pattern matching is also ideal for working with file formats and a bits-and-bytes level; an example of checking for Flash files is shown. Expressions end in commas (delimit expressions), semicolons (delimits clauses), and periods (which end expressions or trigger evaluation in the shell). Anything that starts with a question mark is a compiler macro. Exclamation points mean "send a message to the left".

Modules

Modules exist in a flat namespace, and explicitly export "public" functions. Modules are referenced by name (which is an atom) and compiled to a .beam file. Erlang boasts hot code reloading, and modules take advantage of this. When declaring functions, arity (the number of variables the function takes) is noted. To compile all the modules in your project just do make:all([load]).

Processes

Bob notes that Erlang processes are lightweight and fast, being neither native threads nor pthreads, but rather lightweight green threads. Processes communicate only with asynchronous messages, which they may selectively receive. Because there are no mutable variables, processes are where program state lives. Server processes are written in tail-recursive loops. Matching and logging unexpected messages is considered best practice; unhandled messages will pile up, which is effectively a memory leak.

Distributed

There's little security between nodes; a shared secret cookie is standard. Language semantics are such that communication between nodes works just like local communication. Nodes may connect transitively. It's fairly simple to start multiple nodes and add process monitoring. Erlang is designed around the idea that nodes will fail, and as such it's easy to trap errors and restart nodes.

OTP

Originally designed for high-availability telephony, OTP is now a way of designing fault-tolerant applications in Erlang. An OTP is a tree of supervisor processes which start, monitor, and restart child processes. Also available in OTP are behaviors, which are a kind of enforced inheritance. The example of gen_server and its conventions is shown.

Demo

The presentation was being given from a simple Erlang-based web server, which Bob then load tests to varying degrees, demonstrating excellent performance. Bob mentions that the networking core of Erlang is multiplexed and takes advantage of the OS-level polling features (kqueue, etc.). Performance for their app has been excellent: millions of hits per day per server.

Questions

The "Programming Erlang" book is "an extremely good resource," says Bob.