I've written about how I started using Erlang. A good question is why, after eleven years, am I still using it?

For the record, I do use other languages. I enjoy writing Python code, and I've taught other people how to use Python. This website is statically generated by a Perl program that I had fun writing. And I dabble in various languages of the month which have cropped up. (Another website I used to maintain was generated by a script that I kept reimplementing. It started out written in Perl, but transitioned through at least REBOL, J, and Erlang before I was through.)

One of the two big reasons I've stuck with Erlang is because of its simplicity. The functional core of Erlang can and has been described in a couple of short chapters. Knowledge of four data types--numbers, atoms, lists, tuples--is enough for most programming problems. Binaries and funs can be tackled later. This simplicity is good, because the difficult part of Erlang and any mostly-functional language is in learning to write code without destructive updates. The language itself shouldn't pour complexity on top of that.

There are many possibilities for extending Erlang with new data types, with an alternative to records being high on the list. Should strings be split off from lists into a distinct entity? What about arrays of floats, so there's no need to box each value? How about a "machine integer" type that's represented without tagging and that doesn't get automatically promoted to an arbitrarily sized "big number" when needed?

All of those additional types are optimizations. Lists work just fine as strings, but even the most naive implementation of strings as unicode arrays would take half the memory of the equivalent lists, and that's powerful enticement. When Knuth warned of premature optimization, I like to think he wasn't talking so much about obfuscating code in the process of micro-optimizing for speed, but he was pointing out that code is made faster by specializing it. The process of specialization reduces your options, and you end up with a solution that's more focused and at the same time more brittle. You don't want to do that until you really need to.

It may be an overreaction to my years of optimization-focused programming, but I like the philosophy of making the Erlang system fast without just caving in and providing C-style abilities. I know how to write low-level C. And now I know how to write good high-level functional code. If I had been presented with a menu of optimization-oriented data types in Erlang, that might never have happened. I'd be writing C in the guise of Erlang.

The second reason I'm still using Erlang is because I understand it. I don't mean I know how to code in it, I mean I get it all the way down. I know more or less what transformations are applied by the compiler and the BEAM loader. I know how the BEAM virtual machine works. And unlike most languages, Erlang holds together as a full system. You could decide to ditch all existing C compilers and CPUs and start over completely, and Erlang could serve as a foundation for this new world of computing. The ECOMP project (warning: PowerPoint) proved that an FPGA running the Erlang VM directly gives impressive results.

Let me zoom in on one specific detail of the Erlang runtime. If you take an arbitrary piece of data in a language of the Lua or Python family, at the lowest-level it ends up wrapped inside a C struct. There's a type field, maybe a reference count, and because it's a heap allocated block of memory there's other hidden overhead that comes along with any dynamic allocation (such as the size of the block). Lua is unabashedly reliant on malloc-like heap management for just about everything.

Erlang memory handling is much more basic. There's a block of memory per process, and it grows from bottom to top until full. Most data objects aren't wrapped in structs. A tuple, for example, is one cell of data for the length followed by the number of cells in the tuple. The system identifies it as a tuple by tagging the pointer to the tuple. You know the memory used for a tuple is always 1 + N, period. Were I trying to optimize data representation by hand, with the caveat that type info needs to be included, it would be tough to do significantly better.

I'm sure some people are correctly pointing out that this is how most Lisp and Scheme systems have worked since those languages were developed. There's nothing preventing an imperative language from using the same methods (and indeed this is sometimes the case).

Erlang takes this further by having a separate block of memory for each process, so when the block gets full only that particular block needs to be garbage collected. If it's a 64K block, it's takes microseconds to collect, as compared to potentially traversing a heap containing the hundreds of megabytes of data in the full running system. Disallowing destructive updates allows some nice optimizations in the garbage collector, because pointers are guaranteed to reference older objects (this is sometimes called a "unidirectional heap"). Together these are much simpler than building a real-time garbage collector that can survive under the pressure of giant heaps.

Would I use Erlang for everything? Of course not. Erlang is clearly a bad match for some types of programming. It would be silly to force-fit Erlang into the iPhone, for example, with Apple promoting Objective C as the one true way. But it's the best mix of power and simplicity that I've come across.

Whenever I see this list of memory cell sizes, it strikes me as antiquated:

BYTE = 8 bits
WORD = 16 bits
LONG = 32 bits

Those names were standard for both the Intel x86 and Motorola 68000 families of processors, and it's easy to see where they came from. "Word" isn't synonymous with a 16-bit value; it refers to the fundamental data size that a computer architecture is built to operate upon. On a 16-bit CPU like the 8086, a word is naturally 16-bits.

Now it's 2010, and it's silly to think of a 16-bit value as a basic enough unit of data to get to the designation "word." "Long" is similarly out of place, as 32-bit microprocessors have been around for over 25 years, and yet the standard memory cell size is still labeled in a way that makes it sound abnormally large.

The PowerPC folks got this right back in the early 1990s with this nomenclature:

BYTE = 8 bits
HALFWORD = 16 bits
WORD = 32 bits

That made sense in 1991, and it's still rational today. (64-bit is now common, but the jump isn't nearly as critical as it was the last time memory cell size doubled. The PowerPC name for "64-bits" is "doubleword.")

Occasionally you need to reevaluate your assumptions and not just cling to something because it's always been that way.

For a few months I did freelance humor writing. Greeting cards, cartoon captions, that sort of thing. My sole income was from the following slogan, which ended up on a button:

Once I've gathered enough information for the almighty Zontaar, I'm outta here!

Sitting down and cranking out dozens of funny lines was hard. Harder than I expected. I gave it up because it was too draining (and because I wasn't making any money, but I digress).

Periodically I decide I want to boost my creativity. I carry around a notebook and write down conversations, lists, brainstormed ideas, randomness. I recently found one of these notebooks, so I can give some actual samples of its contents. Below half a page of "Luxury Housing Developments in Central Illinois Farmland" (e.g., Arctic Highlands), there's a long list titled "Ridiculous Things." Here are a few:

salads
spackle
key fobs
wine tastings
mulch
hair scrunchies
asphalt
Fry Daddy^TM
cinder blocks
relish
Frito Pie
aeration shoes

Okay, okay, I'll stop. But you get the idea.

As with the humor writing, I remember this taking lots of effort, and it took real focus to keep going. Did this improve my creativity? I'd like to think so. It certainly got me thinking in new directions and about different topics. It also made me realize something fundamental: technical creativity, such as optimizing code or thinking up clever engineering solutions, is completely different from the "normal" creativity that goes into writing stories or taking photos.

Years ago, I followed the development of an indie game. This was back when writing 3D games for non-accelerated VGA cards was cutting edge. The author was astounding in his coding brilliance. He kept pulling out trick after trick, and he wasn't shy about posting key routines for others to use. Eventually the game was released...and promptly forgotten. It may have been a technical masterpiece, but it was terrible as game, completely unplayable.

I still like a good solution to a programming problem. I still like figuring out how to rewrite a function with half the code. But technical creativity is only one form of creativity.