A funny thing happened today.

Actually, it happened a couple of days ago, but the effects have taken a bit of time to fully hit home. I was reading around in Steve Yegge's blog archives, Lambda the Ultimate, and a few other places that I forget at the moment. The other important bit of background was that I was fooling around with some concepts for the syntax of the emerging Epoch language. Somewhere in all this, something deep in my brain snapped.

I want to use the word "sudden" to describe what happened, but that isn't really quite accurate. It was a slow thing, building over a couple of hours (and in fact is still building a bit even now). There was no one trigger, not even in the sense of a "straw that broke the camel's back." Somewhere, though, a pile of thoughts, peeves, and old questions sort of congealed, and the half-formed answers in the back of my mind passed some ineffable threshold.

Once that marker was passed, my entire view on programming - and, more importantly, programming languages - changed profoundly. As cheesy as it may sound, it honestly has given me an almost giddy sense of euphoria. There is really something verging on the spiritual about reaching such a deep mental clarity. One could almost say that I've found religion.

I've found Lisp.


You have no idea how weird it is to type those words seriously. Heck, it wasn't but a few months ago that I was casually dismissing the world of "smug Lisp weenies" (brownie points to whoever I stole that phrase from) and probing the land of imperative languages for the answer to All My Programming Problems. I've been pondering for years the need for better abstraction facilities in programming languages - although admittedly for much of that time I only knew that I was annoyed at the deficiencies of languages like Java, without being able to really clearly express why I felt them to be so crippled. In fact, a fair bit of my journey along that road is recorded here in this very journal.

Over time, things added up. Then came Tim Sweeney's presentation on programming language needs for next-generation games. That whole thing spurred off a thought in my mind that had been brewing for some time: existing languages suck. So let's build a new one! The results of that particular epiphany are still doing interesting things off in this thread.

Yet even during that discussion, I've mostly (until now) been kind of stand-offish with respect to Lisp. When the language was still called Foo, I wanted to steal some of the "nifty" Lisp features without falling off the deep end into all that crazy talk about sexprs and macros. As things evolved into Epoch, I wanted to steal Lisp's power without looking like Lisp.

I've known for quite some time now of the need for a richer language than what I'm normally stuck with using. In fact, I've often looked at functional languages like Lisp, ML, and Haskell with a sort of wistful longing - I wanted to get into all that awesome potency, but without the scary shapes and incantations that didn't really make sense to my C-addled brain. I could appreciate the need for expressivity, extensible languages, and clean methods for building abstractions. I just hated the syntax.

I think I might have stayed in precisely that position for a very long time - maybe forever - if the Epoch discussion hadn't happened. The Epoch thing got me thinking really hard about syntax, and about clean ways of expressing a very rich and self-extending set of notions. I had started slowly down the trap of making a Really Convoluted Syntax, when something I read (I wish I could recall what) brought me back to reality: a simple, elegant syntax will serve much better.

Actually, I think one other thing contributed powerfully to my Conversion - my work at Egosoft. I've been working on defining a lot of new data formats for some of the features I'm building. In the process, I've had a bit of a struggle - the artists and content creation folks basically want the data to be a huge blob of special-case stuff, with lots of truly disgusting edge-cases, magic symbols, and so on. I've been fighting to take a totally different approach: define all of the data recursively, building complex content in a sort of emergent way from a lot of tiny and similar building blocks. I've played with a huge number of alternatives, and this recursive approach just makes this incredibly better all around. It's more powerful, more robust, and more reliable - it doesn't need a lot of special-case code to back it up, meaning that it decreases (in the long run) the chance of stupid bugs creeping in at edge cases.

So anyways, while I was reading whatever it was that I was reading, the Big Light Bulb Over My Head clicked on. Obviously, this recursive-definition thing isn't just a good idea for game data. It's a good idea for any data. Now I can see clearly why I'm so drawn to the idea of XML but find it to be so annoying and icky at the core - it's almost this recursive thing, but not quite.

Now I can see why Lisp is such an effective language family. Now I can appreciate the syntax and the philosophy behind it. Before, it struck me as just this weird, otherworldly, almost arbitrary thing. Some lunatic had a parenthesis-fetish, and built a language around it. Thanks but no thanks, I prefer the curly braces and four-billion-page specs that can only begin to cover the silly little nuances of syntax.

Unlike any other time before, I can really appreciate the effectiveness of Lisp's syntactical approach. It's not just a random thing - it's a truly beautiful and elegant solution to some very hard problems in language design. I'm not nearly smart enough to solve those problems. Every time I try to hack out a little start at a solution, I look over at Lisp and find out that it's already done a better job, and has been for decades.


At this point, I'm in a weird position. I want a language that has the Lisp-nature, but is not Lisp. It seems like some sort of bizarre Zen mind-rape. Maybe this is the last vestiges of my long history in the C-syntax family still clinging on to my soul for dear life. Maybe I've loaded my bloodstream so thick with curly braces that I need time to be purged.

Or maybe I have this sort of niggling feeling like Lisp just isn't quite it yet. (Eww... I almost feel like a heretic for even suggesting that, when even a few days ago I would have been proud to announce it.) Frankly, I think there's some truth there. Lisp is a brilliant concept. It does a lot of things in very nice ways. And yet I can't help but wonder why it has never taken off, why stupid messy compost heap languages like C++, Perl, and Java still exist.

I've heard a lot of theories as to why Lisp has failed to take over the world. Most of them strike me as total garbage. I think the reality is that Lisp, while Well and Truly Awesome, is still a step short of optimal.

It's funny, really. Two days ago I would have had no reservations about implying that Lisp isn't The Best Thing Forever and Ever. Yet now, as a newly reborn Smug Lisp Weenie, I feel kind of dirty for even bringing it up. I haven't even written any nontrivial software in Lisp yet. Now I can truly sympathize with those who found the Truth before me; you really honestly can't explain it succinctly.

To reach enlightenment requires one to travel down a long, hard road. It takes experience and acute awareness of the deficiencies of other tools. It takes a lot of hard thinking about deep principles of programming. You really can't be led to Lisp Enlightenment by means of a clever article or book. Like nirvana, Lisp Enlightenment must be attained by each individual, walking on his own path.


I doubt I'm smart enough to invent something that is closer to optimal than Lisp, at least not by itself. But maybe there's hope in learning from Lisp. Maybe we can take the pragmatic road, look long and hard at the realities of Lisp's station in life, and see if there just might be some room for improvement. Maybe we can stand upon the shoulders of giants, and just maybe, in so doing, we can see farther.

So now I am a Smug Lisp Weenie in Training. And damned if there isn't a heck of a lot to be smug about.