Lisp, Smalltalk, and the Power of Symmetry

Like many hackers, my first real programming language love was Lisp. Paul Graham, who inspired my own explorations of the language, is a particular advocate and has written quite a bit about Lisp and what makes it different from other programming languages. So what does make Lisp different? Why does Lisp continue to be one of the most powerful, flexible, and concise programming languages in existence, despite the fact that it was invented in 1958–making it the second-oldest high-level programming language in the world?

Paul Graham’s answer is macros:

Many languages have something called a macro. But Lisp macros are unique. […]

Lisp code is made out of Lisp data objects. And not in the trivial sense that the source files contain characters, and strings are one of the data types supported by the language. Lisp code, after it’s read by the parser, is made of data structures that you can traverse.

If you understand how compilers work, what’s really going on is not so much that Lisp has a strange syntax as that Lisp has no syntax. You write programs in the parse trees that get generated within the compiler when other languages are parsed. But these parse trees are fully accessible to your programs. You can write programs that manipulate them. In Lisp, these programs are called macros. They are programs that write programs.

There’s an interesting implication hidden in PG’s argument here. What he’s saying is that macros make Lisp powerful because they allow you to write programs that themselves write programs. But that’s what macros are, not what enables them–Lisp’s macros aren’t the cause of its power, they’re a symptom of it. What makes Lisp powerful isn’t its macros, it’s the fact that Lisp runs in the same context it’s written in. It’s s-expressions all the way down. This leads to an interesting possibility: could there be other ways to achieve similar power? Are macros the only possible way to make programs that write programs, or might there be others?

When I studied computer languages in college, the second language we looked at after Lisp was Smalltalk. I had been so impressed with the power and flexibility of Lisp that the first question out of my mouth that day was “does Smalltalk have macros?” My professor, after thinking it over for a second, replied that “Smalltalk doesn’t need macros.” Huh? Smalltalk doesn’t need macros? Why not? What does it have instead? It took me three years to figure it out: Smalltalk doesn’t need macros because it has classes instead.

It may seem strange to compare object-oriented classes to Lisp macros. Aren’t classes and class structures notoriously rigid, brittle, and prone to spaghetti code? Isn’t all that the opposite of a Lisp macro?

It’s true that the usual flavor of Object-Oriented Programming (“OOP,” as in “OOPs what was I thinking when I designed this class hierarchy”) has issues. Languages such as Java, C++, and even Python seem to think that “object-oriented” means mostly “classes and inheritance.” Which is sort of like saying that “driving” means mostly “buttons and pedals.” What most of these languages seem to miss is that Smalltalk’s class system, like Lisp’s macro system, is a symptom of the power already available in the language, not its cause. If it didn’t already have it, it wouldn’t really be that hard to add it in yourself.

Without Smalltalk’s underlying fundamentals, class inheritance becomes nothing more than a tool for code reuse. As such, it is neither the only nor necessarily the best such tool. The real power of Smalltalk’s object system, including its classes, is not inheritance–it’s reflection. Just as Lisp macros are powerful because they can operate on any Lisp code, including themselves, Smalltalk classes are powerful because they themselves are objects. Smalltalk, like Lisp, runs in the same context it’s written in. It’s objects all the way down.

Lisp is powerful because all Lisp programs are also Lisp data–everything that can be run can be written (and read) as an s-expression. Macros in Lisp are simply what happens when you apply this relationship inductively: they are code that manipulates data that is itself code.

Smalltalk is powerful because all Smalltalk data are programs–all information is embodied by running, living objects. Class programming in Smalltalk is simply data manipulating programs that are themselves data–it’s the inverse of the Lisp philosophy, but the end result is the same. It’s what enables the Smalltalk debugger to freeze, dissect, modify, and resume programs mid-execution. It’s what enables the browser to instantly find all objects that respond to a given message, or all superclasses and subclasses of a given object, or every running instance of a given class. It’s why the Smalltalk IDE isn’t just written in the language, it quite literally is the language.

Lisp, as PG mentions above, has effectively no syntax: because Lisp source code is expressed in the same form as running Lisp code, which is expressed in the same form as Lisp data, the three are interchangeable. Programs can write data which can be run as programs which can write data which can…

Smalltalk goes one further than Lisp: it’s not that Smalltalk’s source code has no syntax so much as Smalltalk has no source code. “Source code,” after all, just means “program that isn’t running,” and there is no such thing as a Smalltalk program that isn’t running! Because there is absolutely nothing in Smalltalk except “data that runs”–which is what an object is, after all–then there is no distinction in Smalltalk between data and programs. Data (objects) can write programs (objects) which write data (objects) which write programs (objects) which…

So Lisp macros aren’t the only way that programs can write programs, after all. The only thing that seems to be required for a language to allow this is a pervasive symmetry between programs and data. If a language allows programs and data to be treated as the same thing, then that language becomes easily and infinitely extensible–a language of the gods. S-expressions, it turns out, are not the only way to do this–you can do it with objects as well. I wonder if there are still more ways we haven’t yet tried?