Intro

As I was reading Crafting Interpreters, I had this idea pretty early on that I wanted to be able to understand what the interpreter was doing visually, step by step. There are lots of graphics in the book that demonstrate how each step connects to the next one, but you are left using your imagination if you want to see how any arbitrary piece of code is converted from the language you write to the one the computer actually knows how to execute.

You can check it out here!

How It Works

On the Lox side I added json logs throughout each phase of the interpretation process – logs when tokens get lexed, logs when tokens get parsed, logs when bytecode gets generated, etc. These logs include an index into the source code so we can trace the path each piece of code takes as it’s transformed all the way into bytecode instructions to be executed. Then we build Lox to wasm and let the web app do the rest of the work.

The web app uses CodeMirror to let you type in any Lox code you want, runs it through the wasm build of the interpreter, and collects each group of log messages to produce visualizations of each compiler phase using D3. The lexer visualization is pretty simple; it’s just syntax highlighting! All we need is to produce a series of spans with the right text and css classes attached.

The parser is the trickiest one. It’s worth noting that the C version of the Lox interpreter just generates bytecode as it parses, never actually producing an AST data structure. Statements are parsed recursively in a way that mirrors the AST. Each statement knows how many children it has. We can process them linearly on the front end and build a tree that we can then hand off to D3. This gets a little trickier for expressions which are not parsed the same way. You can learn more about the Pratt parsing algorithm in the book here, but the takeaway for the AST visualizer is that all the nodes inside expressions get logged backwards. We have to stop every time we see an expression, gather up all its child nodes, and process them in a batch. So the resulting AST visualization doesn’t exactly mirror how the parser works. It seemed like an acceptable simplification to me.

The bytecode visualizer that shows all the VM instructions generated by each parsing step is mercifully straightforward. Since it’s just a list of instructions, we can just show them off in a list. It actually looks pretty much the same as the output printed by the VM to the console with the debug flags turned on. Likewise the stack is just a list of items on the stack. When the VM runs the instructions, we log the contents of the stack, so the playback visualizer can show you what happens as the VM is instructed to jump around to different instructions in the list.

But Why

At the beginning of Crafting Interpreters, author Robert Nystrom muses that compiler books often begin by pleading with readers to value their existence. I spent a lot of time reflecting on that as I read the book. I too felt uncertain how to justify the way I was spending my time both reading the book and working on this project. I was motivated only by the feeling that it’s really cool that we can speak to the computer in whatever way we want, that expresses our ideas the best, because compilers exist. I was motivated to build the visualizer like a hiker stops to take pictures along a trail, as much as to serve any teaching purpose. Maybe AST nodes and bytecode instructions are not as beautiful as waterfalls and vistas, but I enjoyed travelling through them nonetheless.

Of course there are real world use cases for all things I learned here. I have run into situations several times in my career where being able to refactor code by manipulating code as data has proved valuable. Nowadays you can delegate such tasks to an LLM, but it seems to me that it would be better in many situations to produce a deterministic refactoring tool with an LLM than to let the robot refactor each line itself, leaving God knows what monstrosities hidden throughout your codebase. Some phases of the visualizer are a lot like debugging tools that people use every day. I also hope the visualizer can help make compilers seem a little less scary to anyone else interested in learning more about them.