As I was reading Crafting Interpreters, I had this idea pretty early on that I wanted to be able to understand what the interpreter was doing visually, step by step. There are lots of graphics in the book that demonstrate how each step connects to the next one, but you are left using your imagination if you want to see how any arbitrary piece of code is converted from the language you write to the one the computer actually knows how to execute.

You can check it out here!

How It Works

On the Lox side I added json logs throughout each phase of the interpretation process – logs when tokens get lexed, logs when tokens get parsed, logs when bytecode gets generated, etc. These logs include an index into the source code so we can trace the path each piece of code takes as it’s transformed all the way into bytecode instructions to be executed. Then we build Lox to wasm and let the web app do the rest of the work.

The web app uses CodeMirror to let you type in any Lox code you want, runs it through the wasm build of the interpreter, and collects each group of log messages to produce visualizations of each compiler phase using D3. The lexer visualization is pretty simple; it’s just syntax highlighting! All we need is to produce a series of spans with the right text and css classes attached.

The parser is the trickiest one. It’s worth noting that the C version of the Lox interpreter just generates bytecode as it parses, never actually producing an AST data structure. Statements are parsed recursively in a way that mirrors the AST. Each statement knows how many children it has. We can process them linearly on the front end and build a tree that we can then hand off to D3. This gets a little trickier for expressions which are not parsed the same way. You can learn more about the Pratt parsing algorithm in the book here, but the takeaway for the AST visualizer is that all the nodes inside expressions get logged backwards. We have to stop every time we see an expression, gather up all its child nodes, and process them in a batch. So the resulting AST visualization doesn’t exactly mirror how the parser works. It seemed like an acceptable simplification to me.

The bytecode visualizer that shows all the VM instructions generated by each parsing step is mercifully straightforward. Since it’s just a list of instructions, we can just show them off in a list. It actually looks pretty much the same as the output printed by the VM to the console with the debug flags turned on. Likewise the stack is just a list of items on the stack. When the VM runs the instructions, we log the contents of the stack, so the playback visualizer can show you what happens as the VM is instructed to jump around to different instructions in the list.

But Why

At the beginning of Crafting Interpreters, author Robert Nystrom muses that compiler books often begin by pleading with readers to value their existence. I spent a lot of time reflecting on that as I read the book. I too felt uncertain how to justify the way I was spending my time both reading the book and working on this project. I was motivated only by the feeling that it’s really cool that we can speak to the computer in whatever way we want, that expresses our ideas the best, because compilers exist. I was motivated to build the visualizer like a hiker stops to take pictures along a trail, as much as to serve any teaching purpose. Maybe AST nodes and bytecode instructions are not as beautiful as waterfalls and vistas, but I enjoyed travelling through them nonetheless.

Of course there are real world use cases for all things I learned here. I have run into situations several times in my career where being able to refactor code by manipulating code as data has proved valuable. Nowadays you can delegate such tasks to an LLM, but it seems to me that it would be better in many situations to produce a deterministic refactoring tool with an LLM than to let the robot refactor each line itself, leaving God knows what monstrosities hidden throughout your codebase. Some phases of the visualizer are a lot like debugging tools that people use every day. I also hope the visualizer can help make compilers seem a little less scary to anyone else interested in learning more about them.

Coming from the ecosystems of popular interpreted platforms like Python and NodeJS, the tooling around C projects felt very unfamiliar to me. Just like with the jLox, my priority was just to pick something that works so I could move on to actually solving problems. It seemed to me based on some interneting that there are at least 3 reasonable options:

• make
• cmake
• meson

Once again following the “I’ve used it before” principle, I decided once again to go with Make. Make is a very manual build tool that feels only slightly more sophisticated than writing a bunch of bash scripts. That said I was able to come up with a setup that addressed all of my needs and I think works pretty well for a project of this size. This post will cover some useful Makefile concepts and flags for the C compiler.

Choosing a C compiler

Once again, we have a couple of choices. Working on a Linux system I narrowed it down to two choices:

• gcc
• clang

While gcc seems to be the default on Linux I used clang because the error messages seemed slightly better. I can only guarantee that I tested all the command line flags described in this post with clang but this information should apply to gcc as well.

Project Organization

Before I go into detail about the build process I landed on, let’s talk about the goals. Here’s the way I wanted the project laid out:

Goals:

• compile binaries to bin
• support building additional binaries from additional entry points in case you want to build toy programs that play around with pieces of the library
• separate src/lib directory to keep entry points and lib files apart
• compile debug binary with debug flags turned on
• build and run lox with a single command
• pass command line args to lox when running it through the build tool

The rest of this post explains how to set this up.

Compiling C programs with clang

Installing clang should be pretty straightforward if it’s not installed already.

From there all you need to do is point it at a src file like so:

clang src/main.c

This is going to output your binary to a file called… a.out. So obviously we’re gonna want to do something about that.

clang -o bin/lox src/main.c

That’s great until you need to start breaking up main.c into multiple files. Clang will not automatically compile anything just because you #include it. Now you’ve got something like:

clang -o bin/lox src/main.c src/vm.c src/compiler.c

This is the point where I no longer want to function without a task runner

Makefile tasks

You’ve probaby called make install to build and install an open source package before. It’s pretty ubiquitous. That said you don’t really need any prior experience to get started.

Writing your first task is as simple as this:

clox:
        clang -o bin/lox src/main.c src/vm.c src/compiler.c

Important note here: oftentimes when you are programming it does not matter whether you use tabs or spaces to indent. This is not the case for Makefiles. Make will blow up if you use spaces, you have to use tabs.

So now we’ve got a task that builds the program and you can just keep adding your modules one by one to this list if you want.

Task dependencies

Just like with gradle we can use task dependencies to ensure that prerequisite tasks run first. We’ll take advantage of that to add a run task that will make sure your build is up to date before runing lox.

clox:
        clang -o bin/lox src/main.c src/vm.c src/compiler.c
run: clox
        bin/lox

Clang include flags

At the point I want to move all of the additional lib files into a src/lib directory. From my perspective it makes sense to just pile everything at the top level of the project. Instructing the compiler to grab things from src/lib is not an issue. This will work just fine.

clox:
        clang -o bin/lox src/main.c src/lib/vm.c src/lib/compiler.c
run: clox
        bin/lox

The problem is that now you need to update all your includes.

/* #include "vm.h" */
#include "lib/vm.h"

Under normal circumstances this would also be fine. However this project has a weird constraint: you may want to copy code directly from the book unedited. Even if you are typing everything by hand for better memory retention, it’s a pretty useful troubleshooting tool to be able to keep copypasting to verify that you mistyped something. Fortunately clang supports a command line flag that adds directories to the include path so we can both put the files where we want and include them how we want. It could be argued this is a little hacky but this solution made the codebase the most comfortable for me to work in. Just place add an argument prefix the directory you want to add with -I and you can keep the includes in your C code exactly they way they’re written in the book.

clox:
        clang -Isrc/lib -o bin/lox src/main.c src/lib/vm.c src/lib/compiler.c
run: clox
        bin/lox

Clang Macro flags

A couple chapters in the author defines a macro that toggles logging of vm instructions as they get processed.

#define DEBUG_TRACE_EXECUTION

As the author states, this is only for debugging purposes, end users wouldn’t want all of this dumped into the console while they work. If you want you can just comment and uncomment this as necessary. A better solution is to use make tasks to create a different build. Clang makes this easy with define flags that let you define or undefine macros. Comment out the debug flags in your code, and pass the name of your macro prefixed with -D to clang when you invoke it in your make task.

clox:
        clang -Isrc/lib -o bin/lox src/main.c src/lib/vm.c src/lib/compiler.c
run: clox
        bin/lox
clox-debug:
        clang -DDEBUG_TRACE_EXECUTION -Isrc/lib -o bin/lox-debug src/main.c src/lib/vm.c src/lib/compiler.c
run-debug: clox-debug
        bin/lox-debug

Variables and wildcards in Makefiles

Listing out the library files is already pretty inconvenient, but now the list has been duplicated. Ideally we could automatically grab every file in src/lib and pass it to the compiler for every entry point. First let’s solve the duplication problem by adding a variable to the Makefile.

LIB=src/lib/vm.c src/lib/compiler.c
clox:
        clang -Isrc/lib -o bin/lox src/main.c $(LIB)
run: clox
        bin/lox
clox-debug:
        clang -DDEBUG_TRACE_EXECUTION -Isrc/lib -o bin/lox-debug src/main.c $(LIB)
run-debug: clox-debug
        bin/lox-debug

This is already a pretty big improvement. Let’s move some more of the arguments to clang into variables.

APP=src/main.c
LIB=src/lib/vm.c src/lib/compiler.c
FLAGS=-Isrc/lib

DEBUG_FLAGS=-DDEBUG_TRACE_EXECUTION

OUTPUT=bin/lox
OUTPUT_DEBUG=bin/lox

clox:
        clang -o $(OUTPUT) $(FLAGS) $(LIB) $(APP)
run: clox
        $(OUTPUT)
clox-debug:
        clang -o $(OUTPUT_DEBUG) $(DEBUG_FLAGS) $(FLAGS) $(LIB) $(APP)
run-debug: clox-debug
        $(OUTPUT_DEBUG)

This is in my opinion much cleaner and easier to configure. Now use a wildcard so we can finally avoid listing out all the library files.

APP=src/main.c
LIB=$(wildcard src/lib/*.c)
FLAGS=-Isrc/lib

DEBUG_FLAGS=-DDEBUG_TRACE_EXECUTION

OUTPUT=bin/lox
OUTPUT_DEBUG=bin/lox

clox:
        clang -o $(OUTPUT) $(FLAGS) $(LIB) $(APP)
run: clox
        $(OUTPUT)
clox-debug:
        clang -o $(OUTPUT_DEBUG) $(DEBUG_FLAGS) $(FLAGS) $(LIB) $(APP)
run-debug: clox-debug
        $(OUTPUT_DEBUG)

Passing through command line arguments

The lox binary can take a filename as a command line argument. We’ll need to update the Makefile a bit so that we can pass arguments to lox when we invoke it with make. We can define define a new variable at the top of the file called ARGS.

ARGS ?=

APP=src/main.c
LIB=$(wildcard src/lib/*.c)
FLAGS=-Isrc/lib

DEBUG_FLAGS=-DDEBUG_TRACE_EXECUTION

OUTPUT=bin/lox
OUTPUT_DEBUG=bin/lox

clox:
        clang -o $(OUTPUT) $(FLAGS) $(LIB) $(APP)
run: clox
        $(OUTPUT) $(ARGS)
clox-debug:
        clang -o $(OUTPUT_DEBUG) $(DEBUG_FLAGS) $(FLAGS) $(LIB) $(APP)
run-debug: clox-debug
        $(OUTPUT_DEBUG) $(ARGS)

The ?= in ARGS ?= allows conditional assignment, so it will default to vars passed on the command line if they are available. Leaving the right side blank assigns an empty string. Now you can invoke the run task with an argument like this:

make run ARGS=demo_program.lox

Conclusion

Now we’ve got a pretty functional setup. From here adding new tasks for tests etc. is pretty straightforward, and so is adding new flags.

The second half of Crafting Interpreters implements a new interpreter in C. As a high school student I was gifted the canonical text on C, “The C Programming Language”, (nicknamed K&R after the authors Brian Kernighan and Dennis Ritchie). I bounced off of it pretty hard. It’s been a gratifying experience to take another shot at learning the language.

Resources for Learning C

Coming from Javascript or Python, Java is, in my opinion, pretty easy to understand. It’s a little verbose but the semantics were ultimately familiar¹. However with C there were a few fundamental language features used frequently in the book that required me to dedicate some attention to learning them before I could be productive writing C myself vs just copying code out of the book. I resorted to using good old fashioned books to brush up on those features and that’s what I would recommend if you’re in the same boat.

My primary recommendation is “Modern C” by Jens Gustedt. Best practices for C and indeed the language itself have evolved over the decades since K&R was written. “Modern C” is also organized into sections for people with different experience levels sometimes covering the same language concept more than once in more depth each time. I think it’s very well organized for someone who wants to try and grab the language features she needs as quickly as possible at exactly the level of depth she needs at the moment.

The most famous book about C is the aforementioned K&R. I do think that this book is a really fascinating document. You can learn a lot about the mindset the authors of C had when the wrote it. While it’s definitely worth reading, I don’t think this is the best book for learning C today. It’s a great supplement to “Modern C” and I’ve known a few people who love the teaching style in this book.

A Few Features and Concepts

Here’s a quick survey of some of the things I needed to look up to get productive working through the exercises and extending the codebase.

Pointers and Memory Allocation

This is where C deviates the most heavily from other languages. C gives you basically one type of usable collection: fixed sized arrays². Other types of collections like growable array lists and stacks are built on top of pointers and memory allocation. If you’re not familiar with these features you will probably want to dedicate some time to brushing up on this. “Modern C” and K&R both cover this in a lot of detail and I used both to improve my understanding.

The Stack and the Heap³

Learning C it can be helpful to understand a little terminology about two different ways C programs store their variables in memory: stack memory and heap memory. You don’t necessarily need to understand how they work in depth to use the language, but stack memory and heap memory are referenced frequently in teaching materials about C. When you implement C Lox you’ll actually see analogous concepts used to manage the data in Lox programs too.

Stack Memory

Variables that you declare in your functions have a fixed size and they go on the stack. Integers, fixed length strings and array, pointers to addresses. The compiler knows at compile time how much space each function call will need from the stack (except for variable-length arrays, which are outside the scope of this article).

Heap Memory

When you allocate memory with malloc or realloc that goes on the heap. The heap can grow as you ask for more and more memory. This is the memory you’re worried about when you’re learning about memory management in C and the memory that would most likely be responsible for memory leaks.

Function Pointers

In Javascript and Python you can store functions in variables and pass them around. The analogous feature in C is the function pointer. These are used pretty heavily in the parser code in cLox to delegate to different functions when different operators show up in expressions. I found the syntax a little odd so it’s worth reading more about them for that reason, but I think they’re otherwise pretty intuitive.

This is a quick guide to the features from jUnit I used to test jLox. I used the bare minimum number of concepts I felt that I needed to write tests effectively, so this is not a comprehensive guide to jUnit, and not necessarily following best practices. I wrote most of my tests as integration tests once the full interpreter stack was available: feeding Lox code to the interpreter and using print statements to test the output. If I was implementing a real interpreter I would do a much more careful job testing the individual components with unit tests, but those integration tests were good enough to help debug issues that I inevitably introduced.

Test Cases

Test cases are placed in a test directory that sits next to main. Inside it has a directory structure that mirrors main. Tests should be placed in directories mirroring the files they test.

e.g. app/src/test/java/com/craftinginterpreters/lox/InterpreterTest.java

package com.craftinginterpreters.lox;

import org.junit.Test;

public class InterpreterTest {
    @Test public void testStuff() {}
}

Assertions

jUnit comes with a bunch of different types of assertions. I only used a few. They are all pretty self explanatory. Check out the docs for a list of assertion methods available to you: https://junit.org/junit4/javadoc/4.8/org/junit/Assert.html

app/src/test/java/com/craftinginterpreters/lox/InterpreterTest.java

package com.craftinginterpreters.lox;

import org.junit.Test;
import static org.junit.Assert.*;

public class InterpreterTest {
    @Test public void testAsserts() {
        assertEquals("true equals true", true, true);
        assertTrue("true is true", true);
        assertNotNull("true is not null", true);
    }
}

Testing Exceptions

I used one more type of assertion that I thought deserves a little attention. assertThrows allows you to test methods that throw exceptions by passing it a lambda.

app/src/test/java/com/craftinginterpreters/lox/InterpreterTest.java

package com.craftinginterpreters.lox;

import org.junit.Test;
import static org.junit.Assert.*;

import java.lang.UnsupportedOperationException;

public class InterpreterTest {
    @Test public void testException() {
        UnsupportedOperationException exception =
            assertThrows(UnsupportedOperationException.class, () -> {
                throw new UnsupportedOperationException("Not implemented");
            });

        assertEquals("Not implemented", exception.getMessage());
    }
}

Print output

Once you have the interpreter set up, I found that the easiest way to test output from the system was to call the built-in Lox function print, and then check that the correct strings were sent to standard output. It turns out it’s pretty easy to capture the standard output and test it with jUnit.

app/src/test/java/com/craftinginterpreters/lox/InterpreterTest.java

import java.io.ByteArrayOutputStream;
import java.io.PrintStream;
import java.util.List;
import org.junit.Test;
import org.junit.Before;

public class InterpreterTest {
    private final PrintStream standardOut = System.out;
    private final ByteArrayOutputStream outputStreamCaptor = new ByteArrayOutputStream();

    @Before
    public void setUp() {
        System.setOut(new PrintStream(outputStreamCaptor));
    }

    @Test public void testPrint() {
        System.out.println("lol");
        assertEquals("lol", outputStreamCaptor.toString().trim());
    }
}

I got most of this information from here if you are curious: https://www.baeldung.com/java-testing-system-out-println

Focusing a Test case

If you only want to run a single test case (especially useful if you are doing print debugging and drowning in output!), you can easily do that by passing a command line argument. If you are not using gradle, I assume it’s pretty similar:

./gradlew test --tests InterpreterTest.testPrint

Setting up a Mock Clock

At some point Crafting Interpreters will have you implement a clock function that grabs the system time and converts it to seconds. As I struggled to find a way to override and mock out System.currentTimeMillis(), I realized that instead I could just set up the interpreter to accept a clock object as a constructor argument (defaulting to the system clock if none is provided), so I could use a mock clock in the tests. This is kinda overkill, but if you wanted to have this in your life here are the clock classes I used to set that up. Just create an instance of whichever clock and call clock.getCurrentTimeSeconds() instead of calling System.currentTimeMillis() directly inside the interpreter code. Inside your tests pass a TestClock to your interpreter, and then you can control the time with tick().

app/src/main/java/com/craftinginterpreters/lox/LoxClock.java

package com.craftinginterpreters.lox;

import java.util.List;

interface LoxClock {
    double getCurrentTimeSeconds();
}

app/src/main/java/com/craftinginterpreters/lox/SystemClock.java

package com.craftinginterpreters.lox;

import java.util.List;

class SystemClock implements LoxClock {
    public double getCurrentTimeSeconds() {
        return (double) System.currentTimeMillis() / 1000.0;
    }
}

app/src/main/java/com/craftinginterpreters/lox/TestClock.java

package com.craftinginterpreters.lox;

import java.util.List;

class TestClock implements LoxClock {
    int currentTime = 0;

    public double getCurrentTimeSeconds() {
        return (double)currentTime;
    }

    public void tick() {
        tick(1);
    }

    public void tick(int seconds) {
        currentTime += seconds;
    }
}

“Or Whatever Setup”

This is the second of 3 posts on getting set up to work with Java as I go through the book Crafting Interpreters. In this Post we’re covering build tools. There are a lot of ways you could go about building and executing a Java project. I preferred to avoid using an IDE to set up my project, so I needed to choose that setup for myself.

One could, like the cavemen of olden days, just use javac to build her code, and java to run it. Besides the fact that simply building and executing your programs with the java executables is less straightforward than one might prefer, there are some additional benefits to using a build tool. I personally do not want to work on a project of any size without the ability to write tests, and writing tests usually entails dependencies. A few chapters in, the book has you write a code generation tool to get around Java’s tragic verbosity, and its nice to have a task runner and a separate build pipeline to deal with this tool. I’ll explain how I used Gradle to do each of these things.

Gradle vs Maven vs Ant

There are a few different popular build tools to consider when starting a new Java project, and I spent almost zero time evaluating them. I decided to use Gradle because:

1. Gradle is the standard tool for building Android projects
2. Gradle uses Kotlin (or Groovy) instead of XML as the configuration language
3. I have used Gradle before

Are there good reasons to choose one of the others? Maybe!

Getting Started

Install java, install gradle

I won’t go into detail here because this is system dependent. I used my system package manager to install java 21 and the global gradle that you’ll use to initialize your project. If you need more assitance getting gradle installed, check this out: https://gradle.org/install/.

Create a new gradle project

Once you have gradle installed, you can get your project set up pretty easily with this command:

gradle init --package com.craftinginterpreters.lox
            --project-name lox
            --java-version 21
            --test-framework junit
            --dsl kotlin
            --type java-application
            --no-incubating
            --no-split-project

More about gradle init: https://docs.gradle.org/current/userguide/build_init_plugin.html

If you leave out all the options to gradle init, it will prompt you to make a selection for each of these.

• package: Creates directory structure that matches the package name
• project-name: Self explanatory. Not sure if this has any effect on anything other than just being the nanme of the project
• java-version: The version of Java you want to use.
• test-framework: By default it prompts you to choose “junit” which really means junit 4. That’s what I chose, but I was unaware that there is a junit 5, and you can select it here by choosing “jupiter-junit” instead. Since I did not find this out until later, I’ll use junit 4 for my examples in the next post, but it should be easy enough to translate to junit 5 if you prefer to use that instead.
• dsl: You can choose to configure the project with Kotlin or Groovy. Since its plausible I may want to learn Kotlin in the future, I preferred to choose that. The downside is that it’s often easier to find examples online written in Groovy.
• type: This allows you to pick a project template. java-appliation is “a command line application written in java” which is exactly what we want. More about the application plugin: https://docs.gradle.org/current/userguide/application_plugin.html
• no-incubating: You can choose between the bleeding edge and stable versions of Gradle. I decided to go with the stable version since it was pretty likely that I would be going through this book on and off over a long period of time. I want the build to “just work” even if I haven’t touched it in 6 months.
• no-split-project: This allows you to have multiple projects with separate build files. We don’t really need that for a project this small.

Once you’ve run gradle init configured to your liking, you should have java files named App.java and AppTest.java created inside this directory structure. I would change them both to Lox.java and LoxTest.java and rename the classes within. You’ll also want to checkout app/build.gradle.kts and change the main class to match. This will make it a little easier to follow along with the book which names its entrypoint Lox instead of App.

Check this out for more info about initializing Gradle: https://docs.gradle.org/current/userguide/build_init_plugin.html

Dependencies

You probably won’t need to add any more dependencies, since gradle init will install the test framework for you. If you updated the main class to Lox, your app/build.gradle.kts should look like this:

/*
 * This file was generated by the Gradle 'init' task.
 *
 * This generated file contains a sample Java application project to get you started.
 * For more details on building Java & JVM projects, please refer to https://docs.gradle.org/8.12/userguide/building_java_projects.html in the Gradle documentation.
 */

plugins {
    // Apply the application plugin to add support for building a CLI application in Java.
    application
}

repositories {
    // Use Maven Central for resolving dependencies.
    mavenCentral()
}

dependencies {
    // Use JUnit test framework.
    testImplementation(libs.junit)

    // This dependency is used by the application.
    implementation(libs.guava)
}

// Apply a specific Java toolchain to ease working on different environments.
java {
    toolchain {
        languageVersion = JavaLanguageVersion.of(21)
    }
}

application {
    // Define the main class for the application.
    mainClass = "com.craftinginterpreters.lox.Lox"
}

You’ll notice that gradle added an additional dependency called guava. Guava is a kind of supplemental standard library, with lots of neat doodads and utilities. Code from the book won’t take advantage of that library, so you can go ahead and deleted the depenedency. You can read more about guava here if you are curious: https://github.com/google/guava

The Gradle Wrapper

gradle init creates a script called gradlew in the root of your project. This is the primary way you’ll interact with gradle. It will install the correct version of gradle that your project requires if its missing, and use that gradle to run your tasks. This makes it really easy to get set up on another machine.

Unit Tests

You can run your tests with ./gradlew test. I often run them with the --info flag which will allow exceptions thrown and System.out.println to get printed out in the console for you. I’ll discuss writing test cases in a future post.

Configuring tasks

You’ll primarily interact with gradle by running tasks. So far we’ve used tasks that come with gradle like init, and tasks that gradle defines for your project like run and test. The behavior of existing tasks can be configured to our liking. In addition we can create our own tasks.

Configuring run

java-application comes with a run task that executes our application. jLox will accept input in two different forms:

• From a .lox sources files
• As a REPL accepting strings of lox code from standard input

By default gradle will not wire up standard input to your tasks, so we’ll need to enable that explicitly for the REPL to work properly. To test this out you can setup your Lox.java file to look like this:

/*
 * This source file was generated by the Gradle 'init' task
 */
package com.craftinginterpreters.lox;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;

public class Lox {
    public static void main(String[] args) throws IOException {
        System.out.println("Input string:");

              InputStreamReader input = new InputStreamReader(System.in);
              BufferedReader reader = new BufferedReader(input);

              String line = reader.readLine();
              System.out.println("Line:");
              System.out.println(line);
    }
}

If you try ./gradlew run without modifying the run task it should print null without waiting for your input.

Add this to your build.gradle.kts to configure the run task:

tasks.named<JavaExec>("run") {
    // Needed for the repl
    standardInput = System.`in`
}

Now try running that again. It should wait for your input now, but you’ll also see a bunch of “gradle stuff” that interrupts the input experience.

Cleaning up the command line output

A few command line arguments can clean up the output from gradle to give you a better experience interacting with the REPL:

./gradlew run -q --console=plain

Custom tasks

Later in the book we write a code generation tool. We can create custom task to run this code. You can create a placeholder file to test this out and put it in app/src/main/java/com/craftinginterpreters/tool/GenerateAst.java:

package com.craftinginterpreters.tool;

public class GenerateAst {
    public static void main(String[] args) {
        System.out.println("Generate AST");
    }
}

Then add this to your build.gradle.kts:

tasks.register<JavaExec>("generateAst") {
    classpath = sourceSets.named("tool").get().runtimeClasspath
    mainClass.set("com.craftinginterpreters.tool.GenerateAst")
}

You now run this tasks like so:

./gradlew app:generateAst

Passing command line arguments to your tasks

Crafting Interperters will instruct you to accept a directory name as a command line argument where you will output the generated code. Let’s go ahead and test that out too. Let’s tweak the code for GenerateAst.java a bit to print out the first command line arg you pass:

package com.craftinginterpreters.tool;

public class GenerateAst {
    public static void main(String[] args) {
        System.out.println("Generate AST to file: " + args[0]);
    }
}

And now you can pass an argument to your task like so:

./gradlew app:generateAst --args="$PWD/app/src/main/java/com/craftinginterpreters/lox"

Using sourceSets to isolate unrelated code

Ultimately Crafting Interpreters will instruct you to fill this file with code that outputs a new source file with interfaces you’ll import into the rest of your code. One problem you may run into is that if you add new methods to these interfaces, it will immediately break the rest of your code because you’ve failed to implement them yet. But what if you need to run app:generateAst again? app:generateAst shouldn’t really depend on the rest of the code, and indeed Gradle does give us a tool to deal with this called sourceSets.

The first thing we’re going to want to do is to move the tool directory out of app/src/main into its own top level directory app/src/tool. I tried as much as possible to avoid modifying the code in the book, but I think we’re fighting against the tides if we try to keep source code that should be isolated inside of main. Let’s move app/src/main/java/com/craftinginterpreters/tool/GenerateAst.java to app/src/tool/java/com/craftinginterpreters/tool/GenerateAst.java.

Add this sourceSets section and update your generateAst task in your build.gradle.kts file:

sourceSets {
    create("tool") {
        java {
            srcDir("src/tool")
        }
    }
}

tasks.register<JavaExec>("generateAst") {
    classpath = sourceSets.named("tool").get().runtimeClasspath
    mainClass.set("com.craftinginterpreters.tool.GenerateAst")
}

Now you can break whatever you want in the primary codebase without affecting the tool.

Pass command line arguments from the task configuration

A quick quality of life improvement is to configure the default command line arguments for generateAst right in the task configuration.

Update your task configuration like so:

tasks.register<JavaExec>("generateAst") {
    classpath = sourceSets.named("tool").get().runtimeClasspath
    mainClass.set("com.craftinginterpreters.tool.GenerateAst")
    args = listOf(project.file("app/src/main/java/com/craftinginterpreters/lox").getAbsolutePath())
}

Now you can just run:

./gradlew app:generateAst

And you retain the ability to pass the args explicitly if you want.

Task dependencies

Ok was that really necessary? You’ve got ctrl-r after all. We can build on this by making generateAst a task dependency for run and test. Now you don’t have to worry about forgetting to run generateAst at all. You can just run run and test and be confident that all of the generated code is in place.

Using dependsOn for task dependencies

The first thing I tried was to have run and test depend on generateAst directly. Unfortunately this approach has a problem. What we want is to run generateAst not just before run or test execute their code. We actually need it to run before run and test compile their code.

Gradle’s compilation tasks

It turns out, each SourceSet comes with its own set of compilation tasks. For the “main” SourceSet, that task is called “compileJava”. (For the tool SourceSet, the task gets a suffix: “compileJavaTool”. This is a common convention in gradle, where suffixes are omitted when things refer to the “main” SourceSet.) To make sure that the compilation of the “main” SourceSet is dependent on generateAst, let’s fix our build file like so:

val generateAst by tasks.register<JavaExec>("generateAst") {
    group = "build"
    description = "Generates source code using the AST tool"

    classpath = sourceSets.named("tool").get().runtimeClasspath
    mainClass.set("com.craftinginterpreters.tool.GenerateAst")
    args = listOf(project.file("src/main/java/com/craftinginterpreters/lox").getAbsolutePath())
}

tasks.named("compileJava") {
    dependsOn(generateAst)
}

Conceptually, this is a better approach anyway. The code we generate with generateAst should always be generated before will compile “main”, and any future tasks that depend on “main” will get to take advantage of our tool.

fin.

Here’s the finished build.gradle.kts file:

/*
 * This file was generated by the Gradle 'init' task.
 *
 * This generated file contains a sample Java application project to get you started.
 * For more details on building Java & JVM projects, please refer to https://docs.gradle.org/8.12/userguide/building_java_projects.html in the Gradle documentation.
 */

plugins {
    // Apply the application plugin to add support for building a CLI application in Java.
    application
}

repositories {
    // Use Maven Central for resolving dependencies.
    mavenCentral()
}

dependencies {
    // Use JUnit test framework.
    testImplementation(libs.junit)
}

// Apply a specific Java toolchain to ease working on different environments.
java {
    toolchain {
        languageVersion = JavaLanguageVersion.of(21)
    }
}

tasks.named<JavaExec>("run") {
    // Needed for the repl
    standardInput = System.`in`
    dependsOn("generateAst")
}

tasks.named("test") {
    dependsOn("generateAst")
}

tasks.register<JavaExec>("generateAst") {
    classpath = sourceSets.named("tool").get().runtimeClasspath
    mainClass.set("com.craftinginterpreters.tool.GenerateAst")
    args = listOf(project.file("app/src/main/java/com/craftinginterpreters/lox").getAbsolutePath())
}

sourceSets {
    create("tool") {
        java {
            srcDir("src/tool")
        }
    }
}

application {
    // Define the main class for the application.
    mainClass = "com.craftinginterpreters.lox.Lox"
}

At this point I’m pretty satisfied with the project set up. Now we’ve got:

• Unit tests
• Easy to build and run the project
• Control the amount of log output
• Code generation that runs automatically whenever we build

These quality of life improvements improved my experience working through the Java section of the book quite a bit. We love Gradle, we’re all gradle-pilled now.

Crafting Interpreters is divided into two parts: the first half uses Java to build an interpreter, while the second uses to C for a bytecode compiler. One of the things that drew me to the book was that, in addition to learning how to implement programming languages, you also pick up two different programming languages along the way. This post documents some observations about the language itself that stuck out to me when I learned it.

A Note on IDEs

Most people I’ve known in a professional context would use a Java specific IDE to work with the language. I’ve used IntelliJ and its cousin Android Studio in the past. If you have no editor preference at all I think IntelliJ will make your life the easiest. That said, I am not using an IDE to go through this book, and I think it’s totally feasible to use any editor you want to work on a Java project this size.

Package names and Directory Structure

Introductory documentation and tutorials for language beginners often focus on the syntax of the language in question. In my experience, that is rarely where I get stuck. Almost always I run into problems when I try to make the transition beyond a toy project consisting of a single file. With Java one of the first sticking points I ran into was the directory structure. The directory structure of even small java projects always seemed intimidating to me. Actually the way Java projects are usually organized makes a lot of sense if you know what’s going on.

Designed for scale

Java is a language that’s popular at huge companies with huge codebases. New programmers may assume that these companies stick with Java simply out of habit. In fact some of the things that seem unnecessary, like Java’s clunky directory naming, are “features” not “bugs” when you are working at a larger scale.

Maven Standard Directory Layout

Maven is a one of a few competing build tools for working on Java projects. As well as providing the tools to build Java projects, the Maven project also publishes standard conventions for organizing Java projects, and a directory of published Java projects you can use as dependencies. For my project I chose to use Gradle to build my project, which also follows the Maven Standard Directory Layout. It’s worth checking out the documentation yourself, but this layout describes the very top level of the directory structure. We’ll cover the most relevant one real quick:

Not so bad right? Gradle projects further nest this inside an “app” directory, setting you up for success if you need to add any subprojects to your repository.

What’s the deal with “com”

Ok that’s all good and well, but you may remain perplexed when you open up src/main and are greeted with “com”, followed by a seemingly interminable number of useless directories before you find any actual code. There are three principles at work here.

1. Inside src/main the directories should may directly to package names
2. Package names should ideally be unique across all your dependencies
3. If you always name your packages after a website name that you own, you are unlikely to conflict with other package names.

This might seem extreme to you, but consider how awful working with dependencies in environments like nodejs and python can be. This is just one way that the Java ecosystem seeks to make sure things “just work” even on enormous projects. I’ve come to appreciate the reasoning here, even though I do find all the empty directories pretty irritating. Now you’ve got something like this:

Collections

Modern languages usually provide convenient syntax for working with lists and dictionaries. Java instead provides libraries with lots of different collection implementations, optimized for various use cases. I won’t go through all the usefull ones here, but you should expect when the book decides to create a list of object instances, you’re going to be importing something like java.util.List (to type your variables) and java.util.ArrayList (the actual class you instantiate), instead of using []. Actual quote from the book: “We must also chant a minor prayer to the Java verbosity gods since we are using ArrayList now.”

Running the code

The java executable has a ton of options for configuring its behavior. If you plan to use the executable directly, you’re going to have to learn about some of them. I preferred to allow my build tools to handle this for me. Build tools like Gradle and Maven provide a lot of additional benefits anyway, which I will cover in the next post.

UnsupportedOperationException

When a class implements an interface in Java, Java gets upset when any of the methods from the interface are missing. While this is sort of the entire point of interfaces, sometimes it is helpful to run your code before you are done matching the interface. Of course you could mock out the missing methods however you want, but after a little Googling it seemed like the most canonical way to throw an exception for this case is UnsupportedOperationException.

package com.craftinginterpreters.lox;

import java.lang.UnsupportedOperationException;

public class Interpreter implements Expr.Visitor<Object> {
    @Override
    public Void visitFunctionStmt(Stmt.Function stmt) {
        throw new UnsupportedOperationException("Not implemented");
    }
}