I wrote chess_bpa in a short period of time (less than a week) as an experiment in using ChatGPT to generate code in 2024.
Now I’m coming back to the project in an effort to see if there’s anything to re-use there or improve, or to learn from it.
Even though this was just an experimental hack, it is a chance to think about how I would approach a codebase that needed to be maintained or revived. Industry trends seem to suggest that there will be a lot of vibe-coded experiments that end up in production and suddenly have to be maintained seriously. How do we transition a vibe-coded experiment into something maintainable? Ideally the transition should be smooth, painless, and gradual, without a big up-front cost. What should we do with vibe-coded experiments that worked but don’t need to be maintained? Is there some value we can get out of a one-off codebase even if we won’t run it again?
The great thing about chess_bpa is that it’s all in one file, bpa.c, and the file reads from top to bottom and was written from top to bottom.
My workflow at the beginning was to paste the entire file into ChatGPT and ask it to write the next part of the code. We could call this implicit context management. The context for the LLM when generating a line, or a function, is simply everything above that function in the file.
This often leads to a bottom-up structure in the codebase, because implementing a function and then calling it works better than calling a function and then implementing it. This is not particular to LLMs, but is a general feature of software development. The details of implementation may cause an interface to change, so if you start by implementing something before you use it, then you save time rewriting the use of it later. If you implement it and then don’t need it, then you wasted time; avoiding this is the advantage of the top-down approach. With the LLM, the tradeoff is whether it is better to design the thing from the top, fix the interfaces, and then have the LLM write the implementations, or to write implementations first and let those determine the interfaces that you use to build the top-level interface that the program exposes to the world.
There is a raw transcript of my first review of the code, where I took a few notes along the way. I found out the structure of the entire codebase, and the build system, and the amount and location of all the code and its structure, all in a few minutes, because the project is so small, 2691 lines in one C file, half of which may be comments. Even though the project is tiny, 2691 lines is still too much to read line by line. In real products, of course, reading all the code line by line would be even more impossible, but we have to get oriented somehow. It is really appealing to start in a codebase by getting a view of everything that is there. If one is responsible for maintaining something, it is essential. We have to know everything that is there before we can even know if it is working. So one set of questions is about using an LLM to understand a codebase. Because we are talking about sending an entire codebase to an LLM, it can be expensive in tokens, but it’s an interesting question.
At this point we could go in a few different directions:
In short, when we approach an old experimental codebase we are looking for anything we want to take away from it that has value, even if that is an insight into why our approach did not work, or a negative result.
I have some ongoing writing on chess heuristics, and for this writing it would be nice to generate some visualizations, run some queries over a chess database, and consult an engine for accurate evaluations of board positions. All of these things were also done in the chess_bpa project. So I have a concrete question in looking at this codebase: is it more effective for me to adapt, re-use, or scrap this code in order to do the same tasks? If I want to sit down and run a query over a chess database, get evaluations from an engine, and generate some kind of visualizations, should I start with a Jupyter notebook, or an IDE, revive this code I have in a non-trendy language, or write something from scratch? By making any particular tech stack easier to get started with, LLMs actually worsen the paradox of choice. Previously I would have been put off from adopting Python if I didn’t know Python, but now I can use Python even if I don’t know it, because LLMs know it so well. Should we start writing everything in Python because LLMs are better at it?
One of the weirdest things for me about switching from writing code to writing natural language is that it promises to separate you from the question of what language your project is written in. In practice, this hasn’t happened yet, because LLMs will get things wrong, and the things that the LLM gets wrong will depend on the language, and the structure of the program may also depend on the language. Unless you can let the LLM handle the interface to the whole program, all your interactions with the build system, and your usage of any libraries, then you cannot let the LLM pick the programming language. But you could potentially scaffold out all of these language-specific details and implement the same program in multiple languages with minimal extra work.