In this post, I’m exploring the idea that using AI agents to map out the development of prominent open-source codebases can develop judgement in ways that would have required becoming a long-time contributor in the past. The kind of judgement I’m looking at is:

• Which interfaces and abstractions earn their place as a project evolves?
• Where do the trade-offs lie, even for sound decisions?
• How does the concurrency model hold up as things get complex and scale?
• What automation and processes can tame a project’s complexity?

This is the thesis I’m testing: While AI agents take with one hand by making hobby coding even less valuable, they can give with the other by helping us glean wisdom from open-source projects.

I chose to look at a Go project because while I’m not very familiar with the language, it's relevant to my current role as a platform engineer. Most of the technologies I use daily are written in Go. Notably: Docker, Kubernetes, OpenTofu (Terraform fork), and the OpenTelemetry collector. I’ve written some Go on weekends: Advent of Code and the kind of low-stakes projects that don't seem to matter now. I may have nearly a decade of commercial software engineering experience, but I lack Go-specific battle scars. For instance, I've never had someone complain that a Go interface I wrote years ago is now causing them problems, as I have with the public API of Python modules.

Below I ask an agent to summarise the state of etcd (the distributed key-value store underpinning Kubernetes) at key versions, then synthesise it into a narrative of how the system developed in response to user needs and production scale. This is a jumping-off point for exploration and thinking. I acknowledge the risk of regurgitating AI-generated text without truly understanding it, so I endeavour to explore the threads I write about independently.

What I Learned from etcd

I started with some AI-assisted git archaeology. I prompted² Claude Code (Opus 5) to generate a report on each of four etcd versions, v0.1.0 to v3.7.1. I read the individual reports, which were overwhelming because they contained more plausible observations than I could ever hope to verify. To find a thread running through all versions, I used a second prompt³, along with the reports, to generate a summary. The only thing I’ll directly quote from the summary is this table. It shows, at a high level, how the project grew and when each API version arrived and left.

Non-test, non-generated, non-vendored Go lines, measured per tag (git ls-tree -r <tag> filtered to *.go, excluding _test.go, *.pb.go, and third_party/, Godeps/, cmd/vendor/, */vendor/):

What stands out most in this table is that the Raft consensus implementation has had three homes: first as a vendored library, then in the repository, and finally spun out into a separate repository.

Spinning Out a Pure Raft library

Raft⁴ is the consensus algorithm that keeps the keys and values stored in etcd consistent across multiple nodes. Raft means production Kubernetes clusters can deploy their data store across multiple nodes in a highly available configuration. Raft also underpins many of the guarantees etcd provides.

A thread Claude identified from the initial v0.1.0 release to the recent v3.7.1 was the Raft consensus seam, which, as of today, connects the etcd server to a pure logic Raft library that leaves I/O to the consuming system. The same library, extracted from etcd, is used by other distributed databases including CockroachDB and Dgraph.

In v0.1.0, there was a different external Raft library, and its interface wasn’t as clean. Raft wrote to the file system to update the log; it also called time.Sleep. By v2.0.0, this was replaced with an in-repo implementation: the I/O was factored elsewhere and time was represented as a logical tick rather than interacting with a system clock. (When attempting to understand the new Raft library, my puny meatbag brain found the simplified key-value store example code more helpful than current etcd source code.)

At the time of writing, the Raft library README says:

Most Raft implementations have a monolithic design, including storage handling, messaging serialization, and network transport. This library instead follows a minimalistic design philosophy by only implementing the core raft algorithm. This minimalism buys flexibility, determinism, and performance.

When writing software professionally, I try to extract business logic so it doesn’t contain side effects and I/O, favouring pure functions and dependency injection. Even in commercial software, this can be hard to achieve, and the payoff is often limited. For something like Raft, which has a PhD thesis devoted to it, I can imagine it was much harder. The payoff is greater too, as other distributed databases can and do use the system regardless of how they implement I/O.

Not all software components can be pure, however. In etcd, they didn't delete the messy parts; they moved them to the etcd server.

The Messy Details

In version 0.1.0, etcd passed go-raft a directory path to write log entries to. This was paired with a callback function that applied log entries to etcd’s state.

type Log struct {
 ApplyFunc   func(Command) (interface{}, error)
 file        *os.File
 path        string
 entries     []*LogEntry
 results     []*logResult
 commitIndex uint64
 ...
}

Source code here.

The first external Raft library owned I/O and also owned the messy business of ensuring that all log entries were applied to the store. This is simpler in some ways because go-raft could commit changes to the log and apply them to the data store in the same function, managing concurrency with the same mutex.

From version 2.0.0, even before it became a separate library, the Raft component contained only the business rules for keeping multiple nodes’ state consistent. The etcd server was now responsible for ensuring that the database’s state reflected the WAL entries. At this point, though, not much could go wrong. As in version 0.1.0, the state lived in memory, so if a node died, it would rebuild the state from a combination of a snapshot and the WAL.

In version 3.0.0, the database state became durable. The release notes mention “new storage engine”. The part of this engine that stored key-value data on disk was Bolt (a fork of. While this change gave etcd better transaction isolation (the I in ACID) with MVCC, it also created a new durable record on every node: the key-value store in addition to the existing Raft WAL. This introduced a new problem when rebuilding an etcd node’s state: the etcd server needed to track the WAL offset that had so far been committed to the node’s key-value store. etcd committed to Bolt in batches (every 100ms or 10,000 writes), so after a crash the database could lag behind the log, and if it replayed already-applied entries, it could apply them twice.

This is the commit message for saving that offset to Bolt: “etcdserver: save consistent index into v3 storage […] This helps to recover consistent index when restart in the future.” Consistent index doesn’t mean that the index is consistent, so much as that it’s the index of the latest committed Raft WAL entry that’s been written to the etcd server’s on-disk state. At this point, the design is sound: the index is written in the same Bolt transaction as the data, so after a crash both are at the same point, and etcd knows exactly where to resume. From v3 onwards, data consistency rests on this being true in all situations.

Version 3.0.0 burdened the etcd server with ensuring that the consistent index agreed with what WAL entries had in fact been applied. It introduced a new class of potential bug which eventually manifested, resulting in error reports like this one. It is every Kubernetes user’s nightmare: “Some keys exists on node x and y but not on node z, and some on z but not on the other two, e.g. different pods list would be returned from different servers.” The root cause turned out to be this PR, which aimed to ensure the consistent index was saved on each commit. Unfortunately, there were a couple of unintended consequences:

1. The in-memory consistent index value was shared state, read by any commit, not just the one carrying the matching data.
2. This in-memory value was updated before the transaction was committed to the key-value state (Bolt), creating a window in which a concurrent commit could persist it (e.g. the periodic commit that runs every 100ms).

Saving the index and saving the data it pertained to no longer comprised an atomic operation.

Postmortem, Pragmatism and Priorities

The refactor that caused the bug was merged in May 2021 and shipped in v3.5.0 a month later. The first report of diverging etcd nodes arrived in December 2021. A maintainer confirmed the corruption in March 2022; the maintainers sent a public statement to the etcd and Kubernetes mailing lists, and the fix shipped in v3.5.3 in April 2022.

This wasn't the first time this class of bug surfaced. In August 2016, two months after 3.0.0 shipped, a maintainer fixed the same failure mode in a different place (#6247). A snapshot routine could save the consistent index to disk after apply had updated it, but before apply had written the matching data. Their commit message describes the 2022 bug almost exactly, and ends by noting there were other places that needed the same care, “but let us fix this first”. A 2020 bug in the auth store (#11652) was another variation on the theme. The danger was known and named for five years. Each fix patched a particular symptom. Nothing turned “the index is only saved alongside its data” into something the code enforced.

The 3.5.3 fix (#13854) finally did that. It split the in-memory value in two: an applying index for the entry currently in progress, and a consistent index updated only while holding the transaction lock, so it can reach disk only with the data it describes.

The postmortem says the maintainers were “not aware [of] any cases of users reporting a data corruption in production environment”. They took the issue seriously all the same; databases survive on being trustworthy. What’s interesting about the postmortem is that most of the issues they identified were related to people and processes rather than code. Processes around releases had grown lax. Maintainers had moved on. The automated test suite was flaky and incomplete. They reproduced the bug partly by luck, because one maintainer's /tmp was mounted to a real disk rather than memory. The fix took over a week and involved multiple attempts.

The work that followed the postmortem is what I find most instructive. They haven't undertaken a grand refactor of the EtcdServer god struct, which survives to this day, nor any of the other code smells the agent identified in its reports. I think this is because they had more important priorities. Instead, they built machinery for confidence. They reworked corruption checking. A robustness suite now injects failures and checks every run's history for linearisability. The suite’s README lists known correctness bugs, including those it has caught.

Conclusion

This post ended up being more about software engineering than Go. I think that's fine. The judgement I was hunting for is mostly language-agnostic.

Some of this journey was reassuring in a way. Even a high-profile open source project that many depend on has TODO comments in the codebase, and fossils of old implementations that haven’t been cleaned up. Even here, it took a critical bug to restore release discipline and reliable tests. There was no instant fix. It took careful, coordinated work, much of which was later automated.

I don’t want to overstate what I’ve learned. The initial raw AI reports were hard to follow and simply covered too much ground. I was out of my depth, and all I had were reams of plausible observations. The meat of this post came from a dialogue with Claude, where I tried to cut away everything but the most essential threads of etcd’s story. Without that, this post would likely never have been published.

In domains I’m paid to work in, I guide the agent as much as it guides me. While jumping into an unfamiliar domain and relying heavily on the agent is fine for an experimental piece of writing like this, I wouldn’t feel comfortable being so far out of my depth if I were building software for others. Domain expertise still matters. After reading this, you wouldn't hire me to build a new database in Go. You’d pick a database engineer who can recite Designing Data-Intensive Applications from cover to cover and has shipped Go in production. I’ve gained some second-hand wisdom, and some trivia to bore people with. More valuably, I’ve gained a lasting suspicion of any invariant that lives only in tribal knowledge. Such invariants are even more dangerous now that agents can be dropped into a complex codebase with none of that context.

Was it more useful than a toy coding project? Yes, but not by much. There are no shortcuts to expertise. Real fluency in a language still means working up from basic to advanced material until one can read unfamiliar code and run its logic in one's head, rather than relying on a maintainer's postmortem or an AI agent to explain it. That skill matters more now, not less. We're shipping more code we haven’t read, and sooner or later, even teams running fully automated software factories will need someone who can dive into an unfamiliar codebase and understand what it's doing. The scarcer that skill becomes, the more it will be worth.

So the premise I started with was only half right. Hand-written side projects haven’t entirely lost their value. They're just less impressive. They’re like low-intensity cycling: nobody’s excited by it, but it builds the cardiovascular base that makes the hard sessions possible. Digging through a codebase like etcd is the hard session. Without the base, AI was my e-bike. I covered the distance, but my legs didn’t do most of the work. That’s fine for getting somewhere, but not ideal for getting fitter.

1. I’ve chosen the word senior because “expert” is hard to pin down. For instance, managers regularly judge seniority in the industry, though the technical bar for “senior” differs by org; some levelling frameworks try to formalise it. These frameworks cover much more than technical skill, but insofar as they cover the technical side, this from the senior (L50) section of this framework is a good example: “Has the skills to proficiently work on complex and unknown problems under pressure (e.g. debugging complex production issues at speed, or leading the resolution of critical or major incidents)”. Notice it doesn’t mention writing code. This kind of skill isn't automatable yet (though with enough telemetry, an agent can help investigate production issues).

2. The first prompt, run at v0.1.0, v2.0.0, v3.0.0 and v3.7.1: “Explore this codebase and outline its public APIs, its fundamental architecture and software design choices in a Markdown document: etcd-0.1.0.md. References to source code should support all points; priority should be given to where important abstractions are declared/implemented and where they earn their place and, if relevant, where they break down.”

3. The second prompt, used to generate a summary: “Read the markdown documents in the working tree. These all outline the design and trade-offs of etcd at a particular point in time. Generate another document that traces the development of the five most important systems or abstractions over the lifetime of the project. Once one area has been chosen, the source code at the four documented versions should be double-checked and referenced. Pay attention to trade-offs, management of complexity and how abstractions break down as well as design successes.”

4. I’m not a distributed systems expert, so I’m not going to try to summarise Raft beyond saying all nodes maintain write-ahead logs from which state can be built, all writes go through a single leader, and it employs a leadership election mechanism to survive faults. For a more detailed discussion of the Raft algorithm, you can read the original paper. To get some intuition of how leadership and elections work, there’s a web-based simulation here.

John Searle devised his famous Chinese Room thought experiment in response to what he called strong AI. This wasn’t the idea of human-like or superhuman artificial general intelligence (AGI) tech CEOs rave about, but the idea that machines that behave as though they think truly think. One reason these ideas were entertained at the time was that the frontier of artificial intelligence was symbolic rather than numerical, meaning that systems were structured according to formal grammar and logic¹. A scholar who thinks about rules underpinning language all day may be attracted to the idea of a rule-based system that itself thinks. It’s less compelling to imagine that a concatenation of matrix multiplications and other computations understands the sequences of tokens it builds.

I won’t go into detail about the thought experiment, though I’d encourage you to read it in the original article. The basic idea is that someone can produce convincing text-based answers to questions in a language they know nothing about without understanding the questions or answers. All they need are exhaustive formal rules in a language they do know about, on how to respond to sequences of — to them — meaningless symbols.

Because software that lived up to Searle’s room was never actually built², it benefits from an undefined scale. A single person can plausibly follow the potentially vast set of rules because no one formalised them. Due to the number of computations involved in frontier LLM inference and the limited speed at which people can perform arithmetic operations, the subject would need to live for around 380,000 years to generate a token in a conversation with a chatbot. ³ Constrained by human lifetimes, the computation would need to be carried out by a deeply rigid and dogmatic civilisation organised around the purpose of ingesting and producing holy numbers. Even such a dystopia, similar in size to 19th-century England, using telegraph to communicate would produce only five tokens a year. ⁴

I'm not the first to scale this idea up to population size. Ned Block got there before Searle even wrote his paper: his 1978 “China Brain” imagined the entire population of China, linked by radio, simulating a brain neuron by neuron. Searle himself later answered the connectionists with a “Chinese Gym”: a hall of monolingual English speakers hand-cranking a neural network. So the move I'm making here is a well-worn one. What's changed is that the machine on the other side of the analogy now actually exists, which means the scale is no longer a matter of taste: we can put numbers on it.

Let's circle back to Searle's point about incomprehension. I'll try to sketch it out. The scribes or human computers carrying out the calculation wouldn't know what the numbers meant or what they corresponded to. Those privileged few who could access the vaults where the consecrated weight numbers and numbers for divined and generated tokens were stored would not know what symbols the numbers mapped to. The high-ranking bureaucrats who organised inference computation would blindly follow a kind of bureaucratic scripture: their command and control infrastructure operating in the spirit of the Soviet factories that, graded on tonnage, produced chandeliers heavy enough to tear down ceilings, and nail plants that met their quotas with a handful of railway-spike-sized nails.

Scientists caught researching the inference process or building mechanical computers would be publicly executed as heretics. Those who pursued their research in secret would live lonely lives, questioning the consistency of the system they were enmeshed in but lacking a solid basis for any conclusions. Even during the great conjunctions that occur once a generation, the seers who divined new input token numbers from the entrails of sacrificial animals would not know what the numbers meant.

So there it is: an updated Chinese Room, in all its absurdity. It works better as an illustration of just how big large language models are than as a thought experiment. Searle's room has the advantage of not being a closed system, where questions and answers can simply pass under the door. My civilisation has to smuggle its inputs in through an oracle: the divination.

1. This is now known as GOFAI: good old-fashioned AI, a term coined by the philosopher John Haugeland in 1985. Think of SHRDLU, which parsed typed English commands to manipulate a virtual world of coloured blocks; MYCIN, which diagnosed blood infections by chaining through roughly 600 hand-written if-then rules; or Cyc, a decades-long attempt to hand-code common sense as millions of logical assertions. Or ELIZA, the 1966 chatbot that parodied a Rogerian psychotherapist using simple pattern-matching rules, fooling some users into feeling understood, which is precisely the illusion Searle was attacking.

2. This kind of symbolic AI Searle was talking about didn’t make it past R&D, let alone being deployed commercially on a planetary scale like generative AI is today. Despite some early hype, research into expert systems and the like was eventually abandoned, leading to the second AI winter starting in the late 1980s.

3. GPT-4.5, which passed the Turing test most of the time in a 2025 study, had hundreds of billions of parameters; OpenAI never disclosed a figure, but following an educated guess let’s say 600 billion active parameters. Assuming two computations for each parameter to generate a token, it would take an immortal, tireless human 3,169 years to complete a single layer of the transformer that drove GPT-4.5. A single token would take at least 380,000 years. This is based on 0.1 arithmetic operations per second for a human scribe.

4. A significant amount of computation that a transformer does is embarrassingly parallel: the vanilla feed-forward neural network that follows the self-attention magic. This could be parallelised across a galaxy if it weren’t for the speed of light nullifying any parallel gains with a huge communication cost. Barring a mega-structure like a Dyson sphere, the distances signals would need to travel within a solar system for each layer of a transformer would be prohibitive. Within a single small landmass, however, batches of computation can be sent by telegraph to outlying population centres very quickly. A core city of 1 million scribes working around the clock in two 12-hour shifts, 10 million more spread across the land, could generate a GPT-4.5 token in just over two months. See the diagram below for an illustration of this. See also this website for a useful illustration of how transformers and LLMs work.

=====================================================================================

                    O N E   T O K E N   B Y   T E L E G R A P H

             the forward pass at city scale: a capital, its towns, a wire

=====================================================================================

  model      600 B parameters, 120 sequential layers, d_model 20,412
  payload    one hidden state = 20,412 values x 16 bit  ~=  327 kbit
  the wire   100 bit/s telegraph, signals at 0.7c -- the 300 km costs a
             millisecond and a half of flight and hours of keying; at this
             scale bandwidth is the tyrant, never the distance


         _
        /_\       +
   +   |(+)|     _|_
  /_\  | : |    /   \      +                ~ 300 km of wire ~
  | |  | : |   /     \    /_\   ~        ,       ,       ,       ,          +
  | |  | : |  /       \   | |  ~        -+-------+-------+-------+-        /_\
 _| |_ | : | |         |  | |   _  _     |       |       |       |   /\    | |   /\
 | : | | : | |  :: ::  | _| |_|    |     |       |       |       |  /__\  _| |_ /__\
 | : | | : | |  :: ::  | | : || [] |     |       |       |       |  |[]|  | : | |[]|
_|___|_|___|_|_________|_|___||____|_____|_______|_______|_______|__|__|__|___|_|__|_

            C A P I T A L                     the telegraph            T O W N S
       1,000,000 core scribes         100 bit/s, signals at 0.7c   10,000,000 scribes


-------------------------------------------------------------------------------------
                     ONE LAYER  --  k of 120, strictly in order
-------------------------------------------------------------------------------------

  clock         CAPITAL (core)                             TOWNS (periphery)
            1e6 scribes, 10 km wide                     1e7 scribes, 300 km out
    .                  |                                           |
 +0.0 h    +-----------+-----------+                               |
    .      | SELF-ATTENTION        |                               |
    .      | half the city on duty |                               |
    .      | (two 12-hour shifts)  |                               |
    .      | core only: every desk |                               |
    .      | may read the whole    |                               |
    .      | token census          |                               |
    .      |  9.4 h arithmetic     |                               |
    .      | +1.8 h keying across  |                               |
    .      |  the city's own wires |                               |
+11.3 h    +-----------+-----------+                               |
    .                  |                                           |
    .                  |          MLP inputs for layer k           |
+11.3 h                |------- 327 kbit ~ 0.9 h of keying ------->|
    .                  |       (300 km of light-lag: 1.4 ms)       |
    .                  |                                           |
+12.2 h    +-----------+-----------+                 +-------------+-------------+
    .      | MLP: capital's share  |                 | MLP FEED-FORWARD          |
    .      | the on-duty half joins|     .  .  .     | ten million desks, each   |
    .      | 0.9 h                 |     in step     | on its own slice; no town |
    .      |                       |     .  .  .     | consults another  (0.9 h) |
+13.0 h    +-----------+-----------+                 +-------------+-------------+
    .                  |                                           |
    .                  |          MLP outputs for layer k          |
+13.0 h                |<------ 327 kbit ~ 0.9 h of keying --------|
    .                  |                                           |
+13.9 h    +-----------+-----------+                               |
    .      | residual add + norm   |                               |
    .      | file the layer record |                               |
    .      | (negligible)          |                               |
    .      +-----------+-----------+                               |
    .                  |                                           |
    v                  v                                           v
             hidden state to layer                       towns sit idle until
             k+1: back to the top                      the next dispatch keys in


-------------------------------------------------------------------------------------
  THE BILL
-------------------------------------------------------------------------------------

  one layer     9.4 + 1.8 + 0.9 + 0.9 + 0.9   =   13.9 hours
  one token     13.95 h  x  120 layers        =   69.7 days
  the split     74% labour / 26% communication

  the capital keys and computes around the clock: two 12-hour shifts, half a
  million desks always lit. the towns need no relief -- each works 0.9 h in
  every 13.9, and the wire itself schedules their rest.

  every communication hour is serialization -- a fist tapping 327,000 bits
  onto the wire at 100 per second. propagation never shows up on the bill:
  the light-lag budget for the whole token is about a third of a second.

=====================================================================================

Abiogenis is a theorised process whereby nonliving matter develops into prototypical forms of living matter: simple self-replicating structures, RNA, proto-cells. I find this transition from chemistry to biology very interesting, because:

• It could explain the origins of life on Earth (and elsewhere)
• It should be possible to simulate without resorting to extremely simplified models, like rigid body physics for animals.
• The molecular/cellular scale can have an otherworldly aesthetic, which may work with the generative music part of the project.

What am I planning to build?

Here are the rough ideas as they stand: I want to build some web-based software that allows users to navigate space and time in a primordial soup. Time may be folded into space, and then the four dimensions (instead of the usual three) projected onto two. Instead of exerting agency, the viewer will be a passive observer. Generative music, the sophistication of which follows that of the simulated molecules, will be a big part of the viewer’s experience.

I plan to use WebGL and WebAssembly in a relatively low-level manner to handle potentially complex 3D scenes and heavy computation for the state. I’ll probably use Rust, as this is the only low-level language I’ve built anything with. Rust as a technical choice may fall over because I also intend to use as much AI tooling as possible. An older and simpler language like C is likely easier on LLMs and to have greater representation in the training data. I’ll adapt to the constraints of the tools. I’ll mainly rely on AI to augment my basic mathematical knowledge and utilize my experience in building software to ensure that the code is structured sanely. I'll likely write the music-related code in JavaScript and use something like SuperSonic as a synthesis engine; I'm imagining a few subtractive voices and lots of reverb.

While I could have chosen a less ambitious project to test AI coding tools, I feel that enough of my time is spent being pragmatic in my professional life. I don’t want to use my limited spare time to create a React app, Django HTTP API, or some Go microservices.

Why?

I haven’t read much philosophy since attempting a journal article on AI and the machinic unconscious over six years ago. I’ve been busy and have only read humanities books that strongly piqued my interest. One of these was a review of consciousness in recent analytical philosophy. I found the clear communication and modest claims of analytical philosophy refreshing, especially after doing two arts degrees where continental thought is far more popular (with notable exceptions). At worst, continental and continental-inspired writing is a sequence of complex sentences telling the reader how a constellation of different abstract concepts relate to each-other with no definitions or reference to facts. However, there are trade-offs: analytical philosophy can be dull. Much of the book could be summed up as follows: The hard problem of consciousness is indeed hard. There is overwhelming evidence that mental states correlate with brain states; we can’t agree on the details. Let’s find a palatable way of representing the situation that best fits our everyday experience of embodied rational agency. Here is a menagerie of -isms and some thought experiments.

If I wanted satisfaction, I should have read a book about a problem that can be solved. This brings me to abiogenesis. While no one has produced biology from non-living chemicals in a lab, there are a lot of models and theories for how it might work, and more on how primitive life works today. Attempting to make a computer simulation of how consciousness functions in relation to brain activity would be impossible. Where would you start? Large language models have passed the Turing test, but like all machine learning models, we have no reason to believe they’re any more conscious than a wheel of Camembert. We could set about tracing information as it flows through systems that populate a person’s mind; it might be interesting social science, but it wouldn’t address the hard problem of consciousness. Chemical and biological processes are more compatible than physical and mental substances, properties, whatever.

The final thing I’d like to share is a passage from a book that hit the sweet spot between the analytical and continental: Manuel de Landa’s A Thousand Years of Nonlinear History. He adapts and applies some of the most notoriously impenetrable continental ideas from Deleuze and Guattari in a concrete, historically grounded way. The book talks about the material flows that drive history through geology, biology, and language. I remembered a particular passage when thinking about abiogenesis:

[…] positive feedback links between these flows began to form closed-circuits: antimarket money flowed into mining regions and intensified coal production and iron extraction, which triggered a flow of mechanical energy (steam), which in turn triggered a flow of cotton textiles, which created the flow of profits that financed further experimentation with coal, iron and steam technology. These loops of triggers and flows were behind the explosive urban growth in England between 1750 and 1850. […] And precisely these autocatalytic loops were what kept this self-organised structure going.”

Autocatalysis is a likely step in the path from non-living to living matter. I find the idea that this kind of process also emerges at the macro level quite inspiring.

• Writers are at the mercy of the business, which can start charging for things and generally enshitifiying the platform at any time. They could paywall writers' content, like Medium does.
• My blog was associated with the general reputation of content on the platform, which wasn't always favourable.
• I don't only want to write about software engineering, and that is the focus of the platform.

My reasons for choosing a self-hosted WriteFreely are essentially the inverse of the above points, with a couple more:

• Writefreely has an ActivityPub integration, so my blog can be a true citizen of the Fediverse.
• WriteFreely is very simple. It's a Go monolith that can (and in the case of this site, does) run with a SQLite database. It uses Markdown, so no fancy text-editing widgets.

How did I do it?

Now that I've outlined why, I can move on to the unashamedly geeky part of this post. My main objectives were to run Writefreely in a way that was both simple and cheap.

Simplicity

I got a lot of simplicity for free with this WriteFreely Docker image, which neatly encapsulates the initial configuration of WriteFreely as well as data migrations. The image doesn't leverage Writefreely's built-in automatic HTTPS. Luckily, another Go-based project, Caddy, does this, on top of being a solid reverse proxy.

Those of you who have used the likes of Apache or NGINX, but not Caddy, please take a moment to admire the two lines of configuration necessary to reverse proxy this blog:

simoncrowe.blog

reverse_proxy localhost:8080

Yes, this is all that's required to serve this blog and automatically provision TLS certificates using Let's Encrypt!

Frugality

The localhost origin in my Cadddyfile above isn't typical of a Docker-compose setup. It's a result of my trying to save money by not renting a public IPv4 address from Hetzner and only using a free IPv6 address. Caddy needed an internet-routable IPv6 address to get responses from Let's Encrypt's servers, and using the “host” network mode was the simplest way to achieve this.

The laptop I'm typing this on doesn't have an IPv6 address, so I couldn't SSH into the server, even if I wanted to. To get around this for HTTPS traffic, I used Cloudflare as a DNS registry and proxy. This means that simoncrowe.blog resolves to some of CloudFlare's v4 IPs. CloudFlare handles everything between the Hetzner IPv6 origin and its own dual-stack infrastructure, at no cost.

Trade-off time: My insistence on not paying the tiny IPv4 address fee has probably increased the complexity of this. The no SSH constraint is what motivated me to write a Terraform/OpenTofu module. Using the Hetzner web shell to configure even a simple setup like this would have been a miserable experience compared to SSH from the comfort of my laptop's terminal emulator. Because I've been lazy and put the database in the same machine as the HTTP server, I've already consigned the server to a long existence as a pet. Immutable Infrastructure is not happening here, so doing everything via the Hetzner web console and SSH would have been reasonable.

On the other hand, maybe my code will help someone else run their own WriteFreely. Here's the repo.

Cheap compute with a cloud API! What else can Hetzner do?

Since I managed to throw something functional together in a weekend, I'm wondering what one could do with Hetzner and IaC tooling like OpenTofu, given some time.

While I haven't explored their full networking and compute capabilities, the impression I got is that they'd be fine to run an early stage European startup's backend (with some caveats). Anything approaching a scale-up with global ambitions would be better off with a major public cloud provider. The main reason I say this is that they only have data centres in Germany, Finland, the US and Singapore (the latter two being rack-space in 3rd party data centres).

To me, Hetzner feels somewhere between a typical shared LAMP stack web hosting provider one might use for a Joomla/WordPress site and a mail server — and rudimentary AWS (mainly ec2, some s3 and recently a bit of route53). When creating VMs manually on Hetzner, you can install one piece of common software. Like the LAMP hosting, options include WordPress, but also more advanced things like Docker and Prometheus/Grafana. Unlike typical web hosting, you can provision infrastructure using an API, and you get root access to your servers. Rather quaintly, when a server spins up, you get an email with the default root password (luckily, sshd isn't running out of the box).

If I were a frugal European startup founder, the only thing that would stop me from choosing Hetzner is the lack of managed databases. The VMs could easily run stateless workloads where new software versions are rolled out smoothly using OpenTofu lifecycle policies. While one could run a database on such a VM, one would need to be very careful about backups and learn database administration. A tolerable middle ground might be running production databases using AWS RDS/Aurora in AWS regions geographically close to Hetzner locations; sure, there would be some extra latency, but it might be manageable.

I'll keep an eye on Hetzner. I'm impressed with how straightforward everything is with them. At the top of my wish list would be managed stateful services like Postgres and Kubernetes, a container registry, and guarantees around availability zones (high availability in general). Serverless versions of every imaginable thing, I can live without.