Not long ago I wrote about my search for a process (as in daemons) management tool to manage webservers, build watch processes, etc. across many different projects. I was still undecided at that point, but have since landed on Pitchfork. It does a bit more than I need and it is well-maintained. See that previous post for a discussion of that and other options that I considered.

Having settled on Pitchfork for process management, I next needed to choose a task runner. The contenders were: make, just, mise, and go-task/task. I’m not going to do a detailed breakdown, but here’s a post from another site that does. It turns out I came to the same decision they did, for mostly the same reasons. The winner was Mise.

You may notice that both tools, Pitchfork and Mise, come from the same developer. This is a bit of a risk. If the developer (Jeff Dickey) decides they’re no longer interested in maintaining these projects, then I’m suddenly looking for two tool replacements instead of one. But right now, looking at Jeff’s bio, income from his sponsors, support from his employer, etc., I’m really not too concerned.

I picked the top 19 most active repos that I have and migrated all of them to Pitchfork and Mise. Having that consistency across repos has made context switching much easier. It’s been great so far.

Around the same time, I also began using a few new skills. In addition to Pitchfork and Mise skills, I added a markdown formatter and my own writing skill. The writing skill comes with an LLM cliche linter tool as well. During the same migration, I ran the markdown formatter over all my READMEs, so now they’re easier for me to scan and find things in too.

I’ll add just a bit of brutal honesty here at the end in case you made it this far. Whenever I have a little extra time, or I’m just procrastinating on something, I tend to either a) tidy up, or b) try to improve the process. That was exactly the inspiration for this migration. Cheers!

The inertia problem

When I start working on a project, I open a terminal and start a few processes. The web server. A background job worker. The asset watcher that recompiles my JavaScript. Maybe a locally-running database service. Each one might get its own terminal tab, and each tab has a command I have to remember, or dig out of a README, or other docs.

Since I work across multiple projects in different languages, with different stacks, this gets expensive fast. Not expensive in terms of CPU or memory, but in terms of mental overhead. Every project has its own conventions: make dev, ./bin/dev, docker compose up, or something else entirely. You spend more time than you’d like just overcoming the inertia to get back to a running state every time you switch projects.

A process manager gives you a single command to start everything your project needs. And if you can standardize on one tool across all your projects (🤞), you eliminate that context-switching tax entirely: cd project && [tool] start works everywhere, every time.

That’s the goal, so let’s look at the options. (Or just scroll down to the comparison table and the verdict if you want to skip the narrative.)

But before I dive in, it’s worth being clear about what I’m trying to solve for here. Most frameworks give you ways to run individual commands natively: npm run dev, mix phx.server, bundle exec rails server. Those are great and you should use them. Your Procfile entries will often call them directly. A process manager sits on top of those commands and coordinates them, starting all your services at once and multiplexing their output into a single view. The two approaches are complementary, not competing.

There’s also a related but separate problem. Task runners like Make, Rake, Just, and npm scripts handle short-lived commands like running database migrations, executing tests, or triggering builds. You will still reach for those tools for that kind of work, and there’s a whole separate conversation to be had about achieving consistency there across projects. This post is focused specifically on the process management side: starting and keeping multiple long-lived development services running together.

It all started with a Procfile

In 2011, David Dollar released Foreman, a Ruby gem for managing multiple processes during development. The central idea was the Procfile: a plain text file that declares your process types, one per line.

web: bin/rails server
worker: bundle exec sidekiq
assets: npm run watch

Heroku adopted the format and popularized it. Today it’s the closest thing the development world has to a standard for declaring process types. Ports of Foreman exist for Python (Honcho), Go (goreman, forego), and more. Those ports largely replicate the original feature set, so I won’t review them separately, but the ecosystem shows how durable the Procfile idea has been.

What followed were tools that kept the Procfile format but went well beyond what Foreman originally offered. Here’s where things stand today.

The tools

Foreman

Foreman is where it all started for me, and it still does its job. Install the gem, drop a Procfile in your project, and run foreman start. It multiplexes the output of all your processes into a single stream, prefixed with the process name and a timestamp.

gem install foreman
foreman start

One unique feature is that Foreman can export your Procfile to init system formats like systemd, upstart, launchd, and runit. If you want to manage production processes the same way you manage development ones, that export path is a nice convenience. None of the other tools in this review offer it.

Pros: Mature and battle-tested. Dead simple to adopt. Procfile export to init systems is unique among these tools. Wide adoption means good documentation and community support.

Cons: It’s a Ruby gem, which adds a runtime dependency for non-Ruby projects. The bigger issue is how it handles process output. Processes detect they are not writing to a real terminal and suppress their colored output. If you’ve ever run rails server through Foreman and wondered why the logs look flat and grey, that’s why.

Hivemind

Hivemind was built by the team at Evil Martians to fix exactly that problem. It uses a PTY (pseudo-terminal) to capture process output, which means processes think they are writing to a real terminal. So colors work and log output behaves the way you’d expect.

It’s a standalone Go binary with no runtime dependencies beyond the binary itself. Install it with Homebrew or download the binary directly.

brew install hivemind
hivemind

Hivemind is deliberately minimal. It reads your Procfile, starts your processes, and gets out of the way. The README itself will tell you: if you want more features, check out Overmind.

Pros: Standalone binary with no language runtime required. Fixes the color output problem. Extremely easy to adopt, with the same Procfile format and simpler installation than Foreman for non-Ruby projects.

Cons: No interactive process management. If a process crashes, the whole group goes down. There’s no way to restart a single process without restarting everything.

Overmind

Overmind is Hivemind’s more capable sibling, also from Evil Martians. The key difference is that Overmind runs your processes inside a tmux session, which gives it capabilities no other Procfile tool matches.

You can connect directly to any process and interact with it:

overmind connect web

You can restart a single process without touching the others:

overmind restart sidekiq

You can tell Overmind that certain processes are allowed to exit without bringing everything else down, which is useful for one-time setup tasks like database migrations. Auto-restart on crash is configurable per process, and port assignment is automatic and consistent so you can reference one process’s port from another’s configuration.

The tmux integration turns out to be more than a dependency. If you’ve ever wanted to connect to a running Rails process to use the console, or poke at a webpack dev server that’s behaving strangely, overmind connect gets you there without restarting anything.

Pros: Full PTY support, so colors work. Individual process control without affecting the group. Auto-restart on crash. Port management per process. Highly configurable via environment variables so your preferences apply globally across every project.

Cons: Requires tmux. If tmux isn’t already part of your workflow, this is a real dependency, not just a binary download. Nested tmux sessions are manageable if you’re already inside tmux, but worth knowing about upfront.

Overitall

Overitall (oit) takes a different angle entirely. Where the previous tools focus on running processes and multiplexing their output, Overitall treats logs as a first-class concern. It’s a TUI (terminal user interface) that combines process management with what amounts to a built-in log viewer.

You still use a Procfile for defining processes, but a .overitall.toml config file can also point at standalone log files rather than just process output. This is useful for services that write to a file instead of stdout, or for situations where you want to correlate a running process’s output with a framework’s own log file.

The TUI gives you real-time search (regex-capable), per-process visibility toggles, time-based navigation (:goto -5m), and trace detection that can spot correlation IDs and UUIDs in your logs and filter to a specific trace. For log-heavy development workflows, this is a meaningful upgrade over watching interleaved output scroll by.

There’s also an AI integration story: run oit skill install and Overitall installs a skill into .claude/skills/oit/ (or the Cursor equivalent) that teaches your AI assistant how to control the running TUI from the command line. You can then ask Claude to check recent errors, restart a process, or search the logs without leaving your editor.

Pros: Log management as a first-class feature. Regex filtering, time navigation, and trace detection. Interactive TUI with per-process visibility control. AI skill for Claude Code and Cursor. Available on both macOS and Linux.

Cons: Requires an additional .overitall.toml config file alongside the Procfile. The TUI has a learning curve compared to a simple process runner. It’s a newer project and still maturing.

Pitchfork

Pitchfork is the most ambitious tool in this list, and also the most different. Where everything else runs your processes in the foreground and stops them when you close the terminal, Pitchfork manages daemons: background processes that keep running after you walk away. A supervisor process handles them, and you interact with it via a CLI.

The configuration is a pitchfork.toml file:

[daemons.redis]
run = "redis-server --port 6379"
auto = ["start", "stop"]

[daemons.api]
run = "npm run dev"
auto = ["start", "stop"]
depends = ["redis"]
ready_http = "http://localhost:3000/health"

The auto = ["start", "stop"] setting is the most interesting feature for the cross-project consistency goal. With shell hooks activated, Pitchfork starts those daemons automatically when you cd into the project directory and stops them when you leave. You never have to remember to start your services. You just navigate to your project and they’re there.

The feature list goes further: dependency ordering (so Redis starts before the API that needs it), ready checks (so Pitchfork waits until a health endpoint actually responds before declaring a service up), file watching for auto-restart, cron scheduling, resource limits, and a web UI. Pitchfork also ships a built-in MCP server. One JSON config block in your Claude Desktop or Cursor settings wires your AI assistant directly into daemon management.

Color output is available but requires opting in with pty = true per daemon. Since processes run in the background by default, PTY is not on automatically, but enabling it is a one-line change per daemon.

Pros: The most powerful feature set in this review by a wide margin. Auto-start and auto-stop via shell hooks directly address the cross-project consistency problem. Built-in MCP server for AI integration. Dependency ordering, ready checks, and cron scheduling included. Language-agnostic standalone binary.

Cons: Steeper learning curve and a new configuration format to learn. The daemon model is a different mental model than a foreground process runner. You check on services rather than watching them scroll past. Color output requires explicit opt-in per daemon. Actively developed and not yet at 1.0.

Feature comparison

Which tool is right for you?

Choose Foreman if your project is already Ruby-based and you want the simplest possible setup. The gem installs alongside your other dependencies, and the ecosystem familiarity is hard to beat. It’s also the right call if you need to export process definitions to init system formats for production use, since no other tool here does that.

Choose Hivemind if you want a no-frills Procfile runner that works for any project in any language, with proper color output. It’s an easy, zero-cost, drop-in replacement for Foreman for those that don’t need individual process control.

Choose Overmind if you live in tmux and want the ability to connect to, inspect, and restart individual processes without tearing down your whole stack. Once you’ve used overmind connect to drop into a running process’s window, it’s probably hard to let that go.

Choose Overitall if log management is a first-class concern in your workflow. Noisy logs from multiple services, distributed traces to chase, correlation IDs to filter on: this is where oit’s search, filtering, and time navigation start to pay off. I can see this being really useful in a more complex $work environment. The AI skill integration is also a practical bonus if you use Claude Code or Cursor heavily.

Choose Pitchfork if you want services that follow you between projects and start automatically without any deliberate action on your part. The daemon model requires a mental shift, but it has a nice TUI (and Web UI) and it’s the most direct answer to the cross-project consistency problem in this roundup.

The verdict

If you’re looking for a single tool to carry across every project regardless of language or stack, the choice comes down to how much you want to invest in configuration upfront versus how much friction you want to eliminate long-term.

For a lightweight universal runner, Hivemind is the pragmatic choice. Single binary, no runtime dependencies, correct color output, zero learning curve. Drop a Procfile in a project and you’re running. The limitation of no individual process control rarely matters until it suddenly does, and when it does, you know where to go next.

If you are already a tmux user, Overmind is the easy upgrade over Hivemind. Same binary story, same Procfile format, but you gain individual process control and the ability to interact directly with any running process. The additional capability costs almost nothing if tmux is already in your toolkit.

The most interesting answer for the cross-project consistency problem, though, is Pitchfork. Shell hooks that auto-start and auto-stop services as you move between directories solve the problem at a deeper level than any foreground runner can. You stop managing process startup entirely, and your services are just there when you need them. That comes with more upfront configuration and a different operational model, but if you work across many projects and that investment appeals to you, Pitchfork is the tool that most ambitiously reframes what a development process manager can be.

For myself, I’m honestly not sure where I’ll land quite yet. I have a soft spot for the simplicity of Hivemind, I’m not a frequent tmux user so Overmind’s advantages are less compelling, and Pitchfork’s daemon model is intriguing but requires extra work for me and my AI integration.

AI lets you ship faster than ever. That’s great. But six months later, you’re staring at your own codebase wondering who wrote this. Why is this service doing that? Why did you pick this library over the obvious one? What were you thinking?

The problem usually isn’t the code. It’s that AI accelerates the building without capturing the thinking. You evaluate options, make tradeoffs, reject approaches, and move on. The decision happens in a chat window and then it’s gone. What’s left is code without context, and that debt (a form of cognitive debt) compounds quietly until it becomes a real problem.

Though it might help, the best fix isn’t to slow down. It’s to build a few lightweight documentation habits that will hold up in an AI-assisted workflow.

The new documentation problem

Traditional documentation advice assumes you understand everything you’re writing. You wrote the code, you know why it works the way it does, you just need to write it down. AI-assisted development changes that assumption in a couple of ways.

First, you move faster. When you’re evaluating three AI-generated implementations in ten minutes instead of writing one over an hour, there’s almost no natural pause where documentation feels like the obvious next step. The decision-making is compressed, and so is the documentation.

Second, git history gets less useful. It still tells you what changed, but it doesn’t tell you that you picked this background job library because the other one had a deployment constraint you needed to avoid. It doesn’t tell you which of four Stripe webhook implementations the AI suggested, or why you picked that one. It doesn’t tell you that you almost went a completely different direction.

The gap is context. And that’s exactly what these habits are designed to capture.

Document decisions, not implementations

The highest-leverage documentation habit is writing down why, not what. Code already shows what. What it can’t show is the thinking that produced it.

Keep a docs/decisions.md file. No formal template required. No ADR ceremony. Just a running list of decisions, what options you considered, what you ruled out and why, and what you landed on. A few sentences per decision is enough.

Some examples of the kind of thing worth capturing:

• Why you chose Sidekiq over Solid Queue (maybe it was a deployment constraint, maybe it was familiarity, maybe you tried Solid Queue and hit a weird edge case)
• Which Stripe webhook implementation you kept after the AI suggested three different approaches
• Why the data model is shaped the way it is instead of the more obvious structure you considered first

This is especially useful in AI-assisted work because the evaluation often happens fast. You look at a few options, pick one, and move on. Without a quick note somewhere, that reasoning is gone the moment you close the chat.

The daily scratchpad

A scratchpad is not a journal. It’s not a changelog. It’s a place to think out loud while you’re working.

Drop a file at tmp/scratch.md (some people call it journal.md or dev-journal.md) and use it to capture rationale in the moment. What did you try? What didn’t work? How did you arrive at this solution? What were you unsure about?

It doesn’t need to be clean. It doesn’t need to be organized. It just needs to exist.

Git history will tell future you that something changed on a Tuesday in March. The scratchpad will tell future you that you were choosing between two approaches, that one of them had a subtle concurrency problem you figured out mid-afternoon, and that the solution you landed on was a compromise you felt okay about. That’s the stuff that can really help when you return to a project three months later and can’t quite remember what you were even trying to do.

Let AI explain what it built

Sometimes you ship code you don’t fully understand yet. That’s not a great position to be in, but it’s a real one, especially when you’re learning a new language or framework alongside building something.

When that happens, ask the AI to explain what it just built.

Recently I was building a new SaaS product in Elixir while still actively learning the language. The initial architecture involved OTP supervision trees, GenServers, and some patterns that weren’t immediately obvious (to me). Rather than hoping the understanding would come later, asking the AI to write an explainer.md for the architecture made the whole thing legible. Later, an explainer.html with diagrams and a walkthrough made it even better.

You can do the same thing with an existing codebase you’re jumping into for the first time. Drop the relevant files into context and ask for an explainer. It’s faster than reverse-engineering the code yourself, and it gives you something you can update and refer back to as your understanding deepens.

My explainers live in docs/explainers/. They’re not precious documents. They can be rough, they can be wrong in places, and they should be updated as you learn more. But having them at all is the difference between a codebase that feels like yours and one that always feels a little foreign.

Keep your README human

AI tooling has introduced a temptation to stuff READMEs with prompts, agent instructions, and notes about which parts were AI-generated. Resist it.

The README is for humans. What is this project? How do I get it running? Where is the rest of the documentation? How do I contribute?

AI-specific context belongs elsewhere. Agent instructions and project rules for your AI assistant go in AGENTS.md (if you use Claude, a CLAUDE.md symlink pointing to it is handy). Plans and specs go in docs/plans/ and docs/specs/. Decisions go in docs/decisions.md. Explainers go in docs/explainers/.

The README is the front door. Keep it welcoming and keep it human.

Own the code, all of it

You don’t need to mark which lines an AI wrote.

I know that might feel counterintuitive. There’s a reasonable instinct to be transparent about your process. But there’s a difference between transparency and hedging. Marking code as AI-generated is a subtle way of distancing yourself from it, as if to say: if this breaks, that’s not entirely on me. But that’s not how accountability works.

You are responsible for every line of code you ship, regardless of how it was produced. If you don’t understand the code, you shouldn’t ship it. If you’re not sure you understand it, that’s what explainers are for.

The middle ground is being transparent about your development process at a high level. It’s no secret that most developers use AI assistance now, and saying so in your docs or your README is great. That’s transparency. What’s different is using it as a hedge. The tool you used to write the code doesn’t change your responsibility for the result.

The loop

Good documentation helps you understand your codebase. Understanding your codebase means you can give the AI better context when you’re building something new. Better context (human and AI) produces code that really fits the existing architecture. Code that fits is easier to explain and document. Round and round.

None of these habits are heavy. A few sentences in decisions.md. A rough note in scratch.md. An explainer file when you’re in unfamiliar territory. A README that stays focused on humans.

The payoff is that six months from now, when you come back to this project, it feels like yours. You remember what you were thinking. You know why things are the way they are. And you can keep building without spending the first week just figuring out what past-you was up to.

Bonus: automate it with a post-merge hook

All of these habits share a common enemy: you have to remember to do them. Remembering is fine when you’re focused, but documentation is usually the thing that slips when you’re rushing to ship.

Here’s a way to take the remembering out of it. A git post-merge hook fires automatically after every merge. You can use it to shell out to Claude Code in non-interactive mode and ask it to review and update your docs in light of what just changed.

The hook grabs the merge diff, hands it to Claude along with a focused prompt, and lets Claude update README.md, docs/decisions.md, any relevant explainers, and product docs. Changes are left unstaged so you can review them before committing. Nothing gets force-pushed. Nothing gets auto-committed. You stay in control.

Here’s the script:

#!/usr/bin/env bash
#
# .git/hooks/post-merge
#
# After merging to main, asks Claude Code to review and update documentation.
# Changes are left unstaged for your review.
#
# Setup:
#   cp post-merge .git/hooks/post-merge
#   chmod +x .git/hooks/post-merge
#
# Requires Claude Code CLI to be installed and authenticated.

set -euo pipefail

# Only run on main branch
CURRENT_BRANCH=$(git symbolic-ref --short HEAD)
if [ "$CURRENT_BRANCH" != "main" ]; then
  exit 0
fi

echo "📝 Checking documentation after merge to main..."

# Capture what changed in the merge so Claude has context
DIFF=$(git diff ORIG_HEAD..HEAD --stat)

# Build the prompt
PROMPT="A feature branch was just merged to main. Here is a summary of what changed:

$DIFF

Please review the following documentation and update anything that is out of date
or missing given the changes above:

- README.md
- docs/decisions.md (add an entry if a significant architectural decision was made)
- Any relevant files in docs/explainers/
- Any relevant product documentation in docs/

Do not modify AGENTS.md or any files in docs/plans/ or docs/specs/.

After making changes, print a brief summary of what you updated and why.
If nothing needed updating, say so."

# Run Claude Code in non-interactive mode with read/write permissions only
claude -p "$PROMPT" \
  --allowedTools "Read" "Write" "Glob" "Grep" \
  --max-turns 10

# Show which files were changed (if any)
DOC_CHANGES=$(git diff --name-only -- README.md 'docs/**' 2>/dev/null || true)

if [ -n "$DOC_CHANGES" ]; then
  echo ""
  echo "✅ Documentation updated. Review changes before committing:"
  echo "$DOC_CHANGES" | sed 's/^/   /'
  echo ""
  echo "   Run 'git diff' to review, then 'git add' and 'git commit' when ready."
else
  echo "✅ No documentation changes were necessary."
fi

Download here: post-merge

To install it, copy the script to .git/hooks/post-merge and make it executable with chmod +x .git/hooks/post-merge. Since .git/ isn’t tracked by git, you might want to keep the canonical copy in a hooks/ directory in your repo and add a setup step to your README.

A few things worth noting about the script:

The --allowedTools flag scopes Claude to read and write only. It can’t run arbitrary bash commands, can’t touch your git history, and can’t do anything outside the files you’ve pointed it at. Keeping the tool surface small is good hygiene for any automated Claude Code invocation.

The --max-turns 10 flag is a circuit breaker. It caps how many agentic turns Claude can take so a single post-merge hook can’t spin forever if something unexpected happens.

The prompt explicitly excludes AGENTS.md, docs/plans/, and docs/specs/. You don’t want a merge hook rewriting your agent instructions or your design plans. Keep the scope tight.

The --max-turns flag is a circuit breaker, but may need to be tuned based on how much the amount of work that needs to be done. If the circuit breaker is hit, the hook will exit with an error and leave the finished changes unstaged for manual review.

This is a starting point, not a finished product. Your project’s doc structure will differ, and you’ll want to tune the prompt to match. But the pattern holds: capture the diff, give Claude focused scope, leave changes unstaged, and let the human decide what to commit.

Creating a gem that writes your missing tests

No new project survives contact with the real world unscathed. We built TestGenAI, ran it on itself, and it worked well. Then we ran it on another codebase, and two things broke immediately. The fixes turned out to just as interesting as the original build.

This is a walkthrough of how the tool works and what we learned when we took it outside the greenhouse.

The code here is from TestGenAI, a working Ruby CLI gem you can install and run against your own codebase.

The pipeline

The pipeline has five stages:

1. Scan your codebase to find classes and methods without test coverage
2. Build context for each untested method
3. Generate tests using an LLM with the mechanically curated context
4. Validate that the generated tests run and pass
5. Collect the results

Each stage needs to be reliable enough that you can walk away and trust the process to complete. That means handling errors gracefully, providing clear output about what happened, and making it easy to pick up where things left off if something breaks.

Finding untested code

Before you can generate tests, you need to know what needs testing. The right approach depends on whether SimpleCov is available in the project.

If SimpleCov is set up, TestGenAI runs your test suite with COVERAGE=true, reads the resulting coverage/.resultset.json, and uses AST parsing to find methods where every executable line has zero hits. This scanner handles partially-tested files correctly. It reports individual methods that were never exercised, even if other methods in the same file have full coverage.

If SimpleCov isn’t available, the scanner falls back to checking whether a spec or test file exists for each source file. This approach is less accurate. A file tested only through integration tests or through specs for its subclasses will appear fully untested even if its methods are exercised constantly. The SimpleCov scanner is worth setting up.

Both scanners share the same underlying logic for locating methods in source files, which brings up something worth explaining.

Walking the AST

To locate methods, TestGenAI parses each Ruby source file into an abstract syntax tree and walks it recursively. The walker looks for :def and :defs nodes (instance and class methods), tracks the current class/module namespace, and records each method’s file, class, name, and line range.

That line range matters. The SimpleCov scanner uses it to check whether any executable lines in the method had zero hits. A nil in SimpleCov’s coverage array means a line isn’t executable, like a blank line, a comment, or an end. The scanner filters those out before checking for zeros, so it only flags methods where runnable code was never touched.

The parser compatibility problem

To parse Ruby, the gem relies on the parser gem. In older versions, you’d call Parser::CurrentRuby.parse(source) and get back an AST. This worked fine until Ruby 3.4, which switched its internal default parser to prism. Using Parser::CurrentRuby with Ruby 3.4 produces warnings, and in some configurations it fails entirely.

The prism project ships a compatibility shim, Prism::Translation::ParserCurrent, that produces the same AST node types as the old parser gem. The AST-walking code works unchanged. The only question is which one to load.

The solution is a small file that runs at load time and sets a constant:

module Testgenai
  if Gem::Version.new(RUBY_VERSION) >= Gem::Version.new("3.4")
    require "prism"
    CurrentParser = Prism::Translation::ParserCurrent
  else
    require "parser/current"
    CurrentParser = Parser::CurrentRuby
  end
end

The rest of the codebase calls CurrentParser.parse(source) and never thinks about which parser is underneath. The pattern of version check at load time, constant as the abstraction is a clean way to handle the same kind of compatibility gap you’ll run into whenever Ruby ships a significant internal change.

Context, generation, and validation

When you ask an LLM to write tests for a method, you can’t just paste in the method body. It needs the full class, the dependencies that file requires, examples of how the method is called elsewhere in the codebase, and existing test files it can match in style. Context quality is where quick-and-dirty AI test generators fall apart, too little and the tests don’t compile, too much and you hit token limits.

The generator builds a prompt from all of that, sends it to the LLM via the ruby_llm gem (which keeps the generator code provider-agnostic), and strips any markdown fences from the response before passing it to the validator.

The validator writes the code to a temp file, runs bundle exec rspec or the Minitest equivalent, and distinguishes between three outcomes: the file failed to load (syntax errors, undefined constants), the tests ran but failed, or the tests passed. Each outcome needs different handling. A file that doesn’t load gets deleted immediately because it’s useless. A file that runs but fails gets its error output fed back to the LLM for a retry.

The pipeline retries up to three times, passing failure details back each time. LLMs are reasonably good at fixing specific errors when told what went wrong. Undefined constants and wrong require paths almost always resolve in one retry. More complex failures, like incorrect behavior assumptions, may not, and those end up in a failed bucket for manual review.

Then we ran it on a real project

The first external test run revealed two problems, both on the same day.

The first: generated tests were syntactically valid, ran, and passed — but they looked nothing like the rest of the project’s test suite. Wrong authentication setup, wrong factory usage, helpers that weren’t available. Tests that technically pass but violate project conventions create a maintenance burden.

The second: the tool was silently destroying existing tests. When a spec file already existed at the output path, the pipeline would overwrite it with the newly generated content. Any tests already in that file were gone.

Both problems make complete sense in retrospect. The tool had only ever run on its own codebase, where it was always generating new files and where the conventions were deeply familiar to the model from the context it was seeing. A different project broke both assumptions.

Fixing the conventions gap

The core problem is that the LLM knows what your method does, but it doesn’t know how your team writes tests. It doesn’t know that you authenticate in before blocks a certain way, or that you have specific factory traits available, or that you’re not using rails-controller-testing so assigns isn’t an option.

The fix is a conventions system with two parts.

ConventionsExtractor scans your existing test files and pulls out mechanical facts: the most common authentication setup pattern, available factory traits from your factories directory, frequently stubbed objects, whether transactional fixtures are disabled and how cleanup is handled, and whether specific helpers are unavailable based on what’s in your Gemfile. These aren’t judgments, they’re observations extracted directly from the code.

ConventionsSynthesizer takes those raw facts and sends them to the LLM with a prompt asking it to write a concise conventions guide explaining the rule behind each pattern. The result is plain prose. Something like “authentication is handled in before blocks using session[:user_id] = rather than Devise helpers; use this pattern consistently.” That text gets prepended to every generation prompt.

The synthesized guide is cached to spec/conventions.md and invalidated automatically when spec files or Gemfiles change. Regenerating it costs one LLM call.

Enable it with the --conventions flag:

testgenai generate --provider anthropic --model claude-opus-4-7 --conventions

Add spec/conventions.md to your .gitignore. It’s a derived artifact and probably not something to check in.

Fixing the overwrite problem

The overwrite bug was straightforward to diagnose and subtle to fix correctly.

The naive fix would be to skip generation if a spec file already exists. That’s less than ideal. The point is to add tests for untested methods, and partially-tested files are the most common case.

Better behavior is to inject the generated tests into the existing file. The pipeline now reads existing content before doing anything else. It combines existing content with the newly generated code and validates the combined file. If it passes, the combined content is written to the spec file.

If it fails, the pipeline restores the original file exactly as it was and writes the generated-only code to a fallback path. A method in lib/payments/processor.rb that already has a spec/payments/processor_spec.rb would get its fallback at spec/payments/processor_context_spec.rb (using the method name to scope the filename). The failure output tells you where to find it:

  ✗ Payments::Processor#context failed after 3 attempt(s)
    → Generated tests saved to spec/payments/processor_context_spec.rb for manual review

You end up with your original tests intact and the generated attempt sitting somewhere you can look at it and decide what to do.

Running it

Install the gem and point it at your project:

gem install testgenai
cd your_project
testgenai generate --provider anthropic --model claude-opus-4-7

Or add it to your Gemfile in the development group and use bundle exec. Configuration can also come from environment variables:

export TESTGENAI_PROVIDER=anthropic
export TESTGENAI_MODEL=claude-opus-4-7
export ANTHROPIC_API_KEY=your_api_key
testgenai generate --conventions

Three diagnostic commands are available before you commit to a full run:

testgenai scan      # find untested methods without making any API calls
testgenai context   # show what context would be sent to the LLM for each method

scan gives you a picture of your coverage gaps. context is useful for understanding what the LLM will see before spending API credits.

The goal isn’t to replace the developer who understands the code and makes decisions about testing. It’s to handle the mechanical work: setting up describe blocks, wiring test data, writing happy-path coverage. Then you can spend your time on the parts that actually need your judgment. The second project taught us that “mechanical” is more context-dependent than it looks.

A few days ago I built a tool called JoyConf: a real-time audience feedback system that lets speakers see emoji reactions floating up in the corner of their presentation while they’re talking. It was a fun, simple idea and I was pretty excited about it.

I built it in Elixir and Phoenix LiveView, which was a deliberate choice. I mostly write Ruby these days, but this project felt like a good excuse to dig into Elixir and LiveView. Learn something new, build something useful. Two birds, one stone.

I drove the overall design and implementation planning, did active code review, and contributed everywhere I could. But for the Elixir and LiveView specifics, I leaned heavily on Claude. The syntax, architecture decisions, and debugging were Claude’s domain, because I simply didn’t know enough yet to own them. The tool worked and it seemed to work well. But when I got to the end and looked at the codebase, I realized I didn’t really understand the parts Claude had built. I had reviewed the code as carefully as I could, but reviewing code in a language you don’t know, implementing an unfamiliar architecture, only gets you so far. The understanding of those pieces had mostly stayed with Claude.

That’s cognitive debt. And LLMs are very good at generating it.

What cognitive debt actually is

Cognitive debt accumulates when you defer the thinking that should happen now. It’s different from technical debt, which is about the code itself (shortcuts taken, tests skipped, abstractions that didn’t quite work out). Cognitive debt is about the reasoning that never happened. The mental model that never got built. The decision that got made without being understood.

Like financial debt, it doesn’t feel like much at first. You’re moving fast, things are working, you’re shipping. The bill comes later, when you need to debug something you can’t reason about, extend a system you don’t understand, or explain a decision you never actually made. And to be clear, cognitive debt has been around long before LLMs, LLMs just magnify the problem.

LLMs make this disturbingly easy

LLM-generated code is mostly right. That’s what makes it dangerous.

If the code were obviously wrong, you’d catch it. You’d dig in, figure out what went wrong, learn something in the process. But LLM output is usually plausible, often correct, and just coherent enough that it passes the vibe check. You run the tests. They pass. You move on. The mental model of how it works never gets built, because you never needed it… until you do.

There’s a specific failure mode worth naming here. Using an LLM to move faster on things you understand is leverage. Using it to skip understanding altogether is debt. Those feel identical in the short term. Both result in code getting written. One leaves you with understanding you can build on; the other leaves you with output you’re stuck with.

And it catches everyone. Junior developers accept LLM output because they don’t know enough to question it. Senior developers accept it because they had a hundred PRs today and the code looks fine, so they assume it is fine. Both skip the reasoning step. The result is a codebase full of decisions nobody on the team can actually defend.

Back to JoyConf

When I realized I’d built something I didn’t fully understand, I asked Claude to write me an explainer document. Not a summary, but an actual explanation of the architecture, the key concepts, why certain decisions were made, how the pieces fit together. Something I could read, learn from, and come back to later.

It wasn’t a magic pill. I was starting from near zero with Elixir and LiveView, so one document didn’t make me an expert. But it meaningfully closed the gap. I understood the code better than I did before. I had something to refer back to. And I started to feel like the codebase was actually mine.

That experience shaped how I think about using LLMs for coding. The tool works fine. How you engage with it makes all the difference.

Practical ways to keep the debt in check

Ask for explanations before you accept the code. Don’t just run it. Ask the LLM to walk you through what it did and why. This takes an extra minute and catches a surprising number of cases where the code is technically correct but built on assumptions you don’t share.

Ask for an explainer document for bigger decisions. Architecture choices, non-obvious patterns, anything you’re going to need to live with for a while: ask the LLM to write it up in plain language. Keep it in the repo. Future you will thank present you.

Use Simon Willison’s “showboat” approach to document what was built. The showboat tool “creates executable demo documents that show and prove an agent’s work.” (kind of like a Jupyter notebook, but just markdown). The LLM walks through its output with explanation and context. It’s a great way to produce living documentation that captures not just what the code does, but why it was written that way. It’s a great tool, but not suitable for every use case.

Read the LLM’s thinking, especially when debugging. Many LLMs can expose their reasoning process. When you’re stuck on a bug or trying to understand a decision, asking the LLM to think out loud before answering is one of the fastest ways to build genuine understanding rather than just getting an answer.

Write the tests yourself. Even if you let the LLM write the implementation, writing the tests forces you to reason about the behavior you actually want. It’s one of the best ways to make sure the mental model gets built. Of course, it takes more time and it’s not always possible, like with JoyConf where I didn’t know enough about the Elixir environment to write effective tests. But when you can, it’s a great way to stay in the driver’s seat.

Slow down at decision points. LLMs are fast. That’s the point. But speed can accelerate debt. When you hit a fork in the road (an architectural choice, a tradeoff, a “there are a few ways to do this” moment) pause and do the reasoning yourself, even if you use the LLM to help you think it through.

The goal isn’t to use LLMs less

LLMs are genuinely useful and I don’t plan to stop using them. The goal is to stay in the driver’s seat mentally, using them for leverage rather than as a substitute for thinking.

A healthy LLM workflow and a debt-generating one can look identical from the outside. The difference shows up later, when you need to understand, maintain, or extend what you built. If you finish each session understanding what you built and why, you’re using the tool well. If you don’t, you’re taking out a loan.

And like financial debt, cognitive debt is a lot easier to avoid than to pay off.

References

• MIT study on cognitive debt
• Simon Willison’s showboat tool
• JoyConf repo
• JoyConf story

I built a live emoji reaction app for my upcoming talk

I have a talk coming up in a few weeks. It got rescheduled once, which gave me extra time to prepare. The presentation itself is finished, so I used that time to build an application that I think will make the talk more fun and engaging for the audience.

I wanted to add some live audience interaction. The usual options, Slido, Mentimeter, Poll Everywhere, are fine, but they’re designed around Q&A and polls. What I actually wanted was simpler and more visual: live emoji reactions that float up as an overlay on my slides while I’m presenting. None of the existing tools seem to do that. So I built my own, and I called it JoyConf.

It also gave me a reason to finally learn some Elixir.

What it does

The flow is simple:

1. You create a talk in the admin panel and get a QR code
2. You put the QR code on your title slide
3. Attendees scan it and land on a page with pre-defined set emoji buttons: ❤️ 😂 🙋🏻 👏 🤯, etc.
4. They tap a button, and the emoji floats up on their screen in real time
5. A Chrome extension running on your laptop picks up the same broadcast and overlays the emoji directly in the lower right corner of your Google Slides presentation

That’s it. No accounts for attendees, no app to install, no configuration to fiddle with. Scan, tap, react.

Why Elixir and Phoenix

I write Ruby day to day, so the syntax was familiar enough. But Elixir runs on the BEAM, the Erlang virtual machine, which was built for soft real-time systems with lots of concurrent connections. Phoenix LiveView lets you build interactive, server-rendered UIs over WebSockets without writing much JavaScript. And Phoenix PubSub gives you a message bus that lets any process broadcast to any other, regardless of what type of process it is.

All three of those things are important for a system where dozens of phones are sending events to a server that needs to fan them out to a slide presentation and all the phones (for their own consolidated live view) in under a second.

LiveView is pretty magical, by the way. You define your UI as a function of state, and Phoenix handles keeping the browser in sync. You get WebSocket-backed interactivity without writing a single-page app.

How it works under the hood

There are three clients talking to one Phoenix server.

Attendee phones connect via Phoenix LiveView, which manages the WebSocket lifecycle automatically. When an attendee taps an emoji, it fires a phx-click event to the server. The server checks a rate limiter (one reaction per session every 3 seconds) and, if allowed, broadcasts the event to a PubSub topic keyed by the talk slug: "reactions:my-talk".

The Chrome extension connects via a Phoenix Channel, a lower-level WebSocket primitive. The extension can’t use LiveView because it’s not a web page; it just needs to receive events. The ReactionChannel is subscribed to the same PubSub topic, so when the attendee’s broadcast lands, it gets forwarded to the extension automatically.

The admin browser is a standard LiveView protected by HTTP Basic Auth. You create a talk, the server generates a slug from the title, and you get a QR code pointing at the attendee URL.

The PubSub layer is what makes the architecture clean. TalkLive doesn’t need to know the Chrome extension exists. It just broadcasts on the topic, and PubSub delivers it to everyone subscribed, whether that’s a LiveView process, a Channel process, or both.

Here’s the entire broadcast path in code:

def handle_event("react", %{"emoji" => emoji}, socket) do
  if RateLimiter.allow?(socket.id) do
    Endpoint.broadcast!("reactions:#{socket.assigns.talk.slug}", "new_reaction", %{emoji: emoji})
  end
  {:noreply, socket}
end

That’s the whole thing. Five lines. PubSub does the fan-out.

Rate limiting with ETS

The rate limiter is a GenServer that owns an ETS table. ETS is an in-memory key/value store built into the BEAM runtime. It’s extremely fast and, crucially, you can configure it for concurrent reads without going through the GenServer process. This matters because with many attendees tapping at once, you don’t want all those requests queuing up behind a single process.

The table stores {session_id, last_reaction_at}. The allow?/1 function looks up the session, checks if enough time has passed, and updates the timestamp atomically. No database round-trip, no lock contention.

There’s also a client-side rate limit in JavaScript: buttons are disabled for 3 seconds with a visible countdown timer. That’s just UX, the real enforcement is on the server.

The Chrome extension

The extension has two parts: a popup where you enter the talk slug once, and a content script injected into Google Slides pages that handles the actual connection and overlay.

The content script connects a Phoenix WebSocket client to the server, joins the reactions:${slug} channel, and listens for new_reaction messages. When one arrives, it spawns a floating <span> element with a CSS animation that drifts up and fades out.

It never works on the first try

Double emojis on slides. Early on, every reaction appeared twice on the speaker’s screen. The bug was that the Chrome extension was subscribing to PubSub directly as well as receiving the Channel push. Two subscriptions, two deliveries. The fix was removing the redundant subscription and letting the Channel handle delivery exclusively.

Fullscreen mode swallows the overlay. When you go fullscreen in Google Slides, the browser creates a new stacking context. Any position: fixed element on <body> becomes invisible. The fix was listening for fullscreenchange events and re-parenting the overlay <div> into document.fullscreenElement when it fires:

document.addEventListener("fullscreenchange", () => {
  const overlay = document.getElementById("joyconf-overlay");
  if (document.fullscreenElement) {
    document.fullscreenElement.appendChild(overlay);
  } else {
    document.body.appendChild(overlay);
  }
});

Not obvious, but straightforward once you know it.

Button clicks were being swallowed. The initial implementation disabled the emoji buttons immediately on click. That blocked the phx-click handler from firing, so the server never received the event. The fix was deferring the button disable to a setTimeout(..., 0), which lets the click event propagate before the buttons get disabled.

The Chrome extension’s origin. Chrome extensions run from a chrome-extension:// origin, which Phoenix’s check_origin protection rejects by default. One line in endpoint.ex fixes it:

socket "/socket", JoyconfWeb.UserSocket,
  websocket: [check_origin: false]

The tech stack, briefly

The database has one table: talks. Reactions are ephemeral and never persisted. If the server restarts, in-flight reactions are lost, which is fine.

What’s next

JoyConf is an MVP. It does one thing and it’s ready for me to use at my talk. It’s not productized, it’s not super-polished, and it’s likely not easy for the average non-technical user to deploy.

Things that might be nice to add in the future:

• Ease of deployment for less technical users.
• Reaction analytics tied to slides so you can see which parts of your talk landed
• Support for other presentation tools beyond Google Slides

For now, I’m just going to go use it.

The source code is on GitHub. If you want to run your own instance, the README should have everything you need.

Building MockOpenAI: a weekend MVP story

Last weekend I built and published a Ruby gem. From idea to published thing in about four days. Here’s how it went, including the part where I had to reconsider whether I’d built something useful at all.

Friday: 20 ideas, one bet

I’m between jobs right now. Good position to be in if you like building things, terrible position to be in if you like eating. So I’ve been running a little experiment: each weekend, pick one small idea and see if I can ship it.

Friday’s job was to generate and pick an idea. I sat down with an AI and brainstormed 20 candidates. Developer tools. Content products. Micro-SaaS. I narrowed it down to one: a local mock server for testing OpenAI-compatible APIs.

The pitch to myself was simple. I write a lot of Ruby. I write a lot of tests. Testing code that talks to LLMs is a bit annoying because, while the happy path is easy to mock, some of the failure modes and edge cases can be more of a pain. There had to be a better way.

By end of day Friday I had a repo, a gemspec, and a clear plan.

Saturday: build day

The core idea for MockOpenAI is that it’s a real HTTP server running on localhost, not a mock object or a stub. Your application code talks to it exactly the way it would talk to the LLM provider in production. You just point your client at http://localhost:4000 instead of the usaul API endpoint.

That distinction makes a difference, I think. With a real HTTP server you can test things that object-level mocking can’t touch: actual TCP timeouts, truncated streaming responses, retry headers. The kind of failure modes that bite you in production but never show up in your test suite because you stubbed them away.

The architecture is deliberately simple. A Rack server reads a shared JSON state file on every request. Your tests write rules to that file. The server is stateless. No client wrapping, no monkey-patching, no magic.

Here’s what using it looks like:

it "handles a rate limit", :mock_openai_rate_limit do
  expect { MyService.call_llm("Hello") }.to raise_error(RubyLLM::RateLimitError)
end

it "handles mixed outcomes", :mock_openai do
  MockOpenAI.set_responses([
    { match: "Step 1", response: "OK" },
    { match: "Step 2", failure_mode: :timeout },
    { match: "Step 3", response: "Done" }
  ])

  expect(MyService.step1).to eq("OK")
  expect { MyService.step2 }.to raise_error(Timeout::Error)
  expect(MyService.step3).to eq("Done")
end

The failure modes are:

You can also mix success and failure in a single test:

it "handles mixed outcomes", :mock_openai do
  MockOpenAI.set_responses([
    { match: "Step 1", response: "OK" },
    { match: "Step 2", failure_mode: :timeout },
    { match: "Step 3", response: "Done" }
  ])
end

Saturday was productive. By end of day I had all the core classes written TDD-style: Config, State, Matcher, ResponseBuilder, TemplateRenderer, all five failure mode classes. The code was written, the tests passed, and life was good.

Sunday: documentation and shipping

Sunday was docs day. I set up a Jekyll site and wrote the README. I added an Anthropic endpoint too, because my personal projects use both.

I also migrated the first personal project to use MockOpenAI. That went smoothly. The HTTP-level fidelity made a few tests a little more honest than they’d been with simple stubs at the client level.

Monday: the uncomfortable question

Monday I migrated a second personal project. This one used a helper module I’d written a while back to stub LLM calls. Just a few lines of code. It worked fine for that project.

I stared at that code for a while. Here it is:

module RubyLLMMocks
  def mock_ruby_llm_chat(content: nil, error: nil)
    if error
      allow(RubyLLM).to receive(:chat).and_raise(error)
    else
      mock_response = instance_double(
        RubyLLM::Message,
        content: content,
        inspect: "RubyLLM::Message(content: #{content.inspect})"
      )
      mock_chat = instance_double(RubyLLM::Chat, ask: mock_response)
      allow(RubyLLM).to receive(:chat).and_return(mock_chat)
    end
  end
end

# Example:
it "handles general ruby_llm errors gracefully" do
  error = RubyLLM::Error.new(nil, "Unexpected error")
  mock_ruby_llm_chat(error: error)

  generator = described_class.new(options)

  expect { generator.generate }
    .to output(/Error.*Unexpected error/m).to_stdout
    .and raise_error(SystemExit)
end

That’s it. Fifteen lines, no gem dependency, works perfectly for a project that uses RubyLLM as a wrapper. The error case is handled with and_raise. Clean.

So the question had to be asked: did I just build a solution in search of a problem?

After sitting with it, I don’t think so. But I did have to sharpen my thinking about when MockOpenAI actually earns its place versus when a helper method is the right call.

The short version: if you’re using a wrapper library like RubyLLM for all your LLM calls, and you only need happy-path responses and exception simulation in unit tests, the 15-line helper is probably the right answer. It’s less to maintain, has no extra dependencies, and does the job.

MockOpenAI is the right call when you need the actual HTTP layer in the picture. When you’re using the raw OpenAI or Anthropic client directly. When you’re running integration or system tests that make real HTTP connections. When you need to test what happens when TCP actually times out, or when a streaming response gets cut off halfway through, or when your retry logic processes a Retry-After header.

Those are real problems. They’re just not every project’s problems. I added a when not to use page to the docs to make the tradeoffs explicit.

What I’d do differently

One thing I’d change: I’d research the problem space a bit more to make sure I had a better understanding of problem scope and existing solution. (Especially ones I wrote myself!) The tool is solid, but I made some assumptions about the breadth of problems it would solve for. That’s a classic weekend MVP trap I suppose. You’re so focused on building that you skip the a bit of due diligence you think you don’t need.

The gem is published, the docs are live, and it works. The scope is narrower than I originally thought, but the use case is real. That feels like an honest result for a long weekend.

MockOpenAI is a Ruby gem for testing OpenAI-compatible and Anthropic APIs locally. References:

• Source: github.com/grymoire7/mockopenai.
• Docs: grymoire7.github.io/mockopenai.
• Landing page: tracyatteberry.com/mockopenai.

Documentation search: When your Rails app needs which approach

Imagine spending an afternoon watching a developer tear out a perfectly functional search feature. They replace their solid Postgres full-text search with a vector database and RAG pipeline because, well, that’s what you’re supposed to do now, right? The new system is slower, cost them $200 a month in OpenAI API calls, and returned worse results for their specific use case.

This keeps happening. The AI hype cycle has convinced developers that every search problem needs embeddings, vector databases, and agentic loops. Sometimes that’s true. Often it’s not.

Let’s build the same feature four different ways and see what each approach actually costs you.

The use case: searching Ruby gem documentation

We’re building a search feature for a documentation site that indexes about 5,000 Ruby gems. Each gem has README content, API documentation, and code examples. Users ask questions like “How do I upload files to S3?” or “What’s the best gem for handling webhooks?”

This is a realistic scale for most Rails apps. Not Google-sized, not trivial. Just normal business software that needs to help users find information.

I’ll show you four implementations, each adding a layer of complexity. We’ll look at the code, measure the actual costs, and figure out when the added complexity pays for itself.

Approach 1: Traditional search with AI summarization

Start with what works. Postgres full-text search has been solving search problems since before your junior devs were born.

class Documentation < ApplicationRecord
  include PgSearch::Model
  
  pg_search_scope :search_content,
    against: {
      title: 'A',
      content: 'B',
      code_examples: 'C'
    },
    using: {
      tsearch: { prefix: true },
      trigram: { threshold: 0.3 }
    }
end

class DocumentationSearcher
  def initialize(query)
    @query = query
  end
  
  def search
    results = Documentation.search_content(@query).limit(10)
    
    {
      results: results,
      summary: summarize_results(results)
    }
  end
  
  private
  
  def summarize_results(results)
    return nil if results.empty?
    
    prompt = <<~PROMPT
      User question: #{@query}
      
      Here are the top search results:
      #{format_results(results)}
      
      Provide a concise answer to the user's question based on these results.
      If the results don't contain relevant information, say so.
    PROMPT
    
    client = OpenAI::Client.new
    client = OpenAI::Client.new
    response = client.chat(
      parameters: {
        model: "gpt-4o-mini",
        messages: [{ role: "user", content: prompt }],
        temperature: 0.3
      }
    )

    response.dig(:choices, 0, :message, :content)
  end
  
  def format_results(results)
    results.map.with_index do |doc, i|
      "#{i + 1}. #{doc.title}\n#{doc.content.truncate(500)}"
    end.join("\n\n")
  end
end

This approach does one database query and one API call. The search uses proven Postgres features: full-text search with ranking, trigram matching for typos, and weighted fields. Then we send the top results to GPT-4 to generate a summary.

Cost per query:

• Database: ~5ms
• OpenAI API: ~$0.002 (about 1,000 input tokens, 200 output tokens)
• Total latency: ~800ms

When this fails:

• User queries are conceptually different from how docs are written (“async jobs” versus “background processing”)
• Important information is buried in the middle of long documents
• You need to combine information from multiple sources

The failure mode is subtle. Traditional search ranks by keyword matching and field weights. When users phrase questions differently than your documentation uses terminology, they get poor results. You can’t fix this with better prompt engineering because the LLM never sees the relevant documents.

Approach 2: Basic RAG with vector embeddings

RAG (Retrieval Augmented Generation) means embedding your documents as vectors, embedding the user’s query as a vector, and finding documents with similar embeddings. This solves the terminology mismatch problem.

# We need to store embeddings
class AddEmbeddingsToDocumentation < ActiveRecord::Migration[7.1]
  def change
    add_column :documentations, :embedding, :vector, limit: 1536
    add_index :documentations, :embedding, using: :hnsw, opclass: :vector_cosine_ops
  end
end

class Documentation < ApplicationRecord
  has_neighbors :embedding
  
  after_save :generate_embedding, if: :content_changed?
  
  private
  
  def generate_embedding
    text = "#{title}\n\n#{content}"
    
    client = OpenAI::Client.new
    response = client.embeddings(
      parameters: {
        model: "text-embedding-3-small",
        input: text
      }
    )

    self.update_column(:embedding, response.dig(:data, 0, :embedding))
  end
end

class RagSearcher
  def initialize(query, client: OpenAI::Client.new)
    @query = query
    @client = client
  end

  def search
    query_embedding = generate_embedding(@query)
    results = Documentation.nearest_neighbors(:embedding, query_embedding, distance: "cosine").limit(5)

    {
      results: results,
      answer: generate_answer(results)
    }
  end

  private

  def generate_embedding(text)
    response = @client.embeddings(
      parameters: {
        model: "text-embedding-3-small",
        input: text
      }
    )

    response.dig(:data, 0, :embedding)
  end

  def generate_answer(results)
    context = results.map { |doc| "#{doc.title}\n#{doc.content}" }.join("\n\n---\n\n")

    prompt = <<~PROMPT
      Answer the user's question based only on the following documentation:

      #{context}

      Question: #{@query}

      If you cannot answer based on the provided documentation, say so clearly.
    PROMPT

    response = @client.chat(
      parameters: {
        model: "gpt-4o-mini",
        messages: [{ role: "user", content: prompt }],
        temperature: 0.3
      }
    )

    response.dig(:choices, 0, :message, :content)
  end
end

Now we’re making two API calls per search: one to embed the query, one to generate the answer. We’re also using pgvector with HNSW indexing for fast similarity search.

Cost per query:

• Database: ~15ms (vector similarity search)
• OpenAI embeddings API: ~$0.00001 (negligible)
• OpenAI chat API: ~$0.003
• Total latency: ~1,200ms

When this works better: The semantic matching is noticeably better. A query about “background jobs” will match documents about “async processing” and “delayed tasks” because the concepts are similar in vector space. This is real improvement over keyword search.

When this still fails:

• Complex questions requiring information from many documents
• Multi-step reasoning (“compare these two approaches”)
• Questions where the first retrieval doesn’t get the right context

Here’s a concrete failure case I hit: A user asks “What’s the difference between Sidekiq and Good Job?” The vector search retrieves five documents, but three are about Sidekiq and two are about Good Job. The LLM tries to compare them but doesn’t have complete information about both systems. It hedges and gives a vague answer.

Approach 3: Agentic RAG with adaptive retrieval

This is where we let the LLM decide if it needs more information. Instead of one retrieve-then-generate pass, we give the LLM tools to search again, rephrase queries, or combine results.

class AgenticRagSearcher
  MAX_ITERATIONS = 3

  def initialize(query, client: OpenAI::Client.new)
    @query = query
    @client = client
    @conversation_history = []
    @retrieved_docs = []
  end
  
  def search
    initial_prompt = <<~PROMPT
      You are a helpful assistant that searches Ruby gem documentation.
      
      User question: #{@query}
      
      You have access to these tools:
      - search_docs(query): Search documentation with a semantic query
      - get_related(doc_id): Get documents related to a specific document
      
      Think step by step. You can search multiple times with different queries
      to gather complete information before answering.
    PROMPT
    
    @conversation_history << { role: "user", content: initial_prompt }
    
    MAX_ITERATIONS.times do
      response = call_llm_with_tools
      
      break if response[:finish_reason] == "stop"
      
      if response[:tool_calls]
        handle_tool_calls(response[:tool_calls])
      end
    end
    
    {
      results: @retrieved_docs.uniq,
      answer: @conversation_history.last[:content],
      iterations: @conversation_history.length
    }
  end
  
  private
  
  def call_llm_with_tools
    response = @client.chat(
      parameters: {
        model: "gpt-4o",
        messages: @conversation_history,
        tools: tool_definitions,
        temperature: 0.3
      }
    )

    message = response.dig(:choices, 0, :message)
    @conversation_history << message

    {
      finish_reason: response.dig(:choices, 0, :finish_reason),
      tool_calls: message[:tool_calls]
    }
  end
  
  def tool_definitions
    [
      {
        type: "function",
        function: {
          name: "search_docs",
          description: "Search documentation using semantic search",
          parameters: {
            type: "object",
            properties: {
              query: {
                type: "string",
                description: "The search query"
              }
            },
            required: ["query"]
          }
        }
      },
      {
        type: "function",
        function: {
          name: "get_related",
          description: "Get documents related to a specific document",
          parameters: {
            type: "object",
            properties: {
              doc_id: {
                type: "integer",
                description: "The ID of the document"
              }
            },
            required: ["doc_id"]
          }
        }
      }
    ]
  end
  
  def handle_tool_calls(tool_calls)
    results = tool_calls.map do |tool_call|
      function_name = tool_call.dig(:function, :name)
      arguments = JSON.parse(tool_call.dig(:function, :arguments) || "{}")

      result = case function_name
      when "search_docs"
        search_docs(arguments["query"])
      when "get_related"
        get_related_docs(arguments["doc_id"])
      end

      @retrieved_docs.concat(result)

      {
        role: "tool",
        tool_call_id: tool_call[:id],
        content: format_docs_for_llm(result)
      }
    end

    @conversation_history.concat(results)
  end
  
  def search_docs(query)
    embedding = generate_embedding(query)
    Documentation.nearest_neighbors(:embedding, embedding, distance: "cosine").limit(3)
  end
  
  def get_related_docs(doc_id)
    doc = Documentation.find(doc_id)
    Documentation
      .nearest_neighbors(:embedding, doc.embedding, distance: "cosine")
      .where.not(id: doc_id)
      .limit(3)
  end
  
  def generate_embedding(text)
    response = @client.embeddings(
      parameters: {
        model: "text-embedding-3-small",
        input: text
      }
    )

    response.dig(:data, 0, :embedding)
  end
  
  def format_docs_for_llm(docs)
    docs.map do |doc|
      {
        id: doc.id,
        title: doc.title,
        content: doc.content.truncate(1000)
      }
    end.to_json
  end
end

This is a real step up in complexity. We’re now orchestrating multiple LLM calls with tool use. The LLM can search multiple times, explore related documents, and build up context before answering.

Cost per query:

• Database: 30-90ms (multiple vector searches)
• OpenAI embeddings API: $0.00002-0.00006 (2-6 embedding calls)
• OpenAI chat API: $0.015-0.045 (3-5 LLM calls with larger context)
• Total latency: 3-8 seconds

Notice the variance. Some queries get answered in one iteration. Complex ones might do three searches with different phrasings, explore related documents, and make five total LLM calls.

When this works better: That comparison query from before (“What’s the difference between Sidekiq and Good Job?”) now works great. The LLM searches for “Sidekiq background jobs”, gets those docs, then searches for “Good Job background jobs”, gets those docs, then synthesizes a real comparison.

Multi-part questions work too. “How do I set up Stripe payments and handle webhooks?” triggers two separate searches that gather comprehensive information.

When this gets expensive: Every query where the LLM decides it needs more information costs you 3-5x more. If your users ask a lot of complex questions, your API bill climbs fast.

The latency is also noticeable. Eight seconds feels slow in a web UI. You need to stream responses or show progress indicators.

Approach 4: Full conversational agent with external tools

Now we’re building a real agent that can search your documentation, browse external sites, and maintain conversation context across multiple turns.

class DocumentationAgent
  def initialize(session_id, client: OpenAI::Client.new)
    @session_id = session_id
    @client = client
    @conversation_history = load_conversation_history
  end
  
  def chat(message)
    @conversation_history << { role: "user", content: message }
    
    loop do
      response = call_llm_with_tools
      
      break if response[:finish_reason] == "stop"
      
      if response[:tool_calls]
        handle_tool_calls(response[:tool_calls])
      else
        break
      end
    end
    
    save_conversation_history
    
    {
      response: @conversation_history.last[:content],
      sources: extract_sources
    }
  end
  
  private

  def call_llm_with_tools
    response = @client.chat(
      parameters: {
        model: "gpt-4o",
        messages: @conversation_history,
        tools: tool_definitions,
        temperature: 0.3
      }
    )

    message = response.dig(:choices, 0, :message)
    @conversation_history << message

    {
      finish_reason: response.dig(:choices, 0, :finish_reason),
      tool_calls: message[:tool_calls]
    }
  end

  def generate_embedding(text)
    response = @client.embeddings(
      parameters: {
        model: "text-embedding-3-small",
        input: text
      }
    )

    response.dig(:data, 0, :embedding)
  end

  def tool_definitions
    [
      {
        type: "function",
        function: {
          name: "search_internal_docs",
          description: "Search our Ruby gem documentation",
          parameters: {
            type: "object",
            properties: {
              query: { type: "string" }
            },
            required: ["query"]
          }
        }
      },
      {
        type: "function",
        function: {
          name: "fetch_external_url",
          description: "Fetch content from an external URL like GitHub or RubyGems.org",
          parameters: {
            type: "object",
            properties: {
              url: { type: "string" }
            },
            required: ["url"]
          }
        }
      },
      {
        type: "function",
        function: {
          name: "search_github",
          description: "Search GitHub repositories and code",
          parameters: {
            type: "object",
            properties: {
              query: { type: "string" }
            },
            required: ["query"]
          }
        }
      }
    ]
  end
  
  def handle_tool_calls(tool_calls)
    results = tool_calls.map do |tool_call|
      function_name = tool_call.dig(:function, :name)
      arguments = JSON.parse(tool_call.dig(:function, :arguments) || "{}")

      result = case function_name
      when "search_internal_docs"
        search_internal_docs(arguments["query"])
      when "fetch_external_url"
        fetch_external_url(arguments["url"])
      when "search_github"
        search_github(arguments["query"])
      end

      {
        role: "tool",
        tool_call_id: tool_call[:id],
        content: result.to_json
      }
    end

    @conversation_history.concat(results)
  end
  
  def search_internal_docs(query)
    embedding = generate_embedding(query)
    docs = Documentation.nearest_neighbors(:embedding, embedding, distance: "cosine").limit(5)

    docs.map { |d| { title: d.title, content: d.content.truncate(800), source: "internal", id: d.id } }
  end
  
  def fetch_external_url(url)
    # In production, use a proper HTTP client with timeouts and error handling
    response = HTTP.timeout(5).get(url)
    
    {
      url: url,
      content: extract_main_content(response.body.to_s).truncate(2000),
      source: "external"
    }
  rescue HTTP::Error => e
    { error: "Failed to fetch URL: #{e.message}" }
  end
  
  def search_github(query)
    # Use Octokit or similar
    client = Octokit::Client.new(access_token: ENV['GITHUB_TOKEN'])
    results = client.search_code(query, language: "ruby")
    
    results.items.first(3).map do |item|
      {
        name: item.name,
        repo: item.repository.full_name,
        url: item.html_url,
        source: "github"
      }
    end
  rescue Octokit::Error => e
    { error: "GitHub search failed: #{e.message}" }
  end
  
  def load_conversation_history
    cache_key = "agent_conversation:#{@session_id}"
    JSON.parse(Rails.cache.read(cache_key) || "[]")
  end
  
  def save_conversation_history
    cache_key = "agent_conversation:#{@session_id}"
    # Keep last 10 messages to control context size
    trimmed_history = @conversation_history.last(10)
    Rails.cache.write(cache_key, trimmed_history.to_json, expires_in: 1.hour)
  end
  
  def extract_sources
    @conversation_history
      .select { |msg| msg[:role] == "tool" }
      .flat_map { |msg| JSON.parse(msg[:content] || "[]") }
      .select { |item| item.is_a?(Hash) && item["source"] }
      .uniq { |item| item["id"] || item["url"] || item["title"] }
  end
end

This is a different beast. We’re maintaining conversation state, hitting external APIs, and letting the LLM orchestrate complex research tasks.

Cost per conversation turn:

• Database: 15-50ms
• External API calls: 200-2000ms (GitHub, external sites)
• OpenAI embeddings: $0.00001-0.00005
• OpenAI chat: $0.02-0.15 (larger context windows, multiple turns)
• Total latency: 5-15 seconds

When this is worth it: You’re building a research assistant or technical support bot where users have complex, multi-turn conversations. They ask follow-up questions, need you to check external sources, and expect the system to remember context.

A user might ask “What’s the best gem for image processing?”, then follow up with “Show me examples from the ImageMagick wrapper”, then “Is there a more modern alternative?” The agent maintains context and can search different sources for each question.

When this is overkill: Most search features. If users are doing one-off queries and moving on, you’re paying for conversational capabilities they don’t need.

Choosing your approach

I’ve built all four of these systems in production. Here’s how I decide which to use.

Start with enhanced traditional search if:

• You have well-written documentation with consistent terminology
• Queries are mostly straightforward lookup tasks
• You need predictable costs and latency
• Your document corpus is under 10,000 items

The cost difference matters. At 1,000 queries per day, enhanced traditional search costs you $2/day. Basic RAG costs $3/day. Agentic RAG costs $15-45/day. A full agent costs $100-300/day.

Move to basic RAG when:

• Users phrase questions differently than your docs
• Keyword search returns poor matches for valid queries
• You have good quality source documents
• Your corpus is large enough that keyword search becomes unwieldy (50,000+ documents)

You’ll know you need this when users complain that search doesn’t work, and you look at their queries and think “we have docs about that, but they’re using different words.”

Move to agentic RAG when:

• Users ask complex questions requiring multiple sources
• You see patterns of users doing multiple searches in sequence
• Simple RAG returns incomplete answers
• You have budget for 3-5x higher API costs

Watch your analytics. If users do three searches in a row and then give up, they’re manually doing what an agentic system would do automatically.

Build a full agent when:

• You’re building a product feature, not just search
• Users need multi-turn conversations with context
• You need to integrate external data sources
• You have engineering resources for proper tool integration and error handling

The engineering complexity here is significant. You need proper timeout handling, retry logic, conversation state management, and graceful degradation when external APIs fail. This is a feature, not a quick enhancement.

The implementation details that matter

Some practical considerations that aren’t obvious from the code samples.

Chunking strategy for vector search: Don’t just embed entire documents. Break them into logical chunks. For documentation, consider chunking by section with overlap:

class DocumentationChunker
  CHUNK_SIZE = 1000 # characters
  OVERLAP = 200
  
  def chunk(document)
    sections = document.content.split(/^##\s+/)
    
    sections.flat_map do |section|
      break_into_overlapping_chunks(section)
    end
  end
  
  private
  
  def break_into_overlapping_chunks(text)
    chunks = []
    start = 0
    
    while start < text.length
      chunk_end = start + CHUNK_SIZE
      chunks << text[start...chunk_end]
      start += CHUNK_SIZE - OVERLAP
    end
    
    chunks
  end
end

This means each document generates multiple rows in your database with different embeddings. Your vector search returns chunks, not whole documents.

Hybrid search combines the best of both:

def hybrid_search(query)
  # Keyword search results
  keyword_results = Documentation.search_content(query).limit(20)
  
  # Vector search results
  query_embedding = generate_embedding(query)
  vector_results = Documentation.nearest_neighbors(:embedding, query_embedding, distance: "cosine").limit(20)
  
  # Combine with reciprocal rank fusion
  combine_results(keyword_results, vector_results)
end

def combine_results(keyword_results, vector_results)
  scores = Hash.new(0)
  
  keyword_results.each_with_index do |doc, i|
    scores[doc.id] += 1.0 / (i + 60)
  end
  
  vector_results.each_with_index do |doc, i|
    scores[doc.id] += 1.0 / (i + 60)
  end
  
  Documentation.where(id: scores.keys).sort_by { |doc| -scores[doc.id] }
end

This gives you semantic understanding from vectors plus precise keyword matching. The reciprocal rank fusion formula is from research on combining search results. It works better than naive score addition.

Caching saves you money:

class CachedRagSearcher
  def search(query)
    cache_key = "rag_search:#{Digest::MD5.hexdigest(query)}"
    
    Rails.cache.fetch(cache_key, expires_in: 1.hour) do
      perform_search(query)
    end
  end
end

Popular queries get asked repeatedly. Cache the embeddings and the LLM responses. This cuts your API costs dramatically for common questions.

The above caching method uses the raw query as the cache key. If users ask the same question with slightly different wording, they won’t hit the cache. You might want to normalize queries before hashing them for the cache key. For example, you could lowercase the query, remove stop words, etc.

Monitor your failure modes:

class SearchMetrics
  def self.track(query, approach, results)
    SearchLog.create!(
      query: query,
      approach: approach,
      result_count: results.length,
      latency_ms: results[:latency],
      cost_cents: calculate_cost(results),
      user_clicked: false # updated when user clicks a result
    )
  end
end

Track which results users actually click. If they click the first result, your search works. If they reformulate their query three times, it doesn’t. This data tells you whether to upgrade your approach.

What I actually recommend

Build the simplest thing first. Most Rails apps should start with Postgres full-text search plus GPT summarization. It costs almost nothing, has predictable latency, and works fine for straightforward queries.

Add instrumentation immediately. Track user behavior, measure latency, and watch your API costs. You need this data to know if upgrading is worth it.

When you see concrete evidence that simple search fails for your use case, add vector embeddings. This is a real improvement for semantic search. The pgvector extension makes this straightforward in Postgres. You don’t need a separate vector database until you have millions of documents.

Only add agentic features when you can point to specific query patterns that need them. “Users ask comparison questions and we don’t have comparison docs” is a good reason. “Agents are cool and I want to try them” is not.

Save full conversational agents for when you’re building a product feature that needs conversations. This is engineering work, not just adding a library. Budget for it appropriately.

The hype cycle pushes developers toward complexity. Resist it. Your users don’t care about your architecture. They care about getting answers quickly and cheaply. Often the simplest approach gives them exactly that.

But does the code work? See this repo for an implementation of all four approaches.

Disclaimer: The code samples in this post are simplified for clarity. In production, you need proper error handling, timeouts, retries, and security considerations (e.g., sanitizing user input before embedding). The costs and latencies mentioned are estimates based on recent OpenAI pricing and typical response times; your actual costs may vary based on usage patterns and model choices. Always monitor your API usage and costs when deploying AI features. Void where prohibited.

When “just return JSON” isn’t enough

If you’re building a tool that calls an AI API and needs structured data back, at some point you’ll write a prompt that ends with something like: “Return a JSON array. No prose, no explanation, just the JSON.”

And most of the time, that works. But “most of the time” isn’t good enough when a parse error crashes your command mid-run.

I ran into this while building Jojo, a CLI tool that generates job application materials (tailored resumes, cover letters, and a custom landing page) using AI. Several of its commands need structured data back from the model: an array of annotations, an array of FAQ entries, an array of integer indices. Plain text won’t do; the output feeds directly into the next stage of the pipeline.

The problem with prompt instructions

Language models are trained to be helpful. Part of being helpful, from the model’s perspective, is adding context: explaining what it did, noting any assumptions, formatting output to be readable. A prompt instruction to “return only JSON” fights against that training.

The failure modes are predictable:

Here is the JSON array you requested:

```json
[{"question": "What is your experience with Ruby?", "answer": "..."}]
```

I've formatted this as valid JSON with the required fields.

The model followed the spirit of the instruction (it returned valid JSON) but wrapped it in markdown fences and a helpful introductory sentence. JSON.parse on that string raises a ParserError.

The original code defended against this with a quick fence-strip:

cleaned_json = json_string.strip
  .gsub(/\A```(?:json)?\n?/, "")
  .gsub(/\n?```\z/, "")
JSON.parse(cleaned_json, symbolize_names: true)

That handles the fence case. But what about the introductory sentence? What about a model that adds a trailing note after the JSON? The regex grows, cases multiply, and you end up with the same defensive code copy-pasted across three different generators.

JsonExtractor

Instead, we pull this logic into a single place and make it more robust by trying multiple strategies to just find the JSON in whatever the model returned.

module JsonExtractor
  def self.call(content, symbolize_names: false)
    return content if content.is_a?(Hash) || content.is_a?(Array)

    stripped = content.gsub(/\A```\w*\s*|\s*```\z/m, "").strip

    try_parse(content, symbolize_names: symbolize_names) ||
      try_parse(stripped, symbolize_names: symbolize_names) ||
      extract_first_structure(content, symbolize_names: symbolize_names) ||
      raise(JSON::ParserError, "No JSON object found in response")
  end
end

It tries three strategies in order:

1. Parse the raw response directly. If the model actually followed instructions, done.
2. Strip markdown fences and try again. Handles the most common failure mode.
3. Find the first { or [ in the response and parse the complete structure from there, using a depth-tracking parser to handle nested objects correctly. This handles the “Here is the JSON: […]” case regardless of what prose surrounds it.

Each generator now calls JsonExtractor.call(response) instead of rolling its own fence-stripping. One test suite covers all the edge cases.

The cleaner alternative

If you’re using RubyLLM, a Ruby library that wraps multiple AI providers behind a common interface, there’s a more architecturally elegant solution: structured output.

Instead of asking the model to “return JSON” and then parsing whatever it sends back, you define a schema and ask the model to conform to it:

class FaqSchema < RubyLLM::Schema
  array :faqs do
    string :question
    string :answer
  end
end

response = chat.with_schema(FaqSchema).ask(prompt)
# response.faqs is an array of objects with .question and .answer

The model is constrained at the API level, not the prompt level. You get back a parsed object that already matches your schema. No JSON.parse, no fence-stripping, no JsonExtractor. The schema also serves as documentation for what the command expects.

This is genuinely cleaner. So why not use it?

The provider support problem

Structured output is a feature that providers implement, and not all models support it. The major providers (Anthropic, OpenAI, Gemini) support it for their recent models. But older models, like GPT-3.5, Gemini 1.0, Codex, etc., don’t. Some providers, like Hugging Face, don’t support it at all.

Jojo is designed to work with whatever model and provider the user configures. It dynamically picks up provider credentials from environment variables and routes requests through RubyLLM. A user might be running Claude Haiku, GPT-4o, or a locally-hosted model through an OpenAI-compatible endpoint.

RubyLLM’s structured output interface is uniform across providers: chat.with_schema works the same way regardless of which model you’re talking to. But when the underlying model doesn’t support structured output, you don’t get graceful degradation. You get a provider error.

That’s the thing:

• JsonExtractor works on any string. If the model returns prose with JSON buried in it, that’s handled. If the model returns clean JSON, that’s handled. It never throws a provider error because it never makes a structured output request.
• Structured output fails loudly on unsupported models. There’s no fallback. You have to catch the error and decide what to do with it.

For a tool where wide model support is a goal, JsonExtractor is more defensive. The cost is that you’re still at the mercy of whatever the model decides to return, but in practice, every model returns something that contains valid JSON, even if it’s surrounded by other content. The extraction strategy covers the real-world failure modes.

Okay, you also give up strict JSON schema guarantees (if supported) but that’s a tradeoff I’m willing to make for broader compatibility.

When structured output is the right call

None of this is an argument against structured output in general. If you have good control over exactly which model and provider your code will run against, structured output is the better choice. The schema and be explicit, the contract is enforced at the API level, and you’re unlikely to have a parse error at runtime.

You might wonder whether a hybrid approach could get the best of both worlds: check whether the model supports structured output, use it if so, and fall back to JsonExtractor if not. RubyLLM does expose this information. Model objects carry a capabilities array that includes 'structured_output' when the feature is supported. So the check is technically feasible.

The problem is that it doesn’t simplify anything. JsonExtractor still has to exist for the fallback branch, so you haven’t reduced maintenance burden. And now you have two code paths returning different shapes: structured output returns a typed schema object, while JsonExtractor returns a Hash. Every caller has to handle both, or you need a normalization layer on top. You’ve added complexity without removing any.

The tradeoff is essentially: how much do you want to trade portability for guarantees? JsonExtractor is portable and handles the real-world messiness of model output. Structured output gives you stronger guarantees but requires a supported model.

For a tool like Jojo, where users configure their own API credentials and model choices, JsonExtractor keeps things working across the widest range of setups. The “clean” architecture turned out to be less pragmatic once you considered who was actually going to run it. That person is me, btw, and I want it to work with whatever model I’m testing with at the moment, without having to worry about structured output support.

Building Jojo: turning job applications into marketing campaigns

When you apply for a job, you’re competing against hundreds of other candidates. Most of them submit a resume and a cover letter. The ambitious ones tailor those documents to the role. And then everyone waits.

No matter how good your resume is, it’s still a PDF in a pile of PDFs. You’re asking a hiring manager to do the work of figuring out why you’re a fit. What if you did that work for them?

That’s the idea behind Jojo, a Ruby CLI I built to transform job applications into personalized marketing campaigns. Instead of sending documents, you send a package: a tailored resume, a cover letter informed by company research, and a dedicated landing page that shows exactly why you’re a match for the role.

The landing page is the centerpiece. It’s a mini marketing site with an annotated job description that maps your experience to their requirements, portfolio projects selected for relevance to their tech stack, a branding statement written for their company, LinkedIn recommendations, an FAQ section, and a call-to-action to schedule a conversation. It turns a passive application into an active pitch.

Think of it as treating each job application like a product launch. You’re the product. The company you’re applying to is the only customer. Jojo builds the marketing campaign.

How it works

The workflow starts with two inputs: your resume data (a structured YAML file) and a job description (a file or URL). From there, Jojo runs a pipeline of AI-powered generation steps.

# Create a new application workspace
jojo new --slug acme-senior-dev --job posting.txt

# Generate everything
jojo generate --slug acme-senior-dev

The generate command kicks off a sequence:

1. Research — AI analyzes the job description and (optionally) searches the web to build a research document about the company, the role, and how to position yourself.
2. Resume — Your structured resume data is curated and rendered into a tailored resume, emphasizing the most relevant experience.
3. Branding — AI writes a personal branding statement specific to the company and role.
4. Cover letter — Generated from the research and tailored resume, so it references specific things about the company rather than generic platitudes.
5. Annotations — The job description is analyzed requirement by requirement, with each one mapped to your matching experience.
6. FAQ — AI generates role-specific questions and answers based on your background and the job requirements.
7. Website — Everything comes together in a self-contained landing page.
8. PDF — Resume and cover letter are converted to PDF via Pandoc.

Each step feeds into the next. The research informs the resume tailoring. The resume informs the cover letter. The annotations and FAQ feed into the website. It’s a pipeline, not a collection of independent scripts.

Every application gets its own workspace directory organized by slug:

  applications/acme-senior-dev/
  ├── job_description.md
  ├── job_details.yml
  ├── research.md
  ├── resume.md
  ├── cover_letter.md
  ├── branding.md
  ├── faq.json
  ├── job_description_annotations.json
  ├── status.log
  └── website/
      └── index.html

For daily use, there’s also an interactive TUI mode. Running jojo with no arguments launches a dashboard that shows all your applications, tracks which steps are complete, detects when artifacts are stale (because you regenerated a dependency), and lets you generate or regenerate individual steps with a keypress. The staleness detection uses file modification times. If you regenerate your research, the dashboard knows your resume is now stale because it was built from the old research.

  ┌─ Jojo ────────────────────────────────────────────┐
  │  Active: acme-senior-dev                          │
  │  Company: Acme Corp  •  Role: Senior Developer    │
  ├───────────────────────────────────────────────────┤
  │  Workflow                           Status        │
  │  1. Job Description            $   ✓ Generated    │
  │  2. Research                   $   ✓ Generated    │
  │  3. Resume                     $   * Stale        │
  │  4. Cover Letter               $   ○ Ready        │
  │  ...                                              │
  ├───────────────────────────────────────────────────┤
  │  [1-9] Generate item    [a] All ready    [q] Quit │
  └───────────────────────────────────────────────────┘

The $ indicator shows which steps call paid APIs, so you know if an action will cost something before you press the key. Steps that just combine existing artifacts (like website generation) are free.

Architecture: the command pipeline

Jojo is over 5K lines of Ruby source across ~50 source files. Most CLI commands follow the same three-file pattern:

  lib/jojo/commands/{command_name}/
  ├── command.rb    — Orchestration: validates inputs, manages file I/O
  ├── generator.rb  — Content generation: builds context, calls AI
  └── prompt.rb     — AI prompts: system and user prompt templates

So when I need to add a new command, I can create these three files, follow the pattern from the existing commands, and it (hopefully/usually) works. I don’t have to modify a central router or understand the internals of unrelated commands. The pattern helps make the codebase predictable. If you’ve read one command, you understand the shape of all of them. That helps the human and the AI assistant.

This wasn’t the original architecture. The CLI started as a monolith in cli.rb. Thor command definitions were mixed with validation logic, file handling, and generation orchestration. It worked fine for the first few commands, but soon things got messy. Adding a new feature meant navigating a growing code heap and hoping your changes didn’t break something unrelated.

The refactor extracted each command into its own module with a shared base class that provides common behavior (slug resolution, config loading, AI client setup). The CLI file shrank to a thin router with about 150 lines of small methods that delegate to command classes. Interactive mode, which breifly had a circular dependency calling back into the CLI class (eww) now calls command classes directly through a simple adapter.

Dual AI models

Jojo configures two AI models. There’s a reasoning model for complex tasks and a text generation model for simpler ones.

Company research and resume tailoring need the strongest reasoning capabilities as they’re analyzing job requirements, cross-referencing your experience, and making judgment calls about relevance. But extracting metadata from a job description (company name, location, job title) is easier. Using a powerful model for that is like hiring a senior architect to hang shelves.

The reasoning model handles research, resume curation, and cover letter writing. The text generation model handles job description processing, annotations, FAQ generation, and branding statements. Both models are configurable per provider, so you can use a frontier model for reasoning and a faster model for text generation, or whatever suits your budget and quality needs.

Even with the right model architecture, the AI still has a fundamental trustworthiness problem when it comes to factual content (welcome to AI).

Solving the hallucination problem

This was a technical decision that came from a hard fail.

The original resume generation would take the user’s resume data (stored as structured YAML), combine it with the job description and research, and ask the AI to generate a tailored resume in markdown. The prompt included extensive instructions about not fabricating information. It said things like “only include skills the candidate actually has” and “do not add technologies not present in the source data.”

The AI ignored these instructions way too often. I’d review a generated resume and find “Kubernetes” listed in my skills because the AI noticed I mentioned Docker and helpfully inferred I must know Kubernetes too. Or it would embellish a job description with responsibilities I never had. For a resume, this is not good.

The first instinct was to add more guardrails to the prompt. More emphatic instructions. More examples of what not to do. This helped a little, but it didn’t solve the problem. The AI still had the ability to modify anything, and language-level instructions are suggestions, not constraints.

The insight: different fields have different risk profiles

A professional summary should be rewritten for each role. That’s the whole point, but a list of programming languages must not be modified. The years you worked at a company are facts. Your name is your name.

The problem was that “AI shouldn’t have the same permissions everywhere.” Some fields need smart tailoring. Others need strict preservation. And still others might be removed or reordered. The idea was to define a permission system that specifies what the AI can for different kinds of content.

Permission-based curation

The solution was a permission system embedded directly in the resume data:

name: "Bob Denver"               # default: read-only
email: "bob@example.com"         # default: read-only

summary: |                       # permission: rewrite
  Polyglot developer who enjoys solving problems
  with software...

skills:                          # permission: remove, reorder
  - software engineering
  - full stack development
  - AI assisted development

languages:                       # permission: reorder
  - Ruby
  - Java
  - Python
  - Go

experience:                      # permission: reorder
  - company: "Island Adventures Inc."
    role: "Senior Software Engineer"
    start_date: "2020-07"        # read-only (nested)
    description: |               # permission: rewrite
      Full-stack developer delivering a SaaS platform...
    technologies:                # permission: remove, reorder
      - Ruby on Rails
      - Vue
      - Python
      - Docker

Four permission levels:

• read-only (default) — AI cannot modify, delete, add, or reorder. Contact info, dates, company names.
• remove — AI can exclude irrelevant items but can’t modify the ones it keeps. A database skill list can drop SQLite if the role is all PostgreSQL.
• reorder — AI can prioritize by relevance but can’t remove or modify. Your programming languages list stays complete but puts the most relevant ones first.
• rewrite — AI can generate new content using the original as a factual baseline. Professional summary, job descriptions.

In particular, the AI should never add items that aren’t in the source data. Though there is still a risk of hallucination in rewrite fields, the presence of original content in smaller chunks provides a grounding that makes it less likely.

Two-pass pipeline

The curation happens in two passes:

Pass 1: Filter and reorder. The AI receives the full resume data and the job description. It returns a filtered, reordered version that respects the permissions on each field. Skills marked remove, reorder get filtered to ~70% of the most relevant items and sorted by relevance. Lists marked reorder get sorted but all items are preserved.

Pass 2: Rewrite fields. The AI receives the filtered data and generates new content for fields marked rewrite. For example, the professional summary and experience descriptions. It uses the original content as a factual baseline.

Then an ERB template renders the final markdown. The template handles structure and formatting. The AI never touches the output templating.

What makes this work as an engineering solution is that the Ruby code enforces the permissions where possible. If the AI returns a reordered list that’s shorter than the original for a field that only has reorder permission, the Transformer class raises a PermissionViolation error:

unless can_remove
  if indices.length != original_count
    raise PermissionViolation,
      "LLM removed items from reorder-only field: #{field_path}"
  end
end

The permissions are no longer buried in prompt instructions that the AI might ignore. They’re enforced in code. The AI provides suggestions for how to curate the data, and the Ruby code validates those suggestions against the permission rules before applying them. If the AI tries to exceed its permissions, the operation fails rather than silently producing a resume with fabricated content.

The result is a skills section always contains skills I actually have. My job dates are always accurate. But my professional summary is tailored for each role, emphasizing the experience most relevant to that specific position.

What structured data enables

In order to make the permission system work, we had to switch from an unstructured markdown resume to a structured YAML format. This was a significant architectural change and it required reworking the entire resume generation pipeline. However, it was necessary to address the hallucination problem.

The permission system is the most visible benefit of using structured data, but there are other advantages:

• Narrower AI focus — With structured data, the AI can focus on curating specific fields rather than trying to parse and understand a free-form markdown document. This leads to better quality and more consistent results.
• Better output control — The ERB template handles formatting and structure, so the AI only generates smaller pieces of content. This reduces the chances of formatting errors or hallucinated sections and increases the human control over the final output.
• Easier testing — Structured data is easier to work with in tests. You can create synthetic resume data with specific permissions and verify that the output respects those permissions. With unstructured markdown, it’s harder to assert that the AI didn’t add or modify content it shouldn’t have.

Testing as a development discipline

A permission system that enforces constraints in code is only trustworthy if you actually test the enforcement. Jojo has 530 tests across two tiers, with 84% code coverage. Getting there was an intentional investment.

AI coding assistants are enthusiastic about writing features. They’re less enthusiastic about writing tests. This mirrors human tendencies. Tests aren’t as exciting as shipping the next feature, but with AI-assisted development the gap is amplified.

When the first large refactor was needed I noticed that test coverage was sitting at 31%. The code worked, but I had no safety net for refactoring. The push to 84% was a conscious decision to invest in change enablement.

Three kinds of tests

Jojo has three kinds of tests:

All tests run on every ./bin/test (or rake test:all) invocation and in CI.

Testing API-dependent code

The trickiest part of testing Jojo is that a lot of interesting work involves AI and Search API calls. You can’t run those in CI without spending money on every test run, but you also want tests that exercise real response parsing.

The solution was the VCR gem. VCR records real HTTP interactions the first time a test runs and saves them as “cassettes.” On subsequent runs, it replays the recorded responses instead of making real API calls. You get fast, deterministic tests that still exercise the full response-parsing pipeline.

Fixture discipline

One rule that has saved me more than once is that tests (and AI) never touch the inputs/ directory — no matter how much AI would like to. That directory contains real resume data from the user. Tests use test/fixtures/ exclusively, with synthetic data designed for testability.

This is codified in the project’s AI guidelines, which was previously prone to such mistakes. The instructions are explicit, emphatic, and took a few iterations to be effective. This testing discipline was part of the broader experience of building with AI.

Building with AI

There’s a meta quality to this project: it’s a tool that uses AI to generate content, and it was built with AI assistance. Both Claude and Z helped with development.

AI is pretty good at generating boilerplate, brainstorming design alternatives, and automating the tedious parts of refactoring (like updating 50 files when you rename a class).

But the decisions this post is about — the curation system, the architecture, the decision to refactor and when, the test organization — those were human decisions (as was the choice to use em-dashes just then). AI helped implement them a bit faster, but it didn’t tell me they were needed.

One nice thing about AI-assisted development was the ability to explore approaches quickly. When I was designing the permission system, I could describe different architectures, brainstorm, and get working prototypes, all in fairly short order. That kind of rapid experimentation is really helpful. The design thinking, however, still has to be yours.

One not-so-nice thing was needing to prod the AI to write tests for the features it’s helping to build. Also, let’s be honest, there’s a temptation to let the AI go a little too long before reviewing its output. Left to its own devices, an AI assistant will happily build feature after feature, with no test coverage and growing technical debt. Just like a human developer on a deadline, it needs someone to say “we’re not adding anything else until we address the technical debt, and that includes tests.”

What I learned and what’s next

A few things I’d do differently if I started over:

Start with structured data sooner. The original design used a free-form markdown resume as input. This was a frightful battle of prompt engineering from the beginning. The switch to structured YAML data (resume_data.yml) was the right call, but it required reworking the entire resume generation pipeline. If I’d started with structured data, the permission system would have been a natural extension rather than a redesign.

Build the interactive mode earlier. The TUI dashboard made the tool dramatically more usable, but it came in Phase 6 out of 7. Earlier access to the dependency graph and staleness detection would have improved my own workflow during development.

Force TDD from the start, or very near it. I had a test suite from the beginning, of course, but it wasn’t until I hit a major refactor that I made a conscious decision to invest in better test coverage. If I had enforced TDD from the start.

Basically, I would have spent a lot more time up front on planning the architecture and testing strategy, which would have made the development process smoother and more maintainable. AI assistance can be great, but it’s also really good at seducing you into bad habits.

What’s next

A few potential things for the roadmap:

• Interview prep generation — STAR-method examples drawn from your resume data, tailored to the specific role
• More and better theming options — The landing page is Jojo’s UVP, but the current design is pretty basic. More themes and customization options would let users create a landing page that better reflects their personal brand.
• Application tracking — Status tracking across all applications with dates, notes, and follow-up reminders
• Full SaaS product — A Rails app version of Jojo with a user-friendly interface and full job search management features — this would be a much bigger project but could help a wider audience.

Try it out

Jojo is open source and available on GitHub with a documentation site. It’s a Ruby CLI that requires AI and Search provider API keys. Setup takes just a few minutes.

If you’re interested in the code, the architecture, or just want to talk about AI-assisted development, I’d enjoy hearing from you. You can find me on LinkedIn or Mastodon.