Not long ago I wrote about my search for a process (as in daemons) management tool to manage webservers, build watch processes, etc. across many different projects. I was still undecided at that point, but have since landed on Pitchfork. It does a bit more than I need and it is well-maintained. See that previous post for a discussion of that and other options that I considered.

Having settled on Pitchfork for process management, I next needed to choose a task runner. The contenders were: make, just, mise, and go-task/task. I’m not going to do a detailed breakdown, but here’s a post from another site that does. It turns out I came to the same decision they did, for mostly the same reasons. The winner was Mise.

You may notice that both tools, Pitchfork and Mise, come from the same developer. This is a bit of a risk. If the developer (Jeff Dickey) decides they’re no longer interested in maintaining these projects, then I’m suddenly looking for two tool replacements instead of one. But right now, looking at Jeff’s bio, income from his sponsors, support from his employer, etc., I’m really not too concerned.

I picked the top 19 most active repos that I have and migrated all of them to Pitchfork and Mise. Having that consistency across repos has made context switching much easier. It’s been great so far.

Around the same time, I also began using a few new skills. In addition to Pitchfork and Mise skills, I added a markdown formatter and my own writing skill. The writing skill comes with an LLM cliche linter tool as well. During the same migration, I ran the markdown formatter over all my READMEs, so now they’re easier for me to scan and find things in too.

I’ll add just a bit of brutal honesty here at the end in case you made it this far. Whenever I have a little extra time, or I’m just procrastinating on something, I tend to either a) tidy up, or b) try to improve the process. That was exactly the inspiration for this migration. Cheers!

The inertia problem

When I start working on a project, I open a terminal and start a few processes. The web server. A background job worker. The asset watcher that recompiles my JavaScript. Maybe a locally-running database service. Each one might get its own terminal tab, and each tab has a command I have to remember, or dig out of a README, or other docs.

Since I work across multiple projects in different languages, with different stacks, this gets expensive fast. Not expensive in terms of CPU or memory, but in terms of mental overhead. Every project has its own conventions: make dev, ./bin/dev, docker compose up, or something else entirely. You spend more time than you’d like just overcoming the inertia to get back to a running state every time you switch projects.

A process manager gives you a single command to start everything your project needs. And if you can standardize on one tool across all your projects (🤞), you eliminate that context-switching tax entirely: cd project && [tool] start works everywhere, every time.

That’s the goal, so let’s look at the options. (Or just scroll down to the comparison table and the verdict if you want to skip the narrative.)

But before I dive in, it’s worth being clear about what I’m trying to solve for here. Most frameworks give you ways to run individual commands natively: npm run dev, mix phx.server, bundle exec rails server. Those are great and you should use them. Your Procfile entries will often call them directly. A process manager sits on top of those commands and coordinates them, starting all your services at once and multiplexing their output into a single view. The two approaches are complementary, not competing.

There’s also a related but separate problem. Task runners like Make, Rake, Just, and npm scripts handle short-lived commands like running database migrations, executing tests, or triggering builds. You will still reach for those tools for that kind of work, and there’s a whole separate conversation to be had about achieving consistency there across projects. This post is focused specifically on the process management side: starting and keeping multiple long-lived development services running together.

It all started with a Procfile

In 2011, David Dollar released Foreman, a Ruby gem for managing multiple processes during development. The central idea was the Procfile: a plain text file that declares your process types, one per line.

web: bin/rails server
worker: bundle exec sidekiq
assets: npm run watch

Heroku adopted the format and popularized it. Today it’s the closest thing the development world has to a standard for declaring process types. Ports of Foreman exist for Python (Honcho), Go (goreman, forego), and more. Those ports largely replicate the original feature set, so I won’t review them separately, but the ecosystem shows how durable the Procfile idea has been.

What followed were tools that kept the Procfile format but went well beyond what Foreman originally offered. Here’s where things stand today.

The tools

Foreman

Foreman is where it all started for me, and it still does its job. Install the gem, drop a Procfile in your project, and run foreman start. It multiplexes the output of all your processes into a single stream, prefixed with the process name and a timestamp.

gem install foreman
foreman start

One unique feature is that Foreman can export your Procfile to init system formats like systemd, upstart, launchd, and runit. If you want to manage production processes the same way you manage development ones, that export path is a nice convenience. None of the other tools in this review offer it.

Pros: Mature and battle-tested. Dead simple to adopt. Procfile export to init systems is unique among these tools. Wide adoption means good documentation and community support.

Cons: It’s a Ruby gem, which adds a runtime dependency for non-Ruby projects. The bigger issue is how it handles process output. Processes detect they are not writing to a real terminal and suppress their colored output. If you’ve ever run rails server through Foreman and wondered why the logs look flat and grey, that’s why.

Hivemind

Hivemind was built by the team at Evil Martians to fix exactly that problem. It uses a PTY (pseudo-terminal) to capture process output, which means processes think they are writing to a real terminal. So colors work and log output behaves the way you’d expect.

It’s a standalone Go binary with no runtime dependencies beyond the binary itself. Install it with Homebrew or download the binary directly.

brew install hivemind
hivemind

Hivemind is deliberately minimal. It reads your Procfile, starts your processes, and gets out of the way. The README itself will tell you: if you want more features, check out Overmind.

Pros: Standalone binary with no language runtime required. Fixes the color output problem. Extremely easy to adopt, with the same Procfile format and simpler installation than Foreman for non-Ruby projects.

Cons: No interactive process management. If a process crashes, the whole group goes down. There’s no way to restart a single process without restarting everything.

Overmind

Overmind is Hivemind’s more capable sibling, also from Evil Martians. The key difference is that Overmind runs your processes inside a tmux session, which gives it capabilities no other Procfile tool matches.

You can connect directly to any process and interact with it:

overmind connect web

You can restart a single process without touching the others:

overmind restart sidekiq

You can tell Overmind that certain processes are allowed to exit without bringing everything else down, which is useful for one-time setup tasks like database migrations. Auto-restart on crash is configurable per process, and port assignment is automatic and consistent so you can reference one process’s port from another’s configuration.

The tmux integration turns out to be more than a dependency. If you’ve ever wanted to connect to a running Rails process to use the console, or poke at a webpack dev server that’s behaving strangely, overmind connect gets you there without restarting anything.

Pros: Full PTY support, so colors work. Individual process control without affecting the group. Auto-restart on crash. Port management per process. Highly configurable via environment variables so your preferences apply globally across every project.

Cons: Requires tmux. If tmux isn’t already part of your workflow, this is a real dependency, not just a binary download. Nested tmux sessions are manageable if you’re already inside tmux, but worth knowing about upfront.

Overitall

Overitall (oit) takes a different angle entirely. Where the previous tools focus on running processes and multiplexing their output, Overitall treats logs as a first-class concern. It’s a TUI (terminal user interface) that combines process management with what amounts to a built-in log viewer.

You still use a Procfile for defining processes, but a .overitall.toml config file can also point at standalone log files rather than just process output. This is useful for services that write to a file instead of stdout, or for situations where you want to correlate a running process’s output with a framework’s own log file.

The TUI gives you real-time search (regex-capable), per-process visibility toggles, time-based navigation (:goto -5m), and trace detection that can spot correlation IDs and UUIDs in your logs and filter to a specific trace. For log-heavy development workflows, this is a meaningful upgrade over watching interleaved output scroll by.

There’s also an AI integration story: run oit skill install and Overitall installs a skill into .claude/skills/oit/ (or the Cursor equivalent) that teaches your AI assistant how to control the running TUI from the command line. You can then ask Claude to check recent errors, restart a process, or search the logs without leaving your editor.

Pros: Log management as a first-class feature. Regex filtering, time navigation, and trace detection. Interactive TUI with per-process visibility control. AI skill for Claude Code and Cursor. Available on both macOS and Linux.

Cons: Requires an additional .overitall.toml config file alongside the Procfile. The TUI has a learning curve compared to a simple process runner. It’s a newer project and still maturing.

Pitchfork

Pitchfork is the most ambitious tool in this list, and also the most different. Where everything else runs your processes in the foreground and stops them when you close the terminal, Pitchfork manages daemons: background processes that keep running after you walk away. A supervisor process handles them, and you interact with it via a CLI.

The configuration is a pitchfork.toml file:

[daemons.redis]
run = "redis-server --port 6379"
auto = ["start", "stop"]

[daemons.api]
run = "npm run dev"
auto = ["start", "stop"]
depends = ["redis"]
ready_http = "http://localhost:3000/health"

The auto = ["start", "stop"] setting is the most interesting feature for the cross-project consistency goal. With shell hooks activated, Pitchfork starts those daemons automatically when you cd into the project directory and stops them when you leave. You never have to remember to start your services. You just navigate to your project and they’re there.

The feature list goes further: dependency ordering (so Redis starts before the API that needs it), ready checks (so Pitchfork waits until a health endpoint actually responds before declaring a service up), file watching for auto-restart, cron scheduling, resource limits, and a web UI. Pitchfork also ships a built-in MCP server. One JSON config block in your Claude Desktop or Cursor settings wires your AI assistant directly into daemon management.

Color output is available but requires opting in with pty = true per daemon. Since processes run in the background by default, PTY is not on automatically, but enabling it is a one-line change per daemon.

Pros: The most powerful feature set in this review by a wide margin. Auto-start and auto-stop via shell hooks directly address the cross-project consistency problem. Built-in MCP server for AI integration. Dependency ordering, ready checks, and cron scheduling included. Language-agnostic standalone binary.

Cons: Steeper learning curve and a new configuration format to learn. The daemon model is a different mental model than a foreground process runner. You check on services rather than watching them scroll past. Color output requires explicit opt-in per daemon. Actively developed and not yet at 1.0.

Feature comparison

Which tool is right for you?

Choose Foreman if your project is already Ruby-based and you want the simplest possible setup. The gem installs alongside your other dependencies, and the ecosystem familiarity is hard to beat. It’s also the right call if you need to export process definitions to init system formats for production use, since no other tool here does that.

Choose Hivemind if you want a no-frills Procfile runner that works for any project in any language, with proper color output. It’s an easy, zero-cost, drop-in replacement for Foreman for those that don’t need individual process control.

Choose Overmind if you live in tmux and want the ability to connect to, inspect, and restart individual processes without tearing down your whole stack. Once you’ve used overmind connect to drop into a running process’s window, it’s probably hard to let that go.

Choose Overitall if log management is a first-class concern in your workflow. Noisy logs from multiple services, distributed traces to chase, correlation IDs to filter on: this is where oit’s search, filtering, and time navigation start to pay off. I can see this being really useful in a more complex $work environment. The AI skill integration is also a practical bonus if you use Claude Code or Cursor heavily.

Choose Pitchfork if you want services that follow you between projects and start automatically without any deliberate action on your part. The daemon model requires a mental shift, but it has a nice TUI (and Web UI) and it’s the most direct answer to the cross-project consistency problem in this roundup.

The verdict

If you’re looking for a single tool to carry across every project regardless of language or stack, the choice comes down to how much you want to invest in configuration upfront versus how much friction you want to eliminate long-term.

For a lightweight universal runner, Hivemind is the pragmatic choice. Single binary, no runtime dependencies, correct color output, zero learning curve. Drop a Procfile in a project and you’re running. The limitation of no individual process control rarely matters until it suddenly does, and when it does, you know where to go next.

If you are already a tmux user, Overmind is the easy upgrade over Hivemind. Same binary story, same Procfile format, but you gain individual process control and the ability to interact directly with any running process. The additional capability costs almost nothing if tmux is already in your toolkit.

The most interesting answer for the cross-project consistency problem, though, is Pitchfork. Shell hooks that auto-start and auto-stop services as you move between directories solve the problem at a deeper level than any foreground runner can. You stop managing process startup entirely, and your services are just there when you need them. That comes with more upfront configuration and a different operational model, but if you work across many projects and that investment appeals to you, Pitchfork is the tool that most ambitiously reframes what a development process manager can be.

For myself, I’m honestly not sure where I’ll land quite yet. I have a soft spot for the simplicity of Hivemind, I’m not a frequent tmux user so Overmind’s advantages are less compelling, and Pitchfork’s daemon model is intriguing but requires extra work for me and my AI integration.

Building a presentation thingy.

Creating a gem that writes your missing tests

No new project survives contact with the real world unscathed. We built TestGenAI, ran it on itself, and it worked well. Then we ran it on another codebase, and two things broke immediately. The fixes turned out to just as interesting as the original build.

This is a walkthrough of how the tool works and what we learned when we took it outside the greenhouse.

The code here is from TestGenAI, a working Ruby CLI gem you can install and run against your own codebase.

The pipeline

The pipeline has five stages:

1. Scan your codebase to find classes and methods without test coverage
2. Build context for each untested method
3. Generate tests using an LLM with the mechanically curated context
4. Validate that the generated tests run and pass
5. Collect the results

Each stage needs to be reliable enough that you can walk away and trust the process to complete. That means handling errors gracefully, providing clear output about what happened, and making it easy to pick up where things left off if something breaks.

Finding untested code

Before you can generate tests, you need to know what needs testing. The right approach depends on whether SimpleCov is available in the project.

If SimpleCov is set up, TestGenAI runs your test suite with COVERAGE=true, reads the resulting coverage/.resultset.json, and uses AST parsing to find methods where every executable line has zero hits. This scanner handles partially-tested files correctly. It reports individual methods that were never exercised, even if other methods in the same file have full coverage.

If SimpleCov isn’t available, the scanner falls back to checking whether a spec or test file exists for each source file. This approach is less accurate. A file tested only through integration tests or through specs for its subclasses will appear fully untested even if its methods are exercised constantly. The SimpleCov scanner is worth setting up.

Both scanners share the same underlying logic for locating methods in source files, which brings up something worth explaining.

Walking the AST

To locate methods, TestGenAI parses each Ruby source file into an abstract syntax tree and walks it recursively. The walker looks for :def and :defs nodes (instance and class methods), tracks the current class/module namespace, and records each method’s file, class, name, and line range.

That line range matters. The SimpleCov scanner uses it to check whether any executable lines in the method had zero hits. A nil in SimpleCov’s coverage array means a line isn’t executable, like a blank line, a comment, or an end. The scanner filters those out before checking for zeros, so it only flags methods where runnable code was never touched.

The parser compatibility problem

To parse Ruby, the gem relies on the parser gem. In older versions, you’d call Parser::CurrentRuby.parse(source) and get back an AST. This worked fine until Ruby 3.4, which switched its internal default parser to prism. Using Parser::CurrentRuby with Ruby 3.4 produces warnings, and in some configurations it fails entirely.

The prism project ships a compatibility shim, Prism::Translation::ParserCurrent, that produces the same AST node types as the old parser gem. The AST-walking code works unchanged. The only question is which one to load.

The solution is a small file that runs at load time and sets a constant:

module Testgenai
  if Gem::Version.new(RUBY_VERSION) >= Gem::Version.new("3.4")
    require "prism"
    CurrentParser = Prism::Translation::ParserCurrent
  else
    require "parser/current"
    CurrentParser = Parser::CurrentRuby
  end
end

The rest of the codebase calls CurrentParser.parse(source) and never thinks about which parser is underneath. The pattern of version check at load time, constant as the abstraction is a clean way to handle the same kind of compatibility gap you’ll run into whenever Ruby ships a significant internal change.

Context, generation, and validation

When you ask an LLM to write tests for a method, you can’t just paste in the method body. It needs the full class, the dependencies that file requires, examples of how the method is called elsewhere in the codebase, and existing test files it can match in style. Context quality is where quick-and-dirty AI test generators fall apart, too little and the tests don’t compile, too much and you hit token limits.

The generator builds a prompt from all of that, sends it to the LLM via the ruby_llm gem (which keeps the generator code provider-agnostic), and strips any markdown fences from the response before passing it to the validator.

The validator writes the code to a temp file, runs bundle exec rspec or the Minitest equivalent, and distinguishes between three outcomes: the file failed to load (syntax errors, undefined constants), the tests ran but failed, or the tests passed. Each outcome needs different handling. A file that doesn’t load gets deleted immediately because it’s useless. A file that runs but fails gets its error output fed back to the LLM for a retry.

The pipeline retries up to three times, passing failure details back each time. LLMs are reasonably good at fixing specific errors when told what went wrong. Undefined constants and wrong require paths almost always resolve in one retry. More complex failures, like incorrect behavior assumptions, may not, and those end up in a failed bucket for manual review.

Then we ran it on a real project

The first external test run revealed two problems, both on the same day.

The first: generated tests were syntactically valid, ran, and passed — but they looked nothing like the rest of the project’s test suite. Wrong authentication setup, wrong factory usage, helpers that weren’t available. Tests that technically pass but violate project conventions create a maintenance burden.

The second: the tool was silently destroying existing tests. When a spec file already existed at the output path, the pipeline would overwrite it with the newly generated content. Any tests already in that file were gone.

Both problems make complete sense in retrospect. The tool had only ever run on its own codebase, where it was always generating new files and where the conventions were deeply familiar to the model from the context it was seeing. A different project broke both assumptions.

Fixing the conventions gap

The core problem is that the LLM knows what your method does, but it doesn’t know how your team writes tests. It doesn’t know that you authenticate in before blocks a certain way, or that you have specific factory traits available, or that you’re not using rails-controller-testing so assigns isn’t an option.

The fix is a conventions system with two parts.

ConventionsExtractor scans your existing test files and pulls out mechanical facts: the most common authentication setup pattern, available factory traits from your factories directory, frequently stubbed objects, whether transactional fixtures are disabled and how cleanup is handled, and whether specific helpers are unavailable based on what’s in your Gemfile. These aren’t judgments, they’re observations extracted directly from the code.

ConventionsSynthesizer takes those raw facts and sends them to the LLM with a prompt asking it to write a concise conventions guide explaining the rule behind each pattern. The result is plain prose. Something like “authentication is handled in before blocks using session[:user_id] = rather than Devise helpers; use this pattern consistently.” That text gets prepended to every generation prompt.

The synthesized guide is cached to spec/conventions.md and invalidated automatically when spec files or Gemfiles change. Regenerating it costs one LLM call.

Enable it with the --conventions flag:

testgenai generate --provider anthropic --model claude-opus-4-7 --conventions

Add spec/conventions.md to your .gitignore. It’s a derived artifact and probably not something to check in.

Fixing the overwrite problem

The overwrite bug was straightforward to diagnose and subtle to fix correctly.

The naive fix would be to skip generation if a spec file already exists. That’s less than ideal. The point is to add tests for untested methods, and partially-tested files are the most common case.

Better behavior is to inject the generated tests into the existing file. The pipeline now reads existing content before doing anything else. It combines existing content with the newly generated code and validates the combined file. If it passes, the combined content is written to the spec file.

If it fails, the pipeline restores the original file exactly as it was and writes the generated-only code to a fallback path. A method in lib/payments/processor.rb that already has a spec/payments/processor_spec.rb would get its fallback at spec/payments/processor_context_spec.rb (using the method name to scope the filename). The failure output tells you where to find it:

  ✗ Payments::Processor#context failed after 3 attempt(s)
    → Generated tests saved to spec/payments/processor_context_spec.rb for manual review

You end up with your original tests intact and the generated attempt sitting somewhere you can look at it and decide what to do.

Running it

Install the gem and point it at your project:

gem install testgenai
cd your_project
testgenai generate --provider anthropic --model claude-opus-4-7

Or add it to your Gemfile in the development group and use bundle exec. Configuration can also come from environment variables:

export TESTGENAI_PROVIDER=anthropic
export TESTGENAI_MODEL=claude-opus-4-7
export ANTHROPIC_API_KEY=your_api_key
testgenai generate --conventions

Three diagnostic commands are available before you commit to a full run:

testgenai scan      # find untested methods without making any API calls
testgenai context   # show what context would be sent to the LLM for each method

scan gives you a picture of your coverage gaps. context is useful for understanding what the LLM will see before spending API credits.

The goal isn’t to replace the developer who understands the code and makes decisions about testing. It’s to handle the mechanical work: setting up describe blocks, wiring test data, writing happy-path coverage. Then you can spend your time on the parts that actually need your judgment. The second project taught us that “mechanical” is more context-dependent than it looks.

Building Jojo: turning job applications into marketing campaigns

When you apply for a job, you’re competing against hundreds of other candidates. Most of them submit a resume and a cover letter. The ambitious ones tailor those documents to the role. And then everyone waits.

No matter how good your resume is, it’s still a PDF in a pile of PDFs. You’re asking a hiring manager to do the work of figuring out why you’re a fit. What if you did that work for them?

That’s the idea behind Jojo, a Ruby CLI I built to transform job applications into personalized marketing campaigns. Instead of sending documents, you send a package: a tailored resume, a cover letter informed by company research, and a dedicated landing page that shows exactly why you’re a match for the role.

The landing page is the centerpiece. It’s a mini marketing site with an annotated job description that maps your experience to their requirements, portfolio projects selected for relevance to their tech stack, a branding statement written for their company, LinkedIn recommendations, an FAQ section, and a call-to-action to schedule a conversation. It turns a passive application into an active pitch.

Think of it as treating each job application like a product launch. You’re the product. The company you’re applying to is the only customer. Jojo builds the marketing campaign.

How it works

The workflow starts with two inputs: your resume data (a structured YAML file) and a job description (a file or URL). From there, Jojo runs a pipeline of AI-powered generation steps.

# Create a new application workspace
jojo new --slug acme-senior-dev --job posting.txt

# Generate everything
jojo generate --slug acme-senior-dev

The generate command kicks off a sequence:

1. Research — AI analyzes the job description and (optionally) searches the web to build a research document about the company, the role, and how to position yourself.
2. Resume — Your structured resume data is curated and rendered into a tailored resume, emphasizing the most relevant experience.
3. Branding — AI writes a personal branding statement specific to the company and role.
4. Cover letter — Generated from the research and tailored resume, so it references specific things about the company rather than generic platitudes.
5. Annotations — The job description is analyzed requirement by requirement, with each one mapped to your matching experience.
6. FAQ — AI generates role-specific questions and answers based on your background and the job requirements.
7. Website — Everything comes together in a self-contained landing page.
8. PDF — Resume and cover letter are converted to PDF via Pandoc.

Each step feeds into the next. The research informs the resume tailoring. The resume informs the cover letter. The annotations and FAQ feed into the website. It’s a pipeline, not a collection of independent scripts.

Every application gets its own workspace directory organized by slug:

  applications/acme-senior-dev/
  ├── job_description.md
  ├── job_details.yml
  ├── research.md
  ├── resume.md
  ├── cover_letter.md
  ├── branding.md
  ├── faq.json
  ├── job_description_annotations.json
  ├── status.log
  └── website/
      └── index.html

For daily use, there’s also an interactive TUI mode. Running jojo with no arguments launches a dashboard that shows all your applications, tracks which steps are complete, detects when artifacts are stale (because you regenerated a dependency), and lets you generate or regenerate individual steps with a keypress. The staleness detection uses file modification times. If you regenerate your research, the dashboard knows your resume is now stale because it was built from the old research.

  ┌─ Jojo ────────────────────────────────────────────┐
  │  Active: acme-senior-dev                          │
  │  Company: Acme Corp  •  Role: Senior Developer    │
  ├───────────────────────────────────────────────────┤
  │  Workflow                           Status        │
  │  1. Job Description            $   ✓ Generated    │
  │  2. Research                   $   ✓ Generated    │
  │  3. Resume                     $   * Stale        │
  │  4. Cover Letter               $   ○ Ready        │
  │  ...                                              │
  ├───────────────────────────────────────────────────┤
  │  [1-9] Generate item    [a] All ready    [q] Quit │
  └───────────────────────────────────────────────────┘

The $ indicator shows which steps call paid APIs, so you know if an action will cost something before you press the key. Steps that just combine existing artifacts (like website generation) are free.

Architecture: the command pipeline

Jojo is over 5K lines of Ruby source across ~50 source files. Most CLI commands follow the same three-file pattern:

  lib/jojo/commands/{command_name}/
  ├── command.rb    — Orchestration: validates inputs, manages file I/O
  ├── generator.rb  — Content generation: builds context, calls AI
  └── prompt.rb     — AI prompts: system and user prompt templates

So when I need to add a new command, I can create these three files, follow the pattern from the existing commands, and it (hopefully/usually) works. I don’t have to modify a central router or understand the internals of unrelated commands. The pattern helps make the codebase predictable. If you’ve read one command, you understand the shape of all of them. That helps the human and the AI assistant.

This wasn’t the original architecture. The CLI started as a monolith in cli.rb. Thor command definitions were mixed with validation logic, file handling, and generation orchestration. It worked fine for the first few commands, but soon things got messy. Adding a new feature meant navigating a growing code heap and hoping your changes didn’t break something unrelated.

The refactor extracted each command into its own module with a shared base class that provides common behavior (slug resolution, config loading, AI client setup). The CLI file shrank to a thin router with about 150 lines of small methods that delegate to command classes. Interactive mode, which breifly had a circular dependency calling back into the CLI class (eww) now calls command classes directly through a simple adapter.

Dual AI models

Jojo configures two AI models. There’s a reasoning model for complex tasks and a text generation model for simpler ones.

Company research and resume tailoring need the strongest reasoning capabilities as they’re analyzing job requirements, cross-referencing your experience, and making judgment calls about relevance. But extracting metadata from a job description (company name, location, job title) is easier. Using a powerful model for that is like hiring a senior architect to hang shelves.

The reasoning model handles research, resume curation, and cover letter writing. The text generation model handles job description processing, annotations, FAQ generation, and branding statements. Both models are configurable per provider, so you can use a frontier model for reasoning and a faster model for text generation, or whatever suits your budget and quality needs.

Even with the right model architecture, the AI still has a fundamental trustworthiness problem when it comes to factual content (welcome to AI).

Solving the hallucination problem

This was a technical decision that came from a hard fail.

The original resume generation would take the user’s resume data (stored as structured YAML), combine it with the job description and research, and ask the AI to generate a tailored resume in markdown. The prompt included extensive instructions about not fabricating information. It said things like “only include skills the candidate actually has” and “do not add technologies not present in the source data.”

The AI ignored these instructions way too often. I’d review a generated resume and find “Kubernetes” listed in my skills because the AI noticed I mentioned Docker and helpfully inferred I must know Kubernetes too. Or it would embellish a job description with responsibilities I never had. For a resume, this is not good.

The first instinct was to add more guardrails to the prompt. More emphatic instructions. More examples of what not to do. This helped a little, but it didn’t solve the problem. The AI still had the ability to modify anything, and language-level instructions are suggestions, not constraints.

The insight: different fields have different risk profiles

A professional summary should be rewritten for each role. That’s the whole point, but a list of programming languages must not be modified. The years you worked at a company are facts. Your name is your name.

The problem was that “AI shouldn’t have the same permissions everywhere.” Some fields need smart tailoring. Others need strict preservation. And still others might be removed or reordered. The idea was to define a permission system that specifies what the AI can for different kinds of content.

Permission-based curation

The solution was a permission system embedded directly in the resume data:

name: "Bob Denver"               # default: read-only
email: "bob@example.com"         # default: read-only

summary: |                       # permission: rewrite
  Polyglot developer who enjoys solving problems
  with software...

skills:                          # permission: remove, reorder
  - software engineering
  - full stack development
  - AI assisted development

languages:                       # permission: reorder
  - Ruby
  - Java
  - Python
  - Go

experience:                      # permission: reorder
  - company: "Island Adventures Inc."
    role: "Senior Software Engineer"
    start_date: "2020-07"        # read-only (nested)
    description: |               # permission: rewrite
      Full-stack developer delivering a SaaS platform...
    technologies:                # permission: remove, reorder
      - Ruby on Rails
      - Vue
      - Python
      - Docker

Four permission levels:

• read-only (default) — AI cannot modify, delete, add, or reorder. Contact info, dates, company names.
• remove — AI can exclude irrelevant items but can’t modify the ones it keeps. A database skill list can drop SQLite if the role is all PostgreSQL.
• reorder — AI can prioritize by relevance but can’t remove or modify. Your programming languages list stays complete but puts the most relevant ones first.
• rewrite — AI can generate new content using the original as a factual baseline. Professional summary, job descriptions.

In particular, the AI should never add items that aren’t in the source data. Though there is still a risk of hallucination in rewrite fields, the presence of original content in smaller chunks provides a grounding that makes it less likely.

Two-pass pipeline

The curation happens in two passes:

Pass 1: Filter and reorder. The AI receives the full resume data and the job description. It returns a filtered, reordered version that respects the permissions on each field. Skills marked remove, reorder get filtered to ~70% of the most relevant items and sorted by relevance. Lists marked reorder get sorted but all items are preserved.

Pass 2: Rewrite fields. The AI receives the filtered data and generates new content for fields marked rewrite. For example, the professional summary and experience descriptions. It uses the original content as a factual baseline.

Then an ERB template renders the final markdown. The template handles structure and formatting. The AI never touches the output templating.

What makes this work as an engineering solution is that the Ruby code enforces the permissions where possible. If the AI returns a reordered list that’s shorter than the original for a field that only has reorder permission, the Transformer class raises a PermissionViolation error:

unless can_remove
  if indices.length != original_count
    raise PermissionViolation,
      "LLM removed items from reorder-only field: #{field_path}"
  end
end

The permissions are no longer buried in prompt instructions that the AI might ignore. They’re enforced in code. The AI provides suggestions for how to curate the data, and the Ruby code validates those suggestions against the permission rules before applying them. If the AI tries to exceed its permissions, the operation fails rather than silently producing a resume with fabricated content.

The result is a skills section always contains skills I actually have. My job dates are always accurate. But my professional summary is tailored for each role, emphasizing the experience most relevant to that specific position.

What structured data enables

In order to make the permission system work, we had to switch from an unstructured markdown resume to a structured YAML format. This was a significant architectural change and it required reworking the entire resume generation pipeline. However, it was necessary to address the hallucination problem.

The permission system is the most visible benefit of using structured data, but there are other advantages:

• Narrower AI focus — With structured data, the AI can focus on curating specific fields rather than trying to parse and understand a free-form markdown document. This leads to better quality and more consistent results.
• Better output control — The ERB template handles formatting and structure, so the AI only generates smaller pieces of content. This reduces the chances of formatting errors or hallucinated sections and increases the human control over the final output.
• Easier testing — Structured data is easier to work with in tests. You can create synthetic resume data with specific permissions and verify that the output respects those permissions. With unstructured markdown, it’s harder to assert that the AI didn’t add or modify content it shouldn’t have.

Testing as a development discipline

A permission system that enforces constraints in code is only trustworthy if you actually test the enforcement. Jojo has 530 tests across two tiers, with 84% code coverage. Getting there was an intentional investment.

AI coding assistants are enthusiastic about writing features. They’re less enthusiastic about writing tests. This mirrors human tendencies. Tests aren’t as exciting as shipping the next feature, but with AI-assisted development the gap is amplified.

When the first large refactor was needed I noticed that test coverage was sitting at 31%. The code worked, but I had no safety net for refactoring. The push to 84% was a conscious decision to invest in change enablement.

Three kinds of tests

Jojo has three kinds of tests:

All tests run on every ./bin/test (or rake test:all) invocation and in CI.

Testing API-dependent code

The trickiest part of testing Jojo is that a lot of interesting work involves AI and Search API calls. You can’t run those in CI without spending money on every test run, but you also want tests that exercise real response parsing.

The solution was the VCR gem. VCR records real HTTP interactions the first time a test runs and saves them as “cassettes.” On subsequent runs, it replays the recorded responses instead of making real API calls. You get fast, deterministic tests that still exercise the full response-parsing pipeline.

Fixture discipline

One rule that has saved me more than once is that tests (and AI) never touch the inputs/ directory — no matter how much AI would like to. That directory contains real resume data from the user. Tests use test/fixtures/ exclusively, with synthetic data designed for testability.

This is codified in the project’s AI guidelines, which was previously prone to such mistakes. The instructions are explicit, emphatic, and took a few iterations to be effective. This testing discipline was part of the broader experience of building with AI.

Building with AI

There’s a meta quality to this project: it’s a tool that uses AI to generate content, and it was built with AI assistance. Both Claude and Z helped with development.

AI is pretty good at generating boilerplate, brainstorming design alternatives, and automating the tedious parts of refactoring (like updating 50 files when you rename a class).

But the decisions this post is about — the curation system, the architecture, the decision to refactor and when, the test organization — those were human decisions (as was the choice to use em-dashes just then). AI helped implement them a bit faster, but it didn’t tell me they were needed.

One nice thing about AI-assisted development was the ability to explore approaches quickly. When I was designing the permission system, I could describe different architectures, brainstorm, and get working prototypes, all in fairly short order. That kind of rapid experimentation is really helpful. The design thinking, however, still has to be yours.

One not-so-nice thing was needing to prod the AI to write tests for the features it’s helping to build. Also, let’s be honest, there’s a temptation to let the AI go a little too long before reviewing its output. Left to its own devices, an AI assistant will happily build feature after feature, with no test coverage and growing technical debt. Just like a human developer on a deadline, it needs someone to say “we’re not adding anything else until we address the technical debt, and that includes tests.”

What I learned and what’s next

A few things I’d do differently if I started over:

Start with structured data sooner. The original design used a free-form markdown resume as input. This was a frightful battle of prompt engineering from the beginning. The switch to structured YAML data (resume_data.yml) was the right call, but it required reworking the entire resume generation pipeline. If I’d started with structured data, the permission system would have been a natural extension rather than a redesign.

Build the interactive mode earlier. The TUI dashboard made the tool dramatically more usable, but it came in Phase 6 out of 7. Earlier access to the dependency graph and staleness detection would have improved my own workflow during development.

Force TDD from the start, or very near it. I had a test suite from the beginning, of course, but it wasn’t until I hit a major refactor that I made a conscious decision to invest in better test coverage. If I had enforced TDD from the start.

Basically, I would have spent a lot more time up front on planning the architecture and testing strategy, which would have made the development process smoother and more maintainable. AI assistance can be great, but it’s also really good at seducing you into bad habits.

What’s next

A few potential things for the roadmap:

• Interview prep generation — STAR-method examples drawn from your resume data, tailored to the specific role
• More and better theming options — The landing page is Jojo’s UVP, but the current design is pretty basic. More themes and customization options would let users create a landing page that better reflects their personal brand.
• Application tracking — Status tracking across all applications with dates, notes, and follow-up reminders
• Full SaaS product — A Rails app version of Jojo with a user-friendly interface and full job search management features — this would be a much bigger project but could help a wider audience.

Try it out

Jojo is open source and available on GitHub with a documentation site. It’s a Ruby CLI that requires AI and Search provider API keys. Setup takes just a few minutes.

If you’re interested in the code, the architecture, or just want to talk about AI-assisted development, I’d enjoy hearing from you. You can find me on LinkedIn or Mastodon.

Building a Ruby CLI gem for Hyrum’s Law

When you build a public API, users will depend on behaviors you never intended to guarantee. It’s called Hyrum’s Law, and it’s particularly tricky when it comes to error messages. Change “User not found” to “No such user exists” and someone’s regex breaks in production at 2am.

I built Hyrum to solve for this. It’s a Ruby CLI gem that uses AI to generate variations of status messages, ensuring users never become dependent on exact wording. It evolved from a single-provider tool into a multi-provider platform that cut costs by 10x, reduced code complexity from 12 to 1, and added quality validation for AI-generated content.

The problem with predictable messages

Hyrum’s Law states that all observable behaviors of your system will be depended on by somebody. This creates a dilemma for API designers: you want clear, consistent error messages, but you don’t want users parsing them as if they were structured data.

Error codes help with this. Return 404 or E_NOT_FOUND and the message text can evolve independently. But this only works if you’re disciplined about using codes for everything that matters. In practice, some context lives in the message text, and someone will parse it.

The traditional solution is thorough documentation warning against this. The pragmatic solution is accepting that some percentage of users will do it anyway. The solution presented here is to make the messages unpredictable by design.

Building the initial solution

The first version was straightforward. The gem takes a message like “The server refuses the attempt to brew coffee with a teapot” and generates code in your language of choice (Ruby, JavaScript, Python, Java, or JSON) that returns variations at random:

module Messages
  MESSAGES = {
    e418: [
      "Invalid Brewing Method",
      "Teapot not designed for coffee brewing",
      "Please use a suitable brewing device"
    ]
  }.freeze

  def self.message(key)
    MESSAGES[key].sample
  end
end

I started with OpenAI’s API via the ruby-openai gem. It worked well enough for the core use case. But eventually, two problems emerged.

First, cost. Running gpt-4 for simple message generation was using a chainsaw to cut butter. Second, vendor lock-in. Some projects had Anthropic credits, others used local Ollama models. I needed to support multiple providers without maintaining provider-specific code paths.

The migration decision

There were three options:

1. Build custom adapters for each provider
2. Find an abstraction layer that handled the differences
3. Accept the limitation and move on

Building custom adapters would provide complete control, but at the cost of maintaining provider-specific logic as APIs evolved. Option three was tempting but unsatisfying.

I chose option two, migrating to ruby_llm. More than swapping dependencies, this was a fundamental architecture change that would affect testing strategy, error handling, configuration, and the public API.

Key technical decisions

Cost optimization through model selection

The most impactful decision was switching from premium to budget models. For Anthropic, this meant claude-sonnet-4 to claude-haiku-20250514, a roughly 10x cost reduction.

This wasn’t about being cheap. (Okay, maybe it was a little bit about that.) It was (mostly) about matching model capability to task complexity. Generating three variations of “Resource not found” doesn’t require deep reasoning. Budget models handle it perfectly well. The quality remained identical while costs dropped by an order of magnitude.

Testing strategy: mock at the right level

The original implementation used VCR to record HTTP interactions. This is a common pattern, but it had problems:

• Maintaining cassettes for 10+ providers would be tedious
• Tests would break when ruby_llm changed request formats
• We would be testing ruby_llm’s HTTP implementation, not our code

The better approach: mock at the ruby_llm interface level. Instead of recording HTTP traffic, we mock RubyLLM.chat() directly. One mock setup works for all providers. Tests are faster, more maintainable, and focused on our actual logic.

This eliminated the need for VCR and WebMock entirely, removing two dependencies. 🎊

As an additional safeguard, I set up a GitHub Actions workflow that runs the full test suite on every push. This catches regressions before they reach main and provides confidence when accepting contributions. It’s a small addition that pays dividends in long-term maintainability.

Code simplification through extraction

FakeGenerator started at 298 lines with embedded message data. Extracting the messages to an external JSON file and refactoring the logic brought it down to 36 lines. That’s an 88% reduction.

Another big win was in AiGenerator. By letting ruby_llm handle provider differences, the cyclomatic complexity dropped from 12 to 1. Twelve decision points (checking provider types, handling edge cases) collapsed into a single code path.

This is the value of a good abstraction layer. It reduces the lines of code, sure, but it also reduces the number of things you have to think about.

To be honest, this is also currently a pain point. We swapped ruby-openai’s abastraction for a better one, but in an evolving ecosystem, we may need to swap again. The key is that the architecture is now flexible enough to accommodate future changes more easily.

Breaking changes as a design tool

The migration required environment variable changes:

• OPENAI_ACCESS_TOKEN → OPENAI_API_KEY
• OLLAMA_URL → OLLAMA_API_BASE

I considered adding migration helpers to detect old variables and warn users. But the gem was pre-1.0 with minimal adoption. Adding complexity for hypothetical users would hurt future maintainability more than it helped current users.

The cleaner approach: document the breaking changes clearly, provide a migration guide, and move forward with consistent naming. Sometimes the right trade-off is accepting short-term pain for long-term simplicity.

Implementation approach

I followed a disciplined TDD approach for the migration:

1. Write failing tests for AiGenerator
2. Implement minimal code to pass
3. Add error handling tests
4. Implement error handling
5. Verify across multiple providers

Each commit represented a logical unit of work with a clear purpose. The git history tells a story: dependency updates, test infrastructure, new generator implementation, factory updates, cleanup, documentation.

This matters for maintainability. Six months from now, when I need to add a new provider or debug an edge case, the git history explains not only what changed but why.

Validating non-deterministic output

Getting AI to generate message variations is great, but how do you know if the variations are any good?

Building a system that validates the quality of non-deterministic output requires deeper thinking about what “quality” even means in this context.

Defining useful variation

A good variation needs two properties:

1. Semantic similarity - It preserves the original message’s meaning
2. Lexical diversity - It uses different wording than other variations

These goals exist in tension. Perfect similarity means identical text. Perfect diversity means unrelated messages. The sweet spot is variations that mean the same thing but say it differently.

I created a validation system that measures both metrics and combines them into an overall quality score. This lets you validate generated variations automatically.

The initial design mistake

My first implementation compared variations to each other. Generate five variations, measure how similar they are as a group, done. This seemed logical until I tested it.

The problem: variations could be highly similar to each other but completely different from the original message. A set of variations about network timeouts would score well even if the original message was about authentication failures. They were similar to each other, but wrong.

The fix was obvious in hindsight: compare each variation to the original message, not to other variations. Semantic similarity measures how well each variation preserves the user’s intent. Lexical diversity measures how much the variations differ from each other. Two separate concerns, two separate comparisons.

Semantic similarity with embeddings

Measuring semantic similarity requires understanding meaning, not just matching words. “Server error” and “Internal server failure” share minimal text but convey the same concept. Simple string comparison would fail.

The solution: embedding models. These convert text into high-dimensional vectors where semantically similar content clusters together. Calculate the cosine similarity between the original message’s embedding and each variation’s embedding, and you have a numeric score for how well meaning is preserved.

I designed this to be provider-agnostic from the start, learning from the earlier migration experience. The validator uses RubyLLM.embed() which works with any provider that supports embeddings (OpenAI, Google, etc.). When embeddings aren’t available, it falls back to a simpler word overlap heuristic.

This graceful degradation was important. Users without embedding access still get validation, just with reduced accuracy. The feature doesn’t silently fail or block users.

API design for optional features

Quality validation needed to be opt-in. The core workflow is “generate variations and use them.” Adding validation steps would slow things down and require configuration. It needed to enhance the workflow without disrupting it.

The CLI design reflects this:

# Basic usage - no validation
hyrum -s openai -m "Server error" -f ruby

# Opt into validation
hyrum -s openai -m "Server error" -f ruby --validate

# Use in CI/CD with strict mode
hyrum -s openai -m "Server error" -f ruby --validate --strict --min-quality 75

Validation is off by default. Enable it when you want quality metrics. Use --strict to fail builds when quality is too low. Use --show-scores to include metrics in generated output.

Each flag serves a specific use case without cluttering the happy path. This is backward compatible and makes the feature discoverable through --help without overwhelming new users.

What I learned

Shipping a Ruby gem is more accessible than I expected

I hadn’t published a gem before this project. The Ruby ecosystem makes it surprisingly straightforward: follow conventions for directory structure, add a gemspec, and gem build handles the rest. RuboCop enforces community standards, and RSpec provides solid testing patterns.

Most of the learning curve wasn’t in the tooling. It was in the design decisions around versioning, breaking changes, and API stability. Understanding when to bump major vs minor versions, when breaking changes are acceptable, and how much backward compatibility to maintain. These are judgment calls that come with experience, not documentation per se.

Abstractions have a cost and a benefit

Ruby_llm’s abstraction eliminated provider-specific code paths. But it also added a dependency and gave up some provider-specific features (like Anthropic’s prompt caching). The trade-off made sense because the gem’s core use case doesn’t need advanced features. Your mileage will vary.

Model selection is a design decision

Defaulting to budget models wasn’t about minimizing costs. It was about right-sizing capability to task complexity. When your task genuinely needs advanced reasoning, use advanced models. When it doesn’t, you’re paying for capability you’re not using.

Testing at the right abstraction level matters

Mocking at the HTTP level tests the wrong thing. Mocking at the library interface level tests your code. The latter is almost always better unless you’re specifically testing HTTP behavior.

Breaking changes are acceptable in context

Pre-1.0 software with limited adoption is the right time to make breaking changes. Adding backward compatibility for a handful of users creates technical debt that affects every future user. Sometimes the generous thing is to break things cleanly.

User feedback catches design flaws early

My initial quality validation design seemed sound in theory. It measured variation quality as a group property. But the first test revealed the flaw: variations could be similar to each other while being completely unrelated to the original message.

This is why you test with real examples before building the whole system. The fix (comparing to the original message) was trivial to implement early. It would have been painful to retrofit later after building an entire validation pipeline on the wrong assumption.

The lesson: design mistakes are inevitable. What matters is catching them before they become weight-bearing walls in your architecture.

Graceful degradation beats hard dependencies

Embedding models provide superior semantic similarity measurement. But requiring them would block users whose AI providers don’t support embeddings. The word overlap fallback isn’t as accurate, but it’s better than nothing.

This pattern appears throughout the gem. Can’t access embeddings? Use heuristics. Provider doesn’t support structured output? Parse text. Each graceful degradation expands the set of valid configurations.

The alternative is failing fast with clear errors. Both approaches are valid, but for a tool that works across many providers, degradation creates a better experience than strict requirements.

Validation changes what “done” means

Before quality validation, “done” meant “generates variations.” After, it meant “generates variations that preserve meaning while varying wording.” This shift changed the entire value proposition.

The interesting part is that validation makes the AI output more trustworthy without requiring a better AI model. Same model, same cost, but now you have quantitative confidence in the results. That’s the leverage of good metrics.

The result

Hyrum now supports 11 AI providers (OpenAI, Anthropic, Gemini, Ollama, Mistral, DeepSeek, Perplexity, OpenRouter, Vertex AI, AWS Bedrock, GPUStack) through a unified interface. The codebase is simpler, tests are faster, and costs are 10x lower.

Quality validation adds confidence without complexity. Generate variations, validate they preserve meaning while varying wording, and integrate the results into your codebase with quantitative quality metrics. The validation system works across all providers and degrades gracefully when embeddings aren’t available.

Best of all, the architecture can accommodate new providers and capabilities without increasing complexity. When ruby_llm adds support for a new provider, Hyrum gets it for free. When embedding models improve, quality validation automatically benefits. That’s the payoff of choosing the right abstractions.

The project is open source on GitHub. If you’re dealing with Hyrum’s Law in your own APIs, or you’re just curious about the implementation details, check it out.