Building a presentation thingy.

Creating a gem that writes your missing tests

No new project survives contact with the real world unscathed. We built TestGenAI, ran it on itself, and it worked well. Then we ran it on another codebase, and two things broke immediately. The fixes turned out to just as interesting as the original build.

This is a walkthrough of how the tool works and what we learned when we took it outside the greenhouse.

The code here is from TestGenAI, a working Ruby CLI gem you can install and run against your own codebase.

The pipeline

The pipeline has five stages:

1. Scan your codebase to find classes and methods without test coverage
2. Build context for each untested method
3. Generate tests using an LLM with the mechanically curated context
4. Validate that the generated tests run and pass
5. Collect the results

Each stage needs to be reliable enough that you can walk away and trust the process to complete. That means handling errors gracefully, providing clear output about what happened, and making it easy to pick up where things left off if something breaks.

Finding untested code

Before you can generate tests, you need to know what needs testing. The right approach depends on whether SimpleCov is available in the project.

If SimpleCov is set up, TestGenAI runs your test suite with COVERAGE=true, reads the resulting coverage/.resultset.json, and uses AST parsing to find methods where every executable line has zero hits. This scanner handles partially-tested files correctly. It reports individual methods that were never exercised, even if other methods in the same file have full coverage.

If SimpleCov isn’t available, the scanner falls back to checking whether a spec or test file exists for each source file. This approach is less accurate. A file tested only through integration tests or through specs for its subclasses will appear fully untested even if its methods are exercised constantly. The SimpleCov scanner is worth setting up.

Both scanners share the same underlying logic for locating methods in source files, which brings up something worth explaining.

Walking the AST

To locate methods, TestGenAI parses each Ruby source file into an abstract syntax tree and walks it recursively. The walker looks for :def and :defs nodes (instance and class methods), tracks the current class/module namespace, and records each method’s file, class, name, and line range.

That line range matters. The SimpleCov scanner uses it to check whether any executable lines in the method had zero hits. A nil in SimpleCov’s coverage array means a line isn’t executable, like a blank line, a comment, or an end. The scanner filters those out before checking for zeros, so it only flags methods where runnable code was never touched.

The parser compatibility problem

To parse Ruby, the gem relies on the parser gem. In older versions, you’d call Parser::CurrentRuby.parse(source) and get back an AST. This worked fine until Ruby 3.4, which switched its internal default parser to prism. Using Parser::CurrentRuby with Ruby 3.4 produces warnings, and in some configurations it fails entirely.

The prism project ships a compatibility shim, Prism::Translation::ParserCurrent, that produces the same AST node types as the old parser gem. The AST-walking code works unchanged. The only question is which one to load.

The solution is a small file that runs at load time and sets a constant:

module Testgenai
  if Gem::Version.new(RUBY_VERSION) >= Gem::Version.new("3.4")
    require "prism"
    CurrentParser = Prism::Translation::ParserCurrent
  else
    require "parser/current"
    CurrentParser = Parser::CurrentRuby
  end
end

The rest of the codebase calls CurrentParser.parse(source) and never thinks about which parser is underneath. The pattern of version check at load time, constant as the abstraction is a clean way to handle the same kind of compatibility gap you’ll run into whenever Ruby ships a significant internal change.

Context, generation, and validation

When you ask an LLM to write tests for a method, you can’t just paste in the method body. It needs the full class, the dependencies that file requires, examples of how the method is called elsewhere in the codebase, and existing test files it can match in style. Context quality is where quick-and-dirty AI test generators fall apart, too little and the tests don’t compile, too much and you hit token limits.

The generator builds a prompt from all of that, sends it to the LLM via the ruby_llm gem (which keeps the generator code provider-agnostic), and strips any markdown fences from the response before passing it to the validator.

The validator writes the code to a temp file, runs bundle exec rspec or the Minitest equivalent, and distinguishes between three outcomes: the file failed to load (syntax errors, undefined constants), the tests ran but failed, or the tests passed. Each outcome needs different handling. A file that doesn’t load gets deleted immediately because it’s useless. A file that runs but fails gets its error output fed back to the LLM for a retry.

The pipeline retries up to three times, passing failure details back each time. LLMs are reasonably good at fixing specific errors when told what went wrong. Undefined constants and wrong require paths almost always resolve in one retry. More complex failures, like incorrect behavior assumptions, may not, and those end up in a failed bucket for manual review.

Then we ran it on a real project

The first external test run revealed two problems, both on the same day.

The first: generated tests were syntactically valid, ran, and passed — but they looked nothing like the rest of the project’s test suite. Wrong authentication setup, wrong factory usage, helpers that weren’t available. Tests that technically pass but violate project conventions create a maintenance burden.

The second: the tool was silently destroying existing tests. When a spec file already existed at the output path, the pipeline would overwrite it with the newly generated content. Any tests already in that file were gone.

Both problems make complete sense in retrospect. The tool had only ever run on its own codebase, where it was always generating new files and where the conventions were deeply familiar to the model from the context it was seeing. A different project broke both assumptions.

Fixing the conventions gap

The core problem is that the LLM knows what your method does, but it doesn’t know how your team writes tests. It doesn’t know that you authenticate in before blocks a certain way, or that you have specific factory traits available, or that you’re not using rails-controller-testing so assigns isn’t an option.

The fix is a conventions system with two parts.

ConventionsExtractor scans your existing test files and pulls out mechanical facts: the most common authentication setup pattern, available factory traits from your factories directory, frequently stubbed objects, whether transactional fixtures are disabled and how cleanup is handled, and whether specific helpers are unavailable based on what’s in your Gemfile. These aren’t judgments, they’re observations extracted directly from the code.

ConventionsSynthesizer takes those raw facts and sends them to the LLM with a prompt asking it to write a concise conventions guide explaining the rule behind each pattern. The result is plain prose. Something like “authentication is handled in before blocks using session[:user_id] = rather than Devise helpers; use this pattern consistently.” That text gets prepended to every generation prompt.

The synthesized guide is cached to spec/conventions.md and invalidated automatically when spec files or Gemfiles change. Regenerating it costs one LLM call.

Enable it with the --conventions flag:

testgenai generate --provider anthropic --model claude-opus-4-7 --conventions

Add spec/conventions.md to your .gitignore. It’s a derived artifact and probably not something to check in.

Fixing the overwrite problem

The overwrite bug was straightforward to diagnose and subtle to fix correctly.

The naive fix would be to skip generation if a spec file already exists. That’s less than ideal. The point is to add tests for untested methods, and partially-tested files are the most common case.

Better behavior is to inject the generated tests into the existing file. The pipeline now reads existing content before doing anything else. It combines existing content with the newly generated code and validates the combined file. If it passes, the combined content is written to the spec file.

If it fails, the pipeline restores the original file exactly as it was and writes the generated-only code to a fallback path. A method in lib/payments/processor.rb that already has a spec/payments/processor_spec.rb would get its fallback at spec/payments/processor_context_spec.rb (using the method name to scope the filename). The failure output tells you where to find it:

  ✗ Payments::Processor#context failed after 3 attempt(s)
    → Generated tests saved to spec/payments/processor_context_spec.rb for manual review

You end up with your original tests intact and the generated attempt sitting somewhere you can look at it and decide what to do.

Running it

Install the gem and point it at your project:

gem install testgenai
cd your_project
testgenai generate --provider anthropic --model claude-opus-4-7

Or add it to your Gemfile in the development group and use bundle exec. Configuration can also come from environment variables:

export TESTGENAI_PROVIDER=anthropic
export TESTGENAI_MODEL=claude-opus-4-7
export ANTHROPIC_API_KEY=your_api_key
testgenai generate --conventions

Three diagnostic commands are available before you commit to a full run:

testgenai scan      # find untested methods without making any API calls
testgenai context   # show what context would be sent to the LLM for each method

scan gives you a picture of your coverage gaps. context is useful for understanding what the LLM will see before spending API credits.

The goal isn’t to replace the developer who understands the code and makes decisions about testing. It’s to handle the mechanical work: setting up describe blocks, wiring test data, writing happy-path coverage. Then you can spend your time on the parts that actually need your judgment. The second project taught us that “mechanical” is more context-dependent than it looks.

Building a gem.

Building MockOpenAI: a weekend MVP story

Last weekend I built and published a Ruby gem. From idea to published thing in about four days. Here’s how it went, including the part where I had to reconsider whether I’d built something useful at all.

Friday: 20 ideas, one bet

I’m between jobs right now. Good position to be in if you like building things, terrible position to be in if you like eating. So I’ve been running a little experiment: each weekend, pick one small idea and see if I can ship it.

Friday’s job was to generate and pick an idea. I sat down with an AI and brainstormed 20 candidates. Developer tools. Content products. Micro-SaaS. I narrowed it down to one: a local mock server for testing OpenAI-compatible APIs.

The pitch to myself was simple. I write a lot of Ruby. I write a lot of tests. Testing code that talks to LLMs is a bit annoying because, while the happy path is easy to mock, some of the failure modes and edge cases can be more of a pain. There had to be a better way.

By end of day Friday I had a repo, a gemspec, and a clear plan.

Saturday: build day

The core idea for MockOpenAI is that it’s a real HTTP server running on localhost, not a mock object or a stub. Your application code talks to it exactly the way it would talk to the LLM provider in production. You just point your client at http://localhost:4000 instead of the usaul API endpoint.

That distinction makes a difference, I think. With a real HTTP server you can test things that object-level mocking can’t touch: actual TCP timeouts, truncated streaming responses, retry headers. The kind of failure modes that bite you in production but never show up in your test suite because you stubbed them away.

The architecture is deliberately simple. A Rack server reads a shared JSON state file on every request. Your tests write rules to that file. The server is stateless. No client wrapping, no monkey-patching, no magic.

Here’s what using it looks like:

it "handles a rate limit", :mock_openai_rate_limit do
  expect { MyService.call_llm("Hello") }.to raise_error(RubyLLM::RateLimitError)
end

it "handles mixed outcomes", :mock_openai do
  MockOpenAI.set_responses([
    { match: "Step 1", response: "OK" },
    { match: "Step 2", failure_mode: :timeout },
    { match: "Step 3", response: "Done" }
  ])

  expect(MyService.step1).to eq("OK")
  expect { MyService.step2 }.to raise_error(Timeout::Error)
  expect(MyService.step3).to eq("Done")
end

The failure modes are:

You can also mix success and failure in a single test:

it "handles mixed outcomes", :mock_openai do
  MockOpenAI.set_responses([
    { match: "Step 1", response: "OK" },
    { match: "Step 2", failure_mode: :timeout },
    { match: "Step 3", response: "Done" }
  ])
end

Saturday was productive. By end of day I had all the core classes written TDD-style: Config, State, Matcher, ResponseBuilder, TemplateRenderer, all five failure mode classes. The code was written, the tests passed, and life was good.

Sunday: documentation and shipping

Sunday was docs day. I set up a Jekyll site and wrote the README. I added an Anthropic endpoint too, because my personal projects use both.

I also migrated the first personal project to use MockOpenAI. That went smoothly. The HTTP-level fidelity made a few tests a little more honest than they’d been with simple stubs at the client level.

Monday: the uncomfortable question

Monday I migrated a second personal project. This one used a helper module I’d written a while back to stub LLM calls. Just a few lines of code. It worked fine for that project.

I stared at that code for a while. Here it is:

module RubyLLMMocks
  def mock_ruby_llm_chat(content: nil, error: nil)
    if error
      allow(RubyLLM).to receive(:chat).and_raise(error)
    else
      mock_response = instance_double(
        RubyLLM::Message,
        content: content,
        inspect: "RubyLLM::Message(content: #{content.inspect})"
      )
      mock_chat = instance_double(RubyLLM::Chat, ask: mock_response)
      allow(RubyLLM).to receive(:chat).and_return(mock_chat)
    end
  end
end

# Example:
it "handles general ruby_llm errors gracefully" do
  error = RubyLLM::Error.new(nil, "Unexpected error")
  mock_ruby_llm_chat(error: error)

  generator = described_class.new(options)

  expect { generator.generate }
    .to output(/Error.*Unexpected error/m).to_stdout
    .and raise_error(SystemExit)
end

That’s it. Fifteen lines, no gem dependency, works perfectly for a project that uses RubyLLM as a wrapper. The error case is handled with and_raise. Clean.

So the question had to be asked: did I just build a solution in search of a problem?

After sitting with it, I don’t think so. But I did have to sharpen my thinking about when MockOpenAI actually earns its place versus when a helper method is the right call.

The short version: if you’re using a wrapper library like RubyLLM for all your LLM calls, and you only need happy-path responses and exception simulation in unit tests, the 15-line helper is probably the right answer. It’s less to maintain, has no extra dependencies, and does the job.

MockOpenAI is the right call when you need the actual HTTP layer in the picture. When you’re using the raw OpenAI or Anthropic client directly. When you’re running integration or system tests that make real HTTP connections. When you need to test what happens when TCP actually times out, or when a streaming response gets cut off halfway through, or when your retry logic processes a Retry-After header.

Those are real problems. They’re just not every project’s problems. I added a when not to use page to the docs to make the tradeoffs explicit.

What I’d do differently

One thing I’d change: I’d research the problem space a bit more to make sure I had a better understanding of problem scope and existing solution. (Especially ones I wrote myself!) The tool is solid, but I made some assumptions about the breadth of problems it would solve for. That’s a classic weekend MVP trap I suppose. You’re so focused on building that you skip the a bit of due diligence you think you don’t need.

The gem is published, the docs are live, and it works. The scope is narrower than I originally thought, but the use case is real. That feels like an honest result for a long weekend.

MockOpenAI is a Ruby gem for testing OpenAI-compatible and Anthropic APIs locally. References:

• Source: github.com/grymoire7/mockopenai.
• Docs: grymoire7.github.io/mockopenai.
• Landing page: tracyatteberry.com/mockopenai.

Building Jojo: turning job applications into marketing campaigns

When you apply for a job, you’re competing against hundreds of other candidates. Most of them submit a resume and a cover letter. The ambitious ones tailor those documents to the role. And then everyone waits.

No matter how good your resume is, it’s still a PDF in a pile of PDFs. You’re asking a hiring manager to do the work of figuring out why you’re a fit. What if you did that work for them?

That’s the idea behind Jojo, a Ruby CLI I built to transform job applications into personalized marketing campaigns. Instead of sending documents, you send a package: a tailored resume, a cover letter informed by company research, and a dedicated landing page that shows exactly why you’re a match for the role.

The landing page is the centerpiece. It’s a mini marketing site with an annotated job description that maps your experience to their requirements, portfolio projects selected for relevance to their tech stack, a branding statement written for their company, LinkedIn recommendations, an FAQ section, and a call-to-action to schedule a conversation. It turns a passive application into an active pitch.

Think of it as treating each job application like a product launch. You’re the product. The company you’re applying to is the only customer. Jojo builds the marketing campaign.

How it works

The workflow starts with two inputs: your resume data (a structured YAML file) and a job description (a file or URL). From there, Jojo runs a pipeline of AI-powered generation steps.

# Create a new application workspace
jojo new --slug acme-senior-dev --job posting.txt

# Generate everything
jojo generate --slug acme-senior-dev

The generate command kicks off a sequence:

1. Research — AI analyzes the job description and (optionally) searches the web to build a research document about the company, the role, and how to position yourself.
2. Resume — Your structured resume data is curated and rendered into a tailored resume, emphasizing the most relevant experience.
3. Branding — AI writes a personal branding statement specific to the company and role.
4. Cover letter — Generated from the research and tailored resume, so it references specific things about the company rather than generic platitudes.
5. Annotations — The job description is analyzed requirement by requirement, with each one mapped to your matching experience.
6. FAQ — AI generates role-specific questions and answers based on your background and the job requirements.
7. Website — Everything comes together in a self-contained landing page.
8. PDF — Resume and cover letter are converted to PDF via Pandoc.

Each step feeds into the next. The research informs the resume tailoring. The resume informs the cover letter. The annotations and FAQ feed into the website. It’s a pipeline, not a collection of independent scripts.

Every application gets its own workspace directory organized by slug:

  applications/acme-senior-dev/
  ├── job_description.md
  ├── job_details.yml
  ├── research.md
  ├── resume.md
  ├── cover_letter.md
  ├── branding.md
  ├── faq.json
  ├── job_description_annotations.json
  ├── status.log
  └── website/
      └── index.html

For daily use, there’s also an interactive TUI mode. Running jojo with no arguments launches a dashboard that shows all your applications, tracks which steps are complete, detects when artifacts are stale (because you regenerated a dependency), and lets you generate or regenerate individual steps with a keypress. The staleness detection uses file modification times. If you regenerate your research, the dashboard knows your resume is now stale because it was built from the old research.

  ┌─ Jojo ────────────────────────────────────────────┐
  │  Active: acme-senior-dev                          │
  │  Company: Acme Corp  •  Role: Senior Developer    │
  ├───────────────────────────────────────────────────┤
  │  Workflow                           Status        │
  │  1. Job Description            $   ✓ Generated    │
  │  2. Research                   $   ✓ Generated    │
  │  3. Resume                     $   * Stale        │
  │  4. Cover Letter               $   ○ Ready        │
  │  ...                                              │
  ├───────────────────────────────────────────────────┤
  │  [1-9] Generate item    [a] All ready    [q] Quit │
  └───────────────────────────────────────────────────┘

The $ indicator shows which steps call paid APIs, so you know if an action will cost something before you press the key. Steps that just combine existing artifacts (like website generation) are free.

Architecture: the command pipeline

Jojo is over 5K lines of Ruby source across ~50 source files. Most CLI commands follow the same three-file pattern:

  lib/jojo/commands/{command_name}/
  ├── command.rb    — Orchestration: validates inputs, manages file I/O
  ├── generator.rb  — Content generation: builds context, calls AI
  └── prompt.rb     — AI prompts: system and user prompt templates

So when I need to add a new command, I can create these three files, follow the pattern from the existing commands, and it (hopefully/usually) works. I don’t have to modify a central router or understand the internals of unrelated commands. The pattern helps make the codebase predictable. If you’ve read one command, you understand the shape of all of them. That helps the human and the AI assistant.

This wasn’t the original architecture. The CLI started as a monolith in cli.rb. Thor command definitions were mixed with validation logic, file handling, and generation orchestration. It worked fine for the first few commands, but soon things got messy. Adding a new feature meant navigating a growing code heap and hoping your changes didn’t break something unrelated.

The refactor extracted each command into its own module with a shared base class that provides common behavior (slug resolution, config loading, AI client setup). The CLI file shrank to a thin router with about 150 lines of small methods that delegate to command classes. Interactive mode, which breifly had a circular dependency calling back into the CLI class (eww) now calls command classes directly through a simple adapter.

Dual AI models

Jojo configures two AI models. There’s a reasoning model for complex tasks and a text generation model for simpler ones.

Company research and resume tailoring need the strongest reasoning capabilities as they’re analyzing job requirements, cross-referencing your experience, and making judgment calls about relevance. But extracting metadata from a job description (company name, location, job title) is easier. Using a powerful model for that is like hiring a senior architect to hang shelves.

The reasoning model handles research, resume curation, and cover letter writing. The text generation model handles job description processing, annotations, FAQ generation, and branding statements. Both models are configurable per provider, so you can use a frontier model for reasoning and a faster model for text generation, or whatever suits your budget and quality needs.

Even with the right model architecture, the AI still has a fundamental trustworthiness problem when it comes to factual content (welcome to AI).

Solving the hallucination problem

This was a technical decision that came from a hard fail.

The original resume generation would take the user’s resume data (stored as structured YAML), combine it with the job description and research, and ask the AI to generate a tailored resume in markdown. The prompt included extensive instructions about not fabricating information. It said things like “only include skills the candidate actually has” and “do not add technologies not present in the source data.”

The AI ignored these instructions way too often. I’d review a generated resume and find “Kubernetes” listed in my skills because the AI noticed I mentioned Docker and helpfully inferred I must know Kubernetes too. Or it would embellish a job description with responsibilities I never had. For a resume, this is not good.

The first instinct was to add more guardrails to the prompt. More emphatic instructions. More examples of what not to do. This helped a little, but it didn’t solve the problem. The AI still had the ability to modify anything, and language-level instructions are suggestions, not constraints.

The insight: different fields have different risk profiles

A professional summary should be rewritten for each role. That’s the whole point, but a list of programming languages must not be modified. The years you worked at a company are facts. Your name is your name.

The problem was that “AI shouldn’t have the same permissions everywhere.” Some fields need smart tailoring. Others need strict preservation. And still others might be removed or reordered. The idea was to define a permission system that specifies what the AI can for different kinds of content.

Permission-based curation

The solution was a permission system embedded directly in the resume data:

name: "Bob Denver"               # default: read-only
email: "bob@example.com"         # default: read-only

summary: |                       # permission: rewrite
  Polyglot developer who enjoys solving problems
  with software...

skills:                          # permission: remove, reorder
  - software engineering
  - full stack development
  - AI assisted development

languages:                       # permission: reorder
  - Ruby
  - Java
  - Python
  - Go

experience:                      # permission: reorder
  - company: "Island Adventures Inc."
    role: "Senior Software Engineer"
    start_date: "2020-07"        # read-only (nested)
    description: |               # permission: rewrite
      Full-stack developer delivering a SaaS platform...
    technologies:                # permission: remove, reorder
      - Ruby on Rails
      - Vue
      - Python
      - Docker

Four permission levels:

• read-only (default) — AI cannot modify, delete, add, or reorder. Contact info, dates, company names.
• remove — AI can exclude irrelevant items but can’t modify the ones it keeps. A database skill list can drop SQLite if the role is all PostgreSQL.
• reorder — AI can prioritize by relevance but can’t remove or modify. Your programming languages list stays complete but puts the most relevant ones first.
• rewrite — AI can generate new content using the original as a factual baseline. Professional summary, job descriptions.

In particular, the AI should never add items that aren’t in the source data. Though there is still a risk of hallucination in rewrite fields, the presence of original content in smaller chunks provides a grounding that makes it less likely.

Two-pass pipeline

The curation happens in two passes:

Pass 1: Filter and reorder. The AI receives the full resume data and the job description. It returns a filtered, reordered version that respects the permissions on each field. Skills marked remove, reorder get filtered to ~70% of the most relevant items and sorted by relevance. Lists marked reorder get sorted but all items are preserved.

Pass 2: Rewrite fields. The AI receives the filtered data and generates new content for fields marked rewrite. For example, the professional summary and experience descriptions. It uses the original content as a factual baseline.

Then an ERB template renders the final markdown. The template handles structure and formatting. The AI never touches the output templating.

What makes this work as an engineering solution is that the Ruby code enforces the permissions where possible. If the AI returns a reordered list that’s shorter than the original for a field that only has reorder permission, the Transformer class raises a PermissionViolation error:

unless can_remove
  if indices.length != original_count
    raise PermissionViolation,
      "LLM removed items from reorder-only field: #{field_path}"
  end
end

The permissions are no longer buried in prompt instructions that the AI might ignore. They’re enforced in code. The AI provides suggestions for how to curate the data, and the Ruby code validates those suggestions against the permission rules before applying them. If the AI tries to exceed its permissions, the operation fails rather than silently producing a resume with fabricated content.

The result is a skills section always contains skills I actually have. My job dates are always accurate. But my professional summary is tailored for each role, emphasizing the experience most relevant to that specific position.

What structured data enables

In order to make the permission system work, we had to switch from an unstructured markdown resume to a structured YAML format. This was a significant architectural change and it required reworking the entire resume generation pipeline. However, it was necessary to address the hallucination problem.

The permission system is the most visible benefit of using structured data, but there are other advantages:

• Narrower AI focus — With structured data, the AI can focus on curating specific fields rather than trying to parse and understand a free-form markdown document. This leads to better quality and more consistent results.
• Better output control — The ERB template handles formatting and structure, so the AI only generates smaller pieces of content. This reduces the chances of formatting errors or hallucinated sections and increases the human control over the final output.
• Easier testing — Structured data is easier to work with in tests. You can create synthetic resume data with specific permissions and verify that the output respects those permissions. With unstructured markdown, it’s harder to assert that the AI didn’t add or modify content it shouldn’t have.

Testing as a development discipline

A permission system that enforces constraints in code is only trustworthy if you actually test the enforcement. Jojo has 530 tests across two tiers, with 84% code coverage. Getting there was an intentional investment.

AI coding assistants are enthusiastic about writing features. They’re less enthusiastic about writing tests. This mirrors human tendencies. Tests aren’t as exciting as shipping the next feature, but with AI-assisted development the gap is amplified.

When the first large refactor was needed I noticed that test coverage was sitting at 31%. The code worked, but I had no safety net for refactoring. The push to 84% was a conscious decision to invest in change enablement.

Three kinds of tests

Jojo has three kinds of tests:

All tests run on every ./bin/test (or rake test:all) invocation and in CI.

Testing API-dependent code

The trickiest part of testing Jojo is that a lot of interesting work involves AI and Search API calls. You can’t run those in CI without spending money on every test run, but you also want tests that exercise real response parsing.

The solution was the VCR gem. VCR records real HTTP interactions the first time a test runs and saves them as “cassettes.” On subsequent runs, it replays the recorded responses instead of making real API calls. You get fast, deterministic tests that still exercise the full response-parsing pipeline.

Fixture discipline

One rule that has saved me more than once is that tests (and AI) never touch the inputs/ directory — no matter how much AI would like to. That directory contains real resume data from the user. Tests use test/fixtures/ exclusively, with synthetic data designed for testability.

This is codified in the project’s AI guidelines, which was previously prone to such mistakes. The instructions are explicit, emphatic, and took a few iterations to be effective. This testing discipline was part of the broader experience of building with AI.

Building with AI

There’s a meta quality to this project: it’s a tool that uses AI to generate content, and it was built with AI assistance. Both Claude and Z helped with development.

AI is pretty good at generating boilerplate, brainstorming design alternatives, and automating the tedious parts of refactoring (like updating 50 files when you rename a class).

But the decisions this post is about — the curation system, the architecture, the decision to refactor and when, the test organization — those were human decisions (as was the choice to use em-dashes just then). AI helped implement them a bit faster, but it didn’t tell me they were needed.

One nice thing about AI-assisted development was the ability to explore approaches quickly. When I was designing the permission system, I could describe different architectures, brainstorm, and get working prototypes, all in fairly short order. That kind of rapid experimentation is really helpful. The design thinking, however, still has to be yours.

One not-so-nice thing was needing to prod the AI to write tests for the features it’s helping to build. Also, let’s be honest, there’s a temptation to let the AI go a little too long before reviewing its output. Left to its own devices, an AI assistant will happily build feature after feature, with no test coverage and growing technical debt. Just like a human developer on a deadline, it needs someone to say “we’re not adding anything else until we address the technical debt, and that includes tests.”

What I learned and what’s next

A few things I’d do differently if I started over:

Start with structured data sooner. The original design used a free-form markdown resume as input. This was a frightful battle of prompt engineering from the beginning. The switch to structured YAML data (resume_data.yml) was the right call, but it required reworking the entire resume generation pipeline. If I’d started with structured data, the permission system would have been a natural extension rather than a redesign.

Build the interactive mode earlier. The TUI dashboard made the tool dramatically more usable, but it came in Phase 6 out of 7. Earlier access to the dependency graph and staleness detection would have improved my own workflow during development.

Force TDD from the start, or very near it. I had a test suite from the beginning, of course, but it wasn’t until I hit a major refactor that I made a conscious decision to invest in better test coverage. If I had enforced TDD from the start.

Basically, I would have spent a lot more time up front on planning the architecture and testing strategy, which would have made the development process smoother and more maintainable. AI assistance can be great, but it’s also really good at seducing you into bad habits.

What’s next

A few potential things for the roadmap:

• Interview prep generation — STAR-method examples drawn from your resume data, tailored to the specific role
• More and better theming options — The landing page is Jojo’s UVP, but the current design is pretty basic. More themes and customization options would let users create a landing page that better reflects their personal brand.
• Application tracking — Status tracking across all applications with dates, notes, and follow-up reminders
• Full SaaS product — A Rails app version of Jojo with a user-friendly interface and full job search management features — this would be a much bigger project but could help a wider audience.

Try it out

Jojo is open source and available on GitHub with a documentation site. It’s a Ruby CLI that requires AI and Search provider API keys. Setup takes just a few minutes.

If you’re interested in the code, the architecture, or just want to talk about AI-assisted development, I’d enjoy hearing from you. You can find me on LinkedIn or Mastodon.

Building a CLI for turning job applications into marketing campaigns. Jojo helps you create personalized, engaging applications that stand out from the crowd. It generates tailored cover letters, optimizes your resume for ATS, and even creates a custom portfolio website for each application. With Jojo, you can turn your job search into a marketing campaign that gets noticed.

Building a personal URL shortener deployable via SFTP on shared hosting.

Building a Ruby CLI gem for Hyrum’s Law

When you build a public API, users will depend on behaviors you never intended to guarantee. It’s called Hyrum’s Law, and it’s particularly tricky when it comes to error messages. Change “User not found” to “No such user exists” and someone’s regex breaks in production at 2am.

I built Hyrum to solve for this. It’s a Ruby CLI gem that uses AI to generate variations of status messages, ensuring users never become dependent on exact wording. It evolved from a single-provider tool into a multi-provider platform that cut costs by 10x, reduced code complexity from 12 to 1, and added quality validation for AI-generated content.

The problem with predictable messages

Hyrum’s Law states that all observable behaviors of your system will be depended on by somebody. This creates a dilemma for API designers: you want clear, consistent error messages, but you don’t want users parsing them as if they were structured data.

Error codes help with this. Return 404 or E_NOT_FOUND and the message text can evolve independently. But this only works if you’re disciplined about using codes for everything that matters. In practice, some context lives in the message text, and someone will parse it.

The traditional solution is thorough documentation warning against this. The pragmatic solution is accepting that some percentage of users will do it anyway. The solution presented here is to make the messages unpredictable by design.

Building the initial solution

The first version was straightforward. The gem takes a message like “The server refuses the attempt to brew coffee with a teapot” and generates code in your language of choice (Ruby, JavaScript, Python, Java, or JSON) that returns variations at random:

module Messages
  MESSAGES = {
    e418: [
      "Invalid Brewing Method",
      "Teapot not designed for coffee brewing",
      "Please use a suitable brewing device"
    ]
  }.freeze

  def self.message(key)
    MESSAGES[key].sample
  end
end

I started with OpenAI’s API via the ruby-openai gem. It worked well enough for the core use case. But eventually, two problems emerged.

First, cost. Running gpt-4 for simple message generation was using a chainsaw to cut butter. Second, vendor lock-in. Some projects had Anthropic credits, others used local Ollama models. I needed to support multiple providers without maintaining provider-specific code paths.

The migration decision

There were three options:

1. Build custom adapters for each provider
2. Find an abstraction layer that handled the differences
3. Accept the limitation and move on

Building custom adapters would provide complete control, but at the cost of maintaining provider-specific logic as APIs evolved. Option three was tempting but unsatisfying.

I chose option two, migrating to ruby_llm. More than swapping dependencies, this was a fundamental architecture change that would affect testing strategy, error handling, configuration, and the public API.

Key technical decisions

Cost optimization through model selection

The most impactful decision was switching from premium to budget models. For Anthropic, this meant claude-sonnet-4 to claude-haiku-20250514, a roughly 10x cost reduction.

This wasn’t about being cheap. (Okay, maybe it was a little bit about that.) It was (mostly) about matching model capability to task complexity. Generating three variations of “Resource not found” doesn’t require deep reasoning. Budget models handle it perfectly well. The quality remained identical while costs dropped by an order of magnitude.

Testing strategy: mock at the right level

The original implementation used VCR to record HTTP interactions. This is a common pattern, but it had problems:

• Maintaining cassettes for 10+ providers would be tedious
• Tests would break when ruby_llm changed request formats
• We would be testing ruby_llm’s HTTP implementation, not our code

The better approach: mock at the ruby_llm interface level. Instead of recording HTTP traffic, we mock RubyLLM.chat() directly. One mock setup works for all providers. Tests are faster, more maintainable, and focused on our actual logic.

This eliminated the need for VCR and WebMock entirely, removing two dependencies. 🎊

As an additional safeguard, I set up a GitHub Actions workflow that runs the full test suite on every push. This catches regressions before they reach main and provides confidence when accepting contributions. It’s a small addition that pays dividends in long-term maintainability.

Code simplification through extraction

FakeGenerator started at 298 lines with embedded message data. Extracting the messages to an external JSON file and refactoring the logic brought it down to 36 lines. That’s an 88% reduction.

Another big win was in AiGenerator. By letting ruby_llm handle provider differences, the cyclomatic complexity dropped from 12 to 1. Twelve decision points (checking provider types, handling edge cases) collapsed into a single code path.

This is the value of a good abstraction layer. It reduces the lines of code, sure, but it also reduces the number of things you have to think about.

To be honest, this is also currently a pain point. We swapped ruby-openai’s abastraction for a better one, but in an evolving ecosystem, we may need to swap again. The key is that the architecture is now flexible enough to accommodate future changes more easily.

Breaking changes as a design tool

The migration required environment variable changes:

• OPENAI_ACCESS_TOKEN → OPENAI_API_KEY
• OLLAMA_URL → OLLAMA_API_BASE

I considered adding migration helpers to detect old variables and warn users. But the gem was pre-1.0 with minimal adoption. Adding complexity for hypothetical users would hurt future maintainability more than it helped current users.

The cleaner approach: document the breaking changes clearly, provide a migration guide, and move forward with consistent naming. Sometimes the right trade-off is accepting short-term pain for long-term simplicity.

Implementation approach

I followed a disciplined TDD approach for the migration:

1. Write failing tests for AiGenerator
2. Implement minimal code to pass
3. Add error handling tests
4. Implement error handling
5. Verify across multiple providers

Each commit represented a logical unit of work with a clear purpose. The git history tells a story: dependency updates, test infrastructure, new generator implementation, factory updates, cleanup, documentation.

This matters for maintainability. Six months from now, when I need to add a new provider or debug an edge case, the git history explains not only what changed but why.

Validating non-deterministic output

Getting AI to generate message variations is great, but how do you know if the variations are any good?

Building a system that validates the quality of non-deterministic output requires deeper thinking about what “quality” even means in this context.

Defining useful variation

A good variation needs two properties:

1. Semantic similarity - It preserves the original message’s meaning
2. Lexical diversity - It uses different wording than other variations

These goals exist in tension. Perfect similarity means identical text. Perfect diversity means unrelated messages. The sweet spot is variations that mean the same thing but say it differently.

I created a validation system that measures both metrics and combines them into an overall quality score. This lets you validate generated variations automatically.

The initial design mistake

My first implementation compared variations to each other. Generate five variations, measure how similar they are as a group, done. This seemed logical until I tested it.

The problem: variations could be highly similar to each other but completely different from the original message. A set of variations about network timeouts would score well even if the original message was about authentication failures. They were similar to each other, but wrong.

The fix was obvious in hindsight: compare each variation to the original message, not to other variations. Semantic similarity measures how well each variation preserves the user’s intent. Lexical diversity measures how much the variations differ from each other. Two separate concerns, two separate comparisons.

Semantic similarity with embeddings

Measuring semantic similarity requires understanding meaning, not just matching words. “Server error” and “Internal server failure” share minimal text but convey the same concept. Simple string comparison would fail.

The solution: embedding models. These convert text into high-dimensional vectors where semantically similar content clusters together. Calculate the cosine similarity between the original message’s embedding and each variation’s embedding, and you have a numeric score for how well meaning is preserved.

I designed this to be provider-agnostic from the start, learning from the earlier migration experience. The validator uses RubyLLM.embed() which works with any provider that supports embeddings (OpenAI, Google, etc.). When embeddings aren’t available, it falls back to a simpler word overlap heuristic.

This graceful degradation was important. Users without embedding access still get validation, just with reduced accuracy. The feature doesn’t silently fail or block users.

API design for optional features

Quality validation needed to be opt-in. The core workflow is “generate variations and use them.” Adding validation steps would slow things down and require configuration. It needed to enhance the workflow without disrupting it.

The CLI design reflects this:

# Basic usage - no validation
hyrum -s openai -m "Server error" -f ruby

# Opt into validation
hyrum -s openai -m "Server error" -f ruby --validate

# Use in CI/CD with strict mode
hyrum -s openai -m "Server error" -f ruby --validate --strict --min-quality 75

Validation is off by default. Enable it when you want quality metrics. Use --strict to fail builds when quality is too low. Use --show-scores to include metrics in generated output.

Each flag serves a specific use case without cluttering the happy path. This is backward compatible and makes the feature discoverable through --help without overwhelming new users.

What I learned

Shipping a Ruby gem is more accessible than I expected

I hadn’t published a gem before this project. The Ruby ecosystem makes it surprisingly straightforward: follow conventions for directory structure, add a gemspec, and gem build handles the rest. RuboCop enforces community standards, and RSpec provides solid testing patterns.

Most of the learning curve wasn’t in the tooling. It was in the design decisions around versioning, breaking changes, and API stability. Understanding when to bump major vs minor versions, when breaking changes are acceptable, and how much backward compatibility to maintain. These are judgment calls that come with experience, not documentation per se.

Abstractions have a cost and a benefit

Ruby_llm’s abstraction eliminated provider-specific code paths. But it also added a dependency and gave up some provider-specific features (like Anthropic’s prompt caching). The trade-off made sense because the gem’s core use case doesn’t need advanced features. Your mileage will vary.

Model selection is a design decision

Defaulting to budget models wasn’t about minimizing costs. It was about right-sizing capability to task complexity. When your task genuinely needs advanced reasoning, use advanced models. When it doesn’t, you’re paying for capability you’re not using.

Testing at the right abstraction level matters

Mocking at the HTTP level tests the wrong thing. Mocking at the library interface level tests your code. The latter is almost always better unless you’re specifically testing HTTP behavior.

Breaking changes are acceptable in context

Pre-1.0 software with limited adoption is the right time to make breaking changes. Adding backward compatibility for a handful of users creates technical debt that affects every future user. Sometimes the generous thing is to break things cleanly.

User feedback catches design flaws early

My initial quality validation design seemed sound in theory. It measured variation quality as a group property. But the first test revealed the flaw: variations could be similar to each other while being completely unrelated to the original message.

This is why you test with real examples before building the whole system. The fix (comparing to the original message) was trivial to implement early. It would have been painful to retrofit later after building an entire validation pipeline on the wrong assumption.

The lesson: design mistakes are inevitable. What matters is catching them before they become weight-bearing walls in your architecture.

Graceful degradation beats hard dependencies

Embedding models provide superior semantic similarity measurement. But requiring them would block users whose AI providers don’t support embeddings. The word overlap fallback isn’t as accurate, but it’s better than nothing.

This pattern appears throughout the gem. Can’t access embeddings? Use heuristics. Provider doesn’t support structured output? Parse text. Each graceful degradation expands the set of valid configurations.

The alternative is failing fast with clear errors. Both approaches are valid, but for a tool that works across many providers, degradation creates a better experience than strict requirements.

Validation changes what “done” means

Before quality validation, “done” meant “generates variations.” After, it meant “generates variations that preserve meaning while varying wording.” This shift changed the entire value proposition.

The interesting part is that validation makes the AI output more trustworthy without requiring a better AI model. Same model, same cost, but now you have quantitative confidence in the results. That’s the leverage of good metrics.

The result

Hyrum now supports 11 AI providers (OpenAI, Anthropic, Gemini, Ollama, Mistral, DeepSeek, Perplexity, OpenRouter, Vertex AI, AWS Bedrock, GPUStack) through a unified interface. The codebase is simpler, tests are faster, and costs are 10x lower.

Quality validation adds confidence without complexity. Generate variations, validate they preserve meaning while varying wording, and integrate the results into your codebase with quantitative quality metrics. The validation system works across all providers and degrades gracefully when embeddings aren’t available.

Best of all, the architecture can accommodate new providers and capabilities without increasing complexity. When ruby_llm adds support for a new provider, Hyrum gets it for free. When embedding models improve, quality validation automatically benefits. That’s the payoff of choosing the right abstractions.

The project is open source on GitHub. If you’re dealing with Hyrum’s Law in your own APIs, or you’re just curious about the implementation details, check it out.

Building a Phaser 2D grid game with Claude

The experiment

This project started with a simple question: How well could an AI coding assistant help me build a game using a framework I’d never touched before?

I’ve spent years writing JavaScript - the language was familiar territory. I had a passing acquaintance with game development concepts. But TypeScript? I was a novice. Phaser? Absolutely zero experience. I expected Claude to excel with TypeScript (plenty of training data), but suspected Phaser knowledge would be sparse.

The central questions:

• Can Claude navigate a less-common API well enough to build something real?
• Will I understand the generated code well enough to modify and extend it?
• What does it take to give an AI assistant enough context to be genuinely useful rather than just generating plausible-looking code that doesn’t work?

I decided to find out. This is the story of building “Infection: Germs vs White Cells” - a turn-based grid game where players compete to dominate the board through strategic dot placement and chain reaction explosions. More importantly, it’s about what I learned collaborating with AI on unfamiliar territory.

The foundation: Getting started

I chose a tech stack that mixed familiar and new territory: Phaser 3 for the game engine, Vue 3 for UI, TypeScript for type safety, and Vite for fast development builds. Vue would handle menus and overlays while Phaser managed gameplay.

The first challenge was connecting Vue and Phaser - they’re designed for different purposes and don’t naturally integrate. After researching examples, we settled on an EventBus pattern. PhaserGame.vue became the bridge component that initializes the Phaser game and sets up bidirectional communication through events.

// Phaser scene emits events to Vue
EventBus.emit('current-scene-ready', this);
EventBus.emit('level-completed', { winner: 'player' });

// Vue listens and responds
EventBus.on('level-completed', (data) => {
  // Update UI, show victory screen, etc.
});

This pattern worked well throughout the project. Phaser handled game logic, Vue handled UI chrome, and they stayed cleanly separated.

The initial gameplay came together surprisingly fast. Within the first session, we had a working grid where players could click cells, place dots, and see them explode when cells reached capacity. The turn system worked. Player indicators updated correctly. The win condition detected when one player controlled the entire board.

Claude was excellent at generating boilerplate and structure for familiar patterns. The Vue-Phaser bridge? That exists in lots of projects. Basic game loops? Common pattern. TypeScript interfaces for game state? Standard stuff.

Core mechanics: Making it feel like a game

Once the foundation worked, we focused on game feel. The core mechanic is simple: click a cell to add a dot. When dots exceed a cell’s capacity, the cell explodes and distributes dots to adjacent cells. This can trigger chain reactions that flip opponent cells to your color.

Cell capacity depends on position:

• Corner cells hold 2 dots (2 neighbors)
• Edge cells hold 3 dots (3 neighbors)
• Interior cells hold 4 dots (4 neighbors)
• Blocked cells hold nothing and don’t contribute to neighbor capacity

The explosion logic needed timing. If chain reactions happened instantly, players couldn’t follow what was happening. We added a 300ms delay between explosions, just enough to watch the cascade unfold without feeling sluggish.

Next we needed variety, different board sizes, obstacles that blocked certain cells, and increasing difficulty. We needed levels. Rather than storing levels in a database, we defined them in code as a linked list structure. Each level points to the next, making navigation intuitive:

const currentLevel = this.getCurrentLevel();
const nextLevel = currentLevel.next();
if (nextLevel) {
  this.loadLevel(nextLevel);
} else {
  this.handleGameOver(winner);
}

We also added undo functionality, because clicking the wrong cell feels terrible in a strategy game. The game tracks the last 50 moves and can roll back the board state. This feature can also be quite useful for testing and debugging.

Making it polished: UI and UX

A working game isn’t the same as a game that feels good to play. We spent time on visual polish - dots that pulse when placed, smooth animations, satisfying sound effects for placement and chain reactions.

The Settings scene let players configure their experience: sound effects on/off, player colors, which level set to play. The responsive design ensured the grid centered properly on different screen sizes.

The tricky part was state preservation. When players navigated from the game to settings and back, they expected their game to still be there. This required careful management of Phaser’s scene lifecycle - sleep and wake rather than destroy and recreate. We’ll come back to this because it caused one of our biggest bugs.

Building AI opponents: From random to strategic

A game against yourself gets boring quickly. We needed a computer opponent, but I didn’t want to build a perfect player - that’s no fun either. Instead, we implemented four difficulty levels with escalating sophistication.

Easy AI picks random valid moves. No strategy, just valid placement.

Medium AI looks for tactical opportunities:

• Explode fully loaded cells (capacity reached)
• Claim corner and edge cells (harder for opponent to capture)
• Otherwise pick randomly

Hard AI adds offensive tactics:

• Prioritize full cells adjacent to opponent cells (capture on explosion)
• Explode full cells next to opponent’s full cells (trigger counter-chains)
• Fall back to medium strategy

Expert AI evaluates positional advantage using “ullage” (remaining capacity). It seeks cells where adding a dot gives advantage over all adjacent opponent cells, forcing the opponent into difficult positions.

Each level specifies its AI difficulty, so players face escalating challenge as they progress through a level set.

The development approach: start simple, iterate to complexity. We built the dumbest thing that could work (random moves), then made it smarter incrementally based on playtesting feedback.

Architecture evolution: When code gets messy

Here’s where I made my first major mistake: I put too much in Game.ts. In self-defense, I didn’t know yes what the architecture should look like yet. So I deferred those decisions until later by putting everything in one place.

At first, this seemed fine. The game logic lived in the game scene. Makes sense, right? But as features accumulated, Game.ts grew to over 1000 lines. It handled grid creation, cell capacity calculations, explosion logic, AI moves, UI updates, state persistence, settings management, and level progression. Reading it required holding too many concepts in your head at once.

The pain became obvious when bugs appeared. Tracking down a state persistence bug meant wading through explosion logic and UI code. Fixing the play order required understanding grid creation. Everything touched everything.

We needed a separation of concerns. Not for “clean code” aesthetics, but because the cognitive complexity made changes risky and debugging slow.

The refactoring happened incrementally, driven by specific pain points:

GameStateManager emerged when state bugs appeared. We needed one clear place responsible for saving and loading game state to Phaser’s registry, handling undo history, and tracking level progression.

GridManager split out when grid logic got complex. Cell capacity calculations, blocked cell handling, hover states, and visual styling didn’t belong mixed with game logic.

GameUIManager formed when UI updates scattered throughout the code. One change to the player indicator required hunting through multiple methods. Now UI creation and updates live in one place.

SettingsManager centralized the synchronization between localStorage and Phaser’s registry. Settings read priority became explicit: registry first, then localStorage, then defaults.

BoardStateManager extracted the core game logic - explosion mechanics, chain reactions, win condition detection. This became the pure game engine, separate from Phaser rendering concerns.

Each refactoring happened when the pain became clear, not as a planned “rewrite day.” We didn’t wait for the perfect time to refactor. We refactored when the current structure made the next feature difficult.

The linked list structure for levels proved elegant. Rather than tracking level indices and bounds-checking arrays, levels just know their next level. The code reads naturally: if (currentLevel.isLast()) instead of if (currentLevelIndex >= levels.length - 1).

After refactoring, most manager classes stayed under 500 lines with single, clear responsibilities. The Game scene itself still exceeds 800 lines (it orchestrates all the managers and handles complex scene lifecycle), but the cognitive load dropped dramatically. You can now understand GridManager without knowing anything about state persistence or AI strategy.

The testing awakening

I need to confess something: I allowed a lot of code to be written before writing tests.

My rationale seemed sound at the time. I was learning Phaser’s architecture and didn’t want to constantly rewrite tests as I figured out the right patterns. Better to get something working first, then add tests once the architecture stabilized.

This was expensive.

Without tests, every refactoring risked breaking something. I’d extract GameStateManager and then manually click through the entire game to verify level progression still worked. I’d modify explosion logic and hand-test edge cases by setting up specific board states. Bugs appeared, got fixed, then reappeared weeks later because nothing prevented regression.

The wake-up call came during a refactoring that broke the undo system in a subtle way. The game worked for new games, but loading a saved game with undo history crashed. I’d fixed this bug before. Now it was back.

We needed comprehensive test coverage.

Working with Claude, we built a four-phase testing plan:

Phase 1: Core data structures (69 tests)

• Level class: linked list navigation, property access, last level detection
• LevelSet class: level management, traversal, bounds checking

Phase 2: Manager classes (196 tests)

• SettingsManager: localStorage/registry sync, defaults, read priority
• GameStateManager: save/load, undo/redo, move history limits
• LevelSetManager: loading definitions, level set switching
• BoardStateManager: game logic, explosions, win conditions

Phase 3: Game logic (88 tests)

• GridManager: cell capacity, blocked cells, hover states
• ComputerPlayer: all four difficulty levels, move validation

Phase 4: UI layer (49 tests)

• GameUIManager: element creation, updates, positioning

We used Vitest because it’s fast, has excellent TypeScript support, and provides a clean testing API. Tests lived next to their source files: GridManager.ts and GridManager.test.ts in the same directory.

The test suite currently has 514 tests across 20 test files, and they run in about 1.2 seconds. Fast enough to run on every save during development.

Writing tests after the fact taught me something: test-driven development exists for good reasons. The tests we wrote exposed edge cases we’d never considered. They caught bugs that would have appeared weeks later. They made refactoring safe instead of terrifying.

If I started this project over, I’d write tests earlier. Not because tests are “best practice,” but because they would have saved me days of debugging time.

🪲 Key challenges and debugging victories

The real learning happened when things broke. Here are four bugs that taught me the most about Phaser, systematic debugging, and AI collaboration.

Challenge 1: The settings scene reset bug

Symptom: Navigate to Settings, change nothing, click Back. The game resets to the first level. Your in-progress game vanishes.

First instinct: The state isn’t being saved. We added logging to GameStateManager. The state was saving perfectly. The state was loading correctly too. What?

Root cause: We were using scene.start() to transition between Game and Settings scenes. This method destroys the current scene and creates a fresh instance of the target scene. When returning to Game, we got a brand new Game scene that ran its create() method, which loaded the first level by default.

The fix: Phaser scenes have a lifecycle: create() runs once when the scene is first instantiated. wake() runs when a sleeping scene becomes active again. We needed:

// In Game scene
navigateToSettings() {
  this.scene.sleep();  // Not scene.start()
  this.scene.launch('Settings');
}

// In Settings scene
goBack() {
  this.scene.stop();
  this.scene.wake('Game');  // Wake the sleeping scene
}

We also added a settingsDirty flag. If settings actually changed, the Game scene’s wake() method reloads them. Otherwise, it just resumes.

Lesson: Understanding framework lifecycles matters. Claude knew the general pattern but didn’t initially suggest wake/sleep because the Phaser API wasn’t in its training data as heavily. Providing links to current Phaser 3.90 documentation helped tremendously. Without docs, Claude would continue to guess based on older API versions, wasting time.

Challenge 2: Level progression bug

Symptom: Complete the first level, click “Next Level.” You see the first level again instead of level 2.

The investigation: We added logging to track what level was being loaded:

console.log(`[GameStateManager] Saved state:
  ${JSON.stringify(boardState)}`);
console.log(`[BoardStateManager] Setting state:
  ${JSON.stringify(boardState)}`);

The logs revealed the issue: we were saving an empty boardState to the registry, which triggered the “new game” code path that loaded level 1.

Root cause: The level completion logic set a loadNextLevel flag, but the state persistence logic also saw an empty board and saved it. This was a race condition where both actions happened simultaneously, and the empty state won.

The fix: Prioritize the loadNextLevel flag. Check it before looking at board state:

wake() {
  const savedState = this.stateManager.loadFromRegistry();

  if (savedState?.loadNextLevel) {
    const nextLevel = this.currentLevel.next();
    if (nextLevel) {
      this.loadLevel(nextLevel);
      return;  // Exit early
    }
  }

  // Otherwise restore board state
  if (savedState?.boardState) {
    this.restoreBoardState(savedState.boardState);
  }
}

Lesson: The linked list structure actually helped here. The code if (nextLevel) makes it obvious we’re checking if a next level exists. With array indices, we’d have if (currentLevelIndex + 1 < levels.length), which is more error-prone.

Challenge 3: Level set changes not taking effect

Symptom: User selects a different level set in Settings, clicks “Play Game.” They see the old level set instead.

Root cause: The Game scene wasn’t checking if settings changed while it was sleeping. It would wake up and continue with the old LevelSetManager.

The fix: The settingsDirty flag from Challenge 1 solved this too. When settings change, the flag gets set. On wake, if the flag is set, reload all settings:

wake() {
  const settingsDirty = this.game.registry.get('settingsDirty');

  if (settingsDirty) {
    this.reloadAllSettings();
    this.game.registry.set('settingsDirty', false);
  }

  // ... rest of wake logic
}

We also needed defensive logic. What if the user changed both player color AND level set? Both changes needed to take effect together, not sequentially with potential state corruption between them.

Lesson: State synchronization across scene transitions requires explicit change detection. Don’t assume data hasn’t changed while your scene slept.

Challenge 4: Memory leaks from event listeners

Symptom: During manual testing, I noticed the browser memory footprint growing as I transitioned between scenes repeatedly. Something was leaking.

The investigation: We added event listener counting to each scene:

shutdown() {
  console.log(`[${this.constructor.name}] Listeners before cleanup:
    ${this.events.listenerCount('pointerdown')}`);
  this.cleanupEventListeners();
  console.log(`[${this.constructor.name}] Listeners after cleanup:
    ${this.events.listenerCount('pointerdown')}`);
}

The counts kept growing. Event listeners weren’t being cleaned up on scene transitions.

The fix: We added explicit cleanup methods to every scene using TDD. First, write a test that verifies listeners are removed:

it('should clean up all button event listeners on shutdown', () => {
  scene.create();
  const beforeCount = scene.events.listenerCount('pointerdown');
  expect(beforeCount).toBeGreaterThan(0);

  scene.cleanupButtonListeners();

  const afterCount = scene.events.listenerCount('pointerdown');
  expect(afterCount).toBe(0);
});

Then implement cleanup:

shutdown() {
  this.cleanupButtonListeners();
  this.cleanupGridListeners();
  this.cleanupUIListeners();
}

We built a live testing dashboard (npm run test:live) that shows real-time event listener counts during rapid scene transitions. You can watch the numbers and verify they don’t accumulate.

[Screenshot: Live testing dashboard showing event listener metrics would go here]

Lesson: Building testing infrastructure surfaced issues we didn’t know existed. The process of creating the dashboard forced us to think about how to measure cleanup, which led us to Phaser’s listenerCount() API and revealed leaks throughout the codebase. We now have 95% test automation for event cleanup validation and zero known memory leaks.

Common thread: Each bug revealed itself through evidence gathering (logging, instrumentation) rather than guessing. This pattern became the foundation for our systematic debugging approach.

The debugging discipline

These four challenges revealed something important: guessing doesn’t scale, even for AI.

Early in the project, Claude would hit a bug and immediately suggest a fix. Didn’t work? Try another. Still broken? Try a third. This guess-and-check thrashing wasted hours and often made problems worse.

I had access to Jesse Vincent’s systematic debugging “superpowers” (think of them as process discipline plugins for Claude). They just weren’t being enforced. After enough frustration, I made the systematic debugging superpower mandatory in CLAUDE.md - the project documentation file that guides Claude’s behavior. The protocol is straightforward:

1. Gather evidence first - Add logging to understand what’s actually happening, not what you think is happening
2. Analyze patterns - Compare working vs broken implementations
3. Test single hypotheses - Make one targeted change to test a theory
4. Fix root causes - Address the actual problem, not symptoms

It also includes this note: “Systematic debugging is 5x faster than guess-and-check thrashing.”

Before adding this additional directive, Claude would often suggest fixes immediately: “Try changing this API call” or “Maybe add this flag.” After adding it, Claude would first suggest adding instrumentation: “Let’s add logging to see what values we’re actually getting.”

The difference was dramatic. Bugs that previously took hours to solve took 30 minutes. We stopped creating bugs while fixing bugs.

Full disclosure: I’m a senior developer and I knew better. But curiosity got the best of me. One of my goals was understanding AI behavior patterns to collaborate more effectively on future projects. I wanted to see if Claude could guess its way to solutions.

The answer: Sometimes, but unreliably. With sufficient context, guessing (or pattern recognition that looks like guessing) often worked. On unfamiliar frameworks like Phaser, it usually failed.

Key insights:

• For Claude: Evidence before action. Always.
• For me: Enforce systematic approaches through CLAUDE.md, don’t rely on AI self-discipline.

For scene lifecycle bugs, we added comprehensive logging:

create() {
  console.log(`[${this.constructor.name}] ===== SCENE CREATE START =====`);
  // ... scene creation code ...
  console.log(`[${this.constructor.name}] ===== SCENE CREATE END =====`);
}

wake() {
  console.log(`[${this.constructor.name}] ===== SCENE WAKE START =====`);
  const settings = this.game.registry.get('settingsDirty');
  console.log(`[${this.constructor.name}] Settings dirty: ${settings}`);
  // ... wake logic ...
}

This logging made scene transitions visible. We could see exactly when create() ran vs wake(), what data each method received, and what order operations occurred in. Bugs became obvious instead of mysterious.

Working with Claude: What worked, what didn’t

Hiring managers care about productivity and code quality. Here’s an honest assessment of where AI helped, where it struggled, and what that means for development teams.

What worked well

Boilerplate and structure: Claude excels at generating TypeScript interfaces, class structures, and common patterns. Need a manager class with standard CRUD operations? Claude writes it in seconds.

Pattern recognition: “This looks like the command pattern” or “This is similar to the observer pattern we used for events” - Claude connects new problems to solved problems effectively.

Architectural improvements: When I recognized Game.ts had grown too large at 1000+ lines, Claude suggested extraction patterns that made sense. The refactoring strategies were sound once the problem was identified.

Test writing: Once we established a pattern for one test file, Claude could generate similar tests for other classes. The 514 tests would have taken weeks to write manually.

Systematic debugging: Once convinced to use the systematic debugging superpower, Claude followed it reliably. This helped enormously and saved a ton of time.

What required guidance

Phaser API specifics: This was the biggest challenge. Claude’s training data apparently has much less Phaser content than TypeScript or Vue. It would suggest API calls that sounded plausible but didn’t exist, or use patterns from older Phaser versions.

The solution: Provide links to current Phaser 3.90 documentation. When I sent Claude snippets from official docs, suggestions became more accurate. Without docs, Claude would guess, and guessing wasted time.

Project-specific architecture decisions: Claude couldn’t decide whether to use localStorage vs Phaser’s registry, or when to extract a manager class vs keep code together. These decisions required human judgment based on project context. Clearer instructions in CLAUDE.md helped, but some decisions still needed human guidance.

Refactoring timing: Claude would sometimes suggest refactoring when we needed to ship, or suggest shipping when the code really needed cleanup. The “when” required human intuition.

Testing discipline: Without explicit guidance, Claude would tend to neglect testing. It would happily write feature after feature without suggesting tests. The comprehensive test suite only happened because I explicitly requested it and then added requirements to CLAUDE.md.

Claiming victory too early: Claude would repeatedly declare a bug fixed after a potential fix. Before any verification it was ready to mark the item as complete and move on. It needed reminders to verify that changes actually worked.

The CLAUDE.md evolution

CLAUDE.md started as a basic README: project structure, how to run tests, basic architecture notes.

It evolved into the project brain - a comprehensive guide that overrides Claude’s default behavior:

## 🚨 MANDATORY DEBUGGING PROTOCOL

**FOR ANY TECHNICAL ISSUE - ALWAYS use systematic debugging**

**FORBIDDEN PATTERNS (cause more bugs than they fix):**
- "Quick fixes" and guesswork - **STRICTLY PROHIBITED**
- Trying random API calls without understanding root cause
- Making multiple changes at once

Before this protocol existed, Claude would get caught in guessing loops. Try a fix, doesn’t work, try another, still broken, try a third. The mandatory protocol broke this pattern.

We documented every resolved bug: symptoms, root causes, fixes. When similar issues appeared later, the documentation provided patterns to recognize.

The key realization: Documentation is bidirectional. I taught Claude about the project, and the process of explaining things to Claude clarified my own understanding. Writing clear instructions forced me to think clearly about solutions. Some of the lessons learned here will be added to the global ~/.claude/CLAUDE.md file for future projects.

Bottom line: AI assistance works best as a partnership. The human brings judgment, context, and architectural vision. The AI brings speed, consistency, and tireless execution of well-defined tasks. Neither replaces the other.

Lessons learned

Here’s what I’d tell my past self, or anyone starting a similar project.

Technical lessons

1. Test early, test often

Writing tests after code cost significant rework time. Tests exposed edge cases we’d never considered. They caught regressions before they shipped. They made refactoring safe.

If I restarted this project, tests would come first. Not as “best practice” dogma, but as a practical time-saving tool.

2. Scene lifecycle matters

Understanding create() vs wake() vs sleep() vs shutdown() is critical in Phaser. The wrong method causes subtle bugs. The right method makes everything work.

3. Registry over localStorage

We used Phaser’s registry as the single source of truth for runtime state, with localStorage only for persistent settings. This prevented synchronization bugs between storage systems.

4. Separation of concerns reduces cognitive load

This isn’t about “clean code” aesthetics. When Game.ts exceeded 1000 lines, making changes became risky because every change could affect multiple unrelated features. After extracting manager classes, each file became understandable in isolation.

5. Linked lists for sequential navigation

The linked list structure for levels made code readable: currentLevel. next() instead of array index math. It also made the progression concept explicit in the data structure.

6. Event cleanup is not optional

Memory leaks accumulate silently. Without explicit cleanup and testing, the browser’s memory footprint grows. Players might not notice on first play, but the leak still exists.

AI collaboration lessons

1. Documentation is bidirectional

Teaching Claude about the project clarified my own thinking. Writing instructions forced clear problem statements. The CLAUDE.md file became as valuable for me as for the AI.

2. Systematic approaches scale

Ad-hoc debugging doesn’t work on complex projects. The mandatory debugging protocol saved enormous amounts of time by preventing guess-and-check thrashing.

3. Start simple, iterate to complexity

We didn’t architect everything perfectly from day one. We built the simplest thing that could work (random AI, basic grid) then made it more sophisticated incrementally. This approach worked much better than trying to design the perfect system upfront.

4. Context files matter

CLAUDE.md became the project brain. It captured architectural decisions, debugging patterns, resolved bugs, and mandatory workflows. Without it, every conversation started from zero.

5. AI is better with constraints

Claude works best with clear protocols and explicit constraints. “Debug this bug” leads to guessing. “Follow the systematic debugging protocol to investigate this bug” leads to instrumentation and evidence gathering.

Process lessons

1. Git commit messages tell the story

Using conventional commits (feat:, fix:, refactor:, test:) made history searchable. When debugging the level progression bug, searching for “fix: level” immediately found commit 0eb8769. When preparing this blog post, running git log --oneline --reverse showed the project evolution clearly.

2. Refactoring is continuous

We didn’t schedule “refactoring week.” We refactored when current structure made the next feature difficult:

• Game.ts hit 1000+ lines → extracted GameStateManager
• State bugs appeared → extracted BoardStateManager
• Grid logic got complex → extracted GridManager
• UI updates scattered → extracted GameUIManager

Each refactoring addressed immediate pain, not theoretical future problems.

3. Build testing infrastructure

The live testing dashboard (npm run test:live) seemed like overkill for a simple game. But building it forced us to think about how to measure event cleanup, which revealed the listenerCount() API, which exposed leaks we didn’t know existed. The infrastructure paid for itself.

4. Document gotchas immediately

Every resolved bug went into CLAUDE.md immediately. The documentation prevented the same bug from reappearing and provided patterns for similar issues. Future me thanked past me repeatedly.

The final numbers

What AI-assisted development produced:

Code quality metrics:

• 514 tests across 20 test files (started with 0, grew through 4-phase testing plan)
• 95% test automation for event cleanup validation
• ~1.2 second test suite execution time (fast enough to run on every save)
• Zero known memory leaks after systematic cleanup and monitoring

Architecture:

• 9 core manager classes handling distinct concerns (extracted from monolithic Game.ts)
• 10 Phaser scenes managing game flow (Boot → Preloader → Splash → MainMenu → Game/About/Tutorial/Settings → LevelOver → GameOver)
• 4 AI difficulty levels implementing escalating strategic sophistication

Content:

• Multiple level sets with 5-7 levels each
• Variable board sizes and blocked cell patterns for strategic variety
• 100+ commits documenting the evolution with conventional commit format

The game is playable, maintainable, and well-tested. More importantly, the codebase is understandable. A developer new to the project could read GridManager without knowing anything about AI strategy, or modify explosion logic without understanding state persistence. That’s what separation of concerns actually buys you.

Was it worth it?

Absolutely.

I built a working game using a framework I’d never used, and the code is maintainable enough that I’d be comfortable handing it to another developer. That’s the real test.

Claude bridged knowledge gaps effectively where it had training data (TypeScript, design patterns). Where it lacked context (Phaser specifics, project architecture), providing documentation and clear constraints made it productive.

The discipline of testing and documentation paid dividends. The 514-test suite catches regressions before they reach production. The CLAUDE.md file captures institutional knowledge that would otherwise live only in my head. The systematic debugging protocol prevents guess-and-check thrashing that wastes hours.

Key insight: AI works best with clear constraints and feedback. Without the debugging protocol, Claude would guess. With it, Claude would gather evidence. Without test requirements, Claude would skip tests. With requirements, it wrote comprehensive coverage.

The game is playable and reasonably fun. The AI provides genuine challenge. The animations feel responsive. Is the architecture perfect? Honestly, I don’t know. But it’s good enough to ship, iterate on, and extend - which is the point.

Would I do it again? Absolutely, but I’d establish testing discipline earlier and encode systematic approaches in CLAUDE.md from day one.

Try it yourself

Want to see the results or dig into the implementation?

• Play the game - Try it in your browser (no installation required)
• View the source - Explore the code with full commit history
• Read CLAUDE.md
  • See the “project brain” that guided development decisions

If you’re considering AI-assisted development, especially with unfamiliar frameworks, here’s what I learned works:

• Provide current documentation when the AI lacks training data (links to official docs beat guessing every time)
• Establish systematic approaches early (debugging protocols, testing requirements)
• Write tests as you go, not retrospectively
• Use a context file (CLAUDE.md) to capture architecture decisions and patterns
• Expect to teach the AI your project specifics - documentation is bidirectional

The partnership model: You bring judgment, architectural vision, and domain knowledge. AI brings speed, consistency, and tireless execution of well-defined tasks. Neither replaces the other, but together they can tackle unfamiliar territory effectively.

Now go build something.

Building a document Q&A system with Rails 8, SQLite-vec, and OpenAI

Every company wants to plug AI into their proprietary data. But here’s the thing—it’s not just about calling an API. The real challenge is building systems that actually work reliably and are maintainable in the real world.

So I built Ragtime. Think of it as my playground for figuring out how to build modern AI applications the right way. It’s a document Q&A system where you can upload PDFs, Word docs, text files, and Markdown, then ask questions and get answers with actual source citations.

What I really wanted to show wasn’t just that I can build an AI app—anyone can do that these days. I wanted to demonstrate how a senior engineer thinks about building these systems from the ground up, making smart trade-offs and avoiding common pitfalls.

Architecture overview

So what does Ragtime actually look like under the hood? At its core, it’s a RAG (Retrieval-Augmented Generation) system—fancy talk for “find relevant material, then use it to answer questions.”

Here’s how it works: you upload a document, the system pulls out the text, chops it into smart chunks, turns those chunks into mathematical vectors (more on that in a bit), and then when you ask a question, it finds the most relevant chunks and uses them to generate an answer with citations. Simple, right? Well, there are some fun challenges along the way.

Why this architecture?

Let me tell you why I picked this stack. Rails 8 just clicked for the backend because it’s got some really nice improvements for modern apps—better API support out of the box, Solid Queue built right in (no more juggling separate worker processes), and some solid performance tweaks–especially for SQLite. Going API-only gave me this clean separation between frontend and backend, which makes everything easier to maintain and reason about.

For the frontend, Vue.js 3 with the Composition API just feels right for chat interfaces. You get way better state management than server-rendered options, and the component-based architecture makes complex UI stuff like real-time chat and interactive citations so much easier to build.

The vector storage decision: SQLite + sqlite-vec

This was probably the biggest technical decision I had to make—how to store all those vector embeddings. I looked at three options:

1. PostgreSQL + pgvector: This is what everybody uses in production
2. Dedicated vector databases (like Pinecone or Weaviate): The fancy specialized solutions
3. SQLite + sqlite-vec: The simple, “just make it work” approach

I went with SQLite + sqlite-vec, and I have to be honest—it was way more complicated than I expected. Here’s my thinking: when you’re trying to show engineering competence, deployment simplicity matters more than theoretical scalability. But sqlite-vec turned out to be a significant engineering challenge in its own right.

The extension doesn’t just work out of the box. I had to:

• Bootstrap it manually in the docker-entrypoint for production
• Load it programmatically in code rather than declaring it in database.yml
• Add special verification in health check endpoints because if it fails to load, the entire RAG functionality breaks
• Turn off transactional fixtures in tests and create custom test support code

While SQLite gives you that single-file, zero-dependency promise, getting sqlite-vec to work reliably required significant effort. This really highlights a key engineering principle: sometimes the “simple” choice brings its own complex challenges that you need to account for.

Document processing pipeline

Building a document processing pipeline that actually works reliably meant solving some pretty fun challenges:

Challenge 1: Getting text out of different file types

First problem: documents come in all shapes and sizes. PDFs, Word docs, plain text, Markdown—each one needs its own special handling trick. I ended up building a TextExtractor service that’s basically a Swiss Army knife for file formats. It knows how to handle each type, and when things go wrong (which they always do), it fails gracefully and tells you what happened.

# app/services/document_processing/text_extractor.rb
class DocumentProcessing::TextExtractor
  def self.extract(file)
    case file.content_type
    when 'application/pdf'
      PdfReader.new(file).extract_text
    when 'application/vnd.openxmlformats-officedocument.wordprocessingml.document'
      DocxReader.new(file).extract_text
    when 'text/plain', 'text/markdown'
      file.download
    else
      raise "Unsupported file type: #{file.content_type}"
    end
  rescue => error
    Rails.logger.error "Text extraction failed: #{error.message}"
    raise DocumentProcessing::ExtractionError, "Failed to extract text from document"
  end
end

Challenge 2: Splitting text intelligently

Getting the chunking right is crucial for RAG systems. The key challenge is maintaining context—chunks need to be small enough to be relevant but large enough to retain meaning.

I implemented a TextChunker that creates 800-token chunks with 200 tokens of overlap, which is a well-established pattern in RAG systems. The overlap ensures related context spans multiple chunks, and the system respects paragraph boundaries to maintain semantic coherence.

The overlap is particularly important—it keeps related ideas connected across chunk boundaries. I used tiktoken_ruby for accurate token counting because different models count tokens differently, and that accuracy significantly affects chunk quality.

Challenge 3: Implementing vector similarity search

The ChunkRetriever handles the core vector search functionality. It generates embeddings for queries and uses sqlite-vec’s virtual tables to find the most similar chunks. The challenge here was tuning the similarity threshold—too low and you get irrelevant results, too high and you get no results at all.

# app/services/rag/chunk_retriever.rb
class Rag::ChunkRetriever
  DEFAULT_THRESHOLD = 1.2 # L2 distance, tuned for quality results
  DEFAULT_LIMIT = 5

  def initialize(query:, threshold: DEFAULT_THRESHOLD, limit: DEFAULT_LIMIT)
    @query = query
    @threshold = threshold
    @limit = limit
  end

  def call
    # Generate embedding for query
    query_embedding = EmbeddingGenerator.generate(@query)

    # Vector similarity search using sqlite-vec
    chunks = execute_vector_search(query_embedding)

    # Convert to domain objects with metadata
    chunks.map { |chunk| ChunkResult.new(chunk) }
  end

  private

  def execute_vector_search(embedding)
    sql = <<-SQL
      SELECT chunks.*, vec_distance_cosine(chunks.embedding, ?) as distance
      FROM vec_chunks
      JOIN chunks ON vec_chunks.rowid = chunks.id
      WHERE vec_distance_cosine(chunks.embedding, ?) < ?
      ORDER BY distance
      LIMIT ?
    SQL

    sanitized_sql = ActiveRecord::Base.sanitize_sql_array([
      sql, embedding, embedding, @threshold, @limit
    ])

    ActiveRecord::Base.connection.execute(sanitized_sql)
  end
end

Challenge 4: Background job processing

Document processing is computationally expensive—text extraction, chunking, and embedding generation can take significant time. This work needs to happen asynchronously to avoid blocking user interactions.

Rails 8’s Solid Queue with the Puma integration was an excellent choice for this use case. No separate worker processes to manage—everything runs in the background while keeping the app responsive. The in-process approach simplifies deployment significantly.

# app/jobs/process_document_job.rb
class ProcessDocumentJob < ApplicationJob
  retry_on StandardError, wait: :exponentially_longer, attempts: 3

  def perform(document)
    document.update!(status: :processing)

    # Extract text content
    text_content = DocumentProcessing::TextExtractor.extract(document.file)

    # Create chunks with overlap
    chunks = DocumentProcessing::TextChunker.chunk(text_content)

    # Generate embeddings in batches
    embeddings = EmbeddingGenerator.generate_batch(chunks.map(&:content))

    # Store chunks with embeddings
    chunks.each_with_index do |chunk, index|
      document.chunks.create!(
        content: chunk[:content],
        position: chunk[:position],
        token_count: chunk[:token_count],
        embedding: embeddings[index]
      )
    end

    document.update!(status: :completed)
  rescue => error
    document.update!(status: :failed, error_message: error.message)
    raise
  end
end

This in-process approach means way less deployment headache while still giving you reliable background job processing. Sometimes simpler really is better.

Frontend architecture: Vue.js 3 + Composition API

Building a chat interface that doesn’t feel clunky is surprisingly hard. You need to manage conversation state, message history, real-time updates… it gets complicated fast.

Vue.js 3’s Composition API turned out to be perfect for this. It gives you these clean patterns for organizing complex component logic without everything turning into spaghetti code. The chat interface keeps track of the conversation, shows you messages as they come in, and makes citations clickable—click one and it’ll highlight the exact passage in the document.

I used Pinia for state management because it makes debugging so much easier, and the whole component structure follows that separation of concerns principle that keeps you sane when the app gets complex.

// ChatInterface.vue (simplified)
import { ref, computed, onMounted } from 'vue'
import { useChatStore } from '@/stores/chat'

export default {
  setup() {
    const chatStore = useChatStore()
    const message = ref('')
    const loading = ref(false)

    const sendMessage = async () => {
      if (!message.value.trim()) return

      loading.value = true
      try {
        await chatStore.sendMessage(message.value)
        message.value = ''
      } finally {
        loading.value = false
      }
    }

    return {
      message,
      loading,
      currentChat: computed(() => chatStore.currentChat),
      sendMessage
    }
  }
}

Citation extraction and storage

Here’s something that drives me crazy about some AI apps: you get these confident-sounding answers but have no idea where they came from. That’s terrible for user trust.

So I made sure every answer comes with citations. The AnswerGenerator service basically tells the AI “hey, when you answer this, tell me exactly which chunks you used” and stores all that in a nice structured format. This way users can actually verify the answers, which is huge for building trust.

# app/services/rag/answer_generator.rb
class Rag::AnswerGenerator
  def initialize(context:, question:)
    @context = context
    @question = question
  end

  def call
    response = ruby_llm.chat(
      messages: build_prompt,
      temperature: 0.3, # Lower temperature for more consistent responses
      response_format: { type: "json_object" }
    )

    parse_response(response)
  end

  private

  def build_prompt
    PromptBuilder.new(
      context: @context,
      question: @question,
      citation_format: "structured_json"
    ).build
  end

  def parse_response(response)
    data = JSON.parse(response.content)

    AnswerResult.new(
      content: data['answer'],
      citations: build_citations(data['citations'] || []),
      metadata: {
        model: response.model,
        usage: response.usage
      }
    )
  end

  def build_citations(citation_data)
    citation_data.map do |citation|
      {
        chunk_id: citation['chunk_id'],
        document_id: citation['document_id'],
        document_title: citation['document_title'],
        relevance: citation['relevance_score'],
        position: citation['position_in_document']
      }
    end
  end
end

All those citations get stored as JSON in the messages table, which means you can replay conversations later and see exactly how the AI arrived at its answers. Pretty handy for debugging and audit trails.

Production deployment strategy

Getting an AI app to production is… an adventure. You’ve got infrastructure, security, and a million operational concerns to think about. I went with Docker containers and Fly.io because they strike a nice balance between power and simplicity.

Container architecture

The multi-stage Dockerfile was actually pretty fun to build. It optimizes for both development and production, which means faster builds when you’re iterating and smaller images when you’re deploying:

# Multi-stage production Dockerfile
FROM ruby:3.3-slim AS base
WORKDIR /app
COPY Gemfile Gemfile.lock ./
RUN bundle install --deployment --without development test

# Build stage for frontend assets
FROM node:18-alpine AS frontend-build
WORKDIR /app/frontend
COPY frontend/package*.json ./
RUN npm ci
COPY frontend/ ./
RUN npm run build

# Production stage
FROM ruby:3.3-slim AS production
WORKDIR /app
COPY --from=base /usr/local/bundle/ /usr/local/bundle/
COPY . .
COPY --from=frontend-build /app/frontend/dist /app/public/frontend

# Production configuration and startup
RUN bin/rails assets:precompile
EXPOSE 8080
CMD ["./bin/docker-entrypoint"]

Operational stuff that actually matters

Production isn’t just about getting it running—it’s about keeping it running. Here’s what I built in:

• Health checks: Custom endpoints so I can actually tell if the app and database are happy
• Proper logging: Structured logging with correlation IDs because debugging production without context is hell
• Secrets management: Rails credentials for API keys (never commit those to git!)
• Persistent storage: Fly.io persistent volumes for the SQLite database and uploaded files
• Cross-platform builds: Docker buildx so I can build on my ARM64 Mac but deploy to AMD64 servers

All this stuff matters way more than most people think when they’re starting out.

Code quality and testing approach

Look, here’s the thing about AI systems: they’re nondeterministic. The same input can give you slightly different outputs, which makes testing… interesting. But comprehensive testing is still absolutely crucial.

The test suite covers all the important bits:

# RAG Pipeline Integration Test
RSpec.describe "RAG Pipeline Integration", type: :request do
  it "processes document and answers question with citations" do
    # Upload document
    document = create_document_with_file("sample.pdf")

    # Process document through pipeline
    ProcessDocumentJob.perform_now(document)
    expect(document.reload.status).to eq("completed")
    expect(document.chunks.count).to be > 0

    # Ask question
    retriever = Rag::ChunkRetriever.new(query: "What is the main topic?")
    chunks = retriever.call

    # Generate answer
    generator = Rag::AnswerGenerator.new(
      context: chunks,
      question: "What is the main topic?"
    )
    answer = generator.call

    expect(answer.content).not_to be_empty
    expect(answer.citations).not_to be_empty
    expect(answer.citations.first[:document_id]).to eq(document.id)
  end
end

Right now I’ve got 222 passing tests, including:

• Unit tests for all the services and models
• Integration tests that test the whole RAG pipeline end-to-end
• API endpoint tests for every controller
• Frontend component tests for the Vue.js interfaces

That many tests might seem like overkill for a portfolio project, but when you’re dealing with AI systems, you need all the confidence you can get.

Key technical trade-offs

Building Ragtime meant making some interesting calls. Here are the big ones:

SQLite vs PostgreSQL for vector storage

• What I chose: | SQLite + sqlite-vec
• Why: Rails 8’s SQLite optimizations make production-scale deployment viable
• The trade-off: Native extension complexity vs single-container deployment simplicity

Here’s something that might surprise you: Rails 8 ships with SQLite optimizations that can handle 50K concurrent users and up to 50K writes/sec. That’s legitimate production scale that completely changes the old assumption that SQLite is just for small apps.

The real challenge wasn’t SQLite itself—it was the sqlite-vec extension. That required significant engineering effort to make work reliably—manual bootstrapping, custom test support, health check verification. But with Rails 8’s improvements, choosing SQLite for production is actually a defensible decision for many use cases.

Background jobs: Solid Queue vs Sidekiq

• What I chose: Solid Queue with in-process Puma integration
• Why: Rails 8 integration means no separate worker processes to manage
• The trade-off: Less isolation vs way simpler deployment

This leverages Rails 8’s new features while cutting down on operational complexity. If this were a bigger system, I’d probably go with dedicated Sidekiq workers for better isolation and monitoring.

Frontend: Vue.js vs Hotwire

• What I chose: Vue.js SPA
• Why: Better UX for chat interfaces with complex state management
• The trade-off: More complex setup vs staying within the Rails ecosystem

Vue.js gives you better tools for managing conversation state, real-time updates, and interactive citations—all critical for a chat experience that doesn’t feel clunky.

What I’d do differently at scale

Ragtime is perfectly suited for its purpose as a portfolio project that demonstrates solid engineering and trade-offs. But if I were building this for different production scenarios? Some decisions might change:

1. Vector extension: For really large scale, I’d evaluate pgvector for its mature ecosystem, but with Rails 8’s SQLite improvements, SQLite remains viable for many production workloads
2. Background jobs: Dedicated Sidekiq workers for better isolation and monitoring at larger scale
3. Asset serving: CDN integration for static assets
4. Monitoring: Full observability stack with Prometheus, Grafana, and proper alerting
5. Caching: Redis for frequent queries and expensive operations
6. Security: Zero-trust architecture with proper API rate limiting

The key is knowing when to optimize for simplicity and when to optimize for scale—and with Rails 8, that scale threshold is higher than most people think.

Lessons learned

Building Ragtime provided valuable insights into modern AI application development:

Technical learnings

• sqlite-vec is not plug-and-play—it requires significant bootstrapping and error handling
• Rails 8 features significantly improve developer experience for API applications
• Vector similarity tuning is crucial for RAG quality—threshold selection requires testing and iteration
• Container cross-platform builds require careful dependency management
• Native extensions in production containers need special handling and verification

Process learnings

• Comprehensive test coverage is essential for AI systems with nondeterministic outputs
• Documentation as a design tool prevents over-engineering
• Simple deployment strategies accelerate iteration and learning
• Error boundaries and graceful degradation are non-negotiable for production AI systems

Architecture insights

• Modularity enables testing and iteration on complex pipelines
• Background job patterns determine user experience quality
• Separation of concerns simplifies AI integration
• Production readiness requires operational thinking from day one

Conclusion

Ragtime demonstrates how to build modern AI-powered applications with solid engineering practices. The system showcases:

• System architecture: Clean separation of concerns with modern Rails 8 patterns
• AI integration: Practical RAG implementation with production considerations
• Frontend development: Vue.js 3 with proper state management and UX focus
• DevOps practices: Container deployment with operational awareness
• Code quality: Comprehensive testing and maintainable code organization

More importantly, it shows how to make thoughtful technology decisions based on project constraints rather than simply following trends. Sometimes the right solution isn’t the most complex one—it’s the one that solves the actual problem efficiently and maintainably.

Request access to live demo - Password-protected demo View source code - Complete implementation Portfolio & contact - More projects and info

If you need the kind of technical leadership and engineering excellence that balances technical chops with practical constraints to build solutions that actually work in the real world, please reach out.