Code Change Summaries

Save time and get better results

Do you love summarizing code changes for PR descriptions or release notes? Yeah, neither do I. Fortunately, LLMs are pretty good at it.

Here’s a quick, practical use for AI in software development. When you’re working on a project, you often need to communicate a summary of the changes made to the codebase. This can be time-consuming, depending on the scope of the changes. If you’re anything like me, the quality of those summaries can vary widely depending on the scope of changes, how much time you have, and how much sleep you need.

The good news is that AI tools are fairly good at summarizing text. So why not use them to summarize code changes? This can not only save you time, but also produce more consistent summaries and help you communicate more clearly with your team. Spoiler: Fine tuning your prompt (an upfront cost) will be key to getting results that sound the way you want them to.

Use cases

Let’s look at several cases where this technique can save you time and improve your communication:

• Commit messages: Summarize the changes made in a commit.
• Pull requests: Summarize the changes made in a pull request branch.
• Branch summaries: Summarize the changes made in any branch.
• Release notes: Summarize the changes made in a release.
• Change logs: Summarize and format for a changelog.

Now that we know when to use AI summaries, let’s look at how to create them.

Models and tools

There are a lot of models out there that can help you summarize text. Most of them can do a pretty good job of summarizing code changes, as long as you provide them with the right input. Right now, I tend to use Claude for this kind of task due to its facility with both code and natural language processing. That said, GPT-3, or even simpler local models would likely work well for this task. You may want to experiment with different models and tools to find the one that works best for you.

As for tools, you could use a web-based chat interface to start, a simple script (python, bash, etc.) that calls a model to summarize the text, or you could use a more sophisticated tool or script that integrates with your version control system to automatically summarize code changes. The latter option would require more setup, but could save you a lot of time in the long run.

The nitty gritty

Input

The input to the model will be the code changes you want to summarize. I use a git log with both commit messages and diffs. The basic format is:

git log -p <commit_range> > /tmp/changes.txt

This basic command will work for most use cases, but you may need to tweak it depending on your needs. Here are a few quick examples:

# Summarize the changes in the last commit
git log -p -1 > /tmp/changes.txt

# Summarize the changes in the last 5 commits, etc.
git log -p -n 5 > /tmp/changes.txt

# Summarize the changes in a specific commit range
git log -p <commit_hash1>..<commit_hash2> > /tmp/changes.txt

# Summarize the changes in a specific branch
git log -p ...<branch_name> > /tmp/changes.txt

Prompt

The prompt you provide to the model is key to getting good results. You’ll need to experiment with different prompts to find the one that works best for you. Here’s a basic prompt that you can use as a starting point:

Below I've attached the output of a `git log -p` command
describing a set of code changes for a PR. Please provide
a summary of the changes suitable for a PR description.

Let’s say you have a file called ~/tmp/prompt.txt that contains this prompt. Now let’s put it all together.

Execution

As I mentioned above, you have a few options for creating a summary now that you have the prompt and the input. You could start out by just copying and pasting the prompt and the input into a web-based chat interface like Claude or ChatGPT. Personally, I prefer to use a simple script I can run in the terminal. Here’s an example of a simple bash script that uses aichat to summarize the changes:

aichat -f ~/tmp/prompt.txt -f ~/tmp/changes.txt

Of course, you could put all the steps together in a single script to make the process more efficient. Here’s a simple example that handles getting branch changes and generating the summary in one go:

#!/bin/bash

# Verify that a branch name was provided
if [ -z "$1" ]; then
    echo "Usage: ${0##*/} <branch_name>"
    exit 1
fi

BRANCH_NAME=$1

# Create the prompt
cat << 'EOF' > /tmp/prompt.txt
Below I've attached the output of a `git log -p` command
describing a set of code changes for a PR. Please proivde
a summary of the changes suitable for a PR description.

EOF

# Get the changes
git log -p ...$BRANCH_NAME > /tmp/changes.txt

# Summarize the changes
aichat -f /tmp/prompt.txt -f /tmp/changes.txt | tee /tmp/summary.txt

The sky’s the limit when it comes to automating this process. You could handle any of the above use cases, you could do it without tmp files, you could copy the output to the clipboard, you could make sure that BRANCH_NAME wasn’t checked out, etc. The key is to experiment and find a workflow that works for you.

Iteration

Once you have a basic workflow in place, there are two main areas for improvement: the automation process and the prompt itself.

Iterating on the automation process can be fun, but a lot of that will come down to your personal preferences and the tools you’re using. I’ll probably stick with a bash script for now, but I’m sure it will evolve over time. Maybe I’ll go nuts some day and use gum to automate the whole thing. Who knows?

The prompt, however, is where you should spend most of your early iteration time. Make sure to add the kind of sections that you want in your summaries, and give it clear instructions on the kind of tone and phrasing you want. You can also experiment with completely different prompts to see what works best for you.

What didn’t work

While summarizing code changes seems to work well, I’ve also explored other potential uses for this approach. I tried using a similar prompt to create a test plan for the given code changes. Unfortunately, the input only has the code changes and commit messages as context. It doesn’t see the surrounding code context for the changes and it doesn’t know what tests are already in place. I mean, the output may be useful as a starting point for dev testing, or to help you think about what tests you might need, but it’s not going to be a complete test plan by any stretch.

Summary

Using AI to summarize code changes can save you time and produce more consistent results. By using the techniques described here, you can:

• Create better commit messages and PR descriptions
• Maintain more consistent change logs and release notes
• Save mental energy for more important tasks
• Communicate changes more clearly and consistently with your team

Make sure to experiment, especially with the prompt, to find what works best for you. You can also automate the process to save even more time. With a little bit of upfront work, you can make summarizing code changes both efficient and effective.