Yet another tool for updating dependencies

Cover — Go Mod Bump

In Brief

This article is about a small and useful script for updating dependencies called go-mod-bump. You may be interested in the issues, solutions, or the story behind the script. If you want to try the script yourself, it can be found in a public repository.

Disclaimer

  • The script has not been used in CI and may be unsuitable for such use.
  • The story is described retrospectively, after a significant amount of time, so it may contain clear inconsistencies or distortions.
  • The script modifies go.mod and go.sum files in the working git directory and changes the list of staged files; you may lose your changes if you use it carelessly.
  • This tool is not a competitor to popular dependency update automation solutions and is not intended to occupy their niche.

Problem

I work with many repositories both at work and in personal projects, and I regularly need to update dependencies. They are stored in different services for project hosting, in private and public repositories. Unfortunately, not all of these projects have automation set up for updating dependencies, and for some of them, there are no suitable ready-made solutions. Moreover, keeping track of hundreds of repositories is an overwhelming task for me, and I would rather not deal with it.

Limitations

  • Dependency updates are done with a single command.
  • Each module update is in a separate commit.
  • Commit messages resemble those generated by popular dependency update solutions.

Solution

Automate a set of routine operations so that I have minimal or no involvement. Essentially, several steps need to be automated:

  1. Check for new versions of modules.
  2. go get github.com/xorcare/pointer@v1.2.2 - update the module.
  3. go mod tidy - synchronize dependencies in go.mod and go.sum, while also checking for any issues.
  4. go build ./... - perform minimal checks, in this case ensuring that all source code compiles.
  5. git commit -m 'Bump github.com/xorcare/pointer from v1.2.1 to v1.2.2' - create a commit with a clear title.
  6. Repeat these steps for all modules that the maintained module depends on.

History of the Solution

Let’s delve a bit into the history. The task was to create a small helper script for updating dependencies.

Step One

Somehow find out that new versions of dependencies are available. Fortunately, Go already has a suitable command go get -u all, which handles updating everything effectively. From this command’s existence, we can draw two conclusions: Go can check for new package versions, and when using this command, we cannot manage the process at all.

By the time I created the script, I was already familiar with the command go list, so instead of go get -u all, I could dig into it further. After some exploration in go help list, I discovered two important flags:

  • -m, which allows getting a list of modules using go list -m all.
  • -u, which works similarly to go get -u, allowing us to check for new versions.

Thus, we now have a more manageable command: go list -u -m all, which outputs a list of modules along with their current version in the project and the latest version if it differs from the installed one.

Step Two

Having obtained the list of modules and versions, we need to iterate through it. This is not an issue in bash but has some nuances:

  • Each line looks like this: golang.org/x/net v0.22.0 [v0.29.0]; from it, we need to extract three values, and I’m not fond of parsing strings.
  • Only direct modules are needed.
  • Only those modules that have a new version are required.

In the previous step’s go help list, we can apply the parameter -f, which sets the output format. I used it to create something like CSV with the delimiter @.

go list -u -m -f '{{.Path}}@{{.Version}}@{{if .Update}}{{.Update.Version}}{{end}}{{if not .Update}}{{end}}{{if .Indirect}}{{end}}' all | grep -v ''

We obtain a list of modules that our module definitely imports and that require updating:

golang.org/x/net@v0.22.0@v0.29.0

This format can be easily parsed using cut.

Step Three

We now have a list of dependencies ready for updating. At this point, the script looks something like this:

#!/usr/bin/env bash
set -e
MODULES_FOR_UPDATE=$(go list -m -u -f "{{.Path}}@{{.Version}}@{{if .Update}}{{.Update.Version}}{{end}}{{if not .Update}}{{end}}" all | grep -v '')
for mdl in $MODULES_FOR_UPDATE; do
    module=$(echo "$mdl" | cut -f1 -d@)
    current_version=$(echo "$mdl" | cut -f2 -d@)
    latest_version=$(echo "$mdl" | cut -f3 -d@)
    go get "$module@${latest_version}"
    go mod tidy
    go build ./...
    git reset HEAD -- . >/dev/null
    git add go.mod go.sum >/dev/null
    git cm -a -m "${PREFIX}Bump ${module} from ${current_version} to ${latest_version}" >/dev/null
    echoerr "go-mod-bump: upgraded ${module} ${current_version} => [${latest_version}]"
done

At this stage, we have a working tool that can be used.

Step Four

In projects with many dependencies, the script runs very slowly because go list -u checks for updates across all modules regardless of whether they are direct or indirect.

To speed up the script’s operation, we can divide its work into two stages:

  1. Gather only direct modules; similar actions were performed in previous steps.
  2. Save this list in variable $DIRECT_MODULES and request updates only for direct modules:
go list -u -m -f '{{.Path}}@{{.Version}}@{{if .Update}}{{.Update.Version}}{{end}}{{if not .Update}}{{end}}' $DIRECT_MODULES | grep -v ''

This way, we will obtain a list of modules that need updating without iterating through all modules.

Step Five

During usage, it becomes clear that not all modules can be updated; some must be skipped if their updates lead to errors.

This can be implemented by combining commands responsible for checking and updating dependencies into one function:

function update_module() {
    go get "$1"
    go mod tidy
    go build ./...
}

Then in case of failure, simply skip committing by reverting changes in go.mod and go.sum:

if ! update_module "${module}@${latest_version}" >/dev/null 2>&1 >/dev/null; then
    git checkout -f HEAD -- go.mod go.sum
fi

We then proceed to update the next module from the list until it is exhausted.

At this stage, the core mechanics of the script already resemble the published version.

Final Step — But Not Least

Initially, I used the script only myself and there was no reason to embellish it; however later I shared it publicly after enhancing its appearance:

  • Removed hardcoded all allowing developers to choose their own module lists.
  • Added flags -h and -help with usage examples; also added flag -p to prepend a prefix to commit messages if you follow certain naming conventions.
  • For cases where module updates fail, added detailed output messages along with commands that can be copied for manual diagnostics.

Conclusion

The result is a convenient script called go-mod-bump that effectively addresses its intended task well. It is relatively cross-platform as it uses minimal commands and has been tested on macOS, Debian, and Ubuntu.

Perhaps it would have been better to create this tool in true Go fashion by writing it in Go itself; however at that time, I chose bash for simplicity and currently lack inspiration to rewrite it in Go.

Afterword

If you relate to the issues presented here, you might find using this script beneficial. If your git repository hosts provide ready-made tools for automating dependency updates, then use them as they will likely solve your problems better by running on schedule and creating nice merge requests on your behalf.

go  golang  go mod bump  go modules  dependency management  bash