Updating Homebrew formulae when your software gets a new version

• Basic steps
• When things go wrong
• When things (finally) go right
• Submitting your changes
• Closing notes

To get bioinformatics software on my Mac I rely on the Homebrew package manager. Unfortunately, the Brewsci/bio tap, where most biology software lives, doesn’t always update right after a new version is released. While you could wait for someone to update it for you, it’s faster and more fun to do it yourself, and you also get to learn about contributing to Homebrew!

I’ve written a brief tutorial / account that goes over my experience updating the muscle aligner from version 3.8.31 to 3.8.1551. It also covers some things I commonly experience when updating Homebrew formulae, and assumes some basic familiarity with the macOS Terminal.

NOTE: This is the full tutorial which you may find useful, but for uncomplicated revisions (“Basic Steps” only) you can use:

brew install hub && brew bump-formula-pr muscle

with --url=... and --sha256=... or --tag=... and --revision=... arguments to brew bump-formula-pr.

Basic steps

1. If you don’t have one already, sign up for a GitHub account! All Homebrew development is done over GitHub.

2. Install hub via brew install hub in the Terminal.

3. cd $(brew --repo brewsci/bio)

4. Fork the Brewsci/bio repository with hub fork. If this is your first time using hub you will be prompted for your GitHub username and password.

5. Create a branch to work on the updated formula with git checkout -b muscle-3.8.1551.

6. Edit muscle.rb in your preferred editor. brew edit muscle will default to vim. If you have Sublime Text or TextMate installed it might open in those editors. You can change this to e.g., nano with VISUAL=nano brew edit muscle.

7. In muscle.rb, set the url to the URL of the latest release. In the case of muscle I also had to update the version field.

8. Back in the Terminal (outside of your editor), run brew fetch muscle to get the latest release. brew will complain about a hash mismatch as the formula still contains the old hash. Don’t worry about this – just copy the provided hash into the sha256 field of muscle.rb in your editor.

9. The formula has a revision field, so remove that line as it’s a new version.

Now run brew install -vsd --git muscle. Hopefully the new release compiles!

The v turns on verbose mode so you can see what’s happening, s says to build from source instead of using binary bottles, d enables debugging so you can enter the build directory in case anything goes wrong, and --git turns the build directory into a git repository so you can make changes and put them in to the formula as patches to fix minor build problems.

When things go wrong

It didn’t build! Homebrew errors out with a message No such file or directory - src/globalsosx.cpp. If we drop into a shell (thanks to the handy debug option) we can see pretty readily that the src/ directory no longer exists in this updated version of muscle. Exit out of the debugging shell by pressing Ctrl-D twice.

Looking at the muscle formula, we can see that the the error is caused by a patch that fixes build failures on newer versions of macOS:

  def install
    # This patch makes 3.8.31 build on OSX >= Lion.
    # It has been reported upstream but not fixed yet.
    inreplace "src/globalsosx.cpp",
              "#include <mach/task_info.h>",
              "#include <mach/vm_statistics.h>\n#include <mach/task_info.h>"

    # This patch makes 3.8.31 build on RHEL 7.x
    # It ONLY affects Linux (in an "if Linux" clause in the 'mk' script)
    # It is unnecessary to create a static binary
    inreplace "src/mk",
              "LINK_OPTS=-static",
              "LINK_OPTS="

    cd "src" do
      system "make"
      bin.install "muscle"
    end
  end

Usually when adding patches, you would also report the bug and submit any relevant patches to the upstream software developers. Here we can see that the bug was reported, so let’s see if it’s been fixed by removing all the patches. We should also change the cd "src" do line as the src/ directory no longer exists.

  def install
    system "make"
    bin.install "muscle"
  end

Rerun brew install -vsd --git muscle to see if this fixes things. Looks like there’s still a build error:

ld: library not found for -lcrt0.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Build errors can be exotic and difficult to debug, but we’re in luck here: if you examine the Makefile provided by muscle (by dropping into the debug shell and running cat Makefile) you can see that the developer of muscle added a handy tip for macOS users:

# On OSX, using -static gives the error "ld: can't locate file for: -lcrt0.o",
# this is fixed by deleting "-static" from the LDLIBS line.

Let’s include this patch in our new version of muscle. Homebrew provides a function inreplace that lets you make simple changes to files. For more complicated patches, it’s better to run brew install --interactive --git muscle and follow the instructions to include a full diff file into the formula.

Remove all instances of -static and try building it again with brew install -vsd --git muscle.

  def install
    # Fix build per Makefile instructions
    inreplace "Makefile", "-static", ""

    system "make"
    bin.install "muscle"
  end

When things (finally) go right

Success! Test that everything went well with brew test -v muscle and check that the style and syntax of the formula are correct with brew audit --strict --online muscle. test works great, but audit complained and said:

brewsci/bio/muscle:
  * Stable: version 3.8.1551 is redundant with version scanned from URL

Remove the version line and rerun audit to ensure that there are no further issues.

Submitting your changes

Commit the changes with git commit muscle.rb -m "muscle 3.8.1551" and push them up to your fork with git push -u <USERNAME>, replacing <USERNAME> with your own GitHub username. Then run hub pull-request and fill out the details of the pull request template and a short message explaining your changes (if relevant).

You’ll now need to wait while the build infrastructure compiles the changes to your proposed formula. A maintainer will eventually come along and either merge your pull request or ask you to make some changes. Once your new formula is merged, all other users of Homebrew get to enjoy your work! Pat yourself on the back for helping others out and contributing to a important piece of open-source software.

Closing notes

There’s a lot of edge cases to building software, which is why package managers exist so regular folk don’t have to deal with these problems. I’ve spent too much time diagnosing arcane linker errors; I contribute to Homebrew so no one else has to needlessly suffer.

If you regularly use scientific software that isn’t already in Homebrew or Brewsci, please consider contributing a formula, or, if your software is too unstable or specialized, starting a Homebrew tap for your own tools. I maintain my own tap that I add to whenever I come across bio software that doesn’t already exist in Homebrew-science. Consult the Formula Cookbook for tips on crafting Homebrew formulae.