Continuous integration using GitHub Actions for Homebrew on Linux
Homebrew is arguably the most popular package manager for macOS, but it hosts a small and growing userbase on Linux as well! This blog post will explain how the Homebrew maintainers build binary packages (bottles) on Linux.
How binaries used to be built on Linux
Taps, in Homebrew parlance, are special directories in your Homebrew installation, which contain formulae that describe how to build and install software. In other package managers, taps are variously called repositories, channels, streams, and so on. The default tap on macOS is called Homebrew/homebrew-core, while the default tap on Linux is a fork of this called Homebrew/linuxbrew-core. For brevity, I’ll be calling these Homebrew/core and Linuxbrew/core. Bottles are binary packages that we ship to users. This saves a lot of time, as otherwise users would need to build everything from source, an approach that is unsustainable for both users and maintainers.
The above diagram shows how we used to build formulae for Linux. As Linuxbrew/core is a fork of Homebrew/core, maintainers would wait for changes to happen upstream in Homebrew/core, then use a special command brew merge-homebrew to update around 5–10 formulae. This merging process introduces merge conflicts, particularly in the bottle do block that signals the availability of binary packages:
<<<<<<< HEAD
sha256 "bd66be269cbfe387920651c5f4f4bc01e0793034d08b5975f35f7fdfdb6c61a7" => :sierra
sha256 "7071cb98f72c73adb30afbe049beaf947fabfeb55e9f03e0db594c568d77d69d" => :el_capitan
sha256 "c7c0fe2464771bdcfd626fcbda9f55cb003ac1de060c51459366907edd912683" => :yosemite
sha256 "95d4c82d38262a4bc7ef4f0a10ce2ecf90e137b67df15f8bf8df76e962e218b6" => :x86_64_linux
=======
sha256 "ee6db42174fdc572d743e0142818b542291ca2e6ea3c20ff6a47686589cdc274" => :sierra
sha256 "e079a92a6156e2c87c59a59887d0ae0b6450d6f3a9c1fe14838b6bc657faefaa" => :el_capitan
sha256 "c334f91d5809d2be3982f511a3dfe9a887ef911b88b25f870558d5c7e18a15ad" => :yosemite
>>>>>>> homebrew/master
In this diff, Linuxbrew/core‘s version (above the =======) has an :x86_64_linux tag indicating the availability of binaries for Linux, but because upstream’s repository homebrew updated the sha256 hash of its macOS bottles, Git can’t figure out how to merge these changes. These merge conflicts have to be resolved manually by maintainers; in this case, the conflict is resolved in favor of Homebrew/core’s version of the bottle block.
Once the maintainer fixes the merge conflicts, brew merge-homebrew will automatically open a pull request against Linuxbrew/core. To close the pull request and update Linuxbrew/core with these new formulae, the maintainer runs brew pull --clean and git push.
You’ll notice that in fixing the above merge conflict, the Linux bottles are no longer described, meaning that users who brew install that formula will have to build from source rather than downloading binary packages. To actually build bottles for Linux, a maintainer needs to run brew find-formule-to-bottle to identify which formulae need new binaries. This is piped into brew build-bottle-pr which will open individual pull requests for each formula requesting a bottle build. Here’s an example of a small core merge of a single formula and its corresponding bottle pull request:
diff --git a/Formula/mpv.rb b/Formula/mpv.rb
index 5d2a63fe4e8..f7f6ee454be 100644
--- a/Formula/mpv.rb
+++ b/Formula/mpv.rb
@@ -1,3 +1,4 @@
+# mpv: Build a bottle for Linux
class Mpv < Formula
desc "Media player based on MPlayer and mplayer2"
homepage "https://mpv.io"
If you look at the diff, you’ll see that it only adds a comment, so it doesn’t seem like the change in this pull request would actually do anything. But this actually signals to brew test-bot that it should build a new bottle for Linux. Behind the scenes, once the bottle is successfully built, the Azure Pipelines release job uploads the bottles to Bintray in an “unpublished” state. The release job also pushes a commit containing bottle data (its SHA-256 hash) to a fork of the Linuxbrew/core repository, and tags it with a pull request number (e.g., pr-1234).
diff --git a/Formula/mpv.rb b/Formula/mpv.rb
index 5d2a63fe4e8..f73774e9328 100644
--- a/Formula/mpv.rb
+++ b/Formula/mpv.rb
@@ -8,5 +8,6 @@ class Mpv < Formula
bottle do
sha256 "dd0fe84dea1268524e18d210595e31b295906e334ae8114124b94a94d130de60" => :catalina
sha256 "22c3aa2fb8ec77b5125c836badf0ad7889b512280f54f310c5a6ab8e77099fa6" => :mojave
sha256 "0477b20f9a166d746d84c2a7d0b191159c6825512fe66c38ddf9ca6c43403d97" => :high_sierra
+ sha256 "41a811990283f63ce8d2132715ee2ada8b15fd29b11df5427d5ae0b40e947816" => :x86_64_linux
end
After all this happens, a maintainer will then run brew pull --bottle, which cherry-picks the bottle commit to the maintainer’s local copy of Linuxbrew/core, and also publishes the bottles on Bintray. However, prior to pushing the new bottle descriptors to GitHub, the bottling request comment # foo: Build a bottle for Linux needs to be removed, to avoid cluttering Linuxbrew/core with unnecessary comments. Maintainers use a custom command brew squash-bottle-pr to remove these comments when necessary.
Finally, after confirming that everything is correct, the maintainer then pushes the bottling commits to GitHub. Linux users can now update and download the new bottles with brew update and brew install. Success! 🎉
Sidebar
This process, with all its horrors, was actually a significant improvement. Homebrew on Linux previously used CircleCI for its automation, which required an AWS Lambda worker to transfer bottles from Circle to Bintray. After the migration to Azure Pipelines, this Rube Goldberg machine was decommissioned.
How binaries are now built on Linux
You might have noticed how frustratingly manual this process is. In my diagrams, the solid lines indicate manual actions that maintainers need to do, while the dashed lines automatic steps executed by continuous integration. There is a distressing amount of manual intervention needed for Homebrew maintainers on Linux, which can lead to frustration and burnout from the high workload.
Jonathan trying to update and fix Homebrew’s GitHub Actions at the Homebrew annual meeting. “Just one more commit and it’ll work…”
The Homebrew on Linux maintainers have now migrated to a much more sustainable system using GitHub Actions. Thanks to funding from Homebrew’s supporters on GitHub Sponsors and Patreon, a number of maintainers were able to fly to Brussels for Homebrew’s annual meeting and hack on continuous integration. This focused, in-person work paid off huge dividends, as Homebrew on Linux now has a much more manageable and automated bottling infrastructure, as can be seen in this figure:
You’ll notice that much of the previously manual work is now completely automated. After merges from Homebrew/core, the GitHub Actions runners will automatically attempt to build bottles, without a maintainer needing to open individual pull requests to request build jobs. If the automated tests don’t find any problems when building the updated formulae, the maintainer’s work ends there — new binaries are automatically uploaded and published without further intervention.
If one or more bottles fail to build, GitHub Actions notifies the maintainer by posting an issue comment. The maintainer can then open a pull request to fix the build failure, and, once tests pass on that pull request, merge in the changes. GitHub Actions will then automatically publish the binary bottles for the fixed formula.
Conclusion
What, in practice, do these changes mean for maintainers, contributors, and users?
For maintainers, there is far less work needed to get new maintainers up to speed. The only unique aspect of maintaining Linuxbrew/core is now the brew merge-homebrew workflow, and it is a single command with relatively few sharp edges. Everything else just involves fixing Homebrew formulae to build on Linux, which is a relatively straightforward task with our Docker container. We hope to automate merges from Homebrew/core in the short term, which would remove yet another source of manual intervention for Linux maintainers, and in the long-term, to merge Linuxbrew/core into Homebrew/core.
For contributors, the contribution experience is much more straightforward. It’s now very obvious when builds have failed, and the new merge-based workflow means that contributors see their pull requests as “merged”, which I believe is both psychologically rewarding (to “see the purple”) and reduces confusion compared to the old rebase workflow, which marked pull requests as “closed” in red.
For users, the most obvious benefit is that updates from Homebrew/core now happen much more rapidly due to lower maintainer burdens. In addition, we’re now able to bottle many more formulae. The increased number of bottles means more reliable software for users and fewer support requests that maintainers need to handle.
Top panel: Number of bottled formulae (solid line) and total formulae (dashed line) over time. Bottom panel: Percent of total formulae that are bottled on Linux. Download the Linux bottling statistics.
After most of the bugs were ironed out of the new GitHub Actions process, the Linux maintainers attempted to build binary bottles for every single formula, from A to Z. You can clearly see the huge jump in the number of formulae with binaries available in the early part of 2020, after we had migrated to our new continuous integration infrastructure.
Our goal in Brussels was to ensure that a maintainer could merge a pull request with passing tests from their phone’s web browser, and have everything else done behind the scenes by GitHub Actions. I’m proud to say that we achieved that, and we’ll hopefully be able to build on that expertise to modernize the continuous integration infrastructure for Homebrew on macOS.
This work was only made possible thanks to travel funds provided by Homebrew’s sponsors. I’d also like to thank the Homebrew maintainers, especially Issy Long, Michka Popoff, and Sean Molenaar, for feedback on this post prior to publication, and the other Linux maintainers, especially Dawid Dziurla, for checking my work and proposing fixes to my terminally buggy code. Shaun Jackman prototyped the initial workflow in Brewsci, which inspired the solution we settled on.
In my next post, I’ll talk about the work we’ve done to migrate to GitHub Actions on macOS. While you wait, read about how binaries are currently built on macOS, check out the history of Homebrew’s macOS infrastructure, or read about how we use Orka for macOS hosting.