GNU Make parallel build

GNU Make is an important legacy build system. Ninja build system is recommended in general since it’s faster, has more features and generally operates more correctly. Ninja works with CMake and Meson among others.

In general, build systems are useful to:

  • build programs much faster (10’s or 100’s of times faster) by:
    • only recompile parts of the code that changed
    • compile / link in parallel
  • avoid copy-paste mistakes with excessively long compile commands or piles of Bash scripts

GNU Make parallel builds: by default, GNU Make only uses one thread. For most programs, speeding up compilation is desirable and accomplished via the -j or --jobs option:

make -j

The -j option alone tries to compile all targets in parallel at once. This is typically not desirable or efficient due to excessive context switching. On some systems such as Windows Subsystem for Linux, the bare make -j -l2 may overwhelm the computer because WSL1 doesn’t seem to correctly report the load factor. Thus to do parallel builds with GNU Make on WSL1, manually set the number of parallel build threads perhaps equal to or less than the number of physical cores in your PC. For example an 8 physical core laptop would compile with make -j8.

Caveats: in certain systems like older Raspberry Pi computers, the CPU may go into thermal cutback or suffer undervoltage spikes if using an inadequate power adapter. While the better solution is to provide adequate DC power and CPU cooling, one may choose to compromise by manually specifying the number of threads. For example, with the Raspberry Pi we often use 1 or 2 threads.

make -j1

# or

make -j2

RAM limitation: make -j does not consider system RAM, so your computer may become overwhelmed. Try make -j2 or so until the system memory isn’t over-consumed.

Keeping system responsive: a possible downside of building large programs with make -j is that the computer can become non-responsive to input including mouse and keyboard. A symptom of this is that the Caps Lock keyboard light is very slow to respond. To mitigate the problem automatically, disallow a new thread to start if the system becomes too heavily loaded. This is accomplished with the -l or --max-load option.

Linux system load of 0 to 1 means approximately that no tasks are waiting. So if you want to keep your computer responsible to user input (you can continue using your laptop while big program builds in the background), consider something like -l 1. You can set the -l factor higher to allow make to have increasingly higher priority over other tasks, including user input.

Instead of writing Makefiles, we use CMake or Meson on projects involving compiled code. CMake and Meson each have an integrated parallel-executing test suite, which works very well with continuous integration systems for automated self test. Continuous integration should be a part of all software projects of all sizes, from 50 line programs to operating systems. We recommend Azure Pipelines or GitHub Actions for CI in general.

GNU Make parallel docs

Related: CMake fundamentals