Scientific Computing

Visual Studio supports cross-builds on Windows host for Linux targets. This requires either a remote Linux machine connection, or using WSL on the local computer.

A more robust solution without additional setup on developer computers is CI/CD such as GitHub Actions or many other online and offline choices such as Jenkins. When the developer Git pushes, the CD job provides binaries across operating systems.

An example of GitHub Actions CD is the Ninja project. They provide old (CentOS 7) Linux binaries, macOS and Windows. This could easily be extended to ARM etc.

To exclude multiple directories from Black Python code formatter, use the following format in pyproject.toml. The multi-line regex format seems to be required–any other way didn’t take effect.

Edit / add / remove as many directories as desired, using the following multi-line format (indentation is not important). Note the escaping needed for “.” since this is a regex.

This is particularly useful when using Black in a project with Git submodules to not disturb the Git submodule Python code with Black from the top-level project. Likewise for other tools such as flake8 and mypy set exclude in their settings for Git submodules.

[tool.black]
force-exclude = '''
/(
\.git
| \.mypy_cache
| \.venv
| _build
| build
| dist
)/
'''

CMake ExternalProject and FetchContent can download from Git or URL archive. Archive download is usually much faster, especially for projects with a large number of Git commits. Checksum of the archive can optionally be verified with URL_HASH option.

Example

Git submodule

At first glance since Git config can set fetchParallel Git clone submodule in parallel might be something the ExternalProject GIT_CONFIG could do, but we have not tried this.

Recent CMake changelog

CMake 3.26 adds CMake environment variable CTEST_NO_TESTS_ACTION that is good for CI to avoid missing unwanted no test detection. Helpful for debugging is the elimination of CMakeOutput.log and CMakeError.log, replaced by CMakeConfigureLog.yaml. message(CONFIGURE_LOG) allows easy logging to CMakeConfigureLog.yaml.

CMake 3.25 adds workflow presets, making configure-build-test a single command.

CMake 3.24 adds the long-awaited cmake -Bbuild --fresh option to clean the build directory first. The LINK_GROUP generator expression is excellent for resolving complex link dependencies. The CMAKE_COMPILE_WARNING_AS_ERROR boolean option sets most compilers to error if a compile warning occurs, which is generally a good setting for CI systems. CMAKE_COLOR_DIAGNOSTICS environment variable is useful to colorize build system and compiler output.

CMake 3.23 further enhances –debug-find to allow debugging all find_* command, or specific find_package or find_* variables. Header files can be more cleanly installed/affiliated with targets, particularly relevant for packaging and installing with target_sources file sets.

CMake 3.22 adds several CMake Environment Variables that are generally useful. CMAKE_BUILD_TYPE default for single configuration build systems. CMAKE_CONFIGURATION_TYPES defaults available configurations for multi-config build systems like Ninja Multi-Config. CMAKE_INSTALL_MODE makes symlinks with copy fallback a good choice for installing programs from CMake. For CTest, the new ENVIRONMENT_MODIFICATION test property makes modifying environment variables for test(s) much easier.

CMake 3.21 adds more preset features, including making “generator” optional–the default CMake behavior will be used to determine generator. cmake –install-prefix can be used instead of cmake -DCMAKE_INSTALL_PREFIX=. PROJECT_IS_TOP_LEVEL and <PROJECT-NAME>_IS_TOP_LEVEL identify if a project is at the top of the project hierarchy. ctest --output-junit gives test output in standard tooling format.

CMake 3.20 adds support for Intel LLVM compiler and NVIDIA HPC compiler. ExternalProject_Add() learned CONFIGURE_HANDLED_BY_BUILD which avoids CMake commanding a reconfigure on each build. try_compile(... WORKING_DIRECTORY) parameter was added. CMake presets in CMakePresets.json now covers configure, build and test, allowing many parameters to be declared with inheritance in JSON. CMake presets are a key feature for CI, as well as user configurations. ctest --test-dir build option avoids the need to manually cd build. cmake_path allows path manipulation and introspection without actually touching the filesystem.

CMake 3.19 added support for ISPC language. string(JSON GET|SET) parsing is very useful to avoid hard-coding parameters. FindPython/find_package accepts version ranges. Intel oneAPI works with CMake ≥ 3.19.6. Emits warning for cmake_minimum_required(VERSION) < 2.8.12. CMakePresets.json enables configure parameter declarations in JSON.

CMake 3.18 adds CMake profiler Adds REQUIRED parameter to find_*(). Adds file(ARCHIVE_CREATE) and file(ARCHIVE_EXTRACT), which is much more convenient than execute_process(COMMAND ${CMAKE_COMMAND} -E tar ${archive} WORKING_DIRECTORY ${out_dir}) syntax

CMake 3.17 adds Ninja Multi-Config generator. cmake –debug-find shows what find_*() is doing. Eliminates Windows “sh.exe is on PATH” error. Recognizes that Ninja works with Fortran.

CMake 3.16 adds precompiled headers, unity builds, many advanced project features.

CMake 3.15 adds CMAKE_GENERATOR environment variable that works like global -G option. Enhances Python interpreter finding. Adds cmake --install command instead of “cmake –build build –target install”. Added Zstd compression.

CMake 3.14 adds check_fortran_source_runs(). FetchContent was enhanced with simpler syntax. The transitive link resolution was considerably enhanced in CMake 3.14. Projects just work in CMake ≥ 3.14 that fail at link-time with CMake < 3.14.

We don’t recommend use of the older CMake versions below as they take significantly more effort to support.

CMake 3.13 adds ctest --progress and better Matlab compiler support. Lots of new linking options are added, fixes to Fortran submodule bugs. The very convenient cmake -B build incantation, target_sources() with absolute path are also added. It’s significantly more difficult to use CMake older than 3.13 with medium to large projects.

CMake 3.12 adds transitive library specification (out of same directory) and full Fortran Submodule support. get_property(_test_names DIRECTORY . TESTS) retrieves test names in current directory.

CMake 3.11 allows specify targets initially w/o sources. FetchContent is added, allowing fast hierarchies of CMake and non-CMake projects.

Computing file hash checksum allows some verification of file integrity, assuming the file is not maliciously altered to have the same checksum. MD5 and SHA-256 are among the methods available in Matlab-stdlib (works on Matlab or GNU Octave) file_checksum that computes the checksum of a file. The file is read in chunks to avoid overflowing the Java VM memory.

To give secure access to private GitHub repositories on less-trusted systems like CI or HPC or shared workstation, consider GitHub Oauth tokens. The Oauth token can give read-only (or other fine-grained permissions) to all or a specific subset of repositories the GitHub account has access to.

Create a GitHub Oauth token with the desired permissions.

For read-only private GitHub repo access the “repo” permission group is selected.

Copy the text string token and SSH into the remote system where access is desired. Configure the global user Git config to use the Oauth token for the desired GitHub organization or user.

Suppose a coworker “sara” has a private GitHub repo “myrepo” and has added your GitHub username as a collaborator in the “myrepo” settings. On the remote computer, configure Git to use the Oauth token for the “sara” GitHub user:

git config --global url.https://oauth2:OauthToken@github.com/sara/.insteadOf https://github.com/sara/

A similar syntax is used for GitHub organizations or specific repositories.

The text OauthToken is replaced with the actual Oauth token string from GitHub.

Related: Git pull HTTPS push SSH

Coccinelle allows specifying desired matches and transformations in C code. It is a powerful tool for making large-scale changes to programs. Alternatives include regex matching with “sed”, “awk”, IDEs or AI-assisted tools.

Consider Coccinelle examples and AI-based tools before deciding which tool to use. In any case, manual checking will be necessary to ensure that the changes are correct.

Many Git fixup commits are for forgetting to check project code standards like:

• PEP8 compliance
• type annotation checks
• clang-format

Mitigate git commit fixup clutter by making a Git pre-commit hook that applies to all repositories. The pre-commit hook also does a simple check for YaML syntax. This procedure works across operating systems since Python script is used.

Tell Git the directory for Git hooks:

git config --global core.hooksPath ~/.git/hooks

Create executable file ~/.git/hooks/pre-commit from this Python pre-commit script.

This global pre-commit check hook does several things:

1. get a list of Python file names that changed.
2. check PEP8, mypy type hinting and that Python breakpoint debug statements are not present.
3. check YaML syntax
4. clears IPython notebook output cells
5. Checks for trailing whitespaces in any code language

One can easily extend the concept of lint-checking for other programming languages.

These checks can be bypassed at any time for a commit by:

git commit -n

Override this global pre-commit check, substituting a per-repo .git/hooks/pre-commit by in that repo directory typing:

git config core.hooksPath .git/hooks

For example, a website made of Markdown files may wish to run a local-link check via Linkchecker-Markdown.

Troubleshooting

The Python pre-commit script uses the typical Python shebang:

#!/usr/bin/env python3

If on Windows failure occurs like:

cannot spawn .git/hooks/pre-commit: exec format error

Try changing the shebang to:

#!/usr/bin/env python

especially if the command “python3” is not found when typing it in Terminal.

References:

• Git dry run
• Git precommit hooks docs

For macOS developers using Homebrew (possibly other package managers too) there is a TLS v1.3 bug where system cURL was used by build servers and cURL had a bug. One package impacted is CMake, though it’s not a CMake bug as Kitware CMake binaries using newer cURL are fine.

The bug causes some TLS v1.3 download/upload internet operations to fail with code 35. This was due to a bug in macOS system cURL. macOS has already updated cURL for end users. Updating Homebrew CMake fixes this issue.

If stuck, one can set an environment variable per command to disable TLS v1.3 as a temporary workaround:

CURL_SSL_BACKEND=SecureTransport cmake ...

However, it is better to upgrade Homebrew CMake to a fixed cURL version:

brew upgrade cmake

This is not a CMake bug, as the CMake binaries downloaded from Kitware work since they are built with a more recent non-macOS-system cURL. This is a Homebrew packaging bug that can impact other programs distributed by Homebrew (or Macports) due to the cloud build servers using not-yet-updated macOS for the new cURL.

Diagnose: the version of cURL compiled into CMake may be obtained by UserAgent.cmake:

cmake -P UserAgent.cmake

If the cURL version is less than 8.4.0 this macOS SSL 35 bug may exist with TLS v1.3 connections.

To help debug CMake SSL operations, I use BadSSL and TLS check.

The Cisco VPN Secure Client is a small app that runs on the user device to establish and monitor a VPN connection. It requires background services to be allowed to run at login/startup to function. Otherwise, the connection attempt will fail with a message like:

VPN connect capability is unavailable because the VPN service is unavailable

For example, on macOS under settings → Login Items, there may be about three Cisco items that need to be enabled. After enabling, reboot the computer and try the VPN connection again.