Scientific Computing

7-zip LZMA file compression

7-zip is a popular LZMA cross-platform compression algorithm. Install 7zip by package manager. For example, on Windows:

winget install 7zip.7zip

Examples:

  • Compress: 7z a big.7z big.dat
  • Extract: 7z e big.7z
-mmt=on
enable multithreading (far faster compression)
-m0=lzma2
enable LZMA2 compression (more advanced)
-mx=3
like “fast” compression.

Example 7-zip for very large files (100s of gigabytes or terabyte-size files)

7z a -t7z -mx=3 -mmt=on -m0=lzma2  mycomp.7z *.txt

Higher compression settings might not add a lot of compression but can take vastly longer. LZIP advantages vs. 7-zip.

Android 5G SA no data

5G wireless network providers across the United States and elsewhere are turning on 5G standalone mode (SA), that can enable significantly more data bandwidth and lower connection latency. Some phones, even recent phones, are not properly capable of supporting 5G SA even though they work fine in 5G NSA (non-standalone mode) that operates like 4G LTE. Recent Android releases and carrier updates have disabled 5G SA for certain phones that previously had 5G SA enabled due to data connectivity problems when the carrier enabled 5G SA. A symptom of lost data connectivity, possibly due to 5G SA incompatibility, is the phone signal showing 5G signal bars with an exclamation point.

What may possibly but not always help is to turn off 5G in the phone network settings by setting “Preferred network type” to “LTE” or similar instead of “5G”. This may force the phone to use 4G LTE instead of 5G, which may work better for data connectivity. However this does not always help. If a newer phone on the same network carrier works fine with 5G SA, then the phone with data connectivity problems may need a software update or carrier update to fix the problem, or simply a newer phone may be needed. The Network Cell Info app can show the network type and signal strength, and may help diagnose the problem.

CMAKE_TLS_VERIFY global

TLS verification default is ON since CMake 3.31. Users can override this default for all projects with environment variable CMAKE_TLS_VERIFY. or per-project with CMake variable CMAKE_TLS_VERIFY. The default TLS version may be set by CMAKE_TLS_VERSION. If the system TLS certificate location needs to be specified, this can be done by CMAKE_TLS_CAINFO.

Meson build system uses TLS verification by default, warning if verification fails. TLS verification is part of CMake’s internal nightly testing.

The example uses badssl.com, that purposefully has a variety of certificate problem URLs.


Reference: Issues that would have been caught with this default

Find executable path in Python

The full path to executables on the system Path (and cwd on Windows) are discovered by Python shutil.which. On Windows, environment variable PATHEXT is used to search filename suffixes if not specified at the input to shutil.which().

Shell aliases are not found by shutil.which() since the shell is not invoked. Instead append the directory of the desired executable to environment variable PATH, or specify it in shutil.which(..., path="/path/to/exe").

import shutil

# None if executable not found
exe = shutil.which('ls')

Since shutil.which() returns None for non-found executable it is convenient for pytest.mark.skipif

For programs not on PATH where the executable path is known:

shutil.which('myexe', path="/path/to/myexe")

CMake find_program vs. CMAKE_ROLE

CMake find_program does not generally consider NAMES parameter to have file suffixes unless manually specified. For Windows, .com and .exe file suffixes are considered, with search order:

  1. .com
  2. .exe
  3. no suffix

If on Windows and an executable “hello.exe” or “hello.com” exists, then CMake will find it. CMake would NOT find “hello.exe” on non-Windows platforms, where no file suffix is expected.

The full path to executables on the system Path (and cwd on Windows) are found by find_program(). Shell aliases are not found since the shell is not invoked. Instead specify find_program(... HINTS /path/to/exe).

NOTE: The CMake variable CMAKE_EXECUTABLE_SUFFIX ONLY affects find_program() in CMake role PROJECT

find_program(v NAMES hello)

Shell scripts of any file suffix on any operating system are found iff:

  1. (non-Windows) script file executable flag is set (find_program general requirement)
  2. script file suffix is specified as part of find_program(… NAMES) parameter

A complete standalone example:

Detect CMake generator from CMakeCache.txt

The CMAKE_GENERATOR cache variable records the CMake generator used to configure the CMake project in the CMakeCache.txt CMake cache file. Surprisingly, the cmake -LA option does not emit the CMAKE_GENERATOR value.

Thus, parsing CMakeCache.txt will give the previously used CMake generator. This is relevant in automated processes such as CI/CD systems that may build for numerous configurations and generators. This parsing can be trivially done in scripts in many coding languages. Here we give an example in CMake script “detect_gen.cmake”:

cmake_minimum_required(VERSION 3.21)

file(REAL_PATH ${bindir} bindir EXPAND_TILDE)

file(READ "${bindir}/CMakeCache.txt" _cache)

if(_cache MATCHES "CMAKE_GENERATOR:INTERNAL=([^ \n]+)")
  set(CMAKE_GENERATOR ${CMAKE_MATCH_1})
  message(STATUS "Detected CMake generator: ${CMAKE_GENERATOR}")
else()
  message(FATAL_ERROR "Failed to detect CMake generator")
endif()

This script is used like:

cmake -Dbindir=/path/to/build -P detect_gen.cmake

CMake rebuild cache

CMake directory property CMAKE_CONFIGURE_DEPENDS can be used to specify additional dependencies for the configuration step. For example, if a JSON file is used to specify source files, CMake wouldn’t detect if a source file was added, removed, or modified without CMAKE_CONFIGURE_DEPENDS.

Sometimes, the situation is too complicated to specify all dependencies manually. If a change is made that requires CMake to rebuild the cache, two equivalent ways to do this without modifying previously set options are:

cmake -Bbuild
# preserves prior options

or

cmake --build build -t rebuild_cache

Iterate Matlab versions with CMake

These techniques work with any versioned program or library. Here we use Matlab as an example. CMake find_package with a version range would be used to simply select from a known-working version range.

Many Matlab codes require a modern version of Matlab. It’s possible to select from an arbitrary min…max range of Matlab versions with CMake FindMatlab as follows. This technique works with other versioned programs and libraries as well.

foreach(v IN ITEMS 23.2 24.1 24.2)
  find_package(Matlab ${v} EXACT COMPONENTS MAIN_PROGRAM)
  if(Matlab_FOUND)
    add_test(NAME matlab-${v}
      COMMAND ${Matlab_MAIN_PROGRAM} -batch "buildtool"
      WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
      )
  endif()
endforeach()

Git SSL certificate location

Git typically works for HTTPS URLs using the default configuration. If certificates are not working, try configuring Git with the system SSL certificate location.

git config --global http.sslCAInfo /path/to/cert.crt

Git environment variable “GIT_SSL_CAINFO” can override this. This can be used to override missing or incorrect SSL certificate locations for Git operations in programs like Matlab.

To disable SSL certificate verification, set environment variable “GIT_SSL_NO_VERIFY” to “1”, but be aware of the security implications e.g. MITM.

CMake macOS Xcode Environment

Related: GCC / Clang header clash on macOS


When macOS, Xcode, or Command Line Tools upgrades, build directories (including for CMake-based projects) often need to be refreshed. If the user has set custom environment variables concerning Xcode, they may need to be updated as well. Check that Xcode has completed the upgrade by:

xcodebuild -checkFirstLaunchStatus

xcodebuild -runFirstLaunch

For a simple CMake project on macOS, CMakeCache.txt might include:

CMAKE_OSX_DEPLOYMENT_TARGET:STRING=

CMAKE_OSX_SYSROOT:PATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX*.sdk

Having multiple directories under the following is fine:

  • /Library/Developer/CommandLineTools/SDKs
  • /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs

CMake can select the appropriate SDK by the user specifying environment variable SDKROOT or by selecting the SDK:

xcode-select --switch

The XCode versions that Homebrew targets can be found in xcode.rb.

Print the default Xcode version:

pkgutil --pkg-info com.apple.pkg.Xcode

Print the Xcode Command Line Tools version with:

pkgutil --pkg-info com.apple.pkg.CLTools_Executables

Workarounds

If Homebrew GCC breaks after upgrading Xcode or Command Line Tools, try specifying an older SDK. For example, if g++-15 main.cpp -v shows a different (older) SDK than CMake and it works, try specifying that SDK in environment variable SDKROOT. Note that the SDK version corresponds to macOS version, not the XCode version.

export CC=gcc-15 CXX=g++-15 FC=gfortran-15

export SDKROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX16.sdk/

and then source ~/gcc.sh before running cmake with a fresh build directory.

If a CMake build step fails, try copy-pasting the command and removing the -isysroot portion of the command. This is a clear clue the older SDK is (at least temporarily) needed till Homebrew updates its GCC formula.

GCC will tell where included files are coming from by adding the gcc -H flag. This tells what to specify for environment variable SDKROOT.

Example minimal projects: