Scientific Computing

GNU Data Language (GDL) GUI

GNU Data Language (GDL) can use GDLDE Workbench GUI for graphical development.

Download and install GDL.

The Windows installer includes GDLDE Workbench: gdlsetup-Windows-x86_64-standard.exe Simply install and look in Windows Start menu for “GDL Workbench”.

For macOS use gdl-macOS-x86_64-standard.dmg or build GDL from source.

For Ubuntu Linux:

apt install gnudatalanguage

Download and extract GDLDE for Linux.

Install JDK

Install JDK by:

  • Windows: winget install Microsoft.OpenJDK.21 (find exact version by winget search Microsoft.OpenJDK).
  • macOS: brew install openjdk
  • Linux: apt install default-jdk

Restart Terminal, ensure that echo $JAVA_HOME or PowerShell echo $env:JAVA_HOME is not empty.

Eliminate Python __pycache__ directories

Python can put all Python .pyc cache files under one system-wide directory by the PYTHONPYCACHEPREFIX environment variable.

  • Windows: set environment variable PYTHONPYCACHEPREFIX to %TEMP%
  • Linux: in ~/.profile add export PYTHONPYCACHEPREFIX=${TMPDIR}
  • macOS: in ~/.zshrc add export PYTHONPYCACHEPREFIX=${TMPDIR}

Pycache .pyc files can be set to NOT write anywhere with the PYTHONDONTWRITEBYTECODE environment variable.

  • Windows: set environment variable PYTHONDONTWRITEBYTECODE to 1
  • Linux: in ~/.profile add export PYTHONDONTWRITEBYTECODE=1
  • macOS: in ~/.zshrc add export PYTHONDONTWRITEBYTECODE=1

Python 3.12 Apple App Store conflict

LWN.net reports on changes to Python 3.13 urllib standard library. It was deduced that Apple rejected Python 3.12 apps due to a string in the Python stdlib that was rejected, regardless of code execution. There naturally was some very good discussion linked to in the LWN.net article that illustrates the conflict between closed commercial platforms with great financial might and open source software. The Python 3.13 patch has already been merged. A pull request backport patch for Python 3.12 has also been created, and illustrates the clean nature of the patch and new configure flag.

Github Actions dynamic job environment variables

GitHub Actions jobs can dynamically set environment variables with job scope using a run: step to write the variable to an environment file.

Append to PATH: All job steps after the “run:” stanzas have the new PATH value “~/.local/bin” appended. Windows defaults to PowerShell.

- name: set Unix PATH
  if: runner.os != 'Windows'
  run: echo "${HOME}/.local/bin" >> $GITHUB_PATH

- name: set Windows PATH
  if: runner.os == 'Windows'
  run: echo "${HOME}/.local/bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append

Any environment variable can be set in this way. Example: set environment variables “CMAKE_INSTALL_PREFIX” and “CMAKE_PREFIX_PATH” to “~/libs” for the following job steps:

- name: set Unix
  if: runner.os != 'Windows'
  run: |
    echo "CMAKE_INSTALL_PREFIX=~/libs" >> $GITHUB_ENV
    echo "CMAKE_PREFIX_PATH=~/libs" >> $GITHUB_ENV

- name: set Windows
  if: runner.os == 'Windows'
  run: |
    echo "CMAKE_INSTALL_PREFIX=$HOME/libs" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
    echo "CMAKE_PREFIX_PATH=$HOME/libs" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append

Static environment variables in GitHub Actions

GitHub Actions environment variables have distinct scopes:

  • Workflow
  • Job
  • Step

It’s trivial to set static environment variables in each of these scopes. Dynamically setting environment variables is also possible.

Workflow

Set static workflow environment variables in GitHub Actions by using env: at the top level of a “.github/workflows/ci.yml” file like:

name: ci

env:
  CTEST_PARALLEL_LEVEL: 0
  CMAKE_BUILD_PARALLEL_LEVEL: 4
  CTEST_NO_TESTS_ACTION: error
  CMAKE_GENERATOR: Ninja
  CC: gcc

Job

Static job environment variables are set like:

jobs:

  base:
    runs-on: macos-latest

    strategy:
      matrix:
        cc: [gcc-13, clang]

    env:
      CMAKE_GENERATOR: Ninja
      CC: ${{ matrix.cc }}

Step

Set static step environment variables like:


    - run: cmake -B build
      env:
        CMAKE_GENERATOR: Ninja

Install MSYS2 on Windows

MinGW brings GNU compiler tools to Windows since the late 1990s. MSYS2 provides numerous developer tools including MinGW on Windows using pacman package manager.

Install MSYS2 by:

winget install MSYS2.MSYS2

Start the MSYS2 UCRT console in the Windows Start menu. Update MSYS2 to get the latest packages in the MSYS2 terminal. Run this command multiple times until it says “nothing to do”.

pacman -Syuu

To use GCC and other MSYS2 / MinGW64 programs from PowerShell without disrupting other compiler use, create file “gcc.ps1” containing:

$r="$Env:SystemDrive/msys64/ucrt64"
$b="$r/bin"

$Env:CC="$b/gcc"
$Env:FC="$b/gfortran"
$Env:CXX="$b/g++"

# important to put UCRT first to avoid "collect2.exe: error: ld returned 116 exit status" and DLL Hell
$Env:Path = "$b;$Env:Path"

$Env:CMAKE_PREFIX_PATH="$r"

When it’s desired to use MSYS from a PowerShell prompt run “gcc.ps1”.

From MSYS2 command prompt, search for packages like:

pacman -Ss gcc

MSYS2 packages of interest for scientific computing include: gcc, gdb, gcc-fortran, clang, boost, lapack, scalapack, HDF5, ninja, make, pkgconf, aspell. Install packages with environment prefix “mingw-w64-ucrt-x86_64-” for x86_64 Windows applications for example Gfortran “mingw-w64-ucrt-x86_64-gcc-fortran”. On ARM64 Windows, use “mingw-w64-clang-aarch64-” environment prefix for example Clang “mingw-w64-clang-aarch64-clang”.

You may need to reorder directories in the Windows Path variable. For example GNU Octave may need to be moved lower in the Path list or removed from Path.

If you find that MSYS2 is using more 500 MB, try clearing the package cache of old package versions

pacman -Sc

The MSYS2 latest package update feed shows recently updated packages. The MSYS2 Install reference is also useful. PowerShell per-session variable set is useful to set CC, FC, CXX to single intended compiler to build systems.

Alternatives

As compared to Cygwin, MSYS2 works from the Windows Command Prompt or PowerShell. MSYS2 provides native Window binaries. Cygwin does not have a command-line package installer.

Windows Subsystem for Linux is strongly supported, but does not give Windows binaries unless cross-compiling.

Clang MSYS2 environment

Clang, LLVM Flang Fortran compiler, GCC, Boost and many more packages are easily available on Windows via MSYS2. Clang is also available via direct download.

it’s often useful to have separate development environments for each compiler. The Powershell script “clang.ps1” creates a Clang LLVM environment. We don’t permanently put Clang on the user or system PATH to avoid DLL conflicts. Running “clang.ps1” in Powershell enables Clang until that Powershell window is closed.

For MSYS2 Clang and LLVM Flang Fortran compiler, create “clang.ps1” like:

$r="$Env:SystemDrive/msys64/ucrt64"
$b="$r/bin"

$Env:CC="$b/clang"
$Env:CXX="$b/clang++"
$Env:FC="$b/flang"

# important to put UCRT first to avoid "collect2.exe: error: ld returned 116 exit status" and DLL Hell
$Env:Path = "$b;$Env:Path"

$Env:CMAKE_PREFIX_PATH="$r"

For standalone (non-MSYS2) Clang make “clang.ps1” like:

$Env:CC="clang"
$Env:CXX="clang++"
$Env:Path = "$Env:ProgramFiles/LLVM/bin;$Env:Path"

If you need to use the MSVC CL-like clang driver clang-cl, create “clang-cl.ps1” and run it when desired.

$Env:CC="clang-cl"
$Env:CXX="clang-cl"
$Env:Path = "$Env:ProgramFiles/LLVM/bin;$Env:Path"

Detect if program was compiled with optimizations

Users and developers might accidentally build a program or library without optimizations when they are desired. This could make the runtime 10 to 1000 times or more slower than it would be with optimizations. This could be devastating in computational cost on HPC and cause needless schedule delays. Programmatically detecting or using a heuristic to determine if a program was built with optimizations can help prevent this. Such methods are language-specific.

  • CMake, NDEBUG is set if CMAKE_BUILD_TYPE is Release or RelWithDebInfo.
  • Meson: NDEBUG is set if buildtype is release or debugoptimized with
project(..., default_options: ['b_ndebug=if-release'])

C / C++

There is currently no universal language standard method in C / C++ to determine if optimization was used on build. The presence of macro NDEBUG is used by the standard library to disable assertions. One could use if NDEBUG is defined as an indication if optimizations were used.

bool fs_is_optimized(){
// This is a heuristic, trusting the build system or user to set NDEBUG if optimized.
#if defined(NDEBUG)
  return true;
#else
  return false;
#endif
}

Fortran

If the Fortran code is compiled with preprocessing, a method using NDEBUG as above could be used. Fortran iso_fortran_env provides functions compiler_version and compiler_options. These could be used in a fine-grained, per compiler way to determine if optimizations were used.

Python

Distributed Python environments would virtually always be optimized. One can use heuristic checks to help indicate if the Python executable was built in debug mode. I am not yet aware of a universal method to determine if the CPython executable was built with optimizations.

import sysconfig

debug = bool(sysconfig.get_config_var('Py_DEBUG'))