Scientific Computing

Create Docker image

For research reproducibility, a Docker image gives a known software state that should be usable for many years. It isn’t strictly necessary to make a custom image as described below, but it can be useful for a known software state.

Create a Docker image by using a Dockerfile. If desired to set environment variables, use ENV Dockerfile instruction.

FROM alpine:latest

# example for CMake project using Fortran with OpenMPI.
RUN apk add --no-cache ninja-build cmake gfortran openmpi-dev

ENV CMAKE_GENERATOR=Ninja

Create the file above named Dockerfile in the Git repo.

apk
Alpine package manager

Build the Docker image:

docker build -t openmpi-fortran .

from the same directory as Dockerfile

Check the image exists–it may be nearly 500 MB:

docker images

Before committing for upload, invoke the container.

docker run openmpi-fortran

This will almost immediately start and stop, as it didn’t have a command to run or persist.

Get the hexadecimal container ID by:

docker ps -a

The container will have an auto-assigned name. Note the hexadecimal “Container ID”.

upload Docker image

Once the configured Docker container is ready, share this container. This can be done with Docker Hub.

Once ready to upload the image, note the “container ID”, which is the hexadecimal number at the terminal prompt of the Docker container, under docker ps. The container hex ID must appear under docker ps, just being in docker images is not enough.

docker commit -m "Fortran Dockerfile setup" hex_id dockerhub_username/openmpi-fortran

The changes to the image the container made are gathered into a new image. It may take a minute or two for a large image. Ideally with a small image it will take only a couple seconds. The new image has not yet left the computer, it will show up under

docker images

Once uploaded, the Docker image is visible to the world by default.

Login to Docker Hub

docker login -u dockerhub_username

Push (upload) this image to Docker Hub. This image is world-visible by default!

docker push dockerhub_username/openmpi-fortran

If the image is very large > 1 GB, be aware it will take a while to upload, and for the host to download. This is a telltale that it’s important to keep Docker images small.

Docker Fortran image

Docker images are useful for reproducibility and ease of setup and for software binary distribution on platforms not natively available. For example, it may be desired to distribute a statically-linked binary that will run on any Linux system with compatible CPU architecture and kernel system calls.

To setup and maintain Docker images, it’s useful to have Docker Desktop available on the developer laptop to debug and test Dockerfiles. Read the install instructions particular to the laptop OS to understand OS-specific caveats and features. For example, on Linux laptops, to avoid the need for “sudo” in every command, follow the post-install.

Run a Docker container

Docker commands can be run from the system terminal if desired–all commands in this article assume this. Docker images by default will be downloaded to run locally.

Try the Hello World images, which should auto-download and print a welcome message

docker run hello-world

Search for a desired image using Docker Desktop or docker search. Consider the “Official” images first. Let’s use Alpine Linux.

docker search alpine

Get the desired image

docker pull alpine

Verify the images:

docker images

Start the Docker container:

docker run -it alpine
-it
interactive session

Verify the Alpine version running in the container. It will have a # superuser prompt.

cat /etc/os-release

Search for desired APK packages from within the container:

apk update
apk search gfortran

Verify the MUSL C library version like:

ldd

manage containers

These commands are issued NOT from a system Terminal, not the Docker container.

  • list images: docker images -a

  • list containers (running and stopped): docker ps -a

  • stop a Docker container: docker stop container_name

  • start a Docker container: docker start container_name

  • login to a running Docker container: docker exec -it container_name

  • get container environment variables: docker exec container_name env

  • cleanup unused containers docker system prune

Each docker exec command is a new shell instance. Changing directories in one docker exec has no effect on subsequent commands for example.

Docker on GitHub Actions

Docker images are useful for reproducibility and ease of setup and for software binary distribution on platforms not natively available on GitHub Actions runner images. While one can setup a custom Docker image, it’s often possible to simply use an existing official image from Docker Hub.

Example: Ubuntu 20.04

This example GitHub Actions workflow uses the Ubuntu 20.04 image to build a C++ binary with the GNU C++ compiler. For APT operations, the “-y” option is necessary. Environment variable DEBIAN_FRONTEND is set to “noninteractive” to avoid interactive prompts for certain operations despite “-y”. Don’t use “sudo” as the container user is root and the “sudo” package is not installed.

A special feature of this example is using Kitware’s CMake APT repo to install the latest version of CMake on an EOL Ubuntu distro.

name: ubuntu-20.04

on: [push]

# avoid wasted runs
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  ubuntu-20.04:
    runs-on: ubuntu-latest
    container:
      image: ubuntu:20.04
      env:
        DEBIAN_FRONTEND: noninteractive

    strategy:
      fail-fast: false
      matrix:
        gcc-version: [7, 8]

    env:
      CC: gcc-${{ matrix.gcc-version }}
      CXX: g++-${{ matrix.gcc-version }}
      FC: gfortran-${{ matrix.gcc-version }}

    steps:

    - name: install compilers
      run: |
        apt update -y
        apt install -y --no-install-recommends ca-certificates gpg curl ninja-build ${{ env.CC }} ${{ env.CXX }} ${{ env.FC }}

    - name: install CMake
      run: |
        curl -s https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | gpg --dearmor - | tee /usr/share/keyrings/kitware-archive-keyring.gpg >/dev/null
        echo 'deb [signed-by=/usr/share/keyrings/kitware-archive-keyring.gpg] https://apt.kitware.com/ubuntu/ focal main' | tee /etc/apt/sources.list.d/kitware.list >/dev/null
        apt-get update
        test -f /usr/share/doc/kitware-archive-keyring/copyright || rm /usr/share/keyrings/kitware-archive-keyring.gpg
        apt-get install --no-install-recommends -y kitware-archive-keyring
        apt-get install --no-install-recommends -y cmake

    - uses: actions/checkout

    - name: CMake configure
      run: cmake -B build

    - name: CMake build
      run: cmake --build build

    - name: CMake test
      run: ctest --test-dir build

Example: Alpine Linux

This example GitHub Actions workflow uses the Alpine Linux image with the MUSL C library to build a statically-linked binary.

name: alpine-musl

on: [push]

jobs:
  musl:
    runs-on: ubuntu-latest
    container:
      image: alpine

    steps:
    - uses: actions/checkout

    - name: install build tools
      run: apk add --no-cache ninja-build cmake make gfortran openmpi-dev

    - name: print MUSL version
      continue-on-error: true
      run: ldd --version

    - name: CMake configure
      run: cmake -B build

    - name: CMake build
      run: cmake --build build

# Good idea to ensure self-tests pass before packaging
    - name: CMake test
      run: ctest --test-dir build

(Optional) If a CPack archive is desired add step:

    - name: CMake package
      if: success()
      run: cpack --config build/CPackConfig.cmake

The binary artifact or CPack archive can be uploaded by step upload-artifact:

    - name: .exe for release
      uses: actions/upload-artifact@v4
      if: success()
      with:
        name: my.exe
        path: build/my.exe

Open a folder location in macOS terminal

Some folder locations in macOS are not accessible by traversing one level at a time from outside or above that location. In general, it can be convenient to open a terminal window at a specific folder location. The folder may be opened directly by right-clicking on the folder in Finder and selecting “New Terminal at Folder”.

List only file names with PowerShell

By default, listing the contents of a directory with PowerShell shows file metadata along with the path names within the directory. Other shells typically show only the path names by default, which is convenient for copy-pasting lists of files or directories into other commands or scripts. To list only the path names when listing a directory in PowerShell, use the -Name option:

Get-ChildItem -Name

# or

ls -Name

Fortran OpenMP examples

OpenMP and OpenACC are open standards for parallel programming on CPU and GPU shared memory platforms. While OpenMP has wider compiler support, OpenACC also aims to be portable across compilers and computing architectures. OpenMP standard syntax for C, C++, Fortran, and CUDA can work seamlessly with non-OpenMP systems if code is properly written. While OpenMP is typically built in from the factory in contemporary compilers, the compilers may need flag(s) to enable OpenMP. Build systems like CMake and Meson can manage the OpenMP compiler flags.

OpenMP is well-supported in commercial computation performance-oriented Fortran compilers like:

OpenMP support in open-source Fortran compilers includes LLVM and GCC.

Open-source compilers such as LLVM Flang need to have the proper OpenMP-enabling flags set when building the compiler by the distribution maintainers. For example, Homebrew LLVM Flang was missing Fortran libomp.mod support due to misconfiguration in the Homebrew flang formula.

Examples of Fortran OpenMP code:

Git clone a specific branch

“git clone” clones the default branch of the remote repository as determined by the Git server. The default Git branch is typically “main”, but the repository owner can change it to any other branch. To Git clone a specific branch, use the –branch option:

git clone --branch <branch-name> <repository-url>

This can help workaround problems like a default branch that is very large or is broken in some way.

Windows symbolic links and reparse points

Symbolic links are useful in any operating system to shorten long, complicated path names like C:/user/foo/data to just C:/data. If encountering problems with user permission, set user permission to create symbolic links on Windows.

Powershell symbolic link creation syntax:

New-Item -ItemType SymbolicLink -Path "Link" -Target "Target"

# for example:
New-Item -ItemType SymbolicLink -Path "my_program.exe" -Target "path/to/my_program.123.exe"

# also for directories:
New-Item -ItemType SymbolicLink -Path "my_fun_dir" -Target "path/to/my_dir"

For clarity, specify the full path to the target file or directory. Especially avoid target “.” or “..” as these can be confusing.

Symbolic links on Windows are a type of Reparse Points. fsutil can tell the type of reparse point:

fsutil reparsepoint query "my_fun_dir"

Reparse Tag Value : 0xa000000c

The reparse tag value corresponds to a symbolic link IO_REPARSE_TAG_SYMLINK.

Python test_symlink.py shows symlinks using Python standard library pathlib.

App Execution Alias 

fsutil reparsepoint query $Env:LOCALAPPDATA/Microsoft/WindowsApps/wt.exe

Reparse Tag Value : 0x8000001b

The reparse tag value 0x8000001b is a Windows App Execution Alias IO_REPARSE_TAG_APPEXECLINK. App Execution Aliases are not symbolic links, but are a way for Windows CreateProcess to find the correct executable to run from a user-friendly name like “wt.exe” or “bash.exe”.

Not every language works with App Execution Aliases at this time–Java io and nio don’t work with App Execution Aliases currently. Python does work with App Execution Aliases, for example:

python -c "import shutil; print(shutil.which('wt.exe'))"

Unix-like shell

On a Unix-like shell including WSL, softlinks are created like:

ln -s target link