Scientific Computing

John W. Eaton continues to be heavily involved with GNU Octave development as seen in the commit log. The GNU Octave developer community has been making approximately yearly major releases. Octave is useful to:

• run Matlab code to determine if it’s worth porting a function to Python
• use Matlab function from Python with oct2py

Octave allows running Matlab “.m” code without changes for many tasks. “.m” code that calls proprietary toolboxes or advanced functions may not work in Octave.

I generally recommend learning and using Python unless one already has significant experience and a lot of code in Matlab. Practically what happens is that we choose a “good enough” language. What’s important is having a language that most other people are using so we can share results. The team might be building a radar or robot or satellite imager, and what’s being used in those domains is C, C++, Matlab, and Python.

I want a data analysis language that can scale from Cortex-M0 to Raspberry Pi to supercomputer. Yes, Matlab can use the Raspberry Pi as a target, works with software defined radio, etc. Will collaborators have the “right” version of Matlab and the toolbox licenses to replicate results? How can I debug 100 Raspberry Pi’s sitting out in a field? I need to use the GPIO, SDR, do machine learning processing and forward packets, perhaps using coroutines.

Since 2014, MicroPython has been rapidly growing in the number of MPU/SoC it supports. For just a few dollars, numerous IoT wireless modules can run an expansive subset of Python 3 including exception handling, coroutines, etc. For rapid prototyping, one can get the prototype SoC running remote sensing code passed to the cloud before the first planning meeting. Consider the higher-level languages ease of development and tools or inherent memory safety.

Like math systems such as Sage, SciLab allows integrating multiple numerical systems together. However, SciLab is its own language–with convenient syntax, and a Matlab to SciLab converter. SciLab, IDL, Mathematica, and Maple suffer from small audience size and limited number of third-party libraries.

If an SSH session hasn’t been used for a while or the laptop goes to sleep, the SSH session typically disconnects. This leaves a frozen Terminal window that can’t be used. Usually the Ctrl c keyboard combo does not work.

To avoid having to close the Terminal window, unfreeze the SSH client so that the same Terminal window can be used to reconnect to the SSH session. This avoids needless rearranging when you’ve already got a desired tab/window layout for the Terminal.

In the Terminal windows, press these keys in sequence:

Python distutils was removed from Python 3.12 as proposed in PEP632. Setuptools 49.1.2 vendored distutils, but has experienced some friction in setuptools 50.x since so many packages monkeypatch distutils due to the little maintained status of distutils for several years.

With distutils deprecation in Python 3.10, migration to setuptools is a topic being worked on by major packages such as Numpy. Aside from major packages in the Scipy/Numpy stack, I don’t recall many current packages relying on distutils. However, there is code in some packages using import distutils that could break.

I applaud the decision to remove distutils from Python stdlib despite the fallout. The fallout is a symptom of the legacy baggage of Python’s packaging. Non-stdlib packages like setuptools are so much more nimble that sorely needed improvements can be made more rapidly.

Reference: bug report leading to PEP632

Compilers define macros that can be used to identify a compiler and platform from compiled code, such as C, C++, Fortran, et al. This can be used for platform-specific or compiler-specific code. If a significant amount of code is needed, it may be better to swap in different code files using the build system instead of lengthy #ifdef logic. There are numerous examples for C and C++ so here we focus on macros of Fortran compilers.

Gfortran compiler macros

Macro definitions are obtained in an OS-agnostic way by:

echo "" | gfortran -dM -E - > macros.txt

that creates a file “macros.txt” containing all the compiler macros.

commonly used macros to detect operating system / compiler configuration include:

• _WIN32 1
• __linux__ 1
• __unix__ 1
• __APPLE__ 1

CAUTION: these macros are actually not available in the Gfortran compiled programs as they are in GCC. A workaround is to have the build system define these for the particular compiler, OS, etc.

Clang LLVM compiler macros

LLVM-like compilers (including AMD AOCC) macro definitions are obtained in an OS-agnostic way by:

echo "" | clang -dM -E - > macros.txt

that creates a file “macros.txt” containing all the compiler macros.

__VERSION__ and __clang_version__ contain string version information.

Intel oneAPI LLVM compiler macros

oneAPI macros set:

• __INTEL_LLVM_COMPILER 1

to distinguish from oneAPI Classic compiler macros like __INTEL_COMPILER 1

Cray compiler macros

Detect Cray compiler wrapper with compiler macro __CRAYXT_COMPUTE_LINUX_TARGET instead than non-universal __CRAYXC or __CRAYXE.

or use Cray environment variable PE_ENV and check for CRAY or PrgEnv-.

NVIDIA HPC compiler macros

Print compiler macros like:

touch main.c main.cpp main.f90

nvc -dryrun main.c
nvc++ -dryrun main.cpp
nvfortran -dryrun main.f90

NVIDIA HPC macros include:

• __NVCOMPILER
• __NVCOMPILER_LLVM__

Flang-f18 compiler macros

Flang-f18 (flang-new) Fortran compiler macros include __flang__ 1

Other Fortran compiler macros that identify the compiler and platform can be found in CMake source code.

Fortran I/O operations like read() and write() have an optional iostat argument that can be used to capture error codes rather than stopping the program. Fortran iostat integer non-zero error codes don’t have standard values across compilers. Functions like is_iostat_end() allow standard identification of end-of-file conditions across compilers.

There generally aren’t tables published of compiler-specific iostat codes, besides the source code of the compiler itself. Intel oneAPI iostat integer codes are shared with other conditions in the Intel Fortran runtime library.

Git hosting services including GitLab and GitHub can use Git over SSH for enhanced security.

Setup the file “~/.ssh/config” for Git SSH Public Key authentication like:

Host *
   IdentitiesOnly yes
   PubKeyAuthentication yes
   Port 22

Host gist.github.com
  User git
  IdentityFile ~/.ssh/github

Host github.com
  User git
  IdentityFile ~/.ssh/github

Host gitlab.com
  User git
  IdentityFile ~/.ssh/gitlab

if needed, set file permissions (non-Windows OS):

chmod 400 ~/.ssh/config

Create Git SSH key like:

ssh-keygen -t ed25519 -f ~/.ssh/github

For speed, consider Git pull over HTTPS, Git push over SSH.

Multiple accounts on same Git service

To use Git with SSH and multiple public keys e.g. for same Git server with multiple accounts, setup the file “~/.ssh/config” like:

Host github-work
    HostName github.com
    User git
    IdentityFile ~/.ssh/github-work

Host github-personal
    HostName github.com
    User git
    IdentityFile ~/.ssh/github-personal

Then clone the repository like:

git clone github-work:username/repo.git

# or

git clone github-personal:username/repo.git

To set the username and email distinct for this repo if desired, from that repo directory:

git config user.name "Your Name Here"
git config user.email your@email.invalid

PowerShell has functionality somewhat like grep. Use regular expressions to find matching text in files and piped text using Select-String.

Grep files:

Get-ChildItem -Path . -Filter *.txt -Recurse | Select-String -Pattern 'foo'

Grep specific file:

Select-String -Path CMakeLists.txt -Pattern 'project'

Grep piped text:

cat CMakeLists.txt | Select-String -Pattern 'project'

Grep piped text with regex:

cat CMakeLists.txt | Select-String -Pattern 'project\s*\('

Make a grep() function in PowerShell by saving the following to a file “grep.ps1”. To use this function in PowerShell command line, source the file with . grep.ps1 and then use the function from the PowerShell command line like grep pattern filename. One could make this function fancier to be more grep-like.

function grep($pattern, $path) {
    Select-String -Path $path -Pattern $pattern
}

On Windows, VS Code “code” shell command is enabled by default when installing VS Code:

winget install -e --id Microsoft.VisualStudioCode

The installer program adds to “Path” environment variable like:

%LOCALAPPDATA%\Programs\Microsoft VS Code\bin

That directory contains “code.cmd” that invokes “Code.exe” in the directory above. This technique avoids DLL Hell due to to the .dll files in the path above interfering with other programs if that folder was on Path.

For macOS, install VS Code shell command by pressing Shift ⌘ P . For Linux the keys are Shift Ctrl P . Start typing “shell command”. Select “Shell Command: Install ‘code’ command in PATH”. This allows starting VS Code from Terminal by typing code.

RHEL adds a few complexities on top of the “plain” Debian-based Linux distros. SELinux in particular is another layer to consider. Here are common considerations for settings up SSH and/or VNC on RHEL, CentOS or similar RHEL-derived Linux distro.

For SSH connection problems, consider adding to /etc/ssh/sshd_config:

LogLevel DEBUG

then:

systemctl restart sshd.service

then:

journalctl -f

and try to login. This will print a good amount of information streaming and helps reveal .ssh/authorized_keys permissions issues and more.

To determine if SELinux is causing an issue, as a last resort one may temporarily and carefully edit /etc/selinux/config to have

SELINUX=permissive

and reboot. Be sure to put that back to enforcing and reboot when done.

Check that firewalld is allowing the desired SSH port through.

tcpdump port 22 -n -Q inout

should show packets from the client–if not the SSH server firewall may be blocking them.

Waypipe remote desktop

In general current VNC servers are not compatible with the Wayland desktop (that replaces X11). Waypipe is a new remote desktop tool that works with Wayland. However, Waypipe requires another Linux machine with Wayland, so it doesn’t work with Windows or macOS natively. If permissible for your system, you may wish to switch the desktop to X11 instead of Wayland so that traditional VNC servers work.

X11 VNC server

If you are able to switch to X11, then you can use a traditional VNC server like TigerVNC.

Switch to X11 desktop on the server by editing /etc/gdm/custom.conf:

[daemon]
WaylandEnable=false
DefaultSession=gnome-xorg.desktop

Then reboot. Ensure you can locally logon to the X11 desktop as usual.

Install TigerVNC server:

sudo dnf install tigervnc-server

Ensure username is specified with a display number in file: “/etc/tigervnc/vncserver.users”. You do not need or want a file ~/.vnc/xstartup or ~/.vnc/config.

Enable SELinux VNC server:

ausearch -c 'vncsession' --raw | audit2allow -M my-vncsession

semodule -X 300 -i my-vncsession.pp

Set a password for the VNC server:

vncpasswd

Then start the VNC server:

systemctl enable --now vncserver@:2

Logoff the local server, otherwise when you try to view VNC, it will just show a black screen. If you are logged on remotely and try to login locally, the local login gets a black screen. If this happens, you can logout the unwanted black-screen sessions by:

who -u

# gives PID of the local user (:1)

then:

kill <PID of local user>

Reference: RHEL Remote desktop

Linux package managers like Apt and DNF can be set to not install recommended packages by default, which can be useful for headless embedded systems or WSL. This can save 100s of MB or more, which can be important for embedded systems or WSL to save time and disk space. Saving of download/install time is often relevant to CI/CD systems. A caveat is that some expected features may be missing.

Apt

From the command line (without changing default):

apt install --no-install-recommends <package>

Create a file like “/etc/apt/apt.conf.d/95-no-install-recommends” with the following contents:

apt::install-recommends "false";

To override this setting at the command line, to install recommended packages add “–install-recommends” to the apt command line.

DNF

From the command line (without changing default):

dnf install --setopt=install_weak_deps=False <package>

Edit “/etc/dnf/dnf.conf” adding under [main]:

install_weak_deps=False