Scientific Computing

HDF5 file version

When reading HDF5 files, the HDF5 file version bounds chosen by the writing program must be no newer than the reading program’s HDF5 library version. Usually this is not a problem unless using a very old program to read HDF5 files. When necessary to write older version HDF5 files, select the HDF5 file version from the writing program API. At the time of writing, there is no way to introspect the HDF5 library version required to read a specific HDF5 file. The only way to know is to try to read data from the HDF5 file and see if it works.

Python h5py: use libver= when necessary:

with h5py.File(libver='earliest') as f:
   ....

Matlab low-level HDF5 interface H5P.set_libver_bounds() set HDF5 file write library version bounds.

C / C++ / Fortran HDF5 library can set the HDF5 library version bounds by H5Fset_libver_bounds() and H5Pset_libver_bounds().

Related: Check HDF5 files for corruption

List symbols in library / object file

When troubleshooting linking a library or object file, it’s useful to search the symbols defined in the file. The nm program lists the symbols in library and object files. Search for a symbol (e.g. function) by piping nm output to grep. For this example the ncurses library is used.

For an object or static library:

nm /usr/lib/libncurses.a | grep myfunc

For a shared library, the particular version suffix might be needed:

nm -D /usr/lib/libncurses.so.6 | grep myfunc

When the line starts with “T”, the symbol is defined in the library. When the line starts with “U”, the symbol is undefined in the library. Note that header defines, such as used in PDCurses are not listed by nm. That is, PDCurses curses.h is not detected by nm.

#define getch() wgetch(stdscr)

For Fortran, since the “.h” files are not used, the developer needs to provide an interface mapping the functions in Fortran. Note that the developer must use symbol detection code in the build system (e.g. CMake) to ensure the symbol is only defined once. This is important for a library like Curses that has multiple independent implementation. Specifically, the Ncurses library defines “getch” in the .c source, but PDCurses defines “getch” in the .h header.

The Blocktran project shows how to handle this situation with getch() macro.

pkgconf is generally preferred over pkg-config

pkgconf is a modern drop-in replacement for pkg-config. Build systems like CMake, Meson, and Autotools all support pkgconf to help them find dependencies if the package has pkgconf files.

Installing pkgconf:

  • Linux: apt install pkgconf
  • macOS: brew install pkgconf

CMake include-what-you-use (IWYU)

Include-what-you-use IWYU is a static analysis tool that helps find missing and unused #include statements in C / C++ source files. Like any tool, IWYU makes mistakes, so make changes incrementally and be ready to rollback edits. CMake has IWYU support that is enabled in project CMakeLists.txt by including this stanza BEFORE any targets are defined:

option(iwyu "Run include-what-you-use")
if(iwyu)
  find_program(IWYU_EXE NAMES include-what-you-use REQUIRED)
  set(CMAKE_C_INCLUDE_WHAT_YOU_USE ${IWYU_EXE})
  set(CMAKE_CXX_INCLUDE_WHAT_YOU_USE ${IWYU_EXE})
endif()

Upon building the project, CMake will run IWYU on all source files. IWYU emits messages as each source file is built, if there are any issues. IWYU does not emit messages by default when there are no issues.

Troubleshooting

Ensure that IWYU is working by deliberately including an unused header. For example, on MSYS2 sometimes IWYU never emits messages with CMake.

Matplotlib / Matlab label tick at min / max value

For numerical plots, it can be important to label the ticks of axes extrema (minimum or maximum). For example, to clearly show the edges of simulated data axes. This can be easily done in Python Matplotlib or in Matlab. This assumes the typical case that the axes values are numeric.

In this example, the y-axis ticks show the endpoints of the y-data range: -3.375 and 27.0.

The data are in general non-monotonic, so we sort ticks before resetting them.

Python: Matplotlib does not require sorting the ticks.

from matplotlib.pyplot import figure, show
import numpy as np

# synthetic data
x = np.arange(-1.5, 3.25, 0.25)
y = x**3

fg = figure()
ax = fg.gca()
ax.plot(x, y)

# label min and max ticks
yticks = np.append(ax.get_yticks(), [y.min(), y.max()])
ax.set_yticks(yticks)

show()

Matlab: requires sorting the ticks before resetting them.

% synthetic data
x = -1.5:0.25:3.0;
y = x.^3;

fg = figure;
ax = axes(fg);
plot(ax, x, y)

% label min and max ticks
yticks = sort([ax.YTick, min(y), max(y)]);
ax.YTick = yticks;

Install Qt in RHEL for CMake GUI

Qt GUI can be installed in RHEL-like Linux from the CRB repository.

dnf config-manager --set-enabled crb

dnf install qt6-devel

Qt can be used to build cmake-gui like:

# from CMake source directory
cmake -Bbuild -DBUILD_QtDialog=ON

cmake --build build

This results in “build/bin/cmake-gui” being built.

Matplotlib / Matlab log axis plots

In Python Matplotlib or Matlab, making a plot with log-scaled (instead of default linear-scaled) axes can use the functions like “loglog()”, “semilogx()”, or “semilogy()”. Here we show the more general object-oriented syntax for each of a 2-D line “plot()” and 2-D pseudocolor “pcolor()” and then set the axes scale properties.

We arbitrarily use log abscissa scale and linear ordinate scale, but the same syntax applies to other combinations.

Python

2-D line plot:

import numpy as np
from matplotlib.pyplot import figure, show

x = np.logspace(0, 10, 100)
y = np.log(x)**2

fg = figure()
ax = fg.gca()

ax.plot(x, y)
ax.set_xscale('log')

ax.set_xlabel('x')
ax.set_ylabel('y')

show()

pseudocolor pcolormesh(): observe the stretching of the ticks along the y-axis. In some cases it’s helpful to set the axis limits manually to avoid whitespace past the last data point.

import numpy as np
from matplotlib.pyplot import figure, show

d = np.random.rand(10, 10)
x = np.linspace(1, 10, d.shape[0])
y = np.logspace(0, 1, d.shape[1])

fg = figure()
ax = fg.gca()

ax.pcolormesh(x, y, d)
ax.set_yscale('log')
ax.set_ylim(y[0], y[-1])

ax.set_xlabel('x')
ax.set_ylabel('y')

show()

Matlab

2-D line plot:

x = logspace(0, 10, 100);
y = log(x).^2;

fg = figure();
ax = axes(fg);

plot(ax, x, y);
ax.XScale = 'log';

xlabel(ax, 'x');
ylabel(ax, 'y');

pseudocolor pcolor(): observe the stretching of the ticks along the y-axis. In some cases it’s helpful to set the axis limits manually to avoid whitespace past the last data point.

d = rand(10, 10);
x = linspace(1, 10, size(d, 1));
y = logspace(0, 1, size(d, 2));

fg = figure();
ax = axes(fg);

pcolor(ax, x, y, d);
ax.YScale = 'log';

xlabel(ax, 'x');
ylabel(ax, 'y');

Troubleshooting DNS problems

If one suspects a website has been compromised, don’t use a standard web browser to access the site as there could be zero-day malware on the site. Consider Terminal programs that don’t have JavaScript enabled like curl or lynx if necessary to browse the site, preferably from a VM or other isolated computing resource. These programs are also not immune from security vulnerabilities.

DNSViz helps visualize the DNS chain. Keep in mind DNS and nameserver updates can take minutes to hours to propagate.

macOS:

dscacheutil -q host -a name host.invalid

Linux / macOS / WSL:

dig +trace host.invalid

If the DNS entries seem valid, consider that the web hosting server (that sends the HTML files to browser) may be compromised.

Compile Matlab .m code executable

Matlab Compiler compiles existing .m Matlab script to run as an executable on another PC without Matlab. Matlab Compiler Runtime MCR is used on computers that don’t have Matlab to run the compiled Matlab code.

Caveats:

  • Matlab Compiler does not in general speedup Matlab code execution
  • in general, compiled binaries might be disassembled to reverse-engineer the underlying code
  • MCR version on each computer running the executable must match the Matlab version of the compiling Matlab, and the compiling computer must have the same operating system as the MCR running computers.

Compiling computer: ensure Matlab Compiler is installed:

assert(license('test', 'compiler') == 1)

Example program “mymcc.m”:

function Y = mymcc()

X = 0:0.01:2*3.14;
Y = sin(X);
plot(X,Y)
title('Test of MCR')
xlabel('x')
ylabel('y')
disp('I ran an MCR program!')

end

Compile “.m” file in Matlab:

mcc -m -v mymcc.m

Run compiled Matlab program:

./run_mymcc.sh mymcc

I ran an MCR program!

and show a Matlab plot window showing a sine wave. Close the plot window to end the execution of your program.

Notes:

Reference

GNU Octave does not currently have the ability to compile “.m” files. Octave mkoctfile is to distribute C / C++ code that calls Octave functions–and ABI-compatible Octave must be installed on the user computers