Scientific Computing

Git projects using submodules can be set to default shallow Git clone submodules to save space and time. Edit the “.gitmodules” file to have the “shallow = true” option for each Git submodule. This is particularly useful when the top-level project uses third party libraries or libraries with a large Git revision history.

Example .gitmodules file with shallow Git submodules:

[submodule "proj1"]
	path = proj1
	url = https://github.invalid/nobody/proj1
    shallow = true

Then Git clone with the --recurse-submodules option or Git submodule update with the --init --recursive options:

git clone --recurse-submodules <url>

or if already Git cloned

git submodule update --init --recursive

performs a shallow clone of the Git submodules.

Confirm that the submodules are shallow cloned by checking the Git log of the submodule:

git -C ./proj1 rev-parse --is-shallow-repository

These each return “true” indicating that the submodule is shallow cloned.

USB-C dock display adapters (HDMI, DisplayPort) are generally seen as a preferable replacement for proprietary docking ports on old bulky laptops. A symptom is USB-C adapter works on one laptop, but not on another almost identical laptop model.

For Apple hardware, USB-C docks might only work with DisplayPort connection, not HDMI. This may be because some USB-C docks use “DisplayPort Alternate Mode”, which macOS seems not to recognize with an HDMI connection even on the same monitor that works with a DisplayPort cable connection.

General symptoms across hardware and operating systems include the dock display not being detected by the operating system, to showing a black screen, or very low resolution.

The solution seems to be to stick with expensive OEM display adapters, or at least long established brands. The temptation of a cheap adapter can quickly turn to frustration or botched presentations. Another thing to watch for is cheap adapters may fail intermittently when using more than one high bandwidth feature. For example, using Gigabit Ethernet and HDMI on the cheap USB-C adapter simultaneously may fail intermittently during a conference call or teaching a class, which can be frustrating.

Workarounds

Some adapters that charge the laptop with a USB-C input for power may experience improper operations if the display adapter is plugged into the laptop while the USB-C power input is powered. This problem may persist upon re-plugging the adapter to laptop and/or power cycling the monitor and laptop. A workaround we’ve found is to unplug USB-C power input to the adapter, plug into the laptop with all the desired accessories, then finally plug USB-C power input into the adapter. That is unexpected, but has worked for us sometimes.

The COMMENT field of CMake’s add_custom_target() is intended to print messages during the build. However, this field is a no-op when using CMake Generators like Ninja and GNU Make. To reliably print messages during the build regardless of the Generator used, create a custom echo command using CMake’s built-in cmake -E echo functionality.

set(_out ${PROJECT_BINARY_DIR}/output.txt)

add_custom_target(
    COMMAND ${CMAKE_COMMAND} -E touch ${_out}
    COMMAND ${CMAKE_COMMAND} -E echo "Touch ${_out}"
)

GoogleTest internally set CMake compiler flags are too aggressive for end users, and can cause build errors. We experienced this with Intel oneAPI, and created a workaround for the GoogleTest-consuming CMake project to override the offending flags. This technique is useful in general with third-party CMake projects, including those obtained by FetchContent.

For GoogleTest, we determined that Intel oneAPI was experiencing nuisance internal errors:

• Windows “-WX” flag
• Linux “-Werror” flag

We overrode those flags with this CMake script:

FetchContent_MakeAvailable(googletest)

if(CMAKE_CXX_COMPILER_ID STREQUAL "IntelLLVM")
  foreach(t IN ITEMS gtest gtest_main gmock gmock_main)
    if(WIN32)
      # necessary since GoogleTest injects /WX blindly and that fails builds with modern IntelLLVM.
      # https://learn.microsoft.com/en-us/cpp/build/reference/compiler-option-warning-level
      target_compile_options(${t} PRIVATE $<$<COMPILE_LANGUAGE:CXX>:/WX->)
    else()
      # necessary to avoid
      # error: unknown warning option '-Wno-implicit-float-size-conversion' [-Werror,-Wunknown-warning-option]
      target_compile_options(${t} PRIVATE $<$<COMPILE_LANGUAGE:CXX>:-Wno-error=unknown-warning-option>)
    endif()
  endforeach()
endif()

The override “/WX-” nullifies the /WX flag that errors for nuisance warnings internal to GoogleTest. The “-Wno-” flag is the same as the underlying GCC compiler on Linux.

CMake --graphviz can generate GraphViz target dependency graphsfor CMake-supported project code languages including C, C++, and Fortran. Fortran executables and modules are shown in the directed dependency graph.

The “dot” GraphViz program converts the .dot files to PNG, SVG, etc. dot program is available by:

• Homebrew: brew install graphviz
• Winget: winget install -e --id Graphviz.Graphviz

Generating the dependency graph requires CMake configure and generate. Thus, the compiler and generator needed by the CMake project must be working. The project does not need to be compiled before generating the dependency graph. However, the user should select the same CMake configure options as they would for compiling the project.

Example: MUMPS dependency graph is below. SVG vector graphics can be zoomed arbitrarily large in a web browser. The “graphviz/” directory is to avoid making files in the source directory.

cmake -B build

cmake --build build -t graphviz

using script in CMakeLists.txt like:

find_program(GRAPHVIZ_DOT NAMES dot)
if(GRAPHVIZ_DOT)
    file(MAKE_DIRECTORY ${PROJECT_BINARY_DIR}/graphviz)
    set(_graph_name ${PROJECT_BINARY_DIR}/graphviz/${PROJECT_NAME})
    add_custom_target(graphviz
      COMMAND ${CMAKE_COMMAND} --graphviz=${_graph_name}.dot -B${PROJECT_BINARY_DIR} -S${PROJECT_SOURCE_DIR}
      COMMAND ${GRAPHVIZ_DOT} -Tsvg ${_graph_name}.dot -o ${_graph_name}.svg
      COMMAND ${CMAKE_COMMAND} -E echo "CMake ${PROJECT_NAME} targets graph: ${_graph_name}.svg"
    )
endif()

Another example is the h5fortran project graph:

The “dependers” files show only the nodes depending on a node.

Scripts and Viewing output

CMakeUtils graph.py converts a directory of CMake dot diagrams to SVG or PNG and collects them in an HTML document for easy viewing:

python cmakeutils/graph.py ~/myprog/graphviz/

To open a web browser from the Terminal:

python -m webbrowser -t file:///$HOME/myprog/graphviz/index.html

Note that “file:///” has three slashes and the file path must be absolute.

Related: Dependency graphs are also easily created in Python and Matlab.

CMake build targets are declared by “add_[executable,library,custom_target]” commands. Targets can be dynamically set by arbitrarily complex foreach(), if(), etc. logic. A list of CMake targets in the directory scope is retrieved by the BUILDSYSTEM_TARGETS directory property.

The variable “target_names” contains all the target names previously added in the CMakeLists.txt in the DIRECTORY scope. Retrieving the list of targets in a whole project, or in a FetchContent dependency is possible with this CMake function:

function(print_targets dir)
    get_property(subdirs DIRECTORY "${dir}" PROPERTY SUBDIRECTORIES)
    foreach(sub IN LISTS subdirs)
        print_targets("${sub}")
    endforeach()

    get_directory_property(targets DIRECTORY "${dir}" BUILDSYSTEM_TARGETS)
    if(targets)
        message("Targets in ${dir}:")
        foreach(t IN LISTS targets)
            message("  • ${t}")
        endforeach()
    endif()
endfunction()

Use this function like:

print_targets("${CMAKE_CURRENT_SOURCE_DIR}")

Or supposing FetchContent, here using “googletest”:

FetchContent_MakeAvailable(googletest)

FetchContent_GetProperties(googletest)
print_targets(${googletest_SOURCE_DIR})

results in:

Targets in _deps/googletest-src/googletest:
  • gtest
  • gtest_main
Targets in _deps/googletest-src/googlemock:
  • gmock
  • gmock_main

Note: get_property(test_names GLOBAL PROPERTY BUILDSYSTEM_TARGETS) will return an empty list–DIRECTORY scope must be used.

Windows Subsystem for Linux (WSL) uses VHDX files to store each distribution’s filesystem. Over time these disk images grow in size as files are added and deleted. However, the space used by deleted files is not automatically reclaimed, leading to larger disk images than necessary.

There is a PowerShell script to automatically compact WSL VHDX files that works with any of the Windows release levels (including Home).

The jpegtran command line tool allows lossless transformations on JPEG images, including rotation, cropping, and flipping. This is particularly useful when you want to modify JPEG images without re-encoding them, which can lead to quality loss.

Example: lossless clockwise 90 degree rotation of a JPEG image “input.jpg” and save the result to file “output.jpg”:

jpegtran -rotate 90 input.jpg > output.jpg

WiFi captive portals and public networks often block outbound network port traffic. Sometimes even VPNs are blocked. Often only ports 80 (HTTP) and 443 (HTTPS) are allowed.

For Git, one can use Git with HTTPS and Oauth tokens instead of Git over SSH. However, note that Git over SSH has certain benefits for ease of use and security.

To quickly determine if outbound network ports are blocked, portquiz.net is a useful free service. Using Python automates this process for multiple ports concurrently using concurrent.futures.ThreadPoolExecutor threads or asyncio. We provide an example of each method in short scripts. The examples shows that Asyncio using Semaphore with AIOHTTP can be faster than ThreadPoolExecutor with urllib.request.

Solutions to blocked ports for SSH include using SSH ProxyJump with an intermediate TRUSTED server on an allowed port. Some remote SSH systems actually require this, where they the desired server has only LAN access, and a gateway SSH server with no privileges is used as the SSH ProxyJump by network design.

The ultimate workaround would be a mobile hotspot (different network).

LaTeX code highlighting is possible using the minted LaTeX package, which uses Pygments as a backend. Minted is available in Overleaf as well.

If needed, install minted via TeX Live Manager (tlmgr)

tlmgr install minted

Install Python Pygments:

python -m pip install pygments
# or
brew install pygments

If using a GUI, the --shell-escape option may need to be added. In TeXmaker / TeXstudio, under “Options → Configure → Commands” add --shell-escape right after the compiler executable like xelatex --shell-escape or pdflatex --shell-escape or latexmk --shell-escape.

If “pygmentize” isn’t found, under TeXstudio Preferences → Build, check Show Advanced Options in the lower left checkbox and set Additional Search Paths → Commmands ($PATH) to the directory containing “pygmentize”, e.g. /opt/homebrew/bin on macOS with Homebrew or whatever directory comes up for which pygmentize in a Terminal.