LAMMPS Hygon DCU build and performance validation on two platforms

1. Test platforms and verification scope

This report brings together two independent validations built around the same engineering workflow. Platform A covers gfx936 architecture support, GPU package comparisons, and source optimizations; Platform B covers native gfx906/Z100 builds, correctness, and multi-card scaling. Locations, cluster aliases, SSH addresses, partition and node names, and absolute user paths are redacted.

2. Build strategies and issue analysis

2.1 GPU package with the HIP backend

Configuration: -D PKG_GPU=ON -D GPU_API=HIP -D HIP_ARCH=gfx936

First result:Compiled successfully, but hung when running

Initial diagnosis:DTK's CMake configuration applies the -xhip flag to the GPU package, causing hipcc to compile every .cpp file for the device. The resulting executable hung during HIP initialization.

After the fix:Setting AMDGPU_TARGETS="gfx936" and disabling HIP_USE_DEVICE_SORT allowed the build to compile and run. Performance tests nevertheless showed the GPU package to be 4.6–6.3× slower than Kokkos (see the comparison in Section 6).

2.2 Kokkos HIP backend with the default architecture (mismatch)

Configuration: -D PKG_KOKKOS=ON -D Kokkos_ENABLE_HIP=ON

Result:The build succeeds, but execution fails with running kernels compiled for gfx906 on gfx936.

Analysis:Kokkos 4.6.2 does not recognize gfx936 and falls back to gfx906. Because the gfx906 and gfx936 instruction sets are incompatible, execution fails with hipErrorInvalidKernelFile.

2.3 Add gfx936 support to Kokkos

Approach:Add gfx936 support explicitly to Kokkos' CMake configuration and HIP backend.

2.4 gfx906/Z100: native architecture path

Unlike gfx936, gfx906 (Vega 20) has been natively supported by Kokkos 4.6.2, so there is no need to modify the four architecture files of Kokkos and directly enable Kokkos_ARCH_AMD_GFX906=ON and AMDGPU_TARGETS=gfx906. This reduces architecture porting costs, but DTK 26.04 and the cluster toolchain expose another set of build issues.

2.5 DTK 26.04 cuda_wrappers host mode conflict

At about 98% of the build, main.cpp failed because cuda_wrappers/algorithm reported unknown type name '__host__'. In LAMMPS Update 5, cmake/CMakeLists.txt:854 forces the lmp target to -x c++ to address an nlohmann/json HIP SFINAE issue. DTK 26.04's hipcc still injects CUDA wrappers in host mode, causing the standard <algorithm> header to be parsed incorrectly.

target_compile_options(lmp PRIVATE
  "$<$<COMPILE_LANGUAGE:CXX>:SHELL:-xhip --offload-arch=gfx906>")

2.6 HSA library symlink unavailable on the CPU node

The libhsa-runtime64.so symlink resolves through the DCU driver directory /opt/hyhal/, which is absent from the CPU build node. CMake therefore cannot find the HSA runtime. Set HSA_RUNTIME_LIBRARY explicitly to the actual shared-library path; the public commands use the placeholder <DTK_ROOT>/hyhal/lib/libhsa-runtime64.so.

2.7 Broken make interpreter and login-node memory pressure

On Platform B, the system /usr/bin/make references a missing ELF interpreter at /opt/glibc-2.25/lib/ld-linux.so.2; load compiler/make/4.4 instead. The login node has about 251 GB of shared memory for roughly 249 users, with only about 46 GB available during testing. When hipcc compiled roughly 1,000 C++ files, dcc was killed by the OOM killer. The build was moved to the CPU compute queue and submitted with 32 cores and about 111 GB of exclusive memory.

2.8 Restrict the build to the target architecture

DTK 26.04 targets gfx906;gfx926;gfx928 by default, compiling each source file three times and increasing build time and memory use. Set export AMDGPU_TARGETS=gfx906 to generate code only for the target architecture.

2.9 gfx906 GPU-aware MPI path

All 12 peer-to-peer checks among the four Z100 cards on Platform B returned hipDeviceCanAccessPeer=1. DTK 26.04 lacks libhsakmt.a, however, so UCX's ROCm transport cannot be built. The alternative is to build OpenMPI 5.0.7 from source with --with-rocm, which provides accelerator: rocm and mpiext: rocm. Use shared memory and disable UCX's network path with --mca pml ob1 --mca btl vader,self and UCX_TLS=sm,self.

2.10 Build differences between the two platforms

3. Build configuration and commands

# Use hipcc and enable the Kokkos, HIP, and OpenMP packages
cmake ../lammps-22Jul2025/cmake \
  -D PKG_KOKKOS=ON \
  -D Kokkos_ENABLE_HIP=ON \
  -D PKG_OPENMP=ON \
  -D PKG_KSPACE=ON \
  -D PKG_MANYBODY=ON \
  -D PKG_MOLECULE=ON \
  -D PKG_RIGID=ON \
  -D BUILD_MPI=OFF \
  -D CMAKE_INSTALL_PREFIX=$HOME/softwares/lammps/dcu-install \
  -D CMAKE_CXX_COMPILER=hipcc \
  -D CMAKE_C_COMPILER=gcc \
  -D CMAKE_HIP_COMPILER=hipcc \
  -D HIP_PATH=<DTK_ROOT>/hip

3.1 Single-card build on gfx906/Z100

# Build for gfx906 only; public placeholders are used for shared software and install paths
export AMDGPU_TARGETS=gfx906

cmake -S lammps-22Jul2025/cmake -B build-gfx906 \
  -D PKG_KOKKOS=ON \
  -D Kokkos_ENABLE_HIP=ON \
  -D Kokkos_ARCH_AMD_GFX906=ON \
  -D AMDGPU_TARGETS=gfx906 \
  -D HSA_RUNTIME_LIBRARY=<DTK_ROOT>/hyhal/lib/libhsa-runtime64.so \
  -D AMD_HIP_LIBRARY=<DTK_ROOT>/lib/libamdhip64.so \
  -D PKG_OPENMP=ON -D PKG_KSPACE=ON -D PKG_MANYBODY=ON \
  -D PKG_MOLECULE=ON -D PKG_RIGID=ON \
  -D BUILD_MPI=OFF \
  -D CMAKE_INSTALL_PREFIX=<USER_HOME>/softwares/lammps/dcu-install \
  -D CMAKE_CXX_COMPILER=hipcc -D CMAKE_C_COMPILER=gcc \
  -D CMAKE_HIP_COMPILER=hipcc -D HIP_PATH=<DTK_ROOT>/hip

Test job script

#!/bin/bash
#SBATCH -p <partition>
#SBATCH -N 1
#SBATCH --gres=dcu:1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=16
#SBATCH --time=00:10:00

module load compiler/gcc/9.3.0
module load compiler/dtk/25.04.2

export PATH=$HOME/softwares/lammps/dcu-install/bin:$PATH

lmp -k on g 1 -sf kk -in input.lj

4. Performance results

4.1 Test configuration and workload

4.2 Benchmark results

4.3 DCU resource monitoring

5. Main results

5.1 gfx906/Z100 Correctness and Energy Conservation

platform B uses the same input to run Kokkos/HIP and CPU/OpenMP 16 threads respectively to avoid judging the success of the transplant based on performance alone.

LJ comes from the parallel floating point summation order. In the NVE convergence test, LJ can always change from -4.6203 to -4.6213 after equilibrium, with a drift less than 0.02%; EAM changes from -106640.66 to -106640.19, with a drift less than 0.0004%.

5.2 gfx906/Z100 calculation example coverage

5.3 gfx906/Z100 single card weak expansion

5.4 gfx906/Z100 multi-card strong expansion (2M atoms)

GPU-aware MPI changes two-card performance from a slowdown to a 1.35–1.56× speedup, and four-card performance reaches 1.86–2.33×. This result is in stark contrast to Platform A's multi-card non-breakout single card, indicating that the communication stack, P2P topology, problem size, and site configuration are all part of the conclusion.

5.5 Cross-platform performance comparison

The two environments do not have the same input, version and communication stack. This table preserves only the trends reported in the original results and cannot be used as a strict hardware ranking. Platform B's same-node PCIe P2P efficiency is an important factor in multi-card performance.

6. GPU package and Kokkos performance comparison

6.1 Test purpose

This section evaluates whether the GPU package offers a more efficient acceleration path than Kokkos. The GPU package uses hand-written optimized HIP kernels that may theoretically be more efficient than Kokkos' general abstraction.

6.2 GPU package compilation repair

6.3 Performance comparison results (32K atoms, 1000 steps)

6.4 Performance comparison results (256K atoms, 1000 steps)

6.5 Why the GPU package is slower

6.6 Hygon native compiler confirmation

Inspection of the DTK toolchain confirms that this build uses the Hygon native compiler:

# hipcc ultimately invokes Hygon's native dcc compiler
hipcc → <DTK_ROOT>/dcc/bin/dcc
           ↓
  Hygon ISA bitcode: oclc_isa_version_936.bc

Kokkos' AMD_GFX936 is an architecture label. Compilation uses Hygon's dcc compiler and Hygon ISA bitcode, so the resulting code is compiled natively for the DCU.

7. Hotspot analysis and code optimization

7.1 HIP Hotspot Tracking

Hotspots in the 1,048,576-atom dynamic LJ benchmark were analyzed with DTK 25.04.2 hipprof --hip-trace --stats. of the 1,048,576-atom dynamic LJ benchmark.

Pair and Neighbor account for 92.8% of device time and are the only worthwhile optimization targets.

7.2 PMC hardware counter analysis

PMC counters were collected for the Pair and Neighbor hotspots:

Pair does not use LDS and has a high L2 hit rate; Neighbor has a large number of LDS and TCP stall activities, which is the focus of the next optimization step.

7.3 Pair workgroup scan (negative optimization)

tests workgroup 64/128/256/512/1024, staggered operation every three rounds:

improves monotonically as the workgroup increases, and the value of 1024 automatically selected by Kokkos is already the best value in the test. Experimental code has been rolled back.

7.4 Neighbor transpose optimization (+13.6%)

Runtime options neigh/transpose on, no need to recompile, three rounds of interleaved verification:

changes the memory layout of the neighbor list on the GPU (row major → column major) and optimizes the cache access mode of the pair kernel. Attention transpose on is related to the system size: +13.6% for 1M atoms, but slightly slower -1.2% for 32K atoms.

7.5 Neighbor BPT optimization (+4.31%)

tests the number of bins per team in the neighbor build (BPT=1/2/4/8), three rounds of staggered runs:

The default BPT=2 performs worst among the four configurations. Setting BPT=1 improves performance by 4.31% with a one-line change (const int factor = 1;); this setting is retained in the source.

8. CPU and DCU performance comparison

8.1 Limitations of the original “160 cores” comparison

The original report equates one DCU card with a 160-core CPU, based on results from different system sizes:

The normalized throughput ratio is 233.544 / 1.452 = 160.8×, but the CPU and DCU results use different atom counts. The larger DCU case achieves higher GPU utilization, so this cross-scale ratio overstates the speedup for a like-for-like comparison.

8.2 Fair comparison of same scale (32,000 atoms)

At the same scale, one DCU card is about 60× faster, rather than 160×. Follow-up tests measured GPU utilization of 20–41%, depending on the pair style; CPU-side framework overhead was the main bottleneck.

8.3 How problem size changes the GPU speedup

9. Optimization experiments

9.1 Effective optimization

9.2 Invalid optimization (negative optimization or zero benefit)

9.3 Summary of key experiences

10. MPI build and GPU-aware communication

10.1 Background

LAMMPS multi-GPU requires MPI support. There is an ABI mismatch between the OpenMPI that comes with the system (compiled with GCC 8.5.0) and the current GCC 9.3.0, resulting in MPI_Finalize when double free crash. In addition, non-GPU-aware MPI requires CPU↔GPU data copy during each communication step, which results in huge communication overhead.

10.2 Compilation environment

10.3 Compilation steps

# Download and build OpenMPI 5.0.7
wget https://download.open-mpi.org/release/open-mpi/v5.0/openmpi-5.0.7.tar.gz
tar xzf openmpi-5.0.7.tar.gz && cd openmpi-5.0.7

# Load dependencies
module purge
module load compiler/gcc/9.3.0
module load compiler/dtk/25.04.2
module load <UCX_MODULE>

# Configure UCX and ROCm GPU support
./configure --prefix=$HOME/softwares/mpi/install \
  --enable-mpi-fortran=no \
  --with-ucx=<UCX_ROOT> \
  --with-rocm=<DTK_ROOT>

make -j32 && make install

10.4 GPU-aware MPI components

10.5 LAMMPS MPI build

# Build LAMMPS with the locally built MPI
cmake -S lammps-22Jul2025/cmake -B build-mpi \
  -C cmake/presets/hygon-dcu-gfx936.cmake \
  -D CMAKE_CXX_COMPILER=$DTKROOT/bin/hipcc \
  -D BUILD_MPI=ON \
  -D MPI_CXX_COMPILER=$HOME/softwares/mpi/install/bin/mpicxx \
  -D MPI_CXX_INCLUDE_PATH=$HOME/softwares/mpi/install/include \
  -D MPI_CXX_LIBRARIES=$HOME/softwares/mpi/install/lib/libmpi.so

10.6 Runtime configuration

# Key Slurm job settings
module load <UCX_MODULE>
export PATH=$HOME/softwares/mpi/install/bin:$PATH
export LD_LIBRARY_PATH=$HOME/softwares/mpi/install/lib:$LD_LIBRARY_PATH

# Enable GPU-aware communication with -pk kokkos gpu/aware on
mpirun -np 2 lmp -in input -k on g 1 -sf kk \
  -pk kokkos neigh/transpose on gpu/aware on

10.7 MPI version comparison

GPU-aware MPI improves two-card performance by 69% (105 → 177 steps/s) and reduces communication time from 20.2 s to 11.4 s (−43%). The UCX + ROCm path avoids copying data through the CPU.

10.8 UCX selection guidance

DTK 25.04.2 does not include UCX. The tested <UCX_MODULE> provides UCX components with ROCm transport support. Building OpenMPI 5.0.7 with --with-ucx and --with-rocm links these components and enables GPU-aware communication.

UCX module path:<UCX_ROOT>

ROCm path:<DTK_ROOT>

10.9 gfx906/Z100: Same-node solution without UCX ROCm

Platform B lacks the static library libhsakmt.a, so the UCX ROCm transport cannot be built. OpenMPI can still recognize device pointers through its ROCm accelerator extension and use shared memory for communication among cards on the same node:

./configure --prefix=<USER_HOME>/softwares/mpi/ompi-install \
  --enable-mpi-fortran=no \
  --with-rocm=<DTK_ROOT> \
  --with-slurm

export UCX_TLS=sm,self
mpirun --mca pml ob1 --mca btl vader,self \
  -np <GPU_COUNT> lmp -in input \
  -k on g 1 -sf kk -pk kokkos gpu/aware on

This build exposes accelerator: rocm and mpiext: rocm. It was used for intra-node peer-to-peer tests with one to four cards; it does not validate cross-node RDMA.

11. Artifact directory structure

12. Conclusions and scope

12.1 Architecture and compilation

12.2 Effective optimization

12.3 Invalid optimization (rolled back)

12.4 Multi-GPU scaling

EAM 2/1 ratio 0.78→0.87→0.90, trend improving but not breaking 1.0.GPU package has good scalability (2/1=1.32) but single card is slow (~1.8× KOKKOS).

12.5 Fair comparison of CPU and DCU performance

12.6 Single-GPU scaling

Throughput is stable at 80–85 Matom-step/s with near-linear scaling. The 32M-atom case runs out of memory because the neighbor list initially allocates 2,000 entries per atom, exceeding 64 GB. Memory capacity becomes the limit before compute throughput.

12.7 Key lessons

12.8 gfx906/Z100 limitations and workarounds

12.9 Lessons from the two platforms

13. Upstream improvement proposals

13.1 Kokkos upstream

13.2 LAMMPS upstream

13.3 ROCm/HIP compatibility ecosystem