Hashcat HIP Adaptation and Cross-Platform Validation on Hygon DCU

1. Executive Summary

Hashcat v7.1.2 includes a HIP backend, but its built-in HIP ABI is incompatible with Hygon DTK's libgalaxyhip.so, preventing device initialization. This work identified and fixed the compatibility issue on gfx936 with DTK 25.04.2, then applied the same adaptation to gfx906/Z100 with DTK 26.04. The resulting evidence covers ABI analysis, conditional compilation, runtime checks, and cross-version regression.

1.1 Cross-version validation environments

2. Test environment and artifacts

2.3 Build artifacts

The binary example path is /path/to/hashcat/. The OpenCL baseline version is kept separately for regression verification and performance comparison.

3. Compatibility behavior and root-cause analysis

3.1 Login node and compute node behavior

The login node does not expose /dev/kfd, so it can be used to inspect HIP symbols in the binary but not to verify device execution. On an allocated DCU compute node, rocminfo, hy-smi, and OpenCL detect the device as expected.

rocminfo:
  Name:           gfx936
  Marketing Name: BW
  Device Type:    HCU
  Compute Unit:   80

3.2 HIP initialization before adaptation

./hashcat -II --backend-ignore-cuda --backend-ignore-opencl

hashcat (v7.1.2) starting in backend information mode
hipDeviceGetAttribute(): invalid argument
No devices found/left.

OpenCL detects the BW device. This confirms that the hardware, job allocation, driver, and device permissions are working, narrowing the investigation to Hashcat's host-side HIP API compatibility layer.

3.3 HIP ABI mismatch

To preserve runtime loading, Hashcat declares the HIP interfaces it needs in include/ext_hip.h instead of depending on the system HIP headers at build time. Hashcat 7.1.2's bundled declarations follow a newer ROCm ABI, while DTK 25.04.2's hip_runtime_defines.h uses a different, older layout.

A separate probe confirms that DTK reports sizeof(hipDeviceProp_t) = 792. Key field offsets are totalGlobalMem=256, regsPerBlock=272, warpSize=276, clockRate=308, major=328, multiProcessorCount=336, and gcnArchName=396. These offsets differ substantially from Hashcat's bundled structure definition.

4. Code changes

The adaptation is opt-in, so the default build remains unchanged. Changes are limited to the build switch, HIP type definitions, device-attribute initialization, dynamic-library lifetime, and adaptation notes.

DTK_HIP ?= 0

ifeq ($(DTK_HIP),1)
CFLAGS += -DHC_HIP_DTK
endif

#if defined (HC_HIP_DTK)
device_param->regsPerMultiprocessor = prop.regsPerBlock;
#else
device_param->regsPerMultiprocessor = prop.regsPerMultiprocessor;
#endif

hipDeviceGetAttribute(dev=0, attrib=69): invalid argument

5. Fault localization and debugging

1. Confirmed binary contains HIP:Checked hipInit、hipDeviceGetAttribute、hiprtcCompileProgram , and --backend-ignore-hip parameters.
2. Eliminate misjudgment of no-card nodes:The login node is missing /dev/kfd; After switching to the assigned computing node, the DCU and driver are normal.
3. Establish an OpenCL baseline:OpenCL detects the BW device and completes the MD5 benchmark, confirming that the hardware and Hashcat core are functioning.
4. Enhanced logging post-lock enumeration:First failed attribute changes from opaque error to attrib=69, and then use the DTK header file to compile the detection program to check the value and structure offset item by item.
5. First ABI fix:After aligning the property enum and structure layout, device enumeration advanced to the register-property query. Runtime testing confirmed that DTK does not support attribute 69.
6. Register-property fallback: A probe reported prop.regsPerBlock=196608, while attribute 69 returned an error. The adaptation uses this value as a conservative fallback.
7. HIP device enumeration passed:Hashcat reports HIP 6.3.25422, BW, 80 MCUs, 65,520 MB, and the PCI address.
8. HIPRTC Verification:MD5 and NTLM benchmark succeeded, proving that gfx936 .
9. found exit phase SIGSEGV:The calculation result has been output, but the exit code is 139. Use GDB to locate atexit_thread.cc:75, the top of the stack is the unsigned unloaded address.
10. Fix DSO lifecycle:DTK branch retains HIP/HIPRTC DSO until the process exits, and then the benchmark exit code returns to 0.
11. Correctness verification: An actual MD5 dictionary attack passed the self-test and recovered e11c594e6a2f4eb499cceadfca988595:13LEXON.

5.1 Key GDB backtrace

Thread 1 "hashcat-hip-dtk" received signal SIGSEGV
0x000015552f2620e0 in ?? ()
#0  0x000015552f2620e0 in ?? ()
#1  (anonymous namespace)::run
    at libstdc++-v3/libsupc++/atexit_thread.cc:75
#2  __run_exit_handlers () from /lib64/libc.so.6
#3  exit () from /lib64/libc.so.6

The traceback indicates that the dynamic library had been unloaded while a function pointer was still retained during processor shutdown. After correcting the DSO lifetime, the same test exited normally, confirming this diagnosis.

6. Git and compilation process

6.1 Initialize the Git baseline

To ensure that the adaptation process is traceable, the source code baseline can be established first, and then the DTK HIP adaptation changes can be recorded in an independent branch. The following commands show a reference flow.

cd /path/to/hashcat
git init -b dtk-hip-adaptation
git add -A
git commit -m "baseline: import hashcat 7.1.2 source tree"

# After making and verifying the changes
git add .gitignore include/ext_hip.h src/Makefile \
  src/backend.c src/ext_hip.c src/ext_hiprtc.c \
  docs/dtk-hip-adaptation.md
git commit -m "feat: adapt HIP backend for DTK 25.04.2"

6.2 Compilation steps

cd /path/to/hashcat

# Preserve the original binary
cp -p hashcat hashcat-opencl-original

# Clean and build the DTK-specific version
make clean
make DTK_HIP=1 ENABLE_LTO=0
mv hashcat hashcat-hip-dtk

# Basic checks
./hashcat-hip-dtk --version
sha256sum hashcat-hip-dtk hashcat-opencl-original

ENABLE_LTO=0 reduces link complexity during adaptation debugging and keeps builds predictable. The DTK-specific code is enabled with DTK_HIP=1.

6.3 DTK 26.04 toolchain and scheduling notes

On the migration platform, the newer GCC module conflicts with the system toolchain. System GCC 4.8.5 also lacks the AVX512F intrinsics required by the bridges module. The tested build uses DTK's dcc (Clang 17) for source files and system GCC for final archive linking. The bridges module is not built; it is outside the core hashing functionality evaluated here.

6.4 Verification commands

ssh <compute-node>
cd /path/to/hashcat

# Enable HIP only
./hashcat-hip-dtk -II \
  --backend-ignore-cuda \
  --backend-ignore-opencl

# Enable OpenCL only
./hashcat-opencl-original -II \
  --backend-ignore-cuda \
  --backend-ignore-hip

7. Cross-version ABI and platform migration verification

The ABI probe was rebuilt against the DTK 26.04 headers and compared item by item with the DTK 25.04.2 results. All 12 device-attribute enum values used by this adaptation match: 18, 25, 26, 27, 29, 32, 36, 37, 39, 40, 42, and 79.

8. Functional and correctness verification

./hashcat-hip-dtk -m 0 -a 0 example0.hash example.dict \
  --backend-ignore-cuda --backend-ignore-opencl \
  --potfile-disable --restore-disable

Recovered:
e11c594e6a2f4eb499cceadfca988595:13LEXON

The final status code for this dictionary test is 1: the input contains 6494 hashes, 1 of which is restored when the small dictionary is exhausted. Both self-test and target recovery were successful, so this status code reflects task completion conditions rather than a HIP backend exception.

8.1 gfx906/Z100 device verification

HIP.Version.: 6.3.26113

Backend Device ID #01
  Name...........: Device 66a1
  Processor(s)...: 64
  Preferred.Thrd.: 64
  Clock..........: 1700
  Memory.Total...: 16368 MB
  Memory.Free....: 16101 MB
  Memory.Unified.: 0
  Local.Memory...: 64 KB
  PCI.Addr.BDFe..: <redacted>

Device enumeration, HIPRTC kernel compilation, attribute queries, multi-algorithm benchmarking, and exit-code checks all passed. Environment-specific identifiers, including PCI addresses, are redacted.

9. Performance verification and comparison

9.1 Same-card backend comparison on gfx936

To avoid short runs masking differences in sustained performance, both backends ran the same SHA2-256 brute-force workload over the full printable 8-bit character set, with a key space of 95^8 = 6,634,204,312,890,625. Unlikely target hashes kept the DCU fully loaded. Each run lasted about 45 seconds, while hy-smi sampled power, temperature, and utilization every two seconds.

# HIP
./hashcat-hip-dtk -m 1400 -a 3 \
  ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff \
  '?a?a?a?a?a?a?a?a' -O -w 3 --runtime 45 \
  --status --status-timer 5 --potfile-disable --restore-disable \
  --backend-ignore-cuda --backend-ignore-opencl

# OpenCL: same attack parameters; only the backend and binary change
./hashcat-opencl-original -m 1400 -a 3 \
  ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff \
  '?a?a?a?a?a?a?a?a' -O -w 3 --runtime 45 \
  --status --status-timer 5 --potfile-disable --restore-disable \
  --backend-ignore-cuda --backend-ignore-hip

9.2 gfx936 core results

9.2.1 Throughput comparison

9.3 Automatic tuning and resource differences

The two backends selected different workgroups and acceleration combinations, but the final throughput was consistent, indicating that both paths can fully utilize the DCU. HIP's significantly higher host memory footprint was the clearest resource difference in this test.

9.4 Comparing cumulative work counters

HIP completed 266,757,734,400 operations and OpenCL completed 270,092,206,080. Although the OpenCL total is about 1.25% higher, its final status sample occurred at roughly 46 seconds, versus 45 seconds for HIP; initialization, status refresh, and stop times were not identical. The comparison therefore uses Hashcat's ten continuous throughput samples and draws no performance conclusion from cumulative work totals.

9.5 gfx906/Z100 multi-algorithm HIP benchmark

This migration platform provides HIP but not OpenCL. These results verify algorithm coverage and platform stability; they do not support a comparison with OpenCL on this device.

Scheduling parameters are listed as Accel / Loops / Threads / Vector width.

9.6 Sustained load and cross-device comparison

This set of ratios is an actual observation of two complete machine environments, including multiple differences in computing units, architectures, memory subsystems, software versions, and tuning parameters. It cannot be derived solely from the 64/80 computing unit ratio, nor does it represent a fixed ratio for all hash-modes.

10. Conclusions, recommendations and known limitations

10.4 Known limitations

• The adaptation was verified with DTK 25.04.2 / HIP 6.3.25422 and DTK 26.04 / HIP 6.3.26113. Recheck enum values, structure size, and field offsets with future versions.
• gfx936 covers actual attacks and HIP/OpenCL comparison; gfx906 covers 10 algorithms, but lacks OpenCL comparison with the same card.
• regsPerMultiprocessor uses conservative compatibility values; the current algorithm works fine but may affect the tuning upper limit of some cores.
• The DTK build intentionally keeps the HIP/HIPRTC shared libraries loaded instead of calling dlclose. The operating system reclaims them when the process exits; this differs from standard Hashcat library-unloading behavior.
• Runs of 45–60 seconds provide a stable throughput comparison but do not replace reliability testing over hours or days.
• The old GCC migration environment does not build the bridges module, and batch jobs are also affected by the site scheduling policy; neither should be extrapolated to hashcat core limitations.

10.5 Further work

1. Extend testing to common workloads: -m 1000, -m 10900, -m 22000, and -m 3200.
2. Run HIP and OpenCL stability tests for one to four hours while monitoring temperature, power, driver logs, and error counts.
3. Add runtime probes or compile-time assertions for the DTK ABI to catch layout changes after upgrades.
4. To pursue further speedups, investigate HIPRTC compilation parameters, register occupancy, wavefront behavior, and workload-specific tuning.

11. Appendix: Reproduction and Audit Commands

11.1 View code changes

cd /path/to/hashcat
git status
git log --oneline --decorate -5
git show --stat <adaptation-commit>
git show <adaptation-commit>

11.2 Rebuild

make clean
make DTK_HIP=1 ENABLE_LTO=0
mv hashcat hashcat-hip-dtk

11.3 Basic verification

ssh <compute-node>
cd /path/to/hashcat

./hashcat-hip-dtk -II --backend-ignore-cuda --backend-ignore-opencl
./hashcat-hip-dtk -b -m 0 --backend-ignore-cuda --backend-ignore-opencl --runtime 10
./hashcat-hip-dtk -b -m 1000 --backend-ignore-cuda --backend-ignore-opencl --runtime 10

11.4 Key file index