Tuesday, January 15, 2019

Setting up AMD's ROCm From Source

Setting up a Local ROCm Build from Source

These instructions document how to compile and run ROCm from source on a Linux system running either an upstream kernel or a kernel built from the ROCK-Kernel-Driver repository. Users who simply wish to use ROCm for general-purpose GPU programming generally will find it much easier to install from the binary distribution, as detailed in the README. However, those who wish to modify or develop ROCm source code may find this guide useful.

Secondly, I realize that AMD does have a repository supposedly for people who want to set up ROCm from source, but the repository primarily contains a bunch of scripts. These instructions were written under the assumption that people exist who, like me, want to work with ROCm source code, don't work for AMD, and would rather understand what they're doing up front rather than having to dig through a ton of shell scripts.


This guide does not cover installing the ROCm kernel components. Compiling and installing the Linux kernel from source is a sufficiently involved topic and should be covered in other documentation. Instead, the following instructions require that you are already either running a sufficiently new Linux kernel version (i.e. 4.18 or later), or have installed the kernel from the ROCm kernel driver repository.

If you haven't yet added your username to the video group, do so first:

sudo usermod -a -G video $USER

Next, ensure that the video group has write access to /dev/kfd. We will do so by setting the group of the device to video, and ensuring that any member of the group has read and write access to the device:

echo 'KERNEL=="kfd", SUBSYSTEM=="kfd", GROUP="video", MODE="0660"' | sudo tee /etc/udev/rules.d/99-kfd.rules
sudo udevadm control --reload-rules
sudo udevadm trigger

Afterwards, ensure that your changes have taken effect by checking that the kfd virtual device has the correct permissions:

ls -lah /dev | grep kfd

# The above command should output something like:
# crw-rw----   1 root video   240,   0 Jan 15 10:04 kfd

As for software packages, you must first install some additional packages if you don't already have them. On Debian-based systems:

sudo apt install build-essential cmake g++ pkg-config libnuma-dev libpci-dev

Setting up directories

These instructions assume that you want to install ROCm to a directory without requiring root permissions. Start by creating a directory to hold all source code, and create a subdirectory into which we'll "install" ROCm components. Create environment variables holding the full paths of both directories:

mkdir rocm
cd rocm
mkdir install

The remainder of these instructions assume that $ROCM_DIR and $ROCM_INSTALL_DIR have been set as above.

Installing ROCT-Thunk-Interface

The ROCT-Thunk-Interface repository contains a simple C library wrapping ROCm system calls.

Start by cloning the repository if you haven't already:

git clone https://github.com/RadeonOpenCompute/ROCT-Thunk-Interface.git

Next, build the library in a "build" subdirectory:

cd $ROCM_DIR/ROCT-Thunk-Interface
mkdir build
cd build

Finally, install the library and headers to our target directory:

make install
make install-dev

Later on, if you modify ROCT-Thunk-Interface code, re-run the following commands to recompile and reinstall it:

cd $ROCM_DIR/ROCT-Thunk-Interface/build
make install
make install-dev

Installing ROCR-Runtime

ROCR-Runtime wraps the relatively minimal ROCT-Thunk-Interface library to provide more complex functionality, such as userspace memory management. This can only be compiled after you have completed the previous instructions for setting up ROCT-Thunk-Interface.

First, clone the repository if you haven't already:

git clone https://github.com/RadeonOpenCompute/ROCR-Runtime.git

Once again, use cmake to build in a separate build directory. We'll need to specify both where to install the runtime, and where we already installed the thunk interface (they're the same place, assuming you're following along):

cd $ROCM_DIR/ROCR-Runtime/src
mkdir build
cd build

Finally, install the library to our target directory.

make install

Later, if you modify ROCR-Runtime code, re-run the following commands to recompile and reinstall it:

cd $ROCM_DIR/ROCR-Runtime/src/build
make install

Installing the HCC Compiler

This step installs the HCC compiler and other tools such as OpenCL. It requires that you have already set up ROCR-Runtime.

First, clone the repository if you haven't already. Note that this repository is larger than the others due to it also requiring several submodules including LLVM and clang.

git clone --recursive -b clang_tot_upgrade https://github.com/RadeonOpenCompute/hcc.git

As before, build in a separate directory. Also specifiy where to install the binaries:

cd $ROCM_DIR/hcc
mkdir build
cd build
      -D CMAKE_BUILD_TYPE=Release \
make -j8

Finally, install HCC to the target directory:

# Run this with several threads to speed it up--there's a lot to install.
make -j8 install

Later, if you modify the HCC compiler's code, re-run the following commands to update your installation:

cd $ROCM_DIR/hcc/build
make -j8
make -j8 install

Installing HIP

This step installs the HIP compiler, AMD's CUDA-like language.

First, clone the repository if you haven't already:

git clone https://github.com/ROCm-Developer-Tools/HIP.git

As before, build in a separate directory, and specify the install location and the location where we installed hcc previously:

mkdir build
cd build
      -D CMAKE_BUILD_TYPE=Release \
make install

Later, if you modify the HIP source code, re-run the following commands to update your installation:

cd $ROCM_DIR/HIP/build

# Setting LD_LIBRARY_PATH may not be necessary here if you've already followed
# all of these instructions and have added the ROCm libraries to your library
# search path in a more permanent manner.
make install

Installing rocRAND

This may not be necessary for some people, but I was using it in some of my test programs.

First, clone rocRAND as you did the other projects:

git clone https://github.com/ROCmSoftwarePlatform/rocRAND.git

rocRAND has very annoying issues with their build scripts; namely hardcoded paths to /opt/rocm/... in several places that I couldn't figure out how to override using cmake options. So, just fix it with a hammer:

# Find a list of files with hardcoded paths
grep -r "opt/rocm" *

# Now, manually go through every file and replace the /opt/rocm/... paths with
# the location at which you've been installing ROCm.

Next, we'll build in a separate directory again.

mkdir build
cd build
make -j8
make install

These installed to an odd location, so I copied them to a location where they're more easily found by the local installation of hipcc:

cp -r hiprand/lib/* lib/
cp -r rocrand/lib/* lib/

cd ./include
ln -s -T ../hiprand/include ./hiprand
ln -s -T ../rocrand/include ./rocrand

Finally, the headers are wrong in several places (even if you installed the binary distribution of ROCm). If you try to use rocRAND, you'll encounter errors due to "missing" includes from the rocrand or hiprand headers. You'll usually just need to edit the headers and, for each wrong include path, add the directory containing the header to the #include <...> line. There's been an open PR for this issue on the rocRAND repository for a while, but they just don't seem to care about usability very much when it comes to this library.

Additional Steps to Update System Paths

If you plan to run hipcc, hcc, or other tools from your standard command line, you'll need to add your ROCM install location to your PATH. If you're using bash, this can be done by modifying .bashrc:

echo "export PATH=\$PATH:$ROCM_INSTALL_DIR/bin" >> ~/.bashrc
source ~/.bashrc

Some ROCm programs require an HCC_HOME environment variable. This is set to the installation directory of HCC, which was just $ROCM_INSTALL_DIR for us:

echo "export HCC_HOME=$ROCM_INSTALL_DIR" >> ~/.bashrc

Even after running the above step, you'll probably often encounter errors about missing runtime libraries. You can fix this in one of two ways:

  • Adding an entry to /etc/ld.so.conf.d/. If possible, you should start by trying to create a new file in the /etc/ld.so.conf.d/ directory, which contains lists of paths in which the system's dynamic linker searches:

    # Create a config file for most of the ROCm libraries
    echo "$ROCM_INSTALL_DIR/lib" | sudo tee /etc/ld.so.conf.d/ROCm.conf
    # For some reason, the HSA libraries are in a separate directory, so
    # append this line to the config file as well.
    echo "$ROCM_INSTALL_DIR/hsa/lib" | sudo tee -a /etc/ld.so.conf.d/ROCm.conf
    # Finally, update the settings for the dynamic linker:
    sudo ldconfig
  • Updating the LD_LIBRARY_PATH environment variable. This way is easy, but generally discouraged if you are able to add new library search paths in another way:

    echo "export LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:$ROCM_INSTALL_DIR/lib" >> ~/.bashrc
    echo "export LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:$ROCM_INSTALL_DIR/hsa/lib" >> ~/.bashrc
    source ~/.bashrc


For the most part, I attempted to keep invasive system changes to a minimum in these instructions. So, installing should be fairly straightforward:

  1. Delete the ROCM directory you created: rm $ROCM_DIR.

  2. If you updated your PATH environment variable, delete the corresponding lines from .bashrc or whichever configuration file you modified.

  3. If you added new entries to /etc/ld.so.conf.d/, remove them: sudo rm /etc/ld.so.conf.d/ROCm.conf, and run sudo ldconfig. If you instead modified LD_LIBRARY_PATH, delete the relevant lines from .bashrc or wherever you made the changes.


  1. fails at HIP
    localhost:~/rocm/HIP # cd build
    localhost:~/rocm/HIP/build # cmake -D CMAKE_INSTALL_PREFIX=$ROCM_INSTALL_DIR \
    > -D CMAKE_BUILD_TYPE=Release \
    > ..
    -- HIP Platform: hcc
    -- HIP Compiler: hcc
    -- Looking for HCC in: /root/rocm/install. Found version: 3.1.20044-1f5f61e7-8f5654ff00a
    -- Looking for HSA runtime in: /root/rocm/install/hsa
    -- HIP will be installed in: /root/rocm/install
    -- ROCM Installation path(ROCM_PATH): /opt/rocm
    CMake Warning (dev) in CMakeLists.txt:
    A logical block opening on the line

    /root/rocm/HIP/CMakeLists.txt:196 (if)

    closes on the line

    /root/rocm/HIP/CMakeLists.txt:198 (endif)

    with mis-matching arguments.
    This warning is for project developers. Use -Wno-dev to suppress it.

    -- Generating profiling promitives: /root/rocm/HIP/include/hip/hcc_detail/hip_prof_str.h
    /root/rocm/HIP/hip_prof_gen.py Warning: implementation not found: hipMemcpyToSymbolAsync
    /root/rocm/HIP/hip_prof_gen.py Warning: implementation not found: hipMemcpyFromSymbolAsync
    /root/rocm/HIP/hip_prof_gen.py Warning: implementation not found: hipLaunchCooperativeKernel
    /root/rocm/HIP/hip_prof_gen.py Warning: implementation not found: hipLaunchCooperativeKernelMultiDevice
    /root/rocm/HIP/hip_prof_gen.py Warning: implementation not found: hipMemcpyFromSymbol
    /root/rocm/HIP/hip_prof_gen.py Warning: implementation not found: __hipPushCallConfiguration
    /root/rocm/HIP/hip_prof_gen.py Warning: implementation not found: hipGetSymbolAddress
    /root/rocm/HIP/hip_prof_gen.py Warning: implementation not found: hipGetSymbolSize
    /root/rocm/HIP/hip_prof_gen.py Warning: implementation not found: hipExtLaunchMultiKernelMultiDevice
    /root/rocm/HIP/hip_prof_gen.py Warning: implementation not found: hipMemcpyToSymbol
    /root/rocm/HIP/hip_prof_gen.py Warning: implementation not found: __hipPopCallConfiguration
    /root/rocm/HIP/hip_prof_gen.py Warning: 11 API calls missing in interception layer
    CMake Error at CMakeLists.txt:350 (find_package):
    Could not find a package configuration file provided by "amd_comgr" with
    any of the following names:


    Add the installation prefix of "amd_comgr" to CMAKE_PREFIX_PATH or set
    "amd_comgr_DIR" to a directory containing one of the above files. If
    "amd_comgr" provides a separate development package or SDK, be sure it has
    been installed.

    -- Configuring incomplete, errors occurred!
    See also "/root/rocm/HIP/build/CMakeFiles/CMakeOutput.log".

  2. comgr is in https://github.com/RadeonOpenCompute/ROCm-CompilerSupport/tree/amd-stg-open/lib/comgr

    I've not yet figured out how to compile it...

  3. An example bash session to build Comgr on Linux using GNUMakefiles is:

    $ LLVM_PROJECT=~/llvm-project
    $ DEVICE_LIBS=~/device-libs
    $ COMGR=~/support/lib/comgr
    $ mkdir -p "$LLVM_PROJECT/build"
    $ cd "$LLVM_PROJECT/build"
    $ cmake \
    -DCMAKE_BUILD_TYPE=Release \
    -DLLVM_ENABLE_PROJECTS="llvm;clang;lld" \
    $ make
    $ mkdir -p "$DEVICE_LIBS/build"
    $ cd "$DEVICE_LIBS/build"
    $ cmake \
    -DCMAKE_BUILD_TYPE=Release \
    -DLLVM_DIR="$LLVM_PROJECT/build" \
    $ make
    $ mkdir -p "$COMGR/build"
    $ cd "$COMGR/build"
    $ cmake \
    -DCMAKE_BUILD_TYPE=Release \
    $ make
    $ make test

    1. i reckon the paths need updating to fit in...


The "Disparity Extender" Algorithm, and F1/Tenth

Introduction: Autonomous Racing Recently, my team from UNC-Chapel Hill won an F1/Tenth competition, held at CPSWeek 2019, in Montreal. ...