Skip to content

Tutorial

The following tutorial will guide you through the use of the Snitch cluster. You will learn how to develop, simulate, debug and benchmark software for the Snitch cluster architecture.

In the following tutorial you can assume the working directory to be target/snitch_cluster. All paths are to be assumed relative to this directory. Paths relative to the root of the repository are prefixed with a slash.

Building the hardware

To compile the hardware for simulation run one of the following commands, depending on the desired simulator:

# Verilator
make bin/snitch_cluster.vlt

# Questa
make DEBUG=ON bin/snitch_cluster.vsim

# VCS
make bin/snitch_cluster.vcs

These commands compile the RTL sources respectively in work-vlt, work-vsim and work-vcs. Additionally, common C++ testbench sources (e.g. the frontend server (fesvr)) are compiled under work. Each command will also generate a script or an executable (e.g. bin/snitch_cluster.vsim) which you can invoke to simulate the hardware. We will see how to do this in a later section. The variable DEBUG=ON is used to preserve the visibility of all the internal signals during simulation.

Building the Banshee simulator

Instead of running an RTL simulation, you can use our instruction-accurate simulator called banshee. To install the simulator, please follow the instructions of the Banshee repository: https://github.com/pulp-platform/banshee.

Cluster configuration

Note that the Snitch cluster RTL sources are partly automatically generated from a configuration file provided in .hjson format. Several RTL files are templated and use the .hjson configuration file to fill the template entries. An example is /hw/snitch_cluster/src/snitch_cluster_wrapper.sv.tpl.

Under the cfg folder, different configurations are provided. The cfg/default.hjson configuration instantiates 8 compute cores + 1 DMA core in the cluster. If you need a specific configuration you can create your own configuration file.

The command you executed previously automatically generated the templated RTL sources. It implicitly used the default configuration file. To override the default configuration file, define the following variable when you invoke make: 

make CFG_OVERRIDE=cfg/custom.hjson bin/snitch_cluster.vlt

Note: whenever you override the configuration file on the make command-line, the configuration will be stored in the cfg/lru.hjson file. Successive invocations of make will automatically pick up the cfg/lru.hjson file. You can therefore omit the CFG_OVERRIDE definition in successive commands unless you want to override the least-recently used configuration.

Banshee uses also a cluster configuration file, however, that is given directly when simulating a specific binary with banshee with the help of --configuration <cluster_config.yaml>.

Building the software

To build all of the software for the Snitch cluster, run the following command:

# for RTL simulation
make DEBUG=ON sw

# for Banshee simulation (requires slightly different runtime)
make SELECT_RUNTIME=banshee DEBUG=ON sw

# to use OpenOCD semi-hosting for putchar and termination
make DEBUG=ON OPENOCD_SEMIHOSTING=ON sw

The sw target first generates some C header files which depend on the hardware configuration. Hence, the need to generate the software for the same configuration as your hardware. Afterwards, it recursively invokes the make target in the sw subdirectory to build the apps/kernels which have been developed in that directory.

The DEBUG=ON flag is used to tell the compiler to produce debugging symbols. It is necessary for the annotate target, showcased in the Debugging section of this guide, to work.

The SELECT_RUNTIME flag is set by default to rtl. To build the software with the Banshee runtime, set the flag to banshee.

Note: the RTL is not the only source which is generated from the configuration file. The software stack also depends on the configuration file. Make sure you always build the software with the same configuration of the hardware you are going to run it on.

Note: on GVSOC, it is better to use OpenOCD semi-hosting to prevent putchar from disturbing the DRAMSys timing model.

Running a simulation

Run one of the executables which was compiled in the previous step on your Snitch cluster simulator of choice:

# Verilator
bin/snitch_cluster.vlt sw/apps/blas/axpy/build/axpy.elf

# Questa
bin/snitch_cluster.vsim sw/apps/blas/axpy/build/axpy.elf

# VCS
bin/snitch_cluster.vcs sw/apps/blas/axpy/build/axpy.elf

# Banshee
banshee --no-opt-llvm --no-opt-jit --configuration src/banshee.yaml --trace sw/apps/blas/axpy/build/axpy.elf

The Snitch cluster simulator binaries can be invoked from any directory, just adapt the relative paths in the preceding commands accordingly, or use absolute paths. We refer to the working directory where the simulation is launched as the simulation directory. Within it, you will find several log files produced by the RTL simulation.

The previous commands will launch the simulation on the console. QuestaSim simulations can also be launched with the QuestaSim GUI, by adapting the previous command to:

# Questa
bin/snitch_cluster.vsim.gui sw/apps/blas/axpy/build/axpy.elf

For Banshee, you need to give a specific cluster configuration to the simulator with the flag --configuration <cluster_config.yaml>. A default Snitch cluster configuration is given (src/banshee.yaml). The flag --trace enables the printing of the traces similar to the RTL simulation. For more information and debug options, please have a look at the Banshee repository: https://github.com/pulp-platform/banshee.

Creating your first Snitch app

In the following you will create your own AXPY kernel implementation as an example how to develop software for Snitch.

Writing the C Code

Create a directory for your AXPY kernel under sw/:

mkdir sw/apps/axpy

And a src subdirectory to host your source code:

mkdir sw/apps/axpy/src

Here, create a new file named axpy.c inside the src directory with the following contents:

#include "snrt.h"
#include "data.h"

// Define your kernel
void axpy(uint32_t l, double a, double *x, double *y, double *z) {
    for (uint32_t i = 0; i < l ; i++) {
        z[i] = a * x[i] + y[i];
    }
    snrt_fpu_fence();
}

int main() {
    // Read the mcycle CSR (this is our way to mark/delimit a specific code region for benchmarking)
    uint32_t start_cycle = snrt_mcycle();

    // DM core does not participate in the computation
    if(snrt_is_compute_core())
        axpy(L, a, x, y, z);

    // Read the mcycle CSR
    uint32_t end_cycle = snrt_mcycle();
}

The snrt.h file implements the snRuntime API, a library of convenience functions to program Snitch cluster based systems. These sources are located under target/snitch_cluster/sw/runtime/rtl and are automatically referenced by our compilation scripts.

Note: Have a look at the files inside sw/snRuntime in the root of this repository to see what kind of functionality the snRuntime API defines. Note this is only an API, with some base implementations. The Snitch cluster implementation of the snRuntime for RTL simulation can be found under target/snitch_cluster/sw/runtime/rtl. It is automatically built and linked with user applications thanks to our compilation scripts.

We will have to instead create the data.h file ourselves. Create a target/snitch_cluster/sw/apps/axpy/data folder to host the data for your kernel to operate on:

mkdir sw/apps/axpy/data

Here, create a C file named data.h with the following contents:

uint32_t L = 16;

double a = 2;

double x[16] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15};

double y[16] = {1, 1, 1, 1, 1, 1, 1, 1, 1, 1,  1,  1,  1,  1,  1,  1};

double z[16];

In this file we hardcode the data to be used by the kernel. This data will be loaded in memory together with your application code. In general, to verify your code you may want to randomly generate the above data. You may also want to test your kernel on different problem sizes, e.g. varying the length of the vectors, without having to manually rewrite the file. This can be achieved by generating the data header file with a Python script. You may have a look at the sw/blas/axpy/scripts/datagen.py script in the root of this repository as an example. As you can see, it reuses many convenience classes and functions for data generation from the data_utils module. Documentation for this module can be found here.

Compiling the C Code

In your axpy folder, create a new file named Makefile with the following contents:

APP     = axpy
SRCS    = src/axpy.c
INCDIRS = data

include ../common.mk

This Makefile will be invoked recursively by the top-level Makefile, compiling your source code into an executable with the name provided in the APP variable.

In order for the top-level Makefile to find your application, add your application's directory to the APPS variable in sw.mk:

APPS += sw/apps/axpy

Now you can recompile all software, including your newly added AXPY application:

make DEBUG=ON sw

Note, only the targets depending on the sources you have added/modified will be recompiled.

In the sw/apps/axpy/build directory, you will now find your axpy.elf executable and some other files which were automatically generated to aid debugging. Open axpy.dump and search for <x>, <y> and <z>. You will see the addresses where the respective vectors defined in data.h have been allocated by the compiler. This file can also be very useful to see what assembly instructions your source code was compiled to, and correlate the traces (we will later see) with the source code.

If you want to dig deeper into how our build system works and how these files were generated you can follow the recursive Makefile invocations starting from the sw target in snitch_cluster/Makefile.

Run your application

You can run your application in simulation as shown in the previous sections. Make sure to pick up the right binary, e.g.:

bin/snitch_cluster.vsim sw/apps/axpy/build/axpy.elf

Debugging and benchmarking

When you run the simulation, every core will log all the instructions it executes (along with additional information, such as the value of the registers before/after the instruction) in a trace file. The traces are located in the logs folder within the simulation directory. The traces are identified by their hart ID, that is a unique ID for every hardware thread (hart) in a RISC-V system (and since all our cores have a single thread that is a unique ID per core).

The simulation logs the traces in a non-human readable format with .dasm extension. To convert these to a human-readable form run:

make -j traces

If the simulation directory does not coincide with the current working directory, you will have to specify the path explicitly:

make -j traces SIM_DIR=<path_to_simulation_directory>

Detailed information on how to interpret the generated traces can be found here.

In addition to generating readable traces (.txt format), the above command also computes several performance metrics from the trace and appends them at the end of the trace. These can be collected into a single CSV file with the following target:

make logs/perf.csv
# View the CSV file
libreoffice logs/perf.csv

In this file you can find the X_tstart and X_tend metrics. These are the cycles in which a particular code region X starts and ends, and can hence be used to profile your code. Code regions are defined by calls to snrt_mcycle(). Every call to this function defines two code regions: - the code preceding the call, up to the previous snrt_mcycle() call or the start of the source file - the code following the call, up to the next snrt_mcycle() call or the end of the source file

The CSV file can be useful to automate collection and post-processing of benchmarking data.

Finally, debugging your program from the trace alone can be quite tedious and time-consuming. You would have to manually understand which instructions in the trace correspond to which lines in your source code. Surely, you can help yourself with the disassembly.

Alternatively, you can automatically annotate the traces with that information. With the following commands you can view the trace instructions side-by-side with the corresponding source code lines they were compiled from:

make -j annotate
kompare -o logs/trace_hart_00000.diff

If you prefer to view this information in a regular text editor (e.g. for search), you can open the logs/trace_hart_xxxxx.s files. Here, the annotations are interleaved with the trace rather than being presented side-by-side.

Note: the annotate target uses the addr2line binutil behind the scenes, which needs debugging symbols to correlate instruction addresses with originating source code lines. The DEBUG=ON flag you specified when building the software is used to tell the compiler to produce debugging symbols when compiling your code.

The traces contain a lot of information which we might not be interested at first. To simply visualize the runtime of the compute region in our code, first create a file named layout.csv in sw/apps/axpy with the following contents:

            , compute
"range(0,8)",       1
8           ,

Then run the following commands:

# Similar to logs/perf.csv but filters all but tstart and tend metrics
make logs/event.csv
# Labels, filters and reorders the event regions as specified by an application-specific layout file
../../util/trace/layout_events.py logs/event.csv sw/apps/axpy/layout.csv -o logs/trace.csv
# Creates a trace file which can be visualized with Chrome's TraceViewer
../../util/trace/eventvis.py -o logs/trace.json logs/trace.csv

Go to http://ui.perfetto.dev/. Here you can load the logs/trace.json file and graphically view the runtime of the compute region in your code. To learn more about the layout file syntax and what the Python scripts do you can have a look at the description comment at the start of the scripts themselves.

Great, but, have you noticed a problem?

Look into sw/apps/axpy/build/axpy.dump and search for the address of the output variable <z> :

Disassembly of section .bss:

80000960 <z>:
    ...

Now grep this address in your traces:

grep 80000960 logs/*.txt
...

It appears in every trace! All the cores issue a fsd (float store double) to this address. You are not parallelizing your kernel but executing it 8 times!

Modify sw/apps/axpy/src/axpy.c to truly parallelize your kernel:

#include "snrt.h"
#include "data.h"

// Define your kernel
void axpy(uint32_t l, double a, double *x, double *y, double *z) {
    int core_idx = snrt_cluster_core_idx();
    int offset = core_idx * l;

    for (int i = 0; i < l; i++) {
        z[offset] = a * x[offset] + y[offset];
        offset++;
    }
    snrt_fpu_fence();
}

int main() {
    // Read the mcycle CSR (this is our way to mark/delimit a specific code region for benchmarking)
    uint32_t start_cycle = snrt_mcycle();

    // DM core does not participate in the computation
    if(snrt_is_compute_core())
        axpy(L / snrt_cluster_compute_core_num(), a, x, y, z);

    // Read the mcycle CSR
    uint32_t end_cycle = snrt_mcycle();
}

Now re-run your kernel and compare the execution time of the compute region with the previous version.

Code Reuse

As you may have noticed, there is a good deal of code which is independent of the hardware platform we execute our AXPY kernel on. This is true for the data.h file and possible data generation scripts. The Snitch AXPY kernel itself is not specific to the Snitch cluster, but can be ported to any platform which provides an implementation of the snRuntime API. An example is Occamy, with its own testbench and SW development environment.

It is thus preferable to develop the data generation scripts and Snitch kernels in a shared location, from which multiple platforms can take and include the code. The sw directory in the root of this repository was created with this goal in mind. For the AXPY example, shared sources are hosted under the sw/blas/axpy directory. As an example of how these shared sources are used to build an AXPY application for a specific platform (in this case the standalone Snitch cluster) you can have a look at the target/snitch_cluster/sw/apps/blas/axpy.

We recommend that you follow this approach also in your own developments for as much of the code which can be reused.

Using Verilator with LLVM

LLVM+clang can be used to build the Verilator model. Optionally specify a path to the LLVM toolchain in CLANG_PATH and set VLT_USE_LLVM=ON. For the verilated model itself to be complied with LLVM, verilator must be built with LLVM (CC=clang CXX=clang++ ./configure). The VLT environment variable can then be used to point to the verilator binary.

# Optional: Specify which llvm to use
export CLANG_PATH=/path/to/llvm-12.0.1
# Optional: Point to a verilator binary compiled with LLVM
export VLT=/path/to/verilator-llvm/bin/verilator
make VLT_USE_LLVM=ON bin/snitch_cluster.vlt