Test Vectors

The input and golden tensors used by the tests have to be generated. A fresh clone therefore has no test data, and the build will stop with a message telling you to generate it.

Generate everything with:

python scripts/generate_test_vectors.py

Run it once after cloning, and again whenever a generator changes.

This includes the per-shape ITA_dims_*.h headers, which declare which shapes exist and drive the shape selection described below. Nothing under a test’s include/ or src_cluster/ is tracked once it is generated.

Test Vector Sources

The vectors come from two places:

Per-test generators. Most tests ship a generate_vectors.py next to their sources. These need only numpy, seed it, and write the same files every time. They also encode the kernel’s reference model, so they double as executable documentation of what the test checks.

ITA’s test-vector generator. The ITA_MatMul_L1, ITA_MatMul_L2 and ITA_SHA_L2 tests use data laid out in the tiled order the ITA accelerator consumes. That layout is produced by testGenerator.py in pulp-platform/ITA, which is also what feeds ITA’s own RTL testbench. scripts/generate_test_vectors.py clones ITA into a temporary directory, runs its generator for each required shape, and rewrites the per-shape ITA_mem_<shape>.c files.

If you already have a checkout, point at it to skip the clone:

python scripts/generate_test_vectors.py --ita-dir /path/to/ITA

To build only the tests that need no ITA checkout:

python scripts/generate_test_vectors.py --skip-ita

Both sources are deterministic: the per-test generators seed numpy, and ITA’s testGenerator.py seeds it too (--seed, default 0). Re-running produces byte-identical output.

Regenerating from the build

Tests whose generator is registered with CMake can also be refreshed from the build directory:

cmake --build <build-dir> --target regenerate-test-vectors

This covers the per-test generators only; the ITA data needs the script above.

Note

This target cannot bootstrap a fresh clone. Configuring stops as soon as it finds a missing vector, so the target does not exist yet at that point. Run scripts/generate_test_vectors.py first; the target is for refreshing vectors afterwards.

Selecting a shape

Tests that ship several problem sizes pick one at configure time rather than by commenting includes in and out:

cmake -DITA_MATMUL_L1_SHAPE=64x512x64 ...

Available shapes are discovered from the ITA_dims_*.h (or testinputs_*.h) headers a generator has written into the test, so adding a new shape is a matter of teaching the generator about it. An unknown shape, or one whose header and source disagree, triggers a configure-time error.

Adding a new generator

Write a generate_vectors.py in the test directory that takes --output-* arguments defaulting to the paths the test includes, then register it in the test’s CMakeLists.txt:

chimera_test_vector_generator(
  GENERATOR ${CMAKE_CURRENT_SOURCE_DIR}/generate_vectors.py
  OUTPUTS
            ${CMAKE_CURRENT_SOURCE_DIR}/include/testinputs_16x16x16.h
            ${CMAKE_CURRENT_SOURCE_DIR}/include/testoutputs_16x16x16.h
)

OUTPUTS lists the files the generator writes; they are checked at configure time and refreshed by the target above. Finally add the test directory to NUMPY_TESTS in scripts/generate_test_vectors.py and make sure the new outputs are covered by the .gitignore patterns.