RISC-V Certification Test Plan

A RISC-V Certificate is a named set of requirements for a RISC-V implementation, typically defined by a ratified profile and a set of additional requirements. For example, the RVA23 Certificate requires meeting the RVA23S64 profile and having a conforming machine mode implementation.

Certification is the process of testing that a RISC-V implementation meets the requirements of a RISC-V certificate. The goal of certification is to deliver software interoperability across implementations.

There is a fuzzy line between certification and verification. Certification involves testing that an implementation meets all of the normative rules in ratified specifications relevant to the certificate, but does not involve testing that requires detailed knowledge of the microarchitecture. Specifically, certification does not check timing (such as the arrival of interrupts relative to the start, middle, or end of multicycle instructions), pipeline hazards, power management features, etc.

Certification assumes the ratified specification is read and interpreted with knowledge of computer architecture and reasonable judgement, because not every detail is spelled out in normative rules. For example, certification that "ADD performs the addition of rs1 and rs2" involves testing ADD with a reasonable set of values. The test plan author exercises judgement selecting these values because they are not spelled out. For example, the values might involve every combination of {0, 1, -1, maximum positive integer, maximum negative integer, intermediate value} for the two source register values. The intermediate value involves an arbitrary choice. A reasonable author might pick some other sensible edge values, such as some values one more or less than those in the set above. However, certification does not attempt to prove every bit of a carry chain works for every possible input, or that all stuck-at faults would be detected. Similarly, the test plan author may exercise all 32 registers as sources or destinations for each instruction, which is implicitly required but not spelled out in a normative rule.

When in doubt, it is better to test slightly more than strictly necessary for normative rules, rather than slightly less. Over-testing is acceptable, as long as it does not cause false failures or excessive test time. It is often easier to test cross-products of all relevant inputs, even if not all combinations are explicitly named by normative rules. It also reduces the risk of human error missing a required combination.

An implementation that passes certification tests is said to be certified for the relevant certificate. The process of certification is described in TODO CSC page.

Normative rules are the key to certification. They are the features of the RISC-V architecture that need to be tested in order to certify a profile. However, normative rules are human-readable portions of a specification and cannot be checked in an automated manner.

Instead, certification relies on coverpoints, which are machine-readable descriptions of the features to be tested. Every normative rule is tied to one or more coverpoints, such that if the coverpoints are hit, the normative rule is considered to be certified.

Each coverpoint has a set of tests that exercise it, and each test has a set of coverpoints that it hits.

This test plan defines the normative rules that must be hit for each profile, specifies the coverpoints that measure those rules, and outlines the tests that hit those coverpoints. The tests are self-checking, and report pass/fail results. The coverage report confirms that the tests hit all the coverpoints and thus all the normative rules.

Coverpoints are written in SystemVerilog, and are designed to be used with the Extended RISC-V Verification Interface (RVVI). They are written to be independent of the DUT (Device Under Test), so that they can be used with any RISC-V implementation. ^[1] Functional coverage is collected using SystemVerilog infrastructure adapted from riscvISACOV ^[2].

Coverpoints are organized into .svh SystemVerilog coverage files, each of which contain one or more covergroups, each of which contain one or more coverpoints. Coverage files apply to a DUT with a particular set of extensions. For example, the I_coverage.svh coverage file is used for any 32 or 64-bit DUT that supports the I extension. The ZicsrF_coverage.svh coverage file applies to a DUT that supports both the Zicsr and F extensions. Some covergroups in a coverage file apply only when another extension is also supported; for example, the half-double conversion covergroups in Zfh_coverage.svh are compiled only when D is supported.

Tests are written in assembly language .S files. Tests are organized into directories for each extension, and into a privileged directory. The coverpoints are hit by running all of the applicable tests in a directory. For example, RV32I coverage involves running add.S, addi.S, sub.S, etc. RV64I coverage also involves running addw.S, which is unique to RV64.

The following terms are used in this test plan:

Table 1 summarizes the coverage files applicable to each profile. x indicates a mandatory extension, and o indicates an optional extension. The certification test plan handles all mandatory and optional extensions.

Coverage files with multiple extensions in the name apply when all the extensions are supported; for example, ZicsrF is used in a system supporting the Zicsr and F extensions.

All privileged suites require at least version 1.12 of the privileged specification. Features specific to version 1.13 are tested separately in the Sm1p13 and Ss1p13 suites.

Table 2 lists certain extensions whose behavior is presently outside the scope of certification. An x in the table means the extension is required for that profile. Some of these behaviors are not readily visible at the architectural level of observed program behavior, such as constant-time instructions. Measuring the latency of an instruction with a cycle counter is unreliable because instruction fetch, branch prediction, superscalar issue, and out-of-order execution make the count depend on where the instruction is and what surrounds it, not only on its operand data. PMA properties are implicitly tested by executing instructions that depend on the property, but not tested comprehensively across the entire memory map. Entropy generation requires statistical analysis beyond the present scope of certification. These extensions might be added to scope in the future.

Furthermore, the following features are not supported in this version of the certification test plan:

A hart with these features can still be certified if it meets all other requirements; these features are simply not tested.

The DUT is described with a Unified Database (UDB) YAML file. This file lists:

The DUT also needs to know how to boot the core, terminate a test with a success code, send a message to a console, and cause interrupts. These DUT-specific behaviors are defined by RVMODEL DUT-Specific Macros, which are an assembly-language library implementing an API using DUT-specific hardware.

Unified Database allows partially- and fully-configured descriptions. A profile is an example of a partially configured description because it has a mix of required and optional extensions, and does not specify parameter values for most of the extensions. A DUT needs to have a fully configured description including precisely which versions of which extensions are supported and what parameter values are implemented, so that a simulator can exactly match its behavior.

The certification framework requires a fully-configured UDB description of the DUT and RVMODEL macros for the DUT. It uses UDB to produce supplementary configuration files including:

Some RISC-V behaviors are UNSPECIFIED, which means an implementation can do anything. These behaviors are not tested because there is no wrong answer. Table 3 summarizes the normative rules regarding reserved behaviors.

Unimplemented opcodes (except in the custom space) and instructions accessing non-existent CSRs are reserved. The behavior of reserved instructions is UNSPECIFIED, so they should not be tested by most testplans. However, when the Ssstrict extension is enabled, reserved instructions raise an illegal instruction exception. This is tested by the Ssstrict_S test plan.

TODO: Generate this table from norm rules via YAML when Ssstrict is tagged. Expand on definition of Ssstrict in light of github.com/riscv/riscv-isa-manual/issues/3258.

CSR fields can be Write Preserves, Read Ignores (WPRI), Write Legal, Read Legal (WLRL), or Write Any, Read Legal (WARL). Examples of WPRI fields include undefined fields of xstatus, xenvcfg, mseccfg, and xstateen* that could have unintended side effects if they are subsequently defined and unwittingly set to nonzero values. Examples of WLRL fields include xcause.ExceptionCode and hstatus.VGEIN. Other CSR fields are generally WARL.

WPRI fields should be read-only zero if not furnished by the implementation, but are not tested because the test suite may be unaware of subsequently ratified extensions that furnish them. WLRL fields are tested with legal values but no illegal values. Similarly, WARL fields are tested with legal values. The legalization of illegal values is implementation-specific and can be difficult to convey to a reference model for multibit fields, so WARL fields are not tested with illegal values unless it is easy to determine their legality (e.g. 1-bit fields). Reserved values are never tested in CSRs because the behavior is implementation-defined.

Instructions and CSRs in the custom encoding space are never tested because the behavior is implementation-defined.

Table 4 summarizes the normative rules regarding CSR bitfields.

TODO: replace with references to normative rules tagged in spec.

The testplan for each extension contains a list of normative rules applicable to that extension. Normative rules are generally direct quotations from a ratified RISC-V specification describing a single certifiable feature. Normative rules specify the behaviors an implementation must meet in order to be compliant with the standard. In particular, introductory overview and non-normative explanatory text is not quoted as a normative rule.

See tagging_normative_rules for more information about normative rules and IDs and how to tag them in the specification.

Coverpoints are written with one file that covers both RV32 and RV64, to reduce the duplication and risk of becoming out of sync. When a coverage file contains coverpoints that apply only to one XLEN or the other (e.g. 32 or 64-bit edge values), they are separated by `ifdef UDB_MXLEN_32 or UDB_MXLEN_64 directives. Similarly, when a coverpoint applies only to a certain parameter value (PMP NA4 regions are not supported for granularity coarser than 4 bytes), they uses `ifdef to exclude tests based on the parameter value.

Not all normative rules have coverpoints and tests. A normative rule that a behavior is UNSPECIFIED is not tested because there is no wrong answer. For example, writing satp.MODE = Bare with nonzero values in other fields has unspecified behavior and is untested. The CTP may declare that other normative rules are too difficult to test relative to their value (low return on investment) and choose not to test them. For example, if the Zcf extension is supported, misa.C depends on whether misa.F is supported. misa.F is WARL and may be read-only one, so it is not always possible to test the misa.C dependence on misa.F = 0 and, for simplicity, this dependence might never be tested.

To the extent that there are any normative rules associated with instruction opcodes or CSR numbers, if these rules can be tested by coverpoints that execute the instruction or access the CSR, then no other coverpoints are needed. For example, a normative rule that an add instruction has op = 0110011, funct3 = 000, and funct7 = 0000000 is tested by the ordinary coverpoints that exercise the add instruction.

Privileged tests are mostly generated by scripts. They preferably share a single .S file that can be compiled for either RV32 or RV64, again with ifdef directives to separate the two. Unprivileged tests are generated from a template using a Python script, and are divided into RV32 and RV64 directories because the random values differ with XLEN.

Every DUT has S_SUPPORTED, U_SUPPORTED, and H_SUPPORTED parameters from the UDB that indicate which privilege modes are supported. All DUT UDB configs support Sm, but not necessarily standard M-mode (and possibly no CSRs at all). Therefore, the DUT’s rvmodel_macros.h defines STANDARD_SM_SUPPORTED when the DUT implements mandatory portions of the M-mode spec, and leaves it undefined otherwise.

A DUT with no CSRs at all, such as an RV32I microcontroller, still declares Sm in its UDB config, because UDB defines MXLEN only when Sm is implemented, but does not declare Zicsr. It leaves STANDARD_SM_SUPPORTED undefined, so the boot code and trap handler touch no CSRs, and sets include_priv_tests: False in its test_config.yaml, so the privileged tests that the Sm declaration would otherwise select are skipped. config/cores/picorv32/picorv32-rv32imc is an example.

The user is responsible for loading the test ELF onto the DUT and jumping to rvtest_entry_point. The user may optionally run some other firmware or custom boot loader that performs DUT-specific initialization before loading the test ELF.

The test ELF begins with the RVTEST_BEGIN macro, which defines rvtest_entry_point (at an address specified in the link.ld file), and jumps to rvmodel_boot.

rvmodel_boot in turn calls the RVMODEL_BOOT macro, which optionally contains DUT-specific boot code such as turning on a PLL or memory controller. This macro can be empty or undefined in the common case that no DUT-specific boot code is required or that the DUT-specific boot is performed earlier by a separate bootloader.

Next, rvmodel_boot calls RVMODEL_IO_INIT to perform any necessary I/O initialization, such as configuring baud rate and flow control in a UART. This macro can also be empty or undefined if no I/O initialization is needed.

Next, rvmodel_boot calls RVTEST_BOOT_TO_MMODE to set up the trap handlers and initialize M-mode CSRs. This macro first checks if RVMODEL_BOOT_TO_MMODE is defined. If undefined, it runs default boot code assuming standard M-mode. If defined, it runs the DUT-specific RVMODEL_BOOT_TO_MMODE that performs the same general steps for custom M-mode. The default boot code touches CSRs only when STANDARD_SM_SUPPORTED is defined, so a DUT with no M-mode CSRs, such as an RV32I DUT, leaves both STANDARD_SM_SUPPORTED and RVMODEL_BOOT_TO_MMODE undefined.

CSR initialization includes

Even if standard M-mode is not supported or RVMODEL_BOOT_TO_MMODE is defined, the boot code still attempts to turn on mstatus.FS/VS if float or vector is supported, because these are needed for unprivileged tests, even on a RV32I DUT that has no other M-mode features.

The boot process ends in M-mode if the BOOT_TO_MMODE symbol is defined in the test (for tests intended to run in machine mode), or if no lower privilege mode is supported.

If the test is continuing to boot to a lower privilege mode, it next checks if S_SUPPORTED and, if so, calls RVTEST_BOOT_TO_SMODE. This initializes S-mode CSRs, and adjusts M-mode CSRs such as trap delegation and environment configuration. It then enters S-mode. A DUT with a nonstandard M-mode replaces the reference T-SEE rather than this routine.

S-mode CSR initialization includes

The boot process ends in S-mode if the BOOT_TO_SMODE symbol is defined in the test (for tests intended to run in supervisor mode).

If the test is not intended to run in S-mode, and U-mode is supported, rvmodel_boot calls RVTEST_BOOT_TO_UMODE, which switches to U-mode through T-SBI.

Finally, rvmodel_boot clears the trap counter and runs RVTEST_INIT_REGS, which initializes the integer, floating-point and vector registers, then sets up the signature pointer and checks a canary if applicable. It then jumps to rvtest_code_begin to start the test.

Each DUT requires implementing a set of macros to perform the following non-ISA functions:

The macro names begin with RVMODEL, so they are called RVMODEL DUT-Specific Macros. Their functions are sometimes called semihosting or trick box.

The RVMODEL DUT-Specific Macros rely on the capabilities of the SoC and/or testbench. For example, a SoC with a 16550-compatible UART sends a character by writing a byte to the UART’s memory-mapped Transmit Holding Register. A SoC without a UART may have a testbench that monitors writes to a specific memory address and prints the byte written to that address.

Each DUT needs a linker script tailored to its memory map. The linker script in config/<core>/<version>/link.ld defines where to place the code and data regions in the DUT’s memory map. It must contain .text.init, .text.rvtest, .data, and .text.rvmodel sections (in that order), and an ENTRY point named rvtest_entry_point. The first line of the list of SECTIONS in the linker script sets the starting address of the ELF (. = 0x…​) and is the only part of the sample linker script that most users will need to change. Set it to your reset address for simple DUTs. Users with special needs can customize the linker script to start at their own boot code (leaving the RVMODEL_BOOT macro blank) and then jump to rvtest_entry_point.

The .text.rvmodel section contains DUT-specific (RVMODEL) code and must be placed after the .data section. This ensures that variable-size model-specific macro expansions (which may differ between the DUT build and the reference-model build) do not affect the addresses of test data symbols such as scratch and begin_signature, preventing signature mismatches.

On harts that support virtual memory, the test image must be placed where the virtual memory tests leave it mapped. 0x8000_0000, used by the sample configurations, works for both RV32 and RV64. The boot code identity-maps the superpage that contains the test data (a 4 MiB megapage on RV32, a 1 GiB gigapage on RV64), so the whole image must fit inside one aligned superpage. The tests also map fixed virtual addresses, such as 0, all ones, and aliases of the code and data. If one of these falls in the same superpage as the image, the test overwrites the image’s identity mapping and hangs. On RV64, the image must also lie in the Sv39 address space, below 2^38 (256 GiB), even on harts that support Sv48 or Sv57, because the Sv39 tests run on those harts too. The safe ranges are:

For some systems, this may involve configuring a non-default physical address and size for main memory.

Standard macros are defined in tests/env. DUT-specific macros are defined in rvmodel_macros.h, which is in a DUT-specific config directory (e.g. config/<dut>/<version>/rvmodel_macros.h for predefined simulators, or out-of-tree for custom DUTs).

Standard framework macros begin with RVTEST. They include rvtest_setup.h for boot macros and rvtest_trap_handler.h for standard trap handlers, and rvtest_failure_code.h for error reporting. Other macros are defined in ***util.h, …​ describe further

DUT-specific macros begin with RVMODEL. Some RVTEST macros check if corresponding RVMODEL macros are defined and invoke the RVMODEL macro instead for DUTs with custom M-mode, when defined.

The test must implement a custom Test Supervisor Binary Interface (T-SBI) providing an abstraction layer to facilitate S and U-mode tests requesting services from a possibly custom M-mode. These services include testing ECALL to the execution environment and accessing CSRs.

The ratified SBI compiles into a ~1MB OpenSBI firmware binary, which is unnecessarily large to load with every test. Moreover, the SBI calling convention uses a7 and a6, which are not available on RV32E. Therefore, T-SBI is a lighter-weight implementation with a custom calling convention that is incompatible with the ratified SBI.

The SBI is invoked with the ecall instruction. The operation is passed in a0. An additional argument could be passed in a1 (and possibly a2) and a result could be returned in a0 if needed. Other values of a0 are reserved.

For example, an S-mode program can request setting mstatus.TW with a CSR_SET call corresponding to csrrs x0, mstatus, a1 (a0 = 0x3005a073) with a1 containing a 1 in bit 21 (the TW bit) and all other bits 0. The ACT trap handler implements this using an instruction table. If a custom M-mode does not implement mstatus.TW, it must emulate the behavior in some other way. Similarly, S-mode can read mip to a0 with a CSR_READ call corresponding to csrrs a0, mip, x0 (a0 = 0x34402573).

The ACT trap handlers implement an additional set of convenience calls changing between privilege modes in a standard fashion. This includes changing to M-mode. The behavior in M-mode is only predictable if STANDARD_SM_SUPPORTED is true. Another suite with different trap handlers might not support these calls.

As compared to the base RVVI, Extended RVVI adds the following signals:

Some accesses touch more than one physical page. Examples include misaligned accesses wrapping a page boundary, and some vector accesses. Extended RVVI provides the signals corresponding to the base address of the access. For vector, they correspond to the base address of the initial element access.

NTRIGGERS is a new parameter similar to FLEN defining the number of Sdtrig debug triggers supported by the platform.

Coverpoints are described using the open riscvISACOV framework.

The framework collects data coming across the RVVI interface and presents it with ins.current and ins.prev data structures to allow easy access to the architectural state at the end of the current and previous instructions. For example, ins.current.rs1 is the source register for the current instruction, and ins.current.rd_val is the value written to the destination register by the current instruction.

The available fields are given in framework/src/act/fcov/coverage/RISCV_coverage_rvvi.svh. Most commonly used fields are summarized in Table 10. Other fields include floating-point and vector register IDs and values, MMU extended RVVI signals, vector type information, and pending interrupts. TODO: handling register pairs.

Move this and the remainder to ACT repo

Coverage is measured by running tests on the RISC-V Sail model and generating a RVVI-Trace file, then passing this file to an RVVI agent in a SystemVerilog testbench that checks coverage.

The RISC-V Sail model produces the trace file in a standard RVVI-Trace format (see Section Section 7.5). Any other simulator that implements the same trace format can also be used to cross-check coverage.

RISC-V feature behavior can be classified as deterministic, nondeterministic, and unspecified. Deterministic features have a unique expected value for a given configuration, such as the result of sub 1 - 2 on RV32. Nondeterministic features have a fine set of possible outcomes, and might vary unpredictably even on the same DUT. For example, vector tail-agnostic elements can preserve the input value or propagate all 1s, with no guarantee whether the same instruction has the same behavior on all elements or when executed twice in a row. Unspecified features, such as unimplemented instructions, could do anything, from being ignored, to predictably or unpredictably changing architectural state, to halting and catching fire.

All RISC-V simulators, when properly configured to match the behavior of a DUT, should produce the same answers as the DUT and each other for deterministic features. For nondeterministic features, they should produce one of the legal outcomes. For unspecified features, they can do anything.

Self-checking certification tests must check that the result of deterministic features match the reference model. Nondeterministic features must produce one of the finite set of legal outcomes, which must be provided by the test developer because a reference model only produces a single outcome. Unspecified features are not tested because there are no wrong answers.

This is harder than it seems because RISC-V has many features whose behaviors are not fully specified. Many of these features are not yet named, and the set of legal behaviors is not yet enumerated. Unified Database (UDB) is in the process of naming and enumerating all of these features and behaviors to fully describe the behavior of a DUT. Moreover, no simulator yet is fully configurable to match every feature. Some simulators, such as Spike, intentionally have a minimal set of configuration options and are hardwired to particular behaviors.

Nevertheless, the RISC-V Sail model is making rapid progress and is aiming to fully implement the ratified RVA23 profile, read a UDB configuration, and support at least the most reasonable behaviors in time for certification.

If nondeterministic behaviors are tested, the test must accept all legal outcomes. This is costly, so some nondeterministic behaviors may be untested. Nevertheless, agnostic vector operations will definitely be tested and accept all legal results.

When possible, the behavior should be made deterministic by a parameter such that the reference model can match the DUT with that parameter setting.

Every bin in every coverpoint is associated with a specific test ^[3]. Unprivileged tests sweep some feature under test (such as the destination register rd) while randomizing all applicable register IDs and source values (including values to be loaded from memory). For example, the unprivileged test for the cp_rd coverpoint of the add instruction contains 32 add instructions, using the 32 different choices of rd. The rs1 and rs2 register numbers and values are selected randomly. Appendix E gives examples of unprivileged coverpoints and tests meeting these requirements and the coverpoints of Table 18 for the add instruction. Each test is self-checking, checking the destination register value against one provided by a reference model. Certain tests also test other state such as stores or floating-point flags, as described in those sections.

Unprivileged tests are designed to never trap. Their results are independent of the privilege mode in which they are run, so they generally are run only in machine mode. They generally involve no privileged instructions, except that floating-point and vector tests turn on the mstatus.{FS/VS} bits to enable these extensions.

The coverpoints in Table 11 are used in most of the unprivileged test plans in subsequent sections.

Notes:

±2.5 and ±2^p (p is the format’s precision: 11 for half, 8 for BF16, 24 for float, 53 for double) are exact ties for some operations, so they distinguish RMM from RNE. Converting ±2.5 to an integer is a tie between 2 and 3. Adding ±1.0 to ±2^p is a tie between 2^p and 2^p+2.

Table 18 summarizes the coverpoints for the I extension. The Type column is used to generate tests with the appropriate operands. An x in the RV32 or RV64 column indicates that the instruction is supported in that XLEN. The remaining columns refer to coverpoints defined in Table 11. An x in a coverpoint column selects that coverpoint. Any other entry selects a modified coverpoint whose name is the column name followed by an underscore and the entry. For example, nx0 in the cp_rs1 column selects cp_rs1_nx0. Table 19 defines the modified coverpoints.