11. Testing¶

RTEMS developers run test executables when adding new features or testing a bug fix. All tests are run to make sure changes do not introduce regressions. Users can run the RTEMS tests to be certain the build of the kernel they have is functioning.

The section describes using and configuring the RTEMS Tester and RTEMS Run tools, the types of laboratory set ups supported and how to add your BSP to the framework. The tools command line interfaces are detailed in Chapter 13 Section 5 - RTEMS Tester and Run.

An RTEMS Test is an RTEMS executable where the application code is a test. Tests in RTEMS print banners to the console to indicate the configuration of the test and if it has start and finished.

The RTEMS Tools Project provides the RTEMS Tester and RTEMS Run tools. The RTEMS Tester command is rtems-test and the RTEMS Run command is rtems-run. These commands manage the complexity of running embedded executables. The commands provide a consistent command line interface to a testing framework that supports the various run time and testing scenarios we encounter such as simulators, GDB and executing directly on target hardware.

The RTEMS kernel code contains an extensive set of tests to exercise and test the RTEMS kernel. The tests check functionality, provide coverage testing and make sure the kernel is operating as intended on your target system. The testsuite has support to make creating a test simple and uniform.

The tests are built by adding --enable-tests to the RTEMS build configuration command line. There are over 600 tests and building them does extend the RTEMS kernel’s build time and use more disk space but it worth building and running them. The RTEMS test executables have the .exe file extension.

11.1. Test Banners¶

All test output banners or strings are embedded in each test and the test outputs them to the BSP’s console as it executes. The RTEMS Tester captures the BSP’s console and uses this information to manage the state of the executing test. The banner strings are:

*** BEGIN TEST <name> ***: The test has loaded, RTEMS has initialized and the test specific code is about to start executing. The <name> field is the name of the test. The test name is internal to the test and may not match the name of the executable. The test name is informative and not used by the RTEMS Tester.

*** END TEST <name> ***: The test has finished without error and has passed. The <name> field is the name of the test. See the Test Begin Banner for details about the name.

*** TEST VERSION: <version>: The test prints the RTEMS version return by the RTEMS Version API as <version>. All tests must match the first test’s version or the Wrong Version error count is incremented.

*** TEST STATE: <state>: The test is tagged in the RTEMS sources with a special <state> for this BSP. See Test States for the list of possible states. The state banner lets the RTEMS Tester categorize and manage the test. For example a user input test typically needing user interaction may never complete producing an invalid test result. A user input test is terminated to avoid extended delays in a long test run.

*** TEST BUILD: <build>: The test prints the RTEMS build as a space separated series of labels as <build>. The build labels are created from the configuration settings in the Super Score header file rtems/score/cputops.h. All tests must match the first test’s build or the Wrong Build error count is incremented.

*** TEST TOOLS: <version>: The test prints the RTEMS tools version returned the GGC internal macro _VERSION_ as <version>. All tests must match the first test’s tools version string or the Wrong Tools error count is incremented.

11.2. Test States¶

The tests states are:

passed: The test start and end banners have been sent to the console.

failure: The test start banner has been sent to the console and no end banner has been seen when a target restart is detected.

excepted-fail: The test is tagged as expected-fail in the RTEMS sources for this BSP and outputs the banner *** TEST STATE: EXPECTED_FAIL. The test is known not to pass on this BSP. The RTEMS Tester will let the test run as far as it can and if the test passes it is recorded as a pass in the test results otherwise it is recorded as expected-fail.

indeterminate: The test is tagged as indeterminate in the RTEMS sources for this BSP and outputs the banner *** TEST STATE: INDETERMINATE. The test may or may not pass so the result is not able to be determined. The RTEMS Tester will let the test run as far as it can and record the result as indeterminate.

user-input: The test is tagged as user-input in the RTEMS sources and outputs the banner *** TEST STATE: USER_INPUT. The RTEMS Tester will reset the target if the target’s configuration provides a target reset command.

benchmark: The test is tagged as benchmark in the RTEMS sources and outputs the banner *** TEST STATE: BENCHMARK. Benchmarks can take a while to run and performance is not regression tested in RTEMS. The RTEMS Tester will reset the target if the target’s configuration provides a target reset command.

timeout: The test start banner has been sent to the console and no end banner is seen within the timeout period and the target has not restart. A default timeout can be set in a target configuration, a user configuration or provide on the RTEMS Tester’s command line using the --timeout option.

invalid: The test did not output a start banner and the RTEMS Tester has detected the target has restarted. This means the executable did not load correctly, the RTEMS kernel did not initialize or the RTEMS kernel configuration failed for this BSP.

11.2.1. Expected Test States¶

A test’s expected state is set in the RTEMS kernel’s testsuite. The default for a tested is to pass. If a test is known to fail it can have it’s state set to expected-fail. Setting tests that are known to fail to expected-fail lets everyone know a failure is not to be countered and consider a regression.

Expected test states are list in test configuration files that end with the file extension .tcfg. The testsuite supports global test configurations in the testsuite/testdata directory. Global test states are applied to all BSPs. BSPs can provide a test configuration that applies to just that BSP.

The test configuration file format is:

state: test test test

where test test test is a list of tests the state applies too. The state is one of:

include: The test list is the name of a test configuration file to include
exclude: The tests listed are not build. This can happen if a BSP cannot support a test. For example it does not have enough memory.
expected-fail: The tests listed are set to expected fail. The test will fail on the BSP being built.
user-input: The tests listed require user input to run and are not supported by automatic testers.
indeterminate: The tests listed may pass or may not, the result is not reliable.
benchmark: The tests listed are benchmarks. Benchmarks are flagged and not left to run to completion because they may take too long.

11.3. Test Builds¶

The test reports the build of RTEMS being tested. The build are:

default: The build is the default. No RTEMS configure options have been used.

posix: The build includes the POSIX API. The RTEMS configure option --enable-posix has been used. The cpuopts.h define RTEMS_POSIX has defined and it true.

smp: The build is an SMP kernel. The RTEMS configure option --enable-smp has been used. The cpuopts.h define RTEMS_SMP has defined and it true.

mp: The build is an MP kernel. The RTEMS configure option --enable-multiprocessing has been used. The cpuopts.h define RTEMS_MULTIPROCESSING has defined and it true.

paravirt: The build is a paravirtualization kernel. The cpuopts.h define RTEMS_PARAVIRT has defined and it true.

debug: The build includes kernel debugging support. The RTEMS configure option --enable-debug has been used. The cpuopts.h define RTEMS_DEBUG has defined and it true.

profiling: The build include profiling support. The RTEMS configure option --enable-profiling has been used. The cpuopts.h define RTEMS_PROFILING has defined and it true.

11.4. Tester Configuration¶

The RTEMS Tester and RTEMS Run are controlled by configuration data and scripts. The user specifies a BSP on the command line using the --rtems-bsp option as well as optionally specifying a user configuration file using --user-config.

The Figure RTEMS Tester and Run Configuration Files shows the various sources of configuration data and their format. The ini files are the standard INI format, the mc are the internal RTEMS Toolkit’s Macro format, and cfg is the RTEMS Toolkit’s Configuration script format, the same format used by the RTEMS Source Builder.

RTEMS Tester and Run Configuration Files

Configuration data is held in a macro database keyed on the macro name. Macros can be expanded in configuration scripts using the syntax %{name}. The macro database is layered using maps. The defaults and values created when a configure script runs live the in the global map. Values read from the BSP and User INI configuration files are loaded into maps based on the BSP name. This lets a single User configuration file contain specialized configuration values for a number of BSPs and the tester and run commands select the values based on the selected BSP. Macros are expanded using the BSP map first giving those values the highest priority. User defined values are loaded after the BSP configuration values overwriting them letting a user speckles a BSP’s default configuration for their local needs.

Figure RTEMS Tester and Run Configuration Load and Execute Sequence shows the configuration loading and script execution order.

RTEMS Tester and Run Configuration Load and Execute Sequence

11.4.1. Defaults¶

The RTEMS Tester and RTEMS Run are primed using defaults from the file rtems/testing/testing.mc. All default settings can be overridden in a BSP or User configuration file.

11.4.2. BSP and User Configuration¶

The BSP and User configuration files are INI format files. The BSP configuration file has to have an INI section that is the name of the BSP passed on the command line. The section has the following mandatory values:

bsp: The name of the BSP. The BSP name is used to create a macro map to hold the BSP’s configuration data. Typically this is the same as the BSP name used on the command line.

arch: The name of the BSP architecture. This is need for the GDB configuration scripts where the architecture specific GDB needs to run. It is mandatory so the arch/bsp standard RTEMS BSP string can be used.

tester: The tester or run configuration script. This is the name of the configuration script the RTEMS Tester or RTEMS Run executes as a back end. The tester value is typically of the form %{_rtscripts}/<script> where <script> is name of the back end script to be run.

Target commands support expansion of specific tags to provide a convenient way for users to customize a local test environment. The parameters expanded are:

@ARCH@: The BSP architecture.

@BSP@: The BSP’s name set by the bsp value.

@EXE@: The executable name as an absolute path

@FEXE@: The filtered executable if a target_exe_filter is provided else the executable’s file name.

The following are optional and depend on the back end being used and the local target hardware set up:

jobs: The jobs value sets the number of jobs that can be run at once. This setting only effects the RTEMS Tester. The tester can run up to the jobs value of tests concurrently. If the tester back end is a simulator running a job on each available core lowers the total test time. Overloading a machine with too many simulators running in parallel can slow down each simulation and test timeouts may be recorded.

bsp_tty_dev: The BSP’s tty device. This can be a real device on the host machine the executable is being run from or it can be a telnet server and port defined using the stand host format. See Chapter 11 Section 5 - Consoles for details.

target_pretest_command: The pre-test command is a host shell command that is called before each test runs. It can be used to construct a suitable environment or image needed by a simulator or target. The RTEMS executate being run is provided as an argument and the bootloader specific format is the output.
target_posttest_command: The post-test command is a host shell command that is called after each test has finished. It can be used to destroy any environment or image created by the pre-test command.

target_exe_filter: The target executable filter transforms the executable name into a filtered executable name. This filter lets the tester or run command track the name of any generated file a pre-test command may generate. The syntax is a simplified sed regular expression. The first character is a delimiter and there must be 2 sections therefore 3 delimiter. The first section is a Python regular expression and the second section is plain text that replaces anywhere the regular expression matches. For example /\.exe/.exe.img/ will search for .exe in the executable name and replace it with .exe.img. Note, there is no need to escape the text in the second part, it is just plain test.

test_restarts: The number of restarts before the test is considered invalid. Currently not used.

target_reset_regex: The target reset regular expression. This is a Python regular expression used to filter the console input. If a match is made something has happened during the boot process that requires a reset. The target_reset_command is issued to perform the reset. Typically this field looks for boot loader error messages that indicate the boot process as failed.

target_start_regex

The target start regular expression. This is a Python regular expression to filter the console input to asynchronously detect if a target has reset. If a board crashes running a test or at any point reset this filter detects the restart and ends the test with a suitable result.

target_on_command: The target on command is a host shell command that is called before the first test. This command powers on a target. Targets should be left powered off when not running tests or the target may request TFTP downloads that are for another target interfering with those test results. We recommend you implement this command as a target off command, a pause, then a target on command.

target_off_command: The target off command is a host shell command that is called after the last test powering off the target.

target_reset_command: The target reset command is a host shell command that is called when the target needs to be reset. This command can power cycle the target or toggle a reset signal connected to the target. If you are power cycling a target make sure you have a suitable pause to let the target completely power down.

11.4.3. Configuration Scripts¶

Configuration scripts are provided for each supported RTEMS Tester and RTEMS Run back end and console management. The scripts are in the standard RTEMS Toolkit Configuration Script format. Please refer to the RTEMS Source Builder documentation for the basic scripting syntax and usage.

The RTEMS Tester and RTEMS Run specializes the standard configuration syntax providing a directive for the console and each supported back end. The supported directives are:

%console
%execute
%gdb
%tftp

11.4.3.1. Console¶

The %console configures the console used to access the target’s console. The console can be a process’s stdout, a termios tty on Unix and MacOS and Telnet on all hosts. The directive accepts:

stdio

The standard output stream from the executing processing.

tty <dev> <settings>

The name of the tty to open and use. The tty device or <dev> can be a termio device and the <settings> are standard termios values.

The Python termios document provides details of the settings that can be controlled. The settings are a single string where prefix the value with ~ negates the setting. Setting are:

B115200 (an example buadrate)

BRKINT

IGNBRK

IGNCR

ICANON

ISIG

IEXTEN

ECHO

CLOCAL

CRTSCTS

VMIN=<value>

VTIME=<value

A example in a configuration script is:

%define bsp_tty_dev      /dev/ttyUSB2
%define bsp_tty_settings B115200,~BRKINT,IGNBRK,IGNCR,~ICANON,~ISIG,~IEXTEN,~ECHO,CLOCAL,~CRTSCTS,VMIN=1,VTIME=2

A example BSP or User configuration file is:

[bsp-special]
bsp              = example-bsp
bsp_tty_dev      = /dev/ttyUSB2
bsp_tty_settings = B115200,~BRKINT,IGNBRK,IGNCR,~ICANON,~ISIG,~IEXTEN,~ECHO,CLOCAL,~CRTSCTS,VMIN=1,VTIME=2

The console directive is managed in the %{_rtscripts}/console.cfg configuration script. If the %{console_stdio} is defined the console will be stdio else the console will be the BSP console or %{bsp_tty_dev}.

Telnet can be combined with the ser2net daemon to remotely access a target’s physical serial UART interface.

11.4.3.2. Execute¶

The %execute directive executes a command for each rest. The execute forks the command and arguments supplied to the execute directive and captures the stdout stream as the console. If the console directive is set to stdout the sub-processes stdout stream is used as the console.

The RTEMS Tester will run parallel tests as jobs.

An example is:

%execute %{run_cmd} %{run_opts} %{test_executable} %{test_executable_opts}

11.4.3.3. GDB¶

The %gdb directive executes GDB in the machine interface mode give the RTEMS Tester and RTEMS Run commands control. The console is taken from GDB if it is stdout.

The RTEMS Tester will run parallel tests as jobs.

An example is:

%gdb %{gdb_cmd} %{test_executable} %{gdb_script}

11.4.3.4. TFTP¶

The %tftp directive starts a TFTP session on a specified port sending the test executable to the target over a networking using the TFTP protocol.

The RTEMS Tester will run only one test at a time. There is just one physical board running the test.

An example is:

%tftp %{test_executable} %{tftp_port}

11.5. Consoles¶

The RTEMS Tester uses the target’s console output to determine the state of a test. Console interfaces vary depending on the testing mode, the BSP, and the target hardware.

Consoles for simulator work best if mapped to the simulator’s stdout interface. The RTEMS Tester can capture and process the stdout data from a simulator while it is running.

Target hardware console interfaces can vary. The most universal and stable interface target hardware is a UART interface. There are a number of physical interfaces for UART data these days. They are:

RS232
TTL
USB

RS232 is still present on a number of targets. The best solution is to use a RS232 to USB pod and convert the port to USB.

TTL is common on a number of boards where cost is important. A console interface is typically a development tool and removing the extra devices need to convert the signal to RS232 or directly to USB is not needed on production builds of the target. There is a standard header pin out for TTL UART consoles and you can purchase low cost cables with the header and a built in UART to USB converter. The cables come is different voltage levels so make sure you check and use the correct voltage level.

The USB interface on a target is typcially a slave or OTG interface and all you need to a standard USB cable.

We recommend a low cost and low power device to be a terminal server. A Raspberry Pi or similar low cost computer running Linux can be set up quickly and with a powered USB hub and can support a number of USB UART ports. A USB hub with a high power port is recommended that can suppy the Raspberry Pi.

The open source daemon ser2net is easy to configure to map the USB UART ports to the Telnet protocol. There is no need for security because a typical test environment is part of a lab network that should be partitioned off from an enginnering or corportate network and not directly connected to the internet.

A test set up like this lets you place a terminal server close to your target hardware providing you with the flexibility to select where you run the RTEMS Tester. It could be your desktop or an expensive fast host machine in a server rack. None of this equipment needs to directly interface to the target hardware.

The RTEMS Tester directly supports the telnet protcol as a console and can interface to the ser1net server. The telnet console will poll the server waiting for the remote port to connect. If the terminal server ser2net does not have a tty device it will not listen on the port assigned to that tty. A USB tty can come and go depending on the power state of the hardware and the target hardware’s design and this can cause timing issues if the target hardware is power cycled as part of a reset process.

11.6. Simulation¶

Simulation is a important regression and development tool for RTEMS. Developers use simulation to work on core parts of RTEMS as it provides excellent debugging supporting. Simulation run via the RTEMS Tester allows a test to run on each core of your testing host machine lower the time to run all tests.

RTEMS Tester Simulation

The RTEMS Tester Simulation figure shows the structure of RTEMS Testing using simulation. The executables are built and the rtems-test command is run from the top of the build directory. The RTEMS Tester executes the BSP specific simulator for each test capturing the output

11.7. GDB and JTAG¶

GDB with JTAG provides a low level way to runs tests on hardware with limited resources. The RTEMS Tester runs and controls an instance of GDB per test and GDB connects via the GDB remote protocol to a GDB server that interfaces to the JTAG port of a target.

RTEMS Tester using GDB and JTAG

The RTEMS Tester using GDB and JTAG figure shows the structure of RTEMS Testing using GDB and JTAG. The executables are built and the rtems-test command is run from the top of the build directory. The RTEMS Tester executes the BSP architecture’s GDB and expects the user to provide a gdb-script to connect t the JTAG GDB server.

11.8. TFTP and U-Boot¶

TFTP and U-Boot provides a simple way to test RTEMS on a network capable target. The RTEMS Tester starts a TFTP server for each test and the target’s boot monitor, in this case U-Boot request a file, any file, which the TFTP server supplies. U-Boot loads the executable and boots it using a standard U-Boot script.

RTEMS Tester using TFTP and U-Boot.

The RTEMS Tester using TFTP and U-Boot. figure shows the structure and control flow of the RTEMS Tester using TFTP and U-boot. The executables are built and the rtems-test command is run from the top of the build directory.

This test mode can only support a single test job running at once. You cannot add more test target hardware and run the tests in parallel.

11.8.1. Target Hardware¶

The RTEMS Tester TFTP and U-Boot method of testing requires:

A target with network interface.
U-Boot, iPXE or similar boot loader with network driver support for your target hardware and support for the TFTP protocol.
Network power of IO switch.
Network DHCP server.
Console interface cable that matches your target’s console UART interface.
Telnet terminal server. See Chapter 11 Section 5 - Consoles.

The network power or IO switch is a device that can control power or an IO pin over a network connection using a script-able protocol such as Telnet or curl. This device can be used with the target control commands.

11.8.1.1. U-Boot Set Up¶

Obtain a working image of the U-Boot boot loader for your target. We suggest you follow the instructions for you target.

Configure U-Boot to network boot using the TFTP protocol. This is U-Boot script for a Zedboard:

loadaddr=0x02000000
uenvcmd=echo Booting RTEMS Zed from net; set autoload no; dhcp; set serverip 10.10.5.2; tftpboot zed/rtems.img; bootm; reset;

The load address variable loadaddr is specific to the Zedboard and can be found in the various examples scripts on the internet. The script then sets U-Boot environment variable autoload to no causing DHCP to only request a DHCP lease from the DHCP server. The script sets the serverip to the host that will be running the RTEMS Tester then issues a TFTP request. The file name can be anything because the RTEMS Tester ignores it sending the executable image under test. Finally the script boots the download executable and if that fails the catch all reset resets the board and starts the boot process over.

Test the target boots and U-Boot runs and obtains a valid DHCP lease. Manually connect the console’s telnet port.

11.8.2. BSP Configuration¶

The BSP’s configuration file must contain the standard fields:

bsp
arch
jobs - Must be set to 1.
tester - Set to %{_rtscripts}/tftp.cfg

For example the Zedboard’s configuration is:

[xilinx_zynq_zedboard]
bsp    = xilinx_zynq_zedboard
arch   = arm
jobs   = 1
tester = %{_rtscripts}/tftp.cfg

The TFTP configuration supports the following field’s:

bsp_tty_dev: The target’s tty console. For telnet this is a host and port pair written in the standard networking format, for example serserver:12345.
test_restarts: The number of restarts before the test is considered invalid.
target_reset_regex: The target reset regular expression. This is a Python regular expression used to filter the console input. If a match is made something has happened during the boot process that requires a reset. The target_reset_command is issued to perform the reset. This field is typically looks for boot loader error messages that indicate the boot process as failed.
target_start_regex: The target start regular expression. This also a Python regular expression to filter the console input to detect if a target has reset. If a board crashes running a test or at any point in time and reset this filter detects this as happened and end the test with a suitable result.
target_on_command: The target on command is a host shell command that is called before the first test. This command powers on a target. Targets should be left powered off when not running tests or the target may request TFTP downloads that are for another target interfering with those test results. We recommend you implement this command as a target off command, a pause, then a target on command.
target_off_command: The target off command is a host shell command that is called after the last test powering off the target.
target_reset_command: The target reset command is a host shell command that is called when the target needs to be reset. This command can power cycle the target or toggle a reset signal connected to the target. If you are power cycling a target make sure you have a suitable pause to let the target completely power down.
target_pretest_command: The target pretest command is a host shell comment that is called before the test is run

The commands in the listed fields can include parameters that are substituted. The parameters are:

@ARCH@: The BSP architecture
@BSP@: The BSP’s name
@EXE@: The executable name.
@FEXE@: The
. The: @ARCH is the

substituted

Some of these field are normally provided by a user’s configuration. To do this use:

requires = bsp_tty_dev, target_on_command, target_off_command, target_reset_command

The requires value requires the user provide these settings in their configuration file.

The Zedboard’s configuration file is:

[xilinx_zynq_zedboard]
bsp                = xilinx_zynq_zedboard
arch               = arm
jobs               = 1
tester             = %{_rtscripts}/tftp.cfg
test_restarts      = 3
target_reset_regex = ^No ethernet found.*|^BOOTP broadcast 6.*|^.+complete\.+ TIMEOUT.*
target_start_regex = ^U-Boot SPL .*
requires           = target_on_command, target_off_command, target_reset_command, bsp_tty_dev

The target_start_regex searches for U-Boot’s first console message. This indicate the board can restarted.

The target_reset_regex checks if no ethernet interface is found. This can happen if U-Boot cannot detect the PHY device. It also checks if too many DHCP requests happen and finally a check is made for any timeouts reported by U-Boot.

An example of a user configuration for the Zedboard is:

[xilinx_zynq_zedboard]
bsp_tty_dev            = selserver:12345
target_pretest_command = zynq-mkimg @EXE@
target_exe_filter      = /\.exe/.exe.img/
target_on_command      = power-ctl toggle-on 1 4
target_off_command     = power-ctl off 1
target_reset_command   = power-ctl toggle-on 1 3

11.8.3. TFTP Sequences¶

Running a large number of tests on real hardware exposes a range of issues and RTEMS Tester is designed to be tolerant of failures in booting or loading that can happen, for example a hardware design. These sequence diagrams document some of the sequences that can occur when errors happen.

The simplest sequence is running a test. The target is powered on, the test is loaded and executed and a pass or fail is determined:

Test Pass and Fail Sequences

The target start filter triggers if a start condition is detected. This can happen if the board crashes or resets with no output. If this happens repeatedly the test result is invalid:

Target Start Filter Trigger

The reset filter triggers if an error condition is found such as the bootloader not being able to load the test executable. If the filter triggers the target_reset_command is run:

Target Reset Filter Trigger

If the RTEMS Tester does not detect a test has started it can restart the test by resetting the target. The reset command can toggle an IO pin connected to reset, request a JTAG pod issue a reset or turn the power off and on:

Target Timeout