10. Executables

This section discusses what an RTEMS executable is and what happens when you execute it in a target. The section discusses how an application executable is created, what happens when an executable is loaded and run as well as debugging an execiutable.

10.1. RTEMS Executable

Running executables is the most important part of working with RTEMS, it is after all how you run your application and use the RTEMS kernel services.

An RTEMS executable is embedded in a target and executing an embedded executable has challenges not faced when executing software on a desktop or server computer. A desktop or server operating system kernel provides all the support needed to bring an executable’s code and data into a process’s address space passing control to it and cleaning up when it exits. An embedded target has to provide similar functionality to execute an embedded executable.

An RTEMS Source Builder (RSB) built RTEMS tool chain is used to create RTEMS executables. The tool chain executable creates a fixed position statically linked Extendable Loader Format (ELF) file that contains the RTEMS kernel, standard libraries, 3rd party libraries and application code. RTEMS executes in a single address space which means it does not support the fork or exec system calls so statically linking all the code is the easiest and best way to create an executable.

An RTEMS application is constructed vertically with the RTEMS kernel, BSP support code and drivers close to the hardware, above which sit the RTEMS Application Programming Interfaces (API) for control of threads, mutex and other resources an application may use. Middle-ware services like networking, interpreted languages, and protocol stacks sit between the RTEMS APIs and the application components. The software built into an executable can be see as a vertical software stack.

Vertical Software Stack

10.2. Building an Application

RTEMS views any code it is running and using it’s interfaces as an application. RTEMS conforms to a number of international standards such as POSIX and can build and run portable code written in languages such as C, C++ and Ada.

Applications are built from source into ELF object files, 3rd party packages can be built as libraries or they can be imported as source into an application code base. The application, 3rd party packages, RTEMS and standard libraries are linked to create the RTEMS executable. The executable is transferred to the target and a bootloader loads it from the non-volatile storage into RAM or the code is executed in place in the non-volatile storage. The target hardware defines what happens.

Building an Application

The standard and 3rd party libraries are a collection of object files built using the same set of tools the application source is compiled with. The package collects it’s object files into an archive or library.

RTEMS does not provide a standard application build system. The RTEMS ecosystem provides support so a range of build systems can be used. Applications can be built with make, autotools, cmake, waf and more. User should select a build system that meets their project, system, corporate or personal needs.

10.2.1. Machine Flags and ABI

All code in an RTEMS executable must be built with the same machine flags. The machine flags control the instruction set and application binary interface (ABI) the compiler generates. As the executable is statically linked all code must use the same instruction set the hardware is configured to support and all code must conform to the same ABI. Any variation can result in unpredictable behavior such as crashes, failures or lock ups. It is recommend an executable is built with the same or equivalent tool set. Mixing of tool set versions can also result in undefined behavior. The RTEMS tool rtems-execinfo can audit an RTEMS executable and list the machine flags and compilers used.

RTEMS by default does not support instruction emulation for unsupported instructions. RTEMS applications are normally built from source so binary compatibility is not as important as performance. Instruction emulation is costly to execute and rebuilding the executable with the correct instruction set only needs to be done once.

10.3. Target Execution

Fixed position statically linked executables have a fixed address in a target’s address space. The location in the address space for code, data and read-only data is fixed. The BSP defines the memory map and it is set by the BSP developer based on the target’s hardware requirements and it’s bootloader.

Targets typically contains a bootloader that is executed after the target’s processor exits reset. A bootloader is specific to a target’s processor and hardware configuration and is responsible for the low level initialization of the hardware resources needed to load and execute an operating system’s kernel. In the case of RTEMS this is the RTEMS executable.

Bootloaders vary in size, complexity and functionality. Some architectures have a number of bootloader stages and others have only minimal support. An example of a high end system is Xilinx’s Zynq processor with three stages. First a mask ROM in the System On Chip (SOC) executes after reset loading a first stage bootloader (FSBL) from an SD card, QSPI flash or NAND flash depending on signals connected to the device. The FSBL loads a second stage bootloader (SSBL) such as U-Boot and this loads the kernel. U-Boot can be configured to load a kernel from a range of media and file system formats as well as over a network using a number of protocols. This structure provides flexibility at the system level to support development environments such as a workshop or laboratory through to tightly control production configurations.

Bootloaders often have custom formats for the executable image they load. The formats can be simple to keep the bootloader simple or complex to support check-sums, encryption or redundancy in case an image becomes corrupted. A bootloader often provides a host tool that creates the required file from the RTEMS executable’s ELF file.

If RTEMS is to run from RAM the bootloader reads the image and loads the code, initialized data and read-only data into the RAM and then jumps to a known entry point. If the code is executed from non-volatile storage the process to write the image into that storage will have extracted the various binary parts and written those to the correct location.

The important point to note is the binary parts of the executable are somehow loaded into the target’s address space ready to execute. The way this done may vary but the out come is always the same, the binary code, data and read-only data is resident in the processor’s address space at the BSP defined addresses.

10.4. BSP Initialization

The bootloader jumps or calls the RTEMS executable’s entry point, normally a fixed address. The BSP entry point or start up code performs:

1. Low level processor specific initialization that such as setting control registers so the processor is operating in a mode RTEMS is built for
2. Cache flushing, clearing and invalidation
3. Memory management unit (MMU) set up if required
4. Clear the uninitialized data section
5. Process a command line if supported by the bootloader
6. Call bootcard which disabled interrupts, saves away a command line if the BSP supports it then call the RTEMS kernel early initialize entry point rtems_initialize_executive. This call never returns.

Further BSP initialization happens as part of RTEMS kernel’s System Initialization process. The following handlers are declared and if provided are placed at the beginning of the initialization handler list. The BSP can provides:

bsp_work_area_initialize

This function determines the amount of memory that can be given to RTEMS for the workspace and the C library heap which malloc uses. The call typically uses the bsp_work_area_initialize_default to perform actually perform the initialization.

bsp_start

This function is specialized for each architecture and even for some BSPs. It performs the low level initialization RTEMS needs so it can run on the architecture and BSP.

bsp_predriver_hook

This function can be used to initialize hardware drivers depend on such as configuring an interrupt controller. The default version is empty and does nothing.

BSPs all perform similar operations with common functionality and the RTEMS kernel provides common code that can be shared between BSPs. The use of the common code is encouraged for all new BSPs.

10.5. RTEMS Initialization

The RTEMS kernel initialization is:

1. Invoke the registered system initialization handlers
2. Set the system state to up
3. If the kernel supports SMP request multitasking start. All online cores are transferred to the ready to start multitasking state.
4. Start threaded multitasking. RTEMS starts multitasking by getting the first thread to run and dispatching it.

C++ static object constructors are called in the context of the first running thread before the thread body is entered.

10.5.1. System Initialization Handlers

RTEMS supports the automatic registration of services used in applications. This method of initialization automatically configures RTEMS with only the services used in an application. There is no manual configuration of services used and no updating of initialization function tables.

RTEMS uses specialized sections in the ELF executable to perform this task. The system is based on the FreeBSD SYSINT Framework. Ordered initialization is performed before multitasking is started.

The RTEMS Tool rtems-exeinfo can provide some detail about the registered handlers. The following shows the initialization handlers for the hello world sample application in the RTEMS kernel’s testsuite:

$ rtems-exeinfo --init arm-rtems5/c/xilinx_zynq_zedboard/testsuites/samples/hello.exe
RTEMS Executable Info 5.5416cfa39dd6
 rtems-exeinfo --init arm-rtems5/c/xilinx_zynq_zedboard/testsuites/samples/hello.exe
exe: arm-rtems5/c/xilinx_zynq_zedboard/testsuites/samples/hello.exe

Compilation:
 Producers: 2
  |  GNU AS 2.31.1: 14 objects
  |  GNU C11 7.3.0 20180125 (RTEMS 5, RSB e55769c64cf1a201588565a5662deafe3f1ccdcc, Newlib 103b055035fea328f8bc7826801760fb1c055683): 284 objects
 Common flags: 4
  | -march=armv7-a -mthumb -mfpu=neon -mfloat-abi=hard

Init sections: 2
 .init_array
  0x001047c1 frame_dummy
 .rtemsroset
  0x00104c05 bsp_work_area_initialize
  0x00104c41 bsp_start
  0x0010eb45 zynq_debug_console_init
  0x0010ec19 rtems_counter_sysinit
  0x0010b779 _User_extensions_Handler_initialization
  0x0010c66d rtems_initialize_data_structures
  0x00107751 _RTEMS_tasks_Manager_initialization
  0x0010d4f5 _POSIX_Keys_Manager_initialization
  0x0010dd09 _Thread_Create_idle
  0x0010cf01 rtems_libio_init
  0x001053a5 rtems_filesystem_initialize
  0x0010546d _Console_simple_Initialize
  0x0010c715 _IO_Initialize_all_drivers
  0x001076d5 _RTEMS_tasks_Initialize_user_tasks_body
  0x0010cfa9 rtems_libio_post_driver

The section .rtemsroset lists the handlers called in order. The handlers can be split into the BSP initialization handlers that start the BSP:

• bsp_work_area_initialize
• bsp_start
• zynq_debug_console_init
• rtems_counter_sysinit

And the remainder are handlers for services used by the application. The list varies based on the services the application uses.

10.6. Debugging

An RTEMS executable is debugged by loading the code, data and read-only data into a target with a debugger connected. The debugger running on a host computer accesses the ELF file reading the debug information it contains.

The executable being debugged needs to be built with the compiler and linker debug options enabled. Debug information makes the ELF executable file large but it does not change the binary footprint of the executable when resident in the target. Target boot loaders and file conversion tools extract the binary code, data and read-only data to create the file embedded on the target.

An ELF executable built with debug information contains DWARF debug information. DWARF is a detailed description of the executable a debugger uses to locate functions, find data, understand the type and structure of a variable, and know how much entry code every call has. The debugger uses this information to set breaks points, step functions, step instructions, view the data and much more.

We recommend the compiler and linker debug options are always enabled. An ELF file with debug information can be used to investigate a crash report from a production system if the production ELF image is archived. The RTEMS tools chain provides tools that can take an address from a crash dump and find the corresponding instruction and source line. The extra size the debug information adds does not effect the target footprint and the extra size on a host is small compared to the benefits it brings.

A desktop or server operating system’s kernel hosts the executable being debugged handling the interaction with the executable and the debugger. The debugger knows how to communicate to the kernel to get the information it needs. Debugging an embedded executable needs an extra piece called an agent to connect the target to the debugger. The agent provides a standard remote interface to the debugger and an agent specific connection to the target.

Embedded Executable Debugging

The RTEMS tool chain provides the GNU debugger GDB. GDB has a remote protocol that can run over networks using TCP and UDP protocols. The GDB remote protocol is available in a number of open source and commercial debugging solutions. Network debugging using the remote protocol helps setting up a laboratory, the targets can be remote from the developers desktop allowing for better control of the target hardware while avoiding the need to plug devices in to an expensive desktop or server machine.

The following are some examples of GDB and GDB server environments RTEMS supports.

QEMU contains a debugging agent for the target being simulated. A QEMU command line option enables a GDB server and the simulator manages the interaction with the target processor and it’s memory and caches.

QEMU Executable Debugging

OpenOCD is a JTAG debugging package that interfaces to a wide of JTAG pods. JTAG is a low level high speed serial interface modern processors provide as a means of controlling the core processing logic. The features available depend on the architecture and processor. Typical functions include:

1. Processor control and register access
2. System level register access to allow SOC initialization
3. General address space access
4. Cache and MMU control
5. Break and watch points

OpenOCD JTAG Executable Debugging

The RTEMS kernel has a debugging agent called libdebugger. This is a software based agent that runs within RTEMS using network services to provide a remote GDB protocol interface. A growing number of architectures are supported. The RTEMS debugging agent is for application development providing thread aware stop model debug experience.

Libdebugger Executable Debugging