RTEMS / Programs / Google Summer of Code

Go to Issues or rtems/programs/gsoc-merges

Merge Requests Summary

Issues

32 - Conversion to new test suite.

Id

32

State

closed

Type

ISSUE

Author

Amar Takhar

Assignee(s)

Amar Takhar

Created

2017-03-02T19:19:19.000Z

Updated

2025-09-19T01:48:32.330Z

Milestone

6.2

Labels

gsoc, old-indefinite, priority::normal, resolution::invalid, rtems::testing, tickettype::project

Link

https://gitlab.rtems.org/rtems/programs/gsoc/-/issues/32

Merges

0

[Contents, inline)]

Introduction

The current test suite is a combination of different methods and techniques which makes it difficult to maintain. It also relies on screen ouput to determine valid tests.

This project involves converting the current test suite to the new format which has already been setup and prepared.

You will learn about software and hardware testing in embedded computing. The abilities learned during this project will be transferable to any area within software engineering.

Requirements

Knowledge of testing is not required. The ideal candidate should have a firm grasp of the C language and enjoys the occasional challenge. Each test conversion has varied levels of difficulty ranging from 1-10. Some tests will require refactoring of code. Light knowledge of the Python programming language is an asset.

Preparation

It would also be beneficial for anyone applying to briefly familiarise themselves with the following topics:

  • Unit tests.

  • Mock tests.

  • Temporal tests.

  • Continuous Integration

New Build System

A new build system has been written for RTEMS using the Waf build system. In order to familiarise yourself with this please see the following email thread:

https://lists.rtems.org/pipermail/devel/2015-February/009837.html

Waf

Waf is a build system written in Python. Please read over the documents to get a rough idea of how it works. It is not necessary to learn it completely just get an idea of how it is put together.

http://waf.io/

Author: Amar Takhar

2024-04-26T01:34:34.484Z

assigned to @amar

Author: Amar Takhar

2017-03-02T19:32:05.000Z

Add SoC keyword.

Author: Amar Takhar

2017-03-20T17:21:40.000Z

Hello to everyone interested in this project. I will be updating with further details over the next few days. If you are interested in following please add yourself to the CC list after signing up to trac.

Author: Trac Migrate

2017-03-21T13:23:23.000Z

  • Cc added @aadit0402@gmail.com

Author: Amar Takhar

2017-03-23T14:43:55.000Z

  • Description changed

[[Contents, inline)](../wikis/PageOutline(1-5,)]


== Introduction ==

The current test suite is a combination of different methods and techniques which makes it difficult to maintain.  It also relies on screen ouput to determine valid tests.

This project involves converting the current test suite to the new format which has already been setup and prepared.

You will learn about software and hardware testing in embedded computing.  The abilities learned during this project will be transferable to any area within software engineering.



== Requirements ==

Knowledge of testing is not required. The ideal candidate should have a firm grasp of the C language and enjoys the occasional challenge.  Each test conversion has varied levels of difficulty ranging from 1-10.  Some tests will require refactoring of code.  Light knowledge of the Python programming language is an asset.


== Preparation ==

- It would be beneficial for anyone applying to briefly familiarise themselves with the following topics:
+ It would also be beneficial for anyone applying to briefly familiarise themselves with the following topics:
?          +++++


* Unit tests.
* Mock tests.
* Temporal tests.
* Continuous Integration
+
+
+ === New Build System ===
+
+ A new build system has been written for RTEMS using the Waf build system.  In order to familiarise yourself with this please see the following email thread:
+
+   https://lists.rtems.org/pipermail/devel/2015-February/009837.html
+
+ === Waf ===
+
+ Waf is a build system written in Python.  Please read over the documents to get a rough idea of how it works.  It is not necessary to learn it completely just get an idea of how it is put together.
+
+   http://waf.io/

Apologies for the delay, bad timing with work I will have time now for the next few weeks to answer questions.

Author: Amar Takhar

2017-03-23T15:00:07.000Z

Please also refer to this email posted to the users list. I will add the content here for ease:

https://lists.rtems.org/pipermail/users/2017-March/031107.html

This project involves a near ground-up rewrite of the current testsuite harness. Most tests are trivial to convert but many will need refactoring. The new testsuite moves away from the ‘screen’ based testing we do currently to an internal, C-based approach. That is a very general and broad overview it’s not as simple as it sounds.

I do have this working already and the tests have all been renamed.

All the tests have been renamed to use a more intuitive naming scheme no more hidden and random file names. The web portal to the test suite is up temporarily here:

https://test.rtems.org/

This project also involves going into the database to ensure they have been properly renamed and fixing any that need changing.

Feel free to drill down the site. I’m working on better documentation for this project by the 20th I was hoping for sooner but things have gotten in the way.

Anyone working on this project is not expected to convert the entire testsuite, just as much as possible the rest of the work will involve identifying what may be required to complete the rest which can involve refactoring of RTEMS itself.

Author: Trac Migrate

2017-03-29T06:11:00.000Z

The Unity testing framework is by no means good enough for the RTEMS testsuite. It lacks some important features like thread-safety at a basic level.

See also:

https://lists.rtems.org/pipermail/devel/2013-November/004965.html

Nothing changed in this respect since 2013.

I would rather use something like the Google C++ Testing Framework, however, it uses to much resources for our low level targets.

https://git.rtems.org/sebh/rtems-gtest.git/

I don’t think there exists a ready to use testing framework which meets our requirements. What are our requirements by the way? This should be addressed first.

What is the specification input for the tests itself?

How is the output produced at the device level?

How it is ensured the output processing doesn’t influence the test outcome in unexpected ways across targets?

How is the output produced for test suite status consumers? Maybe have a look at XO(1).

How is this synchronized with RTEMS qualification efforts?

How it is guaranteed that we don’t drop test cases accidentally during the conversion? Some tests are hard to understand. Do we keep the existing test suite as is?

And so on.

It is a several man month project if you want to do this right.

Author: Chris Johns

2017-03-29T06:23:07.000Z

Replying to Sebastian Huber:

How is the output produced for test suite status consumers? Maybe have a look at XO(1).

The only consumer on the table at the moment is rtems-test and I would prefer we do not fragment how tests are run. The rtems-test tool is designed to provide a consistent user interface across varying ways we test RTEMS. What happens under the hood is flexible so I hope it can be updated to handle any new test data formats.

Author: Trac Migrate

2017-03-29T16:33:38.000Z

  • Cc added @aadit0402@gmail.com, @cpodonnell8@gmail.com

Author: Amar Takhar

2018-10-21T04:20:12.000Z

  • Resolution set to ~”invalid”

  • Status changed from assigned to closed

Closing this due to it’s irrelevance. There have been far too many chances to this I’ll open a new ticket for the suite I am working on in the future.

Author: Amar Takhar

2024-04-25T20:45:24.909Z

changed the description