RTEMS / Tools / RTEMS Tools

Go to Issues or Merge Requests

Issues

40 - Backport to 6 branch “tester: Avoid relying on external telnetlib”

Id

40

State

closed

Type

ISSUE

Author

Kinsey Moore

Assignee(s)

Kinsey Moore

Closed by

Kinsey Moore

Created

2026-08-14T18:44:01.865Z

Closed

2026-08-27T22:57:32.746Z

Updated

2026-08-27T22:57:32.987Z

Milestone

6.3

Labels

tool::test

Link

https://gitlab.rtems.org/rtems/tools/rtems-tools/-/work_items/40

Merges

0

The following discussion from !90 should be addressed:

  • [] @opticron started a discussion:

    If this is an acceptable solution, it needs to be backported to 6 branch.

Author: Kinsey Moore

2026-08-14T18:44:02.057Z

added rtems#15 as parent item

Author: Kinsey Moore

2026-08-14T18:44:02.173Z

set status to To do

Author: Kinsey Moore

2026-08-14T18:44:02.322Z

assigned to @opticron

Author: Kinsey Moore

2026-08-14T18:47:22.966Z

mentioned in merge request !91

Author: Kinsey Moore

2026-08-27T22:57:32.827Z

set status to Done

30 - RTEMS tester fails with telnet under python 3.13

Id

30

State

closed

Type

ISSUE

Author

Chris Johns

Assignee(s)

Kinsey Moore, Chris Johns

Closed by

Chris Johns

Created

2025-10-02T22:50:32.938Z

Closed

2026-08-13T23:50:09.392Z

Updated

2026-08-13T23:50:09.759Z

Milestone

6.3

Labels

tool::test

Link

https://gitlab.rtems.org/rtems/tools/rtems-tools/-/work_items/30

Merges

1

Summary

When attempting to run the RTEMS tester with Python 3.13 an error about no telnetlib support appears.

Steps to reproduce

Pre-set options

Author: Kinsey Moore

2025-03-02T02:21:55.776Z

assigned to @opticron

Author: Kinsey Moore

2025-02-28T22:39:31.957Z

assigned to @opticron

Author: Kinsey Moore

2025-02-28T22:39:32.416Z

Author: Kinsey Moore

2026-07-31T16:28:04.497Z

On further investigation, this is 2 separate issues. The escapes need to be fixed to resolve the warnings and the actual exit is caused by missing telnetlib since it was deprecated in 3.11 and removed in 3.13.

Author: Kinsey Moore

2026-07-31T16:28:04.497Z

On debian systems, there is currently a workaround for the actual exit issue by installing python3-zombie-telnetlib.

Author: Amar Takhar

2026-07-31T16:28:04.497Z

The current state of telnetlib isn’t great I don’t think there are any maintained ones. This one is its own project but no activity from the maintainer in years: https://github.com/jquast/telnetlib3

The Python one was just removed there is nobody touching the code anymore which is why they called it ‘zombie’ .. apt name.

telnetlib3 is at least installable via pypi. I’m not sure if it’s a drop in replacement does it work can you try it out?

FreeBSD for example has no telnet library for python available and I’m sure that’s true for most OS the Python project itself recommends either telnetlib3 or Exscript which is even more out of maintainership

Author: Kinsey Moore

2026-07-31T16:28:04.497Z

telnetlib3 appears to be a mostly drop-in replacement (s/telnetlib/telnetlib3/g), but I don’t have an example of a tester configuration that actually uses it, so I can’t verify that functionality completely.

Author: Chris Johns

2026-07-31T16:28:04.497Z

https://docs.rtems.org/docs/main/user/testing/configuration.html#console and look for:

%define bsp_tty_dev      1.2.3.4:8989

Author: Kinsey Moore

2026-07-31T16:28:04.497Z

Ah, I do actually have a test configuration that I can verify this on. It’ll probably have to wait until Monday.

Author: Kinsey Moore

2025-03-02T02:21:56.083Z

moved from rtems-source-builder#95

Author: Kinsey Moore

2025-03-02T02:22:42.566Z

mentioned in merge request !53

Author: Kinsey Moore

2025-07-30T18:50:47.695Z

The situation has been improved by 0efe888d3cbdc82fb4ba95c6ad8b2d2e5c960d2e in that the reason for exit is displayed. This still needs work to verify the migration path to telnetlib3.

Author: Chris Johns

2025-10-02T22:41:29.516Z

changed the description

Author: Chris Johns

2025-10-02T22:44:15.955Z

This is issue cover two problem. We need to split it as the syntax issue has been fixed and we should close this issue.

Author: Chris Johns

2025-10-02T22:49:19.321Z

/clone–with_notes

Author: Chris Johns

2025-10-02T22:50:33.404Z

cloned from #22

Author: Chris Johns

2025-10-02T22:51:41.583Z

changed title from

RTEMS tester fails

to run correctly

under python 3.13

to

RTEMS tester fails

with telnet

under python 3.13

Author: Chris Johns

2025-10-02T22:51:41.613Z

changed the description

Author: Chris Johns

2025-10-09T22:25:36.758Z

added rtems#15 as parent epic

Author: Christian Mauderer

2025-11-03T08:54:03.258Z

mentioned in merge request rtems-sis!7

Author: Chris Johns

2026-07-31T00:23:47.657Z

assigned to @chris

Author: Kinsey Moore

2026-07-31T16:32:08.443Z

The quick and dirty fix until we have something better is to vendor telnetlib ourselves.

Author: Kinsey Moore

2026-07-31T17:06:58.029Z

mentioned in merge request !90

Author: Kinsey Moore

2026-07-31T17:20:15.015Z

set status to In progress

Author: Chris Johns

2026-08-13T23:50:11.709Z

set status to Done

15 - rtems-test: Leaks pipes when using a gdb based simulator (opened)

Id

15

State

opened

Type

ISSUE

Author

Joel Sherrill

Assignee(s)

Joel Sherrill

Created

2024-10-15T21:28:33.821Z

Updated

2026-07-31T00:09:49.906Z

Milestone

6.3

Link

https://gitlab.rtems.org/rtems/tools/rtems-tools/-/work_items/15

Merges

0

Summary

When used with a gdb based simulator like psim, rtems-test uses multiple file descriptors – one for gdb.cfg and multiple for pipes. On a server with 56 cores, I encountered the following failure:

[572/675] p:491 f:5   u:6   e:22  I:0   B:3   t:0   L:0   i:0   W:0   | powerpc/psim: spsem_err02.exe
[564/675] p:491 f:5   u:6   e:22  I:0   B:3   t:0   L:0   i:0   W:0   | powerpc/psim: spregion_err01.exe
error: error opening config file: /home/joel/rtems-cron-6.1-rc4/tools/6/share/rtems/tester/rtems/version.cfg

The error opening the config file is because the process rtems-test has exceeded its allowed maximum of file descriptors. The following snippet is from watching lsof with the pid for rtems-test. We can see that gdb.cfg is open 52 times which seems reasonable give a 56 core machine.

I did the rtems-test run in one terminal and the lsof commands in another.

$ lsof -p 1232936 | grep gdb.cfg | wc -l
52

But the process has an increasing number of pipes open as the tests continue to execute.

[joel@gitlab ~]$ lsof -p 1232936 | grep pipe | wc -l
776
[joel@gitlab ~]$ lsof -p 1232936 | grep pipe | wc -l
819
[joel@gitlab ~]$ lsof -p 1232936 | grep pipe | wc -l
842
[joel@gitlab ~]$ lsof -p 1232936 | grep pipe | wc -l
943
[joel@gitlab ~]$ lsof -p 1232936 | grep pipe | wc -l
951

At ~55/675 tests, there are ~165 pipes in the lsof. At ~160/675 tests, there are ~300 pipes in the lsof. It continues to increase.

All go away when the rtems-test fails and exits.

The number of file descriptors open for rtems-test do NOT increase when using SIS.

Steps to reproduce

You should be able to see the number of pipe file descriptors increase on any multi-core machine when using a gdb based simulator. But there must be some mechanism that is able to reclaim them that is able to run. It must not be happening in the higher core environment.

Pre-set options

Author: Joel Sherrill

2024-10-15T21:29:23.567Z

changed the description

Author: Kinsey Moore

2024-10-29T01:26:47.904Z

Interesting, I can’t actually reproduce this on my 8c16t machine when running through all the sptests. It sits at exactly 3x16 pipes and varies a bit around 1x16 gdb.cfg descriptors. I’ll have to go try on the actual machine that Joel was using.

Author: Amar Takhar

2024-10-29T01:26:47.875Z

What OS? Should probably start collecting platform details to see what we’re testing on

Author: Kinsey Moore

2024-10-29T01:55:57.260Z

The OS I just tested on is Debian Bookworm (headless). The machine that demonstrates the issue is Rocky 9.4 (not headless). I was able to cause something to happen on the 8c16t system by setting –jobs=52 which caused the numbers to fluctuate up and down with a peak around 300 pipes which is twice the number that should be visible if things are getting cleaned up in a timely manner. The Rocky machine is also running gitlab, so that may also have an effect.

Author: Amar Takhar

2024-10-29T02:07:31.042Z

Have you taken a look to see how busy the disk is? Is the machine heavily CPU bound? It may be calling for them to be cleaned up but too busy to do it. No idea just throwing ideas out. Memory/swapping may be an issue too.

Author: Kinsey Moore

2024-10-29T02:39:38.221Z

I found a recommendation that when calling proc.kill(), you should follow it with proc.communicate() to ensure that all I/O is finalized and culled. This seems to keep the pipe count to almost exactly 3x jobs, but exposes other problems on my server. The 28c56t machine is not under any load but gitlab, but seems to be less performant than my server is. AArch64 toolchain build times are: * 8c16t(personal server): 13.75m * 28c56t: 22m * 8c16t VM(virtualbox on a laptop): 16m

Something is definitely going on on that machine.

Once I fixed the proc.communicate() issue, other issues start popping up on my server when the tester is coerced to run at 52 jobs:

[232/675] p:152 f:4   u:3   e:21  I:0   B:3   t:0   L:0   i:0   W:0   | powerpc/psim: psxcancel01.exe
error: gdb.cfg:54: macro '%{rtems_version}' not found
error: gdb.cfg:54: macro '%{rtems_version}' not found
warning: switched to dry run due to errors
error: gdb.cfg:54: macro '%{rtems_version}' not found
error: gdb.cfg:54: macro '%{rtems_version}' not found
error: gdb.cfg:54: macro '%{rtems_version}' not found
error: gdb.cfg:54: macro '%{rtems_version}' not found
error: gdb.cfg:60: macro '%{rtems_version}' not found
error: gdb.cfg:60: macro '%{rtems_version}' not found
error: gdb.cfg:60: macro '%{rtems_version}' not found
error: gdb.cfg:60: macro '%{rtems_version}' not found
[233/675] p:153 f:4   u:3   e:21  I:0   B:3   t:0   L:0   i:0   W:0   | powerpc/psim: psxchroot01.exe
...
[249/675] p:170 f:4   u:4   e:21  I:0   B:3   t:0   L:0   i:1   W:0   | powerpc/psim: psxfatal01.exe
error: config error: gdb.cfg:36: No 'target' defined
[253/675] p:170 f:4   u:4   e:21  I:0   B:3   t:0   L:0   i:1   W:0   | powerpc/psim: psxfile01.exe
[251/675] p:170 f:4   u:4   e:21  I:0   B:3   t:0   L:0   i:1   W:0   | powerpc/psim: psxfchx01.exe

Author: Amar Takhar

2024-10-29T02:46:50.979Z

Interesting what area of the code did you put the communicate() in? Are we using popen to open the process to send data you’re right that you typically do want to call communicate() but I wonder if something is getting overwritten or we don’t have timeouts set.

Author: Chris Johns

2024-10-29T02:58:49.155Z

Is the macro not found error new?

Author: Kinsey Moore

2024-10-29T03:04:21.617Z

It doesn’t show up if I run at 16 jobs (with or without the patch).

Author: Kinsey Moore

2024-10-29T03:09:54.736Z

Running at 52 jobs without the proc.communicate() patch shows different errors and backtraces such as:

Traceback (most recent call last):
AttributeError: 'NoneType' object has no attribute 'target_end'

AttributeError: 'NoneType' object has no attribute 'target_end'

During handling of the above exception, another exception occurred:

File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
File "/media/32b58c68-8211-4697-9049-c4881aa9125e/rtems-dev/rtems-tools/tester/rt/config.py", line 487, in capture
Traceback (most recent call last):
self.run()
File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
During handling of the above exception, another exception occurred:
File "/usr/lib/python3.11/threading.py", line 975, in run
self.process.target_end()

^^^^^^^^^^^^^^^^^^^^^^^
self.run()
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.11/threading.py", line 975, in run
File "/media/32b58c68-8211-4697-9049-c4881aa9125e/rtems-dev/rtems-tools/rtemstoolkit/execute.py", line 256, in _readthread
AttributeError: 'NoneType' object has no attribute 'target_end'

Author: Chris Johns

2024-10-29T05:01:32.044Z

The reader is stuck?

Author: Gedare Bloom

2024-12-04T21:40:37.719Z

Author: Chris Johns

2025-10-02T22:36:49.501Z

added rtems#15 as parent epic

Author: Chris Johns

2025-10-02T22:37:50.749Z

assigned to @opticron

Author: Kinsey Moore

2025-10-12T19:04:49.028Z

assigned to @joel and unassigned @opticron

Author: Chris Johns

2026-07-31T00:09:29.102Z

set status to Needs funding

Author: Chris Johns

2026-07-31T00:09:54.359Z

set status to In progress

Merge Requests

91 - tester: Avoid relying on external telnetlib

Id

91

State

merged

Merged by

Kinsey Moore

Merged at

2026-08-14T18:48:16.741Z

Assignee(s)

Kinsey Moore

Created

2026-08-14T18:46:23.193Z

Updated

2026-08-14T18:48:16.731Z

Milestone

6.3

Labels

tool::test

Link

https://gitlab.rtems.org/rtems/tools/rtems-tools/-/merge_requests/91

Commit

adf3e254d08f678d3625a4edccb2ac3485d96519

Summary

tester: Avoid relying on external telnetlib

Kinsey Moore authored 2 weeks ago The telnetlib package has been deprecated and removed from the standard library. This pulls in its last source version so that it can be distributed until a better solution is found.

Closes #40

Generative AI

None

Author: Kinsey Moore

2026-08-14T18:46:23.367Z

assigned to @opticron

Author: Kinsey Moore

2026-08-14T18:46:46.571Z

changed target branch from main to 6

Author: Kinsey Moore

2026-08-14T18:47:22.846Z

changed title from

gitlab: Add link to CI

to

tester: Avoid relying on external telnetlib

Author: Kinsey Moore

2026-08-14T18:47:22.875Z

changed the description

Author: Kinsey Moore

2026-08-14T18:47:29.447Z

enabled an automatic merge when all merge checks for adf3e254d08f678d3625a4edccb2ac3485d96519 pass

Author: Joel Sherrill

2026-08-14T18:48:11.819Z

approved this merge request

Author: Gedare Bloom

2026-08-14T18:48:13.135Z

approved this merge request