Teensy 4.1 Unit Testing Issue

dpd3788 · April 28, 2021, 1:19am

I am trying to run a demo unit test on the teensy 4.1. The unit test stalls after upload with the following message:

Processing * in teensy41 environment
---------------------------------------------------------------
Building...
Uploading...
Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

The platformio.ini file is configured as:

[env:teensy41]
platform = teensy
board = teensy41
framework = arduino
upload_protocol = teensy-cli

The test file is taken from one of the unit test demos, not doing anything fancy:

#include <Arduino.h>
#include <unity.h>

void test_led_builtin_pin_number(void) {
    TEST_ASSERT_EQUAL(13, LED_BUILTIN);
}

void test_led_state_high(void) {
    digitalWrite(LED_BUILTIN, HIGH);
    TEST_ASSERT_EQUAL(HIGH, digitalRead(LED_BUILTIN));
}

void test_led_state_low(void) {
    digitalWrite(LED_BUILTIN, LOW);
    TEST_ASSERT_EQUAL(LOW, digitalRead(LED_BUILTIN));
}

void setup() {
    // NOTE!!! Wait for >2 secs
    // if board doesn't support software reset via Serial.DTR/RTS
    delay(3000);

    UNITY_BEGIN();    // IMPORTANT LINE!
    RUN_TEST(test_led_builtin_pin_number);

    pinMode(LED_BUILTIN, OUTPUT);
}

uint8_t i = 0;
uint8_t max_blinks = 5;

void loop() {
    if (i < max_blinks)
    {
        RUN_TEST(test_led_state_high);
        delay(500);
        RUN_TEST(test_led_state_low);
        delay(500);
        i++;
    }
    else if (i >= max_blinks) {
      UNITY_END(); // stop unit testing
    }
}

The program successfully uploads, and the board blinks N times as expected but does not print out any test results. If I open the serial monitor while the program is running I can see testing outputs flash repeatedly:

-----------------------
11 Tests 0 Failures 0 Ignored
OK

Any suggestions? I have tried on both a Windows and Linux machine with the same results. Ran the same tests on the esp32 with success.

pio core version is 5.2.0a6
teensy version is 4.12.0

maxgerhardt · April 28, 2021, 10:06am

What happens if you restructure setup() and loop() to

uint8_t i = 0;
uint8_t max_blinks = 5;

void setup() {
    // NOTE!!! Wait for >2 secs
    // if board doesn't support software reset via Serial.DTR/RTS
    delay(3000);

    UNITY_BEGIN();    // IMPORTANT LINE!
    RUN_TEST(test_led_builtin_pin_number);

    pinMode(LED_BUILTIN, OUTPUT);

    for(i = 0; i < max_blinks; i++)
    {
        RUN_TEST(test_led_state_high);
        delay(500);
        RUN_TEST(test_led_state_low);
        delay(500);
    }
    UNITY_END(); // stop unit testing
}

void loop() {}

?

dpd3788 · April 28, 2021, 12:38pm

Same thing unfortunately.

dpd3788 · April 28, 2021, 3:58pm

I also tried manually setting the test_port to the port of the device. This results in a “could not open port” error

Uploading...
Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

could not open port 'COM5': FileNotFoundError(2, 'The system cannot find the file specified.', None, 2)
    ============================================================================================ [FAILED] Took 3.85 seconds ============================================================================================
    Test    Environment    Status    Duration
    ------  -------------  --------  ------------
    *       teensy41       FAILED    00:00:03.854
    ====================================================================================== 1 failed, 0 succeeded in 00:00:03.854 ====================

maxgerhardt · April 28, 2021, 4:38pm

Hm yes I can reproduce this with my Teensy 4.0 (that I can also treat as a 4.1 with identical config as yours…)

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Building...
Uploading...
Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

the problem is that after the firmware is uploaded, the MCU reboots and it then initializes its USB peripheral to act as a USB serial device, then Windows has to recognize it and initialize it etc. This takes time, but the Testing.., aka looking at the serial port output, seems to be happening too early. So basically it’s the issue as known in Add test start delay option · Issue #3742 · platformio/platformio-core · GitHub.

Boards with a fixed USB-serial adapter chip don’t have this problem.

Solution per SigFox MKR1200 Unit Testing - If you don't see any output for the first 10 secs, please reset board - #4 by isa56k works for me.

Aka, when I do

[env:teensy41]
platform = teensy
board = teensy41
framework = arduino
upload_protocol = teensy-cli
extra_scripts = extra_script.py

with extra_script.py in the root as

Import("env")

def after_upload(source, target, env):
    print("Delay while uploading...")
    import time
    time.sleep(5)
    print("Done!")

env.AddPostAction("upload", after_upload)

and increasing the delay(3000); to delay(5000); in the firmware, I get…

> Executing task in folder pico_test_2: C:\Users\Max\AppData\Local\Programs\Python\Python38\Scripts\platformio.exe test --environment teensy41 <

Verbose mode can be enabled via `-v, --verbose` option
Collected 1 items

Processing * in teensy41 environment
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Building...
Uploading...
Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

test\test1.cpp:24:test_led_builtin_pin_number   [PASSED]
test\test1.cpp:35:test_led_state_high   [PASSED]        
test\test1.cpp:37:test_led_state_low    [PASSED]
test\test1.cpp:35:test_led_state_high   [PASSED]
test\test1.cpp:37:test_led_state_low    [PASSED]
test\test1.cpp:35:test_led_state_high   [PASSED]
test\test1.cpp:37:test_led_state_low    [PASSED]
test\test1.cpp:35:test_led_state_high   [PASSED]
test\test1.cpp:37:test_led_state_low    [PASSED]
test\test1.cpp:35:test_led_state_high   [PASSED]
test\test1.cpp:37:test_led_state_low    [PASSED]
-----------------------
11 Tests 0 Failures 0 Ignored
========================= [PASSED] Took 16.96 seconds =========================

So it’s ‘just’ a timing issue that will hopefully get resolved in the linked issue.

dpd3788 · April 28, 2021, 5:34pm

The only additional change I had to make to your suggestion was to include test_port = COM5

Thanks so much for your help.

richard · June 6, 2021, 3:56pm

I’ve run into this issue on a Teensy4.0. It happens on my desktop PC but not on my laptop. The difference is that my PC has an additional COM port. Presumably the test task tries to connect to this port when the test port disappears. If I use Device Manager to delete the extra port then it all works as expected.

The fix that @maxgerhardt works for me. (Thanks Max!) FWIW I found that the delay in extra_script.py could be much smaller; 0.35s on my PC.

cheers,
Richard

ivankravets · April 29, 2022, 7:12pm

See the solution to this issue at

github.com/platformio/platformio-core

Add test start delay option

opened 09:20PM - 18 Nov 20 UTC

closed 07:13PM - 29 Apr 22 UTC

maxgerhardt

help wanted unit testing

# Solution Please add to your project a [POST Extra Script](https://docs.plat…formio.org/en/latest/scripting/index.html) with the following contents: **platformio.ini** ```ini [env:seeed_wio_terminal] platform = atmelsam board = seeed_wio_terminal framework = arduino extra_scripts = post:extra_script.py ``` **extra_script.py** ```python import time Import("env") if "test" in env.GetBuildType(): env.AddPostAction("upload", lambda *_, **__: time.sleep(2)) ``` P.S: Do not forget to upgrade to the latest PlatformIO Core via `pio upgrade --dev`. # Related issues: - https://community.platformio.org/t/teensy-4-1-unit-testing-issue/21033/5 - https://github.com/platformio/platformio-core/issues/3956 ### Configuration **Operating system**: Windows 10 x64 **PlatformIO Version** (`platformio --version`): 5.0.4a2 ### Description of problem Reference to topic where this popped up: https://community.platformio.org/t/unit-test-wio-terminal-could-not-open-port-while-testing/17180/ A user wants to do unit testing on a WIO terminal, which has a ~~SAMD21G18A~~ ATSAMD51P19A microcontroller onboard. This microcontroller implements a USB CDC serial connection. When a new firmware is uploaded, or the microcontroller is being reset, the USB connection is temporarily lost, then re-established. The problem is then when testing, PlatformIO does a Build -> Upload -> Test, where there is no delay between uploading and testing when using this specific microcontroller. A problem was problem was observed that when using Linux, after the upload is done, the microcontroller re-enumerates twices on the USB bus, and at the moment the "Test" part of the unit tests wants to access `/dev/ttyACM0`, the Linux kernel or subsystems have not yet fully re-enumerated the device and it's currently owned by root at this point. Then accessing it fails with `[Errno 13] could not open port /dev/ttyACM0: [Errno 13] Permission denied: '/dev/ttyACM0'`. A few milliseconds after it ([exact timing log is here](https://community.platformio.org/t/unit-test-wio-terminal-could-not-open-port-while-testing/17180/9?u=maxgerhardt)), Linux is done processing the device and now the user properly owns `/dev/ttyACM0` and access to it would go through. That's at least how I read these logs. However, since PlatformIO does not have a configurable delay after uploading, or equivalently before testing, `pio test` will always run into this problem. So it would be nice to have such an option to enable these use cases where the to-be-tested MCU is the USB serial port and does some re-enumeration stuff after uploading. #### Steps to Reproduce 1. Create a project for the `seeed_wio_terminal` 2. Add standard unit testing code to it 3. Execute `pio test` ### Actual Results Permission denied error on the serial device. ### Expected Results Testing goes through. ### If problems with PlatformIO Build System: **The content of `platformio.ini`:** ```ini [env:seeed_wio_terminal] platform = atmelsam board = seeed_wio_terminal framework = arduino ``` **Source file to reproduce issue:** ```cpp #include <Arduino.h> #include <unity.h> void TestConvertTemperatureToString(void) { TEST_ASSERT_EQUAL(32, 32); } void setup() { // NOTE!!! Wait for >2 secs // if board doesn't support software reset via Serial.DTR/RTS delay(5000); UNITY_BEGIN(); RUN_TEST(TestConvertTemperatureToString); UNITY_END(); } void loop() { } ``` ### Additional info