PlatformIO Community

Embedded unit tests fail after tearDown?

Hello,

I try to create unit tests, but I can’t get it working. I’ve read the documentation several times, but I can’t find the problem.
In order to limit the sources of errors as much as possible, I made a simple case.

The compilation goes well, the upload goes well, the tests do not succeed.

Here is my configuration: Linux (xubuntu xfce), Clion as IDE, Teensy 4.1, Unity as test framework. PlatformIO Core, version 6.1.4a1

I have read several posts on the forum and issues on Github about similar symptoms. I have tested the solutions described there without success.
for example, I tried this without success: Teensy 4.1 Unit Testing Issue

My platformio.ini configuration file is the following (I tried several combinations and left in comment what I had already tried):

[env:test]
platform = teensy
board = teensy41
framework = arduino
upload_protocol = teensy-cli
upload_port = /dev/ttyACM0
test_framework = unity
lib_deps =
    throwtheswitch/Unity@^2.5.2
test_build_src = false
extra_scripts = post:test/after_upload.py
;extra_scripts = post:test/add_delay.py
;test_port = /dev/ttyACM0
;test_speed = 9600
;monitor_port = /dev/ttyACM0
;monitor_dtr = 1

Extra scripts are:
after_upload.py

###
# Spongman suggestion
# https://github.com/platformio/platformio-core/issues/3742#issuecomment-1003454439
###

import time
import serial

Import("env")

def after_upload(source, target, env):
    port = env.GetProjectOption("monitor_port")
    print("waiting for " + port + " ...")
    while True:
        try:
            s = serial.Serial(port)
            break
        except:
            pass
    print("after_upload script: Done!")

env.AddPostAction("upload", after_upload)

add_delay.py

## 
# Max Gerhardt solution
# https://github.com/platformio/platformio-core/issues/3742
##

import time

Import("env")

if "test" in env.GetBuildType():
    print(env.GetBuildType())
    print("add_delay script: Done!")
    env.AddPostAction("upload", lambda *_, **__: time.sleep(2))

My code to be tested in /lib/ClassToTest/ClassToTest.h:

#ifndef TEST_TESTING_CLASSTOTEST_H
#define TEST_TESTING_CLASSTOTEST_H
#include "Arduino.h"

class ClassToTest {
private:
    boolean _result;
public:
    ClassToTest();

    boolean myFunction() const{
        Serial.println("ClassToTest::myFunction called");
        return _result;
    }

    boolean mySecondFunction() const{
        Serial.println("ClassToTest::mySecondFunction called");
        return _result;
    }
};

#endif //TEST_TESTING_CLASSTOTEST_H

ClassToTest::ClassToTest() : _result(true) {}

My test code in /test/test_ClassToTest.cpp:

#include "Arduino.h"
#include "unity.h"
#include "ClassToTest.h"

ClassToTest ctt;

void setUp(void) {
    ctt = ClassToTest();
    Serial.println("test setUp");
}

void tearDown(void) {
    Serial.println("test tearDown");
}

void test_myFunction(void) {
    boolean result = ctt.myFunction();
    TEST_ASSERT_EQUAL_MESSAGE(true,result, "test myFunction message");
    Serial.println("test myFunction");
}

void test_mySecondFunction(void) {
    // TEST NEVER PASS HERE, WHY?!
    boolean result = ctt.mySecondFunction();
    TEST_ASSERT_EQUAL_MESSAGE(true,result, "test mySecondFunction message");
    Serial.println("test mySecondFunction");
}

int runUnityTests(void) {
    Serial.println("runUnityTests");
    UNITY_BEGIN();
    RUN_TEST(test_myFunction);
    RUN_TEST(test_mySecondFunction);
    // NEVER REACH THAT LINE
    Serial.println("AFTER RUN_TEST");
    int result = UNITY_END();
    Serial.print(result);
    return result;
}


void setup() {
    Serial.begin(9600); //suggestion: https://github.com/platformio/platformio-core/issues/3742#issuecomment-806186699
    delay(5000);
    Serial.println("Starting Tests");
    runUnityTests();
    // NEVER REACH THAT LINE
    Serial.println("Test done");
}

void loop() {
    // LED DON'T BLINK
    digitalWrite(13, HIGH);
    delay(100);
    digitalWrite(13, LOW);
    delay(500);
}

The terminal output is:

(base) _@_:~/CLionProjects/test_testing$ pio test -e "test" -vvv
Collected 1 tests (*)

Processing * in test environment
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Building & Uploading...
CONFIGURATION: https://docs.platformio.org/page/boards/teensy/teensy41.html
PLATFORM: Teensy (4.16.0) > Teensy 4.1
HARDWARE: IMXRT1062 600MHz, 512KB RAM, 7.75MB Flash
DEBUG: Current (jlink) External (jlink)
PACKAGES: 
 - framework-arduinoteensy @ 1.156.0 (1.56) 
 - tool-teensy @ 1.155.0 (1.55) 
 - toolchain-gccarmnoneeabi @ 1.50401.190816 (5.4.1)
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 94 compatible libraries
Scanning dependencies...
Dependency Graph
|-- ClassToTest (License: Unknown, Path: /home/CLionProjects/test_testing/lib/ClassToTest)
|-- Unity @ 2.5.2 (License: MIT, Path: /home/CLionProjects/test_testing/.pio/libdeps/test/Unity)
Building in test mode
arm-none-eabi-g++ -o .pio/build/test/test/test_ClassToTest.cpp.o -c -fno-exceptions -felide-constructors -fno-rtti -std=gnu++14 -Wno-error=narrowing -fpermissive -fno-threadsafe-statics -Wall -ffunction-sections -fdata-sections -mthumb -mcpu=cortex-m7 -nostdlib -fsingle-precision-constant -mfloat-abi=hard -mfpu=fpv5-d16 -O2 -DPLATFORMIO=60104 -D__IMXRT1062__ -DARDUINO_TEENSY41 -DUSB_SERIAL -DARDUINO=10805 -DTEENSYDUINO=156 -DCORE_TEENSY -DF_CPU=600000000 -DLAYOUT_US_ENGLISH -DPIO_UNIT_TESTING -DUNIT_TEST -DUNITY_INCLUDE_CONFIG_H -Iinclude -Isrc -I.pio/libdeps/test/Unity/src -Itest -Ilib/ClassToTest -Itest -I/home/.platformio/packages/framework-arduinoteensy/cores/teensy4 test/test_ClassToTest.cpp
arm-none-eabi-g++ -o .pio/build/test/libcc1/ClassToTest/ClassToTest.cpp.o -c -fno-exceptions -felide-constructors -fno-rtti -std=gnu++14 -Wno-error=narrowing -fpermissive -fno-threadsafe-statics -Wall -ffunction-sections -fdata-sections -mthumb -mcpu=cortex-m7 -nostdlib -fsingle-precision-constant -mfloat-abi=hard -mfpu=fpv5-d16 -O2 -DPLATFORMIO=60104 -D__IMXRT1062__ -DARDUINO_TEENSY41 -DUSB_SERIAL -DARDUINO=10805 -DTEENSYDUINO=156 -DCORE_TEENSY -DF_CPU=600000000 -DLAYOUT_US_ENGLISH -DPIO_UNIT_TESTING -Ilib/ClassToTest -I/home/.platformio/packages/framework-arduinoteensy/cores/teensy4 lib/ClassToTest/ClassToTest.cpp
arm-none-eabi-gcc-ar rc .pio/build/test/libcc1/libClassToTest.a .pio/build/test/libcc1/ClassToTest/ClassToTest.cpp.o
arm-none-eabi-gcc-ar -s .pio/build/test/libcc1/libClassToTest.a
arm-none-eabi-g++ -o .pio/build/test/firmware.elf -T imxrt1062_t41.ld -Wl,--gc-sections,--relax -mthumb -mcpu=cortex-m7 -Wl,--defsym=__rtc_localtime=1658357985 -mfloat-abi=hard -mfpu=fpv5-d16 -O2 .pio/build/test/test/test_ClassToTest.cpp.o -L/home/.platformio/packages/framework-arduinoteensy/cores/teensy4 -L.pio/build/test -Wl,--start-group .pio/build/test/libcc1/libClassToTest.a .pio/build/test/libf88/libUnity.a .pio/build/test/libFrameworkArduino.a -larm_cortexM7lfsp_math -lm -lstdc++ -Wl,--end-group
MethodWrapper(["checkprogsize"], [".pio/build/test/firmware.elf"])
teensy_size .pio/build/test/firmware.elf
arm-none-eabi-objcopy -O ihex -R .eeprom .pio/build/test/firmware.elf .pio/build/test/firmware.hex
teensy_size: Memory Usage on Teensy 4.1:
teensy_size:   FLASH: code:17552, data:6040, headers:9172   free for files:8093700
teensy_size:    RAM1: variables:6784, code:14880, padding:17888   free for local variables:484736
teensy_size:    RAM2: variables:12384  free for malloc/new:511904
<lambda>(["upload"], [".pio/build/test/firmware.hex"])
AVAILABLE: jlink, teensy-cli, teensy-gui
CURRENT: upload_protocol = teensy-cli
teensy_reboot -s
teensy_loader_cli -mmcu=imxrt1062 -w -s -v .pio/build/test/firmware.hex
Teensy Loader, Command Line, Version 2.2
Read ".pio/build/test/firmware.hex": 32768 bytes, 1.6% usage
Error opening USB device: Resource temporarily unavailable
Waiting for Teensy device...
 (hint: press the reset button)
Found HalfKay Bootloader
Read ".pio/build/test/firmware.hex": 32768 bytes, 1.6% usage
Programming.............................
Booting

Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

Starting Tests
runUnityTests
test setUp
ClassToTest::myFunction called
test myFunction
test tearDown
device reports readiness to read but returned no data (device disconnected or multiple access on port?)
-------------------------------------------------------------------------- test:* [ERRORED] Took 17.62 seconds --------------------------------------------------------------------------

======================================================================================== SUMMARY ========================================================================================
Environment    Test    Status    Duration
-------------  ------  --------  ------------
teensy41       *       SKIPPED
test           *       ERRORED   00:00:17.618
======================================================================= 1 test cases: 0 succeeded in 00:00:17.618 =======================================================================

When I use pio device monitor, the output loops with:

Disconnected (device reports readiness to read but returned no data (device disconnected or multiple access on port?))
Reconnecting to /dev/ttyACM0 .   Connected!
Starting Tests
runUnityTests
test setUp
ClassToTest::myFunction called
test myFunction
test tearDown
Disconnected (device reports readiness to read but returned no data (device disconnected or multiple access on port?))
Reconnecting to /dev/ttyACM0 .   Connected!
Starting Tests
runUnityTests
test setUp
ClassToTest::myFunction called

I can’t figure out what I’m not doing right.
Do you have any suggestions?

Could you point me to the docs where we ask users to add these lines? Please note that this is not the root of the problem. I just want to understand where people get these lines as they can lead to other potential problems.

device reports readiness to read but returned no data (device disconnected or multiple access on port?)

It looks like a Teensy issue PJRC (Teensy) Forum


Could you run in verbose mode? pio test -vvv?

Hello Ivan,
Thank you for the answer.

I don’t remember reading it somewhere. Actually it was a message in the terminal I had that the library was not found. I had a similar message about Wire.h and there was a link to PlatformIO Registry
Searching for Unity on registry.platformio.org, I found this PlatformIO Registry where it explains how to install it.
Sorry that this is not a good practice. I don’t know why Clion gave me the message that Unity was not installed.
Based on what you said, I removed thoses line, even more I have recreated a new Clion project.
I created a new Clion test configuration.
I copy and paste the ClassToTest class in Lib/ClassToTest and the test_ClassToTest.cpp script in the test/ folder.
I edited the platformio.ini file with only the following values:

[env:teensy41]
platform = teensy
board = teensy41
framework = arduino
test_framework = unity

By running the pio command:

 pio test -e "teensy41" -vvv

The tests are passed successfully:

Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

Starting Tests
runUnityTests
test setUp
ClassToTest::myFunction called
test myFunction
test tearDown
test/test_ClassToTest.cpp:32:test_myFunction:PASS
test setUp
ClassToTest::mySecondFunction called
test mySecondFunction
test tearDown
test/test_ClassToTest.cpp:33:test_mySecondFunction:PASS
AFTER RUN_TEST

-----------------------
2 Tests 0 Failures 0 Ignored 
OK
0Test done

------------------------------------------------------------------------- teensy41:* [PASSED] Took 9.63 seconds -------------------------------------------------------------------------

======================================================================================== SUMMARY ========================================================================================
Environment    Test    Status    Duration
-------------  ------  --------  ------------
teensy41       *       PASSED    00:00:09.626
======================================================================= 2 test cases: 2 succeeded in 00:00:09.626 =======================================================================

I don’t understand why…

I went back to my first project and copied the (ultra light) contents of the platformio.ini file from the project that passed the tests. As on this project the tests still don’t pass, although I thougth there was nothing different.
So I compared the file trees of both project and deleted every generated files in the project whose tests did not pass.

Then I ran the test command

pio test -e "teensy41" -vvv

again and it worked as expected, all tests are passed correctly.

I can’t figure out what happened.

maybe: “less is more”

Could you try a few attempts?

  1. pio test -e "teensy41"
  2. pio test -e "teensy41" -v