Issue with testing with unity on esp32s2

PlatformIO Core
1

I am not sure if this is the right place to share this, but I ran into an interesting issue with using platformio test and unity with the lolin_s2_mini board. I have a simple workaround, but it seems like there is a possible fix?

Problem

The unity test output is not sent to the serial port, so the test runner doesn’t work

Apparent Cause

The basic issue is the esp32s2 framework contains a unity_config.h in framework-arduinoespressif32/tools/sdk/esp32s2/include/unity/include, and this is being picked up first in the build.

This default config file directs the unity output to stdout, and not the serial port where it needs to go.

The auto-generated unity_config.h file in .pio/build/lolin_s2_mini/unity_config is correct

Why Does This Happen?

The problem seems to be when compiling .pio/libdeps/lolin_s2_mini/Unity/src/unity.c, the include file paths cause the wrong unity_config.h to be included. Interestingly, the include file paths when compiling the test file appear to be correct.

Workaround

Remove ~/.platformio/packages/framework-arduinoespressif32/tools/sdk/esp32s2/include/unity/include/unity_config.h

platformio test works as expected.

Details

When compiling the test case, test/embedded/test_blink/blink.cpp, the include paths are as follows:

-Iinclude -Isrc -I.pio/libdeps/lolin_s2_mini/Unity/src -I.pio/build/lolin_s2_mini/unity_config … etc

When compiling .pio/libdeps/lolin_s2_mini/Unity/src/unity.c (which needs the unity_config.h), the include paths are as follows:

-I.pio/libdeps/lolin_s2_mini/Unity/src -I/home/jim/.platformio/packages/framework-arduinoespressif32/tools/sdk/esp32s2/include/unity/include -I.pio/build/lolin_s2_mini/unity_config … etc

Possible Fix

It seems that changing the include path when compiling unity.c would address the issue.

2

Please show the contents of your platformio.ini and specify the espressif32 platform version used (if not specified in platformio.ini).

The directory .platformio/packages/framework-arduinoespressif32/tools/sdk does not exist on my system. Where does it come from?

3

Wow, thanks for the quick response!

I believe the directory ~/.platformio/packages/framework-arduinoespressif32 is where the VSCode PlatformIO extension downloaded the support package for the esp32 boards on my Debian 13 system. The framework looks to be managed by espressif. The file package.json contains:

{
  "name": "framework-arduinoespressif32",
  "version": "3.20017.241212+sha.dcc1105b",
  "description": "Arduino Wiring-based Framework for the Espressif ESP32, ESP32-S and ESP32-C series of SoCs",
  "keywords": [
    "framework",
    "arduino",
    "espressif",
    "esp32"
  ],
  "license": "LGPL-2.1-or-later",
  "repository": {
    "type": "git",
    "url": "https://github.com/espressif/arduino-esp32"
  }
}

And the file https://espressif.github.io/arduino-esp32/package_esp32_index.json has a lot of information about it. I don’t really understand how the package management works, but it is really cool!

My platform.ini file is as follows:

[env:lolin_s2_mini]
platform = espressif32
board = lolin_s2_mini
framework = arduino
monitor_speed = 115200
check_tool = clangtidy
test_filter = embedded/test_us
test_port = /dev/ttyACM0

I don’t know why espressif would be including the unitity_config.h in the SDK, but it might be easy to avoid when compiling uniity.c

4

Sorry, my fault. I don’t have the folders because I’m using pioarduino’s espressif32 platform to get the latest Espressif Arduino Framework (currently 3.3.4). Here the folder structure is different and the unity files are located at

.platformio/packages/framework-arduinoespressif32-libs/esp32s2/include/unity

Do you need to stick to Espressif Arduino 2.0.17?

If not, try pioarduino:

platform = https://github.com/pioarduino/platform-espressif32/releases/download/stable/platform-espressif32.zip
5

Thank you for the reply.

I have moved to the new pioarduino platform, and it generally looks good, but the problem with Unity output remains.

This framework includes a default unity_config.h:

ls ~/.platformio/packages/framework-arduinoespressif32-libs/esp32s2/include/unity/include
priv                    unity_test_runner.h       unity_test_utils_memory.h
unity_config.h          unity_test_utils_cache.h
unity_fixture_extras.h  unity_test_utils.h

and when compiling ./libdeps/lolin_s2_mini/Unity/src/unity.c, the unity include directory in the framework is included before the directory with the generated unity_config.h:

-I.pio/libdeps/lolin_s2_mini/Unity/src 
-I/home/jim/.platformio/packages/framework-arduinoespressif32-libs/esp32s2/include/unity/include 
-I.pio/build/lolin_s2_mini/unity_config

This causes the output function to be defined as uniity_putc instead of unityOutputChar supplied in .pio/build/lolin_s2_mini/unity/unity_config.cpp

Removing or renaming the unity_config.h file in the framework fixes the problem. But so would changing the order of the 2nd and 3rd include arguments when compiling unity.c

That seems to be what is wanted? But I am not very familiar with this area. I have a workaround, but there may be an opportunity to fix for all.

Thanks

1 Like
6

I don’t get the exact problem here. (I’m not an expert on Unity)

The LOLIN_S2_MINI is using the ESP32’s builtin USB port for Serial output (USB CDC).
In the lolin_s2_mini.json you’ll find:
"-DARDUINO_USB_CDC_ON_BOOT=1" which “routes” the output of the Serial object to the native USB port (USB CDC). Also printf and the output of unity is correctly routed to USB CDC.

What do you mean exactly by “Serial port”? USB CDC or UART ?

7

Good questions. This stuff is a bit obscure and in working to setup unit tests, I am just learning about it. The functionality is a bit hard to follow.

I am using the USB CDC interface. The unity macros and functions write test results to an output device. The platformio macros send the output with Serial.write(), by default going to the USB CDC, and the test runner reads the output.

A good way top understand what is happening is by running the monitor against the target with the test image loaded and comparing the output of a test with and without the workaround in place.

In the first case, where I don’t remove the unity_config.h file from framework-arduinoespressif32-libs, we see the output of my Serial.println() statements added for informational purposes, but not the test results.

 task: platformio device monitor --environment lolin_s2_mini 

--- Terminal on /dev/ttyACM0 | 115200 8-N-1
--- Available filters and text transformations: colorize, debug, default, direct, esp32_exception_decoder, hexlify, log2file, nocontrol, printable, send_on_enter, time
--- More details at https://bit.ly/pio-monitor-filters
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H
Running blink test.
calling run blink test.
Done running blink test.

When I rename the file, clean, and use the test runner to build and load the test onto the target, the monitor task shows the test results reported with the informational statements. The test runner reads the output and declares the test as passed.

Executing task: platformio device monitor --environment lolin_s2_mini 

--- Terminal on /dev/ttyACM0 | 115200 8-N-1
--- Available filters and text transformations: colorize, debug, default, direct, esp32_exception_decoder, hexlify, log2file, nocontrol, printable, send_on_enter, time
--- More details at https://bit.ly/pio-monitor-filters
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H
Running blink test.
calling run blink test.
test/embedded/test_blink/blink.cpp:38:testBlink:PASS
Done running blink test.

-----------------------
1 Tests 0 Failures 0 Ignored

This is a bit confusing I think due to the way the output functions are defined and picked up through unity_config.h. But in the end, without the workaround the test build is picking up the platform unity_config.h and output functions, and not the PlatformIO defined output functions, because of the order of the include directories. And those functions don’t send the test result to the expected output device. Interestingly, the include order is correct when compiling the test file (in this case blink.cpp), but not the file .pio/libdeps/lolin_s2_mini/Unity/src/unity.c, which is the file that needs the macros.

I hope this helps explain the issues. The setup that PlatformIO does for unity seems a bit complex and unique, but very handy.

8

I have absolutely no issues here.
Since I don’t own an ESP32-S2 I’m using my ESP32-S3 DevKitC-1 which have both “ports” (UART and USB CDC). I’m using the USB CDC Port: -DARDUINO_USB_CDC_ON_BOOT=1. So the scenario should be identical to yours.

platformio.ini:

[env:esp32dev]
platform = https://github.com/pioarduino/platform-espressif32/releases/download/55.03.33/platform-espressif32.zip
board = esp32-s3-devkitc-1
framework = arduino
monitor_speed = 115200
build_flags =
  -DARDUINO_USB_CDC_ON_BOOT=1

./test/test_main.cpp

#include <Arduino.h>
#include <unity.h>

void setUp(void) {
    // set stuff up here
}

void tearDown(void) {
    // clean stuff up here
}

void test_led_builtin_pin_number(void) {
    TEST_ASSERT_EQUAL(97, LED_BUILTIN);
}

void test_led_state_high(void) {
    digitalWrite(LED_BUILTIN, HIGH);
    TEST_ASSERT_EQUAL(HIGH, digitalRead(LED_BUILTIN));
}

void test_led_state_low(void) {
    digitalWrite(LED_BUILTIN, LOW);
    TEST_ASSERT_EQUAL(LOW, digitalRead(LED_BUILTIN));
}

void setup() {
    // NOTE!!! Wait for >2 secs
    // if board doesn't support software reset via Serial.DTR/RTS
    delay(2000);
    Serial.begin(115200);
    Serial.println("USB_CDC");

    pinMode(LED_BUILTIN, OUTPUT);

    UNITY_BEGIN();  // IMPORTANT LINE!
    RUN_TEST(test_led_builtin_pin_number);
}

uint8_t i          = 0;
uint8_t max_blinks = 5;

void loop() {
    if (i < max_blinks) {
        RUN_TEST(test_led_state_high);
        delay(500);
        RUN_TEST(test_led_state_low);
        delay(500);
        i++;
    } else if (i == max_blinks) {
        UNITY_END();  // stop unit testing
    }
    delay(10);
}

The board is connected via the native USB port (USB CDC). Output from test:

Executing task: C:\Users\boris\.platformio\penv\Scripts\platformio.exe test --environment esp32dev 

Verbosity level can be increased via `-v, -vv, or -vvv` option
Collected 1 tests

Processing * in esp32dev environment
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building & Uploading...
Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

test/test_main.cpp:36: test_led_builtin_pin_number      [PASSED]
test/test_main.cpp:44: test_led_state_high      [PASSED]
test/test_main.cpp:46: test_led_state_low       [PASSED]
test/test_main.cpp:44: test_led_state_high      [PASSED]
test/test_main.cpp:46: test_led_state_low       [PASSED]
test/test_main.cpp:44: test_led_state_high      [PASSED]
test/test_main.cpp:46: test_led_state_low       [PASSED]
test/test_main.cpp:44: test_led_state_high      [PASSED]
test/test_main.cpp:46: test_led_state_low       [PASSED]
test/test_main.cpp:44: test_led_state_high      [PASSED]
test/test_main.cpp:46: test_led_state_low       [PASSED]
-------------------------------------------------------------------- esp32dev:* [PASSED] Took 32.00 seconds --------------------------------------------------------------------

==================================================================================== SUMMARY ====================================================================================
Environment    Test    Status    Duration
-------------  ------  --------  ------------
esp32dev       *       PASSED    00:00:32.003
================================================================== 11 test cases: 11 succeeded in 00:00:32.003 ==================================================================

How do you start the test?
Do you click on PIO Icon / PROJECT TASKS / [env-name] / Advanced / Test ?

9

Thank you so much for trying to reproduce the problem! (I am running the tests either with the icon in the bottom, the Advanced Tasks, or the command line.) It is terrific to see an example of this working.

I read your previous reply too quickly and missed this:

My tests do not send printf statements to the Serial port. (Edit: clarify just printf, not Serial.printf) If that did happen, I believe using the platform supplied unity macros would work. At this point I can say I didn’t know if it was supposed to do that and have no idea if it is platform dependent. Beyond my limited knowledge.

I do recognize now that perhaps using the platform version of unity_config.h is intentional? In this way the platform package can deliver platform specific unity functions.

I verified that I am using the platform package supplied functions by dumping the symbols from .pio/build/lolin_s2_mini/lib83d/Unity/unity.c.o using the nm command:

...
         U unity_exec_time_start
         U unity_exec_time_stop
         U unity_flush
         U unity_putc
...

I assume that you build would show the same thing. (If not, there must be something else going on)

Perhaps this is desirable for the test cases and I simply need to address the redirection of stdio to the serial port.

Thank you for the assistance!

10

Just to be complete in case anyone is interested, this is what I found for the ESP32S2

from .platformio/packages/framework-arduinoespressif32-libs/esp32s2/sdkconfig

CONFIG_ESP_CONSOLE_UART_DEFAULT=y
# CONFIG_ESP_CONSOLE_USB_CDC is not set
# CONFIG_ESP_CONSOLE_UART_CUSTOM is not set
# CONFIG_ESP_CONSOLE_NONE is not set

So, that seems to explain why I don’t see the Unity output on the USB connection.

11

But the unity output is routed correctly as I have shown above!
It is determined by the buildflag ARDUINO_USB_CDC_ON_BOOT!
If the flag is set to “0” the unity output is routed correctly to the UART port. (I also have tested this).

Perhaps there is a misunderstanding here.

Can you show an example code for your tests?
I am puzzled that your Unity outputs look completely different from mine.

Executing task: platformio device monitor --environment lolin_s2_mini 

--- Terminal on /dev/ttyACM0 | 115200 8-N-1
--- Available filters and text transformations: colorize, debug, default, direct, esp32_exception_decoder, hexlify, log2file, nocontrol, printable, send_on_enter, time
--- More details at https://bit.ly/pio-monitor-filters
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H
Running blink test.
calling run blink test.
test/embedded/test_blink/blink.cpp:38:testBlink:PASS
Done running blink test.

-----------------------
1 Tests 0 Failures 0 Ignored

vs

Processing * in esp32dev environment
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building & Uploading...
Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

test/test_main.cpp:36: test_led_builtin_pin_number      [PASSED]
test/test_main.cpp:44: test_led_state_high      [PASSED]
test/test_main.cpp:46: test_led_state_low       [PASSED]
test/test_main.cpp:44: test_led_state_high      [PASSED]
test/test_main.cpp:46: test_led_state_low       [PASSED]
test/test_main.cpp:44: test_led_state_high      [PASSED]
test/test_main.cpp:46: test_led_state_low       [PASSED]
test/test_main.cpp:44: test_led_state_high      [PASSED]
test/test_main.cpp:46: test_led_state_low       [PASSED]
test/test_main.cpp:44: test_led_state_high      [PASSED]
test/test_main.cpp:46: test_led_state_low       [PASSED]
-------------------------------------------------------------------- esp32dev:* [PASSED] Took 32.00 seconds --------------------------------------------------------------------

==================================================================================== SUMMARY ====================================================================================
Environment    Test    Status    Duration
-------------  ------  --------  ------------
esp32dev       *       PASSED    00:00:32.003

I did not open the serial monitor at all!

When I open the serial monitor, I get

 *  Executing task: C:\Users\boris\.platformio\penv\Scripts\platformio.exe device monitor --port COM9 

--- Terminal on COM9 | 115200 8-N-1
--- Available filters and text transformations: colorize, debug, default, direct, esp32_exception_decoder, hexlify, log2file, nocontrol, printable, send_on_enter, time
--- More details at https://bit.ly/pio-monitor-filters
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H
USB_CDC
test/test_main.cpp:36:test_led_builtin_pin_number:PASS
test/test_main.cpp:44:test_led_state_high:PASS
test/test_main.cpp:46:test_led_state_low:PASS
test/test_main.cpp:44:test_led_state_high:PASS
test/test_main.cpp:46:test_led_state_low:PASS
test/test_main.cpp:44:test_led_state_high:PASS
test/test_main.cpp:46:test_led_state_low:PASS
test/test_main.cpp:44:test_led_state_high:PASS
test/test_main.cpp:46:test_led_state_low:PASS
test/test_main.cpp:44:test_led_state_high:PASS
test/test_main.cpp:46:test_led_state_low:PASS

-----------------------
11 Tests 0 Failures 0 Ignored 
OK

This also looks completely different from the output you have.

12

Posting the code and the exact output of the tests will help clarify the situation. First the code, it has one test assert and a printf statement to see if printing to stdout works:

#include <Arduino.h>
#include <unity.h>

void setUp(void)
{
}

void tearDown(void)
{
}

void testBlink(void) 
{
    for (int count = 0; count <= 20; count++) {
        digitalWrite(BUILTIN_LED, HIGH);
        delay(250);
        digitalWrite(BUILTIN_LED, LOW);
        delay(250);
    }
    TEST_ASSERT(true);
}

void setup() 
{
    Serial.begin(115200);
    delay(4000);
    Serial.println("Running blink test.");
    printf("Testing print to stdout.");

    pinMode(BUILTIN_LED, OUTPUT);
    digitalWrite(BUILTIN_LED, LOW);
    UNITY_BEGIN();
    Serial.println("calling run blink test.");
    RUN_TEST(testBlink);
    Serial.println("Done running blink test.");
    UNITY_END();
    Serial.println("Done unity end.");
}

void loop()
{
}

I ran the monitor just to show the actual output. I found that I could also run the test in verbose mode to see the output. Here is the test output:

Executing task: platformio test --environment lolin_s2_mini 

Verbosity level can be increased via `-v, -vv, or -vvv` option
Collected 6 tests

Processing embedded/test_blink in lolin_s2_mini environment
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building & Uploading...
Total image size: 354662 bytes (.bin may be padded larger)
Note: The reported total sizes may be smaller than those in the technical reference manual due to reserved memory and application configuration. The total flash size available for the application is not included by default, as it cannot be reliably determined due to the presence of other data like the bootloader, partition table, and application partition size.
Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

After that, there is no output until I reset the board. Running the test in verbose mode with Project Tasks > Advanced > Verbose Test:

Executing task: platformio test --verbose --environment lolin_s2_mini 

Collected 1 tests (embedded/test_blink)

Processing embedded/test_blink in lolin_s2_mini environment
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Building & Uploading...
Total image size: 354662 bytes (.bin may be padded larger)
Note: The reported total sizes may be smaller than those in the technical reference manual due to reserved memory and application configuration. The total flash size available for the application is not included by default, as it cannot be reliably determined due to the presence of other data like the bootloader, partition table, and application partition size.

Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

Running blink test.
calling run blink test.
Done running blink test.
Done unity end.

And the test does not end until I reset the board:

device reports readiness to read but returned no data (device disconnected or multiple access on port?)
---------------------------------------------------------- lolin_s2_mini:embedded/test_blink [ERRORED] Took 73.77 seconds ----------------------------------------------------------

===================================================================================== SUMMARY =====================================================================================
Environment    Test                 Status    Duration
-------------  -------------------  --------  ------------
lolin_s2_mini  embedded/test_blink  ERRORED   00:01:13.775
==================================================================== 1 test cases: 0 succeeded in 00:01:13.775 ====================================================================

 *  The terminal process "platformio 'test', '--verbose', '--environment', 'lolin_s2_mini'" terminated with exit code: 1. 
 *  Terminal will be reused by tasks, press any key to close it.

On the verbose test output, we can see the text written with Serial.println, but not text written to stdout. I believe the stdout information is going to the UART, and I could probably connect to the UART and see the output. (I haven’t tried this). This seems to be due to the ESP32S2 and the espresiff libraries (as I briefly pointed to previously in the thread).

I believe my best workaround is to use a custom unity_config.h to override the version in the framework libraries, sending the Unity output to the Serial object. Your example showing that this works and what to expect has been very helpful in clearing up my original confusion about what was going on with the Unity testing.

I am also happy to post any further information or try anything if there is interest. Otherwise I believe I know how to make this work for my needs.

Thank you so much!