Unit Tests not working on Arduino Nano ESP32

I’m trying to create embedded unit tests for my Arduino Nano ESP32 project. Running the app itself works just fine. Here is the output I get when trying to run tests.

 *  Executing task in folder freeRTOS_test: C:\Users\Mnemosyne\.platformio\penv\Scripts\platformio.exe test --environment arduino_nano_esp32 

Verbosity level can be increased via `-v, -vv, or -vvv` option
Collected 1 tests

Processing test_embedded in arduino_nano_esp32 environment
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building & Uploading...
Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

Cannot configure port, something went wrong. Original message: OSError(22, 'A device which does not exist was specified.', None, 433)
----------------------------------------------------------------------------------------- arduino_nano_esp32:test_embedded [PASSED] Took 8.81 seconds -----------------------------------------------------------------------------------------

================================================================================================== 0 test cases: 0 succeeded in 00:00:08.813 ==================================================================================================

Here is the full platformio.ini file. I have also tried setting the test_port value to different COM ports without any help.

; PlatformIO Project Configuration File
;
;   Build options: build flags, source filter
;   Upload options: custom upload port, speed and extra flags
;   Library options: dependencies, extra library storages
;   Advanced options: extra scripting
;
; Please visit documentation for the other options and examples
; https://docs.platformio.org/page/projectconf.html

[env:arduino_nano_esp32]
platform = espressif32
board = arduino_nano_esp32
framework = arduino
monitor_speed = 115200
build_flags =
    -Wall
    -Wextra
    -Werror

Here is my minimal test/test_embedded/test_main.cpp file.

#include <Arduino.h>
#include <unity.h>

#include <am2320.h>

void setUp(void) {
    // set stuff up here
}

void tearDown(void) {
    // clean stuff up here
}

void test_addition(void) {
    TEST_ASSERT_EQUAL(32, 25 + 7);
}

void setup() {
    // NOTE!!! Wait for >2 secs
    // if board doesn't support software reset via Serial.DTR/RTS
    delay(2000);

    Serial.printf("TEST");
    UNITY_BEGIN();
    RUN_TEST(test_addition);
    UNITY_END();
}

void loop() {
}

When I try to run the tests the “TEST” string prints in serial so it clearly manages to upload to the device.

--- Terminal on COM4 | 115200 8-N-1
--- Available filters and text transformations: colorize, debug, default, direct, esp32_exception_decoder, hexlify, log2file, nocontrol, printable, send_on_enter, time
--- More details at https://bit.ly/pio-monitor-filters
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H
TEST

Would appreciate any help figuring this out.

Have you tried

Now it gets stuck at “If you don’t see any output for the first 10 secs, please reset board (press reset button)” and if I press the reset button on the board it fails with an error. Here is the full log.

Building & Uploading...
Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

ClearCommError failed (PermissionError(13, 'The device does not recognize the command.', None, 22))
---------------------------------------------------------------------------------------- arduino_nano_esp32:test_embedded [ERRORED] Took 26.94 seconds ----------------------------------------------------------------------------------------

=================================================================================================================== SUMMARY ===================================================================================================================
Environment         Test           Status    Duration
------------------  -------------  --------  ------------
arduino_nano_esp32  test_embedded  ERRORED   00:00:26.940
================================================================================================== 1 test cases: 0 succeeded in 00:00:26.940 ==================================================================================================

Also if I now try to use the serial monitor it gives the following error: “UserSideException: could not open port ‘COM4’: PermissionError(13, ‘Access is denied.’, None, 5)”

Increase this delay time to 5000 or even 6000.

I tried increasing the delay in steps to 10000 and sadly still getting stuck after the “If you don’t see output…” message.

I’ve just tested this personally on my ESP32-S3-N16R8 board, which should be very similiar to a Arduino Nano ESP32 (also ESP32S3 based).

[env:arduino_nano_esp32]
platform = espressif32@6.12.0
board = esp32-s3-devkitc-1
framework = arduino
extra_scripts = post:extra_script.py
; this is already the default for a arduino_nano_esp32
build_flags = -DARDUINO_USB_CDC_ON_BOOT=1 

extra_script.py

import time

Import("env")

if "test" in env.GetBuildType():
    env.AddPostAction("upload", lambda *_, **__: time.sleep(2))

test/test_embedded/test_main.cpp

#include <Arduino.h>
#include <unity.h>

void setUp(void) {
    // set stuff up here
}

void tearDown(void) {
    // clean stuff up here
}

void test_addition(void) {
    TEST_ASSERT_EQUAL(32, 25 + 7);
}

void setup() {
    // NOTE!!! Wait for >2 secs
    // if board doesn't support software reset via Serial.DTR/RTS
    delay(5000);

    UNITY_BEGIN();
    RUN_TEST(test_addition);
    UNITY_END();
}

void loop() {
}

Then the output of pio test -vv in the Core CLI

Leaving...
Hard resetting via RTS pin...
<lambda>(["upload"], [".pio\build\arduino_nano_esp32\firmware.bin"])

Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

test/test_embedded/test_main.cpp:22:test_addition:PASS

-----------------------
1 Tests 0 Failures 0 Ignored

-----------------------------arduino_nano_esp32:test_embedded [PASSED] Took 18.28 seconds -----------------------------

================================= SUMMARY =================================Environment         Test           Status    Duration
------------------  -------------  --------  ------------
arduino_nano_esp32  test_embedded  PASSED    00:00:18.278
================ 1 test cases: 1 succeeded in 00:00:18.278 ================
PS C:\Users\Max\temp\nanoesp_test>

What espressif32 platform version are you using? I’m on mainline 6.12.0 per platformio.ini.

I’m using 6.12.0. I took your code code and configs and created a new clean project using them, still the same issue.

After you triggered the upload of the unit test firmware, and you kill the pio test process and press the reset button of the board, then quickly start the “Monitor” PlatformIO task, what output does it show, in full?

Presuming I understood your instructions correctly here is the output in full.

 *  Executing task in folder unit_test_debugging: C:\Users\Mnemosyne\.platformio\penv\Scripts\platformio.exe device monitor --environment arduino_nano_esp32 

--- Terminal on COM4 | 9600 8-N-1
--- Available filters and text transformations: colorize, debug, default, direct, esp32_exception_decoder, hexlify, log2file, nocontrol, printable, send_on_enter, time
--- More details at https://bit.ly/pio-monitor-filters
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H

Weird. When you use a src/main.cpp of

#include <Arduino.h>

void setup() {
    Serial.begin(115200);
}

void loop() {
    Serial.println("Hello, world!");
    delay(1000);
}

And use the regular “Upload” task and then the “Monitor” task, does the output appear?

Yes, works correctly.

 *  Executing task in folder unit_test_debugging: C:\Users\Mnemosyne\.platformio\penv\Scripts\platformio.exe device monitor --environment arduino_nano_esp32 

--- Terminal on COM4 | 9600 8-N-1
--- Available filters and text transformations: colorize, debug, default, direct, esp32_exception_decoder, hexlify, log2file, nocontrol, printable, send_on_enter, time
--- More details at https://bit.ly/pio-monitor-filters
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H
Hello, world!
Hello, world!
Hello, world!

I’ve never seen that before. Why would a Serial.println() work in the main firmware but not in the unit test? The created unit test firmware should output things just like the regular firmware.

You could try to use the debug_test platformio.ini option to debug the test_addition unit test. The ESP32S3 on the native USB port does have a USB JTAG controller for debugging.

Not sure if I did what you asked correctly but I set “debug_test = test_embedded” as that was the only valid name according to the ide. I then used the “Run and Debug” command to PIO Debug (unit_test_debugging) and received the following output to debug console.

Processing arduino_nano_esp32 (platform: espressif32@6.12.0; board: arduino_nano_esp32; framework: arduino)
--------------------------------------------------------------------------------
Building & Uploading...
undefinedC:\Users\Mnemosyne\.platformio\packages\toolchain-xtensa-esp32s3\bin\xtensa-esp32s3-elf-gdb.exe: warning: Couldn't determine a path for the index cache directory.

Reading symbols from D:\code\projects\unit_test_debugging\.pio\build\arduino_nano_esp32\firmware.elf...
PlatformIO Unified Debugger -> https://bit.ly/pio-debug
PlatformIO: debug_tool = cmsis-dap
PlatformIO: Initializing remote target...
Open On-Chip Debugger  v0.11.0-esp32-20220706 (2022-07-06-15:48)
Licensed under GNU GPL v2
For bug reports, read
	http://openocd.org/doc/doxygen/bugs.html
adapter speed: 5000 kHz

Info : tcl server disabled
Info : telnet server disabled
Error: unable to find a matching CMSIS-DAP device

.pioinit:11: Error in sourced command file:
Remote communication error.  Target disconnected.: Success.

This is what I get when I try to debug without the debug_test option in platformio.ini

Processing arduino_nano_esp32 (platform: espressif32@6.12.0; board: arduino_nano_esp32; framework: arduino)
--------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/arduino_nano_esp32.html
PLATFORM: Espressif 32 (6.12.0) > Arduino Nano ESP32
HARDWARE: ESP32S3 240MHz, 320KB RAM, 16MB Flash
DEBUG: Current (cmsis-dap) External (cmsis-dap, dfu, esp-bridge, esp-builtin, esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES: 
 - framework-arduinoespressif32 @ 3.20017.241212+sha.dcc1105b 
 - tool-dfuutil-arduino @ 1.11.0 
 - tool-esptoolpy @ 2.40900.250804 (4.9.0) 
 - tool-mkfatfs @ 2.0.1 
 - tool-mklittlefs @ 1.203.210628 (2.3) 
 - tool-mkspiffs @ 2.230.0 (2.30) 
 - toolchain-riscv32-esp @ 8.4.0+2021r2-patch5 
 - toolchain-xtensa-esp32s3 @ 8.4.0+2021r2-patch5
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 34 compatible libraries
Scanning dependencies...
No dependencies
Building in debug mode
Retrieving maximum program size .pio\build\arduino_nano_esp32\firmware.elf
Checking size .pio\build\arduino_nano_esp32\firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [=         ]   9.4% (used 30656 bytes from 327680 bytes)
Flash: [=         ]   9.3% (used 291389 bytes from 3145728 bytes)
Configuring upload protocol...
AVAILABLE: cmsis-dap, dfu, esp-bridge, esp-builtin, esp-prog, espota, esptool, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa
CURRENT: upload_protocol = dfu
Uploading .pio\build\arduino_nano_esp32\firmware.bin
dfu-util 0.11-arduino4

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2021 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to http://sourceforge.net/p/dfu-util/tickets/

Opening DFU capable USB device...
Device ID 2341:0070
Device DFU version 0101
Claiming USB DFU Interface...
Setting Alternate Interface #0 ...
Determining device status...
DFU state(2) = dfuIDLE, status(0) = No error condition is present
DFU mode device DFU version 0101
Device returned transfer size 4096
Copying data from PC to DFU device

Download	[                         ]   0%            0 bytes
Download	[=                        ]   4%        12288 bytes
Download	[==                       ]   8%        24576 bytes
Download	[===                      ]  12%        36864 bytes
Download	[====                     ]  16%        49152 bytes
Download	[=====                    ]  20%        61440 bytes
Download	[======                   ]  24%        73728 bytes
Download	[=======                  ]  29%        86016 bytes
Download	[=======                  ]  30%        90112 bytes
Download	[========                 ]  33%        98304 bytes
Download	[=========                ]  37%       110592 bytes
Download	[==========               ]  40%       118784 bytes
Download	[===========              ]  44%       131072 bytes
Download	[============             ]  48%       143360 bytes
Download	[=============            ]  52%       155648 bytes
Download	[==============           ]  56%       167936 bytes
Download	[===============          ]  60%       180224 bytes
Download	[================         ]  65%       192512 bytes
Download	[=================        ]  69%       204800 bytes
Download	[==================       ]  73%       217088 bytes
Download	[===================      ]  76%       225280 bytes
Download	[====================     ]  80%       237568 bytes
Download	[=====================    ]  84%       249856 bytes
Download	[======================   ]  88%       262144 bytes
Download	[=======================  ]  92%       274432 bytes
Download	[======================== ]  96%       286720 bytes
Download	[=========================] 100%       291760 bytes
Download done.
DFU state(7) = dfuMANIFEST, status(0) = No error condition is present
DFU state(2) = dfuIDLE, status(0) = No error condition is present
Done!
========================= [SUCCESS] Took 5.94 seconds =========================
undefinedC:\Users\Mnemosyne\.platformio\packages\toolchain-xtensa-esp32s3\bin\xtensa-esp32s3-elf-gdb.exe: warning: Couldn't determine a path for the index cache directory.

Reading symbols from D:\code\projects\unit_test_debugging\.pio\build\arduino_nano_esp32\firmware.elf...
PlatformIO Unified Debugger -> https://bit.ly/pio-debug
PlatformIO: debug_tool = cmsis-dap
PlatformIO: Initializing remote target...
Open On-Chip Debugger  v0.11.0-esp32-20220706 (2022-07-06-15:48)
Licensed under GNU GPL v2
For bug reports, read
	http://openocd.org/doc/doxygen/bugs.html
adapter speed: 5000 kHz

Info : tcl server disabled
Info : telnet server disabled
Error: unable to find a matching CMSIS-DAP device

.pioinit:11: Error in sourced command file:
Remote communication error.  Target disconnected.: Success.

Try additionally setting

debug_tool = esp-builtin

in the platformio.ini.

Sadly that didn’t help either.

Processing arduino_nano_esp32 (platform: espressif32@6.12.0; board: arduino_nano_esp32; framework: arduino)
--------------------------------------------------------------------------------
Building & Uploading...
undefinedC:\Users\Mnemosyne\.platformio\packages\toolchain-xtensa-esp32s3\bin\xtensa-esp32s3-elf-gdb.exe: warning: Couldn't determine a path for the index cache directory.

Reading symbols from D:\code\projects\unit_test_debugging\.pio\build\arduino_nano_esp32\firmware.elf...
PlatformIO Unified Debugger -> https://bit.ly/pio-debug
PlatformIO: debug_tool = esp-builtin
PlatformIO: Initializing remote target...
Open On-Chip Debugger  v0.11.0-esp32-20220706 (2022-07-06-15:48)
Licensed under GNU GPL v2
For bug reports, read
	http://openocd.org/doc/doxygen/bugs.html
Info : only one transport option; autoselect 'jtag'
Info : esp_usb_jtag: VID set to 0x303a and PID to 0x1001
Info : esp_usb_jtag: capabilities descriptor set to 0x2000
adapter speed: 40000 kHz

Warn : Transport "jtag" was already selected
adapter speed: 5000 kHz

Info : tcl server disabled
Info : telnet server disabled
Error: esp_usb_jtag: could not find or open device!

.pioinit:11: Error in sourced command file:
Remote communication error.  Target disconnected.: Success.

When you plug in your Arduino Nano ESP32 and go into the Windows device manager and switch to View → Device by container, do you see any device like this?

“Serial USB device” and a “USB JTAG/serial debug unit” is the crucial one.

If yes, you should check with Zadig (https://zadig.akeo.ie/) that you have libusb drivers for that interface 2 loaded. Then OpenOCD should find it.

I do not. Only thing refering to my Nano is “Arduino DFU” specified as Nano ESP32 DFU in the details.

When you connect GPIO0 (“B1”) to GND and plug the board back in, does the device appear differently in the device manager?

Yes, with the pins connected this is what I see.

Then in that mode, you should be able to upload and debug the unit test program via JTAG.

You may additionally need to set uploading via JTAG via

upload_protocol = esp-builtin

in the platformio.ini.

The GPIO0 ↔ GND link should be removed after the device was rebooted with that link intact, so that after the next reboot it won’t again go into bootloader mode.