Specifying test_port breaks the test

zapta · August 19, 2023, 5:30pm

My unity tests uses the CDC serial port of my STM32H7 which is mapped to COM6 on windows. If I run the test without specifying test_port, the tests pass. However if I do specify test_port=6, it fails to open the port and the test fails.

Any suggestions how to fix it? I wonder if this is a timing issue. If I don’t specify the port, platformio spends time to find the port which let the device sufficient time to initialize the CDC link.

(I need to specify the port because the system has additional serial ports that I need to connect).

Error message:

could not open port 'COM6': FileNotFoundError(2, 'The system cannot find the file specified.', None, 2)

Devices:

$ pio device list
COM1
----
Hardware ID: ACPI\PNP0501\0
Description: Communications Port (COM1)

COM6
----
Hardware ID: USB VID:PID=0483:5740 SER=387035733232 LOCATION=1-4.4
Description: USB Serial Device (COM6)

Platformio.ini:

[env:my_env]
build_type = debug
; build_type = release
platform = ststm32
; test_filter = serial_packets/test_serial_packets_client
; debug_test = serial_packets/test_serial_packets_client
test_filter = test_trivial
debug_test = test_trivial
extra_scripts = extra_script.py
board = my_genericSTM32H750VBT6
; Linker file is from cube ide.
board_build.ldscript = STM32H750VBTX_FLASH.ld
debug_tool = stlink
upload_protocol = stlink
; debug_build_flags = -O0 -ggdb3 -g3
; debug_build_flags = -Og -ggdb3 -g3
debug_build_flags = -Os -g3 -ggdb3
; debug_init_break = tbreak app_main
debug_init_break = tbreak none
test_framework = unity
test_port = COM6
monitor_port = COM6
lib_archive = no
check_skip_packages = yes
lib_deps = 
  adc
  data_recorder
  host_link
  io
  misc
  printer_link
  controller
  cube_ide
  serial_packets
  system
build_flags =
  -fmax-errors=5
  -Werror
  -mfpu=fpv5-sp-d16 
  -mfloat-abi=hard 
  -Wl,-Map,${BUILD_DIR}/firmware.map
  -mthumb 
  -D DEBUG
  -D USE_HAL_DRIVER
  -DSTM32_THREAD_SAFE_STRATEGY=4
  -fstack-usage
  -std=gnu11
  -Ilib/cube_ide/Core/Inc
  -Ilib/cube_ide/Core/ThreadSafe
  -Ilib/cube_ide/Drivers/CMSIS/Device/ST/STM32H7xx/Include
  -Ilib/cube_ide/Drivers/CMSIS/Include
  -Ilib/cube_ide/Drivers/STM32H7xx_HAL_Driver/Inc
  -Ilib/cube_ide/Middlewares/ST/STM32_USB_Device_Library/Class/CDC/Inc
  -Ilib/cube_ide/Middlewares/ST/STM32_USB_Device_Library/Core/Inc
  -Ilib/cube_ide/Middlewares/Third_Party/FreeRTOS/Source/CMSIS_RTOS
  -Ilib/cube_ide/Middlewares/Third_Party/FreeRTOS/Source/CMSIS_RTOS_V2/Include
  -Ilib/cube_ide/Middlewares/Third_Party/FreeRTOS/Source/include
  -Ilib/cube_ide/Middlewares/Third_Party/FreeRTOS/Source/portable/GCC/ARM_CM4F
  -Ilib/cube_ide/Middlewares/Third_Party/FatFs/src
  -Ilib/cube_ide/USB_DEVICE/App
  -Ilib/cube_ide/USB_DEVICE/Target
  -Ilib/cube_ide/FATFS/App
  -Ilib/cube_ide/FATFS/Target
  -D CONFIG_MAX_PACKET_DATA_LEN=1000
  -D CONFIG_MAX_PENDING_COMMANDS=5

Log with test_port=COM6 (test fails):

~/projects/daq/repo/controller$ cd platformio/
~/projects/daq/repo/controller/platformio$ pio test 
Verbosity level can be increased via `-v, -vv, or -vvv` option
Collected 10 tests

Processing test_trivial in my_env environment
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Building & Uploading...
xPack Open On-Chip Debugger 0.12.0-01004-g9ea7f3d64-dirty (2023-01-30-15:04)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.org/doc/doxygen/bugs.html
debug_level: 1

hla_swd
[stm32h7x.cpu0] halted due to debug-request, current mode: Thread 
xPSR: 0x01000000 pc: 0x08001958 msp: 0x24080000
** Programming Started **
Warn : Adding extra erase range, 0x0800c520 .. 0x0801ffff
** Programming Finished **
** Verify Started **
** Verified OK **
** Resetting Target **
shutdown command invoked
Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

could not open port 'COM6': FileNotFoundError(2, 'The system cannot find the file specified.', None, 2)
------------------------------------------------------------ my_env:test_trivial [PASSED] Took 25.62 seconds ------------------------------------------------------------

=============================================================== 0 test cases: 0 succeeded in 00:00:25.618 ===============================================================
~/projects/daq/repo/controller/platformio$

maxgerhardt · August 19, 2023, 7:20pm

If the COM port is created by the uploaded test firmware, there could be a timing issue, just like discussed and worked-around in

github.com/platformio/platformio-core

Add test start delay option

opened 09:20PM - 18 Nov 20 UTC

closed 07:13PM - 29 Apr 22 UTC

maxgerhardt

help wanted unit testing

# Solution Please add to your project a [POST Extra Script](https://docs.plat…formio.org/en/latest/scripting/index.html) with the following contents: **platformio.ini** ```ini [env:seeed_wio_terminal] platform = atmelsam board = seeed_wio_terminal framework = arduino extra_scripts = post:extra_script.py ``` **extra_script.py** ```python import time Import("env") if "test" in env.GetBuildType(): env.AddPostAction("upload", lambda *_, **__: time.sleep(2)) ``` P.S: Do not forget to upgrade to the latest PlatformIO Core via `pio upgrade --dev`. # Related issues: - https://community.platformio.org/t/teensy-4-1-unit-testing-issue/21033/5 - https://github.com/platformio/platformio-core/issues/3956 ### Configuration **Operating system**: Windows 10 x64 **PlatformIO Version** (`platformio --version`): 5.0.4a2 ### Description of problem Reference to topic where this popped up: https://community.platformio.org/t/unit-test-wio-terminal-could-not-open-port-while-testing/17180/ A user wants to do unit testing on a WIO terminal, which has a ~~SAMD21G18A~~ ATSAMD51P19A microcontroller onboard. This microcontroller implements a USB CDC serial connection. When a new firmware is uploaded, or the microcontroller is being reset, the USB connection is temporarily lost, then re-established. The problem is then when testing, PlatformIO does a Build -> Upload -> Test, where there is no delay between uploading and testing when using this specific microcontroller. A problem was problem was observed that when using Linux, after the upload is done, the microcontroller re-enumerates twices on the USB bus, and at the moment the "Test" part of the unit tests wants to access `/dev/ttyACM0`, the Linux kernel or subsystems have not yet fully re-enumerated the device and it's currently owned by root at this point. Then accessing it fails with `[Errno 13] could not open port /dev/ttyACM0: [Errno 13] Permission denied: '/dev/ttyACM0'`. A few milliseconds after it ([exact timing log is here](https://community.platformio.org/t/unit-test-wio-terminal-could-not-open-port-while-testing/17180/9?u=maxgerhardt)), Linux is done processing the device and now the user properly owns `/dev/ttyACM0` and access to it would go through. That's at least how I read these logs. However, since PlatformIO does not have a configurable delay after uploading, or equivalently before testing, `pio test` will always run into this problem. So it would be nice to have such an option to enable these use cases where the to-be-tested MCU is the USB serial port and does some re-enumeration stuff after uploading. #### Steps to Reproduce 1. Create a project for the `seeed_wio_terminal` 2. Add standard unit testing code to it 3. Execute `pio test` ### Actual Results Permission denied error on the serial device. ### Expected Results Testing goes through. ### If problems with PlatformIO Build System: **The content of `platformio.ini`:** ```ini [env:seeed_wio_terminal] platform = atmelsam board = seeed_wio_terminal framework = arduino ``` **Source file to reproduce issue:** ```cpp #include <Arduino.h> #include <unity.h> void TestConvertTemperatureToString(void) { TEST_ASSERT_EQUAL(32, 32); } void setup() { // NOTE!!! Wait for >2 secs // if board doesn't support software reset via Serial.DTR/RTS delay(5000); UNITY_BEGIN(); RUN_TEST(TestConvertTemperatureToString); UNITY_END(); } void loop() { } ``` ### Additional info

zapta · August 19, 2023, 9:07pm

Thanks @maxgerhardt, that works. It’s solved a real pain for me, having to disconnect cables before running tests.

This is what I am using.

# Per https://github.com/platformio/platformio-core/issues/3742
# The print messages are available only if running pio with -vv
build_type = env.GetBuildType()
print(f"(extra_script.py): build type = [{build_type}]")
if "test" in build_type:
    print(f"(extra_script.py): adding a post upload delay.")
    env.AddPostAction("upload", lambda *_, **__: time.sleep(2))
else:
    print(f"(extra_script.py): Not adding a post upload delay.")