Embedded testing failed with Broken pipe on Arduino

alex.evseenko · September 21, 2023, 7:30pm

I’ve tried to run unit test in embedded mode using Arduino Micro and Leonardo boards using the Calculator example:

platformio/platformio-examples/blob/develop/unit-testing/calculator/test/test_embedded/test_calculator.cpp

/*
 Copyright (c) 2014-present PlatformIO <contact@platformio.org>

 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
**/

#include <Arduino.h>
#include <calculator.h>
#include <unity.h>

This file has been truncated. show original

On both boards uploaded unit-test works (as I can see fast blinking of the led) but pio test failed with error message:

Testing…
If you don’t see any output for the first 10 secs, please reset board (press reset button)

[Errno 32] Broken pipe

What’s wrong?

Thank you in advance!

maxgerhardt · September 21, 2023, 7:37pm

Do you see the effect of

github.com

platformio/platformio-examples/blob/develop/unit-testing/calculator/test/test_embedded/test_calculator.cpp#L60-L65


      
          void loop() {

              digitalWrite(13, HIGH);

              delay(100);

              digitalWrite(13, LOW);

              delay(500);

          }

on the board, aka a blinking builtin-LED?

alex.evseenko · September 21, 2023, 7:41pm

Yeap, as I wrote in the starting topic I see fast blinking of the led – then it’s uploaded the test code.

So, it works, but not possible to see results of the test – after the Broken pipe error appears the test marked as ERRORED, which is not quite helpful for TDD…

maxgerhardt · September 21, 2023, 7:46pm

I would be quite baffled by this but then the default PlatformIO implementation for the Unity I/O functions kills it maybe. Specifically I can image that Serial.end() which is called during UNITY_END(); shuts down the USB serial port completely, resulting in a broken pipe.

github.com

platformio/platformio-core/blob/fe52b79eb219045e84b3198e7281f0d28ea71d6b/platformio/test/runners/unity.py#L126-L130


      
          #include <Arduino.h>
          void unityOutputStart(unsigned long baudrate) { Serial.begin(baudrate); }
          void unityOutputChar(unsigned int c) { Serial.write(c); }
          void unityOutputFlush(void) { Serial.flush(); }
          void unityOutputComplete(void) { Serial.end(); }

Try and follow Unity — PlatformIO latest documentation (or this) to give it your own implementation of this, i.e.,

create test/unity_config.h and fill it with this content
create test/unity_config.cpp and use the implementation quotes above, but leave unityOutputComplete() as empty function.

alex.evseenko · September 23, 2023, 11:25am

Thank you for the answer!

I’ve added some debugging prints in the setup() and it seems that the Broken pipe error occurred before even the test start executing:

void setup()
{
    // this delay is required, otherwise test results
    // would not be communicated to the host terminal
    delay(1000);

    UNITY_BEGIN();
    RUN_TEST(simple_test);
    UNITY_END();
}

I use the cli command to start test:
pio test --upload-port /dev/ttyACM0 --test-port /dev/ttyACM0 --environment leonardo

and I see the Broken pipe error in the terminal immediately after the upload finished, much before 1 second delay ends up and the RUN_TEST() even started…

maxgerhardt · September 23, 2023, 11:45am

Is another program accessing the port maybe? Something like the infamous brltty service? What’s the output of sudo dmesg | tail -n50 after the pipe breaks?

alex.evseenko · September 23, 2023, 6:04pm

No brltty service, but dmesg shows each time I run the test 2 consecutive disconnections of the Arduino Leonardo with different idProduct:

[ 1810.138979] usb 1-2: USB disconnect, device number 10
[ 1810.488209] usb 1-2: new full-speed USB device number 11 using xhci_hcd
[ 1810.638580] usb 1-2: New USB device found, idVendor=2341, idProduct=0036, bcdDevice= 0.01
[ 1810.638588] usb 1-2: New USB device strings: Mfr=2, Product=1, SerialNumber=0
[ 1810.638591] usb 1-2: Product: Arduino Leonardo
[ 1810.638593] usb 1-2: Manufacturer: Arduino LLC
[ 1810.640662] cdc_acm 1-2:1.0: ttyACM0: USB ACM device

[ 1813.005040] usb 1-2: USB disconnect, device number 11
[ 1813.316350] usb 1-2: new full-speed USB device number 12 using xhci_hcd
[ 1813.469629] usb 1-2: New USB device found, idVendor=2341, idProduct=8036, bcdDevice= 1.00
[ 1813.469634] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 1813.469636] usb 1-2: Product: Arduino Leonardo
[ 1813.469638] usb 1-2: Manufacturer: Arduino LLC
[ 1813.471652] cdc_acm 1-2:1.0: ttyACM0: USB ACM device

Each time these two groups of messages appears and device numbers increased all the time, here they are 10 and 11, next run it would be 12, 13 etc.

In the terminal I see per each test run:

Disconnected (device reports readiness to read but returned no data (device disconnected or multiple access on port?))
Reconnecting to /dev/ttyACM0 .   Connected!
Disconnected (device reports readiness to read but returned no data (device disconnected or multiple access on port?))
Reconnecting to /dev/ttyACM0     Connected!
Disconnected (device reports readiness to read but returned no data (device disconnected or multiple access on port?))
Reconnecting to /dev/ttyACM0     Connected!
Disconnected (device reports readiness to read but returned no data (device disconnected or multiple access on port?))
Reconnecting to /dev/ttyACM0     Connected!
test/test_blink/test_main.cpp:48:simple_test:PASS
test/test_blink/test_main.cpp:49:led_on_test:PASS
test/test_blink/test_main.cpp:50:led_off_test:PASS

-----------------------
3 Tests 0 Failures 0 Ignored
OK

So, the test works but not responded to pio due to this Broken pipe.

maxgerhardt · September 23, 2023, 6:20pm

Ah, no I remember something: Can you apply this to your project?

github.com/platformio/platformio-core

Add test start delay option

opened 09:20PM - 18 Nov 20 UTC

closed 07:13PM - 29 Apr 22 UTC

maxgerhardt

help wanted unit testing

# Solution Please add to your project a [POST Extra Script](https://docs.plat…formio.org/en/latest/scripting/index.html) with the following contents: **platformio.ini** ```ini [env:seeed_wio_terminal] platform = atmelsam board = seeed_wio_terminal framework = arduino extra_scripts = post:extra_script.py ``` **extra_script.py** ```python import time Import("env") if "test" in env.GetBuildType(): env.AddPostAction("upload", lambda *_, **__: time.sleep(2)) ``` P.S: Do not forget to upgrade to the latest PlatformIO Core via `pio upgrade --dev`. # Related issues: - https://community.platformio.org/t/teensy-4-1-unit-testing-issue/21033/5 - https://github.com/platformio/platformio-core/issues/3956 ### Configuration **Operating system**: Windows 10 x64 **PlatformIO Version** (`platformio --version`): 5.0.4a2 ### Description of problem Reference to topic where this popped up: https://community.platformio.org/t/unit-test-wio-terminal-could-not-open-port-while-testing/17180/ A user wants to do unit testing on a WIO terminal, which has a ~~SAMD21G18A~~ ATSAMD51P19A microcontroller onboard. This microcontroller implements a USB CDC serial connection. When a new firmware is uploaded, or the microcontroller is being reset, the USB connection is temporarily lost, then re-established. The problem is then when testing, PlatformIO does a Build -> Upload -> Test, where there is no delay between uploading and testing when using this specific microcontroller. A problem was problem was observed that when using Linux, after the upload is done, the microcontroller re-enumerates twices on the USB bus, and at the moment the "Test" part of the unit tests wants to access `/dev/ttyACM0`, the Linux kernel or subsystems have not yet fully re-enumerated the device and it's currently owned by root at this point. Then accessing it fails with `[Errno 13] could not open port /dev/ttyACM0: [Errno 13] Permission denied: '/dev/ttyACM0'`. A few milliseconds after it ([exact timing log is here](https://community.platformio.org/t/unit-test-wio-terminal-could-not-open-port-while-testing/17180/9?u=maxgerhardt)), Linux is done processing the device and now the user properly owns `/dev/ttyACM0` and access to it would go through. That's at least how I read these logs. However, since PlatformIO does not have a configurable delay after uploading, or equivalently before testing, `pio test` will always run into this problem. So it would be nice to have such an option to enable these use cases where the to-be-tested MCU is the USB serial port and does some re-enumeration stuff after uploading. #### Steps to Reproduce 1. Create a project for the `seeed_wio_terminal` 2. Add standard unit testing code to it 3. Execute `pio test` ### Actual Results Permission denied error on the serial device. ### Expected Results Testing goes through. ### If problems with PlatformIO Build System: **The content of `platformio.ini`:** ```ini [env:seeed_wio_terminal] platform = atmelsam board = seeed_wio_terminal framework = arduino ``` **Source file to reproduce issue:** ```cpp #include <Arduino.h> #include <unity.h> void TestConvertTemperatureToString(void) { TEST_ASSERT_EQUAL(32, 32); } void setup() { // NOTE!!! Wait for >2 secs // if board doesn't support software reset via Serial.DTR/RTS delay(5000); UNITY_BEGIN(); RUN_TEST(TestConvertTemperatureToString); UNITY_END(); } void loop() { } ``` ### Additional info

i.e

extra_scripts = post:extra_script.py

in the platformio.ini and a new file extra_script.py on the same level as the platformio.ini with

import time

Import("env")

if "test" in env.GetBuildType():
    env.AddPostAction("upload", lambda *_, **__: time.sleep(2))

alex.evseenko · September 23, 2023, 6:44pm

Amazing, it fixes the problem, works now – many thanks for your help!