Unit Testing Error

Hello again,

I am trying to run unit tests for a particular project and this is what I got from the command line output

> Traceback (most recent call last):
>   File "main.py", line 7, in init main (/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/tmpUDBPoC/main.c:2287)
>   File "/Users/krishi/.atom/packages/platformio-ide/penv/lib/python2.7/site-packages/click/__init__.py", line 18, in <mo
> dule>
>     from .core import Context, BaseCommand, Command, MultiCommand, Group, \
>   File "/Users/krishi/.atom/packages/platformio-ide/penv/lib/python2.7/site-packages/click/core.py", line 8, in <module>
>     from .types import convert_type, IntRange, BOOL
>   File "/Users/krishi/.atom/packages/platformio-ide/penv/lib/python2.7/site-packages/click/types.py", line 5, in <module
> >
>     from ._compat import open_stream, text_type, filename_to_ui, \

>   File "/Users/krishi/.atom/packages/platformio-ide/penv/lib/python2.7/site-packages/click/_compat.py", line 2, in <modu
> le>
>     import io
>   File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/io.py", line 51, in <modu
> le>
>     import _io
> ImportError: dlopen(/Users/krishi/.atom/packages/platformio-ide/penv/lib/python2.7/lib-dynload/_io.so, 2): Symbol not fo
> und: __PyCodecInfo_GetIncrementalDecoder
>   Referenced from: /Users/krishi/.atom/packages/platformio-ide/penv/lib/python2.7/lib-dynload/_io.so
>   Expected in: flat namespace
>  in /Users/krishi/.atom/packages/platformio-ide/penv/lib/python2.7/lib-dynload/_io.so

Funnily, this is the first time I am seeing such an error.
Oh, by the way this is not the output in any project. I was just trying to get my head around platformio and unit testing and tried to run the calculator example and this is what I got.

Any missing dependancy or library that I should “pip install” perhaps…

1 Like

Fixed in development version before. Could you try Redirecting... ?

Works just fine @ivankravets
:smiley:
Thanks again for the help!!

I noticed that there are two different sets of commands to invoke the test
pio run -t test and pio test

any specific difference between them?
I only ask because the pio test commands works just fine and the pio run -t test throws an error

> test/test_embedded/test_calculator.cpp:1:10: fatal error: 'Arduino.h' file not found
> #include <Arduino.h>
> ^
> 1 error generated.
> *** [.pioenvs/local/test/test_embedded/test_calculator.o] Error 1

This to be exact.
Apart from the wonderful manual available online, is there any other documentation available that explains platformio and unity in detail, rather their integration. I am fairly conversant with unity. I would like to understand is there anyway I can use pio the same way and Cmocks. Does pio support Cmock??

Thanks for your exhaustive patience again to help out noobs and beginners like me.

Of course, pio run -t test will not work. It is an internal target. I’ve just removed it in local development version. You have to use ONLY pio test.

The idea is simple. You organize code as components/modules that are represented as “local project libraries” and then reuse them in tests. Otherwise, you can write modular code if control it with #ifdef UNIT_TEST.

Cmock is the other implementation around UnitTest engine. What is missed in PlatformIO Unit Testing that present in Cmock?

Thank you!!

I understand the modulartity and organization issues. I was used to doing it the conventional way before platformio came along. The manual way of organizing directories and manually running the tests :stuck_out_tongue:

I have just started trying to test code even in platformio and the change in approach is getting some used to.
The examples are crisp and concise. It is my perspective that needs to change. A couple of hours trying and testing should solve it.
:slight_smile:

Cmock for mocking inputs to test simulated inputs. As I understand it, the unittest engine allows to run test code in realtime on the target board to check its functionality. Is my understanding right?
Certain code paths and state machines are not always feasible nor reachable to test them on the target in realtime (Mostly only envelop conditions). Hence my query.

Yep, the tests will be run on a target device (not on localhost).

If you can propose better examples with Unit Tests, please make PR to this repository GitHub - platformio/platformio-examples: PlatformIO Project Examples

Thanks!
Will try to contribute what I can understand in a day or two.
Still need some time to get my around this engine.
It is a powerful tool if I can start using it.
Unit testing on the target board!! thats like the dream come true for an embedded developer.
:smiley:

@ivankravets
Ivan just a follow up question. I have been working on the unit testing part and have some questions.
The vanilla calculator example. No code untouched.

Under the test directory, there are three sub-directories.

  1. test_common
    the .c test file in this directory is able to run the test on the local machine or on the embedded target depending on the parameters passed to it during invocation or in the platformio.ini file. #ifdef Arduino or UNIT_TEST switches.

  2. test_embedded
    Please correct me if my understanding is flawed in any manner. You are simply including the unity test framework in the main test file and running the tests actually on the target embedded system and not on the host PC. This is being done by actually running the asserts as a part of the setup () loop. You are still uploading the sketch and testing?? How does the testing happen on the target board and still is verified on the host PC. This is where I am having some trouble understanding live target testing. You are literally writing a test sketch, including the unity libraries and compiling, uploading and running the tests??

  3. test_local
    This is the classic use for the unity framework where you simply run the tests on the host PC.

Is my understanding of the testing process correct?

Thanks! You have correctly described all examples and their behavior.

Yep, we build project on localhost machine, then upload firmware to target device where all tests will be passed here. Then, we collect results from embedded device (using Serial Port communication) and make nice printing on localhost side.

P.S: When we build firmware for “embedded” target, we wrap your code/test with own tiny Serial-based unit test framework that is a bridge between localhost machine and embedded device.

Got it!! :slight_smile:
Makes sense now!! :smiley:

1 Like

Follow up question!! As i have mentioned I am working with the generic STM32F103C8 bluepills boards, converted one board to act as a blackmagic probe. I am using a upload.gdb file and “upload_protocol=gdb” in the platform.ini file. The compiling and uploading works just fine.

I have run into two problems while trying to implement unit testing with this board / framework combination.

  1. Local host testing is not working.
    This is my platform.ini file

    [env:bluepill_f103c8]
    platform = ststm32
    board = bluepill_f103c8
    framework = mbed
    upload_protocol = gdb

    [env:local]
    platform = native
    framework = mbed

This is my upload.gdb file

target extended-remote /dev/cu.usbmodemBFDEA5F1
monitor swdp_scan
attach 1
load
quit
yes

When I run “pio test --environment=local” this is what I get in the console output.

krishis-MacBook-Pro:DigitalMeter krishi$ pio test --environment=local
PlatformIO Plus (https://pioplus.com) v0.2.3
Verbose mode can be enabled via `-v, --verbose` option
Collected 1 items

====================================================== [test::*] Building... (1/2) ======================================================

[Sun Oct 16 00:13:47 2016] Processing local (platform: native, framework: mbed)
-----------------------------------------------------------------------------------------------------------------------------------------
Please wait...
Please specify `board` in `platformio.ini` to use with 'mbed' framework
======================================================= [ERROR] Took 0.54 seconds =======================================================

=============================================================== [SUMMARY] ===============================================================

Environment local               [ERROR]
Environment bluepill_f103c8     [SKIP]
======================================================= [ERROR] Took 0.54 seconds =======================================================

When I add “board = bluepill_f103c8” option to the local environment this is what I get.

krishis-MacBook-Pro:DigitalMeter krishi$ pio test --environment=local
PlatformIO Plus (https://pioplus.com) v0.2.3
Verbose mode can be enabled via `-v, --verbose` option
Collected 1 items

====================================================== [test::*] Building... (1/2) ======================================================

[Sun Oct 16 00:14:12 2016] Processing local (platform: native, board: bluepill_f103c8, framework: mbed)
-----------------------------------------------------------------------------------------------------------------------------------------
Please wait...

Error: Unknown board ID 'bluepill_f103c8'
======================================================= [ERROR] Took 0.54 seconds =======================================================

=============================================================== [SUMMARY] ===============================================================
Environment local               [ERROR]
Environment bluepill_f103c8     [SKIP]
======================================================= [ERROR] Took 0.54 seconds =======================================================

I am guessing I goofed up some switch / setting somewhere.

The second is to do with the target testing. When I try to run the unit tests on the live target the console is getting through till the third step and is stuck at the third step “testing”

This is my console output.

PlatformIO Plus (https://pioplus.com) v0.2.3
Verbose mode can be enabled via `-v, --verbose` option
Collected 1 items

====================================================== [test::*] Building... (1/3) ======================================================
[Sun Oct 16 00:24:27 2016] Processing bluepill_f103c8 (upload_protocol: gdb, platform: ststm32, board: bluepill_f103c8, framework: mbed)
-----------------------------------------------------------------------------------------------------------------------------------------
Please wait...

=============================================================== [SUMMARY] ===============================================================
Environment local          	[SKIP]
Environment bluepill_f103c8	[SUCCESS]
====================================================== [SUCCESS] Took 2.67 seconds ======================================================

===================================================== [test::*] Uploading... (2/3) =====================================================
[Sun Oct 16 00:24:30 2016] Processing bluepill_f103c8 (upload_protocol: gdb, platform: ststm32, board: bluepill_f103c8, framework: mbed)
-----------------------------------------------------------------------------------------------------------------------------------------
Please wait...
Target voltage: Not Implemented!
Available Targets:
No. Att Driver
1      STM32F1 medium density

=============================================================== [SUMMARY] ===============================================================
Environment local          	[SKIP]
Environment bluepill_f103c8	[SUCCESS]
====================================================== [SUCCESS] Took 6.16 seconds ======================================================

====================================================== [test::*] Testing... (3/3) ======================================================
If you don't see any output for the first 10 secs, please reset board (press reset button)

The console is stuck at that point without any progress, inspite of resetting the target board.
I am having to CTRL + C to abort the operation at that point.
I am guessing that this is owing to my misuse of the embedded target testing protocol, which is not designed for testing using gdb?? Or is it arduino specific?? Or am I not seeing something obvious that is right in front of me…

I have not tested using a St-Link adapter ( I do not have it presently with me, I shall report back my findings with the St-Link programmer in a day or two.)

Thanks again for your time!! :slight_smile:

PS: Forgot to attach my source files
Here is my source
main.cpp

#include "mbed.h"

#ifndef UNIT_TEST
DigitalOut myled(PC_13);

    int main(void)
    {
      while(1) {
        myled=1;
        wait(1);
        myled=0;
        wait(1);

      }
    }
    #endif

And here is my test file
test_main.cpp

#include "mbed.h"
#include "unity.h"

#ifdef UNIT_TEST
DigitalOut myled(PC_13);

void test_led_write_high(void)
{
  myled.write(1);
  TEST_ASSERT_EQUAL(myled.read(), 1);
}

void test_led_write_low(void)
{
  myled.write(0);
  TEST_ASSERT_EQUAL(myled.read(), 0);
}

int main(void)
{
  UNITY_BEGIN();

  for(int i=0;i<10;i++)
    {
      RUN_TEST(test_led_write_high);
      wait(0.5);
      RUN_TEST(test_led_write_low);
      wait(0.5);
      i++;
    }

  UNITY_END();
}
#endif

How are you going to run ARM mbed framework on x86 platform? native means the platform, where you are currently now.

Okay, Finally had a chance to test with my St-link too.
It still fails, but atleast more crisp than earlier.

I removed the “local” environment as you pointed out and modified the platform.ini to use stlink as upload_protocol. This is my console output.

PlatformIO Plus (https://pioplus.com) v0.2.3
Verbose mode can be enabled via `-v, --verbose` option
Collected 1 items
====================================================== [test::*] Building... (1/3) ======================================================
[Tue Oct 18 12:47:24 2016] Processing bluepill_f103c8 (upload_protocol: stlink, platform: ststm32, board: bluepill_f103c8, framework: mbed)
-----------------------------------------------------------------------------------------------------------------------------------------
Please wait...
===================================================== [test::*] Uploading... (2/3) =====================================================
[Tue Oct 18 12:47:27 2016] Processing bluepill_f103c8 (upload_protocol: stlink, platform: ststm32, board: bluepill_f103c8, framework: mbed)
-----------------------------------------------------------------------------------------------------------------------------------------
Please wait...
2016-10-18T12:47:29 INFO src/stlink-common.c: Loading device parameters....
2016-10-18T12:47:29 INFO src/stlink-common.c: Device connected is: F1 Medium-density device, id 0x20036410
2016-10-18T12:47:29 INFO src/stlink-common.c: SRAM size: 0x5000 bytes (20 KiB), Flash: 0x10000 bytes (64 KiB) in pages of 1024 bytes
2016-10-18T12:47:29 INFO src/stlink-common.c: Attempting to write 52916 (0xceb4) bytes to stm32 address: 134217728 (0x8000000)
2016-10-18T12:47:31 INFO src/stlink-common.c: Finished erasing 52 pages of 1024 (0x400) bytes
2016-10-18T12:47:31 INFO src/stlink-common.c: Starting Flash write for VL/F0/F3 core id
2016-10-18T12:47:31 INFO src/stlink-common.c: Successfully loaded flash loader in sram
2016-10-18T12:47:33 INFO src/stlink-common.c: Starting verification of write complete
2016-10-18T12:47:33 INFO src/stlink-common.c: Flash written and verified! jolly good!
====================================================== [test::*] Testing... (3/3) ======================================================
If you don't see any output for the first 10 secs, please reset board (press reset button)
Error: Please specify `upload_port` for environment or use global `--upload-port` option.

Rest of the files are as before.

PS: If you are using a serial connection for sending debug information during live target testing then, you can actually use the blackmagic probe for doing so. BMP firmware implements two serial ports, 1 for running a native GDB server and another for serial passthrough, both on the same USB connection.

Any further help or suggestions???
Has anyone done live target testing on STM32??

Do you see this board in pio device list?

Nope, the st-link does not show up in this device list

This is my console output
krishis-MBP:~ krishi$ pio device list
/dev/cu.Bluetooth-Incoming-Port
-------------------------------
Hardware ID: n/a
Description: n/a
The st-link shows up in the list of attached devices in osx system report.

The blackmagic probe shows up in the device list though.
This is the console output with the blackmagic probe

krishis-MBP:~ krishi$ pio device list
/dev/cu.Bluetooth-Incoming-Port
-------------------------------
Hardware ID: n/a
Description: n/a
/dev/cu.usbmodemBFDEA5F1
------------------------
Hardware ID: USB VID:PID=1D50:6018 SER=BFDEA5F5 LOCATION=20-1
Description: Black Magic Probe (JC66CoreBoard), (Firmware v1.6-rc0-98-g4055a58)
/dev/cu.usbmodemBFDEA5F3
------------------------
Hardware ID: USB VID:PID=1D50:6018 SER=BFDEA5F5 LOCATION=20-1
Description: Black Magic Probe (JC66CoreBoard), (Firmware v1.6-rc0-98-g4055a58)

The BFDEA5F1 is the one that runs the gdb server for debugging and programming, BFDEA5F3 is the one which is a serial pass through.

I got you. Looks like we should add extra option to platformio.ini that explains PlatformIO which port to use for serial communication/monitoring. For example,

[env:mytestenv]
test_port = /dev/cu.usbmodemBFDEA5F3

What do you think?

> Warning! Ignore unknown "test_port" option from "[env:] section"
This is the message in the console output when I use BMP and gdb as upload_protocol and specified the monitor_port option in platformio.ini file.

Secondly, I think I did not explain it properly. The serial passthrough is not the traditional “arduino” serial pass through. Two RX/TX pins are broken out on the bluepill BMP board which I can then connect to the target board UART port and observe the UART on the same USB cable.

The BMP firmware essentially multiplexes the GDB server and a seperate UART monitoring port on the same USB connection.

Did I explain it clearly??

As I understand it, if I should use the unit test protocol in pio then my target firmware should also be streaming data over its UART port and then the BMP UART monitoring port can be used to test the live target.

But then again, the error above shows that the monitor_port is being ignored. Maybe because it is using gdb as the upload_protocol??

  1. Please be careful, "Looks like we should add extra option ". It means, that we SHOULD add and currently this option doesn’t work :slight_smile:
  2. Currently, Unit Testing will work in case when one of Arduino, Energia or ARM mbed framework is used. In this case, you should HIDE own “serial communication” in source code while UNIT_TEST in progress. As I’ve mentioned before, we will use Serial port to fetch test results from target device.

Does for you work pio device monitor -p /dev/cu.usbmodemBFDEA5F3?

:slight_smile:
Got it!!

Regarding point 2, the device monitor is able to see both the ports
When I run the above command, the console is opening up a miniterm session, if thats what you are asking.

This is the exact console output, since neither pio nor the target firmware is using the UART the session keeps waiting for any serial data.

> krishis-MBP:DigitalMeter krishi$ pio device monitor -p /dev/cu.usbmodemBFDEA5F3
> --- Miniterm on /dev/cu.usbmodemBFDEA5F3  9600,8,N,1 ---
> --- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---

Hope this helps. Any chance you can piggyback on gdb protocol itself for live target testing.
The gdb server itself runs on the BMP board in firmware. Unit Testing should only be a matter of extracting data from the gdb output which is available to the pio console.
Anything I can do to help you implement this feature using gdb?? I am only asking particularly about gdb because it is a standardised protocol and help will also be much more easily available.

If you write “Serial.print()” (we don’t say here about implementation), will you see results in pio device monitor?

Are you sure that we should do that? Native Serial is a egood candidate to gather the unit test results. Do you use ARM mbed for this board?