Unit Testing on an Arduino Micro

Advanced Solutions Unit Testing
1

Hi everyone! I am struggling with unit testing, and I am slowly losing my will to live… I use PIO (Core 4.3.4 Home 3.2.3) in VScode (1.47.0) on a 64 bit Windows 10 machine. I am working on a firmware project that uses an Arduino Micro and an Arduino Mega for prototyping (it’s a master-slave system).

A while ago, I started investigating unit testing for that project, and after much wailing and gnashing of teeth and a lot of googling I finally managed to get tests running on my Mega. Now, I am trying to run tests on my Micro. I created a test file much the same as my Mega test file, I run the Test task, the tests compile and the upload starts, but then the Micro disappears from Device Manager, never to be heard of again. No LED, nothing. if I reset it, it goes into the bootloader, but once that times out, it’s gone again. Here’s my ini:

[env:micro]
platform = atmelavr
board = micro
framework = arduino

I think this problem is caused by the fact that the Micro uses a native USB connection rather than UART for serial communication. I can recover the board by resetting it and flashing a blink script while the bootloader is still there.

Fine, I thought, you are really supposed to write your business logic platform independent, and do unit tests natively. So I set out to do just that. However, this time, after much googling and searching this forum, reading the docs and studying the examples, I still can’t get any tests to run. My library uses no AVR headers so it should compile just fine for x86. The native platform doesn’t work at all. I only get:

Building...
'cc' is not recognized as an internal or external command,
operable program or batch file.
*** [.pio\build\native\test\output_export.o] Error 1
'cc' is not recognized as an internal or external command,
operable program or batch file.
*** [.pio\build\native\UnityTestLib\unity.o] Error 1

I also cloned the calculator example and tried to run the native unit tests, same result. No dice.

So I tried to use x86 as platform. First I ran into the test_transport problem, but I managed to figure this out by exactly following the docs. Alas, the rejoicing was brief, because now I get:

Building...
Uploading...
*** Do not know how to make File target `upload' (C:\git_repositories\combivalve_firmware\CombiValve_firmware\upload).  Stop.

And I could not figure this one out yet. I’m stuck, and such is my despair that going back to Atmel Studio is being considered.

So, I have three questions, really. First, can anyone tell me how to run my unit tests on my Micro? Second, can someone tell me how to run my unit tests on Windows?

And third, can anyone recommend a guide or some docs for the whole unit testing business that is actually useful? The examples that are there are really cute and all, but after spending A LOT of time trawling through the PIO docs and example repos and reading blog posts, I still can’t figure out how to make my own unit tests work. I might be a bit daft, in fact, that’s a real possibility, but continually running against walls gets really frustrating. I’d REALLY appreciate a good how-to…

1 Like
2

I have some updates. The problem with the semi-bricked Micro was indeed related to the native USB. I had an int main(void) rather than the setup and loop functions Arduinoland demands. Once I “rectified” this, the Micro is not bricked any more. Alas, still not getting any further. Since the bootloader and the application appear as different COM ports, if I run test as is, I get:

Error: Traceback (most recent call last):
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\platformio\__main__.py", line 109, in main
    cli()  # pylint: disable=no-value-for-parameter
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\click\core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\click\core.py", line 717, in main
    rv = self.invoke(ctx)
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\platformio\commands\__init__.py", line 44, in invoke
    return super(PlatformioCLI, self).invoke(ctx)
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\click\core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\click\core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\click\decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\platformio\commands\test\command.py", line 170, in cli
    "succeeded": tp.process(),
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\platformio\commands\test\embedded.py", line 52, in process
    return self.run()
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\platformio\commands\test\embedded.py", line 83, in run
    line = ser.readline().strip()
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\serial\serialwin32.py", line 273, in read
    raise SerialException("ClearCommError failed ({!r})".format(ctypes.WinError()))
serial.serialutil.SerialException: ClearCommError failed (PermissionError(13, 'The device does not recognize the command.', None, 22))

============================================================

An unexpected error occurred. Further steps:

* Verify that you have the latest version of PlatformIO using
  `pip install -U platformio` command

* Try to find answer in FAQ Troubleshooting section
  https://docs.platformio.org/page/faq.html

* Report this problem to the developers
  https://github.com/platformio/platformio-core/issues

============================================================

If i specify a test_port in my ini, it simply complains:

could not open port 'COM5': FileNotFoundError(2, 'The system cannot find the file specified.', None, 2)

I reckon that latter one could be fixed by just adding a bit of delay before attempting to open the port. It’s past midnight over here so I won’t start diving into that right now, though.

3

OK I couldn’t help myself. I added a two second sleep just to check my hypothesis:

        try:
            sleep(2)
            ser = serial.Serial(
                baudrate=self.get_baudrate(), timeout=self.SERIAL_TIMEOUT
            )
            ser.port = self.get_test_port()
            ser.rts = self.options["monitor_rts"]
            ser.dtr = self.options["monitor_dtr"]
            ser.open()
        except serial.SerialException as e:
            click.secho(str(e), fg="red", err=True)
            return False

This is in the run method of the EmbeddedTestProcessor in embedded.py. Clearly that’s not an elegant solution, I guess trying to open for a while and then timing out might be better. I’ll file a bug report tomorrow. Now, bed time.

4

When using native, the compile is expected to be in the global PATH, aka you’d need to e.g. have MinGW installed and gcc globally available in the commandline. Or use Windows x86 — PlatformIO latest documentation.

Native targets do not implement “Upload” target as there is nothing to upload to. The file is already compiled in .pio\build\environment\<program>.exe which can be run directly.

Works for me.

C:\Users\Maxi>g++ --version
g++ (i686-posix-dwarf-rev1, Built by MinGW-W64 project) 7.2.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
C:\Users\Maxi>cd Desktop\platformio-examples-develop\unit-testing\calculator

C:\Users\Maxi\Desktop\platformio-examples-develop\unit-testing\calculator>platformio test -e native
Verbose mode can be enabled via `-v, --verbose` option
Collected 3 items

Processing test_common in native environment
------------------------------------------------------------------------------------------------------------------------
Building...
Testing...
test\test_common\test_calculator.cpp:48:test_function_calculator_addition       [PASSED]
test\test_common\test_calculator.cpp:49:test_function_calculator_subtraction    [PASSED]
test\test_common\test_calculator.cpp:50:test_function_calculator_multiplication [PASSED]
test\test_common\test_calculator.cpp:51:test_function_calculator_division       [PASSED]

-----------------------
4 Tests 0 Failures 0 Ignored
OK
======================================================================================================== [PASSED] Took 3.05 seconds ========================================================================================================

Processing test_desktop in native environment
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building...
Testing...
test\test_desktop\test_calculator.cpp:48:test_function_calculator_addition      [PASSED]
test\test_desktop\test_calculator.cpp:49:test_function_calculator_subtraction   [PASSED]
test\test_desktop\test_calculator.cpp:50:test_function_calculator_multiplication        [PASSED]
test\test_desktop\test_calculator.cpp:43:test_function_calculator_division:FAIL: Expected 32 Was 33     [FAILED]

-----------------------
4 Tests 1 Failures 0 Ignored
FAIL
======================================================================================================== [FAILED] Took 1.42 seconds ========================================================================================================

Test           Environment    Status    Duration
-------------  -------------  --------  ------------
test_common    uno            IGNORED
test_common    nodemcu        IGNORED
test_common    native         PASSED    00:00:03.052
test_desktop   uno            IGNORED
test_desktop   nodemcu        IGNORED
test_desktop   native         FAILED    00:00:01.419
test_embedded  uno            IGNORED
test_embedded  nodemcu        IGNORED
test_embedded  native         IGNORED
=================================================================================================== 1 failed, 1 succeeded in 00:00:04.471 ===================================================================================================

The one test failure regarding

void test_function_calculator_division(void) {
    TEST_ASSERT_EQUAL(32, calc.div(100, 3));
}

is expected since 100/3 as integer division is 33 of course, not 32.

This issue was known at Wrong upload COM port detected · Issue #55 · platformio/platform-atmelavr · GitHub and closed as a fix was integrated into platform-atmelavr. If it’s again not working, please file an issue there with all the necessary details.

1 Like
5

Hi maxgerhardt, thanks for the quick reply! Things start to make sense, especially once I wasn’t overly tired anymore…

I checked out the issue you linked and it’s actually a distinct issue, albeit with very similar symptoms. I did some more poking and ended up filing an issue here: Unit testing fails with Arduino Micro due to COM port change · Issue #3592 · platformio/platformio-core · GitHub

As for the upload in native, that now makes sense, thanks for explaining! I should’ve checked the source, it becomes quite clear once you look at it.

But the CC problem, sadly, remains. You were right, I did of course not have a separate installation of GCC, I somehow assumed the toolchain would be automagically supplied by PIO. I then installed GCC via MinGW and I have GCC and G++:

C:\Users\Sebastian>gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-cygwin/7.4.0/lto-wrapper.exe
Target: x86_64-pc-cygwin
Configured with: /cygdrive/i/szsz/tmpp/gcc/gcc-7.4.0-1.x86_64/src/gcc-7.4.0/configure --srcdir=/cygdrive/i/szsz/tmpp/gcc/gcc-7.4.0-1.x86_64/src/gcc-7.4.0 --prefix=/usr --exec-prefix=/usr --localstatedir=/var --sysconfdir=/etc --docdir=/usr/share/doc/gcc --htmldir=/usr/share/doc/gcc/html -C --build=x86_64-pc-cygwin --host=x86_64-pc-cygwin --target=x86_64-pc-cygwin --without-libiconv-prefix --without-libintl-prefix --libexecdir=/usr/lib --enable-shared --enable-shared-libgcc --enable-static --enable-version-specific-runtime-libs --enable-bootstrap --enable-__cxa_atexit --with-dwarf2 --with-tune=generic --enable-languages=ada,c,c++,fortran,lto,objc,obj-c++ --enable-graphite --enable-threads=posix --enable-libatomic --enable-libcilkrts --enable-libgomp --enable-libitm --enable-libquadmath --enable-libquadmath-support --disable-libssp --enable-libada --disable-symvers --with-gnu-ld --with-gnu-as --with-cloog-include=/usr/include/cloog-isl --without-libiconv-prefix --without-libintl-prefix --with-system-zlib --enable-linker-build-id --with-default-libstdcxx-abi=gcc4-compatible --enable-libstdcxx-filesystem-ts
Thread model: posix
gcc version 7.4.0 (GCC)

C:\Users\Sebastian>g++ -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-cygwin/7.4.0/lto-wrapper.exe
Target: x86_64-pc-cygwin
Configured with: /cygdrive/i/szsz/tmpp/gcc/gcc-7.4.0-1.x86_64/src/gcc-7.4.0/configure --srcdir=/cygdrive/i/szsz/tmpp/gcc/gcc-7.4.0-1.x86_64/src/gcc-7.4.0 --prefix=/usr --exec-prefix=/usr --localstatedir=/var --sysconfdir=/etc --docdir=/usr/share/doc/gcc --htmldir=/usr/share/doc/gcc/html -C --build=x86_64-pc-cygwin --host=x86_64-pc-cygwin --target=x86_64-pc-cygwin --without-libiconv-prefix --without-libintl-prefix --libexecdir=/usr/lib --enable-shared --enable-shared-libgcc --enable-static --enable-version-specific-runtime-libs --enable-bootstrap --enable-__cxa_atexit --with-dwarf2 --with-tune=generic --enable-languages=ada,c,c++,fortran,lto,objc,obj-c++ --enable-graphite --enable-threads=posix --enable-libatomic --enable-libcilkrts --enable-libgomp --enable-libitm --enable-libquadmath --enable-libquadmath-support --disable-libssp --enable-libada --disable-symvers --with-gnu-ld --with-gnu-as --with-cloog-include=/usr/include/cloog-isl --without-libiconv-prefix --without-libintl-prefix --with-system-zlib --enable-linker-build-id --with-default-libstdcxx-abi=gcc4-compatible --enable-libstdcxx-filesystem-ts
Thread model: posix
gcc version 7.4.0 (GCC)

Yet I still get the complaint from PIO. I also understand now that ‘CC’ is a standin, to be replaced by whatever compiler I have installed. Sorry if this should be obvious, I must confess I haven’t deeply ventured into the mysteries of toolchains yet, so far I have only used pre-packaged toolchains, or Python…

As shown by the console output above, GCC is in my PATH, and I restarted VScode and even my laptop after installing GCC, but to no avail.

6

Hi freezyundead,
I had similar issues with [native] a month ago when I figured out that unity’s unit test wouldn’t allow me mocking dependencies out for a bigger project… then trying to get gtest installed and working was a nightmare. But it works fine now :slight_smile:

If it is just the PATH that’s not working, make sure you added it to the correct environment variable “Path”.
path
Then, restart VS code. If you installed MINGW with the right packets (see nightmare link…), it will not kill you there. Else, use the MINGW’s packet manager to select missing libraries etc.

If this still does not work, let me know; maybe I find some time to compile a step-by-step approach for you.

1 Like
7

Hi schallbert,
Thank you for your suggestion! I added those two packages, but it didn’t change anything. Then, I started digging through the source code again and noticed that what I had locally was quite a bit different from what’s on github. Turns out, I was on platform-native 1.1.0 which, as far as I can tell, just flat out doesn’t work. Since PIO usually gives me a nice red dot icon saying "there’s updates to your packages/platforms/whatever, I assumed I was up to date on all platforms. After updating to 1.1.3 my tests miraculously compile.

With that, almost everything runs. All that’s left is the bug in the Micro test processor, and I guess I’ll fix that and file a pull request…

Thanks to all of you for being awesome rubber duckies! :smiley:

1 Like
8

What about platform = windows_x86 (since you’re on windows)… PlatformIO should then provide all the compiler toolchain itself… native is the odd one out - since it uses the ‘native’ toolchain that you’ve already installed (or so it assumes). :wink:

And it is frustrating… I remember posting earlier about a way to increase the delay between uploading and serial monitor, which seems to be all that’s needed to close that last issue re unit testing. Can’t find that post again now… probably used an extra-script to introduce a brief delay after upload… thus no file modification needed.

EDIT: FOUND IT! :slight_smile: SigFox MKR1200 Unit Testing - If you don't see any output for the first 10 secs, please reset board - #3 by isa56k

9

Yeah, I thought that, too… platform = windows_x86 gives me no end of grief about the test transport. Now that I made native work locally, and also remotely (based on my previous experiences, I fully expected setting up remote unit testing with our GitLab CI to be a pain beyond measure, but it was actually quite straightforward) I think I’ll quit while I’m ahead.

The idea with the extra script is awesome! I’ll try that for sure! Right now, I am working my way through the contribution guide, so I can propose a fix to the issue, but, you won’t guess it… I get errors with tox xD Automated testing is really cursed for me! I think I know what’s causing it tho.

I’ll keep you updated on my progress :wink:

1 Like
10

Your tip with the script was gold! I added the following:

[env:micro]
platform = atmelavr
board = micro
framework = arduino
build_flags = -D MICRO
test_ignore = test_desktop
extra_scripts = post:test/extra_script.py

The extra_script.py:

Import("env")

def after_upload(source, target, env):
    import time
    time.sleep(2)

env.AddPostAction("upload", after_upload)

I found that in order for it to work I needed to place the script in the test folder and point the ini there. There’s an explanation in the docs somewhere. With that, my tests run native and embedded, locally and remotely. Much rejoicing ensues.

Regarding the platformio-core fix: this is priceless… I followed the PIO contribution guide, forked the repo, and ran the unit tests on the code AS IS. And… Drumroll… They failed. That’s it, i give up, I have my solution, I’ll let someone else figure out a proper fix.