Problem with (native) unit testing on a large(r) project

Advanced Solutions Unit Testing
1

After doing trials for several weeks with unit test, I came to the conclusion that there is a problem with native unit tests for larger projects. So I wrote this topic to describe it and maybe some day PlatformIO can come with a solution.

Say you are working on a larger project (>10K lines of code)

  • you organise your code into ‘libraries’ which are under ‘lib’. It cannot be avoided that some of these have hardware dependencies
  • but some are hardware independent, so you want to unit test them in ‘native’
  • problem is that the native environment takes ALL of lib, and tries to compile it, (even if not being used by the test-application) and this build fails because of missing definitions in the native platform…

Simple example : say you have some lib using Arduino Serial.print()
You have another lib doing a circular buffer, with no hardware dependencies.
So you want to unit test the circular buffer, but when compiling all stuff in lib, it fails because Serial is undefined…

Kind regards, Pascal Roobrouck

1 Like
2

So the ringbuffer should be its own lib then.

Alternatively as platformio-examples/unit-testing/arduino-mock at develop · platformio/platformio-examples · GitHub shows, inclusion of the ArduinoFake library provides missing Arduino functionality in native environments so that they can be tested relatively painlessly. I would even argue that that way, the logic closer to the hardware but still ‘only’ relying on Arduino APIs can be tested way better. It also includes FakeIt, a nice mocking software.

Alternatively, libraries can have an extraScript property in which sources can be dynamically excluded or included if they detect that they’re running natively. But alas, the same can be achieved using compile time macros in the code itself. Even simple macros like ARDUINO (which evaluatues to the Arduino IDE version number) can be used for guarding the pieces of code dependent on Arduino.

3

It is in its own subdirectory in lib folder, but the build for the test-application takes all code from lib, including other libraries which don’t make sense in the native environment.

Thanks for the tip on extraScript, I think this will allow to solve many things.

P

1 Like
4

How about creating a “native” configuration environment, disable LDF for it. and manually specify lib_deps names? Also, the other solution could be using lib_ignore.

5

I thought lib_deps is only for external libraries,…
Can it be used for subfolders of /lib ?
Enumerating all libraries which are compatible with [native] would indeed be a solution.

6

Officially - yes, but you can also specify the name of the library. There is a special part of the code that verifies this. Maybe we should document this option. The upcoming PlatformIO Core 5.3 will not process dependencies that are not properly declared (owner/name or external source).

There is a better solution. You can add library.json to each lib/*component* and strictly configure compatible platforms. Please note that you need also to enable strict compatibility mode per a library/component or globally per configuration environment Library Dependency Finder (LDF) — PlatformIO latest documentation

7

Ok, interesting approach. When a lib/component is not compatible with the current platform, what happens ? It is just skipped during that build ? This would be an easy way to exclude hardware dependent libraries from compiling under [native] and would allow me to include mocks for [native] which are then excluded when building for a real target hw, eg [esp32]

Of course a lot can be done with preprocessor as well, but personally I do not like that as it makes the code less readible.

8

Yes, the libraries will be skipped.

9

That’s EXACTLY my problem right now. I can see, it is still not resolved :confused:

Nah, it is the same lib. I just want to test my classes [files] in isolation. I include one file to test it – why does it build all other stuff that I DON’T include?

If I split my code into more libs, it would be artificial, those files belong together. Just not all of them I want/will test.

Yeah, been there. It is OK to get basic global Arduino stuff, but it does not handle all the libs provided by Arduino. For example Servo.h is missing.

So, now, to test a class that does NOT include any Arduino stuff, I must create a big chunk of mocks to satisfy the builder?

I have one “lib”. Actually, I don’t want any libs, that’s just source code of my application, not some “library”. NVM, names don’t matter that much (or do they?). I have my code, I want to test part of it. I don’t want to make artificial structure that will destroy my app design just to test one [or a few] file[s].

Right now I will split my code into 3 “sections” (or “libs” as you call it):

  • pure cpp [that I want to unit test]
  • ArduinoFake–compatibile [that I also can unit test in native]
  • the rest [not testable natively]

I will also make a GH issue to kindly request that files are builded as requested – files that are not being included, should not magically be built [and thus make testing impossible or make TDD slower].

Sorry, I’m a bit bitter now, I just lost 3 days on this issue, read all the docs, and read ALL the testing threads on this forum, just to figure it out…

10
11

For the time being, I have the following solutions:
1* extract your code under test to a separate library, and test it there. As it’s the only code there, you only need one ‘native’ environment.
Disadvantage is you get all the maintenance overhead of another library…

2* test all your native code on the target, io on the host PC.
Disadvantage : It’s slower, because each test needs to be uploaded.

1 Like
12

Yeah, that’s what I meant by saying I will split my code into 3 “sections” (or “libs” as you call it)

Yea, I guess that’s not possible with NodeMCU DevKit – plus, as you said it, it has to be MUCH slower.

How to do TDD on the device? Change one line of test, upload, see it fails, change one line of implementation, upload, see the success.

BTW, I’m surprised that Unit Testing is in the “Advanced” section – I would argue that it should be the basic stuff of every project… Yea, I’m living in a dream world :smiley:

1 Like
13

I am also doing some of my development on ESP32 and I estimate it takes ~ 30 seconds for a build-upload-run cycle.

Default upload_speed is 57600, but on my HW it works fine at 115200, which halfs the upload time.

Further you can set a filter to not run all tests, but only those relevant to the section you are working on.

Finally it’s also faster to have 1 long unitTest application (1 test_ folder) with all the test cases, io splitting the test cases over multiple test_ folders, as each folder needs to have the build - upload - run cycle.

From my experience, in embedded Sw I think unit testing is not yet a common practice, explaining why it is in the advanced section.

2 Likes