Multiple test files for embedded tests

dolfandringa · May 19, 2020, 5:23am

I am running a bunch of embedded tests on my stm32 and for clarity’s sake, I want to split my tests over multiple tests in logical chunks.
What I am using in my native unittests is split the tests into multiple files, each having theyr own namespace, and from the main test file, call the main functions from each namespace. I can do the same in embedded tests by substiuting the main function with setup obviously like this:
test_main.cpp

#if defined(ARDUINO) && defined(UNIT_TEST)
#include <Arduino.h>
#include "test_a.hpp"
#include "test_b.hpp"

void setup() {
    test_a::setup();
    test_b::setup();
}

void loop() {
}
#endif

In test_a.hpp:

#include "unity.h"
namespace test_a {
    void test_something() {
        TEST_ASSERT_EQUAL(1, 1);
    }
    void setup() {
        UNITY_BEGIN();
        RUN_TEST(test_something);
        UNITY_END();
    }
    void loop() {
    }

}

And the same code in test_b.hpp, just renaming the namespace to test_b.

This pattern works fine for native tests, but in embedded tests, it results in only one set of tests to be run:

Starting execution at address 0x08000000... done.
Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

test/test_sample/test_a.hpp:8:test_something    [PASSED]
-----------------------
1 Tests 0 Failures 0 Ignored

I can work around it by calling UNITY_BEGIN/UNITY_END in test_main.cpp and removing it from the other setup functions, but then all tests are reported as coming from test_main.cpp, and I can’t see the original file anymore where the test was defined. Like this:

Starting execution at address 0x08000000... done.
Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

test/test_sample/test_main.cpp:8:test_something [PASSED]
test/test_sample/test_main.cpp:8:test_something [PASSED]
-----------------------
2 Tests 0 Failures 0 Ignored

Is there any good way I can split my embedded tests over multiple files like this? Of course I can make the names of each test unique across all files, but that is what namespaces are meant for, and would make test function names unnecessarily long.

lars.lundh · April 5, 2021, 2:50am

Hello dolfandringa

Did you ever find a solution to this kind of hierarchy?

Can anyone explain how you can get this hierarchy?

test_embedded
-test_a.cpp
-test_b.cpp
-test_c.cpp
test_native
-test_calculator.cpp

But I want to upload only one time to the microcontroller.

I know I can do

test
-test_a/test_a.cpp
-test_b/test_b.cpp
-test_c/test_c.cpp

but that uploads firmware 3 times. and requires 3 lots of boiler plate code.

anyone?

p.s multiple firmware uploads is annoying with ESP32 because it requires a button press each time.
and abstracting common tests into separate files seems like a good practice.

maxgerhardt · April 5, 2021, 9:52am

Upload to the ESP32 should be hands-off automatic if the hardware / dev board has the standard reset circuit via DTR/RTS.

See Unit test recommended structure multi-test - #7 by ivankravets for supported hierarchy. Also see 4th point of doc.

If you want multiple files where the tests are, but only one test run, is not possible to use header files to include everything in one cpp file?

lars.lundh · April 7, 2021, 10:42am

Thank you for your reply,
Turns out its a common problem with “Node32s” development board to need to press the boot load button. hmmm

Thank you very much for the reading material.
I will try again, maybe I came to a conclusion too quickly.
Kind Regards

lars.lundh · April 10, 2021, 1:58am

Hi maxgerhardt

I am failing at putting tests in a .h file
I wrote some example code:
// .h File

#include <unity.h>
#include <string>

class TestA
{
public:

TestA() {}
~TestA() = default;

// #1 Does not work
static void test1(std::string aStringToTest) {
    TEST_ASSERT_EQUAL_STRING("hello", aStringToTest.c_str());
}
// #1 YES Works *** but can not pass variables
static void test2() {
    TEST_ASSERT_EQUAL_STRING("hello", "hello");
}

// #2 Does not work
void test1NotStatic(std::string aStringToTest) {
    TEST_ASSERT_EQUAL_STRING("hello", aStringToTest.c_str());
}
// #2 Does not work
void test2NotStatic() {
    TEST_ASSERT_EQUAL_STRING("hello", "hello");
}

// #3 Would be nice test...
void wouldBeNice1(std::string aStringToTest) {
    TEST_ASSERT_EQUAL_STRING("hello", aStringToTest.c_str());
}
// #3 Would be nice test...
void wouldBeNice2() {
    TEST_ASSERT_EQUAL_STRING("hello", "hello");
}

private:

};

// .cpp File

#include <Arduino.h>
#include <unity.h>
#include "testA.h"

std::string kStringToTest;

void setUp()
{
    kStringToTest = "hello";
}

void tearDown()
{
    kStringToTest = "";
}

void setup()
{
    // #1 YES Works *** but can not pass variables
    UnityTestFunction unityPtrTest1(std::string) = &TestA::test1; //Error
    UnityTestFunction unityPtrTest2 = &TestA::test2;

    // #2 Does not work
    void (TestA::*ptrTest1)(std::string) = &TestA::test1NotStatic;
    void (TestA::*ptrTest2)(void) = &TestA::test2NotStatic;

    // #3 Would be nice test...
    TestA testA; 

    delay(2000);
    UNITY_BEGIN();

    // #1 RUN_TEST expects UnityTestFunction - typedef void (*UnityTestFunction)(void);
    RUN_TEST(unityPtrTest1(kStringToTest));         // Does not work
    RUN_TEST(unityPtrTest2);                        // YES Works *** but can not pass variables

    // #2 Error: a pointer to a bound function may only be used to call the function
    RUN_TEST(ptrTest1(kStringToTest));              // Does not work
    RUN_TEST(ptrTest2);                             // Does not work

    // #3 Would be nice...
    RUN_TEST(testA.wouldBeNice1(kStringToTest));    // Does not work
    RUN_TEST(testA.wouldBeNice2);                   // Does not work

    UNITY_END();
}

void loop()
{
}

The goal is to put test functions in .h File’s testA, testB, testC etc
And run tests from the .cpp

Any help appreciated
hopefully I was just dumb

margaretselzer · September 2, 2022, 8:26am

This was something that was bugging me for a long time too. Today I think I found a solution.

Create one main cpp file for all your tests you want to run with one build/compilation in a test folder.
Put all your tests in separate files
a) in header files somewhere, preferably under the same test folder
b) in cpp files in another test folder, but make sure that in your platformio.ini that specific test folder is added to the test_ignore options.
In your test files, create all your test functions as usual
Add one extra function - e.g. runAllMyTests() - in your test file which should call all other test functions. At the beginning of this function add a call to UnitySetTestFile(__FILE__);
Include all your separate test files in your main test file.
Implement setup() and loop() only in your main test file and call all your runAllMyTests() (you should name them differently or use different namespaces) functions between UNITY_BEGIN() and UNITY_END().

Depending on the granularity of the messages you want, you can choose to use the RUN_TEST() macro in your separate test files to encapsulate your individual test functions within your runAllMyTests() functions or use the macro only in your main cpp file to encapculate only the runAllMyTests() functions themselves (in this case you will get one FAIL/PASS per test file, but the line numbers in an error message will still be correct). In this latter case you have to use unique names though. (Using namespace names within the test macros does not seem to work well, but I did only superficial tests for that.)

I have tested with the following file structure:

/test
   /test_conversion
      test_ConvCalcIfRanges.h
      test_ConvCalcSubranges.h
      test_conversion.cpp

In the headers I put all the individual test functions and all global variables that they depended on in their own unique namespaces, only the runAllMyTests() function remained in global namespace.

test\test_conversion\test_ConvCalcSubranges.h:548: test_find_subranges_and_set_min_max_ok       [PASSED]
test\test_conversion\test_ConvCalcSubranges.h:549: test_find_subranges_and_set_min_max_error    [PASSED]
test\test_conversion\test_ConvCalcSubranges.h:369: test_pairing_primary_and_secondary_subranges_ok: something went wrong        [FAILED]
test\test_conversion\test_ConvCalcSubranges.h:551: test_pairing_primary_and_secondary_subranges_error   [PASSED]
test\test_conversion\test_ConvCalcIfRanges.h:68: test_ifranges  [PASSED]

If you need, you can also set up tearDown_TestFile() functions in all separate test files and then call them from the tearDown() function defined in your main test cpp. One approach for using such a structure could be the following:

using TearDownFuncT = void (*)();
TearDownFuncT _tearDownFunc = nullptr;

void tearDown(void)
{
    // clean stuff up here after tests cases fail
    if (_tearDownFunc)
    {
        _tearDownFunc();
    }
    _tearDownFunc = nullptr;
}

#define SET_TEARDOWN(f) _tearDownFunc = &f
#define NO_TEARDOWN _tearDownFunc = nullptr

Just use SET_TEARDOWN before calling your runAllMyTests() function from a test file where tearDown()is needed.

ingmar · September 26, 2024, 1:30pm

I tried something non-common C style, but it cleans up the place (and it mimics unity’s heavily using macros philosophy):

Put all single testcases (=functions) as static functions in the sub module .c/.cpp files (static to avoid global name clashes).
Call them all with a (non-static) void run_xxx_tests() function
Provide one central “main” test file which contains the main() or setup()/loop() functions, where eventually one function run_all_tests() is called.
The single module run functions are declared as extern (not exactly needed, but for clarity) and then called.
Each module can be enabled or disabled by just commenting.

e.g. in a module:

static void test_one() {
    TEST_ASSERT_TRUE( true );
}

void run_module_1_tests()
{
    RUN_TEST( test_one );
    RUN_TEST( test_two );
    ...
}

And in the central runner:

// Convenience macro declares and calls:
#define RUN_MODULE(run_function) extern void run_function(); run_function();

...

void run_all_tests()
{
    UnityBegin(""); // Do not pass a file name here, since we want the actual module name, not _this_ file name in the output

    // Insert modules' run function that shall be tested:

    RUN_MODULE(run_module_1_tests);
    RUN_MODULE(run_module_2_tests);
    ...

    UNITY_END();
}