Unit test: runs only one testcase on target

[ingmar]

Unity runs only one test case (from two) on the target (Arduino Mega 2560).
That leads to a wrong summary “passed” when the latter testcase had passed, but the other had failed.

When compiled and run under the “native” environment, both cases are executed and show passed / fail correctly.

#include <unity.h>

void test_test1() {
    TEST_ASSERT_TRUE( false );
}

void test_test2() {
    TEST_ASSERT_TRUE( false );
}

...

int main( int argc, char **argv ) {
    UNITY_BEGIN();

    RUN_TEST( test_test1 );
    RUN_TEST( test_test2 );

    UNITY_END();
}

Target:

platformio.ini:

[env:megaatmega2560]
platform = atmelavr
board = megaatmega2560
framework = arduino
build_flags = -I include

[env:native]
platform = native

[ingmar]

Here is the console when run on “native” environment:

[arduhe]

Could you please show us your complete test code - not only the part for the native environment. Without that it’s hard to say where the problem comes from …

[ingmar]

Hi, this was the complete test code, except setUp() and tearDown(). Basically only the boilerplate code that “tests” these two testcases.
Meanwhile, I have extended it to more testcases, and I vary the tests activated in main(). Sometimes, it runs all of them, sometimes only the last one, sometimes only one but this one twice ! Sometimes, it correcly shows the overall “failed”, sometimes it shows “succeed” even when one (not run) testcase should fail.
I cannot see a pattern here, but it seems like a parsing problem.

This setup here runs only test4, and shows “succeeded”, although test3 should fail:

#include <unity.h>

void test_test1() {
    TEST_ASSERT_TRUE( true );
}

void test_test2() {
    TEST_ASSERT_TRUE( true );
}

void test_test3() {
    TEST_ASSERT_TRUE( false );
}

void test_test4() {
    TEST_ASSERT_TRUE( true );
}

void test_test5() {
    TEST_ASSERT_TRUE( false );
}

void test_test6() {
    TEST_ASSERT_TRUE( true );
}

void test_test7() {
    TEST_ASSERT_TRUE( false );
}


void setUp( void ) {}
void tearDown( void ) {}

int main( int argc, char **argv ) {
    UNITY_BEGIN();

    // RUN_TEST( test_test1 );
    // RUN_TEST( test_test2 );
    RUN_TEST( test_test3 );
    RUN_TEST( test_test4 );
    // RUN_TEST( test_test5 );
    // RUN_TEST( test_test6 );
    // RUN_TEST( test_test7 );

    UNITY_END();
}

This setup should run 3 tests:

    // RUN_TEST( test_test1 );
    // RUN_TEST( test_test2 );
    RUN_TEST( test_test3 );
    // RUN_TEST( test_test4 );
    RUN_TEST( test_test5 );
    RUN_TEST( test_test6 );
    // RUN_TEST( test_test7 );

but shows:

[arduhe]

AFAIK arduino programs do not need an explicit main()-method as this is provided by the framework.

Instead you need a setup() (not setUp() ) and a loop() method which will be called from the provided main() method.

Could you please try this example (based on the description you can find here)

#include "Arduino.h"
#include <unity.h>

void test_test1() {
    TEST_ASSERT_TRUE( false );
}

void test_test2() {
    TEST_ASSERT_TRUE( false );
}

int do_test()  {

    UNITY_BEGIN();

    RUN_TEST( test_test1 );
    RUN_TEST( test_test2 );

    return UNITY_END();
}

void setup() {
    do_test();
}

void loop() {
}

[ingmar]

Exact same behavior, i.e. when testcases 3, 5, 6 are activated, 6 is executed twice, the others not.

I followed this tutorial Unit Testing with PlatformIO: Part 1. The Basics | PlatformIO Labs (piolabs.com) and the following post. The charme with my own main() function is that it works for both environments, because, it replaces Arduino’s main(), therefore does not need setup() and loop() any more.

My conclusions:

• It evaluates the list of RUN_TEST macros, since when I comment single testcases out (or in), the result changes, too. Thus, it does not seem like a refresh issue.
• Since it works perfectly on the native platform, but not on the target, I assume that is something with the different toolchain or the macro implementation on the other platform.
• The macro is in \.pio\libdeps\megaatmega2560\unity_internals.h; wheras the implementation for native and Mega2560 is the same. So maybe it is an 8-bit vs. 64-bit issue.
• There is actually a similar post concerning double testcases. Maybe it relates to the same issue Platformio unit test runs the same test case twice - Advanced Solutions / Unit Testing - PlatformIO Community

Whatever… I am happy that I can at least unit test in the native environment. Should do the job me now.

PS: thanks for helping to far!

[maxgerhardt]

Does that happen with the latest core? (CLI → pio upgrade --dev).

If yes → file a bug in https://github.com/platformio/platformio-core/issues/

[ingmar]

I guess I found it.

Turns out:

• all tests are actually executed once. (I have introduced a global counter which is incremented by each testcase, and let that be the “failed” result of the last one to be displayed.)
• Furthermore, I noticed that the result varys when I UnityPrint() something before each RUN_TEST or directly in UnityDefaultTestRun() in unity.c. Depending on the length of that additional text (between 5 and 6 characters), it worked or not. First I thought of kind of a memory issue…

These were the clues that this might actually be a timing problem in a way that the interface that communicates results back to platformio’s test runner is not yet properly initialized, so that the test runner does not get all results or reads from a corrupted buffer (therefore the wrong displayed number of tests, or some outdated content that is read again).

Introducing a little delay before the test run does the job, however requires that the original Arduino main() is used, therefore a small switch between native and target environments:

In platformio.ini:

[env:native]
platform = native
build_flags = -DPLATFORM_NATIVE

Test source file:

#include <unity.h>

#ifndef PLATFORM_NATIVE
#include <Arduino.h>
#endif


//... testcases here


void do_test()
{
    UNITY_BEGIN();
    RUN_TEST( test_case1 );
    RUN_TEST( test_case2 );
    RUN_TEST( test_case3 );
    UNITY_END();
}


void setUp() {}
void tearDown() {}

#ifdef PLATFORM_NATIVE
int main( int argc, char **argv )
{
    do_test();
}
#else
void setup()
{
    delay( 500 );   // Let the communication interface settle
    do_test();
}
void loop() {}
#endif

[maxgerhardt]

@ingmar

void setup()
{
    delay( 500 );   // Let the communication interface settle
    do_test();
}

Oh. But that’s a documented thing.

https://docs.platformio.org/en/latest/advanced/unit-testing/frameworks/unity.html

platformio/platformio-examples/blob/2585734bbf4aaef813079900072aa50bda9a04b9/unit-testing/calculator/test/test_embedded/test_calculator.cpp#L47-L58

[ingmar]

You are right. Now as you write it, I see it in the other examples and issue reports. Seems that I have picked exactly that tutorial that did not mention the timing aspect with the target hardware.
Never mind, I have learned a lot about unity and its internals (by scanning its code). That’s how we learn - sometimes.
Thanks anyway for answering!