How to set up a (potentially) large project?

This is a bit of a broad topic, but I’m trying to find a way to manage a C++17 project with PlatformIO which can deal with lots of “moving parts”: number of µC architectures, target boards, selected features, and test-driven / debug / release builds.

I found the Marlin project, which does appear to cover a huge variety of build variants. There’s also Tasmota, which is more ESP8266-/ESP32-centric.

My main targets are STM32 series µCs, but RasPi Pico and Teensy should preferably also fit in. These are all ARM Cortex, but again, I don’t want to rule out the possibility to handle other architectures, such as Espressif and RiscV. PlatformIO is clearly up to it, so on the build and toolchain side, there appear to be no roadblocks.

Before bringing in existing real code, I have now created a new small setup which lets pio test work on native and a few different STM32 families. The native build is important because it allows much faster development cycles for those parts of the project which don’t deal directly with hardware.

The main puzzle is how to structure the code. Make the source files and directory structure too fine-grained, and you end up with tons of header file dependencies. Make it too coarse, and you get into piles of #ifdef lines all over the place. A useful middle road seems to be to structure the libraries in lib/ in per-architecture groups, with a library.json file in each containing a { "platforms": "..." } line to limit its auto-inclusion by PIO. This hinges on having the following lines in platformio.ini:

[env]
lib_compat_mode = strict

To complicate matters further, I’m hell-bent on using a test-driven approach as much as possible. Because anything as varied as this will break down at some point in time if development progresses without automated testing. I’m using PIO’s “Unity” support for embedded builds and DocTest for native (which is absolutely brilliant, but not compatible with cross-compilation, alas).

And if that’s not enough, let me add that the project uses code-generation to simplify maintenance and avoid writing boilerplate code (scripted in Python). So there are steps in the build process which must go through the source code and generate other parts of the source code before the compilation can run to completion.

All the code for this project is open source, with a very early design on GitHub, but I don’t like the way it’s evolving and slowly starting to grind to a halt. In the current design, I use PyInvoke as a wrapper around PlatformIO, but while it does simplify some things, it’s not that good a fit to really warrant its use. Might as well extend PIO itself using its extra_script feature and advanced scripting in Python.

Development of this project takes place on MacOS and Linux (incl. RasPi).

The Marlin project appears to manage most of its variability through a features.ini file, which specifies which source files to add to a build using src_filter. This is done via fairly elaborate Python scripts, digging deep into the innards of PIO, as far as I can tell. I find it hard to wrap my mind around it all, and haven’t found documentation for most of the env and projenv objects available in PIO’s advanced scripting. My current thinking is that this aspect could be simplified by generating a header with #include’s to define which features and code are to be included in each build.

So my first question is: are there other maintained OSS projects I could look at for insight, which handle a large variety of build targets and use PlatformIO?

My second question is: do you have suggestions on how to set things up for such a wide-ranging & open-ended system? This project has very few dependencies (later extensions will most probably change that). Where would you put say LwIP, if some of the builds were to use networking? Or TinyUSB, if a build needs to support device- or host-USB, for example? Along what dimensions would you structure the variability in lib/: per architecture? per target board? per feature?

Oops, that’s a bit more than a single question :slight_smile:
And of course there is no single “right” answer to all of the above.
But any tips and pointers would help me a lot - and be much appreciated.
Also, collecting a few relevant URLs in this thread could be of use to others.

-jcw

To be honest, I’m unlikely ever to need to do such a project. However, if I were to do it, I would take a good long look at Marlin because that pretty much seems to have cracked it in allowing the “same” code to be applied to numerous 3d printer controller boards.

You might also need to go down the route that Marlin have taken and keep the separate configs, well, separate to avoid polluting the main code. They are two separate areas, for the same thing admittedly.

I’m not a great user of Test Driven Development myself, my projects are tiny and merely playthings, but it is an area I need to develop, in case I get ideas above my station! :grin: I’m assuming that someone at Marlin also does Testing? But it’s a bad idea to assume anything.

If Marlin do do testing, perhaps their source tree will give you a clue.

Sorry I’m not much help here. Good luck.

Cheers,
Norm.

Thanks - yes, I’ve been browsing through the Marlin repository a bit, and it does handle lots of target boards.

I have not found any test code in it so far. Maybe this kind of project is more an “it works or it doesn’t” type which can avoid a separate test setup. Or perhaps testing something which so deeply depends on hardware is too complicated.

1 Like

Ever thought of using Eclipse or Sloeber (Eclipse with Arduino extensions) for your task?
They were designed for huge projects and support multiple configurations dependencies and working sets out of the box.
And Eclipse is well supported by STM, since some of the professional STM IDE’s are just Eclipse variants.
And debugging is quite simple :slightly_smiling_face: It took me 1 hour to have it working.

Thanks for your suggestion but I don’t understand how this would help (also, I’m not using the Arduino runtime but just the CMSIS headers, at least for ARM). It seems to me that in essence, the Eclipse IDE is a GUI on top of existing toolchains.

This is not really about the builds, which PIO handles just fine, it’s about how to structure source and config files such that different target builds can include/exclude a variety of functionality, with each module possibly depending on 3rd party packages/libraries.

Apologies for this somewhat vague description, I realize it’s a bit of a vague/hand-waving Q.