Best practices framework and platform independent structure of lib and code

**EDIT5:** It seems by default the LDF mode chain, only looks at libname.cpp dependencies, so anything not references in this cpp file wont be available, do work with local libraries the way I want you need to change the following setting in platformio.ini:
lib_ldf_mode = deep

I think this whole domain is quite undocumented / unclear and would have loved if there where established best pracises in the main docs of platformio for this kind of setup

Hello, I am struggling getting my project from initial proof of concept to something I am more proud of.

I want all my logic to be testable and not dependent on a specific framework or platform, so I have created a local library called “hal” for my project.

I have struggled back and forth with library.json and extrascript.

I now have the following extrascript


framework = env.get("PIOFRAMEWORK")

print(env.Dump())

if isinstance(framework, list):
    framework = framework[0]
else:
    framework = "native"

platform = env.get("PIOPLATFORM")

env.Replace(SRC_FILTER=["+<*>", "-<framework*>", "-<platform*>", "+<framework/%s>" % framework, "+<platform/%s>" % platform])

This works as I want however, for some weird reason, whenever I have cpp files in subfolders I get the following error trying to compile my hal_arduino.cpp

2 | #include <EEPROM.h>
  |          ^~~~~~~~~~

compilation terminated.
*** [.pio/build/d1_mini/lib224/hal/framework/arduino/hal_arduino.cpp.o] Error 1

if I have this include in my lib/hal/hal.cpp it works, iif i have it in lib/hal/hal_blabla.cpp it doesnt, if i have it in subfolder it does not work.

for some reason Arduino.h works to include, but not EEPROM.h from my esp8266 platform.

Any ideas? Any best practices to structure projects with support for multiple platforms and unit testing of classes?

EDIT: flattening it seems to work… so with this script it seems to work, so probably its the subdirs causing the issue…


framework = env.get("PIOFRAMEWORK")

print(env.Dump())

if isinstance(framework, list):
    framework = framework[0]
else:
    framework = "native"

platform = env.get("PIOPLATFORM")

env.Replace(SRC_FILTER=["+<*>", "-<hal_*>", "-<platform*>", "+<hal_%s>" % framework, "+<hal_%s_%s>" % (framework, platform)])

However, best practices would be nice here, I cant be the only one struggling defining this?

Edit2: Stil does not work, it just didnt pickup my cpp files now so I got linkage errors instead

EDIT3: After hardcoding the src_filter to only care about the files I want for specific build i am back to include EEPROM issue…

    2 | #include <EEPROM.h>
      |          ^~~~~~~~~~
compilation terminated.
*** [.pio/build/d1_mini/lib224/hal/hal_arduino.cpp.o] Error 1```


EDIT4: If I add #include <EEPROM.h> into hal.cpp, including it in hal_arduino.cpp works.... (not very usefull as hal.cpp becomes HW dependent) is ther some sort of "smart dependency scanner" that is messing with me?

When i enable --verbose under build optons i see this for the files

Including EEPROM.h works in hal.cpp but not hal_arduino.cpp

COLLECT_GCC_OPTIONS='-o' '.pio/build/d1_mini/lib224/hal/hal.cpp.o' '-c' '-std=c++17' '-fno-rtti' '-std=gnu++17' '-fno-exceptions' '-v' '-Os' '-mlongcalls' '-mtext-section-literals' '-falign-functions=4' '-U' '__STRICT_ANSI__' '-ffunction-sections' '-fdata-sections' '-Wall' '-Werror=return-type' '-free' '-fipa-pta' '-D' 'PLATFORMIO=60113' '-D' 'ESP8266' '-D' 'ARDUINO_ARCH_ESP8266' '-D' 'ARDUINO_ESP8266_WEMOS_D1MINI' '-D' 'PIO_FRAMEWORK_ARDUINO_LWIP2_IPV6_HIGHER_BANDWIDTH' '-D' 'USE_IPV6' '-D' 'F_CPU=80000000L' '-D' '__ets__' '-D' 'ICACHE_FLASH' '-D' '_GNU_SOURCE' '-D' 'ARDUINO=10805' '-D' 'ARDUINO_BOARD="PLATFORMIO_D1_MINI"' '-D' 'ARDUINO_BOARD_ID="d1_mini"' '-D' 'FLASHMODE_DIO' '-D' 'LWIP_OPEN_SRC' '-D' 'NONOSDK22x_190703=1' '-D' 'TCP_MSS=1460' '-D' 'LWIP_FEATURES=1' '-D' 'LWIP_IPV6=1' '-D' 'VTABLES_IN_FLASH' '-D' 'FP_IN_IROM' '-D' 'MMU_IRAM_SIZE=0x8000' '-D' 'MMU_ICACHE_SIZE=0x8000' '-I' 'lib/configuration' '-I' 'lib/sensors' '-I' 'lib/hal' '-I' '.pio/libdeps/d1_mini/DS18B20/src' '-I' '.pio/libdeps/d1_mini/OneWire' '-I' '.pio/build/d1_mini/core' '-I' '/home/node/.platformio/packages/framework-arduinoespressif8266/tools/sdk/include' '-I' '/home/node/.platformio/packages/framework-arduinoespressif8266/cores/esp8266' '-I' '/home/Compiling .pio/build/d1_mini/lib224/hal/hal_arduino.cpp.o
node/.platformio/packages/toolchain-xtensa/include' '-I' '/home/node/.platformio/packages/framework-arduinoespressif8266/tools/sdk/lwip2/include' '-I' '/home/node/.platformio/packages/framework-arduinoespressif8266/variants/d1_mini'
 /home/node/.platformio/packages/toolchain-xtensa/bin/../libexec/gcc/xtensa-lx106-elf/10.3.0/cc1plus -quiet -v -I lib/configuration -I lib/sensors -I lib/hal -I .pio/libdeps/d1_mini/DS18B20/src -I .pio/libdeps/d1_mini/OneWire -I .pio/build/d1_mini/core -I /home/node/.platformio/packages/framework-arduinoespressif8266/tools/sdk/include -I /home/node/.platformio/packages/framework-arduinoespressif8266/cores/esp8266 -I /home/node/.platformio/packages/toolchain-xtensa/include -I /home/node/.platformio/packages/framework-arduinoespressif8266/tools/sdk/lwip2/include -I /home/node/.platformio/packages/framework-arduinoespressif8266/variants/d1_mini -iprefix /home/node/.platformio/packages/toolchain-xtensa/bin/../lib/gcc/xtensa-lx106-elf/10.3.0/ -U __STRICT_ANSI__ -D PLATFORMIO=60113 -D ESP8266 -D ARDUINO_ARCH_ESP8266 -D ARDUINO_ESP8266_WEMOS_D1MINI -D PIO_FRAMEWORK_ARDUINO_LWIP2_IPV6_HIGHER_BANDWIDTH -D USE_IPV6 -D F_CPU=80000000L -D __ets__ -D ICACHE_FLASH -D _GNU_SOURCE -D ARDUINO=10805 -D ARDUINO_BOARD="PLATFORMIO_D1_MINI" -D ARDUINO_BOARD_ID="d1_mini" -D FLASHMODE_DIO -D LWIP_OPEN_SRC -D NONOSDK22x_190703=1 -D TCP_MSS=1460 -D LWIP_FEATURES=1 -D LWIP_IPV6=1 -D VTABLES_IN_FLASH -D FP_IN_IROM -D MMU_IRAM_SIZE=0x8000 -D MMU_ICACHE_SIZE=0x8000 lib/hal/hal.cpp -quiet -dumpbase hal.cpp -mlongcalls -mtext-section-literals -auxbase-strip .pio/build/d1_mini/lib224/hal/hal.cpp.o -Os -Wall -Werror=return-type -std=c++17 -std=gnu++17 -version -fno-rtti -fno-exceptions -falign-functions=4 -ffunction-sections -fdata-sections -free -fipa-pta -o /tmp/ccXx8WyH.s
COLLECT_GCC_OPTIONS='-o' '.pio/build/d1_mini/lib224/hal/hal_arduino.cpp.o' '-c' '-std=c++17' '-fno-rtti' '-std=gnu++17' '-fno-exceptions' '-v' '-Os' '-mlongcalls' '-mtext-section-literals' '-falign-functions=4' '-U' '__STRICT_ANSI__' '-ffunction-sections' '-fdata-sections' '-Wall' '-Werror=return-type' '-free' '-fipa-pta' '-D' 'PLATFORMIO=60113' '-D' 'ESP8266' '-D' 'ARDUINO_ARCH_ESP8266' '-D' 'ARDUINO_ESP8266_WEMOS_D1MINI' '-D' 'PIO_FRAMEWORK_ARDUINO_LWIP2_IPV6_HIGHER_BANDWIDTH' '-D' 'USE_IPV6' '-D' 'F_CPU=80000000L' '-D' '__ets__' '-D' 'ICACHE_FLASH' '-D' '_GNU_SOURCE' '-D' 'ARDUINO=10805' '-D' 'ARDUINO_BOARD="PLATFORMIO_D1_MINI"' '-D' 'ARDUINO_BOARD_ID="d1_mini"' '-D' 'FLASHMODE_DIO' '-D' 'LWIP_OPEN_SRC' '-D' 'NONOSDK22x_190703=1' '-D' 'TCP_MSS=1460' '-D' 'LWIP_FEATURES=1' '-D' 'LWIP_IPV6=1' '-D' 'VTABLES_IN_FLASH' '-D' 'FP_IN_IROM' '-D' 'MMU_IRAM_SIZE=0x8000' '-D' 'MMU_ICACHE_SIZE=0x8000' '-I' 'lib/configuration' '-I' 'lib/sensors' '-I' 'lib/hal' '-I' '.pio/libdeps/d1_mini/DS18B20/src' '-I' '.pio/libdeps/d1_mini/OneWire' '-I' '.pio/build/d1_mini/core' '-I' '/home/node/.platformio/packages/framework-arduinoespressif8266/tools/sdk/include' '-I' '/home/node/.platformio/packages/framework-arduinoespressif8266/cores/esp8266' '-I' '/home/node/.platformio/packages/toolchain-xtensa/include' '-I' '/home/node/.platformio/packages/framework-arduinoespressif8266/tools/sdk/lwip2/include' '-I' '/home/node/.platformio/packages/framework-arduinoespressif8266/variants/d1_mini'
 /home/node/.platformio/packages/toolchain-xtensa/bin/../libexec/gcc/xtensa-lx106-elf/10.3.0/cc1plus -quiet -v -I lib/configuration -I lib/sensors -I lib/hal -I .pio/libdeps/d1_mini/DS18B20/src -I .pio/libdeps/d1_mini/OneWire -I .pio/build/d1_mini/core -I /home/node/.platformio/packages/framework-arduinoespressif8266/tools/sdk/include -I /home/node/.platformio/packages/framework-arduinoespressif8266/cores/esp8266 -I /home/node/.platformio/packages/toolchain-xtensa/include -I /home/node/.platformio/packages/framework-arduinoespressif8266/tools/sdk/lwip2/include -I /home/node/.platformio/packages/framework-arduinoespressif8266/variants/d1_mini -iprefix /home/node/.platformio/packages/toolchain-xtensa/bin/../lib/gcc/xtensa-lx106-elf/10.3.0/ -U __STRICT_ANSI__ -D PLATFORMIO=60113 -D ESP8266 -D ARDUINO_ARCH_ESP8266 -D ARDUINO_ESP8266_WEMOS_D1MINI -D PIO_FRAMEWORK_ARDUINO_LWIP2_IPV6_HIGHER_BANDWIDTH -D USE_IPV6 -D F_CPU=80000000L -D __ets__ -D ICACHE_FLASH -D _GNU_SOURCE -D ARDUINO=10805 -D ARDUINO_BOARD="PLATFORMIO_D1_MINI" -D ARDUINO_BOARD_ID="d1_mini" -D FLASHMODE_DIO -D LWIP_OPEN_SRC -D NONOSDK22x_190703=1 -D TCP_MSS=1460 -D LWIP_FEATURES=1 -D LWIP_IPV6=1 -D VTABLES_IN_FLASH -D FP_IN_IROM -D MMU_IRAM_SIZE=0x8000 -D MMU_ICACHE_SIZE=0x8000 lib/hal/hal_arduino.cpp -quiet -dumpbase hal_arduino.cpp -mlongcalls -mtext-section-literals -auxbase-strip .pio/build/d1_mini/lib224/hal/hal_arduino.cpp.o -Os -Wall -Werror=return-type -std=c++17 -std=gnu++17 -version -fno-rtti -fno-exceptions -falign-functions=4 -ffunction-sections -fdata-sections -free -fipa-pta -o /tmp/ccZMdnXT.s

I’m not gonna reply to your specific issues here, but to the topic title – because my goal was/is also writing code with clear separation of domain logic and framework as delivery mechanism - while being able to unit test as much as I can.

This is the result:

It is still a mess, but with a few things I am quite happy:

  • there are 3 main separate PIO envs: native (for framework independent code), native_arduino (for code that uses Arduino libs, but I can mock them) and nodemcu (for framework specific code, that I don’t test)
  • as I’ve mentioned before: I’m trying to test everything native plus a bit of native_arduino
  • I’m OOP all the way
  • Folder structure = namespaces
  • Some code is “generic” (not connected to the framework in any way), like DnAppLogger/src/DnApp/Logger
  • Some code is Arduino framework related, like DnAppArduino/src/DnApp/Arduino
  • Some code is my specific hardware related, like DnAppEsp/src/DnApp/Esp
  • Wherever there is an interaction between “my” code and framework’s code, I’m writing “interfaces” (abstract base classes) to define the “framework agnostic protocol”

It’s not perfect, I’ve been learning a lot (basically my first and only contact with C++), some concepts changed with time → hence mess → but I like it. And it works :wink:

Over-engineered for Arduino? Probably yes. But it’s a learning opportunity anyway.

@dvdnwk Great thanks for sharing. I will look for inspiration in your project. =)

I just learned that with the deep+ mode, my footprint with just the ESP32 + clag etc core becomes almost a megabyte already =/

EDIT: Actually, with library.json platform restrictions I got it to build with chain+, but base libs are still eating all flash, so guess its just a cost of using std libs etc and ESP32 lacking support for link time optimisations

I can’t confirm that.
I use the STL libraries (vector, map, algorithm etc.) all the time.

Your project seems to be very extensive and to work with user-defined scripts. Please check this with a simpler project.

@sivar2311
same project, same libs.
ESP8266 flash use: 600KB
ESP32 flash use: 1.1MB

some highlight inspecting usage on ESP32:
xtensa-esp32-elf: 249KB
ESP32S2: 192KB

Sorry, I didnt know that you compare esp8266 against esp32. But even with a plain empty sketch there will be differences in size. Because they are different systems.