How to use custom debug_build_flags per file

gladius · September 5, 2023, 12:45pm

Hello everyone,
I am working on a program that is getting pretty big, and it’s causing some issues when building in debug mode.

I am using a Controllino Mega (Atmel 2560) and I have an ATMEL-ICE attached to it via JTAG to do some debugging using Microchip studio (PlatformIO does not support Atmel2560 JTAG debugging via the ATMEL-ICE AFAIK).

So long as my program was reasonably small, this worked fine when building with:
"debug_build_flags = -O0 -ggdb3 -g3 ".
Now that my program is getting bigger, I end up with 60% RAM used after compiling and less than 1k when running the program, and this is now resulting in undefined behavior. I would like to only compile some files with full debugging info and compile all other files like normal. I know how to do this in other IDES, but I have not been able to figure it out in PlatformIO. I know you can use a Python script to achieve something like this according to the official docs, but I don’t understand how to apply this to my particular problem. If anyone could point me in the right direction, I would be very grateful.
Thank you

maxgerhardt · September 5, 2023, 2:19pm

This should be equivalent Granular optimisation settings - #2 by maxgerhardt.

gladius · September 5, 2023, 4:28pm

This does look like exactly what I need! Sadly, this produces a new error:

Now when building the linker can’t find ANY of my header files. I had this problem once before with this project and fixed it by adding: “lib_ldf_mode = deep+” to my plattformio.ini, but it seems that this is not helping rn.

My platformio.ini looks like this:

[env:controllino_mega]
platform = atmelavr
board = controllino_mega
framework = arduino
monitor_speed = 115200
build_type = debug
lib_ldf_mode = deep+
extra_scripts = pre:modify_debug.py

;debug_build_flags = -Os -ggdb3 -g3
;debug_build_flags = -Og -g
;debug_build_flags = -O0 -g3
;build_flags= -fstack-usage -Wstack-usage=8000

lib_deps =
controllino-plc/CONTROLLINO @ ^3.0.7
br3ttb/PID @ ^1.2.1

And my folder structure is like this:

|–rootdir
|…|–lib
|…|…|–sensors
|…|…- sensor.hpp
|…|…- sensor.cpp
|…|–include
|…|–src
|…|…- main.cpp

The resulting error message when trying to build is:

Verbose mode can be enabled via -v, --verbose option
CONFIGURATION: xhttps://docs.platformio.org/page/boards/atmelavr/controllino_mega.html
PLATFORM: Atmel AVR (4.1.0) > Controllino Mega
HARDWARE: ATMEGA2560 16MHz, 8KB RAM, 248KB Flash
DEBUG: Current (avr-stub) External (avr-stub, simavr)
PACKAGES:

framework-arduino-avr @ 5.1.0

toolchain-atmelavr @ 1.70300.191015 (7.3.0)
LDF: Library Dependency Finder → xhttps://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ deep+, Compatibility ~ soft
Found 16 compatible libraries
Scanning dependencies…
Dependency Graph
|-- CONTROLLINO @ 3.0.7
|-- PID @ 1.2.1
|-- interface
|-- errorHandling
|-- config
|-- SPI @ 1.0
|-- util
|-- statemachine
|-- dilumix
|-- actors
|-- sensors
|-- UART
Building in debug mode
Compiling .pio\build\controllino_mega\src\main.cpp.o
Compiling .pio\build\controllino_mega\lib638\SPI\SPI.cpp.o
Compiling .pio\build\controllino_mega\lib162\CONTROLLINO\Controllino.cpp.o
Compiling .pio\build\controllino_mega\lib4d5\PID\PID_v1.cpp.o
Compiling .pio\build\controllino_mega\lib7b5\util\PID.cpp.o
Compiling .pio\build\controllino_mega\lib7b5\util\analogRead.cpp.o
Compiling .pio\build\controllino_mega\lib7b5\util\filters.cpp.o
Compiling .pio\build\controllino_mega\lib7b5\util\stableCycleTime.cpp.o
src\main.cpp:1:10: fatal error: CInterface.hpp: No such file or directory
Compiling .pio\build\controllino_mega\lib7b5\util\timer.cpp.o

Looking for CInterface.hpp dependency? Check our library registry!

CLI > platformio lib search “header:CInterface.hpp”

Web > xhttps://registry.platformio.org/search?q=header:CInterface.hpp

#include “CInterface.hpp”
^~~~~~~~~~~~~~~~
compilation terminated.
Compiling .pio\build\controllino_mega\lib104\actors\actuonixP16.cpp.o
In file included from .pio\libdeps\controllino_mega\CONTROLLINO\Controllino.cpp:17:0:
.pio\libdeps\controllino_mega\CONTROLLINO\Controllino.h:23:10: fatal error: SPI.h: No such file or directory

Looking for SPI.h dependency? Check our library registry!

CLI > platformio lib search “header:SPI.h”

Web > xhttps://registry.platformio.org/search?q=header:SPI.h

#include “SPI.h”
^~~~~~~
compilation terminated.
Compiling .pio\build\controllino_mega\lib104\actors\dcMotor.cpp.o
Compiling .pio\build\controllino_mega\lib104\actors\valve.cpp.o
.pio\libdeps\controllino_mega\PID\PID_v1.cpp:14:10: fatal error: PID_v1.h: No such file or directory

[…]

I am not sure whether it has something to do with how my file structure is or if I messed up something else…
PS I had to break all links by putting an “x” before them since new users can’t post more than two links, apparently.

maxgerhardt · September 5, 2023, 5:45pm

But the problem immediately goes away when you comment out the extra_scripts = pre:modify_debug.py line?

gladius · September 5, 2023, 5:51pm

Yes, sorry, I should have added that.
It works flawlessly without the extra script but throws errors if I try to use it. I suspect it is the same error again that I originally “fixed” by adding the “deep+” option, but now that the files are treated differently I think the linker once again cannot find the files. Probably, I have to fix the underlying issue that caused this in the first place, but I have no idea on how to do that other than with “deep+”.

maxgerhardt · September 5, 2023, 5:58pm

Yeah so this is actually minimally reproducable with even a pass-through of the original object wrapped in env.Object().

Import("env")

def exchange_debug_for_optimize_flag(node):
    print("Node %s env.GetBuildType()=%s" % (node.name, str(env.GetBuildType())))
    # pass through
    return env.Object(node)

if env.GetBuildType() == "debug":
    env.AddBuildMiddleware(exchange_debug_for_optimize_flag)

So either PlatformIO has a bug here that destroys the library dependency finder or the documenation is wrong: https://docs.platformio.org/en/latest/scripting/middlewares.html.

It does work fine with a very direct return node though, but of course that doesn’t let one modify its compile settings…

gladius · September 5, 2023, 6:12pm

That is quite interesting, but I still suspect I made some stupid mistake and broke the dependency finder in a different way and now because we are using the script to modify the files it is once again broken.
I know PlatformIO does not treat all folders the same and depending on where you store what code it gets compiled differently. Sadly, I don’t understand the details well enough to be sure I am using the folder structure the way it is intended. This solution does seem to work for other people so I am a bit hesitant to call it a bug when the problem is more likely located somewhere between keyboard and chair…

maxgerhardt · September 5, 2023, 6:13pm

No, I’m very sure this bug is in PlatformIO. I was able to reproduce it in an absolute minimalistic setting and opened

github.com/platformio/platformio-core

env.AddBuildMiddleware() breaks library finder

opened 06:12PM - 05 Sep 23 UTC

maxgerhardt

What kind of issue is this? - [X] **PlatformIO Core**. If you’ve found… a bug, please provide an information below. ------------------------------------------------------------------ ### Configuration **Operating system**: Windows 10 x64 **PlatformIO Version** (`platformio --version`): 6.1.12a1 ### Description of problem Per [documentation](https://docs.platformio.org/en/latest/scripting/middlewares.html) it is possible use `env.AddBuildMiddleware(callback, pattern)` to modify the compile settings of specific files (SCons objects). However, even returning `env.Object(node)`, that is without any other options to modify compiler settings, in the callback to this function breaks PlatformIO's ability to identify the library dependencies or at least add the proper include compiler settings to the build of that specific file. Specifically, creating a minimal library `MyTestLib` in the `lib/` folder with one `.h` and `.cpp` file implemeting a simple `return 42;` function and adding `#include "MyTestLib.h"` to the `src/main.cpp` will fail compilation if this `extra_scripts = pre:test.py` is added: ```py Import("env") def pass_through(node): print("Node '%s'" % (node.name)) #print("Type: " + str(type(node))) # pass through works fine #return node # wrapped in env.Object() fails, even when not modfying settings return env.Object(node) env.AddBuildMiddleware(pass_through) ``` Note that `return node` itself works fine, but that eliminates any possibility to modify compiler settings for just that file. #### Steps to Reproduce 1. Download https://github.com/maxgerhardt/pio-middleware-bug-repro 2. Open in VSCode 3. Attempt to compile (--> should fail) 4. Comment in `return node` 5. Build again (--> should work) ### Actual Results ``` Scanning dependencies... Dependency Graph |-- MyTestLib (License: Unknown, Path: C:\Users\Max\temp\gran\lib\MyTestLib) Node 'MyTestLib.cpp' Node 'main.cpp' Building in release mode avr-g++ -o .pio\build\controllino_mega\src\main.cpp.o -c -fno-exceptions -fno-threadsafe-statics -fpermissive -std=gnu++11 -mmcu=atmega2560 -Os -Wall -ffunction-sections -fdata-sections -flto -DPLATFORMIO=60112 -DARDUINO_AVR_MEGA2560 -DF_CPU=16000000L -DARDUINO_ARCH_AVR -DARDUINO=10808 -IC:\Users\Max\.platformio\packages\framework-arduino-avr\cores\arduino -IC:\Users\Max\.platformio\packages\framework-arduino-avr\variants\Controllino_mega src\main.cpp src\main.cpp:2:10: fatal error: MyTestLib.h: No such file or directory ``` ### Expected Results ### If problems with PlatformIO Build System: **The content of `platformio.ini`:** ```ini [env:controllino_mega] platform = atmelavr board = controllino_mega framework = arduino extra_scripts = pre:test.py ``` **Source file to reproduce issue:** ```cpp #include <Arduino.h> #include "MyTestLib.h" void setup() { Serial.begin(9600); } void loop() { int ret = MyTestLib_Function(); Serial.println(ret); delay(1000); } ``` ### Additional info

This could only work where the code did not depend on libraries (at least from lib/).

gladius · September 6, 2023, 12:38pm

Holy crap…
Thanks a lot !