Long compilation "pause" after dependency graph print

lkyslevin · March 28, 2023, 10:49am

Hi there. I am currently developing and thus have to re-compile my code quite offen. Here is how it goes:

Fast progess until the depency graphs have been listed
Quite long pause (>120s)
Fast compilation, linking and upload

Here my compile log:
Executing task: C:\Users\wes39504.platformio\penv\Scripts\platformio.exe run --target upload

Processing firebeetle32 (platform: espressif32; board: firebeetle32; framework: arduino)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/firebeetle32.html
PLATFORM: Espressif 32 (5.2.0) > FireBeetle-ESP32
HARDWARE: ESP32 240MHz, 520KB RAM, 16MB Flash
DEBUG: Current (cmsis-dap) External (cmsis-dap, esp-bridge, esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:
 - framework-arduinoespressif32 @ 3.20005.220925 (2.0.5)
 - tool-esptoolpy @ 1.40201.0 (4.2.1)
 - tool-mkfatfs @ 2.0.1
 - tool-mklittlefs @ 1.203.210628 (2.3)
 - tool-mkspiffs @ 2.230.0 (2.30)
 - toolchain-xtensa-esp32 @ 8.4.0+2021r2-patch3
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain+, Compatibility ~ soft
Found 54 compatible libraries
Scanning dependencies...
Dependency Graph
|-- ArduinoJson @ 6.20.0
|-- Time @ 1.6.1
|-- Adafruit ADS1X15 @ 2.4.0
|-- Adafruit BusIO @ 1.14.1
|-- PubSubClient @ 2.8.0
|-- WiFi @ 2.0.0
|-- config_GPIO
|-- web_and_wifi
|-- NVS
|-- SD_control
|-- FS @ 2.0.0
|-- SD @ 2.0.0
|-- SPI @ 2.0.0
|-- system
|-- AWS
|-- WiFiClientSecure @ 2.0.0
|-- spiffs_config
|-- SPIFFS @ 2.0.0
|-- sleep
|-- FreqCountESP @ 1.0.0
|-- ADC
|-- Wire @ 2.0.0
|-- NTP_and_RTC
|-- Sensor
|-- OTA
|-- Update @ 2.0.0
|-- testing

**LONG PAUSE**

Building in release mode
Compiling .pio\build\firebeetle32\src\system_control.cpp.o
Compiling .pio\build\firebeetle32\libed8\OTA\ota.cpp.o
Compiling .pio\build\firebeetle32\lib80f\Sensor\sensor.cpp.o
Compiling .pio\build\firebeetle32\lib39a\NTP_and_RTC\NTP_and_RTC.cpp.o
Compiling .pio\build\firebeetle32\libd11\ADC\ADC.cpp.o
Compiling .pio\build\firebeetle32\lib00c\spiffs_config\spiffs_config.cpp.o
Compiling .pio\build\firebeetle32\lib8cc\sleep\sleep.cpp.o
Compiling .pio\build\firebeetle32\libf23\AWS\aws_http.cpp.o
Archiving .pio\build\firebeetle32\lib80f\libSensor.a
Archiving .pio\build\firebeetle32\libed8\libOTA.a
Compiling .pio\build\firebeetle32\libf23\AWS\aws_mqtt.cpp.o
Archiving .pio\build\firebeetle32\lib39a\libNTP_and_RTC.a
Compiling .pio\build\firebeetle32\lib344\system\system.cpp.o
Compiling .pio\build\firebeetle32\lib140\SD_control\SD_control.cpp.o
Archiving .pio\build\firebeetle32\lib8cc\libsleep.a
Archiving .pio\build\firebeetle32\libd11\libADC.a
Compiling .pio\build\firebeetle32\lib346\NVS\my_nvs.cpp.o
Compiling .pio\build\firebeetle32\libd0c\web_and_wifi\web_and_wifi.cpp.o
Compiling .pio\build\firebeetle32\lib62f\testing\testing.cpp.o
Archiving .pio\build\firebeetle32\lib344\libsystem.a
Archiving .pio\build\firebeetle32\libf23\libAWS.a
Archiving .pio\build\firebeetle32\lib00c\libspiffs_config.a
Archiving .pio\build\firebeetle32\libd0c\libweb_and_wifi.a
Archiving .pio\build\firebeetle32\lib140\libSD_control.a
Archiving .pio\build\firebeetle32\lib346\libNVS.a
Archiving .pio\build\firebeetle32\lib62f\libtesting.a
Linking .pio\build\firebeetle32\firmware.elf
Retrieving maximum program size .pio\build\firebeetle32\firmware.elf
Checking size .pio\build\firebeetle32\firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [=         ]   8.9% (used 47148 bytes from 532480 bytes)
Flash: [==        ]  16.9% (used 1109917 bytes from 6553600 bytes)
Building .pio\build\firebeetle32\firmware.bin
esptool.py v4.2.1
Creating esp32 image...
Merged 25 ELF sections
Successfully created esp32 image.
Configuring upload protocol...
AVAILABLE: cmsis-dap, esp-bridge, esp-prog, espota, esptool, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa
CURRENT: upload_protocol = esptool
Looking for upload port...
Auto-detected: COM1
Uploading .pio\build\firebeetle32\firmware.bin
esptool.py v4.2.1
Serial port COM1

Some more info:

This is my PlatformIO version: PlatformIO 6.1.7a4
During the “long pause” I am also able to make changes to the code and these changes get compiled. I tested with some print statements.
I noticed this behavior on different Windows 10 machines

Any idea what can cause the long pause and how to improve it?

maxgerhardt · March 28, 2023, 11:15am

May an antivirus software be checking the compiler processes and delaying it?

lkyslevin · March 28, 2023, 11:27am

I had this once and my antivirus software highlights it when it is doing this - I dont think that is the issue.

I just ran my compilation in verbose mod and I am getting a lot of warnings regarding circular dependency issues. Besides, I printed the log to a file and the depency graph listing is more than 100k lines.

10/14 of my my header files have some type if circular dependency. Well, it makes sense that circular depencies are a bad programming practice. But tbh in most cases i think it hard to avoid - or i am just not aware of methods to avoid them. E.g. use of global variables vs dependency injection (which I just googled)

lsangild · March 28, 2023, 11:48am

One of the ways to avoid circular dependencies are using interfaces to decouple your code.

Another more hacky version is to, instead of including a file, just declare the class and make sure someone else defines it.

ivankravets · March 28, 2023, 9:26pm

Could you comment lib_ldf_mode line in your platformio.ini file and try to build again?

lkyslevin · March 29, 2023, 7:56am

So I solved the issue.

Could you comment lib_ldf_mode line in your platformio.ini file and try to build again?

Currently I use chain+ but I tryed with off which raised several Did not find header file issues.

Problem

As described before, I used the pio run -v --target upload command in a PIO-terminal. I recommend adding > somefile.txt to the command (pio run -v --target upload > somefile.txt) because now all the stuff is printed to this file (which is stored in your project) but errors or warnings are still displayed in your build/upload terminal console.

This showed me, that my project, which contains 14 self-made local libraries each consisting of a cpp and h file, that I have a lot of circular dependencies. Since my project only contains one self-made class, which was not introducing any circular dependencie, the problem were my functions introducing the circular dependencies.

Solution

So I had to isolate those functions, which did not have any dependency to one another but only to the core arduino libraries. The other functions, which consisted of these isolated functions were put together into files one hierachy upwards. But I guess the type of circular dependency is different from project to project…

I recommend checking if you have some once in a while and resolve them before it gets too messy.

Finally, my compilation time dropped from >120s to around 20s, which makes development much more enjoyable.