[esp8266] builder tools CheckUploadSize ignores board_upload.maximum_size override in platformio.ini

boxofrox · December 21, 2020, 2:31am

I got nerd sniped by my OCD on this one. I noticed that the program size check always reports a max size of 1044464 despite my override of board_upload.maximum_size = 400 in my platformio.ini.

I tried digging through the code to identify the cause and submit a PR, but I honestly cannot navigate the python code and identify the point at which the board_upload is pulled from the platformio.ini and integrated with the env.BoardConfig() object.

From what I can find, builder/tools/pioplatform.py::PrintConfiguration prints the expected value (400B Flash (see relevant output below)) while builder/tools/pioupload.py::CheckUploadSize receives a different env object that isn’t augmented with the overrides in my platformio.ini. AFAICT, both PrintConfiguration and CheckUploadSize use the same call to env.BoardConfig().get("upload.maximum_size", 0), so I’m lost as to why different different BoardConfig objects are generated.

So I guess my questions are…

Why doesn’t platformio load a single Scons environment for all tasks run in a platformio environment?
For each Scons environment object created, where does platformio load the ini config, identify the board_ entries, and insert them into the BoardConfig object?

# Relevant lines from full output.
HARDWARE: ESP8266 80MHz, 80KB RAM, 400B Flash
...                                ^^^^
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [====      ]  36.0% (used 29500 bytes from 81920 bytes)
Flash: [===       ]  33.3% (used 347872 bytes from 1044464 bytes)
                                                   ^^^^^^^

# Full output
user@comp:project$ pio run -e ota -t checkprogsize
Processing ota (platform: espressif8266; board: d1_mini; framework: arduino)
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/d1_mini.html
PLATFORM: Espressif 8266 (2.6.2) > WeMos D1 R2 and mini
HARDWARE: ESP8266 80MHz, 80KB RAM, 400B Flash
PACKAGES:
 - framework-arduinoespressif8266 3.20704.0 (2.7.4)
 - tool-esptool 1.413.0 (4.13)
 - tool-esptoolpy 1.20800.0 (2.8.0)
 - toolchain-xtensa 2.40802.200502 (4.8.2)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 39 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <PubSubClient> 2.8.0
|-- <Adafruit Unified Sensor> 1.1.4
|-- <Adafruit BME280 Library> 2.1.2
|   |-- <Adafruit Unified Sensor> 1.1.4
|   |-- <SPI> 1.0
|   |-- <Wire> 1.0
|-- <ArduinoOTA> 1.0
|   |-- <ESP8266WiFi> 1.0
|   |-- <ESP8266mDNS> 1.2
|   |   |-- <ESP8266WiFi> 1.0
|-- <SPI> 1.0
|-- <ESP8266WebServer> 1.0
|   |-- <ESP8266WiFi> 1.0
|-- <ESP8266WiFi> 1.0
Building in release mode
Linking .pio/build/ota/firmware.elf
Retrieving maximum program size .pio/build/ota/firmware.elf
Checking size .pio/build/ota/firmware.elf
CheckUploadSize BoardConfig

Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [====      ]  36.0% (used 29500 bytes from 81920 bytes)
Flash: [===       ]  33.3% (used 347872 bytes from 1044464 bytes)
=============================================================================== [SUCCESS] Took 1.87 seconds ===============================================================================

Environment    Status    Duration
-------------  --------  ------------
ota            SUCCESS   00:00:01.866

# platformio.ini
[env:ota]
platform = espressif8266
board = d1_mini
framework = arduino
monitor_speed = 115200
lib_deps =
  mbed-quevedo/ESP8266@0.0.0+sha.77388e8f0697
  knolleary/PubSubClient@^2.8.0
  adafruit/Adafruit Unified Sensor@^1.1.4
  adafruit/Adafruit BME280 Library@^2.1.2
board_build.ldscript = eagle.flash.4m.ld
board_upload.maximum_size = 400

boxofrox · December 21, 2020, 5:52am

Looks like the pieces I was missing are in another repo. This [1] appears to be one of the lines overriding the upload.maximum_size property, and seems to be computing the wrong value in my case.

github.com

platformio/platform-espressif8266/blob/develop/builder/main.py#L263


      
          target_buildprog = env.Alias("buildprog", target_firm, target_firm)
          
          
# update max upload size based on CSV file
          if env.get("PIOMAINPROG"):
              env.AddPreAction(
                  "checkprogsize",
                  env.VerboseAction(
                      lambda source, target, env: _update_max_upload_size(env),
                      "Retrieving maximum program size $SOURCE"))
          # remove after PIO Core 3.6 release
          elif set(["checkprogsize", "upload"]) & set(COMMAND_LINE_TARGETS):
              _update_max_upload_size(env)
          
          
#
          # Target: Print binary size
          #
          
          
target_size = env.AddPlatformTarget(
              "size",
              target_elf,
              env.VerboseAction("$SIZEPRINTCMD", "Calculating size $SOURCE"),

maxgerhardt · December 21, 2020, 11:31am

For ESP8266, the target flash size will always be read from the linker script, since that is what will be effective in the compilation process

github.com

platformio/platform-espressif8266/blob/468a6250fad7f77e1fe6a169b927de29358cfa8d/builder/main.py#L123-L126


      
          def _update_max_upload_size(env):
              ldsizes = _parse_ld_sizes(env.GetActualLDScript())
              if ldsizes and "app_size" in ldsizes:
                  env.BoardConfig().update("upload.maximum_size", ldsizes['app_size'])

So if you want to set a 400 bytes maximum flash size (which… nothing o_O?) then you would have to duplicate the linker script you’re currently using, modify it, put it on your project folder and then reference it with board_build.ldscript = ..

boxofrox · December 21, 2020, 8:33pm

I can’t say it’s very effective in my case of utilizing the “Advanced Memory Usage” to determine whether my program is too large for a serial/OTA upload.

For example, the default ldscript (eagle.flash.4m2m.ld) always reported 1044464 bytes of max flash for program regardless of whether I intended to upload via serial (can flash up to 2MB program), or OTA (can flash up to 1MB program).

When I switched ldscript to eagle.flash.4m.ld, I’d like to compare my program size against ~4MB of flash for serial upload, and ~2MB for OTA upload.

I assumed I could modify upload.maximum_size to adjust the report to my preference, but such wants do not align with an effective compilation process.

My obviously absurd size was meant to be absurd and obvious to point out where the value appears (or doesn’t appear) in the build output.

Thanks. Would I still duplicate the linker script if I only want to affect the Memory Usage report? I feel this would affect other things.

boxofrox · December 21, 2020, 11:37pm

The more I dig, the deeper down the rabbit hole I go…

Unbeknownst to me, while the ESP has 4MB of flash, it is only capable of running a 1MB program due to

a hardware limitation in the ESP because it is the max size of code mem that can be mapped from the flash. [1]

My efforts to extend the maximum size in the Memory Usage report are for naught, as the current linker scripts for the ESP8266 correctly report a 1MB-4kB size limit, and on a ESP8266 with 4MB of flash, there’s an unsused 1MB section to be found for either serial/OTA updates.

So ends this session of nerd sniping.

[1]: Investigate breaking the max sketch size limit · Issue #6681 · esp8266/Arduino · GitHub