ESP weird build and upload issues

Hello,
I am using the development version of platformio core (version 3.6.0a1) with deviot plugin on sublime for development.

After the last upgrade to 3.6 I started seeing some weird issues.
Our team is working on a ESP32 project. When I was on 3.5.3 the compilation and upload worked just fine.
However after I upgraded Pio core, I ran into a couple of issues.

  1. Differences in memory usage during build and during upload.

Here is the terminal output when I run a build command.

Checking size .pioenvs/prod/firmware.elf
Memory Usage -> http://bit.ly/pio-memory-usage
DATA:    [==========]  171.7% (used 506384 bytes from 294912 bytes)
PROGRAM: [==========]  99.5% (used 1798617 bytes from 1807328 bytes)

And this is the error I get when I run the upload command.

Checking size .pioenvs/prod/firmware.elf
Memory Usage -> http://bit.ly/pio-memory-usage
DATA:    [==========]  171.7% (used 506392 bytes from 294912 bytes)
Error: The program size (1798629 bytes) is greater than maximum allowed (1507328 bytes)
PROGRAM: [==========]  119.3% (used 1798629 bytes from 1507328 bytes)

Nothing changed between the build and upload command. and as a result I am unable to upload the firmware.

  1. We are using a custom partition map for the esp32. We have modified the default.csv under .platformio/packages/framework-arduinoespressif32/tools/partition to suit our needs and also modified the sizes in the corresponding board.json file and the board.txt file in the platforms and packages folders respectively.

We are using a app partition size of more than the default and even the 1798629 bytes as reported should fit into the partition. Where does the memory check script get its limits from??

On a related observation, a colleague of mine who is also working on the same project is not having any issue with either compiling or uploading. He is still running the 3.5.3 version of Pio core.

PS: I have tried rolling back the espressif32 version to an earlier version before 1.0.2. I have tried 1.0.0 an 0.12.0 but it does not seem to make any difference.

Ok…
completely uninstalling the platform from Pio home and not specifying a version in the platformio.ini file seems to have fixed the issue.

pio seems to have installed 0.12.0 version which compiles and uploads just fine.

I must have done something wrong earlier when I manually tried to set the version to 0.12.0 in the ini file earlier.

Please don’t modify internal files manually. You will need the latest PIO Core, ESP32 1.0 dev platform and Espressif 32 — PlatformIO latest documentation

1 Like

Ok @ivankravets Will keep that in mind…
I ll reset the packages to the stock and make the changes as you suggested. :slight_smile:

Thank you!!

1 Like

@ivankravets

I tried what you suggested with the board_build.partitions = new_extended.csv
in the platformio.ini file. And the code doesn’t compile at all when I try to specify the partition map.

Here is my platformio.ini file

[DEFAULT]
platform = espressif32
board = hornbill32dev
framework = arduino
upload_protocol = esptool
lib_deps =
# FastLED@3.1.8
	https://github.com/samguyer/FastLED#a631e24
	ArduinoJson@5.11.2
	Adafruit BME680 Library@1.0.5
# Adafruit NeoPixel@1.1.6
	Adafruit Unified Sensor@1.0.2
	Adafruit TSL2561@1.0.2
# HPMA115S0 Arduino Library@1.0.0
# https://github.com/felixgalindo/HPMA115S0.git#dbe8e65
	NTPClient@3.1.0
# https://github.com/markbeee/SHT21.git#2c28aa9

extra_scripts = pre:scripts/load_new_libs.py
build_flags = -fexceptions
	-DFW_VERSION=2
# Threshold for captouch trigger
	-DTOUCH_THRESHOLD=28
# For initializing Wire library (I2C Bus)
	-DSDA_PIN=19
	-DSCL_PIN=21

[env:prod]
build_flags =
	${DEFAULT.build_flags}
	'-DPAYLOAD_HOST="xxxxxxxxxxx"'

[env:dev]
build_flags =
	${DEFAULT.build_flags}
	'-DPAYLOAD_HOST="xxxxxxxxxxxx"'

and the compilation fails on the FASTLed library compilation process…
I tried placing the directive in the [DEFAULT], [prod] and [dev] but no change.
commenting out the directive compiled just fine.

What am I doing wrong?

PS: It is PIO 3.6.0a1, esp32 0.12.0 platform version. I am assuming that I need to be on the 1.0+ dev versions??

However upgrading the platform to 1.0+ is breaking the code compilation. As mentioned on another thread the basic binary size is quite bigger. almost 30%.

Thank you !! :slight_smile:

Custom partitions were added in 1.0.0 Releases · platformio/platform-espressif32 · GitHub

Could you remove sensitive parts of code from your project and try to build with Arduino IDE + the latest ESP32 Core? Do you have the same issue with size? Maybe, is something wrong with our builder?

Thanks!

I ll try compiling with the Arduino IDE and post the results here @ivankravets.

Thank you!! :slight_smile:

Apologies for the delay in getting back @ivankravets
I updated the framework to 1.0.2
Here is the sample sketch I tried compiling with this basic sketch

void setup() {
  // put your setup code here, to run once:
  Serial.begin(115200);

}

void loop() {
  // put your main code here, to run repeatedly:

  Serial.println("Hello World");
  delay(5000);
} 

And here is the compiler output from platformio

And this is the output from Arduino IDE.

There is a ~3KB increase in the PIO binary size, which goes unto almost 300-400 KB for more complex applications.
And this increase in the binary output seems to be increasing with code complexity.
I haven’t had an opportunity to try with a moderately complex firmware, but seeing this sketch and the actual firmware I tried yesterday I am almost certain that PIO is generating a bigger binary.

And this is for a different sketch. its a BME680 basic test sketch.

#include "bsec.h"

// Helper functions declarations
void checkIaqSensorStatus(void);
void errLeds(void);

// Create an object of the class Bsec
Bsec iaqSensor;

String output;

// Entry point for the example
void setup(void)
{
  Serial.begin(115200);

  iaqSensor.begin(BME680_I2C_ADDR_SECONDARY, Wire);
  output = "\nBSEC library version " + String(iaqSensor.version.major) + "." + String(iaqSensor.version.minor) + "." + String(iaqSensor.version.major_bugfix) + "." + String(iaqSensor.version.minor_bugfix);
  Serial.println(output);
  checkIaqSensorStatus();

  bsec_virtual_sensor_t sensorList[7] = {
    BSEC_OUTPUT_RAW_TEMPERATURE,
    BSEC_OUTPUT_RAW_PRESSURE,
    BSEC_OUTPUT_RAW_HUMIDITY,
    BSEC_OUTPUT_RAW_GAS,
    BSEC_OUTPUT_IAQ_ESTIMATE,
    BSEC_OUTPUT_SENSOR_HEAT_COMPENSATED_TEMPERATURE,
    BSEC_OUTPUT_SENSOR_HEAT_COMPENSATED_HUMIDITY,
  };

  iaqSensor.updateSubscription(sensorList, 7, BSEC_SAMPLE_RATE_LP);
  checkIaqSensorStatus();

  // Print the header
  output = "Timestamp [ms], raw temperature [°C], pressure [hPa], raw relative humidity [%], gas [Ohm], IAQ, IAQ accuracy, temperature [°C], relative humidity [%]";
  Serial.println(output);
}

// Function that is looped forever
void loop(void)
{
  if (iaqSensor.run()) { // If new data is available
    Serial.println();
    Serial.println("rTemperature: " + String(iaqSensor.rawTemperature));
    Serial.println("Pressure: " + String(iaqSensor.pressure));
    Serial.println("rHumidity: " + String(iaqSensor.rawHumidity));
    Serial.println("Gas Resistance: " + String(iaqSensor.gasResistance));
    Serial.println("IAQ: " + String(iaqSensor.iaqEstimate));
    Serial.println("IAQ Accuracy: " + String(iaqSensor.iaqAccuracy));
    Serial.println("cTemperature: " + String(iaqSensor.temperature));
    Serial.println("cHumidity: " + String(iaqSensor.humidity));                
//    output = String(millis());
//    output += ", " + String(iaqSensor.rawTemperature);
//    output += ", " + String(iaqSensor.pressure);
//    output += ", " + String(iaqSensor.rawHumidity);
//    output += ", " + String(iaqSensor.gasResistance);
//    output += ", " + String(iaqSensor.iaqEstimate);
//    output += ", " + String(iaqSensor.iaqAccuracy);
//    output += ", " + String(iaqSensor.temperature);
//    output += ", " + String(iaqSensor.humidity);
//    Serial.println(output);
  } else {
    checkIaqSensorStatus();
  }
}

// Helper function definitions
void checkIaqSensorStatus(void)
{
  if (iaqSensor.status != BSEC_OK) {
    if (iaqSensor.status < BSEC_OK) {
      output = "BSEC error code : " + String(iaqSensor.status);
      Serial.println(output);
      for (;;)
        errLeds(); /* Halt in case of failure */
    } else {
      output = "BSEC warning code : " + String(iaqSensor.status);
      Serial.println(output);
    }
  }

  if (iaqSensor.bme680Status != BME680_OK) {
    if (iaqSensor.bme680Status < BME680_OK) {
      output = "BME680 error code : " + String(iaqSensor.bme680Status);
      Serial.println(output);
      for (;;)
        errLeds(); /* Halt in case of failure */
    } else {
      output = "BME680 warning code : " + String(iaqSensor.bme680Status);
      Serial.println(output);
    }
  }
}

void errLeds(void)
{
  pinMode(LED_BUILTIN, OUTPUT);
  digitalWrite(LED_BUILTIN, HIGH);
  delay(100);
  digitalWrite(LED_BUILTIN, LOW);
  delay(100);
}

Platformio output

Arduino IDE output

A 11KB difference.

Could you try to disable library archiving Redirecting... ? Do you see any difference?

The "lib_archive = false " made no difference in the first test sketch.
The binary size remains the same.

The second sketch refused to compile in the second case since I was explicitly linking a library (.a file) with the binary.

Sorry for the late answer. Did you resolve it?

@ivankravets Yes it was resolved automagically a while back . :slight_smile:
My apologies, I forgot to update the issue here…

I dont get the upload issues… we are still using a custom partition map and the code bloat is still apparent. but it works.