Always full rebuild allegedly after V4.0.0 - is my platformio.ini file deficient?

rljonesau · August 27, 2019, 8:57am

Hi There,

Platform IO used to always work as expected, only building the files that were altered or dependent upon an edited file.
Of late it is always re-building all my source files despite only changing some lines of source in one file.

The only real change I can think of is the new V4.0.0 core (since updated to V4.0.2).

Is my platform IO file deficient in some area, causing the full rebuild?


; PlatformIO Project Configuration File
;
; Build options: build flags, source filter
; Upload options: custom upload port, speed and extra flags
; Library options: dependencies, extra library storages
; Advanced options: extra scripting
;
; Please visit documentation for the other options and examples
; https://docs.platformio.org/page/projectconf.html
[env:esp32dev]

platform = espressif32

lib_extra_dirs = ~/Documents/Arduino/libraries

board = esp32dev

framework = arduino

board_build.partitions = min_spiffs.csv

upload_speed = 921600

upload_port = COM16

upload_protocol = esptool

monitor_speed = 115200

extra_scripts = post:add_CRC.py

; replace shitty Arduino millis with a linear time version

build_flags =

-Wl,–wrap,millis

pfeerick · August 27, 2019, 12:13pm

I don’t see anything there that would trigger a rebuild. And I’m not seeing that behaviour on 4.0 or later (also running 4.0.2 atm)… although I’m mainly doing espressif8266 stuff atm, not espressif32.

Just to be clear, you aren’t making changes to platformio.ini when this happens? As that is also a trigger for a full rebuild. And what OS/IDE is this with?

ivankravets · August 27, 2019, 5:42pm

What do you do here? if you generate random or different build flags, PlatformIO will rebuild these files because build environment changes.

rljonesau · August 27, 2019, 8:11pm

Hi Ivan,

I’m adding a CRC across the generated .bin file so when it gets uploaded by others there is an extra level of confirmation the file is really meant for the device.

No I am no changing platformIO.ini, its file date is always older than the source files.

I’m using Windows10, VSCode

Thanks, Ray

rljonesau · August 28, 2019, 10:56am

FWIW, I have also pulled my project to another PC, just in case it was something to do with my PC.
Still exhibits the same rebuild of all the project files - noting the IDF only gets done once.
Yes I did try the build twice

The timestamp on platformIO.ini does not change.
I also tried commenting out my post: - made no difference.

I have read the other instances of full rebuild in the issues and cannot understand why my problem is persisting.

ivankravets · August 28, 2019, 3:02pm

Good! See how we calculate a project checksum platformio-core/helpers.py at develop · platformio/platformio-core · GitHub

Could you help us to debug this issue? Please open PlatformIO IDE Terminal and type:

python -c "from platformio.project import helpers; print(helpers)"

So, you will see a path where is located helpers.py file (not PYC). So, add before this line platformio-core/helpers.py at develop · platformio/platformio-core · GitHub

        if WINDOWS:  # case insensitive OS
            chunks_to_str = chunks_to_str.lower()
        checksum.update(hashlib_encode_data(chunks_to_str))

    print("Checksum = %s" % checksum.hexdigest())
    return checksum.hexdigest()

Now, try to build project a few times. Does checksum change?

rljonesau · August 28, 2019, 8:19pm

Hello Ivan,

It appears the checksum stays the same, always.
One thing I have just noticed though is the Build: (platformio.exe run) seems to behave OK.
Whereas Upload: (platformio.exe run --target upload) is always rebuilding my files, but not the IDF.

THEN, the next time I did Build:, it rebuilt all my source files, then was happy again after that first build.
Hopefully that is useful to you

I have pasted the checksums of a few builds, and commented against each line:
Build: Checksum = 69f678a8d4a118cea67084fca5087c4d8af52468 - rebuilt everything, including IDF files Build: Checksum = 69f678a8d4a118cea67084fca5087c4d8af52468 - did not rebuild after dependency check! Build: Checksum = 69f678a8d4a118cea67084fca5087c4d8af52468 - same, no rebuild . Changed some source . Build: Checksum = 69f678a8d4a118cea67084fca5087c4d8af52468 - rebuilt changed file! Upload: Checksum = 69f678a8d4a118cea67084fca5087c4d8af52468 - rebuilt ALL MY source files - not IDF Upload: Checksum = 69f678a8d4a118cea67084fca5087c4d8af52468 - rebuilt ALL MY source files - not IDF - AGAIN! Upload: Checksum = 69f678a8d4a118cea67084fca5087c4d8af52468 - rebuilt ALL MY source files - not IDF - AGAIN! . Build: Checksum = 69f678a8d4a118cea67084fca5087c4d8af52468 - rebuilt ALL MY source files - not IDF - AGAIN! Build: Checksum = 69f678a8d4a118cea67084fca5087c4d8af52468 - did not rebuild after dependency check!

ivankravets · August 29, 2019, 8:25am

Could you comment extra_scripts = post:add_CRC.py in platformio.ini. Does it work now?

rljonesau · August 29, 2019, 11:32am

It is already commented out

ivankravets · August 29, 2019, 11:46am

Could you try this simple example platform-espressif32/examples/arduino-blink at develop · platformio/platform-espressif32 · GitHub ? Do you have issue with it?

rljonesau · August 29, 2019, 12:21pm

No, that appears to do the right thing, only rebuilds if I edit blink.cpp.

My project does have have quite a few source files.

ivankravets · August 30, 2019, 8:22am

Could you comment other additional options in platformio.ini just to check which triggers project rebuilding? For example, board_build.partitions…

rljonesau · August 30, 2019, 9:10am

Sadly no.

I have reduced it down to the following, any more and I cannot use upload!
Still rebuilds all the files when I use Upload.
As I mentioned before, works OK when I just Build.

[env:esp32dev]
platform = espressif32
;lib_extra_dirs = ~/Documents/Arduino/libraries
board = esp32dev
framework = arduino
;board_build.partitions = min_spiffs.csv
upload_speed = 921600
;upload_port = 192.168.20.40
;upload_protocol = espota
upload_port = COM16
upload_protocol = esptool
;upload_flags =
; --port=3232
;monitor_speed = 115200
;extra_scripts = post:add_CRC.py
; replace shitty Arduino millis with a linear time version
;build_flags =
; -Wl,–wrap,millis

valeros · August 30, 2019, 9:27am

Could you upload your project somewhere?

rljonesau · August 30, 2019, 9:44am

Hello Valeros,

it is the same project you looked at in regards to “the linker line is too long” from me a couple of months back.
Have since implemented you suggestions and streamlined the directory structures.

Ray

valeros · August 30, 2019, 9:50am

OK, I’ve found your project on Gitlab and didn’t notice any problems: the project is compiled exactly once then I can build, upload without recompiling. BTW, what is your OS?

rljonesau · August 30, 2019, 10:07am

Windows 10, VS Code.

rljonesau · August 30, 2019, 10:26am

Hmmmm, prompted by your success, I pulled the repo to a new directory and it now builds correctly there!

There are other non source files and directories floating about that are not in in the repo.
Is it possible platformio is looking at things other than the src, lib and data directories?

I’ll try pruning some stuff to elsewhere for now…

rljonesau · August 30, 2019, 12:23pm

I removed the solution as it is still not clear why the full rebuild takes place.
What is unsettling platform IO it is surely more concerning, and if I can determine that I will certainly then post a true solution.

rljonesau · August 30, 2019, 10:34pm

Well that didn’t last long, now even the new clone is exhibiting the same behaviour.

One thing I did do was add a new library to the /lib directory:
async-mqtt-client & AsyncTCP

Back doing full builds again