PlatformIO Community

Something like `make -jN` for building?


#1

I noticed that platform.io is building one source file at a time, not utilizing all my cpu cores.ltiple jobs

Is it possible to start multiple build jobs?

or this would become a feature request?


#2

How did you check it? Please upgrade to just released PlatformIO 3.0 with the new build system and support for parallel builds.


#3

ok… sorry, my fault…

didn’t notice that 3.0 just out like half an hour ago…


#4

Sorry for bumping the old thread, but I cannot find any information about parallel builds in the documentation.
Since the search for parallel builds returns this topic only, I think it would be good to add some links to the docs about enabling multiple build jobs (rather than creating new topic fully duplicating this one).


#5

PIO Build System automatically decides how many parallels builds could be run. It depends on a number of CPUs/Cores


#6

I know this is an old thread but its 2019, I am using PIO 3.6.4 on Visual Studio, and compiling ESP-IDF projects is insanely slow and painful as it still compiles every single source file individually, how do I enable parallel building on this version?

I even screen captured the compile time, 1 min 53 seconds


#7

PlatformIO automatically detects a number of processors and enables parallel builds. You can open the system Task manager and see it in action.

To check how many CPUs are available for a build process, please open terminal and type:

python -c "from multiprocessing import cpu_count; print(cpu_count())"

#8

So your saying I just have to live with 2 min build times when build clean? I have 8 cores on my system ;).


#9

What’s the native ESP-IDF performance with make -j8 for the same project configuration?


#10

How do you add the -j command using platformio?


#11

No, I mean, the native ESP-IDF. From their build system, as instructed in https://github.com/espressif/esp-idf/ adn https://docs.espressif.com/projects/esp-idf/en/latest/get-started/index.html#installation-step-by-step


#12

No idea, the native build system is totally different then PlatformIO’s, and my project is very massive and doesn’t have the component files that native IDF uses.


#13

This is so fast 2 minutes for 800Mbytes of source files. I’ve never seen so massive SDK as ESP-IDF. Windows XP distribution has less size :metal:


#14

Have you run up a task manager to check what the processor core usage is? Unless you fiddle with the platformio.ini file, does platformio not only rebuild modified files? Meaning that subsequent builds are much, much faster?


#15

I can confirm looking at task manager it is compiling in parallel, but the annoying part is that any new file, removing of a file, or even changing platformio.ini will cause a clean compile, and re-compile the entire IDF when really I see no reason that the IDF should have to be compiled unless you downloaded an update or modified something in the core. Most other make tools for ESP do this, you only get a single compile of the IDF libraries which is then linked against.


#16

Changing the platformio.ini always triggers a re-compile, so it’s best to avoid doing that where possible. As to why adding/removing other files would trigger a re-compile I don’t know the reasoning for, but it does happen. If you simply remove the #include of another file it won’t do a complete re-compile, so it seems adding/removing files is considered a significant enough change to trigger a rebuild… whether it is needed or not. It would be nice to see if that could be made a bit smarter.


#17

Hi,
I have the same problem and observation. I’m rather sure PlatformIO use one core in my compilation. I modified ini file and added this:

build_flags =
j=8

This option, as I expected, dramaticaly increase speed of compilation. But proces crashes during library linking. So two steps forward and one back.


#18

See Something like `make -jN` for building?

How many CPUs does Python print?

python -c "from multiprocessing import cpu_count; print(cpu_count())"

#19

Python said that 8……………


#20

hmmm… But it was on different computer. I’ll check once again it.