Is it possible to say platformio to use all my CPUs during compiling?

sblantipodi · October 18, 2023, 8:07pm

As title,
I have a 7950X3D CPU,
Is it possible to say platformio to use all my CPUs during compiling?

During compiling it uses 16 threads, I would like to try with 32 threads…
is it possible to change this ?

thanks

sblantipodi · October 18, 2023, 8:18pm

it seems that same code uses 100% of my CPU threads while compiling for ESP8266
but it uses only 16 threads when compiling on ESP32… weird…

maxgerhardt · October 19, 2023, 5:36pm

PlatformIO invokes SCons with --jobs (number of jobs to start) with the CPU count by default.

github.com

platformio/platformio-core/blob/5f8c15b96a0fe4d2f9dd490c34834f0fa17295fa/platformio/run/cli.py#L35-L38


      
          try:
              DEFAULT_JOB_NUMS = cpu_count()
          except NotImplementedError:
              DEFAULT_JOB_NUMS = 1

github.com

platformio/platformio-core/blob/5f8c15b96a0fe4d2f9dd490c34834f0fa17295fa/platformio/run/cli.py#L57-L66


      
          @click.option(
              "-j",
              "--jobs",
              type=int,
              default=DEFAULT_JOB_NUMS,
              help=(
                  "Allow N jobs at once. "
                  "Default is a number of CPUs in a system (N=%d)" % DEFAULT_JOB_NUMS
              ),
          )

github.com

platformio/platformio-core/blob/5f8c15b96a0fe4d2f9dd490c34834f0fa17295fa/platformio/platform/_run.py#L72-L81


      
          args = [
              proc.get_pythonexe_path(),
              os.path.join(scons_dir, "scons.py"),
              "-Q",
              "--warn=no-no-parallel-support",
              "--jobs",
              str(jobs),
              "--sconstruct",
              os.path.join(fs.get_source_dir(), "builder", "main.py"),
          ]

As you can see, the pio command exposes this option via the CLI too, so if you do

pio run -j32

it will pass that number on to SCons which is then supposed to start 32 jobs in parallel.

trullock · January 13, 2025, 9:01pm

is it pio run -j32 or pio run -j 32?

I cant seem to make either have any effect (as measured by build time - i cant see it saying how many threads its using even in verbose mode, have I missed it?

pio run -j1 takes the same time as pio run -j16

How can I prove whats going on?

Thanks