Using global Python packages for PlatformIO command line build

I’m trying to get PlatformIO core cli to run on AWS lambda. My ultimate goal is to be able to kick-off an AWS Lambda function, build a firmware using ESPhome cli (which calls PlatformIO under the hood) and then save the firmware bin to S3.

AWS Lambda has some unique aspects in that the filesystem is read-only except for the /tmp directory which is ephemeral storage. I have built a layer for Lambda which includes all of the python dependencies needed, which is linked to /opt/python which is automatically in the search path. I’ve also set PYTHONPATH=/opt/python in the environment. I’m using the Python 3.12 runtime on x86_64 architecture.

My question is, how do I make PlatformIO use a global python installation and package path, rather than using the default install? Two issues I’m facing:

  1. PlatformIO cannot find Pyserial. The build actually succeeds but at the end of the build process, it fails looking for Pyserial in /var/lang/bin/python3.12. It’s not there, because it’s not part of AWS’s Python runtime. Pyserial is installed in /opt/python but pio doesn’t seem to want to look there or respect PYTHONPATH env.
2024-01-12T13:34:59.658Z	Linking .pioenvs/simple-test/bootloader.elf
2024-01-12T13:34:59.737Z	Building .pioenvs/simple-test/bootloader.bin
2024-01-12T13:34:59.814Z	Pyserial is not installed for /var/lang/bin/python3.12. Check the README for installation instructions.
2024-01-12T13:34:59.815Z	Traceback (most recent call last):
2024-01-12T13:34:59.815Z	File "/tmp/.platformio/packages/tool-esptoolpy/", line 31, in <module>
2024-01-12T13:34:59.815Z	import esptool
2024-01-12T13:34:59.815Z	File "/tmp/.platformio/packages/tool-esptoolpy/esptool/", line 42, in <module>
2024-01-12T13:34:59.815Z	from esptool.cmds import (
2024-01-12T13:34:59.815Z	File "/tmp/.platformio/packages/tool-esptoolpy/esptool/", line 14, in <module>
2024-01-12T13:34:59.815Z	from .bin_image import ELFFile, ImageSegment, LoadFirmwareImage
2024-01-12T13:34:59.815Z	File "/tmp/.platformio/packages/tool-esptoolpy/esptool/", line 14, in <module>
2024-01-12T13:34:59.815Z	from .loader import ESPLoader
2024-01-12T13:34:59.815Z	File "/tmp/.platformio/packages/tool-esptoolpy/esptool/", line 21, in <module>
2024-01-12T13:34:59.816Z	import serial
2024-01-12T13:34:59.816Z	ModuleNotFoundError: No module named 'serial'
2024-01-12T13:34:59.823Z	*** [.pioenvs/simple-test/bootloader.bin] Error 1
2024-01-12T13:35:00.772Z	========================= [FAILED] Took 149.28 seconds =========================
  1. Second issue (less critical) is that pio creates a Python venv for the build automatically. This works, but it’s inefficient because it will download the python dependencies every time, increasing the Lambda execution time and cost. I want to include all the python dependencies in my layer, and force platformIO to not create a venv and instead use the installed python packages in /opt/python.
2024-01-12T13:33:15.578Z	Creating a new virtual environment for IDF Python dependencies
2024-01-12T13:33:19.226Z	WARNING: The directory '/home/sbx_user1051/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
2024-01-12T13:33:19.315Z	Installing ESP-IDF's Python dependencies
2024-01-12T13:33:19.647Z	WARNING: The directory '/home/sbx_user1051/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
2024-01-12T13:33:19.800Z	Collecting urllib3<2
2024-01-12T13:33:19.801Z	Obtaining dependency information for urllib3<2 from
2024-01-12T13:33:19.840Z	Downloading urllib3-1.26.18-py2.py3-none-any.whl.metadata (48 kB)
2024-01-12T13:33:19.850Z	━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.9/48.9 kB 5.1 MB/s eta 0:00:00
2024-01-12T13:33:20.233Z	Collecting cryptography<35.0.0,>=2.1.4
2024-01-12T13:33:20.240Z	Downloading cryptography-3.4.8-cp36-abi3-manylinux_2_24_x86_64.whl (3.0 MB)
2024-01-12T13:33:20.275Z	━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.0/3.0 MB 93.5 MB/s eta 0:00:00
2024-01-12T13:33:20.318Z	Collecting future>=0.18.3
2024-01-12T13:33:20.322Z	Downloading future-0.18.3.tar.gz (840 kB)
2024-01-12T13:33:20.329Z	━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 840.9/840.9 kB 235.5 MB/s eta 0:00:00
2024-01-12T13:33:20.414Z	Installing build dependencies: started
2024-01-12T13:33:22.134Z	Installing build dependencies: finished with status 'done'

Is there a way to force PlatformIO to find all Python dependencies in /opt/python?

Thanks in advance!

Sorry, no help here. The system is broken. I’ll be surprised if something works.

Please read this CircleCI — PlatformIO latest documentation

Dang! I was so close. It actually builds the firmware succesfully but I can’t get past the error linking pyserial. It seems like this step is added by platform-espressif32 and it insists on using its own instead of the system installed.

What do you mean by “the system is broken”? Are you referring to AWS lambda or the pio build?

Thanks for the reply. I’m looking at different approaches outside of using lambda.

What does this mean?

If the problem is just that it can’t write to ~/.platformio, you should be able to use the installer script plus the setting of the PLATFORMIO_CORE_DIR global environment variable before that to redirect it to e.g. /tmp/platformio?

The PYTHONPATH environment variable tells the system where to find Python and defines sys.path, see: sys — System-specific parameters and functions — Python 3.12.2 documentation

This isn’t the issue. I have used the PLATFORMIO_CORE_DIR variable already to redirect where platformio writes to, and that’s working fine. On AWS Lambda, only the /tmp directory is writable, so I have it writing to /tmp/.platformio with no problems.

The firmware actually compiles successfully. It’s not until post-compilation where the platfrom-espressif32 calls in the “ElfToBin” builder step, which is defined here: platform-espressif32/builder/ at develop · platformio/platform-espressif32 · GitHub
It’s insisting to use a locally installed package in /tmp/.platformio/packages/tool-esptoolpy/ which doesn’t work instead of the system installed in /opt/python/bin/ (which does work).

Is there a way to tell platformio to NOT create its own python virtual env and download packages? I think this will work if I can install all python packages ahead of time and force it to use system packages.

What does the installer script do differently than just pip3 install platformio?

No. See Understanding the Python Path Environment Variable in Python [Updated]

In your case, the pip3 install platformio is the right solution.

Ok so I may not have fully understood exactly what it does. Now I remember I added it to solve a ModuleNotFound error. If I remove the PYTHONPATH env variable, compilation does not begin and it fails early with this error:

ModuleNotFoundError: No module named 'click':
File "/opt/python/platformio/builder/", line 20:
import click

(click is of course installed)