Using global Python packages for PlatformIO command line build

nate · January 12, 2024, 3:00pm

I’m trying to get PlatformIO core cli to run on AWS lambda. My ultimate goal is to be able to kick-off an AWS Lambda function, build a firmware using ESPhome cli (which calls PlatformIO under the hood) and then save the firmware bin to S3.

AWS Lambda has some unique aspects in that the filesystem is read-only except for the /tmp directory which is ephemeral storage. I have built a layer for Lambda which includes all of the python dependencies needed, which is linked to /opt/python which is automatically in the search path. I’ve also set PYTHONPATH=/opt/python in the environment. I’m using the Python 3.12 runtime on x86_64 architecture.

My question is, how do I make PlatformIO use a global python installation and package path, rather than using the default install? Two issues I’m facing:

PlatformIO cannot find Pyserial. The build actually succeeds but at the end of the build process, it fails looking for Pyserial in /var/lang/bin/python3.12. It’s not there, because it’s not part of AWS’s Python runtime. Pyserial is installed in /opt/python but pio doesn’t seem to want to look there or respect PYTHONPATH env.

2024-01-12T13:34:59.658Z	Linking .pioenvs/simple-test/bootloader.elf
2024-01-12T13:34:59.737Z	Building .pioenvs/simple-test/bootloader.bin
2024-01-12T13:34:59.814Z	Pyserial is not installed for /var/lang/bin/python3.12. Check the README for installation instructions.
2024-01-12T13:34:59.815Z	Traceback (most recent call last):
2024-01-12T13:34:59.815Z	File "/tmp/.platformio/packages/tool-esptoolpy/esptool.py", line 31, in <module>
2024-01-12T13:34:59.815Z	import esptool
2024-01-12T13:34:59.815Z	File "/tmp/.platformio/packages/tool-esptoolpy/esptool/__init__.py", line 42, in <module>
2024-01-12T13:34:59.815Z	from esptool.cmds import (
2024-01-12T13:34:59.815Z	File "/tmp/.platformio/packages/tool-esptoolpy/esptool/cmds.py", line 14, in <module>
2024-01-12T13:34:59.815Z	from .bin_image import ELFFile, ImageSegment, LoadFirmwareImage
2024-01-12T13:34:59.815Z	File "/tmp/.platformio/packages/tool-esptoolpy/esptool/bin_image.py", line 14, in <module>
2024-01-12T13:34:59.815Z	from .loader import ESPLoader
2024-01-12T13:34:59.815Z	File "/tmp/.platformio/packages/tool-esptoolpy/esptool/loader.py", line 21, in <module>
2024-01-12T13:34:59.816Z	import serial
2024-01-12T13:34:59.816Z	ModuleNotFoundError: No module named 'serial'
2024-01-12T13:34:59.823Z	*** [.pioenvs/simple-test/bootloader.bin] Error 1
2024-01-12T13:35:00.772Z	========================= [FAILED] Took 149.28 seconds =========================

Second issue (less critical) is that pio creates a Python venv for the build automatically. This works, but it’s inefficient because it will download the python dependencies every time, increasing the Lambda execution time and cost. I want to include all the python dependencies in my layer, and force platformIO to not create a venv and instead use the installed python packages in /opt/python.

2024-01-12T13:33:15.578Z	Creating a new virtual environment for IDF Python dependencies
2024-01-12T13:33:19.226Z	WARNING: The directory '/home/sbx_user1051/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
2024-01-12T13:33:19.315Z	Installing ESP-IDF's Python dependencies
2024-01-12T13:33:19.647Z	WARNING: The directory '/home/sbx_user1051/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
2024-01-12T13:33:19.800Z	Collecting urllib3<2
2024-01-12T13:33:19.801Z	Obtaining dependency information for urllib3<2 from https://files.pythonhosted.org/packages/b0/53/aa91e163dcfd1e5b82d8a890ecf13314e3e149c05270cc644581f77f17fd/urllib3-1.26.18-py2.py3-none-any.whl.metadata
2024-01-12T13:33:19.840Z	Downloading urllib3-1.26.18-py2.py3-none-any.whl.metadata (48 kB)
2024-01-12T13:33:19.850Z	━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.9/48.9 kB 5.1 MB/s eta 0:00:00
2024-01-12T13:33:20.233Z	Collecting cryptography<35.0.0,>=2.1.4
2024-01-12T13:33:20.240Z	Downloading cryptography-3.4.8-cp36-abi3-manylinux_2_24_x86_64.whl (3.0 MB)
2024-01-12T13:33:20.275Z	━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.0/3.0 MB 93.5 MB/s eta 0:00:00
2024-01-12T13:33:20.318Z	Collecting future>=0.18.3
2024-01-12T13:33:20.322Z	Downloading future-0.18.3.tar.gz (840 kB)
2024-01-12T13:33:20.329Z	━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 840.9/840.9 kB 235.5 MB/s eta 0:00:00
2024-01-12T13:33:20.414Z	Installing build dependencies: started
2024-01-12T13:33:22.134Z	Installing build dependencies: finished with status 'done'

Is there a way to force PlatformIO to find all Python dependencies in /opt/python?

Thanks in advance!

ivankravets · January 13, 2024, 7:08pm

Sorry, no help here. The system is broken. I’ll be surprised if something works.

Please read this CircleCI — PlatformIO latest documentation

nate · January 13, 2024, 10:38pm

Dang! I was so close. It actually builds the firmware succesfully but I can’t get past the esptool.py error linking pyserial. It seems like this esptool.py step is added by platform-espressif32 and it insists on using its own esptool.py instead of the system installed.

What do you mean by “the system is broken”? Are you referring to AWS lambda or the pio build?

Thanks for the reply. I’m looking at different approaches outside of using lambda.

ivankravets · January 17, 2024, 11:04am

What does this mean?

maxgerhardt · January 17, 2024, 11:12am

If the problem is just that it can’t write to ~/.platformio, you should be able to use the get-platformio.py installer script plus the setting of the PLATFORMIO_CORE_DIR global environment variable before that to redirect it to e.g. /tmp/platformio?

nate · January 22, 2024, 7:50pm

The PYTHONPATH environment variable tells the system where to find Python and defines sys.path, see: sys — System-specific parameters and functions — Python 3.12.2 documentation

This isn’t the issue. I have used the PLATFORMIO_CORE_DIR variable already to redirect where platformio writes to, and that’s working fine. On AWS Lambda, only the /tmp directory is writable, so I have it writing to /tmp/.platformio with no problems.

The firmware actually compiles successfully. It’s not until post-compilation where the platfrom-espressif32 calls esptool.py in the “ElfToBin” builder step, which is defined here: platform-espressif32/builder/main.py at develop · platformio/platform-espressif32 · GitHub
It’s insisting to use a locally installed esptool.py package in /tmp/.platformio/packages/tool-esptoolpy/esptool.py which doesn’t work instead of the system installed esptool.py in /opt/python/bin/esptool.py (which does work).

Is there a way to tell platformio to NOT create its own python virtual env and download packages? I think this will work if I can install all python packages ahead of time and force it to use system packages.

What does the platformio.py installer script do differently than just pip3 install platformio?

ivankravets · January 22, 2024, 9:04pm

No. See Understanding the Python Path Environment Variable in Python [Updated]

In your case, the pip3 install platformio is the right solution.

nate · January 22, 2024, 9:44pm

Ok so I may not have fully understood exactly what it does. Now I remember I added it to solve a ModuleNotFound error. If I remove the PYTHONPATH env variable, compilation does not begin and it fails early with this error:

ModuleNotFoundError: No module named 'click':
File "/opt/python/platformio/builder/main.py", line 20:
import click

(click is of course installed)