PlatformIO Community

How do you install python modules in the virtual environment for advanced scripting?

I currently use the extra_scripts section of platformio.ini to run post build python scripts. It’s been working fine, but now I have a new requirement. I need to add a step at the end of the build to copy my build artifacts to Azure File Storage. I can do this in Python by installing the azure-storage-fileshare python module. However, trying to use pip to install the modules in the PlatformIo virtual environment doesn’t seem to work.

What I have been trying to do is install the modules in a pre: build step. Then use the modules in a post build step.
platformio.ini

...
extra_scripts = 
    pre:../preInstallDependencies.py  ;Install python modules if not already installed.
    post:../postBuild_All.py          ;After the build completes, use AddPostAction to upload artifacts to Azure File Storage.

preInstallDependencies.py would look something like this…

import sys
import subprocess
import pkg_resources
import site

python = sys.executable
#c:\users\admin\.platformio\penv\scripts\python.exe

package_path = site.getsitepackages()
#['c:\\users\\admin\\.platformio\\penv'
#,'c:\\users\\admin\\.platformio\\penv\\lib\\site-packages']

installed = {pkg.key for pkg in pkg_resources.working_set}
#{ 'chardet', 'tabulate', 'pyelftools', 'marshmallow', 'click'
#, 'requests', 'certifi', 'pyserial', 'colorama', 'pip'
#, 'setuptools', 'platformio', 'semantic-version', 'wheel'
#, 'bottle', 'urllib3', 'idna'}

if 'azure-storage-fileshare' in installed:
    pass
else:
    subprocess.check_call([python, '-m', 'pip', '--version'])
    #pip 20.2.3 from c:\users\admin\.platformio\penv\lib\site-packages\pip (python 3.7)

    subprocess.check_call([python, '-m', 'pip', 'install', 'azure-storage-fileshare'])
    #ERROR: Could not find a version that satisfies the requirement azure-storage-fileshare (from versions: none)
    #ERROR: No matching distribution found for azure-storage-fileshare

    subprocess.check_call([python, '-m', 'pip', 'install', '--user', 'azure-storage-fileshare'])
    #ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

And finally, postBuild_All.py would look something like this…

from azure.storage.fileshare import ShareFileClient, ShareClient
from azure.core.exceptions import ResourceExistsError

#bla bla bla

However when I try to pip install the ‘azure-storage-fileshare’ module I get pip installation errors.
ERROR: Could not find a version that satisfies the requirement azure-storage-fileshare (from versions: none)
ERROR: No matching distribution found for azure-storage-fileshare

Any ideas how I can make this work?

Seems like your error is specific to the azure package? https://github.com/Azure/azure-cli/issues/1308

Just use

Import("env")

env.Execute("$PYTHONEXE -m pip --version")

@ivankravets answered the question as written in the topic and his approach is cleaner than my approach of using subprocess and sys.executable.

However, @maxgerhardt tipped me off to the problem specific to my issue. Adding the pip -vvv switch gave me more detail and I able to was track down the problem.

Solution…
The azure installation package name should have been “azure-storage-file-share” NOT “azure-storage-fileshare”.

env.Execute("$PYTHONEXE -m pip install -vvv azure-storage-file-share")

Oddly, the Azure module name is non-hyphenated (i.e. ‘azure.storage.fileshare’ NOT ‘azure.storage.file-share’)…

from azure.storage.fileshare import ShareFileClient, ShareClient

My final solution looks like this…
platformio.ini

extra_scripts = 
    pre:../preInstallDependencies.py  ;Install python modules if not already installed.
    post:../postBuild_All.py          ;After the build completes, use AddPostAction to upload artifacts to Azure File Storage.

preInstallDependencies.py

import pkg_resources

Import("env")
installed = {pkg.key for pkg in pkg_resources.working_set}

if 'azure-storage-file-share' in installed:
    pass
else:
    env.Execute("$PYTHONEXE -m pip install azure-storage-file-share")

postBuild_All.py

from azure.storage.fileshare import ShareFileClient, ShareClient
from azure.core.exceptions import ResourceExistsError
#... bla bla bla

This is a bad solution. Please use

try
  import bla lba
except ImportError:
  # do install