How to use own python libraries in extra_script.py

dirkcx · November 12, 2020, 6:27pm

I want to import an own python library in my extra_script.py but when building the code, an error is thrown
ImportError: cannot import name 'postpro'

I installed this python library successfully upfront using
pip install postpro and also with pip install postpro -U

I works well when using standard python libraries:

Import("env", "projenv")
import shutil
...

I am runnning macOS Catalina. thanks in advance for any hint

maxgerhardt · November 12, 2020, 9:13pm

If you do a pip install in the MacOS terminal, that will add it to the Python 2.7 libraries, as that is what MacOS is using. When installing PlatformIO via VSCode, it brings its own self-contained Python3 environment which is separate from the system environment. The libs installed by your system’s pip will not be seen there.

The recommended way to install additional Python dependencies is to attempt an import, and if that fails, call into $PYTHONEXE -m pip install <package> to install it. This is showcased e.g. in

github.com

platformio/platform-espressif32/blob/07919d0c5a731b06f01513766be0e72da38db313/builder/frameworks/espidf.py#L74-L97


      
          try:
              import future
              import pyparsing
              import cryptography
          except ImportError:
              env.Execute(
                  env.VerboseAction(
                      '$PYTHONEXE -m pip install "cryptography>=2.1.4" "future>=0.15.2" "pyparsing>=2.0.3,<2.4.0" ',
                      "Installing ESP-IDF's Python dependencies",
                  )
              )
          
          
# a special "esp-windows-curses" python package is required on Windows for Menuconfig
          if "windows" in get_systype():
              import pkg_resources
          
          
    if "esp-windows-curses" not in {pkg.key for pkg in pkg_resources.working_set}:
                  env.Execute(
                      env.VerboseAction(
                          '$PYTHONEXE -m pip install "file://%s/tools/kconfig_new/esp-windows-curses" windows-curses'

This file has been truncated. show original

and you should add the same style of code in your extra script.

For a more direct way, you can also go into your home directory (/Users/<user> on Mac, idk?) and from there into the .platformio folder. Search this folder for the pip3 executable. Once you find its path, use this pip3 executable to execute the install command, and it will be installed. This is then however not well reproducable accross computers because every user of your extra script will have to do this too; Prefer the first method instead.

dirkcx · November 13, 2020, 8:16pm

Thanks @maxgerhardt for the fast reply, but unfortunately both ways didn’t work for me.
I was able to install the library, but there is another issue, I cannot fix:

% /Users/dirk/.platformio/python3/bin/python3.8 -m pip install postpro
Requirement already satisfied: postpro in /Users/dirk/.platformio/python3/lib/python3.8/site-packages (0.6.5)
ERROR: Error while checking for conflicts. Please file an issue on pip's issue tracker: https://github.com/pypa/pip/issues/new
Traceback (most recent call last):File "/Users/dirk/.platformio/python3/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3021, in _dep_map
    return self.__dep_map
  File "/Users/dirk/.platformio/python3/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2815, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map
...

Maybe the library has errors itself, but it was build for python3.

maxgerhardt · November 13, 2020, 8:20pm

Ehm well that’s new one for me. pip internall error. Does it only do that when you install that library? Does it go away with a .. -m pip uninstall postpro? The way you install it seems right to me if that python path is your $PYTHONEXE.

As a very experimental try, maybe updating pip helps? /Users/dirk/.platformio/python3/bin/python3.8 -m pip install --upgrade pip

dirkcx · November 13, 2020, 8:33pm

update, @maxgerhardt: it was my fault, I called an internal function with an already used variable name.
so your answer was absolutely correct and very helpful. Thanks