Github Actions using a private repository in platformio library manager

Development Platforms
1

I can see a number of questions on VSCode & private libraries, these don’t help…
I have a private github library which works well doing a local build
I just have the https://github.com/{owner}/{libraryname}.git file (should add I am the owner of both the library and the project.
When I come to create a github action to build the project, pio automatically gets all libraries, and fails when trying to get this (private library)

the platformio.ini contains:

platform = espressif8266
board = sonoff_basic
framework = arduino
lib_deps =
    knolleary/PubSubClient@^2.8
    https://github.com/CG/esp8266-utils.git

however the github action fails with:

[57] Library Manager: PubSubClient @ 2.8.0 has been installed!
[58] Library Manager: Installing git+https://github.com/CG/esp8266-utils.git
[59] git version 2.34.1
[60] Cloning into '/home/runner/.platformio/.cache/tmp/pkg-installing-wu7ibbbp'...
[61] fatal: could not read Username for '[https://github.com](https://github.com/)': No such device or address
[62] VCSBaseException: VCS: Could not process command ['git', 'clone', '--recursive', '--depth', '1',

Is there any way of being able to add credentials to plaformio.ini without having to add something like https://username:password@github.com/…

2

This is general question about Github Workflows. Github and Stackoverflow might be better for this – PlatformIO only invokes git and doesn’t have much else to do with it.

python - Cloning private github repository within organisation in actions - Stack Overflow might be a good solution – it checks the repo out in a local path. You can then use file://... in lib_deps to refer to it.

3

Sorry for the thread-necromancing but still, for reference, because we just went through the same:

Generally, you have to handover some kind of access token to platformio with the correct permissions, such that it can fetch the repository via the pre-installed git of a Github Action. By default, an action (including its by-default generated GH_TOKEN environment variable), however, is only scoped to the current repository, not your/your organization’s other repositories.

Otherwise, just doing https://${sysenv.GH_TOKEN}@github.com/<owneruser/orga>/<repo>.git in the corresponding platformio dependency would work!

Due to this limited permission scope, the easiest way for us was to instead create a Github Action for the organization, install it to organization, add its app_id as variable and its private key as secret (see Github’s official docs), and adding it to your github action as descriped via the “generate a token” step and using the “env” parameter for a step:

env:
          GH_TOKEN: ${{ steps.generate-token.outputs.token }}

Using this for your pio run/test... then makes the dream work, e.g.:

from your_github_action.yml:

      - name: Build PlatformIO Project
        run: pio run
        env:
          GH_TOKEN: ${{ steps.generate-token.outputs.token }}

from platformio.ini:

lib_deps=
      https://${sysenv.GH_TOKEN}@github.com/<owneruser/orga>/<repo>.git

Disclaimer: I only tested this on an ubuntu-style Github Action runner, not windows or mac.

What’s also nice about this: local builds on your own machine still work as long as you are correctly authenticated with git and you don’t have a GH_TOKEN environment variable. Then pio just replaces the https://${sysenv.GH_TOKEN}@github.com/... with https://github.com/... and it should fetch the dependency as normal.

4

Would you be able to share an example GitHub Action yml file where you got this to work? I have only been able to get the action to successfully build when I modify platformio.ini file lib_deps to include:

lib_deps =
    https://<owneruser/orga>:${sysenv.GH_TOKEN}@github.com/<owneruser/orga>/<repo>.git

The problem with doing it this way is it does not work for local development. Any help would be appreciated!

5

If you want to use a private github repository locally, you can use “Fine-grained” personal access tokens:

lib_deps = 
  https://<githubuser>:<token>@github.com/<githubuser>/<repository>.git

Creating the token:

  • login to github
  • click on your github user (top right)
  • click on “Settings” (on the right)
  • click on “Developer settings” (on the left)
  • expand “Personal access tokens”
  • click “Fine-grained tokens”
  • click “Generate new token” (top right)
    • name the token
    • choose the expiration for the token
    • click “Only select repositories”
    • select or search the repository you want to use
    • set the permission “Contents” to “Read-only”
    • click on “Generate token”

Now you will get a token that looks like this: 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 which can be used to access the private repository:

Using the token
Example:
Github-User: “sivar2311”
Repository: “sivar2311/priv_lib”

lib_deps = 
  https://sivar2311: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/sivar2311/priv_lib.git
6

I would not include you token in the platformio. it is a security item and should not be readable in your repository.

My solution is to use the action de-vri-es/setup-git-credentials@v2 which references a token created in github

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: de-vri-es/setup-git-credentials@v2
        with:
          # GIT_CREDENTIALS secret should contain https://<username>:<token>@github.com
          # where token is a Personal Access Token generated on https://github.com/settings/tokens
          # It can also contain the account password but this might change in time
          credentials: ${{secrets.GIT_CREDENTIALS}}

      - uses: actions/checkout@v4

      - name: Checkout submodules
        run: |
          git submodule update --init

      - name: Cache pip
        uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
          restore-keys: |
            ${{ runner.os }}-pip-

      - name: Cache PlatformIO
        uses: actions/cache@v4
        with:
          path: ~/.platformio
          key: ${{ runner.os }}-${{ hashFiles('**/lockfiles') }}

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.10"

      - name: Install PlatformIO
        run: |
          python -m pip install --upgrade pip
          pip install --upgrade gitpython
          pip install --upgrade jschon
          pip install --upgrade yamale
          pip install --upgrade platformio

      - name: Run PlatformIO
        run: |
          # install libs before build
          pio pkg install -e ${{ inputs.env }}
          pio run -e ${{ inputs.env }}
7

Maybe I am wrong, but the question was for local development:

8

So a little bit of an update… I got this working in a really clean way that requires no modifications to anything in the platformio.ini file. The benefit to not modifying your platformio.ini file is that it allows the user to set up git authentication on each platform individually.

The tl;dr is to use git credential manager within the GitHub workflow to authenticate and PlatformIO git requests.

Here is the full YAML file:

name: PlatformIO Build

on:
  push:
    branches:
      - main
      - development

permissions:
  contents: write
  issues: write
  pull-requests: write

jobs:
  release:
    name: release
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/cache@v4
        with:
          path: |
            ~/.cache/pip
            ~/.platformio/.cache
            ~/.platformio/packages
          key: ${{ runner.os }}-pio-${{ hashFiles('**/platformio.ini') }}-${{ hashFiles('**/requirements.txt') }}
          restore-keys: |
            ${{ runner.os }}-pio-
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Generate a token
        id: generate-token
        uses: actions/create-github-app-token@v1
        with:
          app-id: ${{ vars.APP_ID }}
          private-key: ${{ secrets.GH_APP_PK }}
          owner: <owneruser/orga>
      - name: Set up git credentials
        run:
          echo "https://<owneruser/orga>:${{ steps.generate-token.outputs.token }}@github.com" > ~/.git-credentials &&
          git config --global credential.username "<owneruser/orga>" &&
          git config --global credential.helper "store"
      - name: Install PlatformIO Core
        run: pip install --upgrade platformio
      - name: Build PlatformIO Project
        run: pio run

There are a few prerequisites before this will run successfully.

  1. Create a GitHub app under your user account or organization depending on where the private repository exists. Follow these instructions from GitHub to create the app and configure the APP_ID and GH_APP_PK secret within the repository of the workflow (credit @csc-sendance ).
  2. Modify the above workflow and change <owneruser/orga> to be the username of the owner or organization the app and private key belong to.

This will use git credential manager to store the credentials within the workflow container inside of the local git config. As a warning, it is not recommended to do this on your local develoment machine as it stores to token in plaintext inside of ~/.git-credentials. This is secure for our use here as the token generated by GitHub is immediately invalidated once the workflow is finished running.

Hope this helps.

1 Like
9

just to react on this:

Maybe I am wrong, but the question was for local development:

We have direct urls to our private repositories (without credentials) in the platformIo and library.json and this works perfectly locally, as long as you have command line access to your github private repositories.

Github workflow never works locally of course, that is meant to be run on a ubuntu server at the github farm. That is why you have to provide credentials through the action I proposed or the method that OP posted

2 Likes