GitLab SaaS Runner can install pkgs from Internet, but not while building

PlatformIO Core
,
1

I’m trying to use a GitLab SaaS runner to build a PlatformIO project within a Ubuntu 20.04 docker container. It works fine locally and on a self-hosted runner.

If the pio build tries to dynamically install a package, it reports an Internet connection error.

Tool Manager: Installing platformio/tool-scons @ ~4.40600.0
InternetConnectionError: You are not connected to the Internet.
PlatformIO needs the Internet connection to download dependent packages or to work with PlatformIO Account.

However, if I add a step to manually install the package before initiating the build, the package is installed OK and the build succeeds.

$ pio pkg install -t tool-scons@4.40600.0
Resolving target dependencies...
Tool Manager: Installing tool-scons @ 4.40600.0
Downloading 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Unpacking 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Tool Manager: tool-scons@4.40600.0 has been installed!

I get the same behaviour with PlatformIO 6.1.4, 6.1.11 & 6.1.13.

It seems that the PlatformIO Internet connection detection during a build is failing when it should not. There are no proxy settings in the docker image or GitLab CI environment and the Internet connection works fine for other uses (such as wget).

Is there a diagnostic check that I could add to help debug the issue?

2

The relevant code for this is:

You could copy-paste the code into its own Python file, add more print statements and execute it on the Gitlab host. Essentially the code is trying to see if any of the given hosts / URLs are connectable on port 80 within a certain timeout. If it returns all errors, it will say you have no internet connection.

Also see https://github.com/platformio/platformio-core/issues/4526.

3

Thanks for the quick response and suggestion Max.

I created a pio_probe_hosts.py script to perform the PlatformIO Internet connection test in isolation. It always succeeds on the first host and reports that no proxy settings are in effect. Nevertheless, pio build fails when it needs to install a package, reporting that there is no Internet connection.

$ pio system info
--------------------------  ------------------------------------
PlatformIO Core             6.1.4
Python                      3.8.10-final.0
System Type                 linux_x86_64
Platform                    Linux-5.4.109+-x86_64-with-glibc2.29
File System Encoding        utf-8
Locale Encoding             UTF-8
PlatformIO Core Directory   /root/.platformio
PlatformIO Core Executable  /usr/local/bin/platformio
Python Executable           /usr/bin/python3
Global Libraries            0
Development Platforms       2
Tools & Toolchains          1
--------------------------  ------------------------------------
$ python3 scripts/test/pio_probe_hosts.py
Socket connection 88.198.170.159:80 succeeded
$ SCRIPT=scripts/build/build_pio.sh
$ $SCRIPT $CI_COMMIT_TAG
PlatformIO Core, version 6.1.4
Building native DMOD app.
Processing native (extends: native; platform: native; lib_ldf_mode: deep+; lib_compat_mode: strict; lib_deps: googletest, dlm, dtl, dlog, dap, dha, dpr, dbl, ws1cmn, ws1util, ws1dip, ws1tlog; build_flags: -std=c++1y, -std=c11, -pthread, -DNATIVE, -DWSONE_DMOD, -DDMOD_APP_EN=1, -I.pio/libdeps/$PIOENV/googletest/googletest/include, -I.pio/libdeps/$PIOENV/googletest/googletest, -Ilib, -Isrc/native/Posix_GCC, -Isrc/native/freertos/include, -Isrc/native/freertos/portable/ThirdParty/GCC/Posix, -Isrc/native, -Isrc/native/mock_stm, -Isrc/common, -Isrc/common/console, -Isrc/bl_common, -Itest/native, -lm -lstdc++, -g --coverage -fprofile-arcs -ftest-coverage -fprofile-abs-path -lgcov, -DTEST_MODE, -DSW_LIVE_RUMBLE_FILTER; check_flags: cppcheck:, -DCPPCHECK=1, --platform=unix32, --std=c11, --inline-suppr, --inconclusive, -j4, -isrc/native/freertos, -isrc/native/Posix_GCC, --addon=scripts/analysis/pio_misra.json, --cppcheck-build-dir=.pio/build/cppcheck/native; build_src_filter: -<*> +<lib> +<native> +<common> +<bl_common>; check_patterns: lib/**/*.h, lib/*/*.c, lib/*/!(test)/*.c, src/native/d**/*.h, src/native/d**/*.c, src/common/**/*.h, src/common/**/*.c, src/bl_common/**/*.h, src/bl_common/**/*.c; test_build_src: True; test_filter: native/*; check_tool: cppcheck)
--------------------------------------------------------------------------------
Tool Manager: Installing platformio/tool-scons @ ~4.40400.0
Error: You are not connected to the Internet.
PlatformIO needs the Internet connection to download dependent packages or to work with PlatformIO Account.

#!/usr/bin/env python3

import socket
import requests

from platformio import util
# from platformio.pipdeps import is_proxy_set

__registry_mirror_hosts__ = [
    "registry.platformio.org",
    "registry.nm1.platformio.org",
]

__check_internet_hosts__ = [
    "88.198.170.159",  # platformio.org
    "185.199.110.153",  # Github.com
    "github.com",
] + __registry_mirror_hosts__


def is_proxy_set(socks=False):
    for var in ("HTTP_PROXY", "HTTPS_PROXY", "ALL_PROXY"):
        value = os.getenv(var, os.getenv(var.lower()))
        if not value or (socks and not value.startswith("socks5://")):
            continue
        return True
    return False


@util.memoized(expire="10s")
def _internet_on():
    timeout = 2
    socket.setdefaulttimeout(timeout)
    for host in __check_internet_hosts__:
        try:
            probe_type = f'No attempt for {host}'
            if is_proxy_set():
                probe_type = f'HTTP get {host} via proxy'
                requests.get("http://%s" %
                             host, allow_redirects=False, timeout=timeout)
                print(probe_type + ' succeeded')
                return True
            # try to resolve `host` for both AF_INET and AF_INET6, and then try to connect
            # to all possible addresses (IPv4 and IPv6) in turn until a connection succeeds:
            probe_type = f'Socket connection {host}:80'
            s = socket.create_connection((host, 80))
            s.close()
            print(probe_type + ' succeeded')
            return True
        except:  # pylint: disable=bare-except
            print(probe_type + ' failed')

    return False


_internet_on()
4

FYI, there is something different about running the docker image in the gitlab.com SaaS runner environment, compared to running the pio_probe_hosts.py script within the docker image on my local Ubuntu 20.04 VM. That required pip install platformio before running the script in order for from platformio.pipdeps import is_proxy_set to succeed.

This did not work on the SaaS runner, nor did python3 -m pip install platformio, so I had to drop the import attempt and use a local copy of is_proxy_set().

I wonder if this tells us anything about the Python environment on the gitlab.com SaaS runner that might lead to different behaviour than expected?

5

Could you dump $ env in your runner?

6

In summary; my GitLab.com SaaS runner started working and I don’t know why!

I added the env command to the job, re-ran the CI Pipeline again and got different behaviour than previously. My pio_probe_hosts.py script reports “No attempt …” for all hosts, which means that is_proxy_set() is throwing an exception. Previously, it successfully made a socket connection to the first host.

Moreover, the subsequent pio run commands don’t report “No internet connection”, but now succeed in installing the required packages and then the overall build succeeds.

One other mystery is that this coverage build was also previously failing because LDF did not correctly detect all of the required source modules (only on the SaaS runner), so the final link failed. This time it succeeded! I suspected this might be connected to the Internet connection problem, and perhaps that is the case.

$ env
CI_PROJECT_NAMESPACE=xxxxxx-xxxxxxx
GITLAB_USER_ID=10495412
CI_RUNNER_VERSION=16.6.0~beta.105.gd2263193
FF_SKIP_NOOP_BUILD_STAGES=true
CI_SERVER_NAME=GitLab
CI_RUNNER_DESCRIPTION=3-blue.saas-linux-small-amd64.runners-manager.gitlab.com/default
GITLAB_USER_EMAIL=xxxxxx@xxxxxx.co.uk
CI_SERVER_REVISION=5d2ce169a5b
FF_USE_WINDOWS_LEGACY_PROCESS_STRATEGY=true
CI_RUNNER_EXECUTABLE_ARCH=linux/amd64
CI_PIPELINE_NAME=
CI_REGISTRY_USER=gitlab-ci-token
CI_API_V4_URL=https://gitlab.com/api/v4
CI_REGISTRY_PASSWORD=[MASKED]
CI_RUNNER_SHORT_TOKEN=zxwgkjAP
CI_JOB_NAME=dmod_coverage_ut
CI_OPEN_MERGE_REQUESTS=xxxxxx-xxxxxxx/ws1dry1!1064
HOSTNAME=runner-zxwgkjap-project-54156664-concurrent-0
GITLAB_USER_LOGIN=xxxxxxx
CI_PROJECT_NAME=ws1dry1
CI_PIPELINE_SOURCE=push
FF_RETRIEVE_POD_WARNING_EVENTS=false
CI_JOB_STATUS=running
CI_PIPELINE_ID=1168978552
FF_DISABLE_POWERSHELL_STDIN=false
CI_COMMIT_REF_SLUG=1617-prepare-ci-pipeline-for-update-to-gitlab-16-7
CI_SERVER=yes
FF_SET_PERMISSIONS_BEFORE_CLEANUP=true
CI_COMMIT_SHORT_SHA=d41e7cfd
CI_JOB_NAME_SLUG=dmod-coverage-ut
RUNNER_TEMP_PROJECT_DIR=/builds/xxxxxx-xxxxxxx/ws1dry1.tmp
FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION=false
CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX=gitlab.com:443/xxxxxx-xxxxxxx/dependency_proxy/containers
FF_USE_GIT_BUNDLE_URIS=true
PWD=/builds/xxxxxx-xxxxxxx/ws1dry1/dmod
CI_RUNNER_TAGS=["gce", "east-c", "linux", "ruby", "mysql", "postgres", "mongo", "git-annex", "shared", "docker", "saas-linux-small-amd64"]
CI_PROJECT_PATH=xxxxxx-xxxxxxx/ws1dry1
FF_USE_NEW_BASH_EVAL_STRATEGY=false
CI_SERVER_TLS_CA_FILE=/builds/xxxxxx-xxxxxxx/ws1dry1.tmp/CI_SERVER_TLS_CA_FILE
CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX=gitlab.com:443/xxxxxx-xxxxxxx/dependency_proxy/containers
CI_COMMIT_REF_PROTECTED=false
FF_USE_POWERSHELL_PATH_RESOLVER=false
CI_API_GRAPHQL_URL=https://gitlab.com/api/graphql
CI_SERVER_VERSION_MINOR=9
CI_COMMIT_SHA=d41e7cfdd05c3732c783849b90b8755eaf485d99
HOME=/root
FF_NETWORK_PER_BUILD=false
CI_DEPENDENCY_PROXY_PASSWORD=[MASKED]
CI_JOB_TIMEOUT=7200
CI_PROJECT_VISIBILITY=private
CI_CONCURRENT_PROJECT_ID=0
FF_SCRIPT_SECTIONS=false
CI_COMMIT_MESSAGE=gitlab-ci.yml: Add env dump and pio probe hosts to DMOD coverage job.
DOCKER_TLS_CERTDIR=
CI_SERVER_SHELL_SSH_PORT=22
monDockerImage=wsone_buildenv_phycore_imx8_a53_m4f:v6.7
CI_JOB_JWT_V1=[MASKED]
CI_JOB_JWT_V2=[MASKED]
FF_USE_DIRECT_DOWNLOAD=true
CI_PAGES_DOMAIN=gitlab.io
FF_PRINT_POD_EVENTS=false
CI_SERVER_VERSION=16.9.0-pre
FF_USE_POD_ACTIVE_DEADLINE_SECONDS=true
CI_REGISTRY=registry.gitlab.com
CI_SERVER_PORT=443
CI_PROJECT_NAMESPACE_ID=79864741
FF_USE_IMPROVED_URL_MASKING=false
CI_PAGES_URL=https://ws1dry1-xxxxxx-xxxxxxx-4fa0ead4973f0e8f81abf62cbac45755537485df.gitlab.io
CI_PIPELINE_IID=14
CI_REPOSITORY_URL=https://gitlab-ci-token:[MASKED]@gitlab.com/xxxxxx-xxxxxxx/ws1dry1.git
CI_SERVER_URL=https://gitlab.com
FF_ENABLE_BASH_EXIT_CODE_CHECK=false
GITLAB_FEATURES=audit_events,blocked_issues,board_iteration_lists,code_owners,code_review_analytics,full_codequality_report,group_activity_analytics,group_bulk_edit,issuable_default_templates,issue_weights,iterations,ldap_group_sync,merge_request_approvers,milestone_charts,multiple_issue_assignees,multiple_ldap_servers,multiple_merge_request_assignees,multiple_merge_request_reviewers,project_merge_request_analytics,protected_refs_for_users,push_rules,resource_access_token,seat_link,usage_quotas,visual_review_app,wip_limits,zoekt_code_search,blocked_work_items,description_diffs,send_emails_from_admin_area,repository_size_limit,maintenance_mode,scoped_issue_board,contribution_analytics,group_webhooks,member_lock,elastic_search,repository_mirrors,adjourned_deletion_for_projects_and_groups,admin_audit_log,auditor_user,blocking_merge_requests,board_assignee_lists,board_milestone_lists,ci_secrets_management,ci_pipeline_cancellation_restrictions,cluster_agents_ci_impersonation,cluster_agents_user_impersonation,cluster_deployments,code_owner_approval_required,code_suggestions,commit_committer_check,commit_committer_name_check,compliance_framework,custom_compliance_frameworks,custom_file_templates,custom_project_templates,cycle_analytics_for_groups,cycle_analytics_for_projects,db_load_balancing,default_branch_protection_restriction_in_groups,default_project_deletion_protection,delete_unconfirmed_users,dependency_proxy_for_packages,disable_name_update_for_users,disable_personal_access_tokens,domain_verification,epics,extended_audit_events,external_authorization_service_api_management,feature_flags_code_references,file_locks,geo,generic_alert_fingerprinting,git_two_factor_enforcement,group_allowed_email_domains,group_coverage_reports,group_forking_protection,group_milestone_project_releases,group_project_templates,group_repository_analytics,group_saml,group_scoped_ci_variables,ide_schema_config,incident_metric_upload,instance_level_scim,jira_issues_integration,ldap_group_sync_filter,merge_request_performance_metrics,admin_merge_request_approvers_rules,merge_trains,metrics_reports,metrics_observability,logs_observability,multiple_alert_http_integrations,multiple_approval_rules,multiple_group_issue_boards,object_storage,microsoft_group_sync,operations_dashboard,package_forwarding,pages_size_limit,pages_multiple_versions,productivity_analytics,project_aliases,protected_environments,reject_non_dco_commits,reject_unsigned_commits,remote_development,saml_group_sync,service_accounts,scoped_labels,smartcard_auth,ssh_certificates,swimlanes,target_branch_rules,type_of_work_analytics,minimal_access_role,unprotection_restrictions,ci_project_subscriptions,incident_timeline_view,oncall_schedules,escalation_policies,zentao_issues_integration,coverage_check_approval_rule,issuable_resource_links,group_protected_branches,group_level_merge_checks_setting,oidc_client_groups_claim,disable_deleting_account_for_users,disable_private_profiles,group_ip_restriction,issues_analytics,password_complexity,group_wikis,email_additional_text,custom_file_templates_for_namespace,incident_sla,export_user_permissions,cross_project_pipelines,feature_flags_related_issues,merge_pipelines,ci_cd_projects,github_integration,ai_chat,ai_config_chat,ai_features,ai_git_command,ai_tanuki_bot,ai_analyze_ci_job_failure,ai_agents,api_discovery,api_fuzzing,auto_rollback,breach_and_attack_simulation,fill_in_merge_request_template,cluster_image_scanning,external_status_checks,combined_project_analytics_dashboards,compliance_pipeline_configuration,container_scanning,credentials_inventory,custom_roles,dast,dependency_scanning,dora4_analytics,epic_colors,enterprise_templates,environment_alerts,evaluate_group_level_compliance_pipeline,explain_code,external_audit_events,experimental_features,generate_description,generate_commit_message,generate_test_file,ai_generate_cube_query,git_abuse_rate_limit,group_ci_cd_analytics,group_level_compliance_dashboard,group_level_analytics_dashboard,incident_management,inline_codequality,insights,issuable_health_status,issues_completed_analytics,jira_vulnerabilities_integration,jira_issue_association_enforcement,kubernetes_cluster_vulnerabilities,license_scanning,okrs,personal_access_token_expiration_policy,pre_receive_secret_detection,product_analytics,project_quality_summary,project_level_analytics_dashboard,prometheus_alerts,quality_management,related_epics,release_evidence_test_artifacts,report_approver_rules,required_ci_templates,requirements,runner_maintenance_note,runner_performance_insights,runner_upgrade_management,runner_upgrade_management_for_namespace,sast,sast_iac,sast_custom_rulesets,sast_fp_reduction,secret_detection,security_configuration_in_ui,security_dashboard,security_on_demand_scans,security_orchestration_policies,security_training,ssh_key_expiration_policy,summarize_mr_changes,summarize_my_mr_code_review,summarize_notes,summarize_submitted_review,stale_runner_cleanup_for_namespace,status_page,suggested_reviewers,subepics,tracing,unique_project_download_limit,vulnerability_auto_fix,vulnerability_finding_signatures,coverage_fuzzing,devops_adoption,group_level_devops_adoption,instance_level_devops_adoption
CI_COMMIT_DESCRIPTION=
FF_USE_ADVANCED_POD_SPEC_CONFIGURATION=false
CI_TEMPLATE_REGISTRY_HOST=registry.gitlab.com
CI_JOB_STAGE=unit_test
CI_PIPELINE_URL=https://gitlab.com/xxxxxx-xxxxxxx/ws1dry1/-/pipelines/1168978552
CI_DEFAULT_BRANCH=master
GITLAB_ENV=/builds/xxxxxx-xxxxxxx/ws1dry1.tmp/gitlab_runner_env
CI_SERVER_VERSION_PATCH=0
CI_COMMIT_TITLE=gitlab-ci.yml: Add env dump and pio probe hosts to DMOD coverage job.
CI_PROJECT_ROOT_NAMESPACE=xxxxxx-xxxxxxx
FF_ENABLE_JOB_CLEANUP=false
FF_RESOLVE_FULL_TLS_CHAIN=false
GITLAB_USER_NAME=Tim Jones
CI_PROJECT_DIR=/builds/xxxxxx-xxxxxxx/ws1dry1
SHLVL=1
CI_RUNNER_ID=12270835
CI_PIPELINE_CREATED_AT=2024-02-08T09:03:35Z
CI_COMMIT_TIMESTAMP=2024-02-08T09:03:31+00:00
CI_DISPOSABLE_ENVIRONMENT=true
CI_SERVER_SHELL_SSH_HOST=gitlab.com
CI_JOB_JWT=[MASKED]
CI_REGISTRY_IMAGE=registry.gitlab.com/xxxxxx-xxxxxxx/ws1dry1
CI_SERVER_PROTOCOL=https
dmodDockerImage=wsone_buildenv_dmod_platformio:v4.9
CI_COMMIT_AUTHOR=Tim Jones <xxxxxx@xxxxxx.co.uk>
FF_POSIXLY_CORRECT_ESCAPES=false
CI_COMMIT_REF_NAME=1617-prepare-ci-pipeline-for-update-to-gitlab-16-7
CI_SERVER_HOST=gitlab.com
CI_JOB_URL=https://gitlab.com/xxxxxx-xxxxxxx/ws1dry1/-/jobs/6124120660
CI_JOB_TOKEN=[MASKED]
CI_JOB_STARTED_AT=2024-02-08T09:03:41Z
CI_CONCURRENT_ID=615
CI_PROJECT_DESCRIPTION=
CI_COMMIT_BRANCH=1617-prepare-ci-pipeline-for-update-to-gitlab-16-7
CI_PROJECT_CLASSIFICATION_LABEL=
FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false
CI_RUNNER_REVISION=d2263193
FF_KUBERNETES_HONOR_ENTRYPOINT=false
FF_USE_NEW_SHELL_ESCAPE=false
CI_DEPENDENCY_PROXY_USER=gitlab-ci-token
FF_USE_DYNAMIC_TRACE_FORCE_SEND_INTERVAL=false
FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR=false
CI_PROJECT_PATH_SLUG=xxxxxx-xxxxxxx-ws1dry1
CI_NODE_TOTAL=1
CI_BUILDS_DIR=/builds
CI_JOB_ID=6124120660
CI_PROJECT_REPOSITORY_LANGUAGES=c,c++,python,qml,motorola 68k assembly
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
FF_SECRET_RESOLVING_FAILS_IF_MISSING=true
CI_PROJECT_ID=54156664
CI=true
GITLAB_CI=true
CI_JOB_IMAGE=registry.gitlab.com/xxxxxx-xxxxxxx/ws1dry1/wsone_buildenv_dmod_platformio:v4.9
CI_COMMIT_BEFORE_SHA=d12f366d04eae751408509a243a8da90eca4d9b7
CI_PROJECT_TITLE=Ws1Dry1
dockerRegistry=registry.gitlab.com/xxxxxx-xxxxxxx/ws1dry1
CI_SERVER_VERSION_MAJOR=16
CI_CONFIG_PATH=.gitlab-ci.yml
FF_USE_FASTZIP=false
CI_DEPENDENCY_PROXY_SERVER=gitlab.com:443
DOCKER_DRIVER=overlay2
CI_PROJECT_URL=https://gitlab.com/xxxxxx-xxxxxxx/ws1dry1
OLDPWD=/builds/xxxxxx-xxxxxxx/ws1dry1
_=/usr/bin/env
$ python3 scripts/test/pio_probe_hosts.py
No attempt for 185.199.110.153 failed
No attempt for 88.198.170.159 failed
No attempt for github.com failed
No attempt for registry.platformio.org failed
No attempt for registry.nm1.platformio.org failed
$ pio run -t clean -e native
Processing native (platform: native)
--------------------------------------------------------------------------------
Tool Manager: Installing platformio/tool-scons @ ~4.40400.0
Downloading 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Unpacking
Tool Manager: tool-scons@4.40400.0 has been installed!
Build environment is clean
Done cleaning
========================= [SUCCESS] Took 2.60 seconds =========================
Environment    Status    Duration
-------------  --------  ------------
native         SUCCESS   00:00:02.604
========================= 1 succeeded in 00:00:02.604 =========================
$ SCRIPT=scripts/test/native_unit_test_pio.sh
$ $SCRIPT
PlatformIO Core, version 6.1.4
Processing native (platform: native)
--------------------------------------------------------------------------------
Build environment is clean
Done cleaning
========================= [SUCCESS] Took 0.24 seconds =========================
Environment    Status    Duration
-------------  --------  ------------
native         SUCCESS   00:00:00.238
========================= 1 succeeded in 00:00:00.238 =========================
Collected 3 tests (native/test_gtest, target/test_dap, target/test_dha)
Processing native/test_gtest in native environment
--------------------------------------------------------------------------------
Building...
Library Manager: Installing throwtheswitch/Unity @ ^2.5.2
Unpacking
Library Manager: Unity@2.5.2 has been installed!
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain+, Compatibility ~ strict
Library Manager: Installing googletest
Unpacking
Library Manager: googletest@1.12.1 has been installed!
Found 13 compatible libraries
Scanning dependencies...
7

The pio_probe_hosts.py script reports “No attempt …” was because I’d omitted import os in this version of the script. Fixing that means that is_proxy_set() works as it should and a socket connection to the first host works.

So, the evidence we have is that SaaS runner “magically” started allowing pio run to detect an internet connection.

I still don’t understand, but at least it’s working…

8

Is this easily reproducable with a local free Gitlab instance plus a machine running the Gitlab Runner that can run docker containers?

9

I’m not sure that it is Max. I’ve just seen the problem disappear, apparently without me changing anything. If we tried to set up a test environment, we might see the same problem or we might not.

I’m happy if we park this issue for now. I’ll certainly let you know if I see it again.