Extra_scripts seems to be executed when I change GitHub branches in VSCode

Yes. Your extra_script could be adding compile / linker settings, so PlatformIO of course executes it when the project is loaded or the platformio.ini itself is changed.

You can determine whether you are in an active build or not as shown in e.g.