Please read through Using esp-idf library within the Arduino Framework (ESP32) - #2 by maxgerhardt, it might solve a lot of your questions.
If you are using Arduino-ESP32, the best way (currently) is to use GitHub - espressif/esp32-arduino-lib-builder to recompile the needed libraries undere a new configuration.
If you want to have different libraries per environment in the same project (I’m really not sure why you would want to do that though) you need to use Advanced Scripting — PlatformIO latest documentation to dynamically redirect the lib folder to where your custom ones are. But I highly recommend starting off with just one recompilation of the libraries for one environment first.