Arduino-ESP32 is a component on top of ESP-IDF. With the latest PlatformIO supported core v2.0.17, that is ESP-IDF v4.4.7. Quite old, considering the v3.x core is built on ESP-IDF 5.1.4, but it still works. There are currently some issues with mainline PlatformIO supporting the 3.x core.
If you only stay within the libraries and core functions of the Arduino core, you won’t notice a difference.