Custom build of esp32 over idf-release/v5.1

Not so easy. Easiest is to build a complete Framework with the libs which can be used in Platformio. I modified the Arduino Lib Builder to generate a fully valid framework.
My fork of the lib Builder. GitHub - Jason2866/esp32-arduino-lib-builder at 5.1_fs_grow
The build is done via GH Actions and generates Releases. This Releases can directly be used with platform_packages