Epsressif team works on a multi-platform solution for ESP-IDF. Recently, they added support for Cmake. We hope that they will provide better interface how to configure ESP-IDF options without a ton of dependencies and menuconfig.
Our goal is to make PlatformIO multi-platform and to work with zero-dependencies to 3rd party software.
Meanwhile, you can control ESP-IDF settings manually in sdkconfig.h. See #define CONFIG_ESP32_DEFAULT_CPU_FREQ_MHZ 160 option in https://github.com/platformio/platform-espressif32/blob/develop/examples/espidf-hello-world/src/sdkconfig.h#L196