Library specific build_flags

Is there a way to enable -Os (or any other build_flags) only for libraries? I only care about debugging my own code and compiling certain libraries in debug causes insane code bloat (and my teensy runs out of memory)
I know that I could edit build.flags inside library.json, but this means I’d need to create my own fork of that library

See + adapt from Granular optimisation settings - #2 by maxgerhardt.