“Something” is using a ton of RAM and it fails the build. Probably some memory size for a buffer is set way too high for the poor ESP8266’s RAM. Try to check your configuration files you use regarding resolutions and buffer sizes.
Otherwise, the ‘hard’ way is to go hack the linker scripts to artificially increase the RAM size and then analyze the .elf / .map file to see what’s taking up the memory, akin to [ESP32] ld: region `dram0_0_seg' overflowed by 156768 bytes.