Possible to write Serial Monitor custom filters to process raw binary data?

PlatformIO Core
1

I am trying to develop a binary serial protocol for my device, and wanted to use PIO Serial Monitor with custom filters to aid in that development, but so far it has only been a hindrance.

It seems that the API for filters assumes the serial data is some form of text, and therefore must have an associated encoding (defaulting to utf-8). Is there any way to specify non-text-encoded, raw binary?
I know I can change the encoding with platformio.ini setting monitor_encoding, but platformio docs don’t specify what all the possible options are (I guess it corresponds with the list here? 6.6. codecs — Codec registry and base classes — Python v3.2.6 documentation )
Even so, I don’t think I want to specify an encoding at all, since that would mean it is some kind of text representation.

Even the builtin hexlify filter is unusable for me because all data is forced to be interpreted as utf-8.
As far as I can tell, when being sent arbitrary bytes, many sequences are seen as invalid utf-8, and get replaced with the unicode replacement character 0xFFFD, which hexlify prints as ?? because the “byte value” from serial.iterbytes is greater than 255.
Having a value greater than 255 is completely nonsensical if you are trying to process raw bytes from the very start.

There is also a monitor_raw setting, but that gives an error if attempted to use with a monitor_filter.

2

I was beating my head against the wall with the same issues you were describing for several hours…

Finally created a Github issue to hopefully make a better future for custom filters.

Turns out that all you need to do is ALSO specify “monitor-encoding: Latin1” in the “platformio.ini”
“monitor-encoding: hexlify” is less than useless, as it simply omits all non-ASCII characters.

With “monitor-encoding: Latin1”, you can use “monitor-filters: hexlify” and see the full range. Grabbing the hexlify Python script code is a great starting basis for making a custom filter.

3

Thanks for opening an issue, it looks like some progress may come of it.
I was trying to meet a deadline, and didn’t have time to continue messing with it.
Instead just moved on to developing the client/desktop software for the device (built on electronjs) to have its own serial debug functionality.