How to transfer a large data array from PSRAM to SPIFFS (ESP32)

1

Hi,
I’ve successfully got OTA firmware updating over GSM (SIM7600) via FTP.
Works every time on small files (blinky) about 260KB, but when I try my application firmware (1114080) the file transfer to SPIFFS slows down around 700KB and finally stops short at 1105404.

FFS available memory:        1378241 bytes
FFS memory used:             1105404 bytes

I’m using this which is probably not the best way:

int filesize = 1114080;
  File otafile = SPIFFS.open("/update.bin", FILE_WRITE);
  for (int i = 0; i < filesize; i++) {
    otafile.print(char(psRamFile[i]));
  }
  otafile.close();

It’s a bit tricky to transfer the files from the SIM7600 modem over serial, this is the process:
FTP site download file to SIM7600 internal flash.
Transfer the file over serial (460800 baud) to PSRAM.
Process and copy to a second PSRAM container to combine the 512 byte data blocks.
Then transfer this to the SPIFFS before calling updateFromFS() function.

Takes the same time as Wi-Fi ota apart from my PSRAM to SPIFFS process is slow.

I’m using two container buffers in PSRAM:

char* psRamBuffer = (char*)ps_malloc(1500000);
char* psRamFile = (char*)ps_malloc(1500000);

I did try this:

File otafile = SPIFFS.open("/update.bin", FILE_WRITE);  
otafile.write((byte *)&psRamFile, sizeof(psRamFile));
otafile.close();

FFS available memory:        1378241 bytes
FFS memory used:                 502 bytes
PSRAM size:                  4194220 bytes
PSRAM available memory:      1194220 bytes

But only wrote 502 bytes, I don’t think I’ve got this quite right tbh, but may be the way to do it.

Any assistance would be appreciated and I will share the code in case anyone else needs it.

2

sizeof(psRamFile) does not do what you want here, it gives you sizeof(char*) = 4, it does not contain the original 1500000 used in the malloc call. Have you tried replacing this with the actual number of bytes of the psRamFile buffer (may be less than 1500000, I’m sure the actual size either calculated or indicated in your update protocol).

3

Hi Max,

I do have the exact bin file size from the FTP file attributes information.
I am using espressif32@3.4.0(v1.0.6), not sure if that is relevant to my issue.
I want to update but there is a problem with pubsubClient library, something for me to deal with another day.

If I change to this:

int BINfilesize = 1114080; // v38.bin
//int BINfilesize = 267856;  // Blink.bin
char* psRamBuffer = (char*)ps_malloc(1500000);
char* psRamFile = (char*)ps_malloc(BINfilesize);

otafile.write((byte*)&psRamFile, BINfilesize);

Then I get:

FFS available memory:        1378241 bytes
FFS memory used:                 251 bytes
PSRAM size:                  4194220 bytes
PSRAM available memory:      1580140 bytes
Error, Empty file

So…
I reconfigured the “esp-wrover-kit.json” file to use the 8MB flash I have on my esp32.
Not sure if these are the best options here.

{
  "build": {
    "arduino":{
      "ldscript": "esp32_out.ld",
      "partitions": "default_8MB.csv"
    },
    "core": "esp32",
    "extra_flags": "-DARDUINO_ESP32_DEV",
    "f_cpu": "240000000L",
    "f_flash": "40000000L",
    "flash_mode": "dio",
    "hwids": [
      [
        "0x0403",
        "0x6010"
      ]
    ],
    "mcu": "esp32",
    "variant": "esp32"
  },
  "connectivity": [
    "wifi",
    "bluetooth",
    "ethernet",
    "can"
  ],
  "debug": {
    "default_tool": "ftdi",
    "onboard_tools": [
      "ftdi"
    ],
    "openocd_board": "esp32-wrover.cfg"
  },
  "frameworks": [
    "arduino",
    "espidf"
  ],
  "name": "Espressif ESP-WROVER",
  "upload": {
    "flash_size": "8MB",
    "maximum_ram_size": 327680,
    "maximum_size": 8388608,
    "protocols": [
      "esptool",
      "espota",
      "ftdi"
    ],
    "require_upload_port": true,
    "speed": 1500000
  },
  "url": "https://espressif.com/en/products/hardware/esp-wrover-kit/overview",
  "vendor": "Espressif"
}

I can now transfer the large application .bin file from PSRAM to SPIFFS the old fashioned way:

File updateBin = SPIFFS.open("/update.bin", FILE_WRITE);  
  for (int i = 0; i < BINfilesize; i++){
    updateBin.print(char(psRamFile[i]));
  }
  updateBin.close();

It is quite slow but usable, perhaps this is as fast as I can get. The bottleneck is the PSRAM to SPIFFS copy that is slow.

Transfer Modem file v38.bin to ESP32 PSRAM… OK
Process file… OK
Transfer update file from PSRAM to Flash… OK
FFS available memory: 1498721 bytes
FFS memory used: 1123225 bytes
PSRAM size: 4194220 bytes
PSRAM available memory: 1580140 bytes
Starting update
Write firmware to flash, 1114080 bytes successfully
OTA finished!
Restart ESP device!

But not with:

otafile.write((byte*)&psRamFile, BINfilesize);

Getting closer now but can’t light up that cigar just yet.
Any help would be appreciated.

4

And did the return value of that function also return the number of bytes written that you wanted?

5

No, it returns 0 with 251 bytes FFS used.

6

What if the & is dropped?

otafile.write((byte*)psRamFile, BINfilesize);
7

The same.

Just wondering if I should be looking at this from a different angle.
WiFi OTA clearly works, is there a way for me to hook into this and stream the data into the that part of the flash?

8

After a look at the Update.h and Updater.cpp files I noticed:

size_t write(uint8_t *data, size_t len);

This appears to do the same without the Stream part.

Update.begin(BINfilesize);
Update.write((byte*)psRamFile, BINfilesize);
Update.end();

Does what I need here, not necessary to copy that psRamFile to the SPIFFS.
OTA update time is now reduced 45 seconds.

1 Like