HTTPS requests to AWS S3 server using an ESP32

[lkyslevin]

I have an ESP32 board and I use the adruino framework in VScode in my platformIO project.

I want to download/upload a file from/to an S3 bucket. The bucket and its content should be private (blocked to public). Since the file size is approx. between 1MB (download) and 10MB(upload) I would like to use HTTPS. The 10MB logfiles are stored on an SD card.

I have found several hints on what could be used to achieve this:

• AWS FreeRTOS
• AWS SDK C++
• esspressif-aws-iot
• arduino-esp32 OTA example
• Several forum posts here and there about the general concept of HTTPS requests to S3

I have already spent at least 10h of research and a little try n error testing some exampels without success and still not knowing what is the most promising approach to accomplish my goals. I already use MQTT to connect to AWS IoT bi-directional which works fine but I am struggling to set up the correct header to get access to my S3 bucket.

Could someone either

• point me in the right direction, which libraries/Frameworks (e.g. AWS FreeRTOS) I should include into my project or

• provide me with a tutorial on how to configure my HTTPS header correctly (especially on how to authenticate correctly to S3 using my AWS Access/Secret key and/or certificates) to access a non-public S3 bucket to download/upload form and to.

[lkyslevin]

So, I would like to use the WiFiClientSecure library and not the HTTPClient library, because the latter one is somehow limited or I simply do not understand how to send my “self-made” HTTPS request header using it.

This is my code:

        WiFiClientSecure ota_client;
        ota_client.setCACert(AWS_CERT_CA);
        String bucket_url = "https://" + bucket_name + "." + service + "." + region + "." + host + "/" + object;
        // Example bucket_url: "https://test_bucket.s3.eu-central-1.amazonaws.com/firmware.bin"
        ota_client.connect(bucket_url.c_str(), 443);
        ota_client.print(http_header);

        // Wait for the response
        while (ota_client.connected() && !ota_client.available());

        // Read and print the response
        while (ota_client.available()) 
        {
            String line = ota_client.readStringUntil('\r');
            Serial.print(line);
        }

The AWS_CERT_CA is the root certificate I am also using for connecting to the AWS IoT MQTT broker.
The http_header I am trying to send has been built following these AWS documentations:

sig-v4-header-based-auth

GetObject

create-string-to-sign

This is my canonical request:

GET
/firmware.bin

content-type:application/octet-stream
host:test_bucket.s3.amazonaws.com
x-amz-date:20230308T151538Z

host;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

This is my string to sign (hash has been altered):

AWS4-HMAC-SHA25
20230308T151538Z
20230308/eu-central-1/s3/aws4_request
08f718a3ffa27f18fad718eab178f3564f1a1c2f389540cc36f17e329304bc45

This is my final header (content has been adapted regarding AWS ACCESS key and signature):

GET /firmware.bin HTTP/1.1
Host: test_bucket.s3.eu-central-1.amazonaws.com
Date: Wed, 08 Mar 2023 15:15:38 GMT
Authorization: AWS4-HMAC-SHA25 Credentials=AKIAZOKIXQT4EXAMPLE/20230308/eu-central-1/s3/aws4_request,SignedHeaders=host;x-amz-date,Signature=0d4e215b75606f5e8ff45f8b87cac62d7c9a2e58b6cead7fdc34886a7417154f

The errors I ger are:

[ 33426][E][WiFiGeneric.cpp:1438] hostByName(): DNS Failed for https://test_bucket.s3.eu-central-1.amazonaws.com/firmware.bin
[ 33430][E][WiFiClientSecure.cpp:135] connect(): start_ssl_client: -1

So I believe at this point something with the certificate but also with the URL is wrong. I checked the firmware.bin URL in AWS and it is exactly the same as the one I use for the connection. Now I hope someone can help me with a few things:

1. Is the certificate I am using the correct one - it was generated when I created a “Thing” in AWS IoT? If not how can I generate a proper one to connect to S3?
2. Does my general signing approach look correct or can you see any issues?
3. To create hashs, I use my AWS SECRET ACCESS key, is this correct? It is not mentioned in the third link I provided.
4. Is there something wrong in my general approach to connect to S3 via HTTPS?

Any help would be great!

[maxgerhardt]

@lkyslevin

        String bucket_url = "https://" + bucket_name + "." + service + "." + region + "." + host + "/" + object;
        // Example bucket_url: "https://test_bucket.s3.eu-central-1.amazonaws.com/firmware.bin"
        ota_client.connect(bucket_url.c_str(), 443);

No. WiFiClientSecure is a wrapper around Just TLS. When you use the connect() method, it expects a hostname. Not the full URL. So you’d just give it test_bucket.s3.eu-central-1.amazonaws.com.

@lkyslevin

ota_client.print(http_header);

Make sure this http_header string ends in two \r\n linebreaks, otherwise the HTTP server won’t recognize that the request has ended.

[lkyslevin]

Thank you for the response.

Make sure this http_header string ends in two \r\n linebreaks, otherwise the HTTP server won’t recognize that the request has ended.

Thanks for the hint, I added them both. Besides, in the AWS documentation they always say:

Create a string by concatenating the following strings, separated by newline characters.

Do you think they mean using \r\n or simply \n?

[maxgerhardt]

It’s CRLF (\r\n). But if your your HTTP request has a message body then a final CRLF CRLF is not needed, only between header end and message body start, just like RFC 2616 says

[lkyslevin]

Ok, thank you for the hint I will update my code.

Regarding the conncetion, I still get the same error:

[ 44752][E][WiFiGeneric.cpp:1438] hostByName(): DNS Failed for test_bucket.s3.eu-central-1.amazonaws.com/firmware.bin
[ 44780][E][WiFiClientSecure.cpp:135] connect(): start_ssl_client: -1

[maxgerhardt]

@lkyslevin

DNS Failed for test_bucket.s3.eu-central-1.amazonaws.com/firmware.bin

This is still using the full URL. Can you show the updated code in full?

[lkyslevin]

Oh, my bad, I forgot to remove the object string at the end of the bucket URL. Here is my code.

                String bucket_url = bucket_name + "." + service + "." + region + "." + host;
                Serial.println("Bucket URL: ");
                Serial.println(bucket_url);
                WiFiClientSecure ota1_client;
                ota1_client.setCACert(AWS_CERT_CA);
                ota1_client.connect(bucket_url.c_str(), 443);
                ota1_client.print(http_header);

                // Wait for the response
                while (ota1_client.connected() && !ota1_client.available());

                // Read and print the response
                while (ota1_client.available()) 
                {
                    String line = ota1_client.readStringUntil('\r');
                    Serial.print(line);
                }
                ota1_client.stop();

And now I am finally receiving a response (altered at sensitive location).

Bucket URL: 
test_bucket.s3.eu-central-1.amazonaws.com
HTTP/1.1 400 Bad Request
x-amz-request-id: 1JTPQC3C4ET8JTXB
x-amz-id-2: qVT0RKD/ScBqX51/TdjtT8B9w1/A+01pkYFalSr1yJ9/jJc4Pan5OV+Q226wVtPQHN/X+qSBjtk=
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Wed, 08 Mar 2023 17:38:02 GMT
Server: AmazonS3
Connection: close

20f
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidArgument</Code><Message>Unsupported Authorization Type</Message><ArgumentName>Authorization</ArgumentName><ArgumentValue>AWS4-HMAC-SHA25 Credentials=AKIAZOKIXQT4DEXAMPLE/20230308/eu-central-1/s3/aws4_request,SignedHeaders=host;x-amz-date,Signature=0cea32460ea4345096d06dc8a35346aadb83d4ceeb1e712cb8185bda197dd</ArgumentValue><RequestId>1JTPQC3C4ET8JTXB</RequestId><HostId>qVT0RKD/ScBqX51/TdU8KlI0O/A+01pkYFalSr1yJ9/jJc4Pan5OV+Q226wVtPQHN/X+qSBjtk=</HostId></Error>
0

[maxgerhardt]

Well, it’s at least something. Now you probably have to corect the authentication headers or the authentication token you used already ran out?

[lkyslevin]

Yes I have to double and triple check the documentation to ensure request signing. Thank you for your help so far.

[lavanya38876]

HTTP/1.1 400 Bad Request

x-amz-request-id: 6J4T37WXJ315HF4N

x-amz-id-2: CajaCwQBmPt9+9ELjq/VzDPYZPJVD3sars2ymtqIPelGuWEW/mj8QPlNXGeBacuvIU39VM5xY16N7OSsaBzliA==

Content-Type: application/xml

Transfer-Encoding: chunked

Date: Fri, 02 Aug 2024 13:09:30 GMT

Server: AmazonS3

Connection: close



132

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidRequest</Code><Message>Missing required header for this request: x-amz-content-sha256</Message><RequestId>6J4T37WXJ315HF4N</RequestId><HostId>CajaCwQBmPt9+9ELjq/VzDPYZP**rs2ymtqIPelGuWEW/mj8QPlNXGeBacuvIU39VM5xY16N7OSsaBzliA==</HostId></Error>

I'm doing same operation, i have also ended up with this issue, did you find any solution?

[lkyslevin]

I would recommend to go through this thread of mine:

AWS S3 HTTPS error SignatureDoesNotMatch

Sorry for the late reply, I have attached my .cpp file with several utility functions. /****************************************************************************** Includes *****************************************************************************/ #include "aws_http.h" /****************************************************************************** Function prototypes *****************************************************************************/ void hashByteToHex(unsigned char *hash…