HTTPS requests to AWS S3 server using an ESP32

I have an ESP32 board and I use the adruino framework in VScode in my platformIO project.

I want to download/upload a file from/to an S3 bucket. The bucket and its content should be private (blocked to public). Since the file size is approx. between 1MB (download) and 10MB(upload) I would like to use HTTPS. The 10MB logfiles are stored on an SD card.

I have found several hints on what could be used to achieve this:

I have already spent at least 10h of research and a little try n error testing some exampels without success and still not knowing what is the most promising approach to accomplish my goals. I already use MQTT to connect to AWS IoT bi-directional which works fine but I am struggling to set up the correct header to get access to my S3 bucket.

Could someone either

  • point me in the right direction, which libraries/Frameworks (e.g. AWS FreeRTOS) I should include into my project or

  • provide me with a tutorial on how to configure my HTTPS header correctly (especially on how to authenticate correctly to S3 using my AWS Access/Secret key and/or certificates) to access a non-public S3 bucket to download/upload form and to.

So, I would like to use the WiFiClientSecure library and not the HTTPClient library, because the latter one is somehow limited or I simply do not understand how to send my “self-made” HTTPS request header using it.

This is my code:

        WiFiClientSecure ota_client;
        String bucket_url = "https://" + bucket_name + "." + service + "." + region + "." + host + "/" + object;
        // Example bucket_url: ""
        ota_client.connect(bucket_url.c_str(), 443);

        // Wait for the response
        while (ota_client.connected() && !ota_client.available());

        // Read and print the response
        while (ota_client.available()) 
            String line = ota_client.readStringUntil('\r');

The AWS_CERT_CA is the root certificate I am also using for connecting to the AWS IoT MQTT broker.
The http_header I am trying to send has been built following these AWS documentations:




This is my canonical request:




This is my string to sign (hash has been altered):


This is my final header (content has been adapted regarding AWS ACCESS key and signature):

GET /firmware.bin HTTP/1.1
Date: Wed, 08 Mar 2023 15:15:38 GMT
Authorization: AWS4-HMAC-SHA25 Credentials=AKIAZOKIXQT4EXAMPLE/20230308/eu-central-1/s3/aws4_request,SignedHeaders=host;x-amz-date,Signature=0d4e215b75606f5e8ff45f8b87cac62d7c9a2e58b6cead7fdc34886a7417154f

The errors I ger are:

[ 33426][E][WiFiGeneric.cpp:1438] hostByName(): DNS Failed for
[ 33430][E][WiFiClientSecure.cpp:135] connect(): start_ssl_client: -1

So I believe at this point something with the certificate but also with the URL is wrong. I checked the firmware.bin URL in AWS and it is exactly the same as the one I use for the connection. Now I hope someone can help me with a few things:

  1. Is the certificate I am using the correct one - it was generated when I created a “Thing” in AWS IoT? If not how can I generate a proper one to connect to S3?
  2. Does my general signing approach look correct or can you see any issues?
  3. To create hashs, I use my AWS SECRET ACCESS key, is this correct? It is not mentioned in the third link I provided.
  4. Is there something wrong in my general approach to connect to S3 via HTTPS?

Any help would be great!

No. WiFiClientSecure is a wrapper around Just TLS. When you use the connect() method, it expects a hostname. Not the full URL. So you’d just give it

Make sure this http_header string ends in two \r\n linebreaks, otherwise the HTTP server won’t recognize that the request has ended.

Thank you for the response.

Make sure this http_header string ends in two \r\n linebreaks, otherwise the HTTP server won’t recognize that the request has ended.

Thanks for the hint, I added them both. Besides, in the AWS documentation they always say:

Create a string by concatenating the following strings, separated by newline characters.

Do you think they mean using \r\n or simply \n?

It’s CRLF (\r\n). But if your your HTTP request has a message body then a final CRLF CRLF is not needed, only between header end and message body start, just like RFC 2616 says

Ok, thank you for the hint I will update my code.

Regarding the conncetion, I still get the same error:

[ 44752][E][WiFiGeneric.cpp:1438] hostByName(): DNS Failed for
[ 44780][E][WiFiClientSecure.cpp:135] connect(): start_ssl_client: -1

This is still using the full URL. Can you show the updated code in full?

Oh, my bad, I forgot to remove the object string at the end of the bucket URL. Here is my code.

                String bucket_url = bucket_name + "." + service + "." + region + "." + host;
                Serial.println("Bucket URL: ");
                WiFiClientSecure ota1_client;
                ota1_client.connect(bucket_url.c_str(), 443);

                // Wait for the response
                while (ota1_client.connected() && !ota1_client.available());

                // Read and print the response
                while (ota1_client.available()) 
                    String line = ota1_client.readStringUntil('\r');

And now I am finally receiving a response (altered at sensitive location).

Bucket URL:
HTTP/1.1 400 Bad Request
x-amz-request-id: 1JTPQC3C4ET8JTXB
x-amz-id-2: qVT0RKD/ScBqX51/TdjtT8B9w1/A+01pkYFalSr1yJ9/jJc4Pan5OV+Q226wVtPQHN/X+qSBjtk=
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Wed, 08 Mar 2023 17:38:02 GMT
Server: AmazonS3
Connection: close

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidArgument</Code><Message>Unsupported Authorization Type</Message><ArgumentName>Authorization</ArgumentName><ArgumentValue>AWS4-HMAC-SHA25 Credentials=AKIAZOKIXQT4DEXAMPLE/20230308/eu-central-1/s3/aws4_request,SignedHeaders=host;x-amz-date,Signature=0cea32460ea4345096d06dc8a35346aadb83d4ceeb1e712cb8185bda197dd</ArgumentValue><RequestId>1JTPQC3C4ET8JTXB</RequestId><HostId>qVT0RKD/ScBqX51/TdU8KlI0O/A+01pkYFalSr1yJ9/jJc4Pan5OV+Q226wVtPQHN/X+qSBjtk=</HostId></Error>

Well, it’s at least something. Now you probably have to corect the authentication headers or the authentication token you used already ran out?

Yes I have to double and triple check the documentation to ensure request signing. Thank you for your help so far.