MQTT 5 integration with PlatformIO

I’m not sure if I post in the right forum/channel.

I just started developing with MQTT 5 and I would like to know what are the plans for the integration of ESP-MQTT 5 “natively” :

mqtt5_client.c : esp-mqtt/mqtt5_client.c at master · espressif/esp-mqtt · GitHub
mqtt5_client.h : esp-mqtt/mqtt5_client.h at master · espressif/esp-mqtt · GitHub
…and all other related files including a new version of mqtt_client.h that include conditionally mqtt5.h the following way :

#ifdef CONFIG_MQTT_PROTOCOL_5
    #include "mqtt5_client.h"
#endif

Meanwhile, how would be the “right” way to override the mqtt_client.h with the new version implementing mqtt5 ?
I’ve read about using “lib_ignore” in platformio.ini this way :

lib_ignore =
  mqtt_client

…and adding the new mqtt_client.h under the project include folder. I also added the new mqtt_client.c in the src folder, but that doesn’t seem to work as I got errors that lead me to think that the mqtt_client.h under the include folder of the project was not used but the old one (under .platformio folders tree).

Ex. of errors :

'esp_mqtt_client_config_t' {aka 'const struct <anonymous>'} has no member named 'credentials'
         if (config->credentials.client_id) {

'esp_mqtt_client_config_t' {aka 'const struct <anonymous>'} has no member named 'session'
     if (config->session.disable_keepalive) {

'esp_mqtt_client_config_t' {aka 'const struct <anonymous>'} has no member named 'network'
     client->config->auto_reconnect = !config->network.disable_auto_reconnect;
...

The new version of mqtt_client.h clearly includes those structs members :

typedef struct esp_mqtt_client_config_t {
  /**
   *   Broker related configuration
   */
  struct broker_t {
      /**
       * Broker address
       *
       *  - uri have precedence over other fields
       *  - If uri isn't set at least hostname, transport and port should.
       */
      struct address_t {
          const char *uri; /*!< Complete *MQTT* broker URI */
          const char *hostname; /*!< Hostname, to set ipv4 pass it as string) */
          esp_mqtt_transport_t transport; /*!< Selects transport*/
          const char *path;               /*!< Path in the URI*/
          uint32_t port;                  /*!< *MQTT* server port */
      } address; /*!< Broker address configuration */
      /**
       * Broker identity verification
       *
       * If fields are not set broker's identity isn't verified. it's recommended
       * to set the options in this struct for security reasons.
       */
      struct verification_t {
          bool use_global_ca_store; /*!< Use a global ca_store, look esp-tls
                         documentation for details. */
          esp_err_t (*crt_bundle_attach)(void *conf); /*!< Pointer to ESP x509 Certificate Bundle attach function for
                                                the usage of certificate bundles. */
          const char *certificate; /*!< Certificate data, default is NULL, not required to verify the server. */
          size_t certificate_len; /*!< Length of the buffer pointed to by certificate. */
          const struct psk_key_hint *psk_hint_key; /*!< Pointer to PSK struct defined in esp_tls.h to enable PSK
                                             authentication (as alternative to certificate verification).
                                             PSK is enabled only if there are no other ways to
                                             verify broker.*/
          bool skip_cert_common_name_check; /*!< Skip any validation of server certificate CN field, this reduces the
                                      security of TLS and makes the *MQTT* client susceptible to MITM attacks  */
          const char **alpn_protos;        /*!< NULL-terminated list of supported application protocols to be used for ALPN */
      } verification; /*!< Security verification of the broker */
  } broker; /*!< Broker address and security verification */
  /**
   * Client related credentials for authentication.
   */
  struct credentials_t {
      const char *username;    /*!< *MQTT* username */
      const char *client_id;   /*!< Set *MQTT* client identifier. Ignored if set_null_client_id == true If NULL set
                         the default client id. Default client id is ``ESP32_%CHIPID%`` where `%CHIPID%` are
                         last 3 bytes of MAC address in hex format */
      bool set_null_client_id; /*!< Selects a NULL client id */
      /**
       * Client authentication
       *
       * Fields related to client authentication by broker
       *
       * For mutual authentication using TLS, user could select certificate and key,
       * secure element or digital signature peripheral if available.
       *
       */
      struct authentication_t {
          const char *password;    /*!< *MQTT* password */
          const char *certificate; /*!< Certificate for ssl mutual authentication, not required if mutual
                             authentication is not needed. Must be provided with `key`.*/
          size_t certificate_len;  /*!< Length of the buffer pointed to by certificate.*/
          const char *key;       /*!< Private key for SSL mutual authentication, not required if mutual authentication
                           is not needed. If it is not NULL, also `certificate` has to be provided.*/
          size_t key_len; /*!< Length of the buffer pointed to by key.*/
          const char *key_password; /*!< Client key decryption password, not PEM nor DER, if provided
                               `key_password_len` must be correctly set. */
          int key_password_len;    /*!< Length of the password pointed to by `key_password` */
          bool use_secure_element; /*!< Enable secure element, available in ESP32-ROOM-32SE, for SSL connection */
          void *ds_data; /*!< Carrier of handle for digital signature parameters, digital signature peripheral is
                   available in some Espressif devices. */
      } authentication; /*!< Client authentication */
  } credentials; /*!< User credentials for broker */
  /**
   * *MQTT* Session related configuration
   */
  struct session_t {
      /**
       * Last Will and Testament message configuration.
       */
      struct last_will_t {
          const char *topic; /*!< LWT (Last Will and Testament) message topic */
          const char *msg; /*!< LWT message, may be NULL terminated*/
          int msg_len; /*!< LWT message length, if msg isn't NULL terminated must have the correct length */
          int qos;     /*!< LWT message QoS */
          int retain;  /*!< LWT retained message flag */
      } last_will; /*!< Last will configuration */
      bool disable_clean_session; /*!< *MQTT* clean session, default clean_session is true */
      int keepalive;              /*!< *MQTT* keepalive, default is 120 seconds */
      bool disable_keepalive; /*!< Set `disable_keepalive=true` to turn off keep-alive mechanism, keepalive is active
                        by default. Note: setting the config value `keepalive` to `0` doesn't disable
                        keepalive feature, but uses a default keepalive period */
      esp_mqtt_protocol_ver_t protocol_ver; /*!< *MQTT* protocol version used for connection.*/
      int message_retransmit_timeout; /*!< timeout for retransmitting of failed packet */
  } session; /*!< *MQTT* session configuration. */
  /**
   * Network related configuration
   */
  struct network_t {
      int reconnect_timeout_ms; /*!< Reconnect to the broker after this value in miliseconds if auto reconnect is not
                          disabled (defaults to 10s) */
      int timeout_ms; /*!< Abort network operation if it is not completed after this value, in milliseconds
                (defaults to 10s). */
      int refresh_connection_after_ms; /*!< Refresh connection after this value (in milliseconds) */
      bool disable_auto_reconnect;     /*!< Client will reconnect to server (when errors/disconnect). Set
                                 `disable_auto_reconnect=true` to disable */
  } network; /*!< Network configuration */
  /**
   * Client task configuration
   */
  struct task_t {
      int priority;   /*!< *MQTT* task priority*/
      int stack_size; /*!< *MQTT* task stack size*/
  } task; /*!< FreeRTOS task configuration.*/
  /**
   * Client buffer size configuration
   *
   * Client have two buffers for input and output respectivelly.
   */
  struct buffer_t {
      int size;     /*!< size of *MQTT* send/receive buffer*/
      int out_size; /*!< size of *MQTT* output buffer. If not defined, defaults to the size defined by
              ``buffer_size`` */
  } buffer; /*!< Buffer size configuration.*/
} esp_mqtt_client_config_t;

Note that I also added this line in platform.ini:

build_flags =
  -D MQTT_PROTOCOL_5=y

Thank you.

Anyone ? How can we override the platformIO included espressif mqtt_client version to use the most recent version supporting MQTT5 (GitHub - espressif/esp-mqtt: ESP32 mqtt component) ? Maybe lib_ignore doesn’t work for this library or maybe I don’t know how to use it.

I also tried adding this line in platformio.ini hoping it will have precedence over the platformio framework library, but it didn’t work either :

lib_deps = https://github.com/espressif/esp-mqtt.git

You need to use ESP-IDF framework. See Espressif IoT Development Framework — PlatformIO latest documentation

I’m actually using Arduino/ESP-IDF “combined” mode and I set those lines at the beginning of my platformio.ini config file in my project :

[env]
		platform = https://github.com/platformio/platform-espressif32.git
		framework = arduino, espidf

If I understand well, the version of ESP-IDF used is the version that is included with the PlatformIO (lastest) version I’m using ?

I was able to make my project work with MQTT5, but I had to get the lastest ESP-MQTT sources from Espressif github repo and put them under /lib project folder. I then included those .h files in my main.cpp file :

#include "../lib/esp-mqtt/mqtt_client.h"
#include "../lib/esp-mqtt/mqtt5_client.h"

mqtt5_client.h is not even included in the version of ESP-IDF framework included with PlatformIO. But it is included in this Github repo of Espressif: https://github.com/espressif/esp-mqtt. This version of MQTT (v5) is more recent than the one that is included in the ESP-IDF version that PlatformIO framework includes.

Is there a better way to use the most recent version of MQTT5 than the way I did it ?

Thank you.

Since that’s an ESP-IDF component, it’s wrong to include it via lib_deps. You must put the component in the components folder of the project, just like the ESP-IDF documentation says and an official example shows.

You then configure the component via the documented menuconfig, not by setting the config macros via build_flags.

1 Like

Thank you very much for your support and patience.

I will follow this.

So this will allow to override the base MQTT ESP-IDF components used by platformIO even if they have the same filename ? (as for example mqtt_client.h and mqtt_client.c that are different in their contents)

Well if it’s built into ESP-IDF, you don’t need to clone it all.

For ESP-IDF + Arduino projects, an older ESP-IDF was used (to match what was used when compiling Arduino-ESP32)

And I do see that in ESP-IDF 4.4.4

so… nothing else should be needed?