🌐 Understand the MQTT Protocol - all you need to know
What is MQTT?
The acronym MQTT stands for "Message Queuing Telemetry Transport". It is a lightweight communication protocol mainly used for machine-to-machine communication in the Internet of Things. It works with the publish-subscribe pattern and is built on the Transmission Control Protocol / Internet Protocol (TCP/IP).
IBM developed the protocol in 1999 to enable reliable data exchange in networks with low bandwidth and high latency. In 2013, OASIS (Organization for the Advancement of Structured Information Standards) officially defined MQTT as a standard for the Internet of Things. In 2016, the protocol became an ISO standard with the designation ISO/IEC 20922:2016.
MQTT is a transport protocol based on TCP/IP. This means that devices connected to each other send control packets to establish and terminate the connection, in addition to the actual data. TCP can also guarantee reliable delivery of data, by setting a quality of service level (Almheiri & Maamar, 2021). For communication, MQTT uses two ports reserved at the Internet Assigned Numbers Authority (IANA): TCP/IP port 1883 for unencrypted communication, and TCP/IP port 8883 for SSL-encrypted communication.
Publish Subscribe Pattern
The publish-subscribe pattern offers an alternative to the traditional client-server architecture. In a client-server architecture, the client communicates directly with an endpoint. In contrast, the sender and receiver in the publish-subscribe model are decoupled. This means they never communicate directly with each other. Instead, communication is facilitated by an intermediary, the broker (MQTT server). The broker's role is to filter incoming data traffic and forward it to the correct destination. In the MQTT protocol, senders are called publishers and receivers are called subscribers. Figure 1 illustrates a simplified MQTT communication. The right side of the figure shows the publisher, a temperature sensor in this case. It publishes a message "temperature: 22" to the topic "temperature". The broker receives this message and forwards it to all subscribers of the topic, in the figure this is the ESP32.
The most important aspect of this architecture is that decouple transmitter and receiver. This decoupling has several advantages: Receiver and sender do not have to know each other, so no IP addresses or ports have to be exchanged. Furthermore, sender and receiver do not have to run at the same time, because the broker can buffer sent messages.
The process of a MQTT communication
- A client device connects to an MQTT broker, which is a server that manages MQTT communication. The client establishes a connection with the broker using a TCP/IP connection.
- The client device subscribes to one or more MQTT topics. A topic is a logical channel that is used to route messages from the client to the broker and from the broker to the client.
- The client device publishes a message to an MQTT topic. The message is sent to the broker, which routes it to all clients that are subscribed to the topic.
- The broker sends the message to all subscribed clients. Each client receives the message and can process it as needed.
- The client device can also receive messages from the broker by subscribing to a topic. When a message is published to a topic that the client is subscribed to, the broker sends the message to the client.
Broker
The broker is the central platform for any MQTT communication. It stores and manages topics, subscribers, clients, IDs, etc. and sends messages. The downside is that if the broker crashes, communication in the network is no longer possible. However, it's possible to connect multiple brokers together, creating a redundant system (Hillar, 2017). The broker also handles authorization and authentication of clients, such as verifying the user ID at the application level. One of the most popular software solutions is the Eclipse Mosquitto Broker, an open source broker that can be installed for free on various distributions (Mosquitto, 2022).
Client
An MQTT client is any device (from a microcontroller to a full-fledged server) that runs an MQTT library and connects to an MQTT broker over a network (Hillar, 2017). The publish-subscribe architecture distinguishes between two types of clients. A client that sends a message to the broker is called a publisher. A client that receives messages from the broker is called a subscriber. The simple implementation of MQTT is the reason why client libraries are already available in many programming languages. Numerous client libraries also exist for the ESP32 microcontroller, one of the best known being the PubSubClient library by Patrick Lapointe (Lapointe, 2022).
Topics - Addressing
With MQTT, messages are published on so-called topics. These topics do not have to be created explicitly. If a client publishes something on a topic, the respective topic is automatically created in the broker. Each message is always assigned to a topic, the topic is included in the payload of the message. A topic consists of one or more topic-levels, the topic-levels are separated from each other by a slash (/), this is shown again in figure 2. The structure is similar to a hierarchical directory structure. Clients can publish something on a topic or subscribe to the topic to receive publication from other clients.
A requirement for a valid topic name is that it consists of at least one character. Even a single slash is a valid topic. Additionally a distinction between upper and lower case is made. Spaces within a topic are also taken into account and accepted by the broker, but should be avoided for readability reasons (Oasis, 2013).
Topic Wildcards
Wildcards allow a client to subscribe to multiple topic levels at once, without having to enter all of the levels individually. A wildcard is a placeholder for all possible topic levels. However, a wildcard can only be used to subscribe to topics, but not for publishing. There are two types of wildcards: single-level and multi- level.
Single-Level Wildcard
This is a wildcard which is placed between two levels. A plus sign is used as a symbol for the wildcard. The wildcard subscribes to all possible topics that exist at that level. An example of this is illustrated in Figure 3.
Multi-Level Wildcard
This is a wildcard that subscribes to all topic levels that come after it. The wildcard must be set as the last character for this. A hashtag (#) character is used as the symbol for the wildcard. Figure 4 shows the structure of the wildcard.
Connection establishment
MQTT uses TCP/IP to establish a connection from the client to the broker. For this, the client first sends a CONNECT packet to the broker. This packet can contain, among other things, a unique client ID, user name, password and a KeepAlive time specification. The broker responds with a CONNACK packet, which then also contains a "return code" among other things. This code indicates whether the connection to the broker was successful (OASIS standard, 2014). Figure 5 illustrates the structure of this connection again.
Publish, Subscribe and Unsubscribe
The publish packet is used to publish messages on the network. Publish packets can be sent from client to server or from server to client to transport application messages (OASIS standard, 2014). Each packet must contain a topic so that the broker can forward it accordingly. Typically, each packet also contains a so-called payload, which contains the actual message. However, MQTT is data-dependent, meaning that the sender decides whether to send binary data, text data, XML, or even JSON. A publish packet consists of several attributes (see Figure 6).
Subscribe package
To receive published messages, a client must subscribe to a topic. This is done by means of a subscribe packet. The packet is sent from the client to the broker to create one or more subscriptions. create one or more subscriptions. The content of the packet is kept simple, it contains a unique packet identifier and a list of subscriptions (OASIS standard, 2014).
The list of subscriptions of a package contains the respective QoSLevel in addition to the topic.
Unsubscribe package
The counterpart to the Subscribe packet is the Unsubscribe packet, it is used to delete a client's subscription in the broker. The packet is sent from the client to the broker and is very similar to a Subscribe packet. It also has a packet identifier and a list of subscriptions to be deleted. Only a QoS level is not sent here.
Quality of Service Level
Quality of Service, or QoS for short, describes the degree of reliability of a message to be sent. The ability to use QoS is one of the key features of MQTT, because QoS makes it possible to exchange messages reliably even in networks with low bandwidth and high latency. The MQTT protocol defines three different QoS levels 0, 1 and 2. At level zero, there is no guarantee that the message will reach the recipient. The message is only sent once and is ignored in further processing, which is why this level is also referred to as "fire-and-forget". At QoS level 1, a message is guaranteed to arrive at the recipient at least once. To ensure this, the sender stores the message until it receives a PUBACK packet from the receiver confirming receipt of the message. Here it is possible that a message is sent or delivered multiple times. QoS 2 guarantees that a message is received only once by the receiver.
Retained Messages
If a client subscribes to a topic on a broker, it would have to wait until another client publishes something on that topic. Before that, the subscribed client has no information or only outdated information. To counteract this problem, the MQTT protocol offers the possibility of sending retained messages. To do this, the publisher of a message must set the "retained flag" in the header of the packet to "true". If the message is now sent to the broker, the broker recognizes that the message should be saved. The broker thus stores the message including its QoS between. If a client now resubscribes to this topic, the broker immediately sends the message that is currently in its cache. If a publisher publishes a message with a retained flag set for a topic for which the broker has already stored a retained message, the old retained message in the broker is overwritten by the new one. This procedure also implies the possibility to delete a retained message. To achieve this, the publisher must publish a message with an empty payload, i.e. a message without data content, with the retained flag set. Using retained messages therefore makes sense if it is intended that newly connected subscribers receive a message immediately (Cope, 2020).
Persistente Session
If a client loses connection to a broker, the topics it subscribed to are lost and it would have to subscribe to them again when reconnecting. Re-subscribing on any connection loss can be a problem, especially for clients with low resources. To avoid this problem, the client can request a "Persistent Session" when it connects to the broker. The Persistent Session allows information to be stored on the server side beyond the connection period. To do this, the cleanSession flag is set to false when the client connects to the broker. To identify the session, the broker uses the clientId provided by the client when the connection is established. The broker then stores the following information:
- Session existence
- All subscriptions of the client
- All messages with QoS levels 1-2 that have not yet been acknowledged by the client
- All messages with QoS levels 1 and 2 that the client missed while it was offline and need to be forwarded
To terminate a persistent session, the client must set the cleanSession flag back to "true" (HiveMQ, 2022).
Last Will and Testament
Normally, a client disconnects from the broker by sending an MQTT DISCONNECT message. However, since MQTT is often used in scenarios with unreliable networks, it can be assumed that some of the MQTT clients occasionally disconnect inappropriately in these scenarios. Improper disconnection can occur due to loss of connection, dead batteries, or many other reasons. To work around the problem, MQTT has incorporated the Last Will and Testament (LWT) mechanism. This functionality sends a notification to other clients when a client loses a connection so that they can respond accordingly. Each client can specify its Last Will message when connecting to the broker. This message is a normal MQTT message, which is stored by the broker. If the broker now detects that the client has not disconnected properly, it sends the last-will message to all subscribed clients of the topic.
If the client properly disconnects with a correct DISCONNECT message, the broker discards the stored LWT message (HiveMQ, 2022).
Keep Alive
One problem that can occur in a communication is when one device has transmission problems or crashes while the second device remains active. In this problem is called a half-open connection. In a half-open connection, the working side of the connection continues to send messages, waiting for confirmation that they have been received, for example. This can go on forever if the device on the other end of the connection is not notified of the failure. For this reason, the Keep Alive function has been integrated into the MQTT protocol. The Keep Alive function defines a so-called Keep Alive interval. This interval is a numerical value that specifies the maximum number of seconds that a client can be inactive and thus still be considered functional by the broker. At these intervals, the client then sends the broker a PINGREQ packet to indicate that it is still available. The broker responds to this packet with a PINGRESP message to the client, so the client knows that the broker is also still available. If the broker does not receive a PINGREQ packet from the client during the interval, the broker terminates the connection to the client. Setting the time to zero disables the keep alive functionality.
Short Summary
MQTT (Message Queue Telemetry Transport) is a lightweight communication protocol that is used for machine-to-machine communication in the Internet of Things (IoT). It is built on the TCP/IP protocol and uses the publish-subscribe pattern for communication. In MQTT, devices communicate through an intermediary called a broker, which receives messages from publishers and routes them to subscribers based on their subscriptions to specific topics. MQTT allows for decoupled communication between sender and receiver, meaning that sender and receiver do not have to be connected at the same time, and do not have to know each other's IP addresses or ports. The broker also handles authorization and authentication of clients, and can support multiple brokers for redundant communication. MQTT is often used in low-bandwidth networks and is useful for sending and receiving small amounts of data.