MQ Telemetry Transport (Part 1)

Going Aloft with Mosquitto

Better known by the acronym MQTT, this lightweight messaging protocol is designed to minimize network bandwidth and device resource requirements. In this article, Jeff sets out to use MQTT via a cloud setup that he can do locally. For this, he turns to Eclipse Mosquitto, an open source message broker that implements the MQTT protocol. Jeff steps through the nitty gritty details of his implementation.

Message Queuing Telemetry Transport. You probably know it as MQTT. It’s a lightweight messaging protocol designed to minimize network bandwidth and device resource requirements. Its publish/subscribe protocol allows a server to handle all data dissemination. In its simplest form, the server can log data published from an external device, called a “node.” A node requires minimal code to packetize its data before sending it to the MQTT server. Multiple nodes can send their data to the same MQTT server. The server is capable of handling a large number of nodes or clients. Packets describe how the enclosed data payload should be handled. Today, there are 16 different types of MQTT command packets. In the simplest system, we may only need three of these—Connect, Disconnect and Publish.

MQTT packets use the Transmission Control Protocol/Internet Protocol (TCP/IP) for communications. The Internet Protocol suite uses ports to identify a network service. Your HTTP Internet activity is most likely directed to port 80. Most MQTT activity uses port 1883. The use of ports simplifies the distribution packets. The Connect and Disconnect packets let the MQTT server know who can be expected to require attention. The server will disregard any packets from clients that have not registered or those who have unregistered. The Publish packet holds the actual data that the server must deal with. If one or more clients only publishes data, then the data aren’t used. There must be at least one subscriber to these data for any real work to be accomplished. Subscribing is the fourth packet type that makes this all worthwhile.

In many cases you will want to record the published data to a log file. This requires a client to subscribe to the same data being published by Client 1. Why go through all of this bending over backward just to log data that could be handled locally on the external node? If that simple scenario handles your project, then I’d take that tack. However, if you look at a bigger picture here, I’ll bet you have more than the one IoT project. You may be using ThingSpeak, or any of the other cloud managers to retrieve and store data from your project. While these are convenient and may have some free storage, many require a monthly maintenance fee. I’m looking for a cloud I can implement locally. Enter Mosquitto.

MOSQUITTO, AN MQTT BROKER
Eclipse Mosquitto is an open source (EPL/EDL-licensed) message broker that implements the MQTT protocol. Many brokers provide cloud-based testing for your project, including test.mosquitto.org. It is easy to move from an external cloud broker to a locally run broker. I have installed Mosquitto on one of my PCs. Mosquitto is available for Windows, Linux, Mac and Raspberry Pi. You can use external testing to get comfortable with the processes discussed here.

Let’s spend a bit of time on the MQTT protocol before installing a simple client device. The MQTT protocol is communication of topics between a client and a broker. The broker (or server) handles the collection and dissemination of topic data. While multiple brokers can be connected, just a single broker is required for most systems. Clients are devices that communicate with a broker. A client may collect sensor data (for example, topic = switch status, topic = temperature, topic = position) and pass it on to the broker (publish a topic). A client may receive particular sensor data (subscribe to a topic) and perform some function (control a digital or analog output) based the topic’s data. A client can be a publisher and/or a subscriber.

Unlike HTTP, which opens and closes communication for each request, the MQTT Broker controls opening and closing a connection over time. Each MQTT packet contains at least two parts—a fixed header and a variable header. Certain commands require a third part—the payload. Table 1 shows these, with a brief description of each.

Packets have three levels of confirmation. A QOS “0” (Quality of Service, level zero) request receives no acknowledgement. A QOS “1” request receives an acknowledgement. A QOS “2” request receives an acknowledgement plus additional packets to acknowledge the acknowledgement! You can choose your level of confirmation based on your requirements. Besides the QOS, a publish packet contains several other flags that can be useful. The Retention flag requires the broker to hold onto this topic’s data after it finishes publishing it to any clients that have subscribed to the topic. This allows new subscribers to get the last known state of the topic whenever they subscribe. The Will (as in “last will and testament”) flag requires the broker to advise all subscribers that the publish client has left or disconnected. Figure 1 shows typical MQTT communications between the clients and the broker.

SIMPLE DEMO
About a year ago, I presented a two-part project entitled “ Texting and IoT Embedded Devices,” using an Espressif Systems ESP8266 to monitor the state of an input and send a text to my phone when a change occurred (Circuit Cellar 332 and 333, March and April 2018). I’ll use the same circuit here as clients communicating with a Mosquitto broker. The first will be the input client and a second the output client (Figure 2). I like using the ESP8266 because it can be programmed as if it were an Arduino using the Arduino IDE. You can use the pubsubclient library to handle all the necessary packet issues. The ESP-01 has one I/O that can be used if you exclude TX, RX, enable, reset, power and ground. Client1 will use IO2 as a (switch) input. Inputs can have an internal pull-up resistor enabled, which will hold the input in a high state. In this example, I’ll be using a toggle switch placed between the input and ground. When the switch is open, the input is at a logic high, and when it is closed, it will short the input to ground, applying a logic low to the input. Client 1 will monitor the input state and report each change of state.

The broker uses “topics” to help compartmentalize different data messages. Each topic can use a tree structure of folders. For instance, my Client 1 node is an ESP8266 with a MAC of “xxxxxx11E9B0,” so I’m using this as the top folder. I’m using one subfolder to divide the client by pin function, and another to narrow this function down to a single pin number. The topic will therefore become “ESP_11E9B0/DIN/IO2”. This is all up to the user—I could have simply named this topic “Client1.” Client 1 will publish a message (the status of the input of IO2) using this topic. In this case, the status is simply the logic state of the input, a “1” if the input is high, and a “0” if the input is low. Sending a binary 1 or 0 is fine, but you won’t be able to “see” it (if you look at a data transmission or log it to a file), because these are non-printable characters. I chose to make them ASCII characters by adding 0x30. This means the message is either character 1 or 0, rather than binary 1 or 0. Messages can be in any format—even JSON (JavaScript Object Notation), if you wish.

CLIENT 1
Let’s look at the highlights of the Client 1 program written on the Arduino IDE for the ESP8266. The first thing we need to do is include the necessary library files and define the network information as shown in Listing 1.

#include <ESP8266WiFi.h>
#include <PubSubClient.h>
//**************************************
// Network info
//**************************************
const char* ssid = “your network”;
const char* password = “network password”;
const char* mqtt_server = “192.168.1.216”; // pc=216, openhanian=194
const char* switchTopic = “ESP_11E9B0/DIN/IO2”;
//const char* lightTopic = “ESP_A3D0DF/DOUT/IO2”;

LISTING 1 – This code includes the necessary library files and defines the network information.

The MQTT_server is Mosquitto running on my PC, so this address is my PC’s IP address on my network. Note: The alternate IP listed here is for openHAB’s openHABian, an open source home automation program for a Raspberry PI. This also has an MQTT broker built in, which means all your MQTT communications are accessible to it. For simplicity, we’ll stick to using the Mosquitto broker installed on my PC. Next you see the definition of the switchTopic (Client 1), so we can publish. The program requires some initial setup (Listing 2).

//**************************************
// setup
//**************************************
void setup()
{
pinMode(switchPin, INPUT_PULLUP);
// Initialize IO@ as a switch input using an internal pull up resistor
switchState = digitalRead(switchPin);

// Initial state of the switch
msg[0] = (char)digitalRead(switchPin) + 0x30;
// Initial message
setup_wifi();
client.setServer(mqtt_server, 1883);
client.setCallback(callback);
}

LISTING 2 – Some initial setup code including setting input pin mode and the using callback function.

The mode of the input pin is defined and the state of the input is saved as the first message. The setup_wifi() function connects to your network. The client.setServer(mqtt_server, 1883) function uses the pubsubclient library to connect to Mosquitto (which must be running on your PC) at port 1883, the standard port for the MQTT protocol over TCP. The last function, client.setCallback(callback), lets them pubsubclient library know that it must use the user’s callback function if it receives any data. What is this callback function? This is only used when a client subscribes to a topic. For now, Client 1 is only publishing, so let’s move on to the MQTT connection routine.

Previously we set up the Wi-Fi and MQTT server info. Now we need to connect to the server. This function will also be called to reestablish the MQTT connection, should it ever be lost. Therefore, I’m naming it reconnect(), and this is shown in Listing 3. To make sure this device is easily identified, I’m using part of the client’s MAC address as its client ID. Once a connection is established, the switch state is sampled, a message is created, and the initial switch state is published to the broker.

//**************************************
// MQTT reconnect
//**************************************
void reconnect()
{
// Loop until we’re reconnected
while (!client.connected())
{
// Create a client ID
char* clientId = “ESP8266Client-11E9B0”;
// Attempt to connect
if (client.connect(clientId))
{
switchState = digitalRead(switchPin);
// publish
msg[0] = switchState + 0x30;
client.publish(switchTopic, msg);
// resubscribe
client.subscribe(lightTopic);
}
else
{
// Wait 5 seconds before retrying
delay(5000);
}
}
}

LISTING 3 – The reconnect() function, shown here, connects to the server. This function will also be called to reestablish the MQTT connection, should it ever be lost.

And now we move on to the main loop, shown in Listing 4. Two basic things are happening here. The first monitors the MQTT connection status. If we only publish when a change occurs, then there may not be any transmissions for a while. The original connection request carries with it a socket timeout, and if the broker doesn’t hear from the client within this period, then the broker initiates a Ping to see if the client is still there. As long as the client answers, the connection stays open (unless the client specifically requests a pubrel). Client.loop() must be called periodically, so the main loop gives up control, allowing the client to process commands received from the broker.

//**************************************
// main loop
//**************************************
void loop()
{
if (!client.connected())
{
reconnect();
}
client.loop();
//
newState = digitalRead(switchPin);
if(switchState!=newState)
{
switchState = newState;
// publish
msg[0] = (char)switchState + 0x30;
client.publish(switchTopic, msg);
}
}

LISTING 4 – Code for main loop is shown here.

The last part of the main loop monitors the input and upon a change in state publishes this change to the broker. The client.publish(switchTopic, msg) command requires two parts, the topic and message. Here the topic is “ESP_11E9B0/DIN/IO2” and the message is the character 0 when OFF and 1 when ON.

The MQTT broker creates a temporary record of each topic’s last message. If any client subscribes to it, it will (re)publish the message to that client. You don’t need to do anything to the broker (Mosquitto) to make this happen, since this is the broker’s intended purpose.

CLIENT 2
The program for Client 2 is basically the same as Client 1. We are including the same library files and defining the same network information. The setup() function (Listing 5) is similar to that of Client 1, except we are defining IO2 as an output. Because Client 2 will be subscribing to Client 1’s publish topic, the MQTT reconnect() function will differ slightly, as shown in Listing 6.

//**************************************
// setup
//**************************************
void setup()
{
pinMode(lightPin, OUTPUT);

// Initialize IO2 as an output
setup_wifi();
client.setServer(mqtt_server, 1883);
client.setCallback(callback);
}

LISTING 5 – Shown here, the setup() function is similar to that of Client 1, except we are defining IO2 as an output.

/**************************************
// MQTT reconnect
//**************************************
void reconnect()
{
// Loop until we’re reconnected
while (!client.connected())
{
// Create a client ID
char* clientId = “ESP8266Client-11E9B0”;
// Attempt to connect
if (client.connect(clientId))
{
client.subscribe(switchTopic);
}
else
{
// Wait 5 seconds before retrying
delay(5000);
}
}
}

LISTING 6 – Client 2 will be subscribing to Client 1’s publish topic, therefore the MQTT reconnect() function will differ slightly, as shown here.

This is a good time to discuss the callback() function, since we’ve just requested the broker to pass on any information it receives on the switchTopic topic “ESP_11E9B0/DIN/IO2”. Originally, Client 1 published the state of the IO2 input using the function, client.publish(switchTopic,msg). This is passed on to Client 2, which is directed by the client.setCalback(callback) function to use the callback() function, shown in Listing 7.

/**************************************
// MQTT reconnect
//**************************************
void reconnect()
{
// Loop until we’re reconnected
while (!client.connected())
{
// Create a client ID
char* clientId = “ESP8266Client-11E9B0”;
// Attempt to connect
if (client.connect(clientId))
{
client.subscribe(switchTopic);
}
else
{
// Wait 5 seconds before retrying
delay(5000);
}
}
}

LISTING 7 – This code shows the callback() function for subscriptions.

This function should be used to select a subscribed topic. Since we have only one topic here, we can simply look at the first character of the payload (received message). We know this will be a single ASCII character consisting of a 0 or a 1. The value of this character is assumed to be the state of Client 1’s switch. The switchState variable keeps track of this information.

Now on to Client 2’s main loop, shown in Listing 8. The first part of the main loop takes care of the MQTT connection and is identical to that of Client 1. The last part of the main loop handles setting and clearing the (light) output. My circuitry has an LED that follows the state of IO2. It reflects the state of the output of Client 2 and the input of Client 1, so it can be used to monitor the actual state of IO2 on both clients.

//**************************************
// MQTT loop
//**************************************
void loop()
{
if (!client.connected())
{
reconnect();
}
client.loop();
if(switchState!=lightState)
{
lightState = switchState;
if(lightState == 1)
{
digitalWrite(lightPin, HIGH); // Turn the port on
}
else
{
digitalWrite(lightPin, LOW); // Turn the port off
}
}
}

LISTING 8 – This shows the code for the main loop for Client 2.

We can call this “open loop” because there is no direct feedback that Client 1’s switch state has actually affected Client 2’s output. However, we are using “QOS”=2 for all this communication, which means the communication between Client 1 and the broker is assured via command handshaking. The communication between the broker and Client 2 also implements this. I can see this demonstrated once in a while, when Client 2’s output fails to follow Client 1 immediately. It will, in fact, correct itself after a short delay.

Should the need arise, you can easily “close the loop” by having Client 2 publish a topic with the state of its output, and have Client 1 subscribe to this topic. If you look back at the Network info section in Listing 1, you find the rem’d out statement //const char* lightTopic = “ESP_A3D0DF/DOUT/IO2”. Remove the rem slashes, and a new topic will be defined. Client 2 can then publish to this topic, and Client 1 can subscribe to it. This will pass back to Client 1 the state of Client 2’s output, which can be used to confirm that Client 2’s output state is the same as Client 1’s input state.

MOSQUITTO
Go to mosquitto.org/download and follow the directions if you want to install Mosquitto on your PC. Mosquitto can be set as a service that is automatically run whenever your PC boots. Alternately, you can run it from the command line. If run from the command line using the mosquitto -v option, you’ll get a feeling for what’s going on in real time (Figure 3).

Unfortunately, Mosquitto does not supported logging to a file on the Windows system. We can get around this by adding the Node-RED application from www.nodered.org to your PC. Node-RED is a browser-based programming tool that looks like a client to a broker. In its simplest form, it can subscribe to an MQTT topic and log the topic to a data file. I’ll use Node-RED here for local logging the publishes of Client 1 and Client 2. The file will record all activity of the clients.

If you want to install Node-RED on your PC follow the directions at nodejs.org/en. Node-RED can’t be run as a service, so you can use the task scheduler to add this to the start-up procedure. If both Mosquitto and Node-RED are running, type “localhost:1880” into any browser, and you will be able to add a local client to work with Mosquitto. Figure 4 shows how easy it is to drag and drop tasks to subscribe to a topic, get the present date stamp, merge the two data streams and log the result to a file.

Note that the two tasks to the right in Figure 4 are taps into the data stream and allow the payloads (messages) to be printed to the debug screen. You can therefore see both the raw message and the message appended to the date/time stamp. This flow subscribes to any topic (“#”) that begins with “ESP_11E9B0” (Client 1). Each message is appended to the file “data.txt” in the “MQTT\ESP_11E9B0\DIN\IO2” on my desktop. Another set of tasks can be added to subscribe to any topic (“#”) that begins with “ESP_A3D0DF” (Client 2). The entries shown in Listing 9 are from the two logs files showing both the actions of Client 1 (switch position) and Client 2 (light status).

from C:\Users\jeff\Desktop\MQTT\ESP_11E9B0\DIN\IO2 data.txt
Thu Jun 06 2019,7:27:16,0
Thu Jun 06 2019,7:28:11,1
Thu Jun 06 2019,7:28:41,0
Thu Jun 06 2019,7:28:46,1

from C:\Users\jeff\Desktop\MQTT\ESP_A3D0DF\DOUT\IO2 data.txt
Thu Jun 06 2019,7:27:16,0
Thu Jun 06 2019,7:28:11,1
Thu Jun 06 2019,7:28:41,0
Thu Jun 06 2019,7:28:46,1

LISTING 9 – The entries shown in shown here are from the two logs files showing both the actions of Client 1 (switch position) and Client 2 (light status).

You might not require a log file of the operations of this simple two-node circuit. However, these basics form a strong foundation for next month’s article, which shows how I moved my IoT from an external cloud (ThingSpeak) to my local MQTT server.

MQTT has great flexibility. Multiple nodes can publish to the same topic, and multiple nodes also can subscribe to the same topic. If I want to control a light from multiple locations, I would use a momentary push button instead of a toggle switch with an LED, to indicate the present status of the light. This way any node could be used to toggle the output. Similarly, multiple lights, such as front and back porch lights, could be controlled from multiple locations if they both subscribe to the same topic.

So much to learn, so little time.

Note: Code for this project can be downloaded from Circuit Cellar’s article code & files webpage.

Read Part 2 Here

Additional materials from the author are available at:
www.circuitcellar.com/article-materials

RESOURCES
Espressif Systems | www.espressif.com
Mosquitto | www.mosquitto.org
MQTT | www.mqtt.org
Node-RED | www.nodered.org
openHAB | www.openhab.org
ThingSpeak | www.thingspeak.com

PUBLISHED IN CIRCUIT CELLAR MAGAZINE • OCTOBER 2019 #351 – Get a PDF of the issue

Sponsor this Article