Bringing it All Back Home
In Part 1, Jeff described the MQTT protocol and how it is used by an MQTT server to keep all your IoT projects tied together and managed from a centralized server running a program such as Mosquitto on a local PC. He presented a simple project connecting two IoT nodes together via communication with the server. In Part 2, Jeff looks at modifying systems he uses to monitor his neighborhood well system and his weather station, for integration into the MQTT server.
At a time when many companies are moving their storage off site, I am a believer in keeping it close. By that I mean under my own roof. As computing is becoming faster and more efficient, storage media is also growing in capacity. Today, gigabyte thumb drives and terabyte hard drives are becoming the norm. When I lose data because I made a mistake or a bad decision, I find it much easier to live with than if all my stuff were to suddenly disappear from the cloud. When things are out of my control, I feel I have no recourse. That’s why I am attempting to bring all my needs back under my control.
In Part 1 last month I discussed the MQTT (Message Queuing Telemetry Transport) protocol and introduced a project with two IoT clients. Client 1 monitors a digital input and “Publishes“ the new state as a “topic” message to the server whenever its input changes state. Client 2 controls a light switch on its digital output and “Subscribes” to Client 1’s topic on the server. Whenever a new topic is Published to the server, all nodes that have Subscribed to that topic get sent the updated message. Client 2 monitors the messages on this topic and sets/clears the digital output based on the message. In this case, I defined Client 1’s topic as “ESP_11E9B0\DIN\IO2\” and the message as an ASC “1” (ON) or “0” (OFF).
While not necessary, I also included the topic “ESP_A3D0DF\DOUT\IO2\” so that Client 2 could Publish the state of its output, and Client 1 could Subscribe to it, to see that the operation was actually completed. While this required two nodes to establish the remote control of a light, the real intention was to show that by shuffling communication through a server—as opposed to just having these two nodes talk to each other directly—the information could be recorded and stored for posterity. You may not care when a light was switched ON and OFF, but what if these were bank transactions or an alarm of some kind? Let’s not forget the potential “bigger picture.”
In the past, I’ve taken a number of different approaches on user interfaces (UIs). For example, I monitor my neighborhood’s well system using ThingSpeak. I have also described a graphics server using HTML5 for my weather station. This month, we’ll look into modifying these two projects for integration into the MQTT server.
WELL ENOUGH ALONE
My neighborhood’s well supplies water for seven families. A deep well pump supplies water to a 360 gal. holding tank. We’re fortunate that the water is potable from the pump, despite a high concentration of iron. Once the suspended iron is exposed to air, it oxidizes and gives the water a rusty color. This is visually unpleasant to drink, and creates rust stains on washed clothing. However, a pair of water softeners remove most of the iron before it gets to the main line that serves the neighborhood.
— ADVERTISMENT—
—Advertise Here—
The softeners periodically require salt as a rinsing agent to rejuvenate (clean) the resin media that collect iron particles. Salt pellets must be added to an external tank, which contains a brine of water mixed with the salt. The brine is drawn by the softeners during their cleaning cycle, and the brine tank is refilled with water. A portion of the salt pellets dissolve during each cycle, until the water becomes saturated. Therefore, the salt lasts several cleaning cycles before it is used up. If the salt is not refilled, the resin is not cleaned and cannot remove any additional iron, giving everyone tinted water. Letting the salt disappear is a no-no, and should be avoided—hence the need for monitoring.
The well monitoring system has seven inputs. The first three are temperature sensors. I take the outside temperature and two inside temperatures—of the upper and lower well house. The well house is an 8’ cube of concrete blocks below ground level. A 3’ wall and roof above ground give access via ladder to the equipment. The upper temperature is in the roof area, and the lower temperature is below ground at floor level. It is interesting to see how wildly the outside and upper temperature vary, whereas the lower temperature hangs around 50°F until winter. With some winter days below zero, temperature becomes an issue. A frozen pipe means no water and the potential of a burst pipe, which has already happened once. Should the temperature fall too low at the lower sensor, a heater can be turned on. Currently, an incandescent light bulb throws enough heat to raise the temperature slightly.
The fourth and fifth sensors are attached to the water softeners. They are paddle-wheel flow sensors used by the softener’s electronics to calculate—based on water usage—when the resin media should be cleaned. They produce a tick for each 6.4 oz. (20/gal.) of fluid. Monitoring the usage can bring to light a number of issues. For instance, when cleaning cycles are activated after midnight—so salt pellets can be added, or the relationship of water usage between softeners—monitoring can help to determine if other maintenance is required.
The sixth sensor measures the pressure in the storage tank. A regulator on the tank turns the pump on when the pressure falls below 40 psi, and turns the pump off when the pressure reaches 60 psi. This can help determine the pump’s efficiency, since it does eventually get clogged with iron deposits.
The seventh sensor is a current probe around the pump lead that indicates when the pump cycles. For longest life, you want the pump to cycle as little as possible. Accordingly, you want to know if the pump starts running too long and isn’t refilling the storage tank. The storage tank is actually pressurized with air above the water, which pushes the water out of the tank when a faucet is turned on. A storage tank with no bladder, or separator between the water in the lower part of the tank and the air in the upper part of the tank, will eventually absorb the air above the water. When this happens the air’s volume at 60 psi will be less than before. This creates more room for water, but also decreases the volume of water that is available to leave the tank before the tank pressure drops to 40 psi. The pump will operate more often for shorter lengths of time—a bad thing. With that in mind, it’s important to maximize draw-down by keeping adequate air in the tank. Most tanks have a bladder to prevent this from happening, but our tank has long since lost its bladder to old age.
MQTT MOSQUITTO
Last month, I began running Eclipse Mosquitto, an open source message broker, on my PC. It turns my PC into a MQTT server that runs in the background. I can reach it through my local LAN/WAN. It will accept MQTT Publish and Subscribe messages sent by any device on my network. All MQTT messages contain a topic and a payload. Messages are categorized by topics such as “Temperature,” with an associated “payload” of some value. Publishers create the data, whereas subscribers consume the data. The MQTT server keeps track of the published topics and sends them to any device that has subscribed to that topic. To learn more about this, please refer to last month’s Part 1 article (Circuit Cellar 351, October 2019).
Getting my well-house device to send MQTT data is pretty straightforward. This application has been using ThingSpeak since 2006. The Arduino library uses TCP over HTTP to communicate with the ThingSpeak server, whereas MQTT uses UDP over HTTP. TCP is connection-based and remains open until closed. UDP is connection-less—that is, each communication is complete in and of itself. In my application, I need to substitute include WiFiUDP.h for include ThingSpeak.h. The Wi-Fi connection remains the same. In last month’s article, we found that the MQTT communications begin with making contact with the MQTT server (at its IPAddress:1883) to establish a link between the two, but not an actual connection. This was discussed and the code was shown.
Round Robin
In this application, the sensors are read round-robin style—one every second. This keeps their values updated with the latest information. You can choose how often you want this information to be sent. Currently, I send it all once per minute. As noted in Part 1, when we want to share information with others, we Publish it to the MQTT server. The following line of code accomplishes this, once we’ve defined the topic and msg.
client.publish(topic, msg);
— ADVERTISMENT—
—Advertise Here—
This next chunk of code is a special case message I use to indicate that the device has just come out of reset. The variable reboot, which starts as true, gets cleared to false after the message has been sent once.
// reboot
if(reboot)
{
t=” reboot”;
t.toCharArray(msg, msgSize);
t = ID;
t.toCharArray(topic, msgSize);
//Serial.print(t);
sendMQTT();
reboot = false;
}
Note that the msg is set to the string reboot and the topic to the “ID” of the device—in this case ID = “esp8266_A14782”, which is its friendly name (including the last 6 characters of the MAC).
Here’s are the messages this device sends:
Topic, Message
ID + “/Revision”, ”Well House Monitor myMQTT 6/21/2019 “
ID + “/Temperature/Outside”, 78
ID + “/Temperature/Attic”, 72
ID + “/Temperature/Cellar”, 56
ID + “/Flow/Softner1”, 27
ID + “/Flow/Softner2”, 20
ID + “/Pressure/Tank”, 45
ID + “/Amps/Pump”, 0
ID + “/Relay”, 0
I always include the ID as the beginning of the topic, so that I can identify its origin. Last month, the project used two devices. The 1st published a switch’s status, and the 2nd subscribed to that topic and controlled a light based on switch status. Mosquitto handles all of this without intervention.
Node-RED is a companion application that allows you visually to connect devices. In Part 1, I detailed how it can be used to subscribe to an MQTT topic and do something with the messages received. In that case, all messages were sent to a log file. Other than viewing the file’s data, there was no visual indication of any device activity. Now let’s see how Node-RED can be used to display the data from the well house.
Node-RED EDITOR
Node-RED provides a browser-based editor for wiring together hardware devices. The node-RED editor is available through your browser at IPAddress:1880. The editor window consists of four components: palette (left), sidebar (right), header (top) and workspace (middle). The palette contains the nodes (objects with possible inputs and/or outputs) available for placement on the workspace. The sidebar contains information about objects to be displayed. The header has a button for deploying a flow (interconnection of objects) and additional menu items. The workspace is where you drag and drop objects, and interconnect them using wires to tie inputs to outputs. Figure 1 shows my browser with three nodes placed, wired and deployed. The sidebar shows the debug output from the debug node as messages are received.
Once you become familiar with all the nodes in the palette, you’ll see that subscribing to an MQTT topic (MQTT input) is just one way of using Node-RED. This project will create a “dashboard” consisting of mainly graphs of the sensor data over time. Let’s look at creating text, gauge and chart displays for one temperature message. Refer to Figure 2 for this discussion.
Our tab FTB 352 shows four nodes added to the workspace. The first node (MQTT input) passes only the message payloads with a topic containing “esp8266_A14782/Temperature/Outside” as defined by editing the node. The next three nodes are from the dashboard in the node’s palette. They are used to display the message payload in text, as a value on a gauge and as a charted value over time. Because the payloads are all strings, the test display includes the payload just as it comes. You may have noticed an “F” (Fahrenheit) was part of this. In the gauge and chart displays, convert the string to a value first, then use the value for displaying.
Some parameters—such as max and min—can be edited in these nodes. Since I have three temperatures, all of them can be wired to the same chart display. With a small bit of massaging we can remove (change node) part of the topic that is used in the chart’s key for identifying the different data lines (Figure 3). The change node is used to set, change, delete or move specific parts of a message, flow or global object. Here specific message topics are truncated. In other words, esp8266_A14782/Temperature/Outside becomes Outside.
The water softeners measure water throughput by ticks (counts) of their paddle wheels. Total counts are sent as messages to be used as data for the chart. Before sending the count data to the chart, I use a function node to alter this. All nodes pass their message to their output, which is then received by the next “wired” node’s input. The function mode allows some JavaScript to be written that can alter this message in some way. Here, I want to convert counts to gallons. The water softener documentation states that the paddle wheel creates 20 counts for each gallon of water that passes through it.
msg.payload = (parseInt(msg.payload) / 20);
return msg;
In a perfect world, the two softeners’ counts would share the flow equally, and their charts would be identical. If displayed on a single chart, the first softener’s data would always be obscured by the second. I chose to display these as separate charts. However, I also want to keep a running daily tally of the total gallons per day. These data aren’t given by the well-house node. Node-RED allows me to total the data and reset the value to zero at the end of each day. Take a look at the following JavaScript code:
var myFlow = flow.get(‘Flow1’) || 0;
myFlow = myFlow + msg.payload;
flow.set(‘Flow1’,myFlow);
msg.payload = myFlow.toFixed(3);
return msg;
— ADVERTISMENT—
—Advertise Here—
I create a local variable myFlow (available to just this node), and assign it the value of a flow variable Flow1 (available from any node within this flow or page on my workspace). The saved value is added to the present message’s value. The total is stored away for next time. Finally, the message sent out is replaced by this new value—fixed at three decimal places. This value updates a text box with Today’s total, but we still need a way to zero out this total every night and chart the daily totals.
This is created with three nodes: “inject”, “function” and “chart.” The “inject” node allows something to happen at a specific time or periodically. I’ve chosen once every day at 11:59 p.m.
var myFlow = flow.get(‘Flow1’) || 0;
var msg1 = {payload:myFlow.toFixed(3)};
msg1.topic = “DailyFlow1”;
flow.set(‘Flow1’,0);
return [msg1];
Another “function” node is used to do this. I create a local variable myFlow (available to this node only) and assign it the value of a flow variable Flow1 (available thanks to the node described above). Now I create a whole new message with a payload of myFlow and a topic of DailyFlow1. The Flow1 variable is then cleared so it can begin with a new total, Tomorrow, which begins in 1 minute. The new message is sent on to the chart node. See the final display “Flow” in Figure 4 and “Dashboard” in Figure 5.
WEATHER REPORT
I’ve had a weather station in various forms prior to the smartphone era, informing me of tomorrow’s weather today. While I’m not a HTML guy, one of the most interesting articles for me was my weather server project that handled its own HTML display. The two part article is “Serving Up HTML (Parts 1 and 2) in June and July 2016 (Circuit Cellar 311 and 312). In keeping with the MQTT theme here, I’ve re-coded that project and have Node-RED code for a new weather display. One of the advantages of using Node-RED for a project is its extensive palette of the available nodes. For instance, the “inject” node lets you simulate the reception of messages as if your external device had Published data.
Here, I use the “switch” node to separate each topic into its own path, which is wired directly to a gauge or text output. Each gauge has three ranges associated with it. These ranges can be color coded, and will display that color depending on the color range settings and the actual payload being displayed. If the range colors are defined but range settings are not, then the color displayed will be a blend of the defined colors proportional to the actual payload. If max and min values are not defined, then the gauge will auto-range based on the payload values.
You might want to separate real-time data (gauges), as in Figure 6, from statistical data (text) on separate flows. Each flow will be offered as tabs on the display page of Node-RED. Clicking on the menu icon on the left side of the blue bar at the top of the UI display page reveals additional “flows” that you have created (Figure 7). You can reach every flow from this one page! The weather station provides plenty of statistical data. It has hourly, weekly, monthly and/or yearly averages for almost every sensor. These might all be displayed from an alternate flow.
SUNNY DAYS
At this time, I have neither solar panels covering my roof, nor a huge solar matrix in the yard. I have a few small panels affixed to my shed roof so I can dabble. But so far I’m not overwhelmed by their energy production. They receive no direct sun in the mornings and late afternoons, so their production is limited. This produces just about enough energy to keep the weather station and solar reporting nodes alive 24/7. The MQTT server also handles the solar node.
The solar charge controller from Epsolar Technology has an RS-485 (ModBus) interface to access its more than 100 registers. These are separated into logical sections: Rated Datum, Real-time Datum, Real-time Status, Statistical Parameters, Holding Registers, Switch Value and Discrete Value. My solar monitor circuitry must poll the charge controller to gather copies of all the registers and publish them to the MQTT server. While the server collects all the registers, I only use a fraction of these to give a real-time display of energy produced, stored and consumed.
The upper-level folder name for all topics from the solar controller node use esp_82952A to identify the device. This topic is appended with the section name and register name as additional folders to give a hierarchy to the topics published. In this Node-RED flow, I subscribe to all esp_82952A# topics. A switch node is used to filter all eight topics by section name. The flow is therefore divided into eight sub-flows. Don’t confuse my term “sub-flow” with Node-RED’s sub-flow. More on this later. Each of these could be sub divided again with eight additional switch nodes, to produce a separate sub-flow for each register in the charge controller. To illustrate this, I am using three of these eight sections, and dividing the three sections into their respective individual registers with switch nodes.
The Real-time Datum section has sub-flows for 10 of the 15 registers. The Statistical Parameters section has sub-flows for seven of the 21 registers. The Discrete Inputs section has sub-flows for both of its two registers. As you can see from the Node-RED editor’s workspace in Figure 8, the diagram is getting quite complicated. For ease of viewing, you can designate portions of the flow to sub-flows, and move them onto separate pages (workspaces), which can then be identified on the flow’s workspace as separate sub-flows, thus simplifying the workspace. However, by doing that you lose your ability to debug each sub-flow because they no longer produce output in the debug window. For that reason, I have not used sub-flows in any of my flows.
My use of the term “sub-flow” has to do with dividing each esp_82952A# topic (they all come in through the same input wire) to the switch node labeled esp_82952A# and are separated into section topic outputs (sub-flows). Each of these is then input into additional switch nodes to further separate the topic into individual outputs (sub-flows) containing just one register
The display of data here is a combination of text, charts and LEDs. You’ll notice there is no LED node in the palette dashboard section. Another great aspect of Node-RED is the palette manager, which allows you to import nodes designed by third parties or create your own. In this case, I imported the node-red-contrib-ui-led node to use in this flow. Although an LED is considered a binary device, its color can be set based on different payloads, binary, other numerical value, string, JSON or buffer. This is similar to most other nodes. For instance, I use yellow for daytime and black for night time. In this case, day/night is a Boolean value, but it could have been the strings yellow or black.
I was caught by surprise by the data I was seeing from the Battery Current register, shown in Figure 9. If I had thought about this a bit more it would have been obvious! The PV Array produces energy. The load consumes energy. The battery both produces and consumes energy. When it consumes energy, it is charging (current is positive). However, it also produces energy (current is negative) once the PV array is no longer producing. Therefore, the Battery chart must show current in both directions. I wanted this to be obvious, by using a green trace when zero or positive and red when negative. I did this by dividing this sub-flow (esp8266_82952A/Statistical Parameters/Battery Current LSW) into two sub-flows. Sub-flow topic positive, when the payload is >=0, else sub-flow topic negative. Both of these sub flows go to the same chart input. The chart plots both as separate topics, positive as green and negative as red. The chart key is disabled, so the topics are not defined in the chart.
HOSTING MQTT
Right from the get-go, I used my PC as the MQTT host. Just recently, I installed openHAB 2 on a Raspberry Pi. This is an open source, home-automation platform, which runs as the center of a Smart Home. My thinking is that openHAB 2 supports MQTT (and Node-RED) which I could use as a permanent location for my MQTT server. The weather and solar nodes are operating on the Pi (running Linux OS). Although I have not played with openHAB 2 at all, the MQTT link allows me to find a path that supports my work in the present, while lighting my path into the future. I’ll be changing the subject next month, but if you are interested in this, please let me know so I can make plans for a future article.
Oh yeah, one more note on using Node-RED and MQTT that was installed with openHAB 2 on the Raspberry Pi: The Pi palette has some additional nodes listed (Figure 10). There are two addition palette sections: “Raspberry Pi” and “Home Automation.” One gives access to the Pi’s I/Os, and the other opens up communications to openHAB 2 giving my sensors life in the future. Too little time, too much to do!
Figure 10 (to the right) – I’m excited about these new nodes available in Node-RED, when using the Raspberry Pi as the MQTT server. I plan to use some of the Pi’s I/O for displaying some LED status of the operational activity. Adding my well, weather and solar data to a Home Control System like openHAB will really tie everything together quite nicely.
RESOURCES
MQTT: MQTT is a machine-to-machine (M2M)/”Internet of Things” connectivity protocol
www.mqtt.org
Node-RED: Flow-based programming for the Internet of Things
www.nodered.org
esp8266: Wi-Fi Module that integrates an SPI flash used for storing user programs, data and firmware
www.espressif.com/en
Espressif Systems | www.espressif.com
Mosquitto | www.mosquitto.org
MQTT | www.mqtt.org
Node-RED | www.nodered.org
openHAB | www.openhab.org
ThingSpeak | www.thingspeak.com
PUBLISHED IN CIRCUIT CELLAR MAGAZINE • NOVEMBER 2019 #352 – Get a PDF of the issue
Sponsor this Article
Jeff Bachiochi (pronounced BAH-key-AH-key) has been writing for Circuit Cellar since 1988. His background includes product design and manufacturing. You can reach him at: jeff.bachiochi@imaginethatnow.com or at: www.imaginethatnow.com.
