Build Your Own IoT Platform – Part 1

The Setup

In this 3-part article series, Dhairya explains how you can build your own IoT platform. In Part 1 he covers how to setup a virtual server, install the necessary software and set up some required components. He details the steps to create a minimalistic and functional IoT platform.

There are a number of great pre-made IoT platforms currently available on the Internet. They have a lot of features that make them very good and easy to use. All that said, the majority of them are paid sites. And, if you are a project developer or a student like me, those pricey platforms are just not affordable. So, what’s the alternative? Well, as developers we know how to build things, right? So, let’s build our own IoT platform!

Please note that doing this will require some investment, but the costs are a lot less than what you’d pay for those paid platforms. And, as an added advantage, you can add any custom feature you want to your own platform to suit your application needs. That’s something most paid platforms do not allow. The cost will be as low as $5 per month and, if you are a student, you will have a free alternative. I will discuss in later in the article in more detail.

FOUR KEY COMPONENTS

So, what are the minimum requirements that an IoT platform should include? Well, just have a look at the Figure 1, which lists all the main required component blocks. We will now discuss the use of the main components in brief and how we can get the most out of our platform with these. All these are hosted on a virtual cloud machine or VPS (virtual private server). We can use any operating system (OS) but using a Linux-based OS is always preferred. We will be using Ubuntu Linux for this article because it provides better performance, stricter security controls and a better ease of access to our platform. Now, lets look at the four key component blocks.

Edge interface, message broker and message bus: These deal with and talk to the physical world—especially heterogeneous devices and sensors. Because devices could be communicating over a multitude of communication technologies—such as Wi-Fi, Bluetooth, LoRaWAN, GPRS and so forth—this module needs to cater to all of them. We can achieve this in a modular format where each type of communication protocol is managed separately. We will just deal with HTTP and MQTT communication protocol in this article.

Message router and communication management: This component has the functionality of enriching existing data messages, rebroadcasting them to the message bus, publishing additional contextual information and other messages after the main message arrives, and tagging them as appropriate. All that is the job of the communication management module. Communication management functions coordinate with the message broker and the rule engine block and interact with the device manager, as required. It also deals with format conversions, for example, from CSV to JSON. For this article, we will communicate between the HTTP REST API and the MQTT Broker using a software called Node-RED.

Time-series storage and data management block: As the name suggests, this component’s function is to store all the received and parsed data that is available on the message bus. While it is not always required, this is very helpful especially due to the increasing popularity of Data Science, which can help us make sense of the stored data and provide ways to make our system more efficient. This is even useful when different services want to fetch some past data from our platform.

Rule engine block: This is the execution block that monitors the message bus and events across the platform and takes action based on set rules.

Now that we know the requirements of our platform, let’s start the fun part: implementing all those components practically! Note that for this tutorial we have chosen the DigitalOcean platform to host the virtual server on which our platform will run. To set up our virtual server, we’ll go through five steps. We will (1) initialize our cloud instance; (2) install basic and advanced software stacks as per our requirements; (3) add some security; (4) create and configure a time series database to store all data traffic on our platform; and (5) give our platform a domain name.

CLOUD INSTANCE INITIALIZATION

This is the first step toward building our platform. We will require a space on the Internet where our platform will be hosted so that we can access it from anywhere. To do this, we will set up a cloud instance on a very popular site called DigitalOcean. It is an infrastructure-as-a-service (IAAS) that lets you host virtual servers and a lot more.

To get started, go to the website www.digitalocean.com and register. After that, you will need to add some credit card credentials for payment. There are a number of referral links that will provide you with free credits, so I recommend you do a search for them. After everything is done, you will see the webpage shown in Figure 2. Now, we will create our virtual server, which is called a Droplet.

Click the blue button on the screen saying “get started with a Droplet” and it will redirect you to a new webpage where you need to specify some things with regard to your cloud instance. The first thing it asks is which OS to choose. DigitalOcean only supports Linux-based OSes out of the box but you can install custom images too. For now, we will stick with the popular Ubuntu OS because it has the biggest community. Next, you’ll need to select the plan you want to use. There are a number of plans available and they are divided into four main categories:

Standard plans: These are the cheapest and are for beginners. They provide a shared CPU space and start from as low as $5 per month.
General purpose plans: These are for people who want to host a medium-scale website for their business. They have dedicated CPUs and start at $40 per month.
CPU optimized plans: These plans are for those who do not require much RAM but they do want more CPU processing power.
Memory optimized plans: These plans are for those who need to perform memory intensive tasks like advanced data analytics.

For this article, we will go with either the $5 or $10 per month standard plan. Please note that if you are using the credit, you might want to choose a higher plan because its validity period is only two months. I am going with the $10 plan, which provides me with 2GB of RAM, one shared CPU, 50GB SSD storage space (more can be easily added) and a 2TB data transfer limit. That transfer limit is truly great considering that the same data transfer limit on a platform like AWS (Amazon Web Services) will cost you around $100.

Next, we will select the datacenter region for our platform. We can actually select any of the given choices, but if our platform is going to be used by everyone all over the world (which is the case we hope!). I would suggest you choose a US-based Datacenter region. In my personal experience they provide the best response times. Next, there are some additional options we need to fill in. And if you have a SSH key, which enables secure SSH access for you, you can just enter the public key you created and you are good to go! I would suggest using PuTTy and PuTTYgen for this. I have provided a link [1] in RESOURCES at the end of this article to a separate tutorial that will describe how to do the SSH key procedure. For this tutorial, we will directly access the platform through the public IP and login with our credentials.

After everything is done, just press Create after finalizing a host name for your Droplet. After that, wait for a few minutes as your server is created. Finally, if everything goes right, you will have a Droplet listed on your dashboard as shown in Figure 3. You can even see the Public IP Address assigned to your cloud instance on the dashboard. Now that it is up and running, we can connect to it via SSH! For Mac, you can use your terminal and type:

# ssh root@<Your Instance IP>

For Windows, you can use a software like PuTTy, which is what I am doing. Just open the software and choose the connection type to SSH and enter your host IP. After that is done, just press the Open button. It will ask for login credentials. The default username is root and a temporary password is sent to you by email. Just enter that and if it’s right, the system will prompt you to set up a new password of your choice. Do that and finally you have access to the command line of your Ubuntu Sever! We can also create floating IPs for our platform if required right from the DigitalOcean control panel.

We will skip that for now because it is not our primary requirement. Next, we will install the required software stacks to our servers. For our use case, we need the “LAMP” stack. LAMP is an acronym that stands for Linux-Apache-MySQL-PHP. As such, LAMP stacks typically consist of the Linux OS, the Apache HTTP Server, the MySQL relational database management system, and the PHP programming language. We will start with Apache Server.

APACHE SERVER

Apache is a free and open-source cross platform web server software. It is one of the most popular and widely used server software in the world. To install it on our virtual server, we only need to type the following two commands:

# apt update
# apt install apache2

We use the Ubuntu’s default package manager to do this task. You will be prompted to allow the use of extra disk space. Keep pressing Y and Enter to continue until the installation is complete.

After it is done, you have to allow the web server to bypass the firewall if it is available. For that, just issue the following commands:

# ufw app list
# sudo ufw allow “Apache Full”

Now that this is done, you can navigate to your Virtual Server’s IP address and you should see the Apache’s default page as shown in Figure 4.

MYSQL DATABASE

We will use MySQL as the database management system for our IoT Platform. It is very easy to install MySQL to our system using a few commands:

# apt install mysql-server

This will also prompt for the extra disk space so just press Y and Enter and let the installation complete. Now, you need to go through a basic setup process to complete the installation. For that, issue the following command:

# mysql_secure_installation

This asks if we want to configure the validate password plugin. We will select Y for yes and continue providing additional passwords as prompted. When we provide a new password, the script shows the password strength for the root password we entered, and we have an opportunity to change it if we want to. We will skip this step and enter N for no at this stage.

For the rest of the questions, we keep pressing the Y and Enter keys at each prompt from the script. This essentially removes some default anonymous users and the test database. Now, you can just start your mysql shell by using the following command:

# mysql

INSTALLING PHP

PHP stands for Hypertext Pre-processor, which is an open source, server-side scripting language for the development of web applications and services. We will use Ubuntu’s apt package manager to install PHP.

# apt install php libapache2-mod-php php-mysql

We will also install some additional packages, which are required to run all the software components together. After this installation is complete, we need to do some additional setup for our Apache Server.

By default, an Apache web server serves HTML files as a preference, and then looks for CGI and Perl files if the HTML file is not available. If the CGI or Perl file is not found, then it checks for a PHP file. However, since we wish to use PHP for our server-side programs in all cases, we need to change this behavior. We can change the Apache directory configuration with the following command:

# nano /etc/apache2/mods-enabled/dir.conf

This opens the configuration file in the default Ubuntu editor, called nano. This file has default file names listed in order, as shown:

<IfModule mod_dir.c>
DirectoryIndex index.html index.cgi index.pl index.php
index.xhtml index.htm
</IfModule>

First, we change the order of the file names, starting with index.php followed by the index.html, and then the rest.

<IfModule mod_dir.c>
DirectoryIndex index.php index.html index.htm index.cgi
index.pl index.xhtml
</IfModule>

Once the changes are done, the file can be saved by pressing Ctrl+X and then typing Y, followed by pressing Enter. This exits us from the editor. We restart the Apache web server to make these changes effective by using the following command:

# systemctl restart apache2

This silently restarts the Apache web server and reloads the configurations with the new changes. However, we still need to validate that these changes are effective. To do so, we create a test PHP program file and verify it within the browser by navigating to the IP address of our cloud instance. To create a new test program, we can open a new file with this command and add a few basic lines in the file as follows:

# nano /var/www/html/test.php
Add these contents to the file
<?php
echo ("Hi...Welcome to my IoT Platform!");
?>

Once finished, we save and close with the Ctrl+X combination followed by typing Y and then pressing Enter. Now when we navigate to http://<INSTANCE_IP>/test.php, we should see the message as shown in the Figure 5. At this stage, our LAMP stack is fully functional, but we need to enable an easy and efficient access to MySQL for which we will be using the popular phpMyAdmin program.

PhpMYADMIN

phpMyAdmin is a free and open-source administration tool for MySQL. As a portable web application written primarily in PHP, it has become one of the most popular MySQL administration tools. To begin the installation, we will first update the package index of our cloud instance. This is followed by the installation of base files for phpMyAdmin with the following two commands:

# apt update
# apt install phpmyadmin php-mbstring

At this stage of the installation process, the system asks a few questions. On the first screen, we will select apache2 as the server. We use the spacebar to mark our selection while we move our choices using arrow keys. Once selected, the installation continues and then asks the next question with another prompt —”configure database for phpmyadmin with dbconfig-common“. Select Yes.

Finally, it asks you to choose and confirm your MySQL application password for phpMyAdmin. After you input that, the installation is complete. At this stage, the installation has added the phpMyAdmin Apache configuration file to the /etc/apache2/conf-enabled/ directory, where it is read automatically. The only thing we now need to do is explicitly enable the mbstring PHP extension, which we can do by entering the following:

# phpenmod mbstring
# systemctl restart apache2

Now we can access the MySQL database with phpMyAdmin by navigating to http://<INSTANCE_IP>/phpmyadmin. It asks for credentials, which we just created. Upon providing the correct credentials, we are able to access our MySQL database in the browser, as shown in Figure 6.

After this, we have a fully functional platform with a running system, a phpMyAdmin based database management system and also PHP programming capabilities. But I wanted to make it even easier, enabling even the absolute beginners to be able to leverage the most out of this. For that reason, we will be using Node-RED to do all the tasks of our platform for now. Note that when you want to take this into production, you will need to move to a more sophisticated language like Python or Node.js for APIs and database management. But for now, we will install Node-RED to our virtual server.

NODE.JS AND NODE-RED

Node.js is a JavaScript platform for general-purpose programming that allows users to build network applications quickly. By leveraging JavaScript on both the frontend and backend, Node.js makes development more consistent and integrated. We will use Ubuntu’s apt package manager to update and install Node.js on our cloud instance with the following commands:

# apt update
# apt install nodejs
# apt install npm

In those three commands, the first command refreshes the local package index, and the next command installs Node.js on our instance. We are also installing the node package manager (NPM), which helps us update and add node packages to our instance as needed. For each of the installations, we are prompted with the amount of disk space being used. We are selecting Y for yes in both cases. Upon successful installation, we can check the installed version of each application with the following commands:

# nodejs -v
# npm -v

If these run without any error, that means node.js has been successfully installed on your server. Now, we will get Node-RED using the NPM or node packet manager using the following command:

# npm install -g –unsafe-perm node-red

After the installation is done, you start Node-RED using the command:

# node-red

The Node-RED software runs on the port 1880 of our IP Address and you can access it easily through your browser. But if you have your firewall enabled, you need to allow TCP connection on the port 1880 to access your Node-RED Dashboard. This can be done using the following command:

# ufw allow 1880/tcp

Now, go to your browser and navigate to the public IP address http://<INSTANCE_IP>:1880 and you should see the Node-RED Dashboard as shown in Figure 7. This enables us to create simple REST APIs and set up MQTT connections using a simple drag and drop programming mechanism. We will cover these in Part 2 of this article series. For now, we will just install the MQTT server to our platform as the last task of this article.

MQTT: MESSAGE BROKER

As we know, we will be using HTTP and MQTT protocols for our IoT platform. HTTP is pretty straightforward and doesn’t require any extra software but when it comes to MQTT, you need a MQTT Broker running on your platform to handle MQTT connections and data. MQTT is a broker-based communication protocol that follows a publish/subscribe mechanism. This paradigm is explained in Figure 8.

Now that we have discussed the fundamentals of a publish/subscribe mechanism and the way that a broker functions, let’s install and configure the MQTT broker on our cloud instance. We will install the Mosquitto broker on our system. The Ubuntu repository has the latest version of Mosquitto available to install through apt package manager.

# apt update
# apt install mosquitto mosquitto-clients

After the installation is completed, Mosquitto starts immediately on port 1883, which can be checked using the following command:

# lsof -i :1883

By default, Mosquitto listens on a standard MQTT port 1883, and we can see in the output that the program is running and listening on that port. At this stage, we can run a quick check to see if everything is working as expected. To do that, we open another terminal window and log in for a second time. Let’s keep the two windows open side by side to see what is going on. On one terminal, we subscribe to test_Topic with the following command:

# mosquitto_sub -h localhost -t test_Topic

Here, -h specifies the hostname and -t specifies the subscription topic. After sending the command, we do not see anything output because the client that we just created is listening for new messages and nothing has been sent yet. On the other terminal, we publish a test message on test_Topic as follows:

# mosquitto_pub -h localhost -t test_Topic -m “Hello world!”

The additional -m option is for specifying a message for the given topic. Once the command is sent, you should see this message pop up on another terminal that has subscribed to this test_Topic.

Please note that you can test it on any device, you just need to have Mosquitto installed and instead of typing “localhost” as the -h argument, use the IP address of your DigitalOcean Droplet and it should work just fine. This confirms that the Mosquitto broker is working fine on our cloud instance. In this article, we set up our own IoT platform on an Ubuntu system hosted on a virtual server. In Part 2, we will use what we started here to develop an IoT project of our own. 

Read Part 2 Here

RESOURCES

Reference:
[1] For the link to the SSH Key setup:
 https://docs.digitalocean.com/products/droplets/how-to/add-ssh-keys

Apache Software Foundation | www.apache.org
DigitalOcean | www.digitalocean.com
Eclipse Mosquitto | www.mosquitto.org
Node.js | www.nodejs.org
Node-RED | www.nodered.org
phpMyAdmin | www.phpmyadmin.net
PuTTY | www.putty.org
PuTTYgen | www.puttygen.com

PUBLISHED IN CIRCUIT CELLAR MAGAZINE • JUNE 2021 #371 – Get a PDF of the issue

Sponsor this Article