Build a Low-Cost Web Server
Jeff explains how HTML came into being and how it’s used to format information. He also presents a project that demonstrates how you might serve up some information to a web browser using an Arduino Mega 2560 and an inexpensive Wi-Fi chip.
It’s been said that a picture is worth 1,000 words. When it comes to a video display, depending on its resolution, this might even be a bit lacking. For such a display, even a picture of words requires gobs of data. Fortunately for us, some tools can reduce this through compression. Small “character” LCDs, on the other hand, have been optimized to allow the display of words with very little overhead. Small “graphic” LCDs still require an abundance of data to produce text or pictures. With graphic or video displays, you have to deal with each pixel via a stream of data. We’ve traded simplicity for flexibility.
Thanks to Samuel Morse, we began our electronic communication’s frenzy by translating letters into a series of varied duration pulses to represent each letter of the alphabet (and digits 0 to 9). As in most businesses, time is money, so the Morse Code was designed in 1836 with brevity in mind. Figure 1 shows how the most used letters require the fewest symbols to transmit, thus allowing for higher throughput. Letters were transmitted by the sender manually pressing (and releasing) a “key” to produce electrical pulses of two basic widths, the dot and the dash. At the receiving end, an electromagnet clicked and clacked out these pulses. The operator translated the sequences back into letters.
This is the key to International Morse Code. The length of a dot is one unit. A dash is three units. The space between parts of the same letter is one unit. The space between letters is three units. The space between words is seven units. Since the letters “e” and “t” are the most used, they were defined as single unit symbols. The timing is such that an “e” requires a duration of a single time unit (dot) and a “t” requires a duration of three time units (dash). Actually, you’ll note that the symbol for the letter “i,” while needing two units (dot-dot), requires a duration of three time units (one for each dot and one for the space between units) and so can be sent in the same amount of time as one dash.
The teletypewriter standardized character transmissions into a 5-bit (symbol) USTTY code. Automated messages could be sent and received at a higher rate and thus replace trained Morse code operators. A teleprinter used control codes to help format the text it was printing (i.e., line feed, LF; carriage return, CR; and form feed, FF) or even signal the operator (i.e., bell, BEL). By 1963 the ASCII 7-bit code replaced USTTY. This expanded the character set, including 33 non-printing control characters that affect how text and space are processed and 95 printable characters including the space (which is considered an invisible graphic).
In the late 1960s, one of the first video replacements for “teletype” printed output was the Computer Terminal Corporation Datapoint 3300, which could display 25 rows of 72 uppercase characters using the ASCII as input without a microprocessor. However, with the introduction of the microprocessor, many understood its potential for creating highly flexible systems driven by software. Today’s computers use higher and higher resolution video displays. The amount to data bytes required to represent one full screen of data can be astronomical. So today it might be more applicable to say, “A picture is worth a 1,000,000 words.”
HTML
When we display a high-definition image, we might be using something like a .JPG format. A screen capture on my PC will produce a 1-MB file (6-MB raw). Imagine if we needed to transmit this each time we sent someone an e-mail or transmitted a webpage. In reality, some large image files might be included in webpages, but for the most part, data can be minimal. How is “minimal” even possible with the screen resolutions in use today?
What began as a network solution to connect resources has grown into what we know now as the Internet. The Internet protocols allow for various information to be passed between two IPs. A short e-mail (less than 50 characters) sent to me had about 4 KB worth of header information to deliver a few words. The message was handled by a mail application specially designed to present me with a screen full of windows to categorize, select, and respond to this small message. The mail application creates a full screen of data and the video monitor displays it. The point here is that it is the local application that is responsible for deciding how the e-mail text is presented on the screen. You might not be using the same mail app as me and so your screen will look entirely different that mine.
— ADVERTISMENT—
—Advertise Here—
Enter the browser application used to access information resources and present the information to the user. The uniform resource locator (URL) or Internet address you type in will commonly start with “http:” to identify a resource to be retrieved over Hypertext Transfer Protocol (HTTP), the default protocol for most browsers. HTML and associated content (e.g., image files and formatting information such as CSS) is passed to the browser’s layout engine to be transformed from markup to an interactive document. This process is known as “rendering.”
Aside from HTML, web browsers can generally display any kind of content that can be part of a webpage. Most browsers can display images, audio, video, and XML files, and they often have plug-ins to support flash applications and Java applets. Like most applications, the app determines how the data sent to it will be displayed. All browser applications have agreed (for the most part) to handle all data based on the instructions sent along with the data. Those basic instructions are part of an HTML standard that describes the structure and cues for presentation, making it a markup language, rather than a programming language.
HTML is only about 20 years old. Its popularity has brought about changes that continue to make it more useful. Let’s begin with a few of the basics. HTML 2.0 was published in 1995 as IETF “RFC 1866.” HTML is an application of Standard Generalized Markup Language (SGML) based on using syntactically delimited characters that are added to the data of a document to represent its structure. There are three different kinds of markup: descriptive markup (tags), references, and markup declarations. Each of these uses special start delimiters surrounding an HTML element.
Tags use the characters <element type> and </element type> to surround the start and end of an HTML element. For instance, <TITLE> and </TITLE> are start and end tags that surround the TITLE element or text that identifies the contents of the document in a global context and may be displayed as a label for the window displaying the document.
References use the & as a prefix to the reference description. For example, < would be used to embed a less-than symbol so as not to be confused with the < used to indicate a Tag (above).
Declarations use <!element> to identify themselves. For example <!DOCTYPE HTML PUBLIC “-//IETF//DTD HTML 2.0//EN”> declares this to be an HTML document using the rules as set by public document HTML.DTD.[1]
Every HTML document has two parts—a document type declaration and an instance. The type declaration, similar to the Declaration above, refers to the rules it will follow. The instance contains a number of “tagged” elements used to give the document structure. The instance represents a hierarchy of nested elements. Each element has a name, some attributes, and some content. Most elements are represented in the document as a start tag, which gives the name and attributes, followed by the content, followed by the end tag (see Listing 1). If you save the text file above with the extension .html and click on it, your PC will execute it using your user agent (browser). You will see something like the image presented in Photo 1.
All browsers should render this example in the same way. It should be noted here that indenting in the text file is often used to help organize the elements and keep track of start and end tags. None of this affects rendering and is unnecessary. All the formatting can be eliminated and the file will still render the same way.
Listing 1
Every HTML document has two parts—a document type declaration and instance.
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html>
<head>
<title>
Webpage example
</title>
</head>
<body>
<p>
Paragraph 1.
</p>
<p>
Paragraph 2.
</p>
</body>
</html>TAGS & ATTRIBUTES
Here’s where it starts getting interesting. Table 1 is a list of many HTML 2.0 tags. <p> and </p> are tags for the paragraph element. There are many other tags that can help customize your web page. For instance, <h1> through <h6> are used to set one of six standard type sizes of an element. The <img> tag does not have an end tag, but it has two mandatory attributes. Attributes help to more fully define an element. These name-value pairs are separated by = and fall within the start tag of an element. For the <img> tag, theses are src=’file location’ and alt=’text’. The ‘src’ attribute defines where the image file is to be found and the alt attribute defines text to be displayed when a file cannot be displayed for some reason. Table 2 is a list of attributes.
-
Table 1
These are the most widely used HTML 2.0 element tags. Tag elements can include multiple tags. Most tags have delimiters surrounding their element. -
Table 2
Attributes are mandatory for some elements and optional for others. For instance, using an ID with an Anchor allows it to be referenced from elsewhere on the webpage. -
Table 3
When I connected the ESP-01 and the ESP-07, I got different responses. The AT+RST is the reset command and AT+GMR is the version command. While the commands included an “OK” in the response, this wasn’t necessarily at the end of the response. Even the version numbers did not have the same format.
CLICK AT TOP OF EACH TABLE TO ENLARGE
— ADVERTISMENT—
—Advertise Here—
To use HTML, you don’t need to know how to use each and every element of the standard. As you can see from the aforementioned code, it isn’t rocket science and this was easily written using just a text editor. Early coders got tired or bored of doing this and were determined to make tools that would simplify the process. You can find plenty of HTML editors out there to choose from, even some free versions. You might try HTML-Kit or Expression Web 4.
The above webpage file can be modified slightly to display an outside temperature reading in a webpage as rendered in Photo 2. If I remove all of the formatting (tabs and white space), what remains is a single string of 164 characters. I divided this into three strings, as you can see in Listing 2. Why divide this into separate strings? If I want my webserver to serve this page to anyone who requests it, all I have to do is serve this string and a browser will display it as an HTML page. Unless I want everyone to see the same unalterable temperature of 78°F, I must be able to stuff in the latest reading. The “big” string is split into three strings where I can easily replace String2 with the latest temperature reading as a STR$(variable). I have a web server using an Arduino with an Espressif Systems ESP8266 as the Wi-Fi device. While an entire Arduino might be overkill for serving 164 characters, it allows room for expansion without having to complicate the design with a peripheral like an SD card to store HTML files.
In its simplest form, the browser can display formatted text to show a temperature reading from an outside device. Note that the actual reading here “78” is a fixed value in the text string that we will want to substitute with the contents of a variable. We can do this by breaking the strings into three smaller strings. All text before the value, the value, and all text after the value.
Listing 2
I divided the code into three strings.
String1=“<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"><html><head><title>Temperature
</title></head><body><h1>”
String2=“78”
String3=“ degrees F</h1><h3>Outside Temperature</h3></body></html>”ARDUINO WEB SERVER
The Arduino platform continues to be very popular. I prefer using the MEGA 2560 as it has a ton of I/O, including four hardware serial ports. While you can employ a software serial port when needed, it will not give the throughput without putting a strain on available processing power. Several preexisting peripheral expansion boards provide immediate support of many I/O devices. Naked prototype expansion boards allow you to use any circuitry not already supported.
The ESP8266 has been a “hot” peripheral for a while now because of its low cost. Shanghai-based Espressif is a fabless semiconductor company that produces the low-power Wi-Fi SoC. First was the ESP-01 board, which allows microcontrollers to connect to a Wi-Fi network and make simple TCP/IP connections using Hayes-style commands through a serial interface. For quite a while, instructions were only available in Chinese, but community-based support has recently sprung up and has branched into many offshoots, including AT, FreeRTOS, MicroPython, Basic, Lisp, and Lua frameworks. Since the ESP-01 (and many other versions) comes with the AT command set preprogramed, I stuck with it for this project. You might want to look at using the ESP8266 with the other frameworks, but be aware that they require reflashing the ESP8266 to turn them into standalone devices.
Recently, the display for my outside weather station crapped out and I’m looking to replace the wireless link with Wi-Fi-enabled circuitry. Sparkfun has a Weather shield available that I’ll be experimenting with. This shield has the short form factor and leaves the far end of the Arduino 2560 free for a short protoboard that can pick up serial ports 2 and 3, power, ground, and more I/O than I would ever need (see Photo 3).
My Arduino Mega 2560 has a short “Weather” shield and a small prototyping board plugged into its expansion bus. The prototyping board picks up a hardware serial port and power and ground for interfacing to a serial device using a universal six-pin header. (For this project I use an ESP8266 Wi-Fi module.)
Having access to only 5 V requires my circuitry to include a 3.3-V regulator, as the ESP8266 despises 5 V. Rather than put the ESP-01 on the protoboard, I added my typical six-pin serial port connector so I can plug in various serial devices. Specifically, I have small prototyping board that holds either the ESP-01 or the ESP-07. While I’m using the ESP-01 here, the ESP-07 offers something not available on the ESP-01, an IPEX external antenna connector, so you can use an external antenna to improve the signal strength when necessary.
I was looking forward to using a support library for the ESP8266, but was disappointed in the fact that I wasn’t able to see all the communication actually happening between the ESP8266 and the Arduino because it was being handled blindly by the library. When things went wrong, I had no idea what was going on. Needless to say, by writing my own functions right in the application, I could print things easily to help debug the interface. It seems that there are a number of different versions out there and they don’t necessarily follow the available datasheet. Once I could see the real response, I could make an adjustment for it.
HOW MAY I SERVE YOU?
Having a Hayes-like AT command set offers a set of APIs that are easy to use. Provide the proper AT formatted command and receive a documented response. All AT commands are capitalized and must end with <CR> <LF>. The ESP-01 defaults to 9,600 bps. If you connect TX and RX, 3.3 V, and ground to a TTL serial USB module (i.e., FTDI TTL-232R-3V3), you can use your favorite terminal program to talk with the ESP8266. Refer to Table 3 for a comparison of the ESP-01 and ESP-07 reset and version commands. This tells me that the two are preprogrammed with slightly different versions and they may respond differently. Since what I see doesn’t match the instruction set manual (V1.4), I use it only as a guide and not a bible. Because I don’t want any library routines to hang waiting on a prompt that will never come I decided to write my own functions. This will allow me to keep a closer eye on all communications. The libraries I’ve seen are looking for a single trigger (character or phrase). Problem is, sometimes there are multiple possibilities.
Connecting the ESP8266 to Serial1 of the Mega 2560 leaves the console port Serial0 open to Logging messages to help debugging the code. You can also use the Arduino as a serial terminal by programming a short application to just pass everything seen on Rx0 to Tx1 and everything on Rx1 to Tx0. Type on the console ((Rx0) and it goes to the ESP8266 (Tx1). Responses from the ESP8266 (Rx1) go to the console (Tx0). You can even have them at different data rates if necessary. The default data rate for Arduino and the ESP-01 is 9,600 bps, so it all plays nice. Here you can play with the ESP8266 command set and see what kind of responses from it. When you feel a bit of confidence, you can begin to let the Arduino send commands to the ESP8266 in your own application.
I like to start off with the AT+RST command just to “reset” the ESP8266 into a known state. Then I use the AT+GMR command to indicate the version of the AT command set that is programmed into the ESP8266. You could use this information to change how you deal with the device based on the version being used. Next we need to tell the ESP8266 how it will be used. There are three options for AT+CWMODE=x. If x = 0, then the device will expect to connect with some existing network. x = 1 means it will take on the responsibility of a network access point. If x = 2, it can do both jobs simultaneously. For this project we want to give the device access to an established network, so it may be reached through an established IP. Its response will be OK, like most responses meaning “success.” Or, if it is already set to the requested state, it will response with “no change.” Be careful.
Next we want the ESP8266 to connect to our network. The AT+CWJAP command must include the SSID of your network and its password to gain access. You can use the AT+CWLAP command to get a list of the available networks within range. Since most are password protected, you will only have access to those in which you know the password. I know the network and password I want to use, so I can skip this step. If you added this to your application, you will need a way to select from the listed networks and input the appropriate password. I wouldn’t want to have to enter this manually every time, so I’m defining these in the application.
Assuming a connection is made, we can find out what IP it has been assigned by using the AT+CIFSR command. This is the IP address one would use to contact the server on this LAN. Remember, assuming your LAN is connected to the internet through a router, this IP is a local address and not accessible from the internet without setting up port forwarding. One last command before making this a server, AT+CIPMUX=1. Here we are instructing the ESP8266 to allow multiple connections (users) signified by the value 1. Because all user requests will come through the same serial connection, each message will be preceded by the ID (connection) number. Thus, you can keep track of who is asking for what.
Now it’s time to enable the server using the AT+CIPSERVER command. This command is used to enable and delete a server. Use =0 to delete the server and =1,<port> to enable it at a particular port (default is 333). Most requests default to port 80. If you want to use another port, remember to append the port to the IP request (i.e., 192.168.1.20:333). All of this takes place in the void setup() routine on the Arduino. Then execution will fall into void loop().
WAITING
The server waits patiently for a user to make a request. When a request to the proper port is received, the data is passed to the serial port with this prepended +IPD,<ID>,<length>,…:. Note there are at least three pieces of information. +IPD is the standard response, followed by the ID or user (see AT+CIPMUX), followed by the message length in characters. This may also include the user’s IP and port (if previously en/disabled using AT+CIPINFO). Everything after the : will be data.
Next month, I’ll explain how the request is handled. Then once these basics are down, we can explore what HTML 5 brings to the table. You could say it’s worth 1,000 words.
— ADVERTISMENT—
—Advertise Here—
Read Part 1 HereREFERENCE
[1] W3C, “HTML DTD,” www.w3.org/MarkUp/html-spec/html-spec_9.html#SEC9.1.
SOURCES
Arduino Mega 2560
Arduino | www.arduino.cc
ESP8266 Wi-Fi Chip
Espressif Systems | www.espressif.com
HTML-Kit
HTML-KIT.com | www.htmlkit.com
Expression Web 4 (Free Version)
Microsoft Corp. | www.microsoft.com/en-us/download/details.aspx?id=36179
PUBLISHED IN CIRCUIT CELLAR MAGAZINE • JUNE 2016 #311 – Get a PDF of the issue
Sponsor this Article
Jeff Bachiochi (pronounced BAH-key-AH-key) has been writing for Circuit Cellar since 1988. His background includes product design and manufacturing. You can reach him at: jeff.bachiochi@imaginethatnow.com or at: www.imaginethatnow.com.
