You have an idea an idea for an innovative microcontroller-based design? Once you start start soldering and wiring, you might want to keep an eye on Bob Japenga’s checklist of essential embedded system design principle. His complete list will appear in Circuit Cellar‘s 25th Anniversary issue, which will be available in early 2013. But since many of you will be attempting to complete projects before January 1, we’re giving you a sneak peek.
We all know that old adage: “If you don’t have time to do it right the first time, where do you find the time to do it right the second?” But this is the nature of developing robust embedded systems. There are literally thousands of little decisions that we make even in the simplest of projects. How can we minimize repeating mistakes?
So my goal in this article is twofold: to celebrate with Circuit Cellar 25 years of great service to us engineers and to hammer home some of those principles that we so often forget. I will divide the essentials into four categories: general essentials, essentials that exist because things (i.e., us and our designs) fail, essentials about testing, and essentials about memory use.
KISS & No Simpler—“Keep it simple stupid (KISS).” How often do I need to hear this? I like the saying about KISS that’s often attributed to Albert Einstein but was actually Roger Session’s paraphrase: “Make things as simple as possible, but no simpler.” I am counting these as our first and second essentials. Keep it simple is number one and no simpler is the second. I find this an immense challenge. When we are faced with schedule deadlines and tight budgets, it is costly to make a design simple. Some people have a gift at taking a problem and abstracting an elegant and simple solution. I remember giving a design specification to one of my employees a number of years ago when I worked for an aerospace company. After several days he came back with over 20 pages of algorithms and charts defining how the specification should be met in software. I wasn’t smart enough to know why it was too complex, but my gut feeling was: “This is too complex. Make it simpler.” Later, I turned it over to another young man who returned with an elegant two-page algorithm that worked perfectly.
How do we do that? “As simple as possible” can get us in trouble if we make it too simple. For example, just recently we were designing a multi-drop serial interface to be incorporated into a medical device. A strong case could be made for the simplicity of using a single-ended interface. But experience tells us that a differential interface will be more robust in the face of defibrillators and all the various noisy electronic instruments it will to play with. Which meets the KISS principle? The same tough decision comes when you’re trying to decide whether to go with a two-wire or a four-wire interface. Two wires has less cabling, but it’s more complex in the interface and forces single-duplex operation. Again, which meets the principle?
Sometimes the trade-off can come down to what you already have in the house. If you have well-debugged libraries for handling the two-wire 485 protocols, the reduced number of wires reduces the overall system complexity even though the software will in fact be more complex.
Sometimes when dealing with algorithm development, the KISS principle can also present ambiguous alternatives. At times, a straightforward linear programming approach can be many times more lines of code and more complex than an elegant algorithm. But the elegant algorithm may be obscure, difficult to maintain, or take too long to come up with. Therein lies the challenge to Keep It Simple Stupid but No Simpler.
Define the Problem/Create Clear Specs—Having a clear set of specs is essential to every part of a design. We all know this and we always belly ache about how we don’t have perfect specifications. But get with it. You won’t always have perfect specs from your customer. But it is essential that you make them as good as possible. And in writing. If your customer is willing, keep pushing back and keep writing it down and refining it as you go.
I’ve found that essential for every phase of a project. Whether it is hardware or software, writing out the spec (on the schematic or in the code) is a wonderful act of discipline. Write out the spec for the module. Write out the spec for the algorithm. Write out the spec for the circuit. Writing it out forces you to think it through. End the belly-aching about the lack of good specs. Start creating them.
Don’t Skimp on the Tools—Tools are our life blood. If you are a manager and your designers don’t have the best tools, you are wasting your money on their salaries. That said, we are not talking about buying tools you don’t use, tools that don’t pay for themselves, or tools that you can rent more cost effectively. Last week we were discussing a problem where one of our cell modem designs exceeded the limit for the second harmonic in spurious emissions. In talking over the problem with the test lab, I discovered that they had a tool that they brought inside the anechoic chamber that could tell the cell modem to transmit on such and such a frequency at maximum power. Naively, I asked, “Shouldn’t we have such a tool?” Someone responded: “Yes, but they cost almost a million dollars.” Oh. But we found we could rent one for $1,000 a day. So, I am not talking about being unwise with our money.
Many years ago while at the aerospace company, I was recommending an HP64000 system that appeared to be a very powerful tool for our software development team. I wrote up the proposal and presented it to the vice president of engineering. His question has haunted me ever since. “Would you buy it if it were your money?” I said then, and continue to say now, “Get the best tools that will allow you to do the job as quickly as possible. If a 200-man-hour job can be done for 100 hours with a $10,000 instrument, is it worth it. Absolutely.”
Read the Documentation—Last year we had a problem that showed up only after we started making the product in 1,000-piece runs. The problem was that some builds of the system took a very long time to power up. We had built about 10 prototypes, tested the design over thousands of power ups, and it tested just fine (thanks to POC-IT). Then the 1,000-piece run uncovered about a half-dozen units that had variable power-up times—ranging from a few seconds to more than an hour! Replacing the watchdog chip that controlled the RESET line to an ARM9 processor fixed the problem. But why did these half dozen fail? Many hours into the analysis we discovered that the RESET line out of the watchdog chip on the failed units would pulse but stay low for long periods of time. A shot of cold air instantly caused the chip to release the RESET. Was it a faulty chip lot? Nope. Upon a closer read of the documentation, we found that you cannot have a pull-up resister on the RESET line. For years we always had pull-ups on RESET lines. We’d missed that in the documentation.
Like it or not, we have to pour over the documentation of the chips and software library calls we use. We have to digest the content carefully. We cannot rely on what is intuitive.
Finally, and this is much more necessary than in years past, we have to pour over the errata sheets. And we need to do it before we commit the design. A number of years ago, a customer designed a major new product line around an Atmel ARM9. This ARM9 had the capability of directly addressing NOR memory up to 128 MB. Except for the fact that the errata said that due to a bug it could only address 16 MB. Ouch! Later we had problems with the I2C bus in the same chip. At times, the bus would lock up and nothing except a power cycle would unlock it. Enter the errata. Under some unmentioned conditions the I2C state machine can lock up. Ouch! In this case, we were able to use a bit-bang algorithm rather than the built-in I2C—but obviously at the cost of money, scheduling, and real time.
If You Can’t Explain it to Mom, It Ain’t Clear—That’s another way to say: “Assume no one reads the user manual.” I recently read a blog post about the City of Boston’s electronic parking meters (http://usabilitylessons.wordpress.com/category/general/). Truly, one wonders who reviewed that user interface. If you want to make robust embedded systems with a user interface, they need to have intuitive interfaces, or you may be surprised at what the user comes up with. This takes time and effort, but it’s well worth it. Try it out on the uninitiated. Engineers are the worst kind of people for testing user interfaces. Try it on kids. My business partner’s one-year-old son found the first bug in our first product.
Be sure to get your hands on the upcoming anniversary issue to learn about the reset of the principles. He covers “Things Fail Essentials,” “Testing Essentials,” “Memory Management Essentials,” and more. Consider using it to create your own design principles checklist that you can keep at your workbench.