Last year we had a problem that showed up only after we started making the product in 1,000-piece runs. The problem was that some builds of the system took a very long time to power up. We had built about 10 prototypes, tested the design over thousands of power ups, and it tested just fine (thanks to POC-IT). Then the 1,000-piece run uncovered about a half-dozen units that had variable power-up times—ranging from a few seconds to more than an hour! Replacing the watchdog chip that controlled the RESET line to an ARM9 processor fixed the problem.
But why did these half dozen fail?
Many hours into the analysis we discovered that the RESET line out of the watchdog chip on the failed units would pulse but stay low for long periods of time. A shot of cold air instantly caused the chip to release the RESET. Was it a faulty chip lot? Nope. Upon a closer read of the documentation, we found that you cannot have a pull-up resister on the RESET line. For years we always had pull-ups on RESET lines. We’d missed that in the documentation.
Like it or not, we have to pour over the documentation of the chips and software library calls we use. We have to digest the content carefully. We cannot rely on what is intuitive.
Finally, and this is much more necessary than in years past, we have to pour over the errata sheets. And we need to do it before we commit the design. A number of years ago, a customer designed a major new product line around an Atmel ARM9. This ARM9 had the capability of directly addressing NOR memory up to 128 MB. Except for the fact that the errata said that due to a bug it could only address 16 MB. Ouch! Later we had problems with the I2C bus in the same chip. At times, the bus would lock up and nothing except a power cycle would unlock it. Enter the errata. Under some unmentioned conditions the I2C state machine can lock up. Ouch! In this case, we were able to use a bit-bang algorithm rather than the built-in I2C—but obviously at the cost of money, scheduling, and real time.—Bob Japenga, CC25, 2013