Electrical engineers often develop “headless” electronic systems—that is, systems without user interfaces. And many of those systems are embedded within product and are generally out of reach when problems occur. Bob Japenga is an engineer with some advice about logging and how it can help you troubleshoot problems as they occur.
Many of our designs are buried in some product or located in remote areas. No one is there when the device hiccoughs. Well defined logging can help make your system more robust because there are going to be problems and you might as well find out as much as you can about the problems when they happen. Here are some general guidelines that we apply here:
• Use an existing logging facility if you can. It should have all of the features discussed here.
• Unless you truly have unlimited disk space, provide a self-pruning cap on all logs. Linux syslog feature has this built in.
• Attempt to provide the most amount of information in the least amount of space. One way we do this is by limiting the number of times the same error can be logged. I cannot tell you how many times I look at a log file and find the same error logged over and over again. Knowing your memory limitation and the frequency of the error, after a set number of identical logs, start logging only one in every 100, or only allow so many per hour, and include the total count. Some failures are best kept in error counters. For example, communications errors in a noisy environment should be periodically logged with a counter; you don’t usually need to know every occurrence.
• Create multiple logs concerning multiple areas. For example, network errors and communications errors are better kept in their own log apart from processing errors. This will help a single error from flooding all logs with its own problem.
• Timestamp all logs—ideally with date and time—but I understand that all of our systems don’t have date and time. As a minimum, it could be in milliseconds since power-up.
• Establish levels of logging. Some logging is only applicable during debugging. Build that into your logging.
• Avoid side effects. I hate it when the designer tells me that if he turns logging on, the system will come to its knees. That’s just a bad logging design.
• Make the logs easy to automatically parse.—Bob Japenga, CC25, 2013