ADC Timing

Ten or twenty years ago when I used to work at Stevens, I ran into a circuit that used a couple of ADCs (Analog to Digital Convertors). I was flummoxed when I started looking at it. ADCs are used to read the voltage of a signal and often that voltage is fluctuating at some frequency, so you want to taking readings often enough that you are getting a picture that reflects what is actually happening. If the signal is varies periodically with a frequency of, say 60Hz, like house current, and you take readings at the same frequency, your readings will all show the same voltage. It might be any value from zero to the maximum value. To get an accurate picture you need to take readings at some multiple of the signal frequency. Two, three or more times faster than the signal frequency.

When I look at the circuit and the ADC chips, I discover that the readings are transferred from the ADC to the CPU as serial data, that is, one bit at a time. Further, the two ADCs were daisy chained together, so the eight bits of data from one ADC were pumped out of one and into the other, and then pumped out of the second and into the CPU. This means that it took at least sixteen ticks of the serial clock to transfer the readings from the two ADCs to the CPU. This means that the serial data clock needs to be running 16 to 20 times faster than the frequency of the signal that you are trying to measure. This sounded nuts to me. Why would you want to handicap your measurement circuit with a 20x speed reduction?

Then I did the math and realized that the serial data clock was running at a frequency of something like 30 Mhz while the signal we were trying to measure was only around 1 megahertz.

This memory just popped into my head last night. Since it made such an impression on me I thought I would write it down.

Stevens Chart Recorders

Stevens Chart Recorders
Got an email from my old employer - Stevens Water Monitoring Systems, Inc. announcing the last build of their chart recorders. Chart recorders were the mainstay of their business for most of the last century, but electronical gizmos are pushing them off the stage.

Drawing of complete chart recorder

These old chart recorders were little marvels of mechanical engineering. The pen moves back and forth, driven by a wheel connected to a float. The paper is driven by a clockwork mechanism that only needed to be wound every six months, when the USGS field guy came by to change the paper. It was conceptually simple, but there are hundreds of little details that made the machine reliable and effective. They are also expensive. I think they cost something like five or ten thousand dollars.

The have been mostly replaced by electronic sensors, electronic data loggers, like the DOT Logger, and radios, like the GHT. Loggers and radios are fairly commonplace these days, but the sensors are still something of a black art. Mostly they use water pressure, and when you want to know the level of a body of water to within an inch, your pressure sensitivity needs to be very high. One inch of water spread over ten square miles of lake is a stink load of water.

House of Electrons

I think I can see the light at the end of the tunnel. I took on a little programming project for Stevens. It wasn't supposed to be a big deal, I originally figured I would be done by Christmas, and I was, sort of. But then the bugs appeared and I spent the last two months tracking them down.

The first one was an optimization problem. I had a delay loop that worked fine under the old compiler, but the new one's optimizer realized that it accomplished nothing, other than waste time, which is the enemy of optimizers, and so deleted it. So I had to look up the correct incantations and spells to use on this compiler to ensure that it would waste time when I told it to.

There were some little bugs involving some changes to commands and responses, but those were easily dealt with. After all, I could see what was wrong there. There was also a problem with trying to herd a large number of ducks down the same path, but that was more of mental technique than a programming problem.

Finally, last Wednesday, I thought it was all working and shipped it off. Finally, relieved of my obligations, at least until they test it and find something wrong. So Thursday I sat down, and just for grins, rewrote a large section of code that had been bugging me ever since I first saw it. It contained about a dozen nearly identical sections which should have been combined into one procedure. It would need a couple of parameters to stand in for the bits that changed in each section, but other than that it was a no-brainer. So I sliced and diced and cut about 200 lines out of a thousand line file, and I was very pleased with the results.

Friday I hear back from Keith and the program I sent them seems to have a very serious shortcoming. Never mind that it got over the first major hurdle, you have to be able to finish the race, and it wasn't doing that. I went back and looked at it again, and by gum, he was right. Load the program on the actual hardware and it croaks whenever you ask it to do some actual work. However, run the program in the simulator on the PC and it works fine.

I originally blamed this discrepancy on the lack of an analog input board on my set of test hardware. Maybe something funny is going on here. Well, no, it fails when run on a complete set of hardware too. What's going on? I pull my hair. I stew. I mull over possibilities. I think of nothing. Fine. Just for grins, let's try the piece of code I reworked on Thursday. There is no reason it should work any better, but hey, let's just try it anyway. And it works.

There is no reason for it to work any better than the previous version, but work it does, so I ship it off. The code size is slightly smaller, so it occupies less flash memory. It uses less RAM, though the same amount is allocated. It was down around 70-80% of available space in both cases, both before and after, so it's not like it was bumping up against any limits. I can't explain it, but I will accept that it works. Maybe the answer will appear in a circle of fire tonight in my dreams.

Work

Posting has been a little light lately. I've been working on a little project for Stevens and it is taking more time than expected. The job seemed simple enough: modify the code so that it would compile with the GNU compiler instead of the old IAR one, and make some small changes to the command interface.

I finished modifying the code so that it would compile using gcc over a month ago. As I didn't have hardware to test it on, I tried it out on a software simulator and it apparently worked fine.

Then there were some delays in testing, but eventually we discovered, that no, the code would not run on the actual hardware. So I drag a set of hardware home, start plugging things in and my little empire starts collapsing. First it was the display screen, whose mode of failure sent me down a number of ratholes before I replaced it. Then there was the dead serial port, which I cured with a new hard drive, a new installation on Windows and a USB to serial port adaptor.

New hard drive from Iguana Micro. $70. With Windows and all my development code loaded I am using three percent of the capacity.

Finally got my development environment put together and that's when it got interesting. Timing seems to be everything. Evidently gcc optimizes code better than the old IAR compiler, and so things that worked before, well, now they don't.

AVR In-Circuit Serial Progammer (ISP or ICSP, depending on who you are talking to).

Cover removed so we can unplug the ribbon cable.

That's a USB cable plugged in the back.

We are using an AVR microcontroller running at 4 megahertz (Geez, did you have to hit so hard? That megahertz!). The AVR uses one or two cycles per instruction, so a single cycle instruction takes one-fourth of one micro-second, or 250 nano-seconds. Some of the time sensitive operations operations require delays running from 100 nano-seconds up about ten milli-seconds. Fortunately, most of these times are minimums. It would be very difficult to deal with delays whose upper and lower limits were less than the length of time it takes to execute one instruction.

Update September 2016. Replaced missing pictures.

DOT Logger

One of my first projects when I joined Stevens was to develop a new data logger. The picture shows both the original enclosure (bottom) and the newer metal chassis (top). The most recent design was getting a little long in the tooth, and users (and sales) were clamoring for more capabilities. My boss found a small microcontroller board that met our requirements:

• Extended temperature range. Our equipment was being used all over the world, from Arctic regions to equatorial deserts. This means basically -40 degrees to +60 degrees Celsius (-40 to +140 Fahrenheit).
• Minimal power consumption. Many locations are accessible only by extraordinary means: hiking, mule, airplane, etc. Most locations are not served by power lines. These sites must depend on batteries and perhaps solar panels. Arctic regions are particularly bad. Batteries do not do well in cold weather, and solar panels need sunlight, which is sometimes hard to find in Northern regions.
• A variety of inputs. We needed at least two serial ports, some digital I/O pins and some ADC's (analog to digital converters).
• Very reliable.

The board we found satisfied all these requirements. It was a little expensive, but it was ready to go. All we needed was to make up an I/O board to mount it to, put it in a box, and write some firmware. Simple enough, though there were a plethora of details that had to be dealt with.

The I/O board was not too difficult. There was minimal active circuitry on it. It was mostly connectors for communications and all the various sensors. Once I had drawn the schematic, we turned it over a layout man, and then shipped the layout off to a board house to have some boards made.

We used a small plastic NEMA4 enclosure, and to keep things simple, we used the I/O board as the "front panel". The CPU board was mounted to the I/O board, and then this assembly was mounted in the lid of the box, so the CPU was hidden and all the I/O connectors were exposed. The body of the box was drilled to accept weather proof cable grommets, and the entire body of the box was used to hold the ends of the sensor cables and their connectors. The idea was that the box could be mounted to the wall of a gauge house and the cables routed to the their sensors. The connectors from the sensor cables would be plugged into the connectors on the I/O board that is mounted in the lid of the box, and then the lid would screwed down onto the body of the box, sealing all the electronics and cable connections inside a weather- and bug-proof container.

Firmware

The firmware was by far the bigger part of this project. As always, the more you can deliver, the more the customer wants. Once you have the basic structure operating, adding one more little feature does not take a whole lot of effort. Previous loggers had only recorded data on one or two channels. This logger started with 18 and eventually was increased to over 200 channels. The whole concept of what a data logger was and how it would be used had to be revised. This led to numerous adjustments to the user interface to accommodate the new capabilities while maintaining the same "flavor" as previous data loggers.

Evolution

This project was supposed to be a stop gap measure. It would allow us to get a new product on the market quickly, but given it's limitations in processor and RAM, it was not going to be long before it became obsolete. Fortunately the environmental market is rather slow moving and so even thought the CPU manufacturer has ended development in this product line, the DOT logger lives on.

Compiler Limitations

The compiler reached its' limits several years ago. It was specific to this vendor and it was fine, as far as it went. It had two modes for compiling: one was direct to the target board and the other was to a file. For development, direct-to-target worked fine, but when you were done with development and wanted to move to production, compiling to a file was what we wanted. At some point, this quit working. It may have been due to the size of the program but the compile-to-file option no long worked. Now we could have had manufacturing compile the source code for each board, it would not have added much time, but it did add a certain amount of complexity, not to mention being error prone. I had written some debug procedures to display a hex dump of memory, so I used these, along with file capture on the host to create a hex image of the code. I then wrote a simple program to convert this hex file into a binary image that could be loaded using the standard vendor supplied loader program. This allowed manufacturing to continue using the same procedure as they had been using on earlier versions of the firmware.

Field Updates

Since we had encountered so many requests for changes and features during the development of the firmware, we decided that it would be a good idea to provide some method for updating the firmware in the field, possibly even over a remote connection, like a phone line. The initial program is loaded into the CPU through a serial port using a vendor supplied loader program. In order to use this program, a jumper had to be set on the CPU board. Once the logger was assembled, this jumper was no longer easily accessible. To avoid having to gain physical access to the logger in order to update the firmware, I developed a download subsystem for the firmware that would perform this operation using standard Xmodem protocol, and without having to set the jumper.

This program was fairly straight forward, though there were a few critical places where the machine instruction sequence was absolutely critical. The CPU board has two 256 KB Flash memory chips. One was used for code, the other for recording sensor data. To perform a firmware update, the new code was downloaded onto the second chip. Then a special piece of code is copied to RAM, execution is transferred to this code in RAM. The RAM based code then copies the new firmware from the second chip to the first. We then execute a reset, and if all went well, the new code takes off and runs.

The major difficulty with this bit of code was in figuring out how to set the memory mapping registers. This is an eight bit processor with a 64 KB address space that is mapped into three windows into a one megabyte physical address space using a couple of memory mapping registers.

VX1100

One of the products Stevens picked up when they bought Vitel was the VX1100 DCP (Data Collection Platform). In its' time it was a remarkable sophisticated device. It was also very complicated and the code that ran it was as complex a mess as I have ever encountered. It combined a set of sensor inputs, a data logger, and GOES transmitter and a packet radio network protocol, not to mention a list of options as long as your arm, and it was all done with a single 8051 microcontroller. Gawd, what a nightmare. There were special versions for several other countries including Venezeula and India. People were always reporting problems with it and I would go root around in the source code and try and find the piece that was causing the problem. Occasionally I would find a problem, but more often than not, another fire would flare up somewhere else and the VX1100 problem would get sent to the back burner, i.e. forgotten about until the next time there was an issue.

We could have designed, built and tested a new logger in less time than I spent fooling around with this thing. Pound wise, penny foolish.

The code was probably pretty good when it started, but it had been through many hands, mutiple revisions and numerous additions. When it got to me, it was more than my little brain could comprehend. On top of all this, there were complex setup procedures that needed to be done in order to test many of these situations, and these complex configuration details were stored in RAM, which forgot everything you told it when you turned the power off, which you had to do every time you changed the EPROM. Now that I think about it, we should have had a separate power switch for the EPROM that would enable us to change it without wiping the RAM. Well, it probably would not have been that simple, but if I had had a chance to work on for more than a couple of weeks at a time, I might have come up with something. Some of the code was pretty good, but they were sections that looked they had been run through a meat grinder. They made absolutely no sense.

The worst part was there was no specification on what it was supposed to produce. The only thing we knew is that people were using it and it worked. So we had to duplicate their setups, run the system and see what it produced, then go back into the source code and see if I could figure out how it had arrived at the answer. What a pile of poo.

Dr. Strangelove

November 13, 2006
Dr. Strangelove

My boss (Mike Fleming) and our chief engineer here at Stevens just got back from a big meeting about GOES satellites in South Carolina. Not much new on the GOES front, more of the same bureaucratic nonsense, but he did bring back another story. Seems the Navy is putting up a constellation of LEO (low earth orbit) satellites to provide broadband communications. They do not expect to fully utilize the capacity of these satellites anytime soon, so are thinking about leasing bandwidth to the commercial sector. The Navy has put up eight of these radiation hardened satellites so far. Designing radiation hardened DSP's is my boss's specialty, so he is especially interested in this. (DSP: Digital Signal Processor. Most all radio communications uses DSP these days.)

You may have heard how the space shuttle has five computers that all do the same job, and the results they produce are a result of a vote amongst these processors. They do this because of radiation. Radiation in space is higher than it is on earth, and can cause errors in computers. Radiation hardening is anything you do to a piece of equipment so that it can continue to operate when exposed to nuclear radiation. The simplest way to do it is to shield the equipment with lead or concrete or something similar. Another way is to make each component large enough that destruction of a few molecules will not impact it's performance. Both these methods mean the device is going to larger and heavier, and heavy is the enemy of space travel.

Redundant devices is another way to deal with this problem. The space shuttle uses multiple computers. Mike's method is similar to the space shuttle method, but instead of duplicating the computers, he uses multiple transistors in the processing chip at the core of the computer itself.

For the military, radiation hardening does not mean just being able to survive the normal incident radiation of outerspace, but also to survive the EMP (Electro-Magnetic Pulse) produced by a nearby nuclear explosion, such as one caused by a North Korean Missle armed with a nuclear warhead that detonates prematurely.

Anyway, with the amount of bandwidth available from these Navy satellites, we could have Sat-phones (Satellite telephone) the size of cell-phones, and we wouldn't need all these cell phone towers. The power required to reach a LEO satellite is about the same that a cell phone produces. So now we are looking at the Sat-phone business.

Since these are military satellites, privacy would be nil, on the other hand, there would be thousands of channels, and how many channels can they actually listen to? And since they are military satellites, the military could shut down the service any time they wanted, similar to their control of the GPS satellites.

http://www.globalsecurity.org/space/systems/muos.htm