I learned the difference between reliability and availability the hard way. It was 2017, and I was standing in a flooded equipment room in Belém, ankle-deep in water, staring at a data logger that was technically “on” but hadn’t recorded a single usable data point in three days. The system was available—the power light was blinking, the processor was running—but it wasn’t reliable. The humidity had crept into the sensor connectors, the SD card was corrupted, and the cellular modem was stuck in a reboot loop. That’s when I realized: in the world of embedded systems for harsh environments, availability is a vanity metric. Reliability is what keeps your data flowing when the grid is drunk and the air tastes like soup.
This article is for the engineers, technicians, and system integrators who deploy electronics in places where the power grid flickers more than a candle, where 4G is a distant dream, and where “weatherproof” is a challenge, not a spec sheet checkbox. We’re going to tear apart the concepts of reliability and availability, ground them in real field conditions, and build a practical framework for designing systems that don’t just survive—they deliver.
Defining the Terms Without the Marketing Fluff
Let’s start with the basics, stripped of academic jargon. Availability is the percentage of time a system is powered on and theoretically capable of functioning. It’s a simple ratio: uptime divided by total time. If your remote weather station is powered up for 8,760 hours a year, it’s 100% available. But if it only successfully logs and transmits data for 4,000 of those hours because the cellular modem keeps dropping out, its availability is still 100%—and its reliability is garbage.
Reliability is the probability that a system will perform its intended function under stated conditions for a specified period. In our world, that means the sensor node in the middle of a sugarcane field in Pernambuco correctly measures soil moisture, stores the data locally, and transmits it when connectivity exists—without corruption, without gaps, and without needing a technician to hike out there every other week. Reliability is about correct operation, not just being turned on.
In the embedded systems community, we often conflate these two because most textbooks assume a stable power source and a benign environment. Throw in a tropical climate, an unreliable grid, and intermittent connectivity, and the distinction becomes the difference between a project that delivers actionable data and one that delivers expensive e-waste.
Why the Grid Makes Availability a Distraction
In regions with unstable power infrastructure, designing for high availability often leads to over-engineering the wrong things. I’ve seen teams obsess over redundant power supplies and hot-swappable batteries while ignoring the fact that their firmware’s file system would corrupt on every unclean shutdown. They’d achieve 99% uptime on paper, but the data was 40% garbage because the system wasn’t designed to handle the reality of frequent power cycles.
Here’s a field truth: in many parts of Brazil, the grid doesn’t just go off—it browns, surges, and oscillates before finally dropping. A simple undervoltage lockout circuit isn’t enough. Your power management IC needs hysteresis, and your firmware needs to treat every power-down as an imminent event. I now design all my remote nodes with a supercapacitor-backed real-time clock and a few milliseconds of hold-up time to safely close files and park the SD card before the voltage rails collapse. That’s a reliability feature, not an availability feature.
Consider the typical solar-powered installation. The availability calculation says: if the battery is sized for three days of autonomy and the sun shines every two days, you’re golden. But reliability asks: what happens when the battery’s internal resistance rises due to heat aging, and the voltage sags under load even though the state-of-charge says 60%? I’ve measured battery internal resistance doubling after just six months in a sealed enclosure under direct sun in Ceará. The system was “available” but couldn’t power the modem during transmission bursts. Reliability cratered.
Intermittent Connectivity: The Silent Data Killer
Availability metrics love to count “connected hours,” but in the field, connectivity is a binary mess. Your device might have a perfect 4G signal for 10 minutes, then nothing for six hours. If your firmware isn’t designed for store-and-forward with resilient retry logic, you’re losing data. Period.
I learned this lesson with a fleet of water quality monitors along the Rio São Francisco. The initial firmware used a simple MQTT publish with QoS 0—fire and forget. When the cellular network was up, it worked beautifully. But during the rainy season, when the network would drop for days, the device’s circular buffer would overflow, and we’d lose the oldest data first—exactly the data we needed to understand the flood event. We switched to a priority-based storage system that preserved high-resolution data during events and downsampled older data. We also implemented MQTT QoS 1 with persistent sessions and local SQLite storage. Availability didn’t change. Reliability—measured as the percentage of generated data points successfully delivered to the cloud—jumped from 62% to 97%.
For systems with truly intermittent connectivity, I now recommend a tiered storage approach: RAM buffer for real-time data, FRAM or MRAM for critical state variables that must survive power loss, and high-endurance SD cards (SLC or pseudo-SLC) for bulk storage. Avoid consumer-grade TLC cards—they’ll wear out in months under constant write cycles. I’ve had good results with Swissbit and Apacer industrial cards, though they’re not cheap. The cost of a field visit to replace a failed card far exceeds the price difference.
Environmental Stress: When “IP67” Isn’t Enough
Here’s where I get irritated with datasheet-driven design. An IP67 rating means the enclosure survived a 30-minute dunk in still water under lab conditions. It doesn’t mean it’ll survive years of thermal cycling, UV exposure, and condensation in a tropical environment. I’ve opened IP67 enclosures to find a swimming pool inside because the pressure differential from daily heating and cooling sucked moisture past the seals.
The fix isn’t a higher IP rating—it’s a system-level approach to environmental protection. I now use Gore-Tex vents on every enclosure to equalize pressure while blocking liquid water. I conformally coat all PCBs with a silicone-based coating (I prefer Electrolube’s DCA for its reworkability). I specify connectors with gold-plated contacts and use dielectric grease on every mating surface. And I test every prototype in a thermal chamber that cycles between 10°C and 60°C at 95% relative humidity for at least 200 cycles before I even consider deploying. That’s not a standard test—it’s a survival test I developed after too many failures.
One specific failure still stings: a soil moisture sensor network in a coffee plantation in Minas Gerais. The sensors used capacitive measurement, which should be more resilient than resistive types. But the PCB’s guard ring design was inadequate, and condensation on the board surface created leakage paths that swamped the femtofarad-level capacitance changes we were trying to measure. The data looked plausible—the system was “available”—but the readings were nonsense. We redesigned the board with proper guarding, added a hydrophobic coating, and included a diagnostic channel to detect board leakage. That’s reliability engineering: anticipating failure modes, not just meeting specs.
Measuring What Matters: Field Metrics That Don’t Lie
If you’re not measuring reliability, you’re guessing. I’ve settled on a few key metrics that actually reflect field performance:
- Data Delivery Ratio (DDR): The percentage of expected data points that successfully arrive at the server, correctly timestamped and within acceptable error bounds. This is my north star. If DDR drops below 95%, something is wrong—even if every device is reporting “online.”
- Mean Time Between Data Gaps (MTBDG): How long the system runs before experiencing a gap longer than a defined threshold (e.g., 1 hour for hourly data). This catches intermittent failures that availability metrics miss.
- Sensor Drift Rate: Tracked through periodic calibration checks or redundant sensors. In tropical conditions, I’ve seen electrochemical gas sensors drift by 20% per year—far faster than datasheet predictions based on temperate lab conditions.
- Power Cycle Survival Rate: The probability that the system resumes correct operation after an uncontrolled power loss. I test this by yanking the power cord 1,000 times and checking for file system corruption, RTC drift, and configuration loss.
These metrics require you to instrument your system properly. I include a “health packet” in every data transmission: battery voltage, internal temperature/humidity, flash wear level, last reset cause, and a checksum of the configuration. This data is gold for diagnosing field issues remotely. Without it, you’re flying blind.
Design Patterns That Prioritize Reliability Over Availability
Over the years, I’ve developed a set of design patterns that shift the focus from “keep it running” to “keep it working.” These aren’t theoretical—they’re battle-tested in the Brazilian backlands.
1. Graceful Degradation Instead of Binary Failure
When a sensor fails or a communication link drops, the system shouldn’t just stop. It should fall back to a reduced-functionality mode that preserves core data. For a weather station, if the wind speed sensor fails, the system should still log temperature, humidity, and pressure—and flag the wind data as invalid rather than sending zeros or stale values. This requires your firmware to actively monitor sensor health (checking for open circuits, out-of-range values, or communication timeouts) and maintain a status register for each subsystem.
2. Watchdogs That Actually Watch
I’m tired of seeing external watchdog timers used as a crutch for bad firmware. A watchdog should be the last line of defense, not the first. Your firmware should have internal sanity checks: task heartbeats, stack overflow detection, memory pool monitoring. If the system resets, it should log the reset cause and attempt a controlled recovery—restoring the last known good configuration, running a self-test, and only then resuming normal operation. I’ve seen systems that reset every 10 minutes for months because of a memory leak, and the watchdog just kept kicking them back into the same failure loop. That’s not reliability; that’s a zombie.
3. Time Synchronization Without Assuming Connectivity
NTP is great when you have internet. In the field, you often don’t. I use a combination of GPS time synchronization (even a cheap module can provide accurate time if it can get a fix once a day) and a temperature-compensated RTC. The firmware maintains an estimate of RTC drift and applies corrections. When connectivity is available, it syncs with NTP and updates the drift model. This ensures timestamps are accurate even after weeks of isolation—critical for correlating data across nodes.
4. Data Integrity From Sensor to Server
End-to-end checksums are non-negotiable. I CRC every data packet at the point of generation, store the CRC with the data in local flash, and verify it on read-back before transmission. The server then validates the CRC before ingestion. Any mismatch triggers a re-transmission request or, if the data is already gone from local storage, a gap marker. This catches bit flips from cosmic rays, flash degradation, and noisy communication channels. I’ve seen single-bit errors in flash storage cause sensor readings to jump by orders of magnitude—errors that would go undetected without end-to-end integrity checking.
Field-Proven Component Selection for the Tropics
Datasheets lie—or at least, they omit the conditions you’ll actually face. Here’s what I’ve learned about component selection through years of post-mortems on failed devices:
- Connectors: Gold-plated contacts are mandatory. Tin-plated connectors develop high-resistance oxide layers in humid conditions. I’ve measured contact resistance increases from milliohms to tens of ohms in six months. Use connectors with IP67 or better rating when exposed, and always apply dielectric grease.
- Capacitors: Avoid Y5V and Z5U dielectrics—their capacitance drops by 80% or more at temperature extremes. X7R is the minimum; X8R or C0G/NP0 for timing-critical circuits. Electrolytics dry out fast in heat; I’ve switched to polymer electrolytics or large MLCCs where possible.
- SD Cards: As mentioned, SLC or pseudo-SLC industrial cards only. Consumer cards use TLC or QLC flash with poor wear-leveling and will fail in months under continuous logging. I’ve had good results with Swissbit S-46 and Apacer industrial microSD.
- Enclosures: Stainless steel or powder-coated aluminum. Avoid plastic enclosures in direct sun—they degrade and warp. Use Gore-Tex vents to equalize pressure and prevent condensation. Double-bag desiccant inside the enclosure during assembly.
- Conformal Coating: Silicone-based (SR) for reworkability, acrylic (AR) for cost-sensitive projects. Apply after all connectors are mated to avoid coating contacts. I’ve seen uncoated boards develop dendritic growth between closely spaced pads in high humidity—effectively shorting out sensitive analog circuits.
Testing Like You Mean It: Beyond the Lab Bench
Lab testing is necessary but insufficient. I’ve had devices pass IEC 60068 environmental tests with flying colors and then fail within weeks in the field. Why? Because lab tests are standardized, repeatable, and don’t capture the chaotic combination of stresses that real deployments impose.
My field-testing regimen now includes:
- Thermal-Humidity Cycling: 200+ cycles between 10°C and 60°C at 95% RH, with the device powered and actively logging. This exposes condensation issues, connector corrosion, and component drift.
- Power Cycling Under Load: 1,000+ cycles of abrupt power removal and restoration while the device is writing to flash and communicating. This catches file system corruption and brown-out issues.
- Vibration Testing on a Shaker Table: Simulating transport on unpaved roads. I’ve had connectors work loose and crystals fracture from vibration that didn’t show up in drop tests.
- Solar Radiation Simulation: Using UV lamps to age enclosures and exposed cables. I’ve seen cable ties disintegrate and LCD displays become unreadable after a year of tropical sun.
But the most valuable test is a beta deployment in the actual target environment for at least one full seasonal cycle. There’s no substitute for real rain, real insects, real voltage spikes from nearby lightning, and real humans interacting with the equipment in ways you never anticipated.
When Availability Metrics Actually Help
I don’t want to give the impression that availability is useless. It has its place—specifically, in systems where uptime is the primary requirement and the function is simple. For example, a pump controller that just needs to turn on when the water level drops. If it’s available, it works. But even then, you need to ensure that “available” means “capable of correct operation,” not just “powered on.”
Availability metrics are also useful for fleet management. If 10% of your devices are offline, you have a logistics problem—even if the remaining 90% are perfectly reliable. I use availability as a coarse filter: if a device’s availability drops below 90%, I investigate. But I don’t confuse high availability with a healthy system.
Building a Reliability Culture in Your Team
This is the hardest part. Reliability isn’t just a set of design rules—it’s a mindset. It means prioritizing field data over lab data, admitting failures openly, and resisting the urge to declare victory when the dashboard shows green lights.
I’ve instituted a “failure review” process for every field failure. No blame, just a systematic analysis: what happened, what was the root cause, how was it detected, and what design or process change will prevent it in the future. These reviews have generated a living document of design guidelines that now spans over 100 pages. Every new engineer on the team reads it before touching any hardware.
I also insist on field visits for every designer. There’s no substitute for seeing your device covered in mud, with ants nesting in the enclosure, and a farmer telling you it stopped working six months ago but he didn’t know who to call. That experience changes how you design.
FAQ: Reliability and Availability in Harsh Field Deployments
What’s the simplest way to explain the difference between reliability and availability to a non-engineer?
Availability is whether the device is turned on. Reliability is whether it’s doing its job correctly. A device can be available 100% of the time but completely unreliable if it’s sending garbage data. Think of a car: availability means the engine starts; reliability means it gets you to your destination without breaking down.
How do I choose between improving reliability and improving availability when I have a limited budget?
Always prioritize reliability. A system that’s available but unreliable generates bad data, which erodes trust and can lead to wrong decisions. A system that’s occasionally unavailable but reliable when it’s up still generates trustworthy data. Focus on data integrity, graceful degradation, and resilient storage before spending money on redundant power supplies or hot-swap capabilities.
What’s the most common reliability failure you see in tropical deployments?
Condensation inside enclosures, hands down. It causes corrosion, short circuits, and sensor drift. The fix is a combination of pressure-equalizing vents, conformal coating, and proper enclosure sealing. Second place is SD card corruption from unclean power-downs—solved with supercapacitor-backed safe-shutdown circuits and industrial-grade flash storage.
Can I use consumer-grade components if I add enough protection?
Sometimes, but it’s a gamble. I’ve used Raspberry Pi boards in field deployments with conformal coating, industrial SD cards, and resilient power supplies, and they’ve survived for years. But I’ve also had the USB connectors corrode, the HDMI port short out from humidity, and the polyfuse trip repeatedly in high temperatures. If you go this route, test extensively and budget for higher failure rates. For critical systems, industrial-grade components are cheaper in the long run.
How do I convince management to invest in reliability testing?
Show them the cost of a field failure. Calculate the expense of sending a technician to a remote site—travel, labor, downtime, lost data. One field visit often costs more than the entire reliability testing budget for a project. I keep a spreadsheet of every field failure and its cost, and it’s the most persuasive tool I have. When management sees that a $500 testing investment could have prevented a $5,000 field repair, the conversation changes.
Next Steps: From Theory to Practice
This article has laid out the conceptual framework and practical techniques for prioritizing reliability over availability in harsh field deployments. The next logical step is to dive deeper into specific implementation details. In upcoming articles, I’ll cover:
- Designing a bulletproof power supply for solar-powered remote stations—including MPPT charger selection, battery chemistry tradeoffs, and supercapacitor hold-up circuits.
- Implementing end-to-end data integrity—from sensor CRC to server-side validation, with code examples for embedded C and Python.
- Building a field failure database—how to systematically collect, analyze, and learn from every failure to continuously improve your designs.
If you have specific questions or failure stories of your own, I’d love to hear them. The best lessons come from the field, not from textbooks. Reach out through the contact page, and let’s build a community of practice around designing embedded systems that actually work where they’re needed most.