Software Architect

Everything will ultimately fail

Book: 97 Things Every Software Architect Should Know
Publisher: O’Reilly Media
Author: Richard Monson-Haefel
97 Things Every Software Architect Should Know – 38/97

'Coz sharing is caring

Hardware is fallible, so we add redundancy. This allows us to survive individual hardware failures, but increases the likelihood of having at least one failure present at any given time.

Software is fallible. Our applications are made of software, so they’re vulnerable to failures. We add monitoring to tell us when the applications fail, but that monitoring is made of more software, so it too is fallible.

Humans make mistakes; we are fallible also. So, we automate actions, diagnostics, and processes. Automation removes the chance for an error of comission, but increases the chance of an error of omission. No automated system can respond to the same range of situations that a human can.

Therefore, we add monitoring to the automation. More software, more opportunities for failures.

Networks are built out of hardware, software, and very long wires. Therefore, networks are fallible. Even when they work, they are unpredictable because the state space of a large network is, for all practical purposes, infinite. Individual components may act deterministically, but still produce essentially chaotic behavior.

Every safety mechanism we employ to mitigate one kind of failure adds new failure modes. We add clustering software to move applications from a failed server to a healthy one, but now we risk “split-brain syndrome” if the cluster’s network acts up.

It’s worth remembering that the Three Mile Island accident was largely caused by a pressure relief value—a safety mechanism meant to prevent certain types of overpressure failures.

So, faced with the certainty of failure in our systems, what can we do about it?

Accept that, no matter what, your system will have a variety of failure modes. Deny that inevitability, and you lose your power to control and contain them. Once you accept that failures will happen, you have the ability to design your system’s reaction to specific failures. Just as auto engineers create crumple zones—areas designed to protect passengers by failing first—you can create safe failure modes that contain the damage and protect the rest of the system.

If you do not design your failure modes, then you will get whatever unpredictable—and usually dangerous—ones happen to emerge.

'Coz sharing is caring

By Swatantra Kumar

Swatantra is an engineering leader with a successful record in building, nurturing, managing, and leading a multi-disciplinary, diverse, and distributed team of engineers and managers developing and delivering solutions. Professionally, he oversees solution design-development-delivery, cloud transition, IT strategies, technical and organizational leadership, TOM, IT governance, digital transformation, Innovation, stakeholder management, management consulting, and technology vision & strategy. When he's not working, he enjoys reading about and working with new technologies, and trying to get his friends to make the move to new web trends. He has written, co-written, and published many articles in international journals, on various domains/topics including Open Source, Networks, Low-Code, Mobile Technologies, and Business Intelligence. He made a proposal for an information management system at the University level during his graduation days.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.