Your facebook wall is filled with articles about “<n> traits of successful <x>”, where n is in natural numbers, and X is an arbitrarily defined category to keep n small enough to make it look exclusive, but large enough to retain the illusion of deducing some kind of pattern from it. Not a day goes by that I’m not quoted one more self-help book or biography-of-famous-person as some kind of formula or pattern.
I wrote once before on how people hide behind the word “process”. Once you define a “process”, you can now overlook the outcomes – even when they are statistically defying your expectations. I wish I could give a whole lecture on “expectations” someday, but I’m too cynical for that. As a side-note, some of the most annoying conversations I get into is when people don’t know the semantic difference between desire, expectation, and outcome.
Today’s observation is about something similar. It is partly about critical-thinking (the word is too MBA for me) and partly about some common sense.
We all love formulas. They’re easy to follow, easy to justify and impose no responsibility upon us. They’re easy to follow because well, it’s a formula. It comes with a manual. Easy to justify because of the responsibility angle later. Also because statisticians can prove anything (my dad’s one so I say this with some authority.) You can pick just the right sample, and the right variables in the data, and most people will never wonder if you left out any variables or what sampling method was used. Which is why these articles will never say, ‘n traits of all successful CEOs’ or even ‘n traits of most successful CEOs’. There is never a reverse-correlation either – if you picked a sample of people demonstrating those traits and followed them through life, can you demonstrate a higher chance of success?
Finally, formulas free us from responsibility because…. “formula!” One doesn’t see such people in America, but in India I used to find it difficult to get a real opinion out of anyone at all! If you asked what the best restaurant is, you’d be told some restaurant is rated highly on yelp. If you asked what algorithm to use, you’d be quoted a paragraph by Knuth where he briefly mentions why he preferred said algorithm. Why are we doing something this way? Because that’s the process to do it.
Anyone who took modelling 101 in college (mathematical modelling, I mean; anyone who took runway modelling 101 is probably not in the same world as the people reading this blog) would tell you that processes and models are always subject to intense scrutiny and that’s the only way to do it. We know that even today, occasionally people will post about observations that don’t fit predictions made by Einstein’s model. Why does such news gain mainstream attention, despite it being proven incorrect a few weeks later? Because Einstein himself was interested in defining a model that predicts verifiable accurate outcomes, and not just to define a model with his name on it.
The point I’m trying to drive hard is that a process that is not exactly followed to the letter (the letter can always have acceptable tolerances to deviation) is a useless expense. It costs to implement a process, and a process is created with some assumptions in mind. If you break those assumptions, change the process instead. A process almost always has a desired outcome. For instance, a process for clinical trials, always desires to have zero false positives in approving drugs. Expectation is what we realistically expect the false-positive rate to be.
Why is Process important?
This is one thing that never really got discussed during those Software Engineering courses. Why are processes even important? If it were simple distrust of an employee in the chain, there wouldn’t be such a heated debate over it. That’s why bureaucracies have processes – to keep checks and balances.
I can’t look back and remember anyone ever coming up with a very good example for why process is important beyond just making every employee go through the loops. Are there scenarios where analysis of the process itself is a valuable tool in quality? Turns out industrial manufacturing depends on that!
There are some people who claim to hate “process” when in reality they probably hate bureaucracy. They are called “mavericks” by the bureaucrats who really can’t remember why something was put into place and whether that is still relevant.
Even those who don’t like “process”, are still following a process. They do open up an IDE. They still do follow some coding convention. They still do breathe and eat and sleep. They do take a break to think about a sorting implementation before coding it. (When someone says a particular person doesn’t think things through that’s not quite literal – else the sort just wouldn’t work. Even to copy a sort out of a textbook, there is still a process there – you need to find a quality textbook, you need to adopt the function from whatever pseudo-code to your specific language/framework/platform.)
So lets look at some classic simple problems which simply cannot work without a model and a process. These extend to software development as to anything; hat-making, event-hosting, party-hosting, cooking, etc.
Back in my college days (and I think it’s even MORE relevant today), we’d have a problem of figuring out the lifetime of a light-bulb, given that we need to get it into the market in six months. It’s really one of those fun academic questions as well as a practical one. An LED light has a stated life of upto 40 years. I sure as hell know they didn’t have LEDs back in the 70′s and even if they did, I know they didn’t have time for a repeat experiment.
So how the hell DO they do it anyway? What kind of advanced algorithms and advanced physics and advanced time-travel is used? The answer is pretty simple – they control the quality of the process used to manufacture the light, and rely on the process quality to produce a quality outcome. This is where modelling of the bulb helps. When you model a system, you create an equation or description of it’s observable behaviour. Knowing the chemical/physical degradation of each element under various circumstances, allows us to make predictions of how said element will behave under circumstances it will be exposed to during its lifetime – the air pressure, ambient temperature, sunlight, humidity, etc.
Really the only way we have of making any sort of reasonable guarantee on the lifetime of that bulb depends on the quality of the process. This is a very real problem. We pay good money for those things, we sure as hell expect them to last for 40 years. But we also want a decade-old LED technology in our homes TODAY!
Another example is the classic Hollywood military/heist/intelligence scenario. Let’s say you have 10 planes ready to “scramble” at the first sign of attack at an airbase. You want to prevent the enemy from scrambling their planes. The amateur aims for taking out the pilots or sabotaging the planes – which are probably very highly defended just against such amateurs. However, a small weakness in processes can be exploited better with less risk. If there is only one attendant who fuels the planes, or if there is only one guy who uses his flags to guide the planes out to the runway, there’s a higher chance that fellow is much less protected than the pilots. Of course modern models build in redundancies in all situations, but these have been flaws exploited by some of the greatest military generals or thieves or spies throughout history. This has happened already.
The same applies to software. A process doesn’t need to have a documentation stage or a high-fiving stage. There are no traits or “key principles” or “top features” that make a process effective. What makes it effective is when each stage is put in place with only one objective in mind – that the expected outcome moves closer to the desired outcome. That only comes through conviction of why you’re putting it in – copying a self-convenient trait from an arbitrary sample of arbitrarily defined successful people will do little to improve your quality.
They are invaluable in predicting and preparing for events that are difficult to reproduce. They will also fail occasionally, and so long as the failures are within expected limits (either number of failures, or deviation from ideal quality, etc.) everything’s fine. Achieving perfection is practically impossible. As you move towards the right side of the normal curve, the law of diminishing returns begins to work against you. Each linear gain in quality, requires an exponential investment of resources.
Why is knowing important?
This is mainly for the bureaucrats. Why is understanding all this important? I’ve been told before to keep my mouth shut and do my job – just make sure you follow our established procedures and also deliver on time, and make sure there are no bugs. For one, it builds trust, and for another, it allows people on the front-lines to improvise.
I’m taking a scuba diving course this week and while I was skeptical of the 300-page book I have to read, I was sold when my instructor told me that we have to go over exactly how each piece of equipment operates. Because when you’re a 100 feet below the surface, knowing WHY you’re doing something allows you to respond with higher quality when something does go wrong.
