I wrote once before on how people hide behind the word “process”. Once you define a “process”, you can now overlook the outcomes – even when they are statistically defying your expectations. I wish I could give a whole lecture on “expectations” someday, but I’m too cynical for that. As a side-note, some of the most annoying conversations I get into is when people don’t know the semantic difference between desire, expectation, and outcome.

Today’s observation is about something similar. It is partly about critical-thinking (the word is too MBA for me) and partly about some common sense.

We all love formulas. They’re easy to follow, easy to justify and impose no responsibility upon us. They’re easy to follow because well, it’s a formula. It comes with a manual. Easy to justify because of the responsibility angle later. Also because statisticians can prove anything (my dad’s one so I say this with some authority.) You can pick just the right sample, and the right variables in the data, and most people will never wonder if you left out any variables or what sampling method was used. Which is why these articles will never say, ‘n traits of all successful CEOs’ or even ‘n traits of most successful CEOs’. There is never a reverse-correlation either – if you picked a sample of people demonstrating those traits and followed them through life, can you demonstrate a higher chance of success?

Finally, formulas free us from responsibility because…. “formula!” One doesn’t see such people in America, but in India I used to find it difficult to get a real opinion out of anyone at all! If you asked what the best restaurant is, you’d be told some restaurant is rated highly on yelp. If you asked what algorithm to use, you’d be quoted a paragraph by Knuth where he briefly mentions why he preferred said algorithm. Why are we doing something this way? Because that’s the process to do it.

Anyone who took modelling 101 in college (mathematical modelling, I mean; anyone who took runway modelling 101 is probably not in the same world as the people reading this blog) would tell you that processes and models are always subject to intense scrutiny and that’s the only way to do it. We know that even today, occasionally people will post about observations that don’t fit predictions made by Einstein’s model. Why does such news gain mainstream attention, despite it being proven incorrect a few weeks later? Because Einstein himself was interested in defining a model that predicts verifiable accurate outcomes, and not just to define a model with his name on it.

The point I’m trying to drive hard is that a process that is not exactly followed to the letter (the letter can always have acceptable tolerances to deviation) is a useless expense. It costs to implement a process, and a process is created with some assumptions in mind. If you break those assumptions, change the process instead. A process almost always has a desired outcome. For instance, a process for clinical trials, always desires to have zero false positives in approving drugs. Expectation is what we realistically expect the false-positive rate to be.

Why is Process important?

This is one thing that never really got discussed during those Software Engineering courses. Why are processes even important? If it were simple distrust of an employee in the chain, there wouldn’t be such a heated debate over it. That’s why bureaucracies have processes – to keep checks and balances.
I can’t look back and remember anyone ever coming up with a very good example for why process is important beyond just making every employee go through the loops. Are there scenarios where analysis of the process itself is a valuable tool in quality? Turns out industrial manufacturing depends on that!

There are some people who claim to hate “process” when in reality they probably hate bureaucracy. They are called “mavericks” by the bureaucrats who really can’t remember why something was put into place and whether that is still relevant.

Even those who don’t like “process”, are still following a process. They do open up an IDE. They still do follow some coding convention. They still do breathe and eat and sleep. They do take a break to think about a sorting implementation before coding it. (When someone says a particular person doesn’t think things through that’s not quite literal – else the sort just wouldn’t work. Even to copy a sort out of a textbook, there is still a process there – you need to find a quality textbook, you need to adopt the function from whatever pseudo-code to your specific language/framework/platform.)

So lets look at some classic simple problems which simply cannot work without a model and a process. These extend to software development as to anything; hat-making, event-hosting, party-hosting, cooking, etc.

Back in my college days (and I think it’s even MORE relevant today), we’d have a problem of figuring out the lifetime of a light-bulb, given that we need to get it into the market in six months. It’s really one of those fun academic questions as well as a practical one. An LED light has a stated life of upto 40 years. I sure as hell know they didn’t have LEDs back in the 70′s and even if they did, I know they didn’t have time for a repeat experiment.

So how the hell DO they do it anyway? What kind of advanced algorithms and advanced physics and advanced time-travel is used? The answer is pretty simple – they control the quality of the process used to manufacture the light, and rely on the process quality to produce a quality outcome. This is where modelling of the bulb helps. When you model a system, you create an equation or description of it’s observable behaviour. Knowing the chemical/physical degradation of each element under various circumstances, allows us to make predictions of how said element will behave under circumstances it will be exposed to during its lifetime – the air pressure, ambient temperature, sunlight, humidity, etc.

Really the only  way we have of making any sort of reasonable guarantee on the lifetime of that bulb depends on the quality of the process. This is a very real problem. We pay good money for those things, we sure as hell expect them to last for 40 years. But we also want a decade-old LED technology in our homes TODAY!

Another example is the classic Hollywood military/heist/intelligence scenario. Let’s say you have  10 planes ready to “scramble” at the first sign of attack at an airbase. You want to prevent the enemy from scrambling their planes. The amateur aims for taking out the pilots or sabotaging the planes – which are probably very highly defended just against such amateurs. However, a small weakness in processes can be exploited better with less risk. If there is only one attendant who fuels the planes, or if there is only one guy who uses his flags to guide the planes out to the runway, there’s a higher chance that fellow is much less protected than the pilots.  Of course modern models build in redundancies in all situations, but these have been flaws exploited by some of the greatest military generals or thieves or spies throughout history. This has happened already.

The same applies to software. A process doesn’t need to have a documentation stage or a high-fiving stage. There are no traits or “key principles” or “top features” that make a process effective. What makes it effective is when each stage is put in place with only one objective in mind – that the expected outcome moves closer to the desired outcome. That only comes through conviction of why you’re putting it in – copying a self-convenient trait from an arbitrary sample of arbitrarily defined successful people will do little to improve your quality.

They are invaluable in predicting and preparing for events that are difficult to reproduce. They will also fail occasionally, and so long as the failures are within expected limits (either number of failures, or deviation from ideal quality, etc.) everything’s fine. Achieving perfection is practically impossible. As you move towards the right side of the normal curve, the law of diminishing returns begins to work against you. Each linear gain in quality, requires an exponential investment of resources.

Why is knowing  important?

This is mainly for the bureaucrats. Why is understanding all this important? I’ve been told before to keep my mouth shut and do my job – just make sure you follow our established procedures and also deliver on time, and make sure there are no bugs. For one, it builds trust, and for another, it allows people on the front-lines to improvise.

I’m taking a scuba diving course this week and while I was skeptical of the 300-page book I have to read, I was sold when my instructor told me that we have to go over exactly how each piece of equipment operates. Because when you’re a 100 feet below the surface, knowing WHY you’re doing something allows you to respond with higher quality when something does go wrong.

I had to mention this, in the interest of fairness, that I’m highly impressed at Nokia. If you believe this hurts their reputation, you probably don’t understand the consumer very well. I’ve personally worked in teams which were squeamish about their product, and in others where responding fast was the culture. I assure you owning up to a problem and responding to it head-on may seem like a dumb move for a slick used car salesman in “business school”, but I’ve been involved in forums and blogs where I know it to work wonders.

Nokia has done what Microsoft is probably incapable of pulling off. When a problem arises, I believe there are two ways to go:

Either you just don’t say anything – and this works if you’re the incumbent (Apple in iPhone 4, MSFT in Vista.)  Overtime there is enough uptake that you don’t have to defend your position anymore. You just quote “It is installed on 90% of desktops” and don’t elaborate on that. There is nothing wrong with this approach, because when you’re on 90% of desktops or 90% of phones there’s going to be a few of truthers uncovering conspiracies that don’t exist and you don’t want to encourage the perception that anyone can make stupid claims and the company will spend time and money responding.

The other way to deal with this is head-on. By responding on your own terms rather than being cornered into doing it. That way, you control the narrative and you control the perception. That’s how Google got their market-cred amongst geeks. Rumour was, Google’s “feedback” form never went into a black hole. Issues were responded to head-on.

Nokia’s timeline of events is one that has left me in awe. They launched a certain phone on one Sunday. By Monday, people had “perceived” some issue with connectivity. By Tuesday Nokia had figured out what it was, given a 100$ credit of good-will to each buyer, committed to delivering a fix by the following Monday which can be downloaded, or if you don’t even want that, you could just outright replace your phone with a fixed phone. I’m sure this came as a pleasant surprise to everyone who bought the phone even if only 5% faced the issue (and incidentally helped avoid any numbers or percentages from being discussed.)

I think if Nokia ever does come back from the dump, instead of all the trash-talking and iPhone funerals by Microsoft, I’m going to mark this specific moment as the inflection point when it all changed. If I ever wondered “Why Nokia? Couldn’t anyone make good quality phones?” I just got my answer. Nokia because… this! I don’t know of any other company who could have pulled this off.

Instead of having weird arguments or being cornered by journalists, and before there were 10 articles that pointed to “Nokia’s Data Connectivity Problem”, in about 24 hours, there were a thousand more which said, “Lumia 900 practically free. Minor issue, but fix coming in 5 days. 100$ rebate.” Instead of a blog post by “General Manager of Nokia Lumia 900 customer experiences division” assuring us of “appropriate action being taken and a fix coming soon”, and got the entire company giving us a coherent line: “Nokia has noticed. Nokia has the fix. Plus here’s a good-will gesture.”

Even on technical grounds, it’s fairly impressive. There are a handful of companies on this planet who would figure out the software bug in less than 24 hours, and commit to delivering it to all handsets before the end of the week.

I now believe any OS could piggyback on this company and gain a good marketshare, and remembered why even I, before the iPhone, was a hardcore Nokia phone user all my life. The moment they match my phone spec-for-spec and add software that matches my phone spec-for-spec, I no longer have concerns about being left to die in the wilderness for lack of support – if Microsoft gets upto their usual antics, Nokia’s got my back.

It’s a silly thing to stand out, but a lot of times, you come across non-information or non-opinions. There’s speech. There’s words. There’s even text or written language. Its effective information content is zero.

Every notice when someone comes up to you, presents an idea and says, “We think this idea may or may not work.”

And you’re staring back and nodding seriously, but you’re thinking of Steve Martin in your head and trying not to laugh. “Hmm… let me see here. That’s a tough one… it may work, or it may not work…. Wow… Need to really think of this here. Boy you’ve really narrowed it down to some close contenders, I’ll give you that.”

Background: Today morning Seattle City Lights politely sent me a mail saying, “You’ve paid your light bill successfully. Thank you.” I rarely ever open those mails, but for some or the other reason, I clicked on it. The initial reaction was shock, and then panic. My bill was almost $900. After some careful analysis, I realized a large part of it had to be my stupid circuitry. The amount of load my wine heater was drawing from mains was… let’s just say, ambitious. (Cops, if you’re reading this, you’re welcome to inspect my home as many times as you wish. I have no meth lab going on here. If those white vans across the street are of surveillance teams, could you come in? I’ve been dying to join some awesome secret organization, but they never seem to want me in.)

After the initial jerk, I decided to go out and replace each and every one of the light bulbs in my house with CFL variants. I emptied out QFC and my bulbs won’t end. I never knew how much crap I had in the house. At last count, I have eight100-watt lights from the kitchen (replaced with 12-watt equivalents), and get this – eighteen 60-watt bulbs including the bathrooms and kitchen (I have some elaborate bathrooms with decorative light-strips like the ones found in movie-star dressing rooms.)

WHOA! So everytime I was in the kitchen, I was burning 800 watts. Every time I was brushing, I was burning nearly 600+ watts. My 65″ LCD TV runs at about 600 watts!

Forget the long-term savings, just replacing everything with an, on-average, 15-watt equivalent brings the consumption down to 400 watts overall for the entire freakin’ house if I’m extravagantly keeping every single light on. And that’s still less than what I was burning just while brushing.

More updates to come. I emptied the local QFC and replaced only about half my apartment. There’s still at least 15 or so 100-watt bulbs that could use replacing. Out I go…..

Always good indicator of Microsoft’s product quality is the employee’s Facebook feeds. Managers at MSFT don’t have to panic – none of my friends leak any information.

For anyone with plenty of time and academic curiosity to test my hypothesis, they should go analyse the Facebook/Twitter timelines of employee posts. One of the things I learnt the hard way in my almost-five years there is that very very few Microsofties are what you might call – true academicians (though those that exist are the ones behind what you see succeeding today.) My respect and admiration for Terry Myerson is a matter of Google’ing my past posts, and I’d dare you to find anyone of the thousands of people to whom I complain about WP to, to quote even a single instance where I have spoken against him.

The trend there is to throw around the word ‘innovation’, ‘synergy’, ‘dynamism’, etc. a lot and quote past examples, and find new process names to try. It’s pretty annoying to hear a softie promote their product at times (there was one blog out of the thousands who really questioned whether all softies speak the way Charlie Kindel does – most do in fact.) I have many analyses as to why most Microsoft people appear to be totally out of touch with reality (heck I worked on Messenger and Hotmail – you don’t even want to go there!) The problem is, for a large part of the last decade, all you had to do was build stuff, and it’d get a cool few million users without any effort and you post-justify success with MBA-speak. Windows was the vehicle. You slap some WPF on there, and it inevitably gets picked up. You never have to convince or sell developers on, how or why, WPF improved the things you could do compared to other alternatives.

Which is why when WP was announced, the first thing they did was to hold an iPhone funeral. My FB feed was practically filled with death-threats. Try and suggest one feature missing in your Windows Phone, and you get attacked with all kinds of straw-men. The same rhetoric over and over distills down to one or more of the following:

1. Whatever you ask for is irrelevant. Yes, it’s only perception that you need a compass in maps. People got along fine before, and people will get along fine without it. Don’t be such a prick and feel entitled to anything. We don’t owe you anything, so don’t suggest features.

2. The iPhone never had it in 2007. (Not sure what they try to say here, but in their world apparently, that’s a defense)

3. Maybe it’s not for you (and this is said with some kind of pride of elitism – we’re the 1% few who appreciate the true value of a magical world-saving device)

4. It’ll be successful in 2015 (this one really confuses you – I could go on and on about how utterly ridiculous it is to expect me to pay $300 bucks for your phone NOW because it’s 3rd model four years from today will leap frog the competitions model then after you’ve paid for a couple of additional $300 upgrades.)

I got sidetracked. My apologies. As I was saying, the academicians – let me clarify here. There is a mistaken belief that Academicians are people without purpose. Heck they know purpose and they know money. What it really means is people who are dedicated to building a good product, just ‘because’! In its day Microsoft did come out with some pretty good stuff just because – COM being the best one I can name right now. Even Windows Mobile was pretty impressive for its time.

I think at this CES finally, there is a Windows Phone device that really makes me “think”. I’m not saying I’ll buy it – because I recently threw out my Focus for a 4S (yeah, if you made me pay $300 bucks for a shitty device, I’m waiting till a WP offers me TWICE as much as my iPhone to make up for the difference. Fool me once, shame on you, fool me twice, shame on me.) But for once, I see a device I would have been genuinely (as opposed to defensively) proud of, if I worked at either Microsoft or Nokia.

Many of you may not know this, and it seems like a pretty childish thing to do, but one way to get softies to shut up about their ‘superior’ OS for the past year, was to carry a few hundred dollars or a couple of blank checks in your pocket. Anytime someone used the word ‘superior’, I would pull out a check for $700 bucks and put my iPhone on the table and ask them to give me their replacement. Don’t argue with me. Don’t debate with me. You know better than me. Here’s my money. I’m waiting to be impressed.

An indication of good times though, is clearly from the lack of those outrageous death-threats flying around on FB this time. After CES, they have gotten factual, and that’s a mighty good sign. The N900 actually looks pretty good, and if they can fix the app situation (which is still bad) they just might do well. I was also impressed at HTC adding the 16mp camera. Now don’t you go telling me it may not add quality. We dished it out to them when they defended lack of dual-core, and we must face it. When it comes to specs, a WP has the most powerful camera I’ve heard of in a phone.

The software though really has to deliver fast and has to finish off the remaining ‘magic’ of the equation. Apps, including Facebook’s own, must provide full-fidelity (a concept I learnt is very important, and I learnt it at and from Microsoft.) Skype MUST match the Facetime/iMessage magic. If not none, then at least minimal, signin dialogs, and stuff. Just detect what’s on the other side, and enable video and Messenger IMs without the two users first having added each other as contacts on Messenger or Skype. If they pull that off, and do it fast, and make sure they remain feature-compatible with other phones – they just may expect my few hundred bucks for my next upgrade in 2 years.

I do see a good change in attitude. Instead of speaking of killing things and being the No 1 in 10 years, they are slowly learning the reality that others have pretty decent products too and will probably be here for a while. Making a good product and selling a decent amount of phones in the short-term may not be a bad compromise to dreams of world domination in four years.

After five years of watching shitty products being promoted with zeal, I’m happy to see great products promoted calmly for what they are – simply good products. I’m impressed!

I was stuck trying to build my other wine-heater thingy in the meantime, and the job and house move was keeping me busy. Beginning next week, I’ll get back to working on the promised library to read data and allow any apps to process it. My particular interest is in plugging it into sci-lab if I can, so people can have a free signal-processing environment to process PendantEEG data and experiment with it.

That’s all for now, folks! 

As is bound to happen, I began building something of my own. It’s not until you do the math that you realize just how much specific heat water has. It’s a miracle we get it to boil at all. Pumping in that much energy into a thermodynamic system that fast was a problem. My heating elements would have to heat up too rapidly and I needed some way to control them.

The next step was obvious. A trip to radio shack, an Arduino and a fast-switching MOSFET, turned into a pulse-wave modulated heating element  - beyond which the original intentions of heating wine fast no longer mattered – I was chasing some crazy high.

I ended up building a device that gets multiple feedback readings plots them on a logarithmic function, and injects heat at the rate at which said liquid can absorb safely, without the boundaries over-heating, and can compute the rate of heat loss from the system through radiation or evaporation (before you ask, yes, at that point I was caught up in the feature-hole – just adding stuff because I could.) Microcontrollers in the hands of a software engineer are dangerous. Once you have one, you find yourself with limitless power and the ability to add frivolous sensors and features, for cheap.

Anyway, a quick survey among friends (which is a pretty biased sample – since they always tell you what you want to hear)  indicated that it may be quite a nifty little thing for home use. There are plenty of liquids that needs to be regulated in a small temperature window. Thermostats are typically fairly brutal in their operation – they turn on and suddenly heat up your elements rapidly, and abruptly shut off. You can’t really regulate them to say turn on 1/5th of a second, each second, so that you don’t get the massive temperature variance.

It’s been quite an adventure, and I practically burnt through my life savings. I am a trained programmer, but electronics has been a hobby. I’m worse still when it comes to fabrication. Fabrication is an art, and as such cannot be objectively studied. As much as I’d like to be, I’m not MacGyver. My initial tools of choice were test tubes, and beakers – stuff you’d find easily, but not stuff that makes a good product. I learnt a hard lesson – material science is a “thing”, and one that deserves more respect than I had given it for a long time. Finding a material with all the thousand properties that make a good food-safe barrier between your electronics, and… well, food, is a challenge.

So I stand here today – owner of the world’s arguably most advanced device ever possessed for the purpose of heating wines – and one that I dare not use, for fear of accidentally starting fires.

As I mentioned above, when you really do the math, it takes about 4 KJ to heat 1 liter of water by 1 degree C. Assuming you want it at near boiling (say 95C) from room temperature of say, 30C, you’re looking at a 65C difference. You need about 230 KJ to heat it up. That’s about 230 watts per second for a thousand seconds (about 16 minutes) not counting any heat losses. No way do I have the guts to  run that kind of power through anything I built at home. I have some experience with microelectronics, but those who handle power electronics are in a different league altogether. It’s a wonder we live such safe lives given how much power circulates through our grids around us at every moment.

I am however, looking for investors. I’ve filed a provisional patent, and would welcome any contacts, startups, large companies, kitchen appliance companies to revolutionalize the world of… heating wine?

Until my next project then….

An early behind-the-scenes video. The high-power circuitry comes next, and I’m not confident I won’t blow up my board though I’m taking every precaution to protect circuits with sensitive fuses. Time will tell. There isn’t anything here you can’t build out of the box. I’d say a college-student is much better equipped than me – I actually forgot that connecting resistors in parallel reduces overall resistance, and am glad to say the fuses came in handy.

http://youtu.be/YDDQ_LFfHm8

On a related note, Windows Phone just wasted the last hour trying to make the video upload and Zune’s clunky confusing interface, made finding the file location difficult when I managed to copy it with physical thethering. I could go on and on about the entire series of steps I followed (but MSFT folks know me all too well, a factual argument is one they can’t win – I’m sure I’m looking at this from the wrong “lens” or “prism” or whatever.) God damnit, I spent my savings on this project which felt more important than moving to the iPhone ASAP. Glad I got that out of my system. Phew.

Anyhow, I learnt increasing an order of magnitude for an NP-Hard exponential problem doesn’t help (an N+1 input will kill you anyway.) Couldn’t think of a probabilistic method in the 2 days I spent on it. There’s a couple of optimizations I wasn’t able to do due to the DS I choose. A sparse grid or matrix would have been the right way to go here, but reverting state is difficult on those. The one thing that would speed this up further is to proactively look at the shape of the unfilled area in the candidate box, and find one of the unplaced rectangles that would never fit in that shape. That would trim a large number of candidate boxes fast.

For what it’s worth, here’s the dirty code. More of the BCI signal-processing code to follow over the weekend. Here’s the file: RectanglePacker

Quick overview – languages have rules, structures, etc. Sometimes, the rules become too complex, or at times, they are so specialized, they turn into a look-up table (i.e., not a lot of generaization.) Whenever you can’t generalize, you add entropy. Putting aside, for a moment, the poetic beauty of a language and the art of eloqution, many rules are redundant.

Consider language as simply a tool, a means to an end rather than the end in itself, designed to express a thought. If so, the less ambiguity something has, the easier it is, and the better it solves its purpose. When one first begins to learn computer languages, or even when they think of “parsing” English, every single person that I know goes through the thought process above. Why not just have a language that isn’t as nuanced? Why not design a simpler language? Esperanto certainly came out of a need, but building out a complete new language may not have been the solution. It appears that the need is already being met by modification to English itself.

I am beginning to believe that the very efficiency computational linquists want in a simple-to-parse language, is also the kind of simplicity the human brain wants. There is a certain idea you want to express. The nuances of whether I will do something, as opposed to whether something will be done by me, while undoubtedly helpful, may not be as necessary as we think. Facebook/Twitter are helping reinforce that idea. If you look at most non-proofread contemporary speech, it almost feels like a context-free language. It appears that what NLP wants, NLP may end up getting, simply because what makes NLP so hard is what also makes language itself so hard for most people.

Texting is the classic blatant example. Most texts are simply a gathering of words put together. There is a certain amount of context and syntax present to avoid ambiguity, but overall, the tools used to elimiate ambiguity are the ones that can do it in as blatant a way as possible, with as little simplicity subtlety as possible. Similarly, few FB/Twitter posts seem to be carefully crafted treatises, but generally just words that present an idea. The less context necessary for the idea to be parsed, the better it is communicated. Five years ago, a lot of “old school” people, including me, would complain of the utter lack of punctuation in sentences. Instead of adapting punctuation correctly, I found that people learnt to phrase their text in such a way that addition of commas and full-stops became unnecessary. A modern FB post is as decipherable without punctuation, as it is with. That’s some creative adaptation, right there.

Another reason for this is search engines. Very rarely do you search for something like, “Give me movie times for today evening in Redmond.”

The same idea is expessed as simply as, “Movies redmond today”

Over time, it is not difficult to imagine this is how I might begin communication with a friend. Even the verb is implied and not explicitly stated! The parsing rules for this language are just ridiculously simple – tokenize the sentence, and you know what it’s saying.

Then again, I’m not blaming the internet or machines for this phenomenon. I think it is simply been the first time that a large population the entire earth is literate (even 30 years ago, when I was born, I knew plenty of people who couldn’t read or write.) Written language was, no matter how many people may dislike this, an elite previledge – and to some extend, an end in itself. When you are a club of handful people, you can end up in an ego-pissing match. What we might call spoken ‘peasant’ language was always utterly simple and efficient (though I find a lot of ideas I cannot express to them due to the lack of a vocabulary that can convey subtle differences.)

I’m not advocating anything here, but we have to admit that any complex and large system always tends towards reducing entropy over time. It does not mean literary art will have no appreciation, but it is an interesting thought. This would be an interesting hypothesis to test out, if only for the academic validity of the idea. Is modern human language finding a path towards reduction in the energy and ambiguity required to express an idea? Is it a dual-feedback loop where NLP systems are getting better with feedback, but also driving certain generalizations back into the human world?