Archis's Blog

September 14, 2011

Windows 8

Filed under: Technology — archisgore @ 12:27 am

Disclaimer: I work for Microsoft. Full-time. Apply all your bias/prejudice/loyalty/whatever filters while you read this. Also, I have intentionally avoided getting “insider info” on Win8, which means what I know at this point, is much less than what the general public knows.

Most of you are aware of my lack of love for many things Microsoft. I am happy with what I saw today, and I am in love with what the Windows team has built, and how fast they built it (24 months ago, we had Win7.) Any programmer worth his 2 cents has to know how tough that is.

When I first joined in 2006, I joined in the middle of what was heating up to be Vista. It was a different time and age in the company but there was certainly energy. I wouldn’t say much else, but there certainly was energy. A Windows that was 6 years in the making. A lot of new stuff.

Back then, and coming from heavy Linux usage for five years, I could have commented on MANY, MANY things that could have been done differently. A sort of “wake up, there is an entire world outside of Microsoft” call.

Windows 8 though – blew my mind. On more fronts than one. They seem to have picked up literally every piece of goodness I loved about all the companies/products I’ve admired and made one big culture/product/direction change – the development models, the quality bars, the communication and announcement model, the feedback model, the structure and discipline, the humility, the confidence, and the list is endless.

On the first count, the team is structured – intensely focussed, small and targetted teams, determined to win, and dedicated to a quality bar (as Sinofsky commented in his keynote a few minutes ago, much to my pleasure, “we are bound by quality, not time.”) Everyone knows what they are doing, and why they are doing it, and how it fits in for the customer. The team is also, unlike many other (or perhaps older) MSFT orgs, not looking for technology resume-fillers. Today’s keynote was about what the whole “thing” can do instead of how cool their new memory-management system is. Windows Live was about how your photos are accessible instead of how their data-access APIs are serialized on-the-wire. Plus, demo’s, not hypothetical scenarios for some  feature nobody else will have in 2016. It’s out there. It’s now. You can download it yourself in less than 10 hours. Now that’s something that I can feel proud of, and brag about as a fellow employee. The best way to respond to a critic is to give them a URL, not predictions or prophecies.

On the other hand, unlike a certain other structured company out there, the team can’t be accused of not being accessible to you either. Developer preview goes out tonight. Feedback is welcome, and will be listened to, acted upon. While the above reminds you of Apple, this part distinctly reeks of the open development models out there. I think this is the perfect balance between the Cathedral and a Bazaar. A purely democratic Bazaar leads to little polish. A Cathedral doesn’t scale. There are multiple official blogs where the teams are listening for feedback. Discussions that criticize the product are not banned or hushed-up.

However, and this is the cool part, if you do want to know about the actual APIs and the on-the-wire formats, it is coming, and it shall be given too. I’m just very excited how the scenario was the primary driver. In my opinion, when I think about whether or not I want to go on a certain platform, my first instinct is “who the heck will use this?” While a lot of people will disagree with me, and if you do, you may be making a big mistake, developers don’t choose a platform because it provides dynamic typing and annotations, or XML-defined UIs. Developers use platforms because: a) Their customers will use it and b) They can delight their customers with what they can build even if it needs to be written in Assembly Language. They will build runtimes or platforms to ease their life, sure. To steal and paraphrase a quote from Jurassic Park: “You can’t contain developers. Developers… will find a way!”

Just because I hate maintaining hyperlinks, you can figure out everything here: http://www.buildwindows.com/

I’ve spoken to plenty of people all over the company, and I loved the fact that Win seems to be the least obnoxious (if at all) of the teams out there. If you have opinions, they want to know. If you have issues, you will get guidance. If there is a decision, you will hear a justification of why they came to a certain conclusion, and if you can show a flaw in their logic, are open to reconsidering that conclusion. If you have nothing but praise, they will humbly thank you, and tell you that there is more that needs to be done, and get back to their coding. :-) That’s one admirable team, and I am proud to work beside them.

September 1, 2011

On code reuse and maintainability

Filed under: Science, Technology — Tags: , , , — archisgore @ 12:32 am

A wise man once said, “Any procedural program, given a sufficient level of complexity, will end up implementing some form of Lisp.” (If you can’t find a paraphrase of that quote, then attribute it to me – but I’m pretty sure I read it somewhere 10 years ago.)

Today, we continue on the rant against frameworks, and look into code reuse and maintainability. Unlike regular posts, this is one area I’m not too sure about, and would love comments or counter-examples. Spare no punches!

What is reuse exactly? Syntax or semantics?

Let’s start with an example. Given a procedural language, I write a framework to do the following. Instead of the coding having to decide at coding-time what operation they want to do on a float, they can delegate it to my run-time operation-definition framework that allows data-driven dynamically loaded operations.
1. The code itself will look like this:
y = f(x)
2. The executable will add a configuration file for my framework that says:
<define function=”f” definition=”System.Math.Sin”/>

This one raises a lot of questions that I want to ask, but for now, I’d like to know, would you call this code reuse? I can certainly make great arguments for this style of coding (keep quiet functional-programmers, this one’s for someone who _has_ chosen a procedural language.) It allows me to change the definition of ‘f’ at runtime. I don’t have to worry if tomorrow my computation changes, because a simple config-change will make my code work for Cos, or Tan or whatever else the user needs. I can replace the definition of “Sin” to use a different implementation whenever I choose.

For me personally, this is bullshit! It’s the worst kind of code I would ever had the displeasure of dealing with. Only the last argument made any sense, and there are ways around that. It is the most irresponsible style of coding too – the programmer, instead of taking responsibility for ensuring correctness of code, delegates every function out of their own scope. If you really want to do that, use a functional language already! People have been advocating them for over four decades now, and this is exactly the reason why! Stop contaminating my procedural code with a smarty-pants half-assed implementation of something for which robust implementations exist already. You can replace your code at runtime and any interpretor worth its two cents has a decent JITer. Semantically, what would be the difference in sending the interpretor a new file to interpret, versus changing configuration for a running program? y=f(x) is certainly not going to have bugs (and can be tested easily.) Your probable bugs are going to be inside ‘f’ anyway. So while your core ‘executable’ can be assured of being stable, it’s a false perception.

The problem with this snippet, functional language or not, is that instead of ensuring correctness, it actually reduces it.

For one, what you see above, is an example of syntactic reuse. You are reusing the syntax for making a function call. You may disagree with me on this, but to me personally, what is really valuable isn’t syntax reuse but semantic reuse. Implementing a good Sine function is damned difficult. That’s what I want to reuse. Calling into the Sine function isn’t what I worry about when I open up my editor. The correctness of my Sine function is what I want to reuse. If there’s a bug, and someone fixes it, I want the new Sine function. If I may ever need to use a Sine function from a different library/implementation, well, seriously – change references to the new library and recompile your code (I know I’m making some atrocious demands here.)

The second problem is the really serious one. When I’m writing code as y = f(x), what the heck am I thinking? I mean seriously. If I am writing a program, I write it for a specific purpose. If I’m computing some vector component across one axis, I know why I’m computing it along that axis. Which means, when I write f(x), I had better damned well know what that ‘f’ should be. If that ‘f’ is ever going to change to ‘g’, then that’s because my problem statement has changed. It alters completely what I am doing (two axes are never the same.) If I start computing Cos(x), it is very very different from computing Sin(x) and I would have serious justifications for why I want the Cosine now. I sure as hell don’t want to reconfigure a running program to do that. I may do a host of things with a running program – use a more accurate Sin implementation, use a faster Sin implementation. If I’m fundamentally changing the definition of the function, I’m in big problems from the outset because I’m changing what my code is guaranteed to do.

Copy-fidelity

I know a lot of people don’t consider the fidelity their code preserves when it is xcopied from one place to another, but I assign it a very high value. The problem with the above snippet, is that 9 out of 10 times, someone’s going to only pick up the code file, without caring what the configuration file is. This is certainly not unreasonable, regardless of how senior or experienced you are. If you see a file called “eigenvalues.java”, you think to yourself, “Hmm…. maybe I should copy eigenvalues.java and use it to compute my eigenvalues.”

I find nothing wrong with this thinking. Very soon though, you see a compiler error: “Function ‘f’ not found.” You spend a couple of days (and if you’re lucky, you’d find a comment) figuring out that you have to host this class file in another loader called ‘function-replacement-framework.jar’. No big deal you say, I love myself some helper tools! This is when you go mad. “functionReplacementParserError: Please define ‘f’ in the configuration file.”

Now you are, in effect, figuring out how to compute eigenvalues so you can define what ‘f’ should be. I’m sorry, but you really should NEVER have to write a config file to define _what_ your code does! Regardless of how much computer-sciency stuff your parser and interpretor are doing, and how they’re generating binary classes at run-time which are processor-optimized by the JITer, this is some pretty bad design there.

Maintainability

So I’ve been thinking about this for a few weeks now, and want opinions on. I came to a good working definition that I think I’m going to use for a while in the near future. I draw this definition from how real-time systems are defined.

A brief overview for the uninitiated – while the common-sense notion of real-time systems is ‘they’re really really fast’, the working definition is ‘time-deterministic’. Meaning that, given what they do, they must do it in deterministic time – meaning, they have to guarantee that something will happen at (‘by’ if you’re soft-realtime) a specific time. Even in light-hearted situations such as filling ketchup in a bottle, you need hard time-determinism, to ensure the bottle is under the nozzle ‘at’ a certain time, not ‘by’ a certain time (if it passes from under it too early, you’ve got a mess to clean up.)

In a similar fashion, I was wondering if maintainability is “doing less work”, or “ensuring correctness even if it is more work.” If I had to make the choice between say…
1. Being able to replace all definitions of “Sin” with “Cos”, using a single config change, which is admittedly a lot less work, but depends on the hope that everyone has taken care to handle the cases where x=0,
2. Or replace Sin with Cos in all the code manually, which is a lot more work, but you ensure that that specific place really is worthy of a “Cos” function as per your replacement-intention. This would guarantee determinism in correctness, but increase work significantly.

If the above sounds unlikely I can certainly come up with more concrete stuff. We frequently use stdout and stderr to send output of a program. Would you prefer to control what goes where implicitly, or would you prefer to modify code yourself (there’s a difference between code consciously having an if-then based on a configuration parameter, vs, your code just having fprintf(outstream, “<stuff>”) where you don’t know during code-time what outstream could be, and can’t function without a complex config file.)

I intentionally chose the stderr vs stdout example because it is vague. I can see it from both ways. In some cases I am conflicted as to whether a warning should go to stderr or stdout. It depends on what kind of tools are going to capture the output, and how they may want to parse/interpret it (some tools, for instance, may consider anything being spit to stderr as indication of program failure.)

How would you define maintainable (or is it maintenable) code? How do you draw the line between abstraction of details vs. core purpose of the tool which is what makes the tool what it is. Does configuration at some point, become complex enough that you’re really just programming in a declarative language through your config files, whereas your “code” is simply an interpretor at that point? If so, is it configuration any longer? If declarative code is more maintainable, why not use a declarative language from the ground up?

August 22, 2011

Why doesn’t anyone speak of Law-enforcement in India?

Filed under: philosophy, Politics, Preaching — archisgore @ 2:13 am

I wrote once before about Raj Thakery and had a lively debate with a friend. My objection was simple – he broke a law, and he was not held accountable for it. For all my admiration for Batman, I certainly respect the fact that he is always portrayed as an anti-hero by the writers. A moral conflict that is never really quite resolved.

This month it is the Jan Lokpal Bill in vogue. I understand that the laws we have are not sufficient. I understand that there is a need for anti-corruption law reform. But what we really, truly, desperately and urgently need, is the ability to enforce the law. I took some time to go over the Jan Lokpal Bill discussion on the site, and most of it sounds like one of my algorithm-proving design documents (first we send an integer as a parameter, then we check the integer for non-negativity, thereby ensuring it is always >= 0.) My computer, fortunately, is 100% reliable in performing that integer non-negativity check, which is why the algorithm works. That isn’t how India works though. We have rules. We have the law. It isn’t being followed.

I wrote before about Middle-class morality, and how we like to believe we’re better than everyone else. There is a  reason why Gandhi was such a hard-ass on most people. Before criticizing someone else, he was willing to criticize himself. After he was done with identifying all his own faults, we would narrow down on exactly what it is he wants out of others. The fight with the British was distinctly different from what is happening today, and it didn’t quite begin as an Independence struggle. If I remember correctly, “Quit India”, was a  late-comer to the party once he realized every other recourse had been attempted and proven unsuccessful.

The original conflict, the same as in South Africa was, “If we are under the British rule, we are British citizens. Consequently, we must be subject to British rule. We must enjoy all the rights, previledges and responsibilities of British rule. We want British rule. We _demand_ British rule to the last letter of the law.” This isn’t all that different from the American argument either. It started with demanding a colonial representation in the British Parliament, we they were to pay British taxes. When the Brits refused, did they make demands for independence from the British rule.

What is happening in India is similar, but not the same. Unlike the British, there are constitutional laws designed to protect us from corrupt officials. There is law designed to achieve a lot. What is lacking, is the implementation of the law. Now logically speaking, if the law were implemented, we would never find ourselves in this predicament to begin with. The Parliament represents the Will of the people. Some would claim that isn’t true. I had an argument with my mom about this yesterday, and I must disagree. The Prime Minister is right. The Parliament has no obligation to table a bill they don’t want to. If that is not the will of the people of a constituency, the MP would be afraid of not being reelected. It is a self-regulating system. The interpretation against Anna Hazare, that they are holding the country hostage is not all that far-fetched. Democracy is a hard pill to swallow when it doesn’t go our way.

Enforcement:

The first thing I did to uncover more, was to read the FAQ on the website www.indiaagainstcorruption.org. One of the more disturbing statements on that site was, “The government’s agreeing to Anna’s demands was a democratic (not coerced) victory because the tiny fraction of the people of India who participated in the action represented the hearty desires of the masses against corruption.” I find that statement a bit presumptuous. I could say Osama Bin Laden’s tiny fraction who participated in the action against the US represented the hearty desires of the masses. In fact that’s exactly what Osama himself claimed. If we are to be a democracy, and a representative democracy at that, we can never presume anything. If an inconvenient law, rule, measure is taken against us, we must live by it because that is what we are signing up for. First, we must look at the election process – a lot of measures which benefited the masses have made it for 60 years. If the masses want it, it happens.

So either of two things is happening. Either election process is screwed up – in which case, if the very laws that are supposed to protect our democracy are not enforced, what makes you think any array of new laws would be enforced? Or, as the government is saying, the lack of introduction of the bill is, in fact, the will of the people.

There’s a reason I say this. Are the masses truly against corruption? Do you know that there have been plenty of law-enforcing people I’ve seen in my life, that the masses have kicked out of office? Arun Bhatia is my classic example. We speak of lack of infrastructure in our city of Pune (where I was born.) We speak with cynicism of all that could be done if the city planning laws were followed. Arun Bhatia became the commissioner of Pune for exactly 48 hours. As soon as he took office, and began enforcing the law, the masses – yes we’re speaking of the masses – the common masses – the regular middle classes, lower-income classes, upper classes, etc. – EVERYONE kicked him out of office in 48 hours. What does that say about the will of the masses? In fact, if I had heard that not a lot of people were against corruption, I’d readily believe it. To think that people from my city of birth are against corruption is a joke! Did they hold agitations to enforce the city planning laws? Did they hold agitations to enforce laws against a certain faction that went into people’s homes and beat them up? I know hundreds of “middle classes” that lie and cheat on their income taxes. What right do you think they have of holding someone else accountable? Why should a politician be treated by a different law? Why must he not get the right to lie and cheat? Why the double standard?

What I’m leading into, is this – if we do pass a bill, will it be enforced? What happens when hundreds of these middle classes are caught in land-deals or property purchases whose value is not honestly declared? What happens when hundreds of these people’s undeclared income is brought under investigation? Will we ignore it? Will we demand that it is inconvenient to us, and it must not be enforced?

This is a very real and dangerous possibility, that may bind our country in chains for another century with a big grand farce. Do the masses with the candles on the streets realize the consequences for themselves? This is not about the Members of Parliament or the Chief Ministers or the Prime Minister. This one’s going to hit home, and hit us all where it hurts. Illegal land deals. Illegal constructions. Illegal electricity bill manipulations. Illegal cooking gas cylinders. Undeclared incomes of doctors, lawyers, businessmen, farmers. I’m not sure all those supporters have thought this through. The fight is politically and diplomatically framed against politicians, and we’re all up in arms “against” an entity that we have clearly bounded and defined. I know plenty of people who have moved vehicles across state boundaries without paying the proper taxes. It is because of a corrupt cop who is happy to take $2 that they avoid paying heavy fines. Would they really want that corruption gone? When it starts to hit us, we’re going to demand leaner laws. We’re going to have talk shows and debates about confiscating whether a poor farmer’s undeclared income is ‘fair’. We’re going to cave in. And like the host of other laws that exist, we will have another one that won’t be taken seriously.

Visibility:

Visibility is a big part of the Jan Lokpal bill being promoted. Allegedly, it will allow corruption to be brought to light, which implies that we don’t yet know that India has corruption. Do you really buy that? Seriously? So you’re sayimg, there is corruption in India, not because it is not prevented, but rather because people don’t _know_ that it happens? Are you kidding me?!

To prevent the Jan Lokpal from abusing its powers, there will be populist measures like video recordings of meetings. We’re back to the point above. Do you think I don’t act against police abuses because I don’t know they happen? Do I not prosecute people in power because there is lack of evidence? So that given a video recording of a misdeed, I’m better equipped to fight irregularities there? I’m afraid of the kind of world we live in, if this belief is widespread.

I can’t prosecute those who abuse powers, because I don’t have the means to. Whether I have a video tape or not, I don’t have the time, resources or guarantee of remaining alive long enough, to fight a case in the courts. Visibility was never the problem. I remember in my own University, I had evidence of breaking of rules and regulations by the director. I have a hatred for that entire institution not because I didn’t have video recordings, but because I could find nobody who would act on the evidence that did exist!

Such populist measures frighten me because if I ever do have a grievance, I will have that video recording thrown in my face, and told to shut up because it shows no irregularity.

Enforce, Enforce, Enforce:

I love the support this issue is getting. I know people are pissed. I know people are frustrated. We’ve had enough. We taken this for over six decades. More bills and laws and authorities are not the way to go. We need to enforce what we have. Enforce laws. Enforce rules. I’m totally in support of adding new laws. But do so consciously knowing what we are all giving up. The sacrifice isn’t the fasts we’ll do, or the protests we’ll hold in the safety of American cities (for us NRIs), or the candles we’ll light. The sacrifice we’ll give up is our own little bribes we’re so used to.

We must ENFORCE! if this is ever to work. Sadly, I’m not convinced that’s going to happen, but am holding out hope that I be proven wrong. I’ll get back to you in one year and we shall see where we stand.

August 5, 2011

BCI updates 3

Filed under: Brain-Computer Interface — archisgore @ 2:50 am

What’s happening on the BCI front….

Well, I have sufficient data from myself. I’m looking for subjects who wouldn’t mind me taking some of their recordings for an hour or two worth of data (and while I shall try to maintain full anonymity, I’m not making any such promises outright.)

I’m spending lots and lots of time reading for the next 3 weeks. I realized just how rusty I had become. Fixing it. Current reading list includes large-data pattern mining. Bayesian decision theory. I always go through “Introduction to Algorithms” once every year (and I hadn’t read their latest edition.) General math and calculus.

Fortunately thanks to library deadlines, I have to finish all this before the 9th of August.

Also, my dad’s a statistician, so I’ve sent some of the BCI data to him in case he can find a good classifier.

This is the non-exciting phase of any project. I know what to do. The blockers are gone. It’s just week after week of thankless work that does not demonstrate linear returns. You can help by joining in the effort and keeping me going.

July 21, 2011

Philosophical rant against frameworks

Filed under: Uncategorized — archisgore @ 4:15 am

I began this rant in one of my older BCI-related posts, about why I’m not going to use “Frameworks”. This is a first-class post into why I hate most frameworks and platforms out there, and why people should really just take it up with their companies how they can improve the promotion-systems if writing small tools doesn’t seem “fancy enough” (seems to me like either an HR problem for workers, or a faulty evaluation system for academicians.)

So what’s wrong? Some frameworks are good. Some are great even. I wouldn’t think of doing matrix algebra without Matlab (and I appreciate the fact that they allow us to write plugins to add new features and extend it.) I love how I run most of my code above the .Net CLR and it is “managed” for me. No more processes gone awry. Very few memory leak opportunities. Great library to work with. Same can be said for the JVM, or Python.

If you notice, each of the above is a platform with very little (or none) missing pieces. However, every so often, I come across a platform or framework, that is composed entirely of missing pieces. Something very typical I commented on in that BCI post is as simple as file-copy. Back in college, if I needed a file-copy, you’d see someone building a five-line program to copy files from one place to another. A good program would have buffering, network FS mounting, etc. etc. What it would still not have, and this is something that would never even cross the mind of the developer, is to make it a “data copying platform, where file-copying is a subset of the functionality.”

So what’s wrong with building a platform beyond your core problem? Lots. First, if you don’t know a requirement, you can’t plan for it. I’m making that a very hard statement. When you read the above, you have no idea what that “data” is going to be. If it is just binary opaque data, then it maps to file-copying. If it isn’t binary opaque data, then you have no clue what the requirement will be. So you end up providing features for extending events such as “precopy”, “postcopy”, “copyprogress”, etc. Chances are (and I’d call it a near-certainty) that when that specialized data really needs to be copied, your provided hooks are at best completely useless (in which case, a decision not to use your platform is easy to justify), or they miss some last-mile option which requires a complex hack. I’d bet almost everyone reading this has used at least one fancy solve-it-all framework which lacked one key feature, which required elaborate mechanisms to fit within that framework. If the framework were a tool instead, you could write your mechanism around it, instead of within it.

Taking the file-copy example above, I could generalize that task to something even more abstract. “Operation.” Instead of a file-copy program, I write a program to “Do Operations”, one of them being “File Copy”. So I’d have a framework for operations and each OperationProvider must provide a well-defined entry point called “DoOperation”. That way, when I need to do any other operations such as “FileZip” in the future, I can just write a plugin and leverage my framework. Either you keep the parameters to this entry-point super strict (a list of files), or you keep them super-generic (a Variant, or Object.) Either way, there is one common theme that begins to emerge. What you’re writing is an operating system shell!

The shell provides a generic method for “Doing Operations”, which are represented by distinct plugins called “executable files”, which are loaded dynamically at runtime, whose parameters are generic but are strongly type-checked within, and each plugin provides an entry-point, and blah blah blah…..

What I’ve observed is, 9 out of 10 frameworks, over a few iterations, end up becoming shell replacements. There is only a percieved benefit of abstraction and you have none of the loose-binding power of the system shell. Whatever features the framework itself was intended to provide, remains a small part of the system, largely superseeded by a magnitude of plugins, which now can’t be used for other tasks because they depend on the whole framework loading mechanisms.

I’m not without my own skeletons in the closet. During college, and in my first three years at Microsoft, I’ve tried every single time, (and once almost got away with it), to build these generic monsters. It’s very tempting really. When you’re working on a problem, you start to see generalizations. To be fair, a lot of them are real, while a small number of them are percieved - or you can just make something up like “Do Operation” which is so generic, you can fit anything in there. There’s a certain elegance in the Unix-style design (and even today, I won’t use a machine unless I have trusty VI and GREP installed.) The fact that tools are just that – tools. They do stuff they’re supposed to do, while remaining agnostic to the environment they run in. They can be combined or reconfigured to build bigger tools. And they depend on a framework too – the system shell.

Any modern OS comes with a shell capable of powerfull script execution. The fact is, platforms only work when they specialize more than the shell. The specialization has to be fairly targetted. The .Net CLR collects garbage. It is a framework that provides garbage collection to programs. The garbage collection itself is not arguable. The Type system is not up for plugin-based changes. Matlab’s language is immutable. Its matrices are not what you can plugin and modify.

The deeper problem that once begins to face very fast is that of coupling. Imagine a complex workflow where you define a billion operations that depend on a million other operations. Sounds like a perfect problem for “Framework Man” (if I were any good at photoshop, you’d see a superhero here.) In fact, that’s what most frameworks do. That generic “DoOperation” wrote about above isn’t as apparently ridiculous to others as one would think. A lot of what I’m about to say depends on interpretation, so many of you may disagree.

For the layperson, think of the coupling as how under the hood of your car, your sparkplug depends on the cylinder size you have, and your cylinder size dictates the power you get, and also the fuel injection, and also the the cooling required, and the cooling system eats some of that power, so that takes away a little from what you can use to run. The actual situation is a lot more complicated. Now here’s the thing with cars or operating systems – people fight tooth and nail to reduce coupling as much as humanly possible, and then some. Unless absolutely justified without no way out, you want to reduce coupling. I’m not exaggerating how seriously this is done.

Problem with that file-copy example I gave above is, it seems almost intentional, or at worst a tongue-in-cheek joke at how many people may end up using it because it provides: 1. Filecopy and 2. Extensibility.

Problems with coupling are well-known. One thing depends on another, depends on another, depends on another…. It is a nightmare to deploy, understand, maintain and most importantly fix! You see, if I found a better tool for file-copy, the best I’d do in an OS script is modify parameters to that tool. For a framework, I either write a shell wrapper (if so, what was wrong with a shell script anyway?), or modify the code to be hosted in it, and if the code is in a different language than what it supports, use another framework to write marshalling code around it.

I think what I’m saying here is, management and tasks must never mix. If you need to manage and handle complex workflows for file-copies, that’s fine. Have a framework that manages tasks. Let each of those tasks be handled by the native OS. Even if you can write a fancy-powerful task, write it as an independent tool that can be plugged out and replaced or reused. Call it on the shell. That way, others can quickly reproduce any problems they face with that task in the future when they need to debug. Fixing the task if code isn’t available, is as easy as finding another tool and fitting it in. Undoing the action of that task are simpler. These scenarios aren’t as far-fetched as one would think. Imagine if my n’th task in a workflow were writing registry keys, that my n+1th task were reading. If my n+1th task fails, it’s quite a nightmare to reproduce that failure without creating another workflow that can feed the whole chain (or mocking the task prior to it and hosting the whole framework beast). Trying different parameters involves running the mock workflow over and over again, and depending on what kind of checks, initializations it does, it’s a massive waste of time (is it too hard for the tool to first check for updates, then check for updates to each component, then checking security keys, etc. etc.?)

Small distinct stateless tools, data piped by the OS, a lightweight scripting langage for glue, and console control FTW!

June 21, 2011

Fact

Filed under: philosophy, Science — Tags: , , — archisgore @ 1:14 pm

This one comes from long introspection. Over the course of my career, I’ve been involved in sufficient debates and arguments where a misinterpretation of semantics has lead to difficulty in communication. I try to be very precise in choice of language and generally demand that others do the same. Any student of mathematics ought to appreciate the value of accurate semantics, and interpretation of data. However, not everyone is a mathematician, and predicate logic isn’t for everyone.

Last weekend I had the opportunity to meet someone whose opinion I value highly, and someone I hold in high regard in terms of intellect. Those discussions helped me formulate this with precision.

Regular readers will know that I frequently do change my opinions and definitions, or the premises upon which I build arguments. Therefore, if this conflicts with any previous posts, I would be glad to have that pointed out to me, as a reminder that I am in fact, learning.

I’ve been in plenty of debates where I try to get my opponent to state their premises. In many cases, they find me rude for interrupting them, because when I find a premise inaccurate, flawed, or misstated, I feel the need to get that clarified (and I forsee the need to write a post on how a good premise build up could make arguments efficient.) In many cases, premises are assumed to be facts (or at least, in a good argument they should be.)

So what is fact? I’ve heard plenty of opinions. I’ve heard people say, “What is fact for you, may not be fact for others.”

I think it’s worth dedicating a paragraph to clarify that fact is not Truth. Truth is best left for philosophers, and perhaps even theologists. While it is common that mathematical truth is based on fact, we shall not open that can of worms here.

Going back to what fact is. I must heavily insist that I disagree with the comment, “What is fact for you, may not be fact for others.”

For one, if that were true, there would never be progress in the debate, and there would be no point in holding a debate. As I understand it, a debate operates on information that is an order higher than fact. It operates on two things: Interpretation of fact, and Opinion. I won’t go into analyzing opinion here.

After various attempt at defining fact, I arrived at a definition that fits the need quite snugly – Fact is observation (as in a scientific experimental observation.) There is a reason fact may not be truth – and that’s because the observation process may contain a flaw.

First lets look at whether it satisfies the rigorous requirements of building a premise.

1. An observation is, to put it in terms of signal processing, an ‘impulse response’. An observation is the effect a process has on a quantity you can measure. An observation is, whether correct or incorrect, indisputable by parties in an argument. Let me explain. If we were debating why a certain number of jobs are reduced, we may disagree on the reason of decline, however, our observation cannot change.

2. If the observation does change based on who is making the observation, it is all the better. Now we reach the very depths of predicate logic. The reason some arguments seem to go around in “circles” is not because two parties have incompatible observations, but because different parties have incompatible processes to make those observations. The power of defining fact as an observation comes into forcing both parties to agree on the process used to make the observation. You may argue that such deep definitions are irrelevant for a debate. I disagree. I argue that almost all debates are very simple and can follow simple First-Order-Predicate-Logic if we agreed on our observations. The real debate is in the details – can we agree on observation? Here we reach an opinion (and my job ends.)

So why this clarification? I strongly believe, and have observed, that many of the truths we cling to depend on our personal convenience (misquoting Obi-Wan a bit.) We try to appear to debate on a higher-order plane because it is a convenient arena where interpretations and opinions can be inserted without premises.

You can’t really disagree with observation. You can disagree with the process used to make the observation. A classic example is clinical trials. Common arguments about drug-approval testing go, “While the FDA thinks the drug is unfit for use, they don’t know the whole story.”

The fact of the matter is, the FDA isn’t interested in the whole story. They know the story they care about. Their process is well-documented and available for peer review. It is open to criticism and suggestions for improvement. Whether or not that process is followed for a certain clinical trial is also a matter of opinion (and to a large extent quantifiable.) However, having raised no objections to the process of the trial, nor any concerns regarding its execution, there really isn’t a different observation you can arrive at, and therefore, the fact cannot change for you.

A final question I want to address is, “Is observation good enough?” Observation is the “effect” a process is having on a measurable quantity. That is what I meant by ‘impulse response’ in DSP terms. Observation is all we really have. Almost all of science depends on observation. Do not misinterpret observation as ‘seeing’ (as in, ‘to see’.) Observation is quite literally measurement of an effect. When we observe gravity, we observe the effect gravity has on an object (change of force, shape, etc.) Even if you strive for truth, you will end up at a dead-end – observation.

June 17, 2011

Civilization

Filed under: Uncategorized — Tags: , , , , , — archisgore @ 2:52 am

I reaffirm what we already know of western countries, or should I say, developed countries. I’m sure different people will have a different comparison point, but I’ll explain mine.

I come from a remote village in India – one of those places where there used to be one bus that ran a few times a day into the city, and we had 4 hour power cuts daily, and one full day of power cuts. A small island-like place in between 3 rivers which, when flooded, would cut off a square-mile region from all civilization for a couple of days. If we got rain, the transformers would frequently blow, and we’d get power back the next evening. That was just routine life – it takes as long as it takes.

Today morning (well at noon), Redmond lost power. Outside by window a block from my apartment, I see a thick black column of smoke. A fire of proportions that clearly indicated a week’s outage in my timescale, but adjusting it to “America”, I estimated the next day.

I went to work at one hoping to camp out there as long as I could, and when I found office had no power either, decided to drive back. This was barely an hour after I lost power. On the way back, the traffic lights were already working. When home, I found power back on. Where I come from, this would be considered a miracle. If I were to narrate this back to the same place, they’d laugh in disbelief – it would sound like a ridiculous hyperbole from some “americanized desi who thinks he’s better than us.”

Also, when it happened, I could see a response in minutes. I still recall those phone calls we’d make to the power company back in India, only to get a very rude lecture (if ever they picked up) on how ungrateful I was for the little power that I did get during that day. There were cops on the streets, the traffic maintained control despite traffic lights going out, and well, civilization kicked in.

Pretty amazing how we set our expectations huh? This probably means I have to go back to work though…

June 9, 2011

Brainwaves are beautiful!

Filed under: Brain-Computer Interface — archisgore @ 9:53 am

There’s something so mesmerizing about looking at a real-time FFT of your own brain-waves. I found a good plotter library and with a sliding window of 64 samples (approximately the last second of data at 50Hz sampling,) I am seeing not only my own brainwaves being plotted in real-time (well, 5 times a second at least due to the bad code I wrote,) but I see a direct correlation between limb movements and the spikes. I’m plotting it too fast to actually make any sense of the spikes and I presume a lot of the energy is useless as a discriminant of individual limbs. I also notice a rather large DC component which I intend to get rid of.

For now, I’m just excited. What was five years in the dream-list, an year into planning, and about three months in coding has finally worked! I’m going to continue staring at them tonight just for fun.

Some upcoming experiments will be performing the FFT over larger windows (and higher frequency resolution.) Just lots and lots of raw recordings which I could move inMatlab and perform analysis there to compute a baseline. Recording at different states – sleepy, tired, excited (which is now), hungry, thinking, not-thinking, depressed, etc. Finally, perhaps even recording all night to find validate pop-culture know-how about Alpha, Beta sleep patterns.

If you want recordings, ping me. I can’t say for sure what kind of experiments I’ll perform, so it may be a bit random, but it’s a lot of data per second, so I’m going to run out of space real fast.

For now… live long and prosper!

June 6, 2011

First sanity test of EEG data

Filed under: Brain-Computer Interface — Tags: — archisgore @ 12:25 am

It’s been long since I last posted on what the BCI stuff is leading to. The delay was partly due to my random travels which seemed to have peaked out (I missed a lot of flights on my latest trip, and that’s an indication for me to stick around for a while.)

A main reason was trying to find a good source for capturing raw data. The situation wasn’t as hopeless as I initially thought – it appeared that there’s very few drivers that can read PendantEEG data. However, digging a little deeper, I found plenty of open-source projects to pick from, all of them on Google Code (I’ve been out of this world so long, I still default to sourceforge for all my needs.) A combination of legal and practical issues required plenty of thought. My binding contract at Microsoft requires that any code I write, is owned by the company. The boundary between personal pet projects, and moonlighting, is somewhat complicated. Let’s not go into the details (but you can ask me if you’re curious), because I’m eager to get onto all the latest cool stuff I’ve been doing.

I really loved all the stuff out there – BioEra and Pyofeedback, but they were somewhat too deeply integrated and exposing them as COM objects, or re-writing them to work on my box with minimal libraries would have taken a lot of work, and even if I could do it, would have put the whole thing in a legal mess of ownership, copyrights, and my job-security. The tool I settled on is PendantEEGToolset, which reads data and exposes it over a TCP channel. A TCP boundary is as safe as you can get in terms of de-coupling, and any code I write, is now owned by MSFT but at least it isn’t contaminated with GPL code. That means it’ll take some time getting out, but I can go about doing it “right”.

I finally was able to plot (and .Net isn’t good for fast graphing out of the box) signals and get garbage. When developing the drivers, I first tried to capture data without plugging in any electrodes to test the protocol works. I was a bit apprehensive because without electrodes, instead of getting a stream of zeros as I had expected, I was getting large steps and large falls. The FFT showed large numbers which made no sense. I wrote a signal generator that allows me to generate a custom signal and fake the protocol so I could ensure by FFT was running correctly.

However, when plugged onto a human being (and my first test subject was a visiting couchsurfer – with his consent of course), the readings normalized and I was able to observe a change in pattern in the time domain (I still haven’t been able to plot my FFTs fast enough, so will have to read the numbers tonight.)

All in all, a very successful sanity test. The device works (so far), and I’m capturing data. More next weekend.

June 2, 2011

Astrologers…

Filed under: Entertainment, philosophy, Science — Tags: — archisgore @ 2:10 am

Did I spell that right? I’m supposed to write a document that’s going to take 2 hours, and I’ve pushed it too far. Good time to get all my thoughts out to the world one at a time. Today, let’s rip on Astrologers a bit.

To give you an idea of the motivation, I picked up a hillarious book at the airport during my last India visit called “Am I a Hindu.” That’s going to lead to a few posts, but you’ll have to wait until the next time I run out of things to do, and face the inevitable document-writing task. Today, I began reading this book to take my mind off some blocking issues. I had read a part of it during my flight, and I recalled an emotional rollercoaster between humor, apathy and perhaps anger (or annoyance). I’m giving you this context because this post is regarding one argument that book made (I’m willing to discuss other arguments.)

Lets get this out of the way - lack of disproof, is not a proof. I’d love to talk to anyone who believes that isn’t true (that was sarcastic; if you think lack of disproof is proof itself, I probably don’t ever want to speak to you in my life). The Indian Government proclaims it’s a science, and I claim I’m king of the world. Neither of the clauses is relevant for this discussion. A common argument we hear from pro-Astrology people is, “Why is it so difficult to believe stars could affect the physical processes within you?”

It’s not difficult to believe at all. I never claimed a remote planet doesn’t have gravitational influence on me. I’m claiming you’re full of crap. When I rip on Astrologers and Prophesizers, I’m making fun of them. I’m claiming they’re full of bullshit. It’s about them! That’s as direct as I can say it (offense intended). I have no problem believing that we might be able to model those interactions, and what the result of that influence would be. I’m not saying it can’t be done. I’m saying you’re not the ones doing it.

“If people can predict weather, why can’t people predict fortunes?”

We could extend this argument infinitely – If people can predict weather, and people can predict fortunes, why can’t people predict when we’ll get a man on Mars? If people can predict weather, the stock market, the next Tsunami, and some Earthquakes, sure, it may be possible to predict fortunes too. Doesn’t change the fact that you’re not the one to do it.

I think the modern Astrology debate has gotten too impersonal. Perhaps we’re trying to be too politically correct, or the Astrologers are just better at reframing the problem than we are at noticing that it got reframed right under our noses. I believe in open-heart surgery, however, if you’re any one of the people reading this blog, I can safely say I’m not letting you come anywhere near my heart. If you tried to convince me, I’d find it midly humorous and highly annoying. It’s the same with Astrologers – science doesn’t deny modelling. Modelling is a fundamental tenet of Science. What we’re saying is, you’re no good at it, and that you have no idea what you’re talking about.

It’s true that all models are merely approximations. That’s why in addition to predicting weather, weather-predictors are also constantly ‘learning’ from the outliers. They’re on the search for new variables, and better sampling methods. I’ve not seen major publications that have indiciated the discovery of any new variables or processes, or models that provide a better fit than what historic Astrology demonstrates. Even if that’s accomplished, a theory that does not demonstrate a prediction record significantly higher than a random process, is not considered a theory at all. Are astrologers willing to submit themselves to a controlled experiment where they can demonstrate their predictors are any better than a random predictor?

Hence the title of this post – ‘Astrologers’. Don’t make this about the “Science of Astrology”. I don’t claim it won’t work. This is about you – I claim you don’t work.

« Newer Posts — Older Posts »

Theme: Silver is the New Black. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 122 other followers