http://news.cnet.com/8301-10805_3-10139597-75.html?tag=mncol;txt
…. and the winner is Live Mesh! It’s always good to see such recognition for stuff one works on. Follow the link above to read all about it.
January 13, 2009
January 1, 2009
Live Mesh nominated for TechCrunch Best Technology Innovation/Achievement award
My first post in the first few hours of the new year starts off with some encouraging news. Live Mesh has been nominated on TechCrunch for the Best Technology Innovation/Achievement award. Always feels good to find what you work on is still relevant out there to a lot of people.
If you feel Mesh is something that deserves this award, please do vote for it on the TechCrunch site. You can vote once per IP every 24 hours.
And here’s wishing you a very happy new year. May the force be strong with you…
June 8, 2008
College project idea: Failure-tolerant systems enabled by "sync"
Back in the days of DOS, shutting down a machine was merely about turning off the power, and at times, directly the mains. One didn’t have fancy file systems that needed maintaining consistency and atomicity. If something was bad, you just ran chkdsk or scandisk to fix it. Consistency was maintained very optimistically – we just assumed the best and prepared for the worst. If files got lost, we would have xcopy’d backups elsewhere and we’d compared the whatever little was saved against our backup and do a merge….. sounds very familiar doesn’t it?
Then came relational databases, normalization, concurrency control, etc. All kinds of fancy computer science was dedicated to complex programming techniques and systems that were debugging nightmares. As if designing and maintaining a system that acquires locks over time, then makes changes, and rolls back failed transactions wasn’t fun enough, we went ahead and created distributed versions of those systems. Either everything was in realtime, synchronous and consistent, or nothing was happening at all. But this can go only so far – trying to maintain redundancy and failover in such a complex system isn’t just non-trivial, but impossible on the scale the web today requires.
Then again, all the million people on your site don’t really operate on the same data at the same time – and perhaps such tight locking across all resources wasn’t only unnecessary, but also inefficient. Perhaps you could have two broken databases that would periodically exchange information with each other about the changes they underwent in the meantime. After many years of research in locking and concurrency control, and write-ahead logging, and such stuff, we ended up with the same model we used in the early 80′s – sync!
It’s not only a lot simpler to implement, maintain, fix, patch, test, but also a lot simpler to understand and use as well. Moreover, every user, every device, every application, has it’s own version of their personalized history. It allows every user to view what’s going on in the world from their own point of view (Obiwan Kenobi, et al).
Yesterday evening I met a friend who is in India for a couple of months and the HoD of Fergusson College’s CS Dept over dinner. We began discussing the kind of cool projects kids could do for fun over the summer, and Atul mentioned beginning by creating a simple MPI cluster out of all machines in the lab (for the uninitiated, for kids in B.Sc. computer science, it would be a first in the history of the course.) Since we’ve both worked-on/interested-in loosely coupled massively scalable distributed architectures (and my project at ISSC was to attempt to build one called YODA), we all began pushing our agenda (before you assume anything, “sync” wasn’t really my original agenda).
The outcome came to be something so awesome and cool that we’re meeting up on Monday (16th of June 2008) to discuss something constructive. As with anything I do, I’m on the lookout for enthusiastic crazy kids who want to do something fun just for the heck of it.
Mrs. Page, the HoD, wanted some real tangible experience for her students. I still have the itch to complete YODA to fruitition – I still believe it was the most awesome project I ever worked on with the best project group in my life with a lot of potential. Atul has his interests in high-perf computations for language processing (the perfect candidate for MapReduce).
The idea is exactly what YODA was all about, but perhaps at a less ambitious level – handing off the network/node/data management jobs to MPI (yes, we take the hit to have a geographically localised cluster). To have a thin “loading” framework on top (my agenda) so that you can submit “jobs” (concepts stolen from BOINC) which may be of the MapReduce type (Atul’s interest), but with one major twist to it – instead of the conventional master-slave topology, why not go for a peer-to-peer sync topology? (perhaps working on Live Mesh has made me a religious sync fanatic)
Look at it this way – today, MapReduce/BOINC keeps working if any/all worker nodes go down. What happens if the entire data center went down? Including all your master nodes? The last few months, if Ray Ozzie has taught me anything, it is that you can add all the UPSes you want and all the security guards you want, but the only way to truly protect data is to simply replicate it as far and wide as possible. Its cheap, it’s effective, and it’s simple. GFS (Google File System) claims to do it. Then why not do it for compute-intensive tasks to share merge results?
Let’s say a BOINC-like cluster where every machine is a peer. You “inject” your task onto one of these machines and based on availability of bandwidth the task starts replicating across all machines like a torrent (the parts that have not gone out before, go out first). When any node finishes it’s processing, it simply “sync’s” the results with any peer nodes that may have completed results. When it has any meaningful results that it can merge (“Reduce”), it does so, or it acts as a proxy to sync those results further through the swarm. (You may notice this follows a lot of principles of git as well.)
Instead of being “assigned” a workunit by a master, each peer simply picks up a random workunit out of the ones it currently has but has no recorded solution for. When it gets more than one unit that is mergable, it merges them. This is like dynamic-programming on stereoids. If the bittorrent implementation is done right, each node should have as minimum overlap as possible, and yet, we do need some overlap which will be a function of the design as opposed to any deliberate actions by the master. (The overlaps are used to validate results in case something went wrong.)
Just imagine if your college lab turned into a swarm of independently operating units who are sync’ing with each other and processing results, and when you go at any node in the system, you have part of the answer. Depending on how long you give it, every node will have the final answer in the end. You can naturally optimize by assigning a “special” result node which periodically gets partial results from all boxes in the swarm and thereby create a weak master.
The good part is, if you have multiple problems to solve you just go on injecting them into any node you have accessible. Over time, you go to any node you can find and get results. All without any fancy coding to maintain consistency, accuracy, integrity, etc.
Naturally, there may have been things I’ve overlooked, but for a 10-minute discussion, doesn’t seem like a bad idea for some college students to try out as a few-weeks project. In case you’re interested, let me know. All debates, arguments, flames, criticisms are appreciated.
June 6, 2008
The earnest wait to release…. of movies or software or books!
We all know how eagerly we all waited for the final Harry Potter book. And my friends should tell you my almost maniacal obsession with “The Dark Knight“. And we always wonder why these people project dates that are a couple of years ahead or at least eighteen months. I used to get frustrated and angry at such dates and I also frequently comment on blogs/forums about this issue.
And yet, something different happened a few minutes ago. I was reading about J J Abrams’ upcoming Star Trek movie and it said “eleven months to go”, and for once, I didn’t feel as if this was an irrational timeline. Perhaps because we’re having meetings at work daily planning for the next 18 months.
You really come to realize things when you’re on the other side of the barrier. People keep asking so many queries about Live Mesh, and where it will go, and whether it will do a special feature that they want. And I used to get frustrated – after all, we are working as hard as possible, but it takes time to write code, test it, validate it, approve it, sign it off, push it to a distribution channel, and provide that seamless install experience for users. Suddenly, eleven months doesn’t sound all that much.
And I also got a glimpse of what the producer/director/actors are feeling like. I’m sure they want to get it out as soon as humanly possible, but there are things that just take physical chronological time and there’s nothing you can do about it.
Now that’s something to think about……
April 17, 2008
The world wide techno-mess
As devices penetrate our ever-increasingly connected lives, we struggle all the more to keep them in control. Devices whose purpose is ever-increasingly vague, if not converging. Phones have become portable VCR’s, and palm tops have phones in them, while PC’s are home theatre systems, and I don’t even know what to make of gaming consoles that replace supercomputers!
What makes up the mess?
Devices: As devices converge (or diverge), we acquire new skills to use them effectively. When phones handle your workflows, can transparently beam presentations to bluetooth-enabled projectors, and play your favourite movies on the television, it’s not just enough to simply be able to do these things. Where does your favourite movie come from on your phone? From your PC? If so, which one? Your personal desktop at home? Your kid’s desktop? Your wife’s desktop? Your laptop? One of your many computers at work? Imagine you’re about to leave on a weekend trip and want a fresh movie on your phone. You know you own a few movies, but where are they?
All that thousand-dollar piece of hardware being able to play movies, is worthless if you can’t get your favourite movies on the damn device to begin with. This is just one scenario on one device! Let me make it even more fun – say you were already on your way and had no physical access to the machine on which the said movie existed. Your phone with the fancy 3G/Wifi connections, TV-out and hardware 3d acceleration, just turned into a piece of junk! And we’re just talking about plain old big files!
Data: Extend this to thousands of office documents you need to process. Add the millions of songs. Add the pictures which come from various sources – all your own cameras, family, friends… you get the picture. All these “objects” have contextual data associated with them – data which each device wants to process uniquely. This may include comments, ratings, rankings, tags, etc on your photos or songs. Copying these files around, in the conventional sense of the phrase, is like passing around stone tablets with caveman-drawings on them. Soon, the sheer magnitude of managing this data is overwhelming enough for us to just “give up!” one day.
That was just files. We also deal with abstract “objects”. I mean stuff that can’t be just dismissed as blobs of files – such as email, appointments, notes, annotations, contacts, blogs, etc. My blog is more than just a big text file somewhere – it has pictures, it has links, it has tags. The “blog” so to speak, makes sense only when all these elements are put together. As plain binary data streams, it’s just random noise. Although objects are files at the backend, what really brings them to life are the applications used to view them. This blog entry, without blogger, is junk. A contact, without an application that can leverage it to make phone calls or send mail, is an irritation, if not a gross frustration!
Applications: Features are intrinsically reflected in data (regardless of what certain critics may have to say). Go pick up your phone from 3 years ago and tell me if it can aggregate RSS feeds from your friends – it’s contact storage, no matter how “standardized” it may be, will not include a link for RSS feeds. And I wonder if you had to put in a feature that was never imagined by the original application writers? How do we extend objects to cater to future features?
Connections: And finally, we have links – objects are meaningless without a context. The purpose of these devices, files, objects is to ensure they can be communicated. A phone number is linked to the person who owns it. If not updated regularly, it’s a random series of digits – hardly an appealing feature to have. So does music and movies. They all work a lot better in a social context – what your friends have to say about them, how they rank them, and so on…
Even within the context of applications without the data, maintaining settings and configuration coherence across all of them is certainly a challenging proposition. Imagine taking an hour setting up your video player “just right” for that saturday-evening movie and having them lost after reinstalling it! How many board room meetings have you felt embarrassed in because the settings weren’t consistent with the computer on which your presentation was designed? (this happened to me today afternoon at the most recent)
The popular solution? Web-apps: One solution, and one gaining recent popularity, is to dumb down all devices into a web browser. Turn them all into an HTML reader, move all logic to the cloud, and you’re done. Given the current intractability of the problems, this solution is an appealing compromise, if not the one most desirable. You buy a gaming console which can displace a supercomputer, and you run a web-browser on it! You buy a phone with 3d hardware acceleration, and again, the best you can do is run a browser on it! And have 5-6 computers that you deal with on a daily basis, and you do everything in your browser! Why? Because with one centralized place to store, manage, organize your data and applications, you get convenience.
Are webapps desirable? I doubt it. I admit they may be necessary, but they’re certainly not desirable. Surely with all those “innovation” keywords flying around out there, we could do better? If I buy a supercomputing-capable gaming console, I expect it to do something…super! If my friends come over, I want to show off some of that supercomputability! I sure as hell don’t want to be browsing bland HTML pages and have my friends go, “Eh…*yawn*”
Another popular solution? Closed Ecosystem. Every device you buy needs to be “certified” for every other device you presently own. Given this precondition, everything “just works”. In the worst case you have to physically plug them together with a wire, and in the best case, you don’t care how the hell they do it, but they just talk to each other!
Is a closed ecosystem desirable? Honestly? I don’t care. I’m certainly not philosophical about this kinda thing. If I could afford it, I’d go buy all pre-certified devices. It’s just about money for me – what is the perfect ecosystem worth to me? I bought my previous PC intending to heavily mod it throughout its lifetime. Five years later, I disposed it three weeks ago with the seal still intact. Couldn’t care less if it allowed me to tinker with it. However, there seems to be a lot of vocal resistance to closed ecosystems, although people still do go buy these devices – and I don’t blame them – I prefer a $200 worth of hardware which gives me $100 worth of value instead of a $50 device which didn’t do the task I bought it for!
So what comes next? I don’t know… yet. Everyone’s struggling to figure this one out, and it should be fun to see what people come up with. We certainly do live in a nightmare of devices, and every day, we get categorized into:
1. Those who take this mess in their stride. They can use all kinds of gadgets, perfectly configured to “just work” to make for a pleasurable lifestyle.
2. Those who still have some hope and are struggling.
3. Those who’ve given up and either gone entirely into the clouds, and compromise on the experience their powerful devices could provide, for the convenience of having it all “just work”. And of course, those who’ve bought into the closed ecosystem of devices which give a great experience and just work too!
