hack, life, hack, life


Some Real Life and Hacking interleaved this week.  I added file-upload via the browser, and then ran into a problem with a misconfigured machine poisoning my Paxos voting.  It’s not Paxos, but I kept endlessly triggering a new round of voting, for no good reason.  That took a day to settle out on a reasonable approach.

AllState came out to look at my car, and haul it off to the shop… for 3 weeks.  Ugh.  Bent the door reinforcement beam, so I need a new door.  The Evo’s been a rock-solid reliable car with 120K miles on it (and yes I drive it like an Evo should be), so I’m not going to skimp on the repair.  But 3 weeks driving my family beater van.  Ugh.  Also shot about half a day to talk with the various people (appraiser-scheduler, appraiser, claims manager, body shop).  (fyi – the beater van is a Toyota previa with 165K miles on it, purchased explicitly to train teenage drivers; its slow & reliable and cheap).

I got our new hire to do a nice hack for better HTML generation from inside the Cloud.  We want a way to let H2O developers (that’s mostly me, Jan Vitek & the new guy right now) churn out HTML pages with internal stats on the Cloud.  So we need an easy way to write fairly pretty HTML, and also fill-in-the-blanks with some Plain Olde Java Code, and install with no extra files (a single JAR install) and be rock-solid reliable on ANY network & browser config (means: No JavaScript! for those security-conscience networks), and be debugable in-house (means: no downloading large packages e.g. JSPs, etc; been there, done that, watched several good engineers lose their lives debugging that crud).  So what did we come up with?

You write a Java class for each unique browser page:

public abstract class HTMLIndex extends HTMLUtils {

You have one big HTML String for the page:

final static String html =
+ "<html>"
+ "  <head>"
+ "     <title>Welcome to H2O</title>"
+ CSS.tables
+ "  </head>"
+ "<body>"
+ "<p><a href='Store.View'>Surf the local K/V Store</a>"
+ "<p>The Local Cloud has %subnet_size members"
+ "<p><a href='Upload'>Upload a file</a>"

(ugh: not sure how to get code to not auto-html-ize here.  It’s a plain Java String with plain HTML embedded in it).  Notice a few boilerplate strings already folded in, including some CSS.  Notice I got href links in there, other html and that funny %subnet_size token.  This isn’t the whole HTML string for this page, more later.  But first:

You write the fill-in-the-blanks function, filling in the %subnet_size token with H2O.Cloud._subnet_size:

public static H2ONanoHTTP.Response serve( H2ONanoHTTP server, String uri, Properties header ) {
  String source = html;
  source = replace(source,"subnet_size",H2O.Cloud._subnet_size);
  return server.wrap(source);

Simple, dumb, frightfully easy to use.  I’m sure it’s been thought of a zillion times before.  (and thanks to NanoHTTP for a 1-file web server)  Oh, and for tables your HTML string looks like:

final static String html =
    + "<p>The Local Cloud has %subnet_size members"
    + "<p><a href='Upload'>Upload a file</a>"
    + "<p><table>"
    + "<tr><th>Local Nodes<th>CPUs<th>FreeMem<th>TotalMem<th>MaxMem<th>FreeDisk<th>MaxDisk</tr>"
    + "%tableRow{"
    + "  <tr class='%rowClass'>"
    + "    <td><a href='Node=%host'>%node</a></td>"
    + "    <td>%num_cpus</td>"
    + "    <td>%free_mem</td>"
    + "    <td>%tot_mem</td>"
    + "    <td>%max_mem</td>"
    + "    <td>%free_disk</td>"
    + "    <td>%max_disk</td>"
    + "  </tr>"
    + "}"
    + "</table>\n"

And the fill-in code:

    for (H2ONode h2o : H2O.Cloud._members.values()) {
      // only replace the table line, we will insert the values later one by one
      source = multiReplace(source, "tableRow");
      String host = h2o._inet.getHostAddress();
      source = replace(source,"rowClass",(alt++&1)==0 ? "rowOdd" : "rowEven");
      source = replace(source,"host",host);
      source = replace(source,"node",host);
      source = replace(source,"num_cpus" ,            h2o.get_num_cpus () );
      source = replace(source,"free_mem" ,toMegabytes(h2o.get_free_mem ()));
      source = replace(source,"tot_mem"  ,toMegabytes(h2o.get_tot_mem  ()));
      source = replace(source,"max_mem"  ,toMegabytes(h2o.get_max_mem  ()));
      source = replace(source,"free_disk",toMegabytes(h2o.get_free_disk()));
      source = replace(source,"max_disk" ,toMegabytes(h2o.get_max_disk ()));
    source = remove(source,"tableRow");

That ‘replace’ function does the obvious thing.  Very nice.  A for-loop in the Java-code to fill in the HTML table.  Clean, simple to use.  Yes I am stupidly happy about this hack.  Thanks Petr.

More Real Life: changing jobs means changing insurance.  I COBRA’d the skipped month between jobs, but it doesn’t kick immediately (instead it kicks in retroactively).  This means all those bills I take for this entire month have to be re-filed with each individual provider so they can be back-billed to COBRA.  Also, all my providers need to know about the COBRA this month, and about the new plan next month.  TOTAL pain in the butt.  My teenager crushes her glasses weekly, typically using her face to cushion  her headlong rolling dive in the grass, or to put that wrestling move over on her siblings, or to test the power of belt-sanders & power tools, stuff like that.  I glad she’s enjoying life vigorously but there’s a limit to how far an epoxy repair job can go, so I finally had to pony up for another $400 pair of glasses… during the COBRA-hell-month.  My youngest is in for another round of orthodontics.  My cholesterol prescription ran out.  My colonoscopy (after insurance) is running $600.  And so on: each is another shot at my wallet when I’m in the 1-month no-paycheck limbo, and the fact that COBRA cashed my premium check is just ironic.

My hack(s)?  First was a major cleanup of my async RPC stuff; greenfield coding is great and all that, but after some QA & a little hammering it needed to go from “fun hack” to “production ready” which needs a whole ‘nother level of thinking.  I’ll save the other major hack for next blog.

Last bit of ‘life’: I’ve been listening to the music group Bond.  Awesome stuff and easy on the eyes…  😉



Hit and Run

I was driving in Palo Alto with 2 of my kids in the car heading towards Fry’s, looking to get the largest monitor I could reasonably buy.  I was driving down one of the many fairly narrow 4-lane side streets, heading towards El Camino Real when a driver pulled into my blind spot.  With a block-and-a-half to go to the red light, the driver very slowly started to overtake me… then he started edging into my lane.  I honked and edged up and over to give me some space… with now much less than a block to go the driver suddenly floored his vehicle and came over into my lane (his lane was ending quickly, being filled with large yellow Caddie parked at the light).  I hit the brakes but couldn’t get any more over (no shoulder, sidewalk full of telephone poles and other crud).  Crunch… I was fairly certain he had hit the Caddie, and I knew he had hit me.

I checked my kids, all ok.  I looked out the window at the other car & the Caddie.  He was between the Caddie and me, and was trying to back down the street – but got blocked by cars coming up behind him.  He stopped moving and started getting out of his vehicle, so I took a breather and got a clearer look – there were lots of cars, lots of honking; my car looked drivable and in the way and there was a Jiffy Lube at the corner, so I waved to him and rolled around the corner and parked.

I got out and headed over to his car… and he was busy yelling and screaming at the Caddie driver (who had been parked at a red light the whole time!).  He looked at me and got back into his car and rolled up to light and around the corner, clearly to join me at the Jiffy Lube… and took off instead.  Gone.  I was dumbfounded, and stood looking at his vanishing car like an idiot.  One of the Jiffy Lube guys came up to me and asked if I was ok; I nodded and went over to check out the Caddie guy.  He was a timid old man and was really shook up by the screaming-at he just had.  It was clear his vehicle was blocking the road and looked untouched.  I stuck with him until he felt better, then he got in his car and drove off.  I walked back to mine to survey the damage and double-checked the kids.  Sure enough, a big dent with lots of missing paint.

Then the Jiffy Lube guy volunteered that he had heard the crash, had come around the corner, had watched the driver chew out the Caddie… and decided the driver looked “off”.  He had the plate number written down and encouraged me to call the cops.  So I did; and sure enough in about 5 minutes a Palo Alto policeman showed up.  He was very polite and called in the plate#, and started taking statements.  In another 10 minutes he reported that they had caught the guy!  Then he offered me a chance to eye-witness the driver where they had stopped him… so I followed the cop back across Palo Alto and did a slow drive-by… and indeed it was the same guy.  They offered me the chance to press charges – but they also reported he didn’t seem drunk/drugged – so I decided to do this with just insurance instead.  Maybe he was just having a bad day.

Today AllState came out and looked at my car, and hauled it off to the body shop.  Maybe in a week I have a car with a fresh paint job?  Kudos to the Palo Alto police department; that was quick work catching the guy.  Kudos to the Jiffy Lube guy who had the brains to get the plate number.  And I did eventually get my monitor.

You didn’t think I was going to talk about technology, did you?



Where’d The Week Go?

Where’d the week go?  I managed to implement an async RPC call based on UDP packets; that was kinda slick.  Its more-or-less stateless, so I can re-send the call or re-send the ack+result repeatedly without anybody getting confused.  I added an HTML RESTful Get/Put API, plus some Key surfing.  Based on this+Paxos, I have the starter bits for a distributed K/V store!  Seems like everybody has one of these, so I figured I’d better go make one also.  😉

Then I tried my toy on Linux…it’s been running great on Windows & Mac with 3 to 5 machines.  Linux should be a piece-o-cake!  Or not so.  First stumbling block was Java network self-discovery is painful on Linux: lots and lots of network devices are enabled but few are actually connected to the network.  Then I ended up enabling an IPV6 device… I’m not quite IPV6 ready (close, but not there).  Also, I kept finding loopback addresses.  Then I hit Java signed-byte issues (gah!  a COMPLETELY ridiculous language design decision; bytes are 99.9% used in an unsigned fashion).  Took me another day to pull out all the bugs… then we immediately ran and demo’d it!  (shakes head)   There’s like 1 week’s worth of work in there!  I’m not sure what to make of it all; we were completely upfront with where the product was… and the end-user is loving it.

We ran back to the office High-Five-ing each other and stared at our big empty room.  And then a friend of Sris called: an unrelated startup in S.F. has stumbled and is selling off new office furniture at firesale prices.  We ran to a U-Haul shop & got a $20 box truck (plus $1/mile … that adds up fast).  An hour later we were stuffing desks and chairs into the thing.  We got 6 desks and 6 nice ergo chairs for $1500…. and that is fantastically higher than our rent… I think I’m losing my sense of money here.  (shakes head again)  We ran away with our booty and an hour later were unloading into the not-so-empty office. Those desks have slick wheels and the floor is good and the room is big; after I got up a good head of steam I hopped on the desks for a ride across the room.  Brings new meaning to the word “execu-glide”  Holy moly!  The office is starting to look… office-like.  Needs a microwave (but not a coffee machine; the coffee press came in the building with me on that first visit; must have priorities).  I left Sris to return the truck, and I ran home to a long promised date-night.  Made it just in time and a nice romantic dinner was had, complete with thunderous rain and lightning (umm, excuse me Mr. California?  I pay my cost-of-living taxes!!!  It’s mid-April… where’s the warm sunshine we were promised?)

Next day was mostly spent doing paperwork.  And I’m only doing the new employee paperwork – Sris’ got the backend of this mountain.  Sigh.  Startups are 90% perspiration, 9% inspiration and another 90% paperwork.  Health insurance, payroll, building insurance, PGE wants to be paid, the office water got shut off (fixed now), W-4’s, I-9’s, 401K’s… the list goes on and on.  My home printer died.  My SO’s netbook died (fixed by pulling the battery and power-cycling to drain the caps so it REALLY got a cold-start).

Happy-not-sigh: my kids came back from their mom’s after a week-long spring-break vacation.  More Halo with the boys, more stupendously silly dinner conversation.  The refurbished bathroom looks amazing, and everybody got to apply giant fish stickers on the walls.  Today I am playing taxi-Dad and taking one kid to Sea Scouts, then taking the others shopping for giant monitors, hacker keyboards and Magic-the-Gathering card decks.



Hackers Weekend, Dining Philosophers

We had a hackers weekend at 0xdata, inviting over some potential new hires (and all the handful of employees) and then having a weekend long whiteboard “discussion” – a long excited romp through the innards of what 0xdata is building, and what we can do better, and plain old how-to-solve some problems, which markets to go after first, etc, etc.

The furniture we bought isn’t due to show up yet so I tossed down some folding chairs and beanbags I bought from home, plus an aged whiteboard I had lying around.  We used an iPhone for a local wi-fi hotspot (the building’s Real Internet is coming Tuesday) – and the connectivity sucked.  Great for testing out a budding Cloud’s connection & repair logic.  Paxos worked flawlessly.  We stopped by Safeway for the required junk food and Starbux coffee, and ended up snacking in the sunshine, sipping coffee, and talking distributed & concurrent algorithms.  Dinner rolled through coding exercises, beer, the Right Way to run a programmer’s interview, more distributed algorithms, more beer, the comical errors of past VCs, and whether sex, beer or coding is better (drugs & rock-n-roll never made the list).  Holy crap!  If I had known running my own startup would be this much fun, I’d done it years ago!!!




What the Heck is Cliff Up To?

I finally “Got With The Program” – the standard Silicon Valley program that is – and founded a new startup, 0xdata. That’s zero-ex-data like hexadecimal but with ‘data’ instead of ‘decimal’ – and pronounced “hexdata”.  I’ve been having a blast doing all those things founders do.  In particular, I’m hacking code seriously again for the first time in years.  It’s green-field hacking and I’m pumping out the code.  I hacked a Paxos algorithm over the weekend, directly writing bits into UDP packets… fun!  We’re doing distributed systems stuff and using Paxos for auto-Cloud-discovery.  I tested with Paxos by randomly killing JVMs with “kill -9” and re-launching in a tight loop.  Nodes come & go, but the Cloud remains.  Paxos Rocks!

Today I did Yet Another Interview, 5th or 6th this week, in RedRock, Mountain View.  Sharp guy, but well encased in Generic Big Company… and bored silly.  We’ll see.  Did I mention we have office space now (I mean besides RedRock)?  And basically for free?  The owner took a modest amount of stock in lieu of rent.

We went furniture shopping… at the “startup furniture store”.  This guy cycles used office furniture in and out through startups.  Interesting business model and he had the Right Stuff for us, for cheap. The guy was hysterical!  He was cracking a joke every few seconds, and any possible straight-line was immediately played on.  I think he’s seen one too many startups in his time; once he sold some furniture to a startup guy, bought it back 6 months later on fire sale, then sold it again to the same guy again.  His business model must be one of the more reliable ones around, sorta like the gold-rush merchants who made it rich selling shovels to miners.

What am I doing technically?  Alas, that mostly has to remain quiet for a bit longer.  But the Universe has a ‘hole’ in it, a vacuum, an imbalance of available fast cheap fragile X86 servers with plenty of DRAM, big disks and fast networks – and on the other side a real absence of easy-to-use programming tools.  Those big disks have filled up over the years with Big Data, and thanks to MapReduce, Hadoop and NoSQL we are able to store and address a bunch of connected machines as a single resource.  In other words: we got plenty of unreliable CPU, memory, disk & network… but we can’t get at with the same ease the hardware guys made possible when going to dram from multiple CPUs connected over internal buses.  Until we break that ease-of-use barrier, we’ll never get every-day programmers coding distributed systems as easily as we do single machines now.

Programmers are crying out for a Better Answer, a better way to convert Big Data into Knowledge.  This is the beginning of a the next epoch in distributed computing – and it is only in its first generation.  The best is yet to come.

I’m On The Job.