Introverts, Emotional Processing, Self-Esteem and Salary Negotiations

Introverts, Emotional Processing, Self Esteem and Salary Negotiations

This is a story about myself. But every time I tell it to a room of introverts (e.g. programmers), it resonates with many in the audience.

Has this ever happened to you?

I’m quietly hacking away on something new when:

“Hey Cliff! WTF! I went to show off the new module (which is alpha quality) to the customer and it crashed! I got egg all over my face!”

and now I’m sorting out why I feel ashamed, and before I can respond I get:

“Your supposed to be some hot-shot programmer! Why didn’t you test it!”

and of course I did, but it’s alpha quality and it has many known bugs…

“Your crap is crap, get outta my sight!”

and now I’m really reeling…
and the guy walks off.  About an hour later I figure out what snappy comeback I should have said, but I never figure it out in the moment.

What just happened?  Why was I stun-locked by the verbal assault?  The speaker should have known the code was barely baked and known unstable; the entire dev team certainly did.

And so why yell at me for doing something foolish?  Obviously the fool was not me…
(but wrong question, because the answer to why Fools yell is not important)

Right question is:
Why was I stun-locked?
Why couldn’t I think of the snappy response?

Because I can’t do verbal processing and emotional processing at the same time!!!

This has profound meaning to me, and figuring it out has literally changed my life – hugely for the better – because I can now (and do) compensate for it.

Let’s replay the above conversation, showing my inner thinking as well:

I’m peacefully coding. My emotional guard is down. My creativity is up.  I’m building something and enjoying working “In The Flow”.

“Hey Cliff! WTF! ….”

There’s a verbal assault on my ego, without any warning. I’ve been surprise attacked (with words). My ego is rebounding from the assault, and now I’m trying to figure out why this guy is mad at me (hasn’t sunk in yet he’s off trying alpha-quality code on a customer). Before I get much farther…

“Your supposed to be some hot-shot programmer! Why didn’t you test it!”

There’s a second assault, this time directly at my ego & self-esteem.  Like many programmers I am proud of my abilities. The “supposed to be” comment is directly derogatory against my self esteem and the “why didn’t you test it?” is being presented as an example of my poor coding skills.

Again, I am emotionally processing the assault. By now I’ve sorted out that his original action was unreasonable (presenting alpha code to customers without setting expectations), and he’s engaging in ad-hominem attacks against me.

“Your crap is crap, get outta my sight!”

My emotional barrier is up. The further ego assault is blocked. I begin making a judgement call on his character… and he walks off. The sudden attack ends, as rapidly and unexpectedly as it begins.

I’m now emotionally wrecked, and pulled completely out of The Flow.  My adrenaline has charged me up, and I’m highly stressed.  It takes me a good hour to calm down before my ego rebounds somewhat (there’s a definite hit).

After I calm down, I can begin to replay the conversation in my head, intellectually, and recognize an unreasonable assault… and start to sort out all kinds of intellectual responses to his emotional attack.  The “snappy comeback” eventually materializes in my head, hours too late.

I can’t do verbal processing and emotional processing at the same time.  The verbal assault starts me down the path of emotional processing, and I’ll not have any words until I sort myself out.

What can I do about it?

I can (and do) memorize words, that I can say without any verbal processing.  Rote words; blind words; suitable for all such attacks. Here’s mine, but the actual words are less important than the ability to get them out blindly:

“I need some space to sort this out. I’m leaving.”

And I walk away.

There’s a key add-on phrase, if you need to do this with somebody who you have a relationship you value:

“… and I’ll be back”

That’s it. Hang up the phone. Block the calls & texts. Crucially you must walk away, to end the assault. Run if need be, or call 911 if the attacker runs after you. Nobody needs that kind of abuse, and that’s what it is, pure and simple: abuse.

This same pattern shows up in my love life as well!

I’ve been dating for awhile now, since getting divorced. Sometimes during the first few months of a new relationship, while things are going really well (from my point of view) I get the question: “Cliff, do you love me?”

And I’m speechless.

I’m thinking “what happens if I say ‘no’? Will she walk away? Is she so concerned with ‘love’ at this point in the relationship? I like her alot, but ‘love’ after a few months? No takes longer… We’re still sorting out the basics of the relationship.

And as for my actual feelings towards this person… that’s a deep search of the soul which I can’t do while she’s hanging on me and whispering in my ear.  I’m highly distracted.

And in my silence, she interprets what? Rejection? More worries, more concerns from me… and yes in the past actual real miss-communication because of the silence.

i.e., I’m “stun locked” doing emotional processing and have no words.

So now I this prepared answer (which doesn’t take any emotional processing):

“I don’t know really. I have to search my soul to answer, and I can’t do that right now… that takes days. Last time I checked myself, I really really like you… but it was not love, at least not then. Let me get back to you on this in a few days”.

And then there’s some confusion, so the whole of this talk is generally presented somewhere along the way; about my “no emotional and verbal processing at the same time”, and the need for space to understand myself.  I have found that my partners are all understanding and accepting of this – it’s something direct and concrete about how I process emotions, and easy to understand.  (Of course, if they are not understanding of how I’m built, that’s something for me to sort out as well).

This same pattern shows up in my sex life as well

Sex is highly emotionally charged, at least for me.

I cannot talk during the act, and no amount of poking and prodding has been able to change that.

What can happen is that my partner demands a verbal response from me.  And in the demand of it, I do not feel accepted for what I am.  And then I’m doing more emotional processing- but of my self-esteem, and my relationship and not of sex.  My sex drive is halted.  A demand for “sweet nothings” leads to a cold bed.

While I can’t talk during the act, I can listen, hear and act on words; especially the gentle guiding words.  I can “speak” with my actions.  I just cannot bring up my own words, until after my own climax.

The compensation for this is the same as before: prepare the words before the emotions hit; talk it out in the light of day, while fully clothed.  I make sure she knows that at some point during the evening, I’ll become less and less verbal (and more and more physical) until after the event.

This same pattern shows up in my office life – in other people

Somebody is excited about a solution they’ve come up with, and they’re sharing.  Loudly and excitedly! They are often saying the same things over and over again.  A few other people get excited, and so start talking about problems with the solution, and other solutions and alternatives, and there’s a wild brainstorming session rolling on…

…and quietly sitting in the corner, is somebody with a bright idea, maybe the best answer, who’s too afraid to speak up.

They’re thinking: “what if I’m wrong? I’ll look like a fool!” or maybe “I wish I could talk, but I can’t!” or maybe “They won’t believe me anyways…”

They’re doing emotional processing, and aren’t able to come up with the words, nor the courage to speak up.

As a leader, it falls upon me to spot this pattern – the verbal action from the loud few on one hand, and the quiet listeners on the other – and take action.  I don’t want to stop the brainstorming session, not yet, but I want others to have a chance to join in . I usually end up saying something like this: “Larry, I heard you suggest XXX as a solution, is that right?” – I want Larry to know he’s been heard so perhaps he’ll back off promoting the idea so loudly if we’re already on the same page.  I need don’t say it’s the right plan or wrong plan, only that the concept has gotten across.

And after that acknowledgement, I’ll offer “I see some quiet listeners – anybody else want to add in?” – and we’ll all pause talking, which makes a verbal space – a gap in the flow of words – which can be easier for a shy person to step into.

Generally, I work with a lot of people who are shy with words, and I have to be careful not to be the “Loud Larry” myself – lest I shut out the brilliant suggestion from the quiet person. I have to spot the pattern where one person is “verbally dominating” another, and give all folks a “verbal space” to step into.

Many of the people I work with – programmers and introverts of all kinds – appear to have a fairly low “emotional IQ” – they’re not necessarily aware of why something has upset them, only that it did.  I end up doing a lot of quiet conversations, where I help people understand themselves.

The Aspects of Farmer, The Mage, and the Warrior

We all exhibit some of each these aspects:

  • The Farmer enjoys the hard work, and building with his labor.  His ego is boosted by what he accomplishes with his physical labor.  “Today I built a house!  Some family will live in it someday!”  Tradesmen are an exemplar.
  • The Mage enjoys the “smart work” and tool-building “magical” things.  His ego is boosted by being “smarter” or “working smarter”.  Programmers often have this aspect strongly, but also Doctors and Experts in all walks of life.
  • The Warrior engages in a fight, and might indeed revel in it.  Politicians, CEO’s, Lawyers, wheeler-dealers, and used car salesmen fall into this camp.  The fight’s the thing, not really what is being fought over.

Programmers (and introverts?) often do not practice with their Warrior very much.  Their Warrior is unskilled and naive.  They can be pushed into the defensive very easily and dominated by those with more “fighting” (verbal and otherwise) skills – “alpha males”.

What would a well functioning Warrior think of my opening conversation?  Well, what do you think would happen if a Lawyer was on the receiving end of:
“Hey Cliff! WTF! ….”
The Lawyer might respond:
“You’re a f*cking idiot! That was a fool thing to do!!!!”

The assault is returned immediately – an ad-hominem reply to match the opening tone of the fight, and then very not-subtle pointing out who was the real fool.  The Lawyer’s Warrior is ready for the verbal fight, and engages willingly.

Stress and Anger Management

“Hey Cliff! WTF! … Your crap is crap. I’m outta here.”  And he walks away.  The fight is over for the Fool, but not for me.  My adrenaline is raging, my Warrior has picked up my Ego armor, and my body is spoiling for a Fight or Flight response . That adrenaline and stress doesn’t just fade way immediately.  It sticks around, perhaps for a long long time. The stress and anger might stick around for years. I’m angry now.  I want to act, my body is expecting, demanding that I act.

You’ve heard the term “kicking the dog”?  I’m angry.  I want to Attack. Somebody, something, anything.  But I’ve enough brains not to attack somebody who might hurt me, so I want to attack somebody who is safe.  The dog.  My S.O., my kids, the person next to me.  I’m angry at the Fool, but I’ll attack whoever is convenient and safe.

Wrong twice. Once because I end up attacking the very people who are trying to help me. And twice because my anger isn’t directed at the source in a useful way.  The Fool is safe to attack me again some day.

And yet, we still need to get the Anger out – lest it turn inwards.  Have you ever gone home after such an event and stewed on it?  That’s anger turned inward, and it leads to depression.  High stress leads to health issues.  We need to get rid of the anger and stress in a useful way.

The Key: Get the Anger Out, but not At

Out, not At

Getting anger out requires some physical activity to get the adrenaline to metabolize.
Not at means picking an activity that doesn’t focus towards another being.

Go for a run.  Go to the gym. Walk around the block. Get too a tennis court, imagine the Fool’s face on the ball and smash it.  Again and again, until his face fades, and the anger fades, and you’re actually doing something aerobic.  I’ve grabbed a pick axe and gone to my backyard, and smashed stones.  Bang!  Rock chips fly, sparks fly, bits and pieces go in all directions.  There the Fool is again!  Wham!  More sparks!  It’s very therapeutic and actually really hard exercise.  After 10 or 15 minutes I’m hot and sweaty, I’m over being angry, and I’m looking at a hole… and my Mage kicks in and asks… “what can we do with this hole?  Plant a tree?”  And my Farmer speaks up: a tree-hole needs to be more round, and about 2 feet deep, and a little to the left.  More sweat, happily applied (and now I have 2 large mature apple trees growing in my back yard).

Working Through The Emotions

There’s another step, once the immediate anger has faded, and that’s working through the emotions and the experience – and accepting what happened (I got verbally attacked by a co-worker), and moving on.  For me, I have found that talking to The Village is a good solution – the Village of whoever happens to be around me; friends and co-workers, family, strangers, whoever.  For other people it might be different: e.g. artistic outlets like painting or singing.  A long quiet time alone (such as hiking) with lots of Zen time, perhaps in a beautiful place.

Know that you need time to work through the emotions and give yourself that space.  When the hub-bub of life settles down, and the kids are in bed, and the quiet descends on the world – don’t reach for noise-makers to fill the space (TV, movie, loud friends with alcohol) – instead, give the time to your inner self.  Work through what happened in whatever way is best for you.  Don’t stew – reflect.  If you find yourself stewing on it, you need to find another way to work through it.  Maybe a conversation with a close friend or parent or other safe person.

Just like any body injury you need to heal internally.
Give your soul some healing time.

Red, Yellow and Green Flag People

Make this judgement call on the people around you: are they Red, Yellow or Green?

Green flag people are safe to be around. It’s the friends you trust.  The ones you can bounce ideas off of, without fear of getting sarcastic responses.  You’ll get safe honest answers, no “hurting” words, and in return you can share your inner feelings and honest opinions back.

Red flag people are unsafe to be around.  You cringe when you see them walk up; you raise your emotional guards. Your creativity and openness ends… because sometimes they attack you, or they are passive-aggressive, or making endless (seemingly reasonable) demands. They are dangerous to your self-esteem and ego.

Yellow flag people are somewhere in the middle; folks you can be around but maybe you only drop your guard carefully.

Why make this call?

Because it brings your Mage into your relationships, to help you figure out which ones are useful and which ones carry a burden – and now you can make an intelligent decision about which the usefulness vs cost trade-off.

Example:  Angry Ann always has some biting sarcastic thing to say about the people around you (and you wonder what she says about you behind your back), but also she’s a crazy fun party animal.  Thinking about your relationship lets you make the decision  intellectually: is Angry Ann worth hanging around?  Sometimes?  Always?  Never?  Bring your Mage to bear.

Same call for Green people: mine is my brother.  Divorce is on, boss yelled at me, health claim denied, got a ticket doing the same speed as every other person, whatever.  I know who to reach out for for support.  Knowing that, means I know when and where and how fast I can get emotional help… and that alone mitigates most of my emotional trauma. Know Thy Greens, and Use Them.  People really do like helping somebody else, and it’s always very life affirming when I end up being somebody else’s Green.

What To Do

Lose The “Reds”.  Really.  Try really hard to lose all the Red flag people in your life. You aren’t going to change them (people can only change themselves), and you don’t need the stress and negativity in your life.  Life is too short to run around with your armor always up, getting ready to duck from the next verbal assault.  Your creativity, your passion, your love and skill – all will suffer while you are constantly on the defensive.

It gets worse if the Red is somebody close in your life (parent, S.O., boss) – you can get trapped in “siege mode”, constantly on the deep defensive. Only being reactive, never active.  Slowly getting beaten down into a shapeless mass of sadness.  Internally you are screaming for help, but externally nobody hears you – and you need help to get out of this relationship.  Get That Help!  Councilors, Green friends, whatever, wherever – Act to Get Out.  Make that internal scream a real audible one.

How To Lose The “Reds FLAGS”?

Depends on where they are in your life.

Kinda-sorta-friends?  Pick your social activities with care.  Same for abusive family members when you are an adult: don’t bother hanging out with your abusive relative.

Boss/co-worker? Complain uphill (facts are crucial here, not just your words; emails, texts or co-complaints; starting a he-said-she-said fight is generally worthless).  And if that fails, ponder walking away.  Life is to short have a crappy job experience; go get another one and work happy.

Significant Other?  That’s a lot harder, and harder yet when kids are involved.  But know that there’s something wrong, if you are red-flagging some parts of your S.O. – it’s probably time for talk therapy or a marriage councilor.

Love, Change, and Mind Reading

No matter how much you Love somebody,
you cannot Love them into Changing

Change comes from within, not without.  No matter how much I cajole You, the Gentle Reader, with these words, I cannot change you.  Only you can change you.  Same for every body else in your life, and for you in their lives.  Somebody else might want you to Change, and you might even be aware of it… but unless you decide to Change yourself, their desires are for nought.

No matter how much you Love somebody,
you cannot Read Their Mind, nor they yours

So much broken expectations arise from this. Your S.O. just did something that hurt you. Did he do it on purpose? Why would he hurt me?  “I just need to love him more, and he’ll stop doing that.”

It’s much more likely that your S.O. has no clue that you got hurt – and if you mention it “hey, you know I really got hurt here” – your SO will probably say “what??? wait, no! I didn’t mean to hurt you!”, and maybe even “how did that hurt? I don’t understand!”.  If they are concerned about your well-being, then they’ll seek understanding of “how” that hurt, and Change – but the Change is driven from within – not by you Loving them, but by your S.O. Loving you.

Salary Negotiations

Funny joke I heard recently:

“Fighting with a Lawyer is like wrestling with a pig in mud. The Lawyer enjoys
it, you’re going to get really dirty and you are very unlikely to win.”

Same with used car salesmen: they sell cars to non-Warriors each and every day.  It’s what they do – nothing emotional about, nothing personal.  It’s just their job, and they probably enjoy it and they are certainly good at it.

Same with HR: they negotiate salaries each and every day.  Nothing personal, and no emotional processing, it’s just their job – to get your skills at the best price for the company.

The HR rep has a salary range in mind, and perhaps a recommendation from the team that’s trying to hire you. Their job is to get the requested skills (you’re only one of many they are looking at) hired at the lowest possible salary, and after that – they step out of the picture – no hard feelings, and no regrets… and no real interaction with the hiring team except to check-in now and then. Your actual work colleagues generally aren’t part of this process and probably don’t know what you end up going through (except that they, themselves, went through the HR hiring process years ago).

But it IS your life, and your standard of living perhaps for years to come.

After the customary pleasantries, the HR rep, like any good Warrior, launches into an (acceptable verbal) attack: “So, Cliff, how much are you making now?” or the very similar “So, Cliff, how much do you need?”

Immediately I’m thinking: “If I tell him, he’ll only offer me a little above that – but I know I’m worth more!” and “how is my self-worth being judged by a single number?” and “I bet he’s willing to pay more, but if I ask to much I won’t get a job!”…

i.e., I’m emotionally processing, and my un-skilled Warrior is on the defensive.  Pretty soon I’m backed into the “I really need this job! I better tell him!” corner and the fight is lost before it hardly begins.  I get the expected lame offer, and I’m feeling under-appreciated for years to come.

What can we do about it?
Well, we do what any Warrior should do: prepare and practice for the fight!

Prepare! Know Thy Worth. It’s easy to find out the basics: check out similar skill sets on,,, etc… job boards everywhere will have other examples of salaries. There are a zillion books on interviewing skills and practice interviews and questions and strategies.  Getting that better salary is worth far more than the price of the book and a few hours to read it.  Spend The Time.

Practice!  Get a partner to play the HR role and set up as much as possible like the Real Deal: dress up nice (so you’re out of your comfort zone, just like the real interview), find an empty office and have your partner sit in the “Power Seat” behind the desk – another way you get put on the defensive – and play out a salary negotiation.

Practice (unlike prep) hits you in a different place, and trains a different Warrior muscle. Prep gives you intellectual ammo, and lets you think through what you should be asking, how to approach the problem and so forth: your Mage is at work. Practice is where your Warrior turns Knowledge into Action. Without the Practice you are very likely to stumble trying to apply your new-found knowledge. Don’t skimp on it!

More “Prepare” Advice:

BATNA: “Best Alternative To a Negotiated Agreement”.  If you don’t already know the term look it up.  It’s your “Plan B”, your fall-back position, it’s what you get if you fail to get the job, or fail to get the negotiated raise or whatever.  Knowing your fall-back position tells you where your “negotiation space” is, your “wiggle room”.  Your Warrior will need this information during the “fight”, readily and without thinking.  Prepare it ahead of time.  Your Mage can tell you if it’s really weak, and perhaps needs shoring up before going into the fight.

Example: I like my job and I’m good at it, but my pay sucks and I know others are getting paid more – I want a raise.  But if I walk in and they say “No” – what is my option?  Suck on the No?  I need a better BATNA – so perhaps I should go looking for another job.  A serious look, where in that other new job interview I can know that I don’t need it… but if I want it, I can have it.  Then when I walk into the salary negotiation on the job I really want, my BATNA is now “Suck on the No OR Walk Away to a New Job”.  It’s a MUCH stronger bargaining position.

For the “how much are you making” questions – prepare an answer ahead of time.  You never need to answer the “how much am I making now”, but you do need to politely answer his question.  Something along the lines of “what I’ve been making in the past is not really important here, what’s important is what fantastic skills I’m bringing to the table – what are they worth to you?”  Or “I’m asking $xxx, because I know I’m worth it” – and aim a little high.  The worse that can happen is: “we’ll have to get back to you on that salary request” – generally followed up by a counter-offer – one that you can think about in the safety of your own home on your own time.

This is the first time I’ve put this into words, but not the first time I’ve talked this out.
Sorry if it seems long and disorganized, I’m still sorting it all out myself.

Good luck in all that you do!


Down from Eight Miles High… and Landing!

I’ve landed at Neurensic!  First I’d like to thank all the people entertained my request for a job – there were lots and lots.  Competition was stiff, and as the month went on, the offers got wilder and wilder and closer and closer to my dreams.  Ultimately I had the choice for taking VC funding and starting something of my own (still Out There, and will probably happen yet, and could be Flying Cars), and venturing into the financial tech sector, something I’ve long thought about. Fin Tech won the day, but I’d like to give a special shout out to Google (at least 4 different groups courted me, I really like the feel of Google).

The job hunt experience gave me an unparalleled view into the Valley’s outstanding tech problems.  Several problems really stood out – as things a lot of people wanted help solving, and I thought I’d mention them here as a sort of “State of the Valley’s Better Mousetrap Hunters”.

NodeJS – 3 different companies made clear offers to help make a server-side version of JavaScript targeting NodeJS.  This is right up my alley: adding threading (but keeping the single-threaded coding style), adding a low latency high performance GC, and adding A Better JIT.  Looking at the state of JS JIT’ing I cant help but think there’s a lot of performance being left on the table – especially for “long” running server apps.  Maybe some easy tweaks and V8 cleans up here… but so far the NodeJS audience ain’t feeling the love.

Python – At least 6 stalled attempts to speed up Python are Out There (PyPy appears to be “less stalled”) – and I got clear offers from 2 companies and sorta clear offers from 2 more to make “Python Go Fast”.  I also reached out to the Python core community – and got a resounding Silence.  I’m not fond of pushing rocks uphill; if the core community isn’t interested then I’m not (yet) willing to tilt at this windmill.  Nonetheless, given the response from 4 companies (some quite large) I’m willing to bet Somebody’s interested.  Sorta needs the Kickstarter-from-Hell to pay my bills, but then could be a fun gig.

Both the NodeJS and Python speed problems appear to not be getting solved anytime soon.  I might come revisit both of these areas down the road- perhaps more as a quiet easy-going slam-dunk job.

After that, mix in 4 different ML-related offers, 8 different startups (literally at least 8, more than one network-security-meets-ML), and several We’ll-Fund-Whatever-Cliff-Does startups.  And flying cars, no lie; that was the last-ditch almost winner.  But I need to be in a better state financially before I can bite into that apple.  All told I sorted through over 20 serious job offers. It’s good to have First World Problems.

Why Neurensic?  Why Financial Tech?

This story goes waaaay back – I’ll skip past the “I was born a poor black child…” part of the story and start in when I was about 26 or 27.  I was just finishing up my PhD and was starting to look at the job market – and Everybody was saying all the money was in being a stock market trader.  It looked too much like giving up on helping the world – just living for myself, so mercenary and narcissistic.  I still wanted to Save the World, and I liked my mix of theory and practice… so I ran off to do compiler research at HP Labs, then beat IBM on IBM hardware with my mad compiler skilz, and a few years later headed out to California to make this new-fangled Java language Go Fast.

Fast forward 20 years, and I got fame… but not really fortune.  I am all for changing that now, at least a bit, and for Save the World.  I have been courted repeatedly by the traders over the years, and certainly talked with dozens of high frequency trader groups about Java performance and low latency GC.  Basically I’ve had the Fin Tech itch for more than 20 years, and I’m finally going to scratch it – but not in trading directly, but in making trading safe.

Neurensic has some lucky timing and some serious financial market Quant talent. They put together a Machine Learning-based product to find financial market cheaters – and just in time to match the new Dodd-Frank compliance rules.  Their timing is impeccable; suddenly everybody who is responsible for traders’ good behavior (i.e. the Banks) wants their stuff.  Their first generation product is just coming out, and I plan on bring my own skill and H2O’s Big Data and ML abilities to dramatically up Neurensic’s game.  The Goal: keeping the Stock Market (well, all financial markets) safe for everybody.

Neurensic Round Two is also right up my alley: a Rosetta Stone for the 40 or 50 back-end systems found in any large Bank.  Many of these systems date from the 80’s and 90’s, and use a level of tech I’m very comfortable with.  The Goal here is to simply bring together all of this disparate data – it’s Big Data, so my Big Data experience applies here also – and give decision makers a grand overview of their entire enterprise, updated in seconds instead of days.  The term that springs to mind is a colonoscopy, a birds-eye view of a Banks’ backside.  In the longer term, we aim to change the fabric of the financial system as a whole, at the transactional layer.

I will remain in Silicon Valley, and will be shortly recruiting engineers looking to blend machine learning, financial tech, and some “catch the bad guys” mentality.

But Wait, There’s More!

I have a another offer I am actively pursuing – to keep extending H2O in the Open Source.  This will be a funded H2O-Center-of-Excellence with room for 5 to 10 salaried engineers solely focused on moving H2O forward.  Funding hasn’t landed yet, so I can’t say more – but I’m hopeful that I can start offering positions here in the next month or so.  Note that Neurensic is both fully on-board with this effort, and is not funding it nor controlling it in any way. Basically, it will be my second day job, and it happens to align well with my other day job.

Related-but-not-related, I’ve really enjoyed my engagement with the Skymind guys (hi Chris and Adam!) and DeepLearning4J – and am also angling to be in a position to get DL4J and H2O working together.

Hey!  This means I’m running at least 2 teams of of 5 to 10 engineers each!  Isn’t Cliff known for both his code hacking skilz and lack-of-people skilz?  Well, I’ve been running the engineering team at H2O for several years – and I really like it, and by all accounts people like how I run a team.  So yeah, I’ll be running some teams.  I’ll also be coding – I am a Builder after all, I have to Make Stuff to save my soul – so yeah, I’ll be coding still.  Just blending in some people-time also.

Oh yeah, AND I’m teaching a course at Stanford on Machine Learning. Stanford CME250A

If any of this interests you, come knock on my door!  For anything open source, or H2O, or heck Java Performance or compilers or flying cars, my email of 30 years still works:

For anything related to Neurensic or Fin Tech try me at:

Looking forward to hearing from y’all!!!



Words of Parting, a Fond Farewell

I resigned from H2O last week – with a fair parting and I fully support H2O’s success in future.  Srisatish and I worked together since 2004, and I learned a tremendous amount from him.  He has an amazing passion and vision.

I’d like to take this opportunity to celebrate our work together, in building something from nothing.  As Srisatish recalls, we did a million little things together; I also fondly remember the uHaul trip retrieving our earliest furniture (where we forgot to close the rear door and got a lot of other drivers’ wide-eyed panic looks before figuring it out!), and cranking out code furiously – and Taco Bell as the “company cafeteria”.  And of course the many many team-building lunches eating in the warm California sun at Fiesta Del Mar.

A lot of how we built the company was playing off each others strengths.  Srisatish had a precognition of a Machine Learning wave sweeping the industry, and with his passion he sold the vision to me, the investors, and then the industry.  For my part, I built the backbone of that vision: not just the foundation, but the powerhouse engineering team which built up from that foundation to produce the milestone Machine Learning product that is H2O.  That partnership produced amazing stuff, and I’m sorry it had to come to an end.  I only wish him the very best in the future.

It’s all of you, the H2O team, I’ll miss the most…. over the past four years, you have built an amazing technology – who’s strength is only just being tapped.  I have witnessed huge personal growth in all of you, as coders, as engineers and as good people, and am very happy to have had a chance to be a part of that growth. Your creativity, hard work and dedication have built something wondrous.  Your passion has become your outreach, you are leading the world into an era of ML and AI – a better way to do things, build things, change things, faster, more intelligent, and yet more human as well.  I applaud all that you have done, and continue to do – and I wish you the very best in your personal journey to make the world a better place.



Captain’s Log Days 8, 9 & 10

Captain’s Log Day 8

We’re on the road by 10am, this time a full day’s drive to Montgomery AL from Katy TX.  I forget how big Houston freeways are; at one point I count 9 lanes *in each direction* (18 total lanes!).  I’ve never seen so much concrete.  It’s otherwise mostly uneventful, though.  Traffic is fair to light and the road is good.  We stop at a random lakeside park by Lake Charles for lunch.  It smells of the ocean and has an alligator pond/cage/viewing area.

While I typically encourage the kids to drink a lot (to survive the desert heat & dry), I don’t check on how much they eat, just that they eat a reasonably balanced diet.  So I missed out that Matt hadn’t eaten all day, and was constantly staring heads-down on his IPod on silly flash games.  Well, towards afternoon he starts feeling sick, and near dinner he barfs and refuses to eat or drink anything.  He cannot even keep down tiny bits of bread or Gatoraide; the 2nd barf happens on our bedsheets and pillow.  At this point he decides to camp out by the RV toilet and do any more barfing into that (uggh!!!  poor guy!!!!), and we decide to cut it short and look for camping for the night.  By dinner he’s still unable to keep anything down; we grab to-go food from a collection of fast-food joints and keep rolling to the nearest campsite.

We get 1/2 way between Mobile & Montgomery, AL and pull over into a nice full-service RV park.  Shelley & I decide to camp outside in a tent, so Josh can get off the floor (he’s 17 and 6ft tall, lean and flexible… and does not fit in any of the RV pullout/fold-down beds, so he’s been sleeping in the aisle).  We want Josh off the floor so Matt can make an emergency run from his foldout bed to the bathroom without interference.  It’s beastly hot and humid outside, but I figure it will cool off as the night wears on.  Boy was I wrong! It remains 80+ & 80% humidity all night long outside, while the kids were sleeping in air-conditioned luxury.  And we get a late night visit from the camp kitten – he’s adorably cute and caterwauls at us, and starts climbing the tent with his razor claws until Shelley takes him for a walk.  He follows her like a shadow all over the park until she finally has to lock him in the campground bathroom.


Captain’s Log Day 9

Finally dawn breaks and we move back into the cool RV air.  Ahhh, blessed relief.  Also, Matt is much better – it’s a common kid 24-hour tummy bug.  I start him back in on the BRAT diet, with sips of water – and now he’s very hungry, a good sign.  He continues to improve throughout the day and is eating normal by dinner.  We pull up camp (we’re getting quite expert at this) and head for Stone Mountain, GA.

Stone Mountain is a giant mountain-sized chunk of granite outside of Atlanta, with a park and a lake.  It’s been carved with a 50ft high sculpture and has been slowly improved over the years to include many hiking trails, a sky tram system, lots of outdoor adventure activities and an amusement park.  Apparently the “ducks” (amphibious vehicals) are fantastic.  We are going there for the July 4th extravaganza – and as a sign that I’m on vacation, I barely know that today is the 3rd and I’ve no idea what day of the week it is.  We get there about 3pm and check in to a nice RV camp site.

Shelley cooks a fantastic spaghetti dinner.  My brother Eric drives out to camp with us, bring his best friends’ two small girls (ages 6 & 7) with him (he’s been watching the girls when the parents are working since they were 2 & 3) and we all enjoy a nice picnic dinner.  As the evening rolls on we’re deciding on whether or not to see the laser & fireworks show this evening (there’s a bigger one tomorrow) – when the thunderstorm hits.  It’s a real downpour, big lightning and thunder, blowing wind, the works.  We wait that out, and then try to take a walk about the park.  Eric & I, the two girls and my middle two kids walk over to the clubhouse (to check out the water-taxi ride to the main park area) but the rain has other ideas.  We make it to the clubhouse but we’re fairly wet, so we treat the girls to hot chocolate while we dry out.  We wait for the rains to end but it’s no good – the rain has turned into a steady drizzle; we just as wet by the time we make it back and there’s no end in sight.  We give up any idea of tent camping or seeing the laser show and settle for watching a Disney movie (the Sword in the Stone) and having a lazy evening with all 10 of us huddled in the RV).  Sleeping arrangements are “cozy” to say the least!  But at least everybody is dry.


Captain’s Log Day 10

It’s the 4th of July!  We breakfast, cleanup & head over to the water taxi. The rains have stopped and the sun is out.  It’s gonna be a hot & humid day. The water taxi is nice, it’s cooler near the lake.  We make it to Stone Mountain’s main attraction area and decide to walk to the bell tower.  The park is already busier than Eric has ever seen it before.  There’s a large Indian family setup under the bells already (and I see more people of the same persuasion walking over to the tower all morning – I think they figured out a cool shady semi-private place to hang out at all day).

We’ve walked maybe a half a mile and it’s not even noon and we’re already soaked with sweat when we make it back to the Plantation Inn.  The Inn isn’t open for lunch (although the AC is nice), but the helpful counter lady tells us there’s RV parking closer in.  We walk up to Memorial Hall.  Immediately two things strike me as really odd: there’s at least 1000 people hanging around looking for food (and more pouring in all the time), it’s 11:30 and *none* of the dozen or so restaurants are open yet – and there’s bus & RV parking open
right in front of the main Hall.

I hand the kids my credit card (to get lunch at noon when the restaurants open) and Shelley and I hightail it back to the RV: across 1/2mile of hot trails & roads, ride the water taxi (we miss the one in front of us by literally seconds even with me sprinting across the landing area), and finally the 1/4 mile hike from the taxi dock to the RV.  We pull the hookups as fast as we can and roll out & down the road.  Nessie does NOT sprint, she *proceeds*, but we made her proceed as fast as possible.  We took the short way around the lake, only to discover the road was closed: the attendant at the barricades explains “the road fell in a hole”.  Nothing to be had for it; Shelley makes a 3-pt turn on a narrow park road and we go the long way around.  Finally, a full hour later, we make it back to the bus/RV parking in front of Memorial Hall – and Lo! it’s open.  We take the most premier parking spot in all of Stone Mountain, at noon-thirty on the 4th of July.  (A short time later one other RV takes the next spot, then the road is closed behind us).

The amazing thing about the Stone Mountain concessions was the astronomical price for food; hotdogs: $7-$10, drinks also $7 or so.  (And they denied a hot and hungry hoard for at least an hour???).  But finally we all sat down and finished our food and plotted our next move.  Shelley, Eric & I all want a big hike.  Last Christmas Shelley & I hiked the Grand Canyon down to Phantom Ranch and back out in two days, and Eric has hiked both the Pacific Crest Trail and the Appalachian Trail end-to-end.  We head out for the top of Stone Mountain on a hot & muggy day.  There’s lots of other folks with the same idea, but it really is a long hot hike.  Most of my kids bail out after a mile or so, voting to go hang out in the AC (which is really a good plan); Eric and his two young charges make it to the path-up cutoff but it’s a killer hike in the heat so they turn around also.

It ends up as Shelley, Laura (age 15) and I heading on, and we decide to head for the bird sanctuary.  It’s another couple of miles and we gave most of the water to Eric & the girls.  The three of us head down the far side of the mountain to a kids playground and finally drag ourselves into the park and help ourselves to the water fountain.  We drink a quart each, and fill a couple more quart bottles we’re carrying.  We hike the 1/2mile more to the bird sanctuary – mostly carrying on now because of what Shelley would call “Mission” – her ex-Marine training to “complete the Mission” no matter the cost.  i.e., we’re all too collectively silly to claim the end goal is ridiculous, so we hike it anyways. It’s a decent enough little woodsy trail, with plenty of songbirds – but far to beastly hot to really enjoy.  By the time we make it back to the kids’ park we’ve drunken all our water (another 1/2gal between the 3 of us), so we reload (and re-drink our fill) and back up the mountain to cross it in reverse.  We make it back in good time, although it was really pushing our the limits to
hike so far on such a hot day.

There is much lounging around and napping in the RV’s AC to wait out the heat of the day.  Matthew (age 12) introduces the two little girls to the joys of Minecraft.  Eric & Laura nap.  Everybody else surfs the (very very slow) park Internet, eating popcorn & chips.  Finally as the heat starts to fade and twilight sets in we get enough gumption to make & eat hotdogs.  Then we pack it up and prepare to leave the relative safety and peace of the RV for the slowly building hoard.

The lawn below Memorial Hall faces the giant sculpture carved into the face of Stone Mountain.  The only open spaces are at the very front, so that’s where we head.  I estimate 100,000 people eventually filled that lawn; in any case it was a colossal crowd.  It was also actually quite a peaceful crowd; no rowdies (no alcohol allowed), zillions of little kids running pell-mell, picnic blankets, soap bubble makers and glowing flashing LED lights.  It’s cooler now, so we settle down on our blankets and chairs, listen to the music and wait for the show.  At various times I let Josh or Karen & Luke wander off for snacks (a little nerve-wracking that; they are out of sight in the crowd within seconds and gone for 30mins or more, but everybody returns fine).

The fireworks show starts promptly at 9:30 and is possibly the best I’ve ever seen.  There’s a laser & light show on the mountain, there’s a Civil War tribute, (there’s ads for all of Georgia’s major sports teams), there’s music and of course fireworks.  The actual fireworks where downright amazing; you get a double-echo from the Bang! works, one directly and one bounced off the mountain.  They used plenty of the big fireworks and absolutely tons of rising sparks kind; the entire mountain was a sheet of fire for minutes at a time. The finale left us breathless.

Unwinding back to the camp was a slow but uneventful crawl; I’ve sure we beat the campers on foot (who had to wait for the river-taxis and the report was to expect a 2.5 hr wait).  Eric took his to charges home and we collapsed tired but triumphant for a full nights sleep.



Captain’s Log Daze 3 & 4

Captain’s Log, Day 3

Next day we take a lazy breakfast and then decide to visit the Biosphere with Grandpa.  We head out to Nessie and observe the new tire is looking mighty flat.  Humm….  we hook up the air pump… and it’s at 80 psi, spot-on the normal max pressure.  Looking further: the inside tire is flat.  CRAP.  But wait!  It really WAS fine in Casa Grande, I checked it before we drove off.  And pretty quickly it’s clear that the rubber value stem is leaking, probably
banged too hard during the change and now it’s going flat overnight.  We call up Ed.  He agrees to change it under warranty… but he doesn’t want to drive out to meet us.  But he Does The Right Thing, and calls GCR Tire, a Tucson local who WILL come out to Grandpa’s.  Ed’s covering the whole cost.  So now we’re basically stuck at Grandpa’s waiting on GCR Tire (who’s promised to get there “in an hour” and it’s already 10am).

Meanwhile something triggered in Shelley’s brain about tires aging, so I go read up on them.  I learned something new today: all tires age.  After 6 years you should replace them, completely independent of tire wear.  Pretty much no tire is expected to last 10 years except under “ideal” circumstances.  And tire manufacturers have to stamp the date of manufacture on the tire, so you can tell how old your tires are.  (read up on it, but it’s the week & year of manufacture as a 4-digit number in an oval after the “DOT” stamp).  So we go look at our tires (mostly meaning Shelley crawling under the RV in the 110-heat to read between the dualies).  Sure enough, the youngest tire is 8 years, and the oldest is 12 years old.  Good tread, but expected to blow at any moment.

Crap, crap, crap.  Another round of planning & family voting.  We decide to limp over to Big-O tires to replace the remaining 5 tires, never mind fixing the old one.  GCR Tire shows up for the repair while I’m finishing negotiations with Big-O (and yes I asked GCR and no they did not have the tires we need in stock).  So the GCR guy politely fills our inside tire (it’ll last maybe an hour) and we roll over to Big-O.  We drop everybody off at Costco, where we do some shopping and eat a delicious Costco lunch (which is actually pretty dang cheap and a decent enough hot dog), and wait 2 hours for me to blow another $900 on tires.  After a while we’re back at Grandpa’s house with 6 brand-spanking new tires, waiting for a thunderstorm to pass before we go swimming.  It’s too late for the Biosphere, that will have to wait for another visit.

The thunderstorm takes too long to pass and we miss swimming also.  We have some more family over for a nice dinner, then we hit the road again for more night driving.  This time we’re heading for Carlsbad Caverns.  It’s a long haul out of Tucson but utterly uneventful.  We even give Luke (19 yrs old!) a turn at the wheel.  He’s a natural driver and handles this big rig fine.  We make a long drive of it but Carlsbad is just too far to make in one day.  We end up in the backside parking lot of a Walmart somewhere just inside the Texas border (Walmart mostly has a “RV friendly” policy).  It turns out that while our GPS has many useful features, finding RV campsites is not one of them.  Also when we turn off I-10 and head into the countryside we lose all cell phone service and can’t call ahead.

Captain’s Log, Day 4

It’s a 3hr early-morning drive or so to the Caverns.  We get there just before the heat starts getting oppressive again.  This time we decide to leave the generator on and the AC running while we spend the hot part of the day underground.  I used to see this all the time and wonder about it: RV’s with the generator going constantly.  Now I get it – Nessie will be in tolerable shape when we return to her, but without the AC Nessie would heat up like a tin
box in the hot sun.

Several of my kids are really nervous about entering the Caverns; they’ve had some scary cave experiences in the past.  We have to gently encourage several down the switchbacks into Carlsbad, but they master their fears and soldier on down into the cool cave air.  Carlsbad does not fail to deliver.  The Caverns are immense on a scale that’s hard to imagine; all of downtown San Jose could comfortably fit in them.  The trails wander on for miles in there (the sections closed to the public are probably 100x larger than the miles of public sections).  There’s a section where the roof soars over 300ft overhead and single rooms covering many acres with lines-of-sight of perhaps a quarter-mile underground.  And it’s all a fairyland of cave growths and little pools, with eerie lighting everywhere; flowing stone sculptures with names like “Temple of the Sun” or “Doll Theater”.  For the younger generation: it’s the largest Minecraft cave you’ll ever see.  🙂

We ride the 800ft (!) elevator lift back to the surface and decide to stay for the evening bat swarm (it’s still to hot to drive).  Every evening at dusk between 250 thousand and a few million bats leave to go eat mega-tons of insects up and down the local rivers (the numbers fluctuate so much because the bats migrate frequently).  We hang out in the local gift shop & cafe for a few hours (always a bad plan when on a budget), then try to watch a movie in Nessie (AC keeps it tolerable in there, but it’s still pretty warm), and finally evening rolls around.  We settle in to listen to the rangers and then finally the main show: 250 thousand bats fly out of the cave like smoke on the wind.  There’s a faint odor of bats in the air, and an endless murmuring of chirping bats and the little winged creatures are flitting everywhere overhead before flying off the escarpment edge and off into the darkness.

We do another (not so long) night of driving, stopping at midnight in Fort Stockton, TX.  We get a longer nights’ sleep tonight, even if the location isn’t as glorious.



Captain’s Log Days 1 & 2

Captain’s Log, Day 1

Today’s the day for the start of our epic 3-week 7000-mile cross-country RV trip of doom!  I’m up (fairly) early as I need to pick up all my kids – and their extra clothes, toiletries, games, meds, etc – by 9am.  Then I take them back to my house to begin packing in earnest, except for Josh who I need to take to the eye doctor’s to replace his glasses (broke under warranty) and Laura – who left her drawing pad behind.  I also need to drop my ex-Sprint AirWave back at the UPS store, and go by the pharmacy for a month’s worth of meds, and get fresh fruit for the RV, and… and … and … you get the picture.

Meanwhile Shelley is busy doing last-minute packing of Nessie, our 7-ton 31′ Class C RV – all the fruits & veggies & cold-stuff go in in the last minute.  While I’m running around frantically driving kids all over creation, Matt figures out he’s got a total of 3 pairs of underwear at my place, so Shelley is out driving him to get some undies (and other stuff we need) while I’m running my errands.  Despite all the crazy start and hasty lunches we actually hit the road as planned right at noon.

So on this trip we have: Me & Shelley (a red-head), my eldest daughter Karen, Luke (another red-head), my son Josh, my 2nd daughter Laura (also a red-head but no relation to Shelley) and my youngest Matt.  We’re off to see the country and all my scattered relations.  I’ve got my Dad (& Jane) living in Tucson AZ the kids’ other grandpa Zade in Luling TX (outside of San Antonio TX), my sister (Aunt Ruth) and mom (Pat Ireland) in Katy TX (outside of Houston), my brother in Atlanta GA, and my Uncle Bill and his 4 daughters (all my age) and their 15 kids (all my kids’ ages) in eastern Connecticut.

We’re starting out of San Jose, heading over Pacheco Pass to I-5, then south towards LA – but we badly do NOT want to hit LA right at rush hour, so we eventually cut over to Bakersfield and then follow some long long slow farm road across the central valley to Barstow… and up to Calico, a ghost town.

Now when Shelley was a kid, her mom would drive this very road (to visit her grandparents in Vegas) and they would stop by Calico once a year or so.  She has some fond memories from her childhood so visiting Calico is somewhat of a pilgrimage to her.  We arrive there right at dusk and can’t find anybody manning the entrance booth, so we sheepishly drive (our 31′ RV) quietly into the town – and promptly find the RV campground.  It’s basically deserted (there’s 1 other camper there, and space for maybe 100 vehicles), has power hookups and bathrooms with showers and running water… and it’s free, at least for people
arriving as late as we did.  We got out, stretched our legs and enjoyed the beautiful pink sunset over the red red hills, made sloppy joes on Nessie’s stove and ate on the picnic tables in picture-perfect weather.  Laura got the neighborhood dogs to howl back at her, Karen & Luke made videos of the epically blowing Laura’s hair, Matt climbed the hills and Josh & I ninja-sparred.

It was a picture-perfect ending to the 1st day.

Captain’s Log, Day 2

We walk though Calico the next morning.  It’s cool desert morning air, with some wonderful history.  The town’s been cleaned up a fair amount since Shelley was last there but remains a really nice tourist trap.  Mission accomplished, we head out for the long hot desert drive to Tucson to visit my Dad (Grandpa).  It’s a *long* boring drive down I-40.  Shelley is an awesome long-haul truck-and-horse-trailer driver, so driving this RV thing is a piece of cake.  (and while I’m up getting Shelley a nectarine, Laura types in my blog: “Moo” and “He has yet to notice.”)  Karen is talking about whale sperm shampoo (*not* sperm whale shampoo)… and the generator cuts out – it’s overheated.  That means the main compartment AC cuts out.  Oh – did I mention that on the long uphill grades the cab also AC cuts out?  (I assume because the engine is working too hard?).  So we pressed on in the 110-degree heat, across I-40, down “highway” 95 (looks like asphalt thinly spread over desert dirt, there’s a whole lotta “dips in road”).  Back on I-8 and heading west, and 2hrs out of Tucson and we’re all baking on-and-off (as the cab AC cuts in and out, and the cabin is slowly climbing above 90degrees)… when we blow a tire.

Yup, 20 miles from Nowhere, AZ, down that long & lonely road… we suddenly pick up a shake & shimmy… and a list.  We hove Nessie over to port and off the side of the road.  I tremulously step out to survey the damage.  Outside right rear tire has blown big, completely come apart.  It’s 1 of a dually, and the other is squashed under the load but holding.  Time for some quick thinking; we are baking and a long way from anywhere… and lame.  We check the phones: we have cell service; Thank You T-Mobile.  We call AAA.  They don’t do RV tires but they do give us the number of RV Medic in Casa Grande… which isn’t open after hours.  We get the answering machine & another number to call… also no answer.  So now we’re calling all about (at least 3 phones making active calls at this time, plus Google map’s are in hot action).  We decide to limp into Casa Grande.  We dump the tanks (not the black!) and push the kids over to the “good” side to lighten the load.  We also batten down the hatches, as Shelley points out that if the remaining tire blows we’ll “drop hard”.  Casa Grande is about 20miles down the road, and we decide that 40 mph is probably a good max-speed so we start off.

Then the dust-storm hits.  NO I AM NOT KIDDING.  We’re lamely limping along when the wall of dust hits, obliterating the “Blinding Dust Storms” road sign.  So now we’re limping blindly along getting buffeted by 40mph winds and dust (and tumbleweeds ARE blowing by, queue lonely wild-west music please) when the rain hits.  Yes: thick dust on our windshield AND ITS RAINING NOW WITH THE BLOWING DUST.  Nessie soldiers on.  20min later we pull off I-8 and out of the storm and head down some lonely farm road… but with the lights of Casa Grande clearly in the distance.  We pull into the first big lot we see (Big Tires empty lot) step out and see a rainbow.  Back around to calling RV Medic we get a human, we tells me to call Ed W’s who DOES do after-hours work.  $200 minimum charge.  Ed (who also requires 3 or 4 phone calls to reach) promises he can work on us, but can we get to town?  No problem.

While we wait at Ed’s shop for an hour (his mobile guy is on another call), Grandpa & Grandma drive up from Tucson and take the 3 younger kids back to their place and feed them all manner of treats.  We (4 remaining) older kids mosey over to a nearby restaurant and get dinner and some heat relief.  Another hour later and I’m $400 poorer and sporting a brand-new tire.  We pile in and make it to Grandpa’s.  Much sighs of relief, and a good nights sleep was had by all.



Progress + Vacation

It’s been a freak’n month since I last blogged!  Where’s the time gone???

Mostly I’ve been furiously coding.  ‘wc *java’ of our ‘src’ directory now reports 31500 lines.  We’ve cleaned up and CSS’s the web interface.  We added LevelDB to handle zillions of small K/V pairs (larger ones go to the local file system directly, and of course we still handle S3 and HDFS natively (either using an existing hadoop install, or directly *being* a distributed hadoop)).  We’re still 100% peer-to-peer, even for the direct HDFS stuff.  Last week I hacked a concurrent Patricia Trie (leaving the making of a *distributed* concurrent Trie for later, but now I know how to do it…). Then we ran all 36Gig of Wikipedia data through WordCount, using that Trie – it took less than an hour on 1 node.

This week it’s about running a Linear Regression *distributed*, using distributed Fork/Join as the programming paradigm.  Also, integrating a HashMap-in-a-Value (so we can pass about & maintain the Map interface in the Value piece of our K/V store – think: distributed JS objects), plus the final bits of VectorClocks (all behind the scenes; the VCs will let us do atomic update and strong coherence of Keys but they’re a horrible API to expose).  We’re building a toolkit approach to solving the problem of building a reasonable database over the Cloud.  Either (distributed) Patricia Tries or (distributed) Concurrent Skip Lists for range queries, plus JS-like objects in Values, plus atomic (transactional) update of individual JS objects using a Compare-And-Swap like approach (instead of locking: CAS is much faster under load, as threads can optimistically make progress).

More on all of the above later this week – as we have a hard deadline to finally *open* our Open Source project.  Yeah, yeah, yeah, I’ve been hassled plenty about calling ourselves Open Source and not (yet) having any open source… we’ve been trying to get the basics done first… but the real news:  I’m finally going on Vacation!!!

Yes, Nessie, the 31′ 7-ton Class C RV of Doom is being prepared for our 7000 mile Epic Cross-Country Journey.  I’ve been wanting to do this for a decade now: take the entire clan (7 of us!) across country, touring all the junk tourist traps we can and visiting our scattered family as we go.  We got family in Tucson AZ, San Antonio TX (well, Luling really), Houston, Atlanta, DC area, and Connecticut.  I’m giving an invited lecture at UIUC on our way back, and have been assured I can use that lecture as a reason to declare this a “business trip”, and deduct all the gas and mileage costs – I figure about $3500 in gas alone.  We stopping at Stone Mountain in GA over the 4th of July, visiting my brother and camping at the lakeside facing the mountain where we’ll watch the fireworks and lazer show from the RV roof.  We’re going to visit Carlsbad Caverns.  We’ll pass through DC and maybe attempt the Smithsonian (not sure about that one; depends on the schedule and how badly I want to fight the RV through DC traffic).  We’re visiting my Uncle’s classic family farm in Connecticut where my 4 cousins live – all my age, all married with 3 to 4 kids each… all about the same age as my 4 kids.  We’re talking now about 15 to 20 neices and nephews, plus Aunts & Uncles galore, and of course pigs and chickens and horses.  It’ll be a regular zoo.

So if you see a large white whale heading east on I10 with a frazzled Shelley or my excited 19yr-old at the helm, honk, wave Hi and give us a wide berth…



Quote(s) of the Month from Kevin Normoyle (Sun/Sparc & Azul L2 Cache Designer Extraordinaire, Cache Coherence Advisor to 0xdata):

Reminds me of CS101, on one of my first programs.  The grader wrote in big red letters over my big comment block:
“Don’t document your bugs, Fix them”

So I asked Kevin if I could quote him, and I got this response back:

ah that’s fine…I spout “Advice” left and right to everyone… Many dismiss it as “Rant”.  There’s always that fine line between being a Prophet, and just another crazy guy standing on the corner yelling.  One could argue that everyone who every posts to Twitter is an “Advisor” of some sort, to the world.

Sound advice, from a (reluctant) adviser to the world.

What’s Going On?

As alluded to in my last blog, here’s my fun hack de-jeur: “Whats going on?”

I’ve got a multi-node setup with UDP packets slinging back and forth, and each node itself is a multi-cpu machine.  UDP packets are sliding by one another, or getting dropped on the floor, or otherwise confused.  I’m in a twisty maze of UDP packets all alike (yes, I played the game back in the day).  Then something crashes, and pretty quickly the network is filled with damage-control packets, repair & retry packets, more infinite millions of mirror reflection packets.  What just happened?  I press my handy little button and…

… a broadcast of “dump, ship and die” hits the wires (a few extra times for good measure).  All my busy Nodes stop their endless chatter and dump the last several seconds of packets towards my laptop, slowly & reliably, via TCP.  Each node has been gathering all the packets sent or received to (well, the first 16 bytes of each) in a giant ring-buffer, along with time-stamp info and the other party involved.  After I ship all this data from every node to the one poor victim (that I pressed my button on), every other node dies (to prevent further damage).

The Last Survivor gathers up a bunch of very large UDP packet dumps and starts sorting them.  Of course, you can’t just sort on time, that would be too easy.  No, all the nodes are running with independent clocks; NTP only gets them so close in time to each other.  Instead I have to sort out a giant Happens-Before relationship amongst my packets.  I am helped (above and beyond some sort of home-brew wire-shark) by my application understanding it’s own packet structure.  I *know* certain packets must be strongly ordered in time, never mind what the clock says.  For example, I only send out an ACK for task#1273 strictly after I receive (and execute) task#1273.  Paxos voting protocols follow certain rules, etc, etc.

In the end, I build a very large mostly-correctly-ordered timeline of what was just going on, as seen by each Node itself, and then HTML’ify it and pop it up on the browser.  Voila!  There for all the world to see is the blow-by-blow confusion of what went wrong (and generally, the follow-on error “recovery” isn’t all that healthy, so more broken behavior follows hard on the heels of broken behavior).

Basically, I’m admitting I’m a tool-builder at heart.  As soon as I realized that standard debuggers don’t work in this kind of situation, and wireshark couldn’t sort based on domain-specific info (and pretty-print the results, again using domain-specific smarts), I went into tool-building mode.  As of this blog, I’ve found several errors in my cloud setup already; e.g. a useless abort-and-restart of a Paxos vote if a heartbeat arrives mid-vote from an ex-cloud-member (that’s alive and well and wants to get back in the Cloud), and some infinite-chatter issues getting key replication settled out as nodes come and go.

On other fronts, my car came back from the body shop, only to turn around and go back to the engine shop: the timing belt had slipped.  The work was done under warranty and I’ll go pick up my car on Monday.  I can hardly wait!!!

My GFs car’s brakes have been squealing for weeks; they finally started shuddering and we decided it was time to fix them.  She’s driving a 1993 Nissan Maxima with 220K miles on it; weird things start breaking at that age, but mostly the car just soldier’s on.  But it was time for the brakes.  We pulled the rear pads & looked at the rotors: one of them was shot.  Fortunately a new rear rotor was only $25, plus another $22 for pads (tax, brake grease, still under $50).  We couldn’t get the dang pistons to move back! We tried at least 5 different wrench/jig/clamp combos to no avail.

We figured the pistons must have been jammed with debris, so with great trepidation we pulled the brake fluid line, the emergency brake cable and pulled the whole unit to my workbench.  I popped the piston out manually.  It looked clean and good… and had this funny thing in the middle… stupid me, failed to check the internet again… it’s the anti-slip mechanism for the emergency brake.  You have to *spin* the piston to screw it back into the cylinder.  Sigh.  It took us another 1/2hr to find the right tool to spin the dang thing, but it finally went in without too much trouble.  After that it was another hour to reassembly all the parts, and then we had to bleed and bleed and bleed the line.  As of this writing, the pedal is still to soft, I suspect we need to bleed it some more.

Daughter is at the Old Salts Regatta, plus a ton of driving to meet people for 0xdata, plus a much needed dinner out… and down 2 cars (GF’s brakes-in-progress and my car in the shop), made for a very complicated week.



Hit and Run

I was driving in Palo Alto with 2 of my kids in the car heading towards Fry’s, looking to get the largest monitor I could reasonably buy.  I was driving down one of the many fairly narrow 4-lane side streets, heading towards El Camino Real when a driver pulled into my blind spot.  With a block-and-a-half to go to the red light, the driver very slowly started to overtake me… then he started edging into my lane.  I honked and edged up and over to give me some space… with now much less than a block to go the driver suddenly floored his vehicle and came over into my lane (his lane was ending quickly, being filled with large yellow Caddie parked at the light).  I hit the brakes but couldn’t get any more over (no shoulder, sidewalk full of telephone poles and other crud).  Crunch… I was fairly certain he had hit the Caddie, and I knew he had hit me.

I checked my kids, all ok.  I looked out the window at the other car & the Caddie.  He was between the Caddie and me, and was trying to back down the street – but got blocked by cars coming up behind him.  He stopped moving and started getting out of his vehicle, so I took a breather and got a clearer look – there were lots of cars, lots of honking; my car looked drivable and in the way and there was a Jiffy Lube at the corner, so I waved to him and rolled around the corner and parked.

I got out and headed over to his car… and he was busy yelling and screaming at the Caddie driver (who had been parked at a red light the whole time!).  He looked at me and got back into his car and rolled up to light and around the corner, clearly to join me at the Jiffy Lube… and took off instead.  Gone.  I was dumbfounded, and stood looking at his vanishing car like an idiot.  One of the Jiffy Lube guys came up to me and asked if I was ok; I nodded and went over to check out the Caddie guy.  He was a timid old man and was really shook up by the screaming-at he just had.  It was clear his vehicle was blocking the road and looked untouched.  I stuck with him until he felt better, then he got in his car and drove off.  I walked back to mine to survey the damage and double-checked the kids.  Sure enough, a big dent with lots of missing paint.

Then the Jiffy Lube guy volunteered that he had heard the crash, had come around the corner, had watched the driver chew out the Caddie… and decided the driver looked “off”.  He had the plate number written down and encouraged me to call the cops.  So I did; and sure enough in about 5 minutes a Palo Alto policeman showed up.  He was very polite and called in the plate#, and started taking statements.  In another 10 minutes he reported that they had caught the guy!  Then he offered me a chance to eye-witness the driver where they had stopped him… so I followed the cop back across Palo Alto and did a slow drive-by… and indeed it was the same guy.  They offered me the chance to press charges – but they also reported he didn’t seem drunk/drugged – so I decided to do this with just insurance instead.  Maybe he was just having a bad day.

Today AllState came out and looked at my car, and hauled it off to the body shop.  Maybe in a week I have a car with a fresh paint job?  Kudos to the Palo Alto police department; that was quick work catching the guy.  Kudos to the Jiffy Lube guy who had the brains to get the plate number.  And I did eventually get my monitor.

You didn’t think I was going to talk about technology, did you?



What I Did For My Christmas Vacation

Warning – contains no technical content …

My GF and I both love to drive (I own an Evo which I love, I drive really hard, and it’s been a rock-solid car, now with over 120K miles).  The kids were with their mom for a week-and-a-half, so GF & I took off for Bryce Canyon and the Grand Canyon.  She did most of the driving on the way out – both because she loves to drive my toy! – and because I was stuck doing PLDI reviews (sign up now!).  I did half my reviews from the road, uploading reviews at passing Starbucks.  Along the way we passed through Casa de Fruita, a 100-yr-old over sized farm stand, classic California at its best – complete with toy riding trains, carousel, restaurant, hotel, and of course plenty of fruits & nuts (not referring to myself at all here).  I basically run on chocolate and coffee, so this was a fuel-stop for my brain.  GF stopped to feed the pet peacocks.   Since this vacation is being done “on the cheap” we stayed at a smaller hotel outside Bryce.  It was fine (if a bit plain) and the price was right.  No WiFi in theory – but it was available in the lobby, and we finagled a room just behind the lobby area which DID have WiFi… so I finished off my PLDI review uploading from our hotel room after all.

Typical Bryce Canyon weather prevailed for several days: snow flurries with lows down to 9!  After a day recovering from driving (we had put a LOT of miles under our belts!), we checked out Bryce Canyon.  Fantastical views (and GF found a raven to feed)… but obscured by snow flurries and low slung clouds.  Next day dawned with crystal clear-blue skies and winter wonderland style snow everywhere; red red rock, “hoodoos” and dark evergreens.  The colors still sparkle in my minds eye… pictures here.  We took a 4 mile hike along the rim, then down in and around the “hoodoos” and back out the other side.  Being the brilliant person I am, I forgot the camera battery charger (and none for sale in any of the local shops)… so the digital camera died at the bottom of the hike (we bought a throw-away camera for the next hike, but the photos aren’t back yet).

Next we headed for the Grand Canyon.  We’ve been planning on hiking it for a couple of months now, and we lucked into a cancellation in the Phantom Ranch barracks (typically you need to reserve space a year in advance!).  Along the way there we made good time… except for the one dreaded moment when that SUV on your bumper suddenly spouts red & blue lights, and the sheriff wants to talk about safe driving speeds in the mountains of Utah (warning only: maybe 2nd time in my entire life I’ve been pulled over and only received a warning).  We got in very late (got stuck behind a SUV doing between 15 and 25mph on clear roads labeled at 45mph, but it was night, on national park roads w/double yellow strip; very frustrating that the person would not pull over!) and so collapsed on the hotel bed without seeing the canyon that night.  Next day we walked over and ogled the canyon.  It’s still beyond my words to describe, even if I’ve seen it (hiked it!) before.  Its one of those things you just got to experience in person.  GF’s never hiked it before, so we took our time…

Day One.  After checking out the trail heads and talking with the rangers we decided to buy crampons.  BEST DECISION EVER.  Hiking in Bryce on the snow taught us that there’s ice under all that pretty snow… dang slick ice.  We hiked down a full mile of ice from the South Kaibab trail, then the trail dried up and opened up into the amazing views.  Only place I know of you can stand with a 1000+ foot dropoff to one side, and yet be looking at a 1000+ cliff towering above you on the other hand…  and both be yet another 1000+ further down from the rim and yet ANOTHER 1000+ feet above the river!  It’s a real-life 3-d eye candy mind bender.  We hiked it down in 5 hrs.  Not bad for this 50yr old pile-o-bones.  Sore and triumphant we walked into the Phantom Ranch main room.  Our (segregated) bunk beds awaited… but they had had another cancellation.  For a mere $30 more, we got a spacious 10-sleeper cabin to ourselves!  We rushed over to the cabin, and sat down on a bed to “rest a bit”.  Two hours later, we awoken from a deep nap and realized that dinner was fast approaching…  Dinner was delicious (as only a hearty steak dinner can be after a day of hiking).  After dinner we took a star-light stroll in the pleasantly cool evening, laid down besides the rushing creek and star-gazed awhile.

Day Two.  Day One, of course, is the day you act like an idiot and climb down this great big hole.  It’s on Day Two that you have to fix the situation by climbing out.  The first hour of waking we spent discovering all those muscles (and joints!) that got overworked yesterday… and being ravenously hungry despite the giant meal of the day before.  So after a big breakfast we started back up, the Bright Angel trail this time.  Other than the amazingly scary river crossing on the swinging suspension bridge (the Colorado is wide and fast there, and the bridge very clearly was swinging more than 5ft side to side), the upward hike was … upward.  Hard work undoing all those steps yesterday, although the uphill is MUCH easier on the joints.  The crampons came out again for the three miles of snow and ice near the rim… and we made it up in 7.5 hours.

Next day we drove back home, 750 miles in one day (again).  Next day we cleaned up, picked up, packed up… and the kids arrived here for Christmas!  We had a wonderful family Christmas; GF made a fantastic dinner and there was much regaling of tales and playing of video games for all…