Fast Bytecodes for Funny Languages

A weekly podcast with Cliff Click talking about all things to do with programming, programmers and computer performance.

This is a short talk on generating fast bytecodes and concurrency library throughput and some performance failures to look out for

 

Debugging Data Races

A weekly podcast with Cliff Click talking about all things to do with programming, programmers and computer performance.

This is a short talk on debugging one of the most confounding bugs in programming: the Data Race.

 

Java vs C/C++ – The Podcast

A weekly podcast with Cliff Click talking about all things to do with programming, programmers and computer performance.

A podcast of a blog I did here in the past, which turned into a great talk that I’ve given lots of times over a decade.

 

Introverts, Emotional Processing, Self-Esteem and Salary Negotiations

Introverts, Emotional Processing, Self Esteem and Salary Negotiations

This is a story about myself. But every time I tell it to a room of introverts (e.g. programmers), it resonates with many in the audience.


Has this ever happened to you?

I’m quietly hacking away on something new when:

“Hey Cliff! WTF! I went to show off the new module (which is alpha quality) to the customer and it crashed! I got egg all over my face!”

and now I’m sorting out why I feel ashamed, and before I can respond I get:

“Your supposed to be some hot-shot programmer! Why didn’t you test it!”

and of course I did, but it’s alpha quality and it has many known bugs…

“Your crap is crap, get outta my sight!”

and now I’m really reeling…
and the guy walks off.  About an hour later I figure out what snappy comeback I should have said, but I never figure it out in the moment.

What just happened?  Why was I stun-locked by the verbal assault?  The speaker should have known the code was barely baked and known unstable; the entire dev team certainly did.

And so why yell at me for doing something foolish?  Obviously the fool was not me…
(but wrong question, because the answer to why Fools yell is not important)

Right question is:
Why was I stun-locked?
Why couldn’t I think of the snappy response?

Because I can’t do verbal processing and emotional processing at the same time!!!

This has profound meaning to me, and figuring it out has literally changed my life – hugely for the better – because I can now (and do) compensate for it.


Let’s replay the above conversation, showing my inner thinking as well:

I’m peacefully coding. My emotional guard is down. My creativity is up.  I’m building something and enjoying working “In The Flow”.

“Hey Cliff! WTF! ….”

There’s a verbal assault on my ego, without any warning. I’ve been surprise attacked (with words). My ego is rebounding from the assault, and now I’m trying to figure out why this guy is mad at me (hasn’t sunk in yet he’s off trying alpha-quality code on a customer). Before I get much farther…

“Your supposed to be some hot-shot programmer! Why didn’t you test it!”

There’s a second assault, this time directly at my ego & self-esteem.  Like many programmers I am proud of my abilities. The “supposed to be” comment is directly derogatory against my self esteem and the “why didn’t you test it?” is being presented as an example of my poor coding skills.

Again, I am emotionally processing the assault. By now I’ve sorted out that his original action was unreasonable (presenting alpha code to customers without setting expectations), and he’s engaging in ad-hominem attacks against me.

“Your crap is crap, get outta my sight!”

My emotional barrier is up. The further ego assault is blocked. I begin making a judgement call on his character… and he walks off. The sudden attack ends, as rapidly and unexpectedly as it begins.

I’m now emotionally wrecked, and pulled completely out of The Flow.  My adrenaline has charged me up, and I’m highly stressed.  It takes me a good hour to calm down before my ego rebounds somewhat (there’s a definite hit).

After I calm down, I can begin to replay the conversation in my head, intellectually, and recognize an unreasonable assault… and start to sort out all kinds of intellectual responses to his emotional attack.  The “snappy comeback” eventually materializes in my head, hours too late.

I can’t do verbal processing and emotional processing at the same time.  The verbal assault starts me down the path of emotional processing, and I’ll not have any words until I sort myself out.


What can I do about it?

I can (and do) memorize words, that I can say without any verbal processing.  Rote words; blind words; suitable for all such attacks. Here’s mine, but the actual words are less important than the ability to get them out blindly:

“I need some space to sort this out. I’m leaving.”

And I walk away.

There’s a key add-on phrase, if you need to do this with somebody who you have a relationship you value:

“… and I’ll be back”

That’s it. Hang up the phone. Block the calls & texts. Crucially you must walk away, to end the assault. Run if need be, or call 911 if the attacker runs after you. Nobody needs that kind of abuse, and that’s what it is, pure and simple: abuse.


This same pattern shows up in my love life as well!

I’ve been dating for awhile now, since getting divorced. Sometimes during the first few months of a new relationship, while things are going really well (from my point of view) I get the question: “Cliff, do you love me?”

And I’m speechless.

I’m thinking “what happens if I say ‘no’? Will she walk away? Is she so concerned with ‘love’ at this point in the relationship? I like her alot, but ‘love’ after a few months? No takes longer… We’re still sorting out the basics of the relationship.

And as for my actual feelings towards this person… that’s a deep search of the soul which I can’t do while she’s hanging on me and whispering in my ear.  I’m highly distracted.

And in my silence, she interprets what? Rejection? More worries, more concerns from me… and yes in the past actual real miss-communication because of the silence.

i.e., I’m “stun locked” doing emotional processing and have no words.

So now I this prepared answer (which doesn’t take any emotional processing):

“I don’t know really. I have to search my soul to answer, and I can’t do that right now… that takes days. Last time I checked myself, I really really like you… but it was not love, at least not then. Let me get back to you on this in a few days”.

And then there’s some confusion, so the whole of this talk is generally presented somewhere along the way; about my “no emotional and verbal processing at the same time”, and the need for space to understand myself.  I have found that my partners are all understanding and accepting of this – it’s something direct and concrete about how I process emotions, and easy to understand.  (Of course, if they are not understanding of how I’m built, that’s something for me to sort out as well).


This same pattern shows up in my sex life as well

Sex is highly emotionally charged, at least for me.

I cannot talk during the act, and no amount of poking and prodding has been able to change that.

What can happen is that my partner demands a verbal response from me.  And in the demand of it, I do not feel accepted for what I am.  And then I’m doing more emotional processing- but of my self-esteem, and my relationship and not of sex.  My sex drive is halted.  A demand for “sweet nothings” leads to a cold bed.

While I can’t talk during the act, I can listen, hear and act on words; especially the gentle guiding words.  I can “speak” with my actions.  I just cannot bring up my own words, until after my own climax.

The compensation for this is the same as before: prepare the words before the emotions hit; talk it out in the light of day, while fully clothed.  I make sure she knows that at some point during the evening, I’ll become less and less verbal (and more and more physical) until after the event.


This same pattern shows up in my office life – in other people

Somebody is excited about a solution they’ve come up with, and they’re sharing.  Loudly and excitedly! They are often saying the same things over and over again.  A few other people get excited, and so start talking about problems with the solution, and other solutions and alternatives, and there’s a wild brainstorming session rolling on…

…and quietly sitting in the corner, is somebody with a bright idea, maybe the best answer, who’s too afraid to speak up.

They’re thinking: “what if I’m wrong? I’ll look like a fool!” or maybe “I wish I could talk, but I can’t!” or maybe “They won’t believe me anyways…”

They’re doing emotional processing, and aren’t able to come up with the words, nor the courage to speak up.

As a leader, it falls upon me to spot this pattern – the verbal action from the loud few on one hand, and the quiet listeners on the other – and take action.  I don’t want to stop the brainstorming session, not yet, but I want others to have a chance to join in . I usually end up saying something like this: “Larry, I heard you suggest XXX as a solution, is that right?” – I want Larry to know he’s been heard so perhaps he’ll back off promoting the idea so loudly if we’re already on the same page.  I need don’t say it’s the right plan or wrong plan, only that the concept has gotten across.

And after that acknowledgement, I’ll offer “I see some quiet listeners – anybody else want to add in?” – and we’ll all pause talking, which makes a verbal space – a gap in the flow of words – which can be easier for a shy person to step into.

Generally, I work with a lot of people who are shy with words, and I have to be careful not to be the “Loud Larry” myself – lest I shut out the brilliant suggestion from the quiet person. I have to spot the pattern where one person is “verbally dominating” another, and give all folks a “verbal space” to step into.

Many of the people I work with – programmers and introverts of all kinds – appear to have a fairly low “emotional IQ” – they’re not necessarily aware of why something has upset them, only that it did.  I end up doing a lot of quiet conversations, where I help people understand themselves.


The Aspects of Farmer, The Mage, and the Warrior

We all exhibit some of each these aspects:

  • The Farmer enjoys the hard work, and building with his labor.  His ego is boosted by what he accomplishes with his physical labor.  “Today I built a house!  Some family will live in it someday!”  Tradesmen are an exemplar.
  • The Mage enjoys the “smart work” and tool-building “magical” things.  His ego is boosted by being “smarter” or “working smarter”.  Programmers often have this aspect strongly, but also Doctors and Experts in all walks of life.
  • The Warrior engages in a fight, and might indeed revel in it.  Politicians, CEO’s, Lawyers, wheeler-dealers, and used car salesmen fall into this camp.  The fight’s the thing, not really what is being fought over.

Programmers (and introverts?) often do not practice with their Warrior very much.  Their Warrior is unskilled and naive.  They can be pushed into the defensive very easily and dominated by those with more “fighting” (verbal and otherwise) skills – “alpha males”.

What would a well functioning Warrior think of my opening conversation?  Well, what do you think would happen if a Lawyer was on the receiving end of:
“Hey Cliff! WTF! ….”
The Lawyer might respond:
“You’re a f*cking idiot! That was a fool thing to do!!!!”

The assault is returned immediately – an ad-hominem reply to match the opening tone of the fight, and then very not-subtle pointing out who was the real fool.  The Lawyer’s Warrior is ready for the verbal fight, and engages willingly.


Stress and Anger Management

“Hey Cliff! WTF! … Your crap is crap. I’m outta here.”  And he walks away.  The fight is over for the Fool, but not for me.  My adrenaline is raging, my Warrior has picked up my Ego armor, and my body is spoiling for a Fight or Flight response . That adrenaline and stress doesn’t just fade way immediately.  It sticks around, perhaps for a long long time. The stress and anger might stick around for years. I’m angry now.  I want to act, my body is expecting, demanding that I act.

You’ve heard the term “kicking the dog”?  I’m angry.  I want to Attack. Somebody, something, anything.  But I’ve enough brains not to attack somebody who might hurt me, so I want to attack somebody who is safe.  The dog.  My S.O., my kids, the person next to me.  I’m angry at the Fool, but I’ll attack whoever is convenient and safe.

Wrong twice. Once because I end up attacking the very people who are trying to help me. And twice because my anger isn’t directed at the source in a useful way.  The Fool is safe to attack me again some day.

And yet, we still need to get the Anger out – lest it turn inwards.  Have you ever gone home after such an event and stewed on it?  That’s anger turned inward, and it leads to depression.  High stress leads to health issues.  We need to get rid of the anger and stress in a useful way.

The Key: Get the Anger Out, but not At

Out, not At

Getting anger out requires some physical activity to get the adrenaline to metabolize.
Not at means picking an activity that doesn’t focus towards another being.

Go for a run.  Go to the gym. Walk around the block. Get too a tennis court, imagine the Fool’s face on the ball and smash it.  Again and again, until his face fades, and the anger fades, and you’re actually doing something aerobic.  I’ve grabbed a pick axe and gone to my backyard, and smashed stones.  Bang!  Rock chips fly, sparks fly, bits and pieces go in all directions.  There the Fool is again!  Wham!  More sparks!  It’s very therapeutic and actually really hard exercise.  After 10 or 15 minutes I’m hot and sweaty, I’m over being angry, and I’m looking at a hole… and my Mage kicks in and asks… “what can we do with this hole?  Plant a tree?”  And my Farmer speaks up: a tree-hole needs to be more round, and about 2 feet deep, and a little to the left.  More sweat, happily applied (and now I have 2 large mature apple trees growing in my back yard).

Working Through The Emotions

There’s another step, once the immediate anger has faded, and that’s working through the emotions and the experience – and accepting what happened (I got verbally attacked by a co-worker), and moving on.  For me, I have found that talking to The Village is a good solution – the Village of whoever happens to be around me; friends and co-workers, family, strangers, whoever.  For other people it might be different: e.g. artistic outlets like painting or singing.  A long quiet time alone (such as hiking) with lots of Zen time, perhaps in a beautiful place.

Know that you need time to work through the emotions and give yourself that space.  When the hub-bub of life settles down, and the kids are in bed, and the quiet descends on the world – don’t reach for noise-makers to fill the space (TV, movie, loud friends with alcohol) – instead, give the time to your inner self.  Work through what happened in whatever way is best for you.  Don’t stew – reflect.  If you find yourself stewing on it, you need to find another way to work through it.  Maybe a conversation with a close friend or parent or other safe person.

Just like any body injury you need to heal internally.
Give your soul some healing time.


Red, Yellow and Green Flag People

Make this judgement call on the people around you: are they Red, Yellow or Green?

Green flag people are safe to be around. It’s the friends you trust.  The ones you can bounce ideas off of, without fear of getting sarcastic responses.  You’ll get safe honest answers, no “hurting” words, and in return you can share your inner feelings and honest opinions back.

Red flag people are unsafe to be around.  You cringe when you see them walk up; you raise your emotional guards. Your creativity and openness ends… because sometimes they attack you, or they are passive-aggressive, or making endless (seemingly reasonable) demands. They are dangerous to your self-esteem and ego.

Yellow flag people are somewhere in the middle; folks you can be around but maybe you only drop your guard carefully.

Why make this call?

Because it brings your Mage into your relationships, to help you figure out which ones are useful and which ones carry a burden – and now you can make an intelligent decision about which the usefulness vs cost trade-off.

Example:  Angry Ann always has some biting sarcastic thing to say about the people around you (and you wonder what she says about you behind your back), but also she’s a crazy fun party animal.  Thinking about your relationship lets you make the decision  intellectually: is Angry Ann worth hanging around?  Sometimes?  Always?  Never?  Bring your Mage to bear.

Same call for Green people: mine is my brother.  Divorce is on, boss yelled at me, health claim denied, got a ticket doing the same speed as every other person, whatever.  I know who to reach out for for support.  Knowing that, means I know when and where and how fast I can get emotional help… and that alone mitigates most of my emotional trauma. Know Thy Greens, and Use Them.  People really do like helping somebody else, and it’s always very life affirming when I end up being somebody else’s Green.

What To Do

Lose The “Reds”.  Really.  Try really hard to lose all the Red flag people in your life. You aren’t going to change them (people can only change themselves), and you don’t need the stress and negativity in your life.  Life is too short to run around with your armor always up, getting ready to duck from the next verbal assault.  Your creativity, your passion, your love and skill – all will suffer while you are constantly on the defensive.

It gets worse if the Red is somebody close in your life (parent, S.O., boss) – you can get trapped in “siege mode”, constantly on the deep defensive. Only being reactive, never active.  Slowly getting beaten down into a shapeless mass of sadness.  Internally you are screaming for help, but externally nobody hears you – and you need help to get out of this relationship.  Get That Help!  Councilors, Green friends, whatever, wherever – Act to Get Out.  Make that internal scream a real audible one.

How To Lose The “Reds FLAGS”?

Depends on where they are in your life.

Kinda-sorta-friends?  Pick your social activities with care.  Same for abusive family members when you are an adult: don’t bother hanging out with your abusive relative.

Boss/co-worker? Complain uphill (facts are crucial here, not just your words; emails, texts or co-complaints; starting a he-said-she-said fight is generally worthless).  And if that fails, ponder walking away.  Life is to short have a crappy job experience; go get another one and work happy.

Significant Other?  That’s a lot harder, and harder yet when kids are involved.  But know that there’s something wrong, if you are red-flagging some parts of your S.O. – it’s probably time for talk therapy or a marriage councilor.


Love, Change, and Mind Reading


No matter how much you Love somebody,
you cannot Love them into Changing

Change comes from within, not without.  No matter how much I cajole You, the Gentle Reader, with these words, I cannot change you.  Only you can change you.  Same for every body else in your life, and for you in their lives.  Somebody else might want you to Change, and you might even be aware of it… but unless you decide to Change yourself, their desires are for nought.

No matter how much you Love somebody,
you cannot Read Their Mind, nor they yours

So much broken expectations arise from this. Your S.O. just did something that hurt you. Did he do it on purpose? Why would he hurt me?  “I just need to love him more, and he’ll stop doing that.”

It’s much more likely that your S.O. has no clue that you got hurt – and if you mention it “hey, you know I really got hurt here” – your SO will probably say “what??? wait, no! I didn’t mean to hurt you!”, and maybe even “how did that hurt? I don’t understand!”.  If they are concerned about your well-being, then they’ll seek understanding of “how” that hurt, and Change – but the Change is driven from within – not by you Loving them, but by your S.O. Loving you.


Salary Negotiations

Funny joke I heard recently:

“Fighting with a Lawyer is like wrestling with a pig in mud. The Lawyer enjoys
it, you’re going to get really dirty and you are very unlikely to win.”

Same with used car salesmen: they sell cars to non-Warriors each and every day.  It’s what they do – nothing emotional about, nothing personal.  It’s just their job, and they probably enjoy it and they are certainly good at it.

Same with HR: they negotiate salaries each and every day.  Nothing personal, and no emotional processing, it’s just their job – to get your skills at the best price for the company.

The HR rep has a salary range in mind, and perhaps a recommendation from the team that’s trying to hire you. Their job is to get the requested skills (you’re only one of many they are looking at) hired at the lowest possible salary, and after that – they step out of the picture – no hard feelings, and no regrets… and no real interaction with the hiring team except to check-in now and then. Your actual work colleagues generally aren’t part of this process and probably don’t know what you end up going through (except that they, themselves, went through the HR hiring process years ago).

But it IS your life, and your standard of living perhaps for years to come.

After the customary pleasantries, the HR rep, like any good Warrior, launches into an (acceptable verbal) attack: “So, Cliff, how much are you making now?” or the very similar “So, Cliff, how much do you need?”

Immediately I’m thinking: “If I tell him, he’ll only offer me a little above that – but I know I’m worth more!” and “how is my self-worth being judged by a single number?” and “I bet he’s willing to pay more, but if I ask to much I won’t get a job!”…

i.e., I’m emotionally processing, and my un-skilled Warrior is on the defensive.  Pretty soon I’m backed into the “I really need this job! I better tell him!” corner and the fight is lost before it hardly begins.  I get the expected lame offer, and I’m feeling under-appreciated for years to come.

What can we do about it?
Well, we do what any Warrior should do: prepare and practice for the fight!

Prepare! Know Thy Worth. It’s easy to find out the basics: check out similar skill sets on glassdoor.com, monster.com, dice.com, etc… job boards everywhere will have other examples of salaries. There are a zillion books on interviewing skills and practice interviews and questions and strategies.  Getting that better salary is worth far more than the price of the book and a few hours to read it.  Spend The Time.

Practice!  Get a partner to play the HR role and set up as much as possible like the Real Deal: dress up nice (so you’re out of your comfort zone, just like the real interview), find an empty office and have your partner sit in the “Power Seat” behind the desk – another way you get put on the defensive – and play out a salary negotiation.

Practice (unlike prep) hits you in a different place, and trains a different Warrior muscle. Prep gives you intellectual ammo, and lets you think through what you should be asking, how to approach the problem and so forth: your Mage is at work. Practice is where your Warrior turns Knowledge into Action. Without the Practice you are very likely to stumble trying to apply your new-found knowledge. Don’t skimp on it!

More “Prepare” Advice:

BATNA: “Best Alternative To a Negotiated Agreement”.  If you don’t already know the term look it up.  It’s your “Plan B”, your fall-back position, it’s what you get if you fail to get the job, or fail to get the negotiated raise or whatever.  Knowing your fall-back position tells you where your “negotiation space” is, your “wiggle room”.  Your Warrior will need this information during the “fight”, readily and without thinking.  Prepare it ahead of time.  Your Mage can tell you if it’s really weak, and perhaps needs shoring up before going into the fight.

Example: I like my job and I’m good at it, but my pay sucks and I know others are getting paid more – I want a raise.  But if I walk in and they say “No” – what is my option?  Suck on the No?  I need a better BATNA – so perhaps I should go looking for another job.  A serious look, where in that other new job interview I can know that I don’t need it… but if I want it, I can have it.  Then when I walk into the salary negotiation on the job I really want, my BATNA is now “Suck on the No OR Walk Away to a New Job”.  It’s a MUCH stronger bargaining position.

For the “how much are you making” questions – prepare an answer ahead of time.  You never need to answer the “how much am I making now”, but you do need to politely answer his question.  Something along the lines of “what I’ve been making in the past is not really important here, what’s important is what fantastic skills I’m bringing to the table – what are they worth to you?”  Or “I’m asking $xxx, because I know I’m worth it” – and aim a little high.  The worse that can happen is: “we’ll have to get back to you on that salary request” – generally followed up by a counter-offer – one that you can think about in the safety of your own home on your own time.


This is the first time I’ve put this into words, but not the first time I’ve talked this out.
Sorry if it seems long and disorganized, I’m still sorting it all out myself.

Good luck in all that you do!
Cliff

 

Down from Eight Miles High… and Landing!

I’ve landed at Neurensic!  First I’d like to thank all the people entertained my request for a job – there were lots and lots.  Competition was stiff, and as the month went on, the offers got wilder and wilder and closer and closer to my dreams.  Ultimately I had the choice for taking VC funding and starting something of my own (still Out There, and will probably happen yet, and could be Flying Cars), and venturing into the financial tech sector, something I’ve long thought about. Fin Tech won the day, but I’d like to give a special shout out to Google (at least 4 different groups courted me, I really like the feel of Google).

The job hunt experience gave me an unparalleled view into the Valley’s outstanding tech problems.  Several problems really stood out – as things a lot of people wanted help solving, and I thought I’d mention them here as a sort of “State of the Valley’s Better Mousetrap Hunters”.

NodeJS – 3 different companies made clear offers to help make a server-side version of JavaScript targeting NodeJS.  This is right up my alley: adding threading (but keeping the single-threaded coding style), adding a low latency high performance GC, and adding A Better JIT.  Looking at the state of JS JIT’ing I cant help but think there’s a lot of performance being left on the table – especially for “long” running server apps.  Maybe some easy tweaks and V8 cleans up here… but so far the NodeJS audience ain’t feeling the love.

Python – At least 6 stalled attempts to speed up Python are Out There (PyPy appears to be “less stalled”) – and I got clear offers from 2 companies and sorta clear offers from 2 more to make “Python Go Fast”.  I also reached out to the Python core community – and got a resounding Silence.  I’m not fond of pushing rocks uphill; if the core community isn’t interested then I’m not (yet) willing to tilt at this windmill.  Nonetheless, given the response from 4 companies (some quite large) I’m willing to bet Somebody’s interested.  Sorta needs the Kickstarter-from-Hell to pay my bills, but then could be a fun gig.

Both the NodeJS and Python speed problems appear to not be getting solved anytime soon.  I might come revisit both of these areas down the road- perhaps more as a quiet easy-going slam-dunk job.

After that, mix in 4 different ML-related offers, 8 different startups (literally at least 8, more than one network-security-meets-ML), and several We’ll-Fund-Whatever-Cliff-Does startups.  And flying cars, no lie; that was the last-ditch almost winner.  But I need to be in a better state financially before I can bite into that apple.  All told I sorted through over 20 serious job offers. It’s good to have First World Problems.


Why Neurensic?  Why Financial Tech?

This story goes waaaay back – I’ll skip past the “I was born a poor black child…” part of the story and start in when I was about 26 or 27.  I was just finishing up my PhD and was starting to look at the job market – and Everybody was saying all the money was in being a stock market trader.  It looked too much like giving up on helping the world – just living for myself, so mercenary and narcissistic.  I still wanted to Save the World, and I liked my mix of theory and practice… so I ran off to do compiler research at HP Labs, then beat IBM on IBM hardware with my mad compiler skilz, and a few years later headed out to California to make this new-fangled Java language Go Fast.

Fast forward 20 years, and I got fame… but not really fortune.  I am all for changing that now, at least a bit, and for Save the World.  I have been courted repeatedly by the traders over the years, and certainly talked with dozens of high frequency trader groups about Java performance and low latency GC.  Basically I’ve had the Fin Tech itch for more than 20 years, and I’m finally going to scratch it – but not in trading directly, but in making trading safe.

Neurensic has some lucky timing and some serious financial market Quant talent. They put together a Machine Learning-based product to find financial market cheaters – and just in time to match the new Dodd-Frank compliance rules.  Their timing is impeccable; suddenly everybody who is responsible for traders’ good behavior (i.e. the Banks) wants their stuff.  Their first generation product is just coming out, and I plan on bring my own skill and H2O’s Big Data and ML abilities to dramatically up Neurensic’s game.  The Goal: keeping the Stock Market (well, all financial markets) safe for everybody.

Neurensic Round Two is also right up my alley: a Rosetta Stone for the 40 or 50 back-end systems found in any large Bank.  Many of these systems date from the 80’s and 90’s, and use a level of tech I’m very comfortable with.  The Goal here is to simply bring together all of this disparate data – it’s Big Data, so my Big Data experience applies here also – and give decision makers a grand overview of their entire enterprise, updated in seconds instead of days.  The term that springs to mind is a colonoscopy, a birds-eye view of a Banks’ backside.  In the longer term, we aim to change the fabric of the financial system as a whole, at the transactional layer.

I will remain in Silicon Valley, and will be shortly recruiting engineers looking to blend machine learning, financial tech, and some “catch the bad guys” mentality.


But Wait, There’s More!

I have a another offer I am actively pursuing – to keep extending H2O in the Open Source.  This will be a funded H2O-Center-of-Excellence with room for 5 to 10 salaried engineers solely focused on moving H2O forward.  Funding hasn’t landed yet, so I can’t say more – but I’m hopeful that I can start offering positions here in the next month or so.  Note that Neurensic is both fully on-board with this effort, and is not funding it nor controlling it in any way. Basically, it will be my second day job, and it happens to align well with my other day job.

Related-but-not-related, I’ve really enjoyed my engagement with the Skymind guys (hi Chris and Adam!) and DeepLearning4J – and am also angling to be in a position to get DL4J and H2O working together.

Hey!  This means I’m running at least 2 teams of of 5 to 10 engineers each!  Isn’t Cliff known for both his code hacking skilz and lack-of-people skilz?  Well, I’ve been running the engineering team at H2O for several years – and I really like it, and by all accounts people like how I run a team.  So yeah, I’ll be running some teams.  I’ll also be coding – I am a Builder after all, I have to Make Stuff to save my soul – so yeah, I’ll be coding still.  Just blending in some people-time also.

Oh yeah, AND I’m teaching a course at Stanford on Machine Learning. Stanford CME250A

If any of this interests you, come knock on my door!  For anything open source, or H2O, or heck Java Performance or compilers or flying cars, my email of 30 years still works:

cliffc@acm.org

For anything related to Neurensic or Fin Tech try me at:

cclick@neurensic.com

Looking forward to hearing from y’all!!!

Cliff

 

Words of Parting, a Fond Farewell

I resigned from H2O last week – with a fair parting and I fully support H2O’s success in future.  Srisatish and I worked together since 2004, and I learned a tremendous amount from him.  He has an amazing passion and vision.

I’d like to take this opportunity to celebrate our work together, in building something from nothing.  As Srisatish recalls, we did a million little things together; I also fondly remember the uHaul trip retrieving our earliest furniture (where we forgot to close the rear door and got a lot of other drivers’ wide-eyed panic looks before figuring it out!), and cranking out code furiously – and Taco Bell as the “company cafeteria”.  And of course the many many team-building lunches eating in the warm California sun at Fiesta Del Mar.

A lot of how we built the company was playing off each others strengths.  Srisatish had a precognition of a Machine Learning wave sweeping the industry, and with his passion he sold the vision to me, the investors, and then the industry.  For my part, I built the backbone of that vision: not just the foundation, but the powerhouse engineering team which built up from that foundation to produce the milestone Machine Learning product that is H2O.  That partnership produced amazing stuff, and I’m sorry it had to come to an end.  I only wish him the very best in the future.

It’s all of you, the H2O team, I’ll miss the most…. over the past four years, you have built an amazing technology – who’s strength is only just being tapped.  I have witnessed huge personal growth in all of you, as coders, as engineers and as good people, and am very happy to have had a chance to be a part of that growth. Your creativity, hard work and dedication have built something wondrous.  Your passion has become your outreach, you are leading the world into an era of ML and AI – a better way to do things, build things, change things, faster, more intelligent, and yet more human as well.  I applaud all that you have done, and continue to do – and I wish you the very best in your personal journey to make the world a better place.

Cliff

 

Winds of Change

I’ve left H2O.  I wish them all the best.  I’ve left a longer farewell here.

I’m a loose cannon, at least for a few months, and am looking for (the right kind of) trouble.

So I’m here, posting on my blog again, to see if I can get some suggestions on what to do with my life.  🙂

Here are a few of my crazy Cliff ideas I’m sorta chasing around:

  • Python Go Fast: Do Unto Python as Thou hast Done Unto Java. I hack the guts of Python; add a high power JIT, a full-fledged low-pause GC, true multi-threading support, i.e. make Python as fast and as parallel as Java (about the same speed as C).  This blog is really a request for an open discussion on this topic.  Is the Python community interested?  How does this get funded?  (uber Kickstarter?)  I’ll only go here with the full support of the core Python committers, and general “feel goods” from the general python community – and I’m hoping to start a discussion.  At this point I’m a premier language implementer, and making Python Go Fast is well within my abilities and past experiences. Take about 2 years & $2M for this effort to be self-sustaining (build all the core new tech and hand it off to other contributors).
  • H2O2: I left a lot of unfinished technical work at H2O – and H2O has plenty of technical growing room.  I could continue to contribute to the Open Source side of H2O, with some Big Company footing the dev bill.  Big Company gets kudos for supporting Open Source, H2O gets the next generation of cool new features.
    • Plan B, Big Company funds some new core closed-source innovations to H2O and monetizes that.  H2O still gets some Open Source improvements but not all core tech work is open.
  • Teach: I bow out of the Great Rat Race for a year and go teach Virtual Machines at Stanford or Berkley.  Fun, and makes for a nice sabbatical.  (as a bonus, I’ll probably have 3 kids in college next year, and the whole Stanford Pays Faculty Kids’ College thing sounds really good).  Might could do this while hacking Python guts at the same time.
  • Jetsons: I know how to Do the Flying Car Thing Right.  Million people, million flying cars in the air, all at once, and nobody can crash anything.  Feels like you’re flying but the Autopilot-o-Doom holds it all together.  Got figured out how to handle bad weather, ground infrastructure lossage (e.g. the Big Quake wipes out ground support in 10sec, how do you land 1million cars all at once?), integration into existing transportation fabric, your driveway as a runway, playing nice with the big jets, etc.  Bold statement, needs bold proof.  Lots more stuff here, been thinking on this one for a decade.  Needs 3 or 4 years to put together the eye-popping this-might-actually-work prototype.  Thus needs bigger funding; probably in the $10M range to get serious.
  • Something Random: by which I mean pretty much anything else that’ll pay the bills and is fun to do.  I got some pretty darned broad skillz and interests…

Gonna be a long-needed Summer-o-Crazy-Fun for me.  Heck, maybe I’ll post my 2nd tweet (shudder).  🙂

Cliff

 

How does Java Both Optimize Hot Loops and Allow Debugging

This blog came about because an old friend is trying to figure out how Java can do aggressive loop and inlining optimizations, while allowing the loading of new code and setting breakpoints… in that optimized code.

On 2/21/2015 11:04 AM, IG wrote:

Safepoint. I’m still confused because I don’t understand what is the required state at a safepoint that can’t be ensured at any random point of execution.  What is in-flight that must be permitted to complete? Given a cycle-by-cycle map you can find all the pointers at *any* cycle – is safepointing just a way to reduce the volume of pointer-locating info?

Reducing OOP-map volume happens, is useful – but is not the main thing being achieved.

It’s flipping backwards from the machine state to the language architected state, unwinding from super-aggressive optimization to reach the theoritical Java (or C/C++) language state at some points.  Toooo expensive to preserve that mapping line-by-line, call-by-call, execution-point-by-execution-point. So you keep the mapping at a few places, and optimize the bejeebers out of things in-between.  If somebody wants to:

  • set a breakpoint in optimized code (yes, works for Java all the time)
  • inject new classes, new targets for some virtual-call

then you need to stop the physical machine (actual CPU/Mill) at some place where you can find all the language architected pieces.  This mapping is far more complex than the oop/no-oop mapping for GC… but isn’t why you make it rare.  You make it rare because you have to hang onto all the language state at these safepoints, but don’t need it between safepoints.

An example might help:

  for( int i=0; i<N; i++ )
    sum += A[i]  // set a breakpoint here, for "i>1e6" or maybe "i>1e7"

So intent is to run awhile… then somebody sets a breakpoint… and wants to inspect “sum”, or change “i” in the debugger, then continue execution.  What’s the machine code look like, with no safepoints?  Something like this (bogus X86-like assembly code, pardon non-perfection):

 ... some setup, zero-trip test, range-check on "N" vs "A.length", etc
 ld  RA=&A        // Classic strength-reduced pointer into array A
 add RC,RA,#N*8   // Stopping value of A
 ld  RB=0         // sum
 loop:
 ld  RD=[RA+off]  // Load from A
 add RB=RB,RD     // Sum into RB
 add RA=RA,8      // Advance our pointer into A
 cmp RA,RC        // Stop when at the end
 blt loop
 // RB contains sum

 

But actually, with some loop unrolling:

... some setup, zero-trip test, range-check on "N" vs "A.length", etc
 ld  RA=&A        // Classic strength-reduced pointer into array A
 add RC,RA,#N*8 & (-1-3)  // mask of low 2 bits of iteration - unroll by 4
 ld  RB=0 // sum
 loop:
 prefetch [R1+off+32]  // Fetch next cache line... overlap memory & compute
 ld  R0=[RA+off+ 0]    // Get 4 values array A
 ld  R1=[RA+off+ 8]
 ld  R2=[RA+off+16]
 ld  R3=[RA+off+24]
 add R4=R0,R1  // add-tree to allow more parallel adders
 add R5=R2,R3  // add-tree to allow more parallel adders
 add RB=RB,R4  // partial sum into RB
 add RB=RB,R5  // full sum into RB
 add RA=RA,8*4 // Skip over 4 elements of array A
 cmp RA,RC     // Stop when at the end
 blt loop
 ... cleanup code for 0-3 more iterations
 // RB contains sum

 

Now I want to break on the “sum += A[i]” line… where is that, exactly, in that unrolled loop?

And which register contains “i”?  (Hint: none of them).

Do I even want to keep a shadow copy of “i” in parallel with my “A” pointer I’m rolling through the loop?  It’s pure overhead, unless I breakpoint and want to have some place to read “i”…. but even then I cannot *change* this shadow “i” because it won’t change my “A” pointer.

What about changing N?  Do I have enough info in the register mapping to describe all possible kinds of updates I might want to do to register RC when I tweak N?  This example is simple, but optimizing compilers can do hella complex code changes.

How does a Safepoint help here?  Well, for starters it points out that I do indeed need some way to find “i”, and if I change it then propagate those changes back into the loop.  Lets look at this code with a Safepoint in it.

... some setup, zero-trip test, range-check on "N" vs "A.length", etc
 ld  RA=&A        // Classic strength-reduced pointer into array A
 add RC,RA,#N*8 & (-1-3)  // mask of low 2 bits of iteration - unroll by 4
 ld  RB=0         // sum
 ld  RI=0         // Concession to Safepoint: keep "i" around
 loop:
 prefetch [R1+off+32]  // Fetch next cache line... overlap memory & compute
 ld  R0=[RA+off+ 0]    // Get 4 values array A
 ld  R1=[RA+off+ 8]
 ld  R2=[RA+off+16]
 ld  R3=[RA+off+24]
 add R4=R0,R1  // add-tree to allow more parallel adders
 add R5=R2,R3  // add-tree to allow more parallel adders
 add RB=RB,R4  // partial sum into RB
 add RB=RB,R5  // full sum into RB
 add RA=RA,8*4 // Skip over 4 elements of array A
 add RI=RI,4   // Consession to Safepoint: keep "i" around
 SAFEPOINT: RI contains "i", RB contains "sum", "some place in the stack" contains "N"
 cmp RA,RC     // Stop when at the end
 blt loop
 ... cleanup code for 0-3 more iterations
 // RB contains sum

 

So I added 1 op in my hot loop to compute “i” on the fly.  Not too expensive; 1 clock per cache-line of data.  In general we end up keeping alive what is otherwise programmatically dead “just in case” the user stops the running code and wants to inspect (dead) variables – but this is cheap.  Just spill ’em off to stack somewhere and flag ’em in the Safepoints variable map.

How do we trigger the Safepoint?  Easiest way is to have the safepoint code test some thread-local bit for a “stop” bit, and if found… stop.  If we have to stop we might have a lot of cleanup to do, but stopping is rare so the cost (of mostly not stopping) is low.  There’s lots of ways to make the safepoint check cheap.  Here’s one that depends on keeping a 2Meg (TLB-sized) window above the stack pointer mapped in or out – perfect for an X86 TLB:

  ld RX,[RSP+2M] // will trap if we map this page out, then catch the SEGV & handle

 

How do we use the Safepoint?  Once we’ve stopped the running thread mid-loop, we expect the trap handler to have saved off all the registers.  Later we can inspect the register map to find “i” or “sum”.  Note that a Java-level query cannot ask for our “RA” register – as it’s the sum of “&RA” and “i*8” and isn’t visible at the language level – so “RA” does not appear in the Safepoint’s map.

How do we change “i” or “N” and have it All Just Work?  Basically, we re-enter a clone of the loop from above…. a clone specially made to be re-entered at any iteration.  The loop setup code in the clone will rebuild “RA” and “RB” and any other intermediate state as needed, before falling into essentially the same code as the normal loop (and indeed they might be shared here!).  Or in other words – since the optimizer did something complex to build “RA” from “&A” and “i” – have the optimized do it again for any arbitrary new “i”.

The general plan is simple (from the optimizing compiler’s point-of-view): Safepoints are treated like a function-call only taken on a very very rare execution path.  Safepoints take as inputs all visible language-level state, including the state of all inlined function calls, and generally are assumed to update all memory state but not any local vars (if you are also hacking local variable state via the debugger interface, then we need to go a step further and deopt/reopt – leading to the loop clone mentioned above).

With this magical “low frequency function call” in hand, the optimizer attempts the best optimizations it can, subject to the normal constraints that the call arguments are available somehow.

Hope this helps,
Cliff