Down from Eight Miles High… and Landing!

I’ve landed at Neurensic!  First I’d like to thank all the people entertained my request for a job – there were lots and lots.  Competition was stiff, and as the month went on, the offers got wilder and wilder and closer and closer to my dreams.  Ultimately I had the choice for taking VC funding and starting something of my own (still Out There, and will probably happen yet, and could be Flying Cars), and venturing into the financial tech sector, something I’ve long thought about. Fin Tech won the day, but I’d like to give a special shout out to Google (at least 4 different groups courted me, I really like the feel of Google).

The job hunt experience gave me an unparalleled view into the Valley’s outstanding tech problems.  Several problems really stood out – as things a lot of people wanted help solving, and I thought I’d mention them here as a sort of “State of the Valley’s Better Mousetrap Hunters”.

NodeJS – 3 different companies made clear offers to help make a server-side version of JavaScript targeting NodeJS.  This is right up my alley: adding threading (but keeping the single-threaded coding style), adding a low latency high performance GC, and adding A Better JIT.  Looking at the state of JS JIT’ing I cant help but think there’s a lot of performance being left on the table – especially for “long” running server apps.  Maybe some easy tweaks and V8 cleans up here… but so far the NodeJS audience ain’t feeling the love.

Python – At least 6 stalled attempts to speed up Python are Out There (PyPy appears to be “less stalled”) – and I got clear offers from 2 companies and sorta clear offers from 2 more to make “Python Go Fast”.  I also reached out to the Python core community – and got a resounding Silence.  I’m not fond of pushing rocks uphill; if the core community isn’t interested then I’m not (yet) willing to tilt at this windmill.  Nonetheless, given the response from 4 companies (some quite large) I’m willing to bet Somebody’s interested.  Sorta needs the Kickstarter-from-Hell to pay my bills, but then could be a fun gig.

Both the NodeJS and Python speed problems appear to not be getting solved anytime soon.  I might come revisit both of these areas down the road- perhaps more as a quiet easy-going slam-dunk job.

After that, mix in 4 different ML-related offers, 8 different startups (literally at least 8, more than one network-security-meets-ML), and several We’ll-Fund-Whatever-Cliff-Does startups.  And flying cars, no lie; that was the last-ditch almost winner.  But I need to be in a better state financially before I can bite into that apple.  All told I sorted through over 20 serious job offers. It’s good to have First World Problems.


Why Neurensic?  Why Financial Tech?

This story goes waaaay back – I’ll skip past the “I was born a poor black child…” part of the story and start in when I was about 26 or 27.  I was just finishing up my PhD and was starting to look at the job market – and Everybody was saying all the money was in being a stock market trader.  It looked too much like giving up on helping the world – just living for myself, so mercenary and narcissistic.  I still wanted to Save the World, and I liked my mix of theory and practice… so I ran off to do compiler research at HP Labs, then beat IBM on IBM hardware with my mad compiler skilz, and a few years later headed out to California to make this new-fangled Java language Go Fast.

Fast forward 20 years, and I got fame… but not really fortune.  I am all for changing that now, at least a bit, and for Save the World.  I have been courted repeatedly by the traders over the years, and certainly talked with dozens of high frequency trader groups about Java performance and low latency GC.  Basically I’ve had the Fin Tech itch for more than 20 years, and I’m finally going to scratch it – but not in trading directly, but in making trading safe.

Neurensic has some lucky timing and some serious financial market Quant talent. They put together a Machine Learning-based product to find financial market cheaters – and just in time to match the new Dodd-Frank compliance rules.  Their timing is impeccable; suddenly everybody who is responsible for traders’ good behavior (i.e. the Banks) wants their stuff.  Their first generation product is just coming out, and I plan on bring my own skill and H2O’s Big Data and ML abilities to dramatically up Neurensic’s game.  The Goal: keeping the Stock Market (well, all financial markets) safe for everybody.

Neurensic Round Two is also right up my alley: a Rosetta Stone for the 40 or 50 back-end systems found in any large Bank.  Many of these systems date from the 80’s and 90’s, and use a level of tech I’m very comfortable with.  The Goal here is to simply bring together all of this disparate data – it’s Big Data, so my Big Data experience applies here also – and give decision makers a grand overview of their entire enterprise, updated in seconds instead of days.  The term that springs to mind is a colonoscopy, a birds-eye view of a Banks’ backside.  In the longer term, we aim to change the fabric of the financial system as a whole, at the transactional layer.

I will remain in Silicon Valley, and will be shortly recruiting engineers looking to blend machine learning, financial tech, and some “catch the bad guys” mentality.


But Wait, There’s More!

I have a another offer I am actively pursuing – to keep extending H2O in the Open Source.  This will be a funded H2O-Center-of-Excellence with room for 5 to 10 salaried engineers solely focused on moving H2O forward.  Funding hasn’t landed yet, so I can’t say more – but I’m hopeful that I can start offering positions here in the next month or so.  Note that Neurensic is both fully on-board with this effort, and is not funding it nor controlling it in any way. Basically, it will be my second day job, and it happens to align well with my other day job.

Related-but-not-related, I’ve really enjoyed my engagement with the Skymind guys (hi Chris and Adam!) and DeepLearning4J – and am also angling to be in a position to get DL4J and H2O working together.

Hey!  This means I’m running at least 2 teams of of 5 to 10 engineers each!  Isn’t Cliff known for both his code hacking skilz and lack-of-people skilz?  Well, I’ve been running the engineering team at H2O for several years – and I really like it, and by all accounts people like how I run a team.  So yeah, I’ll be running some teams.  I’ll also be coding – I am a Builder after all, I have to Make Stuff to save my soul – so yeah, I’ll be coding still.  Just blending in some people-time also.

Oh yeah, AND I’m teaching a course at Stanford on Machine Learning. Stanford CME250A

If any of this interests you, come knock on my door!  For anything open source, or H2O, or heck Java Performance or compilers or flying cars, my email of 30 years still works:

cliffc@acm.org

For anything related to Neurensic or Fin Tech try me at:

cclick@neurensic.com

Looking forward to hearing from y’all!!!

Cliff

 

Words of Parting, a Fond Farewell

I resigned from H2O last week – with a fair parting and I fully support H2O’s success in future.  Srisatish and I worked together since 2004, and I learned a tremendous amount from him.  He has an amazing passion and vision.

I’d like to take this opportunity to celebrate our work together, in building something from nothing.  As Srisatish recalls, we did a million little things together; I also fondly remember the uHaul trip retrieving our earliest furniture (where we forgot to close the rear door and got a lot of other drivers’ wide-eyed panic looks before figuring it out!), and cranking out code furiously – and Taco Bell as the “company cafeteria”.  And of course the many many team-building lunches eating in the warm California sun at Fiesta Del Mar.

A lot of how we built the company was playing off each others strengths.  Srisatish had a precognition of a Machine Learning wave sweeping the industry, and with his passion he sold the vision to me, the investors, and then the industry.  For my part, I built the backbone of that vision: not just the foundation, but the powerhouse engineering team which built up from that foundation to produce the milestone Machine Learning product that is H2O.  That partnership produced amazing stuff, and I’m sorry it had to come to an end.  I only wish him the very best in the future.

It’s all of you, the H2O team, I’ll miss the most…. over the past four years, you have built an amazing technology – who’s strength is only just being tapped.  I have witnessed huge personal growth in all of you, as coders, as engineers and as good people, and am very happy to have had a chance to be a part of that growth. Your creativity, hard work and dedication have built something wondrous.  Your passion has become your outreach, you are leading the world into an era of ML and AI – a better way to do things, build things, change things, faster, more intelligent, and yet more human as well.  I applaud all that you have done, and continue to do – and I wish you the very best in your personal journey to make the world a better place.

Cliff

 

Captain’s Log Days 8, 9 & 10

Captain’s Log Day 8

We’re on the road by 10am, this time a full day’s drive to Montgomery AL from Katy TX.  I forget how big Houston freeways are; at one point I count 9 lanes *in each direction* (18 total lanes!).  I’ve never seen so much concrete.  It’s otherwise mostly uneventful, though.  Traffic is fair to light and the road is good.  We stop at a random lakeside park by Lake Charles for lunch.  It smells of the ocean and has an alligator pond/cage/viewing area.

While I typically encourage the kids to drink a lot (to survive the desert heat & dry), I don’t check on how much they eat, just that they eat a reasonably balanced diet.  So I missed out that Matt hadn’t eaten all day, and was constantly staring heads-down on his IPod on silly flash games.  Well, towards afternoon he starts feeling sick, and near dinner he barfs and refuses to eat or drink anything.  He cannot even keep down tiny bits of bread or Gatoraide; the 2nd barf happens on our bedsheets and pillow.  At this point he decides to camp out by the RV toilet and do any more barfing into that (uggh!!!  poor guy!!!!), and we decide to cut it short and look for camping for the night.  By dinner he’s still unable to keep anything down; we grab to-go food from a collection of fast-food joints and keep rolling to the nearest campsite.

We get 1/2 way between Mobile & Montgomery, AL and pull over into a nice full-service RV park.  Shelley & I decide to camp outside in a tent, so Josh can get off the floor (he’s 17 and 6ft tall, lean and flexible… and does not fit in any of the RV pullout/fold-down beds, so he’s been sleeping in the aisle).  We want Josh off the floor so Matt can make an emergency run from his foldout bed to the bathroom without interference.  It’s beastly hot and humid outside, but I figure it will cool off as the night wears on.  Boy was I wrong! It remains 80+ & 80% humidity all night long outside, while the kids were sleeping in air-conditioned luxury.  And we get a late night visit from the camp kitten – he’s adorably cute and caterwauls at us, and starts climbing the tent with his razor claws until Shelley takes him for a walk.  He follows her like a shadow all over the park until she finally has to lock him in the campground bathroom.

 

Captain’s Log Day 9

Finally dawn breaks and we move back into the cool RV air.  Ahhh, blessed relief.  Also, Matt is much better – it’s a common kid 24-hour tummy bug.  I start him back in on the BRAT diet, with sips of water – and now he’s very hungry, a good sign.  He continues to improve throughout the day and is eating normal by dinner.  We pull up camp (we’re getting quite expert at this) and head for Stone Mountain, GA.

Stone Mountain is a giant mountain-sized chunk of granite outside of Atlanta, with a park and a lake.  It’s been carved with a 50ft high sculpture and has been slowly improved over the years to include many hiking trails, a sky tram system, lots of outdoor adventure activities and an amusement park.  Apparently the “ducks” (amphibious vehicals) are fantastic.  We are going there for the July 4th extravaganza – and as a sign that I’m on vacation, I barely know that today is the 3rd and I’ve no idea what day of the week it is.  We get there about 3pm and check in to a nice RV camp site.

Shelley cooks a fantastic spaghetti dinner.  My brother Eric drives out to camp with us, bring his best friends’ two small girls (ages 6 & 7) with him (he’s been watching the girls when the parents are working since they were 2 & 3) and we all enjoy a nice picnic dinner.  As the evening rolls on we’re deciding on whether or not to see the laser & fireworks show this evening (there’s a bigger one tomorrow) – when the thunderstorm hits.  It’s a real downpour, big lightning and thunder, blowing wind, the works.  We wait that out, and then try to take a walk about the park.  Eric & I, the two girls and my middle two kids walk over to the clubhouse (to check out the water-taxi ride to the main park area) but the rain has other ideas.  We make it to the clubhouse but we’re fairly wet, so we treat the girls to hot chocolate while we dry out.  We wait for the rains to end but it’s no good – the rain has turned into a steady drizzle; we just as wet by the time we make it back and there’s no end in sight.  We give up any idea of tent camping or seeing the laser show and settle for watching a Disney movie (the Sword in the Stone) and having a lazy evening with all 10 of us huddled in the RV).  Sleeping arrangements are “cozy” to say the least!  But at least everybody is dry.

 

Captain’s Log Day 10

It’s the 4th of July!  We breakfast, cleanup & head over to the water taxi. The rains have stopped and the sun is out.  It’s gonna be a hot & humid day. The water taxi is nice, it’s cooler near the lake.  We make it to Stone Mountain’s main attraction area and decide to walk to the bell tower.  The park is already busier than Eric has ever seen it before.  There’s a large Indian family setup under the bells already (and I see more people of the same persuasion walking over to the tower all morning – I think they figured out a cool shady semi-private place to hang out at all day).

We’ve walked maybe a half a mile and it’s not even noon and we’re already soaked with sweat when we make it back to the Plantation Inn.  The Inn isn’t open for lunch (although the AC is nice), but the helpful counter lady tells us there’s RV parking closer in.  We walk up to Memorial Hall.  Immediately two things strike me as really odd: there’s at least 1000 people hanging around looking for food (and more pouring in all the time), it’s 11:30 and *none* of the dozen or so restaurants are open yet – and there’s bus & RV parking open
right in front of the main Hall.

I hand the kids my credit card (to get lunch at noon when the restaurants open) and Shelley and I hightail it back to the RV: across 1/2mile of hot trails & roads, ride the water taxi (we miss the one in front of us by literally seconds even with me sprinting across the landing area), and finally the 1/4 mile hike from the taxi dock to the RV.  We pull the hookups as fast as we can and roll out & down the road.  Nessie does NOT sprint, she *proceeds*, but we made her proceed as fast as possible.  We took the short way around the lake, only to discover the road was closed: the attendant at the barricades explains “the road fell in a hole”.  Nothing to be had for it; Shelley makes a 3-pt turn on a narrow park road and we go the long way around.  Finally, a full hour later, we make it back to the bus/RV parking in front of Memorial Hall – and Lo! it’s open.  We take the most premier parking spot in all of Stone Mountain, at noon-thirty on the 4th of July.  (A short time later one other RV takes the next spot, then the road is closed behind us).

The amazing thing about the Stone Mountain concessions was the astronomical price for food; hotdogs: $7-$10, drinks also $7 or so.  (And they denied a hot and hungry hoard for at least an hour???).  But finally we all sat down and finished our food and plotted our next move.  Shelley, Eric & I all want a big hike.  Last Christmas Shelley & I hiked the Grand Canyon down to Phantom Ranch and back out in two days, and Eric has hiked both the Pacific Crest Trail and the Appalachian Trail end-to-end.  We head out for the top of Stone Mountain on a hot & muggy day.  There’s lots of other folks with the same idea, but it really is a long hot hike.  Most of my kids bail out after a mile or so, voting to go hang out in the AC (which is really a good plan); Eric and his two young charges make it to the path-up cutoff but it’s a killer hike in the heat so they turn around also.

It ends up as Shelley, Laura (age 15) and I heading on, and we decide to head for the bird sanctuary.  It’s another couple of miles and we gave most of the water to Eric & the girls.  The three of us head down the far side of the mountain to a kids playground and finally drag ourselves into the park and help ourselves to the water fountain.  We drink a quart each, and fill a couple more quart bottles we’re carrying.  We hike the 1/2mile more to the bird sanctuary – mostly carrying on now because of what Shelley would call “Mission” – her ex-Marine training to “complete the Mission” no matter the cost.  i.e., we’re all too collectively silly to claim the end goal is ridiculous, so we hike it anyways. It’s a decent enough little woodsy trail, with plenty of songbirds – but far to beastly hot to really enjoy.  By the time we make it back to the kids’ park we’ve drunken all our water (another 1/2gal between the 3 of us), so we reload (and re-drink our fill) and back up the mountain to cross it in reverse.  We make it back in good time, although it was really pushing our the limits to
hike so far on such a hot day.

There is much lounging around and napping in the RV’s AC to wait out the heat of the day.  Matthew (age 12) introduces the two little girls to the joys of Minecraft.  Eric & Laura nap.  Everybody else surfs the (very very slow) park Internet, eating popcorn & chips.  Finally as the heat starts to fade and twilight sets in we get enough gumption to make & eat hotdogs.  Then we pack it up and prepare to leave the relative safety and peace of the RV for the slowly building hoard.

The lawn below Memorial Hall faces the giant sculpture carved into the face of Stone Mountain.  The only open spaces are at the very front, so that’s where we head.  I estimate 100,000 people eventually filled that lawn; in any case it was a colossal crowd.  It was also actually quite a peaceful crowd; no rowdies (no alcohol allowed), zillions of little kids running pell-mell, picnic blankets, soap bubble makers and glowing flashing LED lights.  It’s cooler now, so we settle down on our blankets and chairs, listen to the music and wait for the show.  At various times I let Josh or Karen & Luke wander off for snacks (a little nerve-wracking that; they are out of sight in the crowd within seconds and gone for 30mins or more, but everybody returns fine).

The fireworks show starts promptly at 9:30 and is possibly the best I’ve ever seen.  There’s a laser & light show on the mountain, there’s a Civil War tribute, (there’s ads for all of Georgia’s major sports teams), there’s music and of course fireworks.  The actual fireworks where downright amazing; you get a double-echo from the Bang! works, one directly and one bounced off the mountain.  They used plenty of the big fireworks and absolutely tons of rising sparks kind; the entire mountain was a sheet of fire for minutes at a time. The finale left us breathless.

Unwinding back to the camp was a slow but uneventful crawl; I’ve sure we beat the campers on foot (who had to wait for the river-taxis and the report was to expect a 2.5 hr wait).  Eric took his to charges home and we collapsed tired but triumphant for a full nights sleep.

Cliff

 

Captain’s Log Daze 3 & 4

Captain’s Log, Day 3

Next day we take a lazy breakfast and then decide to visit the Biosphere with Grandpa.  We head out to Nessie and observe the new tire is looking mighty flat.  Humm….  we hook up the air pump… and it’s at 80 psi, spot-on the normal max pressure.  Looking further: the inside tire is flat.  CRAP.  But wait!  It really WAS fine in Casa Grande, I checked it before we drove off.  And pretty quickly it’s clear that the rubber value stem is leaking, probably
banged too hard during the change and now it’s going flat overnight.  We call up Ed.  He agrees to change it under warranty… but he doesn’t want to drive out to meet us.  But he Does The Right Thing, and calls GCR Tire, a Tucson local who WILL come out to Grandpa’s.  Ed’s covering the whole cost.  So now we’re basically stuck at Grandpa’s waiting on GCR Tire (who’s promised to get there “in an hour” and it’s already 10am).

Meanwhile something triggered in Shelley’s brain about tires aging, so I go read up on them.  I learned something new today: all tires age.  After 6 years you should replace them, completely independent of tire wear.  Pretty much no tire is expected to last 10 years except under “ideal” circumstances.  And tire manufacturers have to stamp the date of manufacture on the tire, so you can tell how old your tires are.  (read up on it, but it’s the week & year of manufacture as a 4-digit number in an oval after the “DOT” stamp).  So we go look at our tires (mostly meaning Shelley crawling under the RV in the 110-heat to read between the dualies).  Sure enough, the youngest tire is 8 years, and the oldest is 12 years old.  Good tread, but expected to blow at any moment.

Crap, crap, crap.  Another round of planning & family voting.  We decide to limp over to Big-O tires to replace the remaining 5 tires, never mind fixing the old one.  GCR Tire shows up for the repair while I’m finishing negotiations with Big-O (and yes I asked GCR and no they did not have the tires we need in stock).  So the GCR guy politely fills our inside tire (it’ll last maybe an hour) and we roll over to Big-O.  We drop everybody off at Costco, where we do some shopping and eat a delicious Costco lunch (which is actually pretty dang cheap and a decent enough hot dog), and wait 2 hours for me to blow another $900 on tires.  After a while we’re back at Grandpa’s house with 6 brand-spanking new tires, waiting for a thunderstorm to pass before we go swimming.  It’s too late for the Biosphere, that will have to wait for another visit.

The thunderstorm takes too long to pass and we miss swimming also.  We have some more family over for a nice dinner, then we hit the road again for more night driving.  This time we’re heading for Carlsbad Caverns.  It’s a long haul out of Tucson but utterly uneventful.  We even give Luke (19 yrs old!) a turn at the wheel.  He’s a natural driver and handles this big rig fine.  We make a long drive of it but Carlsbad is just too far to make in one day.  We end up in the backside parking lot of a Walmart somewhere just inside the Texas border (Walmart mostly has a “RV friendly” policy).  It turns out that while our GPS has many useful features, finding RV campsites is not one of them.  Also when we turn off I-10 and head into the countryside we lose all cell phone service and can’t call ahead.

Captain’s Log, Day 4

It’s a 3hr early-morning drive or so to the Caverns.  We get there just before the heat starts getting oppressive again.  This time we decide to leave the generator on and the AC running while we spend the hot part of the day underground.  I used to see this all the time and wonder about it: RV’s with the generator going constantly.  Now I get it – Nessie will be in tolerable shape when we return to her, but without the AC Nessie would heat up like a tin
box in the hot sun.

Several of my kids are really nervous about entering the Caverns; they’ve had some scary cave experiences in the past.  We have to gently encourage several down the switchbacks into Carlsbad, but they master their fears and soldier on down into the cool cave air.  Carlsbad does not fail to deliver.  The Caverns are immense on a scale that’s hard to imagine; all of downtown San Jose could comfortably fit in them.  The trails wander on for miles in there (the sections closed to the public are probably 100x larger than the miles of public sections).  There’s a section where the roof soars over 300ft overhead and single rooms covering many acres with lines-of-sight of perhaps a quarter-mile underground.  And it’s all a fairyland of cave growths and little pools, with eerie lighting everywhere; flowing stone sculptures with names like “Temple of the Sun” or “Doll Theater”.  For the younger generation: it’s the largest Minecraft cave you’ll ever see.  🙂

We ride the 800ft (!) elevator lift back to the surface and decide to stay for the evening bat swarm (it’s still to hot to drive).  Every evening at dusk between 250 thousand and a few million bats leave to go eat mega-tons of insects up and down the local rivers (the numbers fluctuate so much because the bats migrate frequently).  We hang out in the local gift shop & cafe for a few hours (always a bad plan when on a budget), then try to watch a movie in Nessie (AC keeps it tolerable in there, but it’s still pretty warm), and finally evening rolls around.  We settle in to listen to the rangers and then finally the main show: 250 thousand bats fly out of the cave like smoke on the wind.  There’s a faint odor of bats in the air, and an endless murmuring of chirping bats and the little winged creatures are flitting everywhere overhead before flying off the escarpment edge and off into the darkness.

We do another (not so long) night of driving, stopping at midnight in Fort Stockton, TX.  We get a longer nights’ sleep tonight, even if the location isn’t as glorious.

Cliff

 

Captain’s Log Days 1 & 2

Captain’s Log, Day 1

Today’s the day for the start of our epic 3-week 7000-mile cross-country RV trip of doom!  I’m up (fairly) early as I need to pick up all my kids – and their extra clothes, toiletries, games, meds, etc – by 9am.  Then I take them back to my house to begin packing in earnest, except for Josh who I need to take to the eye doctor’s to replace his glasses (broke under warranty) and Laura – who left her drawing pad behind.  I also need to drop my ex-Sprint AirWave back at the UPS store, and go by the pharmacy for a month’s worth of meds, and get fresh fruit for the RV, and… and … and … you get the picture.

Meanwhile Shelley is busy doing last-minute packing of Nessie, our 7-ton 31′ Class C RV – all the fruits & veggies & cold-stuff go in in the last minute.  While I’m running around frantically driving kids all over creation, Matt figures out he’s got a total of 3 pairs of underwear at my place, so Shelley is out driving him to get some undies (and other stuff we need) while I’m running my errands.  Despite all the crazy start and hasty lunches we actually hit the road as planned right at noon.

So on this trip we have: Me & Shelley (a red-head), my eldest daughter Karen, Luke (another red-head), my son Josh, my 2nd daughter Laura (also a red-head but no relation to Shelley) and my youngest Matt.  We’re off to see the country and all my scattered relations.  I’ve got my Dad (& Jane) living in Tucson AZ the kids’ other grandpa Zade in Luling TX (outside of San Antonio TX), my sister (Aunt Ruth) and mom (Pat Ireland) in Katy TX (outside of Houston), my brother in Atlanta GA, and my Uncle Bill and his 4 daughters (all my age) and their 15 kids (all my kids’ ages) in eastern Connecticut.

We’re starting out of San Jose, heading over Pacheco Pass to I-5, then south towards LA – but we badly do NOT want to hit LA right at rush hour, so we eventually cut over to Bakersfield and then follow some long long slow farm road across the central valley to Barstow… and up to Calico, a ghost town.

Now when Shelley was a kid, her mom would drive this very road (to visit her grandparents in Vegas) and they would stop by Calico once a year or so.  She has some fond memories from her childhood so visiting Calico is somewhat of a pilgrimage to her.  We arrive there right at dusk and can’t find anybody manning the entrance booth, so we sheepishly drive (our 31′ RV) quietly into the town – and promptly find the RV campground.  It’s basically deserted (there’s 1 other camper there, and space for maybe 100 vehicles), has power hookups and bathrooms with showers and running water… and it’s free, at least for people
arriving as late as we did.  We got out, stretched our legs and enjoyed the beautiful pink sunset over the red red hills, made sloppy joes on Nessie’s stove and ate on the picnic tables in picture-perfect weather.  Laura got the neighborhood dogs to howl back at her, Karen & Luke made videos of the epically blowing Laura’s hair, Matt climbed the hills and Josh & I ninja-sparred.

It was a picture-perfect ending to the 1st day.

Captain’s Log, Day 2

We walk though Calico the next morning.  It’s cool desert morning air, with some wonderful history.  The town’s been cleaned up a fair amount since Shelley was last there but remains a really nice tourist trap.  Mission accomplished, we head out for the long hot desert drive to Tucson to visit my Dad (Grandpa).  It’s a *long* boring drive down I-40.  Shelley is an awesome long-haul truck-and-horse-trailer driver, so driving this RV thing is a piece of cake.  (and while I’m up getting Shelley a nectarine, Laura types in my blog: “Moo” and “He has yet to notice.”)  Karen is talking about whale sperm shampoo (*not* sperm whale shampoo)… and the generator cuts out – it’s overheated.  That means the main compartment AC cuts out.  Oh – did I mention that on the long uphill grades the cab also AC cuts out?  (I assume because the engine is working too hard?).  So we pressed on in the 110-degree heat, across I-40, down “highway” 95 (looks like asphalt thinly spread over desert dirt, there’s a whole lotta “dips in road”).  Back on I-8 and heading west, and 2hrs out of Tucson and we’re all baking on-and-off (as the cab AC cuts in and out, and the cabin is slowly climbing above 90degrees)… when we blow a tire.

Yup, 20 miles from Nowhere, AZ, down that long & lonely road… we suddenly pick up a shake & shimmy… and a list.  We hove Nessie over to port and off the side of the road.  I tremulously step out to survey the damage.  Outside right rear tire has blown big, completely come apart.  It’s 1 of a dually, and the other is squashed under the load but holding.  Time for some quick thinking; we are baking and a long way from anywhere… and lame.  We check the phones: we have cell service; Thank You T-Mobile.  We call AAA.  They don’t do RV tires but they do give us the number of RV Medic in Casa Grande… which isn’t open after hours.  We get the answering machine & another number to call… also no answer.  So now we’re calling all about (at least 3 phones making active calls at this time, plus Google map’s are in hot action).  We decide to limp into Casa Grande.  We dump the tanks (not the black!) and push the kids over to the “good” side to lighten the load.  We also batten down the hatches, as Shelley points out that if the remaining tire blows we’ll “drop hard”.  Casa Grande is about 20miles down the road, and we decide that 40 mph is probably a good max-speed so we start off.

Then the dust-storm hits.  NO I AM NOT KIDDING.  We’re lamely limping along when the wall of dust hits, obliterating the “Blinding Dust Storms” road sign.  So now we’re limping blindly along getting buffeted by 40mph winds and dust (and tumbleweeds ARE blowing by, queue lonely wild-west music please) when the rain hits.  Yes: thick dust on our windshield AND ITS RAINING NOW WITH THE BLOWING DUST.  Nessie soldiers on.  20min later we pull off I-8 and out of the storm and head down some lonely farm road… but with the lights of Casa Grande clearly in the distance.  We pull into the first big lot we see (Big Tires empty lot) step out and see a rainbow.  Back around to calling RV Medic we get a human, we tells me to call Ed W’s who DOES do after-hours work.  $200 minimum charge.  Ed (who also requires 3 or 4 phone calls to reach) promises he can work on us, but can we get to town?  No problem.

While we wait at Ed’s shop for an hour (his mobile guy is on another call), Grandpa & Grandma drive up from Tucson and take the 3 younger kids back to their place and feed them all manner of treats.  We (4 remaining) older kids mosey over to a nearby restaurant and get dinner and some heat relief.  Another hour later and I’m $400 poorer and sporting a brand-new tire.  We pile in and make it to Grandpa’s.  Much sighs of relief, and a good nights sleep was had by all.

Cliff

 

Progress + Vacation

It’s been a freak’n month since I last blogged!  Where’s the time gone???

Mostly I’ve been furiously coding.  ‘wc *java’ of our ‘src’ directory now reports 31500 lines.  We’ve cleaned up and CSS’s the web interface.  We added LevelDB to handle zillions of small K/V pairs (larger ones go to the local file system directly, and of course we still handle S3 and HDFS natively (either using an existing hadoop install, or directly *being* a distributed hadoop)).  We’re still 100% peer-to-peer, even for the direct HDFS stuff.  Last week I hacked a concurrent Patricia Trie (leaving the making of a *distributed* concurrent Trie for later, but now I know how to do it…). Then we ran all 36Gig of Wikipedia data through WordCount, using that Trie – it took less than an hour on 1 node.

This week it’s about running a Linear Regression *distributed*, using distributed Fork/Join as the programming paradigm.  Also, integrating a HashMap-in-a-Value (so we can pass about & maintain the Map interface in the Value piece of our K/V store – think: distributed JS objects), plus the final bits of VectorClocks (all behind the scenes; the VCs will let us do atomic update and strong coherence of Keys but they’re a horrible API to expose).  We’re building a toolkit approach to solving the problem of building a reasonable database over the Cloud.  Either (distributed) Patricia Tries or (distributed) Concurrent Skip Lists for range queries, plus JS-like objects in Values, plus atomic (transactional) update of individual JS objects using a Compare-And-Swap like approach (instead of locking: CAS is much faster under load, as threads can optimistically make progress).

More on all of the above later this week – as we have a hard deadline to finally *open* our Open Source project.  Yeah, yeah, yeah, I’ve been hassled plenty about calling ourselves Open Source and not (yet) having any open source… we’ve been trying to get the basics done first… but the real news:  I’m finally going on Vacation!!!

Yes, Nessie, the 31′ 7-ton Class C RV of Doom is being prepared for our 7000 mile Epic Cross-Country Journey.  I’ve been wanting to do this for a decade now: take the entire clan (7 of us!) across country, touring all the junk tourist traps we can and visiting our scattered family as we go.  We got family in Tucson AZ, San Antonio TX (well, Luling really), Houston, Atlanta, DC area, and Connecticut.  I’m giving an invited lecture at UIUC on our way back, and have been assured I can use that lecture as a reason to declare this a “business trip”, and deduct all the gas and mileage costs – I figure about $3500 in gas alone.  We stopping at Stone Mountain in GA over the 4th of July, visiting my brother and camping at the lakeside facing the mountain where we’ll watch the fireworks and lazer show from the RV roof.  We’re going to visit Carlsbad Caverns.  We’ll pass through DC and maybe attempt the Smithsonian (not sure about that one; depends on the schedule and how badly I want to fight the RV through DC traffic).  We’re visiting my Uncle’s classic family farm in Connecticut where my 4 cousins live – all my age, all married with 3 to 4 kids each… all about the same age as my 4 kids.  We’re talking now about 15 to 20 neices and nephews, plus Aunts & Uncles galore, and of course pigs and chickens and horses.  It’ll be a regular zoo.

So if you see a large white whale heading east on I10 with a frazzled Shelley or my excited 19yr-old at the helm, honk, wave Hi and give us a wide berth…

Cliff

 

Quote(s) of the Month from Kevin Normoyle (Sun/Sparc & Azul L2 Cache Designer Extraordinaire, Cache Coherence Advisor to 0xdata):

Reminds me of CS101, on one of my first programs.  The grader wrote in big red letters over my big comment block:
“Don’t document your bugs, Fix them”

So I asked Kevin if I could quote him, and I got this response back:

ah that’s fine…I spout “Advice” left and right to everyone… Many dismiss it as “Rant”.  There’s always that fine line between being a Prophet, and just another crazy guy standing on the corner yelling.  One could argue that everyone who every posts to Twitter is an “Advisor” of some sort, to the world.

Sound advice, from a (reluctant) adviser to the world.

http://www.cs.tau.ac.il/~shanir/nir-pubs-web/Papers/OPODIS2006-BA.pdf

What’s Going On?

As alluded to in my last blog, here’s my fun hack de-jeur: “Whats going on?”

I’ve got a multi-node setup with UDP packets slinging back and forth, and each node itself is a multi-cpu machine.  UDP packets are sliding by one another, or getting dropped on the floor, or otherwise confused.  I’m in a twisty maze of UDP packets all alike (yes, I played the game back in the day).  Then something crashes, and pretty quickly the network is filled with damage-control packets, repair & retry packets, more infinite millions of mirror reflection packets.  What just happened?  I press my handy little button and…

… a broadcast of “dump, ship and die” hits the wires (a few extra times for good measure).  All my busy Nodes stop their endless chatter and dump the last several seconds of packets towards my laptop, slowly & reliably, via TCP.  Each node has been gathering all the packets sent or received to (well, the first 16 bytes of each) in a giant ring-buffer, along with time-stamp info and the other party involved.  After I ship all this data from every node to the one poor victim (that I pressed my button on), every other node dies (to prevent further damage).

The Last Survivor gathers up a bunch of very large UDP packet dumps and starts sorting them.  Of course, you can’t just sort on time, that would be too easy.  No, all the nodes are running with independent clocks; NTP only gets them so close in time to each other.  Instead I have to sort out a giant Happens-Before relationship amongst my packets.  I am helped (above and beyond some sort of home-brew wire-shark) by my application understanding it’s own packet structure.  I *know* certain packets must be strongly ordered in time, never mind what the clock says.  For example, I only send out an ACK for task#1273 strictly after I receive (and execute) task#1273.  Paxos voting protocols follow certain rules, etc, etc.

In the end, I build a very large mostly-correctly-ordered timeline of what was just going on, as seen by each Node itself, and then HTML’ify it and pop it up on the browser.  Voila!  There for all the world to see is the blow-by-blow confusion of what went wrong (and generally, the follow-on error “recovery” isn’t all that healthy, so more broken behavior follows hard on the heels of broken behavior).

Basically, I’m admitting I’m a tool-builder at heart.  As soon as I realized that standard debuggers don’t work in this kind of situation, and wireshark couldn’t sort based on domain-specific info (and pretty-print the results, again using domain-specific smarts), I went into tool-building mode.  As of this blog, I’ve found several errors in my cloud setup already; e.g. a useless abort-and-restart of a Paxos vote if a heartbeat arrives mid-vote from an ex-cloud-member (that’s alive and well and wants to get back in the Cloud), and some infinite-chatter issues getting key replication settled out as nodes come and go.

On other fronts, my car came back from the body shop, only to turn around and go back to the engine shop: the timing belt had slipped.  The work was done under warranty and I’ll go pick up my car on Monday.  I can hardly wait!!!

My GFs car’s brakes have been squealing for weeks; they finally started shuddering and we decided it was time to fix them.  She’s driving a 1993 Nissan Maxima with 220K miles on it; weird things start breaking at that age, but mostly the car just soldier’s on.  But it was time for the brakes.  We pulled the rear pads & looked at the rotors: one of them was shot.  Fortunately a new rear rotor was only $25, plus another $22 for pads (tax, brake grease, still under $50).  We couldn’t get the dang pistons to move back! We tried at least 5 different wrench/jig/clamp combos to no avail.

We figured the pistons must have been jammed with debris, so with great trepidation we pulled the brake fluid line, the emergency brake cable and pulled the whole unit to my workbench.  I popped the piston out manually.  It looked clean and good… and had this funny thing in the middle… stupid me, failed to check the internet again… it’s the anti-slip mechanism for the emergency brake.  You have to *spin* the piston to screw it back into the cylinder.  Sigh.  It took us another 1/2hr to find the right tool to spin the dang thing, but it finally went in without too much trouble.  After that it was another hour to reassembly all the parts, and then we had to bleed and bleed and bleed the line.  As of this writing, the pedal is still to soft, I suspect we need to bleed it some more.

Daughter is at the Old Salts Regatta, plus a ton of driving to meet people for 0xdata, plus a much needed dinner out… and down 2 cars (GF’s brakes-in-progress and my car in the shop), made for a very complicated week.

Cliff

 

Hit and Run

I was driving in Palo Alto with 2 of my kids in the car heading towards Fry’s, looking to get the largest monitor I could reasonably buy.  I was driving down one of the many fairly narrow 4-lane side streets, heading towards El Camino Real when a driver pulled into my blind spot.  With a block-and-a-half to go to the red light, the driver very slowly started to overtake me… then he started edging into my lane.  I honked and edged up and over to give me some space… with now much less than a block to go the driver suddenly floored his vehicle and came over into my lane (his lane was ending quickly, being filled with large yellow Caddie parked at the light).  I hit the brakes but couldn’t get any more over (no shoulder, sidewalk full of telephone poles and other crud).  Crunch… I was fairly certain he had hit the Caddie, and I knew he had hit me.

I checked my kids, all ok.  I looked out the window at the other car & the Caddie.  He was between the Caddie and me, and was trying to back down the street – but got blocked by cars coming up behind him.  He stopped moving and started getting out of his vehicle, so I took a breather and got a clearer look – there were lots of cars, lots of honking; my car looked drivable and in the way and there was a Jiffy Lube at the corner, so I waved to him and rolled around the corner and parked.

I got out and headed over to his car… and he was busy yelling and screaming at the Caddie driver (who had been parked at a red light the whole time!).  He looked at me and got back into his car and rolled up to light and around the corner, clearly to join me at the Jiffy Lube… and took off instead.  Gone.  I was dumbfounded, and stood looking at his vanishing car like an idiot.  One of the Jiffy Lube guys came up to me and asked if I was ok; I nodded and went over to check out the Caddie guy.  He was a timid old man and was really shook up by the screaming-at he just had.  It was clear his vehicle was blocking the road and looked untouched.  I stuck with him until he felt better, then he got in his car and drove off.  I walked back to mine to survey the damage and double-checked the kids.  Sure enough, a big dent with lots of missing paint.

Then the Jiffy Lube guy volunteered that he had heard the crash, had come around the corner, had watched the driver chew out the Caddie… and decided the driver looked “off”.  He had the plate number written down and encouraged me to call the cops.  So I did; and sure enough in about 5 minutes a Palo Alto policeman showed up.  He was very polite and called in the plate#, and started taking statements.  In another 10 minutes he reported that they had caught the guy!  Then he offered me a chance to eye-witness the driver where they had stopped him… so I followed the cop back across Palo Alto and did a slow drive-by… and indeed it was the same guy.  They offered me the chance to press charges – but they also reported he didn’t seem drunk/drugged – so I decided to do this with just insurance instead.  Maybe he was just having a bad day.

Today AllState came out and looked at my car, and hauled it off to the body shop.  Maybe in a week I have a car with a fresh paint job?  Kudos to the Palo Alto police department; that was quick work catching the guy.  Kudos to the Jiffy Lube guy who had the brains to get the plate number.  And I did eventually get my monitor.

You didn’t think I was going to talk about technology, did you?

Cliff

 

What I Did For My Christmas Vacation

Warning – contains no technical content …

My GF and I both love to drive (I own an Evo which I love, I drive really hard, and it’s been a rock-solid car, now with over 120K miles).  The kids were with their mom for a week-and-a-half, so GF & I took off for Bryce Canyon and the Grand Canyon.  She did most of the driving on the way out – both because she loves to drive my toy! – and because I was stuck doing PLDI reviews (sign up now!).  I did half my reviews from the road, uploading reviews at passing Starbucks.  Along the way we passed through Casa de Fruita, a 100-yr-old over sized farm stand, classic California at its best – complete with toy riding trains, carousel, restaurant, hotel, and of course plenty of fruits & nuts (not referring to myself at all here).  I basically run on chocolate and coffee, so this was a fuel-stop for my brain.  GF stopped to feed the pet peacocks.   Since this vacation is being done “on the cheap” we stayed at a smaller hotel outside Bryce.  It was fine (if a bit plain) and the price was right.  No WiFi in theory – but it was available in the lobby, and we finagled a room just behind the lobby area which DID have WiFi… so I finished off my PLDI review uploading from our hotel room after all.

Typical Bryce Canyon weather prevailed for several days: snow flurries with lows down to 9!  After a day recovering from driving (we had put a LOT of miles under our belts!), we checked out Bryce Canyon.  Fantastical views (and GF found a raven to feed)… but obscured by snow flurries and low slung clouds.  Next day dawned with crystal clear-blue skies and winter wonderland style snow everywhere; red red rock, “hoodoos” and dark evergreens.  The colors still sparkle in my minds eye… pictures here.  We took a 4 mile hike along the rim, then down in and around the “hoodoos” and back out the other side.  Being the brilliant person I am, I forgot the camera battery charger (and none for sale in any of the local shops)… so the digital camera died at the bottom of the hike (we bought a throw-away camera for the next hike, but the photos aren’t back yet).

Next we headed for the Grand Canyon.  We’ve been planning on hiking it for a couple of months now, and we lucked into a cancellation in the Phantom Ranch barracks (typically you need to reserve space a year in advance!).  Along the way there we made good time… except for the one dreaded moment when that SUV on your bumper suddenly spouts red & blue lights, and the sheriff wants to talk about safe driving speeds in the mountains of Utah (warning only: maybe 2nd time in my entire life I’ve been pulled over and only received a warning).  We got in very late (got stuck behind a SUV doing between 15 and 25mph on clear roads labeled at 45mph, but it was night, on national park roads w/double yellow strip; very frustrating that the person would not pull over!) and so collapsed on the hotel bed without seeing the canyon that night.  Next day we walked over and ogled the canyon.  It’s still beyond my words to describe, even if I’ve seen it (hiked it!) before.  Its one of those things you just got to experience in person.  GF’s never hiked it before, so we took our time…

Day One.  After checking out the trail heads and talking with the rangers we decided to buy crampons.  BEST DECISION EVER.  Hiking in Bryce on the snow taught us that there’s ice under all that pretty snow… dang slick ice.  We hiked down a full mile of ice from the South Kaibab trail, then the trail dried up and opened up into the amazing views.  Only place I know of you can stand with a 1000+ foot dropoff to one side, and yet be looking at a 1000+ cliff towering above you on the other hand…  and both be yet another 1000+ further down from the rim and yet ANOTHER 1000+ feet above the river!  It’s a real-life 3-d eye candy mind bender.  We hiked it down in 5 hrs.  Not bad for this 50yr old pile-o-bones.  Sore and triumphant we walked into the Phantom Ranch main room.  Our (segregated) bunk beds awaited… but they had had another cancellation.  For a mere $30 more, we got a spacious 10-sleeper cabin to ourselves!  We rushed over to the cabin, and sat down on a bed to “rest a bit”.  Two hours later, we awoken from a deep nap and realized that dinner was fast approaching…  Dinner was delicious (as only a hearty steak dinner can be after a day of hiking).  After dinner we took a star-light stroll in the pleasantly cool evening, laid down besides the rushing creek and star-gazed awhile.

Day Two.  Day One, of course, is the day you act like an idiot and climb down this great big hole.  It’s on Day Two that you have to fix the situation by climbing out.  The first hour of waking we spent discovering all those muscles (and joints!) that got overworked yesterday… and being ravenously hungry despite the giant meal of the day before.  So after a big breakfast we started back up, the Bright Angel trail this time.  Other than the amazingly scary river crossing on the swinging suspension bridge (the Colorado is wide and fast there, and the bridge very clearly was swinging more than 5ft side to side), the upward hike was … upward.  Hard work undoing all those steps yesterday, although the uphill is MUCH easier on the joints.  The crampons came out again for the three miles of snow and ice near the rim… and we made it up in 7.5 hours.

Next day we drove back home, 750 miles in one day (again).  Next day we cleaned up, picked up, packed up… and the kids arrived here for Christmas!  We had a wonderful family Christmas; GF made a fantastic dinner and there was much regaling of tales and playing of video games for all…

Cliff

 

The Greatest Trip Report

Divorce.

So ends the Greatest Trip of my life.  22 years ago I fell in love and married the woman of my dreams.  I was happy and content in love, full of energy and hope for the future.  I had a needed talent; I sallied forth to Save The World or at least make it a better place – while building a safe, secure and rich life for us.  We had a child, then another, and still more until we had 4 – each child just as precious as the last.  They are great kids.  I found my talents for programming, for complex computer language implementation skills,  for singing and for public speaking.  Life was rich, busy and fulfilling – to me.

To the other person in my life, however, things were different.  20 years later I discover she felt that her life was stifling and constraining – her dreams too long had gone unfulfilled, nay, unnoticed.  Her pain, held inside too long, turned to anger and resentment.  Too late we tried marriage counseling, honeymoon style vacations, long heart to heart talks- but too much water had flowed under that bridge.  About two years ago we gave up the hope of reconciliation and started the process of dividing two lives that had lived together for 20 years.

That process is finally ending.  We have a final legal resolution now and all that remains is the raising of our 4 kids independently.  The situation for the last two years has taken most of my time and all of my emotional energy.  It has been a place of personal growth and introspection, of deep thinking, of tears and sadness.  It has also been (eventually, after a long time of sadness) a journey of joy and discovery; of doing things I have long denied myself (just returned from my 2nd year at Burning Man!!!); and of new celebrations of life.

I’m enjoying my half-time with the kids and becoming more of a Dad and more a part of their lives.  I’m getting my energies back and am feeling more ready to slay more dragons than I have in years.

Look Out World, Here I Come!
Cliff