Skip to content

Git Together ’10

Oh, this looks like fun.  After the GSOC mentor summit wraps, there’ll be another Git Together in Mountain View and there are already a few neat ideas for talks.  Maybe by then I can prototype some git-based object storage and deployment in Corgi, and then I can have something to spout off about instead of just gawking.

I have this vague vision of Heroku-style deployment in the GoGoEgo world: checkout the objects backing a site, make local changes, push changes.  Use a branch to develop something crazy, merge it in (with attendant content) to make it public.  All the plumbing is there — Gerrit proves that — it’s just a matter of trying to imagine a sensible UI that can be reduced to something with Heroku levels of Zen.

The likelihood that it’ll be anywhere near public by October is nil.  But it could give good demo before then.

Notice #3028: node i-bbd142d3 down

OpenNMS complains a lot.  It’s a touchy little bugger, and it’s supposed to be.  HTTP not responding here, SSH not responding there … and usually some ISP in the middle is just slow to pass packets.  The noise is useful.  All good stuff to know.

I absolutely hate this particular message, though.  When all services on all interfaces for a given host go away, that’s just never a good thing.  Here, as it happens, the message signalled the ugly and untimely death of my last instance-store based Amazon instance.  Ah, i-bbd142d3, I hardly knew ye.

Fortunately, everyone did as they were supposed to, and kept their databases and files on EBS, so it wasn’t a very grave undertaking to bring up a new (EBS-backed!) instance, move the volumes, and repoint the elastic IP.  There was a bit of downtime, but at least no dinosaur-killing disaster.  And I’m really happy to be rid of that last instance-store.  I always live in terror that someone’s going to put something mission critical on an instance-store by mistake, and then be really upset — with me — when it’s just gone with the wind one day.

Still, on balance, I really prefer days that don’t feature a “node down” message.

How Gerrit Code Review Changed My Life

Gerrit, the code review tool from the Android Open Source Project, came to the forefront of my attention only recently. When Grzegorz Kossakowski joined the Scala+GWT project, he proposed it as the keystone of the project’s workflow.  Grzegorz had a unique perspective; he interned last year with Gerrit’s creator, Shawn Pearce.

I started experimenting with Gerrit, and I confess it was love at first install.  I think I jumped in at a good time; it sounds like some early adopters had challenges.  But for me, at this mature point in Gerrit’s lifecycle:  Run war file.  Get working system.  Register with OpenID.  Start administering and using.

Of course, if you want to do a complicated JEE setup, you can.  But Gerrit trivially runs as its own standalone application with an embedded Jetty and H2 database.  Wow … does this sound familiar?  Minus the Restlet and OSGi guts we love so much, Gerrit is based on most of the same stack as our own internal applications.  Jetty, H2, GWT … shared by VergePay and GoGoEgo.  Soul brothers, we are.

There’s even a hidden gem inside Gerrit: gwtorm, a lightweight persistence library that Shawn and/or his confederates quietly created to support cross-database, ORM-style operations without all the cognitive incompatibilities that arise when trying to use something like Hibernate with GWT.  This little solution reminds me of why we have been reliant on com.solertium.db for so many projects.  There is, I’m convinced, room for a simple POJO based object mapper somewhere in the world.  More on those musings later, though.

Anyway.  Love.  Not just because the architecture made my heart swell with fraternal pride.  Also because I’d really tried over and over again to institute a code review process based on Rietveld or Review Board, and just couldn’t demand it with a straight face.  I admire the GWT guys for having the discipline to religiously develop via an Subversion and Rietveld process, but Carl and I tried it on GoGoEgo and just couldn’t muster the strength.

Subversion itself was probably 90% of our problem.  I feel that my svn-fu is too weak to do branches and merges in the Subversion world.  The primary issue, for me, is that the cost of failure is so incredibly high.  Mistype a command, or guess wrong in a Team Provider dialog box, and you’ve made a semi-permanent ruin of your repository.  I like to think I’m surrounded by a bunch of smart people, and they’re even worse at the svn-fu than I am.  So, the practical outcome is: we don’t branch, we don’t merge.  We just fight over trunk.  There is occasionally blood.

We were fumbling, in Rietveld, towards a model much like Gerrit’s.  But instead of having a Subversion topic branch, a patchset would land in Rietveld and live its own iterative life there, with an unreasonable amount of work by the contributing developers to maintain and post versions of the patch, just because of slight differences between how our Eclipse based process viewed the repository, and how Rietveld viewed it.

Gerrit’s tight coupling with git really reduces all this pain.  First of all, I like git a lot better than Subversion.  I may be alone in that at the office!  … but I’m not alone in the world.  I just like distributed version control in general, and git’s command line interface makes perfect sense to me.  It’s trivially easy to create a topic branch, reasonably easy to rebase changes into a clear patch, and again trivial to push that change into Gerrit.

And then, a miracle happens.  Nothing.

Gerrit is a quality control speed bump.  Between prolific developers, and the truth that is the repository, a speed bump is Good.  Even on my own private project, where nobody’s reviewing my code, by going through Gerrit, I’m guaranteed a chance to visualize my changeset in a nice Web interface and make sure it contains what I think it contains.  I get a chance to read it and see: Did something nefarious happen with newlines?  Did an unintended file get committed by accident?  Did a cat or dog walk across some keys and “changed” a file in a way I didn’t know about?  All three of these things have happened — to me personally — in the past two weeks.  And I spotted the problem in Gerrit before the attendant poop landed in my repository.

In a team environment, code review itself is great.  It can exist on lots of levels.  In my daily work travels, when I say “code review,” the usual assumption is that I mean some level of rigorous proposal and defense, like a financial audit.  For me, really, I almost never mean anything that stringent.  I think a second pair of eyeballs is always good, even if it’s just a quick sanity check.  When I say that all code should be reviewed, I don’t mean it should necessarily be dissected.  Just read by somebody other than the person who wrote it.  Like blogs, am I right?

With good support for useful email notifications that users can control themselves, Gerrit makes it easy for this heads-up, second-pair-of-eyeballs code review to proceed. Since switching to Gerrit on a couple of large projects, and subscribing to scads of notifications:

  • I’ve gotten a much better feel for the pace of development
  • I’ve spotted commits that had impact (either positive or negative) beyond their intended scope
  • I’ve been a lot more useful in ad hoc discussions when questions come up

I’ve also interceded a couple times to -1 patches that would have done something bad, for reasons that were occasionally subtle.  As a result, nothing bad has landed in the Gerrit managed repositories yet.  Suddenly a genuinely trunk-stable development strategy is much more achievable.

I feel revolutionized.  I really do love Gerrit.  The 2.1.3 release removes my last few quibbles with the UI (thanks to the new Review+Submit feature, and branch listings for non-administrators)  A few more big projects remain to convert to this workflow, and I can’t wait to get there.

Android For Retail?

When we began developing products for the point of sale space at Solertium and RMR, we made a couple of key, no-turning-back technology decisions:

  • The client would be browser based
  • The server would be cross-platform and distributed

We started development at about the same time as Google announced its Chrome OS initiative, which looked like it might eventually be a great operating environment for client stations. We also started out demonstrating our browser based capabilities on the iPhone, which was the dominant smartphone in the market, firmly occupying the category it redefined.

“Way back” then — early 2009 — Google’s Android platform existed largely in the lab.  The T-Mobile G1 was the only generally available hardware device, and it had met with only a lukewarm reception.  How things have changed since then!  Android has developed considerable momentum, overtaking iPhone in US market share in Q1 2010 — though iPhone 4 will surely try it’s darnedest to reclaim the title in Q2.

Let’s ignore mindshare for a moment.  Capability-wise, Android has also caught up, and then some.  A year ago, it lacked multi-touch support, and lagged considerably behind iOS in performance.  But the Android 2.2 (FroYo) release has pulled ahead in some respects.  A demo at Google I/O showed an Android phone demolishing an iPad in browser JavaScript performance.  Of course browser JavaScript is just one piece of the performance puzzle, but it’s still a great demo.

Android is definitely not just for phones any more; in May 2010, Google announced its Google TV initiative, which is essentially a reimagining of the Tivo concept on the Android operating system. It’s notable because big partners like Sony, Dish Network, Logitech, and Best Buy have committed to it.

But, even if it’s useful for more than phones, the official builds of Android still only run on the low-power ARM chips found in phones and other small embedded devices.

Enter the very recent news — which seems like it follows naturally from the Google TV announcement — that Intel will support an official port of Android 2.2 to Intel chips.

Until this release, to be made available this summer, the only way to run Android on a non-ARM device is to use an unofficial port done by the awesome volunteers at  As the expression goes, your mileage may vary.

The Intel port will focus on the Atom chips present in many netbook and slate form factor devices, and, we can expect, they will provide good support for Intel video controllers, sound controllers, and the multi-touch screens found on coming Intel-powered netbooks and slates.  Intel has committed to contribute its vendor overlay to the AOSP directly, which provides some insurance of the port’s longevity if Intel’s interest wanes.

Meanwhile, Chrome OS still doesn’t exist.  Based on the Chromium OS dailies, its focus remains very clearly on the browser.  It presents a net-top OS in which the browser is the entire view to the world. This might be the future, but in a world of “App” explosion, it also might not be.  Certainly I feel unwilling to make this leap of faith myself.


We recently came into possession of some used PioneerPOS equipment that was used to run VergePayPOS beta with a customized Ubuntu Linux, sporting Firefox and a locked-down window manager. This environment can run both the client and server pieces of the software.

I installed the 1.6r2 build of android-x86 on the PioneerPOS, which booted and ran fine. Specific drivers for our hardware weren’t included in the build, so the display was a basic 800×600 VESA and the touchscreen only marginally worked — axes reversed, etc.

Nevertheless, I was able to easily open the VergePay client in the display and use it normally, with some scrolling, given that it was designed for 1024×768 screens. The speed of the client application is dramatically better than on the Firefox/Ubuntu version. All of the efficiency packed into Android for small and slow devices really shows up at the macroscopic level when running a full-screen, Javascript intensive application.

I can’t overstate this in terms of impact and feel.  The GWT application running in Android’s browser on an x86 machine responded as if it were truly a native application.  Really slick.

Given this performance, it’s easy to conclude that a properly supported Android 2.2 on Intel would be a fantastic client for VergePayPOS’s browser interface.

Server Side Possibilities

The VergePayPOS server is developed using a typical embedded-Java approach. It targets the GoGoEgo application server, which is run under standalone Java. To run it on a standard Mac or Linux system, we install and configure Java as necessary, then run the GoGoEgo executable from the command line. Updates, etc. need to be generally done by hand, or mediated by the software itself. During beta we have been pushing updates out by sending OSGi modules over rsync (ssh).

Android offers us the opportunity to run the server side as an app, which can be downloaded, installed, updated, etc. by the operating system. The code need not change much; it’s still Java (Android is a Java based OS). Android has some restrictions that must be observed, though, and there are differences; Android Java is a clean-room implementation (Apache Harmony and the Dalvik VM) unrelated to Sun Java.

But I find something seductive about the idea of re-imagining the VergePayPOS server side as a collection of Android apps. This would provide an ability to remix the platform at a consumer level without requiring much if any involvement by an integrator. Features like c-store support, processor integration, etc. could all be delivered as discrete apps. The App Store / Android Market purchase model is well understood by consumers at this point.

I see no reason yet to abandon the generic multi-platform Java server install for projects like VergePayPOS — certainly it is the traditional way of doing things — but it seems that a broad range of opportunities is about to open up in the Android space. Dozens of compatible devices are about to hit the market in the next year, and at least some of them will have capabilities that make them ideal for retail environments.

I’m looking forward to taking the experiment to the next level with an Atom-powered FroYo device later this summer. Any point of sale hardware vendors willing to give it a shot?