Last week was my fourth week here at Genuitec, and they are still saying things like "I'm sure you've done [* one or another practice *] at other places you've been."

Well, I haven't. Not like this anyway.

You see, most of us work on corporate software. Departmental stuff or experimental stuff or this would be cool stuff or things for one or groups of users.  Or we work on small feature sets for long periods of time. Software at this level can be very relaxed and even quite pedantic. Most shops have a process that you either follow or pretend to follow, but nobody much cares about perfection because there are rarely any high stakes at the UI level, only back end.

My New World: A Million Users

So what is it like to have a million people download your latest release ?  In a word,  Fear.

There's just something very tangible and real about getting your butt kicked. It happened here at Genuitec once, and these guys are insanely dedicated to making sure it never happens again. MyEclipse had a release some years ago that still has a profound effect today.

No one here would mis-understand if you reported that you tried us at that one release and never came back. It took us two weeks to fix all of the bugs that happened then, and we probably lost a lot of customers forever in those two short weeks. Up to then we had been a normal software company, one that ran like any other shop I am familiar with.

QA was a few clicks just to make sure all the screens looked like they were going to come up. Throw it over the wall, call it done until someone proves otherwise.

That was then, this is now.

So what's it like doing a release at Genuitec now ? We test and test and test. It's refreshing, at first, then it really grinds on you. We test so long and so hard the bug reports get really stupid sounding at times. But that's what makes it look so good. That's what makes us all crazy as developers, too. It's like that feeling you get when your eyes are popping out of your head from staring at the screen too long.

It's day and night for two weeks, and I can't say I enjoyed it. We didn't even get it all perfect.

But I can say something I have never been able to say before. I feel really confident in this release, at a level that I have never taken the time to do before. This is the way to do software, or at least that's what it feels like after doing it both ways.

Blue GA Release

The result of months of work by a top notch team, the MyEclipse Blue GA release is something the entire team can be proud of.  These guys are the best, and I am greatly honored to be a part of that process.

We released it at 11:30 PM CST on Sunday March 16th, 2008. I know that time well, because I was so tired I kept one screen open to the clock and one to the work at hand, until we hit the go button.

A New Twist: Developer as Customer's Best Ally

I found another very intriguing pattern to this release that I don't recall having seen before. This time the feature-itis that one normally sees wasn't inspired by the customer team, it was straight from the team leads themselves.

Time and again the only conflicts were between me, as Product Manager, and the developers working in the trenches trying to just add that one more feature for this release. These guys care intensely about their customers, and don't want even the slightest obstacle in usage of our product. Bug report after bug report was about features that were absolutely necessary and must be added to this release. I went along for a while, then I finally had to shut if off or we wouldn't ever get the release out the door.

The Future: Where Can It Be Improved ?

If it's great that we do all this testing, the part that seems least great is doing it all by hand. At the behest of one the founders, we're looking at some automated UI testing for some of our more brain-draining repetitive UI testing.

I'm also going to look at some non-UI testing, such as I've been doing at other shops. If you're just using the UI to feed in series of repetitive and closely similar work in the background, sometimes that is easier to do one time at the junit level, then just run that in the PDE tester like you would in a conventional test first shop. Apparently there are guys that have integrated this into something like Cruise Control, or at least that is what one of the attendees at EclipseCon was telling me over beers Wed night.

All in all, its been a great learning experience, this new process of releasing software like you really care about it intensely. I like it.

Comments ?

Is my experience different than yours ?

Do you have any particular experience with UI testing tools or any other relevant testing tools and practices ?

Any other comments on the whole issue of testing vs TDD or any other approaches ?