Thursday, January 22, 2009

You are totally not going to believe this

For some reason, Grimm thought I'd be the perfect person to write this article. I have no idea why, but I'm always willing to do my part, especially considering the lack of DPS Warlock love in the recent patch.

The unthinkable has happened. I totally agree with Tobold. A recent article published at WoWInsider is completely out of line. I could have posted my rebuttal there, and really wanted to, but one voice in thousands, as opposed to one voice with few readers, the latter is still the better deal. Hi.  /wave

I enjoy reading WoWInsider, especially Guild Watch - O teh drama! - but there are times that they rub me the wrong way. Mike Schramm has done so, in spades, this time around.
In short, patch 3.0.8 has been a disaster. If Blizzard feels that this patch lives up to the quality of content they released in Northrend (or if they, unbelievably, somehow though this was meant to be a bugfix patch for Northend, that ended up screwing up more things than it fixed), then they need to take a long, hard look at their quality assurance system again.

"Mike, you ignorant slut."
Why is it, when people want to act like they know what goes on in software development, start trash talking QA? Whenever software crashes, "it's always QA's fault", as far as they're concerned, and they ponder - in a superior tone of voice - at how anyone could have possibly missed such an obvious bug.

It's like watching a monkey, throwing poo at the wall, until one sticks, at which point the monkey gets excited.

Did it ever occur to you, Mike, that it wasn't missed? Did it ever occur to you that maybe someone knew about it and decided that schedule trumped quality, once again?

I'm sure you are thinking by now, "well, Miss Smarty Pants, how do you explain this fine mess?"

Don't mind if I do.


Purveyor of FAIL for over 25 years!
First, some bona fides: I have worked in the industry for over 25 years; 10 spent in the area of hardware reliability, a field that allows for very little in the way of error (you can't patch a hardware failure). The next 10 have been spent in software quality, first at a mid-level position (planning etc) and later in a senior position.  Prior to those I spent 8 years as a technical supervisor in a high-pressure environment that tolerated very little down time, and did so with a great degree of success (Medals were involved.). In short, I know my way around the swamp.


This is what it sounds like when code cries
Having gotten that out of the way, here is my theory, based - in large part - on actual working knowledge of the process.

First, you need to understand something: QA is not the gatekeeper. QA rarely, if ever, has the means to prevent software from shipping.  This is acceptable to everyone because QA prefers to be testing software, not participating in launch planning meetings with upper management.

QA's primary job in the software industry is to measure the quality of the software, and to inform the project manager as to that quality. They may also be required to "sign off" on the test results.

But if they don't, and upper management wants to ship ... it'll ship. Somehow.


I've seen the big picture, and you're not in it
Management is infamous for this sort of thing, both in my personal experience as well as by reputation.

We have seen Blizzard's commitment to software quality in the past. Heck, the game itself was delayed so that they could get it right. Entire instances and features have been held back until they can be done right.  The software has been of consistently high quality and usability for years. And QA, once it finds a process that works, will hold on to it with both hands AND a full set of teeth. QA loathes change, especially when it comes to a working process (Of course, a change to the process might have been made "to improve our time to market". Yep, you smell it too, I bet. A manager).

On the other hand, this patch has all the hallmarks of Management on it.

Sudden, apparently unplanned deployment? Check.

Incomplete? Apparently.

Undocumented? Sorta.

Not staged as patches usually are (You will usually see the patch in the downloader a good week ahead of release day. Not in this case)? True, dat!

All hallmarks of a manager deciding that getting "something" out was more important than getting something GOOD out.


Professor Plum with the Monkey Wrench in the Library
So why might that happen? Well, look at how long 3.0.8 was in testing, and look at the continuing issues with lag that were laid at Wintergrasp's feet. I can see some PR or marketing flak hitting up the project manager with a dozen emails a day over how imperative it was to get a fix for the "Wintergreen issue". I can see the project manager pushing back, saying they hadn't met testing requirements or something. I can see a more senior exec being drug into the dispute to "facilitate" this little misunderstanding ("Facilitate" being a euphamism for "explaining to the project manager who signs his paycheck.") . Not that I've seen it before or anything.

My conclusion, therefore, is that this patch was willingly rushed out before it was ready, and that management knew full well that it was not ready. However, management was getting beat up about something, pressed until the correct "key words" were spoken by someone, and then proceeded to push the software out the door.

And here we are.


In cyberspace, no one cares if you scream
There are many reasons why this could have happened. You can come up with a fantastic theory about how a system that is designed to prevent oversight of obvious stuff (QA), broke down and failed anyway (missed several obvious bugs). Or you can explain this away as plain human nature (bad managerial call).

In the end, it pays to remember one of the many Murphy's laws, because the simplest explanation is almost always the right one.

To err is human.

To really screw things up, requires a manager.