Stories about Quality. 1. Who owns it?

A while ago I was asked to post my thoughts about quality. I mulled over this request for quite a while, and realized that I did not spend enough time on the subject to produce anything really deep.

However, I was working in this industry for almost 20 years now (next year will be exactly 20 years since I have made appreciable amount of money on software, writing a piece of code for an artillery guidance system in 8080 assembly, but that's quite another story).

Not surprisingly, I have accumulated quite a few stories that could help someone maybe learn something from somebody else's experience, which is always the best way to learn. So I will be posting them here, periodically, as they come to mind.

And so it goes...

My first real experience interacting with testers was at Microsoft. The company I was working before did have a test team, but they were all black-box testers. The company produced CAD software, so the testers were part former drafters, part computer enthusiasts, but not engineers. They knew something about the product, but there was no process, no test plans, nor any accounting whatsoever. All testing was done ad-hoc by just banging on the keyboard. And they clearly could not be considered to be owners of the quality of the product.

On my first day at Microsoft, I was presented the book "Writing Solid Code" by Steve Maguire. Steve was a tester during a tumultuous period at Microsoft around late '80s, when the company was mired in bugs, and it looked like every new fix introduces two more regressions. But he also was a developer. And judging from the book, a very good one!

Unlike my previous company, Microsoft employed real devs to do testing. This was a big change.

In fact, the mantra at Microsoft was
(1) SDETs (software developers in test) should be as qualified as developers as SDEs (software design engineers)
(2) Testing is an equal participant in product development, it has the same competency and career ladder, and testers on level X are paid the same as devs on the same level.
(3) As a discipline, testers own equal share of responsibility for a product success. They do not report to developers, and are equals to developers in every respect.
(4) There should be 3 testers for every 2 developers.

This was 1998, the stock was dancing around $120, Microsoft was swimming in money and could hire the best of the best - so we could almost afford this in its full, unabridged by reality form!

The test manager in Windows CE, which was my first team at Microsoft, was Pat Copeland (he's now at Google as well, although in the Mountain View campus). He believed that the test team's job is TO PREVENT PRODUCT FROM SHIPPING. That is, the product ships when the test team can no longer prove that it is not shippable. Wow! Not only test team at Microsoft is a co-equal branch, they hold the keys to the ultimate goal of any software company - shipping the product.

This lasted for a few years, the test team was very strong, they kept the quality gates high, and I came to believe that this is how it really should be. To this day I think it was actually a good system, a system of perfect checks and balances. If there is an org that builds the product, there should be a different org to verify it - otherwise there are plenty of opportunities of the conflict of the interests.

Then Pat left for the greener pastures, and right about the same time Windows CE went through a reorganization. Windows Me just shipped, and was to be the last in Windows 9x code line, so they needed to do something with the massive number of people who were now free to do something else. I have no idea what happened with Windows PMs and devs, but we got all of their testers - lock, stock, and barrel.

Now, Windows CE at the time was strictly an arms vendor. Not to be confused with Pocket PC line which was handled by a different arm of the company, Windows CE org produced OS kernel (in the form of libraries to be linked together in various combinations) and dev tools, that could then be used, as a LEGO set, to produce more or less anything embedded. Pocket PC was one of our client orgs, so was Auto PC, Web TV, and a few other projects. The product was also available to 3rd party companies, of course.

The point is, we produced a dev kit, which had practically no UI (just a bunch of samples), and all testing was done at the API level. Which meant that all testers were SDETs.

Windows 9x was a consumer product. Almost all of its testing was UI-based, and so there was an enormous lab, and a lot of people who could bang on a keyboard, but could not program. The new test team was 3-4 times bigger than what we had before.

As it turns out, the new test manager also had a radically different approach to testing. He believed that every release cycle he is given a fixed time to test the product. He does his best to test whatever he can during this time, and then whatever is left, ships.

This was a major culture clash. Most of the original testers left, and the dev team was to deal with the change. Most of us took this as a disaster - nobody owned the quality of the product any more!

I remember having yelling matches with the new test manager in the hallways. But to no avail - the new PUM was also from Win9x, this was the life as they knew it, and that's how it was going to be. The shipping date was set at January 4th, and it shipped on the clock.

This was the worst release of Windows CE ever. Half of it didn't work, and the half that did was twice slower than the previous version. That was version 4.0, so 4.1 shipped in 6 months and was basically a gigantic service pack. But the quality was actually OK.

And then 4.2 shipped in another 6 months, and had more bug fixes and a new feature now and then, and it was very good - higher quality than any other release. Suddenly, the life was pretty good.

What happened was devs stopped thinking about the quality as strictly the test team's business. They realized that the product will work IFF they produce the code that has few bugs, and write they own tests (or samples) to ensure it. Using QA team as a safety net no longer worked, but people got it, and started working around it.

Of course, part of the test development team was still there - they were smaller, and they mostly handled the tasks that could not be handled by engineers - long-term testing, stress, certain kinds of integration testing, and automation.

So what did I learn from this experience?

1. Quality is a feature. If nobody owns it, it won't ship.


2. In a great team, an owner will emerge and fill the void if it exists.


3. The sky may be falling, but in a well-functioning team things will work themselves out. "As if by magic..."

To be continued...