As Systems Get Better, the Problems Get Harder

By Al Riske

13.Mar.08 - Chris Gerhard could have made things much easier on himself.

Awhile back, Gerhard -- now Chief Technologist and Principal Engineer in the Systems Technical Service Center in Sun Services -- wrote a program that enabled him to easily fix a problem customers were having with a database application.

"It never struck me as being rocket science, but it turns out nobody else was doing it," he says. "The mistake I made from an ease-of-life point of view is I gave it to other people. Had I kept it to myself, then all these problems would still be happening and I could solve them very very quickly."

But what would be the fun of that?

"That program became the Diskomizer and is now being used across the whole of Sun," Gerhard says. "It allows you to test any system that will store data."

As a result of his innovation -- and many more from engineers across the company -- Sun products have gotten better and better with each release of hardware, each release of software.

"Now the problems we get are more subtle," he says. "The volume is going down but the complexity is going up."

Gerhard works in what he describe as the break/fix part of Sun.

"Problems that can't be resolved are escalated and eventually end up on my desk," he says. "The exciting part of my job is I don't know where those problems are going to come from at any time, what area of technology I'm going to be involved in from one day to the next."

He estimates that the vast majority of cases -- "99.9 percent of them" -- are solved by just one person: The engineer on the frontline who answers the phone.

The remaining one-tenth of one percent end up in the hands of the chief technologist or one of his colleagues who has the experience, tenacity, and inspiration to solve it.

"I solve problems by applying techniques and having inspiration. It's the inspiration that is difficult to document and difficult to describe," Gerhard says. "But it's not by a long shot just me. It's working with your colleagues (who are all remote) and bouncing ideas off each other and pulling in the right experts, because you can't be an expert on everything."

That, he says, pretty much sums up his official job description.

"The bit that has never been written on my job description is thinking about how we can solve problems better," he says.

One of the things he has been pushing over the past few years stems from a simple observation: "If somebody sets up a test system [to replicate a customer's environment], the probability of them having set it up somewhere close to where you are is very considerably less than them having set it up a long way away."

In other words, the lab is almost always going to be in a remote location. Accept it and support it. In fact, turn it into an advantage.

"In that respect I did some work to evangelize within our organization that we should have a lab infrastructure that supports people being remote, that supports being able to do all the things you generally need to do to machines, remotely," Gerhard explains.

That includes everything from easy console access to the ability to power systems on and off remotely.

"From that, you realize we only need three labs around the world. For reasons of not wanting people to have to work at night, you have three labs, one in each geography, and everyone accesses them remotely," he says.

So how do you turn that into an advantage?

Say a customer reports a problem late in the day. You don't want to work all night, but you can't very well wait until morning to begin.

No problem. Someone in the next geography is just starting his or her day.

"If we can work on a problem during the customer's nighttime and have it solved for them by 9 o'clock in the morning they're very very happy," Gerhard says. "We do that to some extent but we don't do it nearly enough in my opinion."

People aren't meant to pick up new cases after 4 p.m., but the gray area is the call that comes in midday. Maybe it can be resolved quickly, maybe not.

"Once you gather the information, then you're supposed to work the problem. But often it takes you a few hours to gather the information and then you're at the end of your day," Gerhard explains.

So another of the problem-solving approaches he has pushed for is a standardized way of collecting and forwarding the facts of the case. "You need to do it in a standard form so the people who are writing it can write it quickly and the people who are reading can analyze it quickly," he says.

The Sun Global Resolution Troubleshooting System, or SGRT, was initially established because, as Gerhard puts it, "once you start applying rational troubleshooting to a problem a lot of problems disappear."

"When you train people who are good at problem solving to use SGRT, they all go, 'Well, that's what I do anyway.' Even if you're doing it in your head, you realize that you're asking these questions, these same questions, over and over again. Where is the problem seen? Where is it not seen? What is unique about where it's seen? What is unique about where it's not seen?" he says.

But even if you can do it in your head, Gerhard realized, there's a real, unforeseen advantage to formalizing the process.

"Going through these in a standard way should allow us -- and does allow us -- to pass cases between people in a way that enables them to understand and pick up on the documentation of the problem much faster than if you just write it in a free-text form."

In short, SGRT isn't just a good way of solving problems, it's also a very good way of documenting and communicating them.

"It's just like programming really. You need a standard calling convention between subroutines," he says. "Well, SGRT is the calling convention between service engineers."

Problem solved.