Xbox 360 defects: an inside history of Microsoft’s video game console woes

After the write-off

There has been a lot of executive turnover in Microsoft’s game division, but none of it has been laid directly at the feet of the defect problem. Moore left Microsoft a year ago to become head of the sports label at Electronic Arts, while Don Mattrick, the former head of EA’s worldwide game studios, took the top games job under Bach at Microsoft. Bach said that Moore’s departure had nothing to do with the RROD problem, and Moore said the reason for his departure was that he and his family wanted to return to the San Francisco Bay Area. Bach remains one of the top executives at Microsoft, running the Entertainment & Devices group, (including the IPTV, Games for Windows, Xbox, and Zune businesses) which reported its first-ever profit in the fiscal year that ended June 30, 2008. Allard is chief technology officer of the E&D group, while Todd Holmdahl is head of games hardware.

Oddly enough, demand for Microsoft’s console continued, partly because gamers loved playing games so much that they kept on buying replacement consoles. By the fall of 2007, Microsoft had its Falcon-based machines ready. It was able to cut the price of the console, but not so much that it hurt its bottom line. The launch of “Halo 3” in the fall helped spur console sales, helping Microsoft stay ahead of Sony and at least stay competitive with the Wii in terms of game sales. In a development that was reminiscent of the “Dead Rising” failures, a number of Halo 3 buyers said the game appeared to fry their consoles.

Microsoft never announced how many machines had been returns. Based on an estimated $60 repair price, the company’s reserve can cover lots of machines. In early 2008, SquareTrade, which sells warranties for electronics, reported that there was a 16.4 percent failure rate for Xbox 360 systems. Michael Pachter, an analyst at Wedbush Morgan, estimated that the number is 3 million, or about 15 percent of the total. It isn’t clear if the numbers should count consoles that fail and aren’t repaired for free (only those with the three red lights are replaced for free); also retailers such as GameStop have begun repairing many of the units themselves. Bach said that Microsoft’s own research shows that fans continued to love the console and buy games, in spite of the problems with the defective units. Pachter believes that is true.

“In the ordinary course of something like this, you would expect it to show up in the customer reaction data,” Bach said. “We just haven’t seen that. It speaks to the fact that they love their games and Xbox Live. Does it frustrate them? Yes. On the other hand, they know we’re taking care of them. People have a certain amount of respect for that. If it had happened on a product that had less baseline customer satisfaction, it would have had a bigger impact. We really haven’t seen that.”

At this point, the Wii has begun to far outsell its rivals on a worldwide basis. It’s not inconceivable that it will sell as many units as its two rivals combined. It isn’t clear how much the RROD problem hurt Microsoft, which remains ahead of Sony. The scorecard so far: the Wii, 29 million; Xbox 360, 20 million; PS 3 (pictured left), 14.4 million.

Was it worth it to go first, despite the concerns of the engineers and the inevitable consumer return problems?

“If you take the question of whether it was the right thing to try to be first, the answer to that is definitely yes,” Bach said in July 2008. “It has given us a leg up in a number of places that are super important. It has given us a leg up with game developers. It has given us a leg up from an economics perspective. It helped us expand Xbox Live quickly. At a strategy level, if you asked if we wanted to be first again, I would say yes. It’s easy for me to go back and say, if I knew what would transpire over the next two years, would I go back and do something different, I think that’s an obvious answer. But the fact is, based on the data we had at the time and all the hard work we put into it, there was no way to see what actually happened.”

The problem for Microsoft was that it sacrificed something to rush into the market and establish the Xbox 360 as the premiere platform of the living room. What it sacrificed was the good will of consumers, who are actually critical in terms of establishing a lasting platform. As for profits, Microsoft has finally turned a fiscal-year profit in the Entertainment & Devices group. It did so because it wrote off the costs of the defective units and warranty program in the previous fiscal year. (The $1.15 billion write-off covered several years of repair costs, some of which would have otherwise been incurred during the profitable fiscal year.) But Bach said that the business model for the Xbox 360 isn’t permanently ruined.

“It doesn’t make it good,” he said. “Would I like the billion dollars back? Of course, the answer is yes. But there is still plenty of economics in the business for us to be successful.”

Microsoft hasn’t been as aggressive at cutting prices this generation, but Sony has a more expensive box and so it hasn’t been putting price pressure on Microsoft. Had Sony been more aggressive, the consequences could have been much worse in terms of customer defections due to the RROD. Even with the $50 to $79 price cuts announced by Microsoft on Wednesday, it is taking a lot longer for prices to fall in this generation.

Jerry Knoben, a former Intel manufacturing chief, joined Microsoft in January this year as head of Xbox 360 manufacturing. He recently sent out a memo to the staff saying that the yield on Xbox 360 production had improved from the mid-80-percent-range to 92 percent. But he said that there were still some surprises in manufacturing that Microsoft had to fix.

The improved consoles use a new motherboard, dubbed Jasper. This one has both a 65nm IBM microprocessor as well as a 65nm graphics chip from AMD. The improved graphics chip may finally run at acceptable thermal limits. But it’s very late in coming, compared to the rest of the chip industry, in part because Taiwan Semiconductor Manufacturing Co., the chip contract manufacturer making the graphics chips, was slow to shift to 65nm factories. Late or not, Jasper is what enabled Microsoft to cut its prices this week.

By next year at the earliest, Microsoft is scheduled to launch a new iteration of the Xbox 360 with a board dubbed Valhalla. That board will have a single chip that combines both the graphics chip and microprocessor on a single device. This is the kind of technical tour de force that Sony achieved with the chips for the PlayStation 2, a move that allowed the company to significantly reduce manufacturing costs. Microsoft still has an advantage over the PS 3 in manufacturing costs, since it doesn’t have an expensive Blu-ray player.

Shane Kim, (pictured left) until recently head of Microsoft Game Studios and now head of the game division’s business development efforts, recently said that it was too early to call the console war in favor of Nintendo. We won’t know the winner, he says, until somebody sells 100 million units. Microsoft’s plan is to sell 75 million Xbox 360s. The only way they could ever do that is to be more aggressive on their pricing, which means driving down the costs of the console as low as possible. And by doing the right thing for their customers from now on.

Sony and Nintendo aren’t safe just yet. Bach has always talked about how Microsoft has a 10-year to 15-year plan for success. Maybe Xbox 3.0, just like Windows 3.1, will do the trick.

To conclude, the video game industry has never seen a consumer problem as bad as the “red rings of death” and the size of the $1.15 billion charge stands as one of the biggest liability glitches in consumer electronics history. How Microsoft handled the flaw may provide a lesson for all modern electronics companies; that is, if you are going to promote the hell out of something, it better work the way you say it does and you better have a strong customer support and engineering debugging team to back it up.

Amazon and eBay have their periodic outages. Sony had its problems with batteries, Intel had flawed Pentium chips that were bad at math, Nvidia took a $200 million charge to deal with graphics failures on laptops, and Apple has had its share of glitches on the iPhone. As electronics become and more complicated, it becomes a practical impossibility to test for every single flaw. But with the force of the Internet behind them, angry customers can exact a revenge that can be extremely damaging to a company’s brand image.

One thing is clear. Microsoft has to move beyond its mentality of being a software company that can launch fast and fix later. With global markets and global launches, the consequences of such a cavalier approach to hardware quality can start to pile up. The company clearly has the cash and the size to compete among the best hardware companies out there. It cannot afford to tarnish its brand when competitors on various fronts — such as Google on the web side and Apple, Sony, and Nintendo on the hardware side — are there to exploit its stumbles.

At this writing, it seems clear Microsoft’s hegemony in operating systems and productivity software won’t last forever. The company has limited time to get the rest of its house in order. It has to diversify into fields such as video games and make them pay off with profits. Microsoft has demonstrated an incredible tenacity in a lot of markets. It took almost a decade for the company to march to victory with its Office productivity software. I have no doubt the company can execute. It just has to decide that quality is something that matters.

If Bach is correct and Xbox 360 gamers are loyal, then Microsoft has been lucky. In the future, I suspect consumers won’t be as forgiving as fans such as Chris Szarek, who doesn’t own any other console and has bought three dozen Xbox 360 games.

Szarek said, “They got me. There was a time when I was furious with them.”

So why did he stay with the Xbox 360?

“What can I say?,” he said. “I like the games.”

If you liked this story, check out our interview with Spore creator Will Wright.