Xbox 360 defects: an inside history of Microsoft’s video game console woes

Microsoft’s mea culpa

In many ways, the Xbox 360 was a big achievement. Microsoft had beaten its rivals Sony and Nintendo to market by a year. Sony had outsold Microsoft five-to-one in the previous generation of consoles, but Microsoft has outsold Sony’s PlayStation 3 with the Xbox 360. Microsoft’s to game executive Bach (left) had repeatedly promised that Microsoft would be profitable in his Entertainment & Devices group in the fiscal year ending June 30, 2008. The company met that target. Big losses have turned into profits.

But when the company launched its Xbox 360 video game console in November, 2005, it didn’t have a handle on product quality and it was not prepared to systematically analyze its product returns and debug bad consoles.

Microsoft has admitted those mistakes. On July 5, 2007, the company said it would take a billion-dollar write-off to pay for free replacement of Xbox 360s by up to three years from the date of purchase. Peter Moore, who at the time was head of the games business at Microsoft, said in an interview the decision to take the write-off was due to an “unacceptable number” of returns. The company made improvements to the console’s quality, but Moore said it was unclear how many units would fail in the field. Microsoft then followed up its new policy with numerous public apologies, including a statement from Bach, its game executive.

Microsoft never disclosed its actual return rates. But according to data obtained by VentureBeat, the total number climbed above 1.2 million consoles in early 2007. That is a huge amount, considering Microsoft had only shipped 11.6 million into stores by the time of the announcement in mid-2007.

The company was dragged kicking and screaming to its admission of widespread defects — with gamers and the press doing the dragging. In interviews in April and May of 2007, Microsoft executives denied that quality problems were haunting them. Todd Holmdahl, (pictured left) general manager in charge of the Xbox platform for Microsoft in Redmond, Wash., said in an interview with me on May 9, 2007, while I was at the San Jose Mercury News, that the “vast majority” of Microsoft’s Xbox 360 customers were having a great experience with their consoles. He declined, however, to say what percentage of consoles were being returned for defect reasons. In doing so, he was as coy as any of the console makers are about such sensitive data. But everyone wanted an answer to that question.

It’s not clear why it took Microsoft so long to announce its warranty extension last year, almost 19 months from the time when the first complaints arose at the outset of the launch. As it was investigating the cause of the problems, the company stayed quiet, acknowledging problems only little by little. The company delayed its free-replacement announcement until it understood the scope of the problem.

In July, game executive Bach said that the number of returns was not measurably big until a year after the launch. It was a problem that became evident only over time, he said.

“It wasn’t related to things we were seeing in testing or some judgment call we had to make about whether the product was ready,” Bach said at the dinner. “We were confident the product was ready. We did a lot of testing. The problem that shows up with the three red lights on the console is a complex interaction with some very complex parts.”

Ignoring warnings

But the evidence for the quality debacle was there to see even before Microsoft shipped any machines. In August, 2005, as Microsoft was gearing up production, an engineer raised a hand and said, “Stop. You have to shut down the line.” This wasn’t just a brief moment. The engineer spoke up repeatedly.

That engineer, who asked not to be identified, had deep experience in manufacturing. When production results were really off kilter, stopping a line and tracing a problem back to its roots was the answer. But the higher-ranking engineers, managers and executives chose to risk going forward. There wasn’t a universal backlash from the engineering ranks, according to one engineering source.

Nobody listened to that engineer — who spoke on condition of anonymity — apparently because console launches are always hurried affairs. Yields — the percentage of working products in a given batch of total products produced — generally start low. As the manufacturers conduct statistical analysis and tight controls on every step in assembly, they learn how to drive the yields up.

Still, the picture wasn’t pretty. The defect rate for the machines was an abysmal 68 percent at that point, according to several sources. That meant for every 100 machines that Microsoft’s contract manufacturers, Flextronics and Wistron, made at their factories in China, 68 didn’t work. At the recent dinner, Bach denied that there was a big concern about defects at that point in time.

At that point, it is likely true that the engineers weren’t raising enough red flags for the executives to pay attention. Early yields on electronic goods are almost always lousy. Those veteran engineers figured that they would be able to debug the problems and bring the yields up quickly. But the expected rapid improvement in quality just didn’t happen. The communication between upper management and the engineers wasn’t clear. Nor was the strategy aligned between marketing and customer support.

There were plenty of warning signs. Early reports on the problems were myriad. In an Aug. 30, 2005 memo, the team reported overheating graphics chip, cracking heat sinks, cosmetic issues with the hard disk drive and the front of the box, under-performing graphics memory chips from Infineon (now Qimonda), a problem with the DVD drive, and other things. At that point, the contract manufacturers were behind schedule and had only built hundreds of units. They were supposed to have been in high gear, making thousands every week.

The illusion of a well-oiled machine

Yet around that same time, in mid-August 2005, Microsoft gave me a tour of the hardware test center in Mountain View, Calif. Hundreds of boxes were being tested in heat chambers and test labs. Leslie Leland, an engineering manager, said all was on schedule. Microsoft wanted to convey its own confidence to its partners, game developers, and gamers. To my untrained eye, it looked like a smooth operation.

But on the inside, it was a frenetic time. The initial yield on the most critical chip, the three-core microprocessor designed and manufactured by IBM, was only 16 percent. For every 100 produced, only 16 worked. This low yield was surprising because a plaque on the wall in Microsoft’s campus in Mountain View proclaimed that IBM had “taped out,” or completed the design, of its first microprocessor prototype on Dec. 8, 2004. The timing of the tape out was right on schedule, which was good because it is normal for six months to a year of debugging to follow a tape out. IBM had an easier time than the maker of the graphics chip.

ATI Technologies (a graphics chip maker that has since been acquired by Advanced Micro Devices) was late to deliver working graphics chips, or GPUs. Like IBM, it had promised to deliver one of the most complex chips it had ever made in about half the time it usually took to create such chips. But ATI only had half the number of the 400 engineers IBM had on the project.

The fact that both companies had designs done at all was the result of a Herculean effort. Microsoft’s engineers started working on the Xbox 360 at least a year after Sony’s engineers began work on the PlayStation 3, yet Microsoft wound up shipping a year ahead of Sony. Everything Microsoft did was under time pressure, including the creation of the IBM microprocessor and the ATI graphics chip. The only way to pull it all off was through a virtual organization, where Microsoft outsourced many of the tasks to companies that specialized in particular tasks. Microsoft’s hardware engineers in Redmond, Wash. and Mountain View, Calif. were the master integrators of all of the suppliers. The Microsoft chip designers in Mountain View also designed an important video processing chip themselves.

Considering all of the work, Microsoft had too few hardware people. Some of the designers of the Xbox 360, including engineering chief Greg Gibson, were stretched thin. Gibson and J Allard (pictured left), who led the console design effort, had begun work on Zune, Microsoft’s portable media player. Top brass had approved the project to dethrone Apple’s iPod, but Microsoft kept Zune secret from the outside world until much later. Some engineers were pulled of the Xbox 360 at a critical moment to join Allard’s effort to create a music player. Those who were left to work on the test team worked around the clock, traveling to China to work in the factory.