Xbox 360 defects: an inside history of Microsoft’s video game console woes

Go or no go?

Leading up to the launch in the fall of 2005, the number of defective units would soon grow to tens of thousands. Any other consumer electronics company would likely have postponed a launch with such low yields. But Microsoft had more money in the bank than anyone else. The decision this time would fall to Bach and Moore (pictured left). The costs of launching with low yields — where you take big losses on every product sold — could bankrupt other companies. But Microsoft could afford to do so. Microsoft did delay the launch date from October until November. But some inside the company still believed returns would be out of control.

Some of the defects were “latent.” That means they may not show up for some time after the consumers starts playing with the machine. Latent defects can be as high as 50 percent of all defects. That means if you have a low yield to start with, you expect to have a lot of returns when the latent defects kick in later on. Bach said that the real problems with the Xbox 360s didn’t really show up until a year after the launch. That’s when returns were mounting, when they should have been declining.

The yield problem was only discussed internally, and so the public at large was left wondering whether Microsoft was intentionally creating a shortage of consoles by making just a small number of machines. The truth was that Microsoft had to produce a lot of units — many of which failed — to get working consoles that it could ship. It was trying to get as many machines to the market as it could.

A bad decision on testing machines?

The problems began before the Xbox 360s got to the market. The testing machines were not ready, and the battery of tests that they ran were not fully developed. That meant that the testing machines would inspect the Xbox 360s coming off the line and approve them for shipment, even though there were likely flaws.

The test machines were not properly debugged, due to an ill-advised cost-cutting initiative that shaved $2 million from $25 million paid to Cimtek, a test machine maker in Canada. The Microsoft team decided not to pay the consulting fee to Cimtek to build, manage and debug the test machines. Sources familiar with the matter said there were only about 500 test machines at the time of launch, a third of the 1,500 needed.

“There were so many problems, you didn’t know what was wrong,” said one source of the machines. “The [test engineers] didn’t have enough time to get up and running.”

The Xbox 360’s yield climbed somewhat throughout the fall of 2005, as Microsoft readied the worldwide launch in November. But the “first pass yield” (before machines were taken off the line to be reworked inside the factory) was never over 70 percent. Microsoft was counting on its virtual organization of chip designers, assembly houses, and Chinese factory workers to solve the problems. It considered this low-cost infrastructure to be one of its main advantages over Sony, which had high-cost operations under its own roof. But the Chinese manufacturing machine wasn’t as honed as it is today.

If it had shut down the line until it properly debugged everything, Microsoft might have had to delay the launch in Europe or Japan. The Japanese market was considered important to crack, but the games being developed for that market were late anyway. But Robbie Bach had repeatedly said that it was strategically important for Microsoft to fulfill its promises.

If Microsoft had an outstanding test process, then the problems with the console might have been more glaring and Bach might have had more reason to delay.

A tough but successful launch

On Nov. 22, 2005, Microsoft started selling the consoles. It invited 3,000 gamers to come to a hangar in the middle of the Mojave Desert. Gamers rushed into the hangar doors like Greeks rushing into Troy. There, they played the first games for 30 hours and were rewarded with the chance to be the first to buy the machines. Across the country, gamers lined up at stores and waited for midnight. They took them home and obsessively played games like Activision’s World War II shooting game, “Call of Duty 2.”

The hype around the launch was so strong that gamers snapped up every machine. Microsoft had hoped to sell millions, but it was running short in Europe and the U.S., while machines in Japan were barely moving off the shelves. By the end of 2005, Microsoft had only shipped 1.5 million consoles worldwide. That compared to 1.4 million shipped in 2001 with the original Xbox, when it launched in only the U.S. There was now much more demand. Consumers were understandably upset with the relative shortage, since an equal number of units was now shared across three major territories. In Japan, the lack of home-grown Japanese games to go with the Xbox 360 consoles meant that demand was tepid; machines sat unsold in stores even as consumers in the U.S. were paying hundreds more beyond retail on auction sites such as eBay.

The consoles started failing almost immediately. Consumers were excited about being the first on their block to own the new system. They could run graphically beautiful games on high-definition flat-screen TVs, which were just becoming affordable. But their disappointment upon encountering the red rings of death was evident in posts on the Internet. Some customers swore off Microsoft. At the time, Microsoft said that it had received “isolated reports” of console failures and that returns were within the normal range.

Many consumers complained that Microsoft’s customer support people often tried to talk them out of doing a return. (The support people could come across as condescending. They would caution users to avoid putting the console on a plush carpet, or inside a home entertainment center with no air flow). It would be many months before the company set up an easy process where customers could track their returns and replacement units on the web.

According to the Consumer Electronics Association, the average return rate for products where the consumer gets their money back is about 2 percent. That is based on a study conducted in 2005 which showed the return rate for any consumer electronic product is fairly constant, said Steve Koenig, a senior analyst at the CEA.

According to Microsoft’s internal data, the assumption for the long term was that 6 percent or 7 percent of the consoles would be defective before shipping into the market, meaning the yield would be about 94 percent or 93 percent. The rate of return was expected to be low as well. But even after the holidays, Microsoft continued to struggle to ship enough units.

Good quality is taken for granted as a requirement for doing business these days, but it is still difficult to assess what is considered a “normal rate of product returns or yield rates,” said Shukri Souri, a manufacturing expert at Exponent, a Menlo Park, Calif. company that figures out why products fail.

“A significant factor that determines rate of return is how complex a product is to build,” Souri said.

The Xbox 360 was in fact more difficult to build than anything the team had built before. It had 1,700 parts, many of them hard to make. There were more than 200 suppliers. The chips and the box (insides pictured left) were designed to balance performance, aesthetics and power dissipation. As Holmdahl pointed out, a problem with any one of the parts or any one of the vendors could result in a problem that affected the whole box.

In 2006, it only got worse

Despite the problems, some results were turning out better than expected, such as how quickly Microsoft would sign gamers up for its Xbox Live online games service. (That figure has now hit 12 million subscriptions for both paid and unpaid subscriptions, more than originally forecast).

But some things, Microsoft clearly didn’t plan for. The complaints of shortages and problems with consoles continued into the spring of 2006. By the end of March 31, Microsoft said it had shipped more than 3.3 million consoles to retailers. There was a growing “bone pile” in a warehouse at Wistron and at a repair center in Texas.

Microsoft had more than 500,000 defective consoles that sat in warehouses. They were either duds coming out of the factory or they were returned boxes, according to inside sources. The yield was climbing, but far too slowly. The company stood by its statement that returns were within “normal rates for consumer electronics products.”

In the spring of 2006, Moore said with some relief that Microsoft was finally readying a surge of new consoles for the market. Celestica, a third manufacturing partner, was bringing its factory online. It seemed that Microsoft was finally turning a corner on production.

Because Sony and Nintendo didn’t have their new machines ready yet, Microsoft had a wide-open field in 2006. It was racing to snare the most customers before the rivals could compete. But the cheaper priced and older PlayStation 2 continued to outsell the Xbox 360 throughout 2006.

Customer complaints gather steam

Szarek, the previously mentioned resident of Chicopee, Mass. gained some notoriety when he said that he suffered three failed consoles by May, 2006. His first console had lasted only 28 days after he picked it up. Then he became even more frustrated with the returns process.

Under the glare of publicity, Moore personally apologized to Szarek and made sure he got a free game and wasn’t charged for the repairs. Moore’s comments were conciliatory and he managed to calm some of the critics. One of the problems: though the return rate was predictably high, Microsoft had not made big investments in its return infrastructure to enable a smooth return process. One PR woman said she had to do shifts doing support calls.

“When you have tens of millions of people who own something, you’ll have people using the Internet as a forum,” Moore said at the time. “I’m not hiding behind the fact that people have had problems with the console and at times we have been deficient with the way we’ve handled them. We are improving our customer service. There were times when we did not handle complaints the way we should have. I take accountability for that. I have not seen since that any further issues.”

The denial of widespread problems infuriated Szarek and others who felt Microsoft was stonewalling. As the inventory surge materialized, Microsoft was able to send replacement units out more quickly and deal with those who were unhappy about the shortage. Sales topped 5 million by June 30, 2006. Moore had personally set a goal of selling 10 million machines before his rivals sold their first machines. That was going to be tough, since Sony and Nintendo had set their launches for the fall of 2006.