The Chicopee, Mass. gamer was accustomed to the hardware failures that became known throughout the Internet as RROD, or the “red rings of death” which flash when the console becomes inoperable.
[aditude-amp id="flyingcarpet" targeting='{"env":"staging","page_type":"article","post_id":92872,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"games,mobile,social,","session":"A"}']A 40-year-old photographer, Szarek was a hardcore Microsoft fan who spent more than $1,000 on his games. But each time one of his Xbox 360 consoles failed, he had to spend time convincing Microsoft’s tech support that they should send him a new console. Each time he got a refurbished console as a replacement (a machine that had been returned to a repair center in Texas, fixed as much as possible, and then shipped back out). When he complained on the Internet and to the media about the shoddy product and poor customer service, people branded him a cry baby and wrote him off as a statistical anomaly. But by the spring of 2008, Szarek was vindicated. There were at least a million or two other people like him.
Szarek’s fourth machine lasted almost two years, experiencing the same short life that many other Xbox 360s suffered. Microsoft replaced these machines for free under the warranty that it announced on July 5, 2007, for defective Xbox 360s exhibiting what it more politely called the “three flashing red lights.” That warranty program cost Microsoft up to $1.15 billion, but the loss of face and loyalty among gamers in the fierce console war with Nintendo and Sony has been immeasurable. Szarek, who became a spokesman for dispossessed defective Xbox 360 owners, played a part in making Microsoft acknowledge its console quality problem.
AI Weekly
The must-read newsletter for AI and Big Data industry written by Khari Johnson, Kyle Wiggers, and Seth Colaner.
Included with VentureBeat Insider and VentureBeat VIP memberships.
This is the unauthorized tale of how Microsoft lost its chance to become the leader in the biggest market it has attacked beyond its twin monopolies in Office and Windows software. Rival game console maker Nintendo out-thought the larger players Microsoft and Sony by designing the Wii game console with a clever, intuitive game controller. Even so, Microsoft could have captured more gamers during this product generation, yet the RROD problem held it back. The Xbox 360’s defect problem will go down as one of the worst snafus in consumer electronics history.
Its own worst enemy
Microsoft knew it had flawed machines, but it did not delay its launch because it believed the quality problems would subside over time. With each new machine, the company figured it would ride the “learning curve,” or continuously improve its production. Even though Microsoft’s leaders knew their quality wasn’t top notch, they did not ensure that resources were in place to handle returns and quickly debug bad consoles. There were plenty of warning signs, but the company chose to ignore them. The different parts of the business weren’t aligned.
It reminds me of the German war machine just before World War I, as chronicled by Barbara Tuchman in the classic history book, “The Guns of August.” The German generals were intent on keeping their trains on time; but the leaders overlooked their chances for stopping the war altogether. The Schlieffen plan called for them to strike first. Once the Russians and French mobilized, the Germans had to move into action. They marched off blindly into tragedy.
Likewise, Microsoft’s strategy depended on beating its rivals to market. It couldn’t afford to stop and delay the launch in order to solve its quality problems, or so upper management believed. What Microsoft’s leaders didn’t realize was that getting to market first with a flawed machine would only win them a battle; and it risked the loss of the war.
[aditude-amp id="medium1" targeting='{"env":"staging","page_type":"article","post_id":92872,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"games,mobile,social,","session":"A"}']
“They got enamored with the idea of the Microsoft army rolling everything out at the same time,” said one knowledgeable source who asked not to be identified.
The quality problem negated much of the advantage of going first, and it has delayed the company’s plan to aggressively market the console and slash its prices. (Microsoft disputes this point; it cut the price of all three versions of its Xbox consoles by $50 to $79 on Wednesday. And the company believes it will sell more boxes than Sony will. But prices ought to be lower still during this stage of the console life cycle). That has stopped the company from reaching the broader market of consumers that Nintendo has won over. It has lowered its ambitions, hoping instead just to get a clear edge on third-placed Sony. The future profits that the company once hoped for are now likely to wind up in Nintendo’s pockets.
Microsoft’s top game executive, Robbie Bach, president of the Entertainment & Devices group, said at a dinner in July that Microsoft’s own research shows that gamers have largely forgiven the company for defective Xbox 360s. Microsoft has still sold more Xbox 360 consoles than Sony to date. But there is no doubt that the company has lost considerable good will among gamers. Before Microsoft offered free replacements, connsumers grumbled that they had to turn to forums, such as those on Ars Technica, to vent and to find solutions to problems that the company didn’t openly discuss. And for a couple of months now, Sony’s PlayStation 3 has been outselling the Xbox 360 in the U.S. for the first time.
“Fundamentally, their thinking shows that they are a software company at heart,” said one veteran manufacturing executive. “They put something out and figure they can fix it with the next patch or come up with a bug fix.”
[aditude-amp id="medium2" targeting='{"env":"staging","page_type":"article","post_id":92872,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"games,mobile,social,","session":"A"}']
The terrifying part of the story is that this kind of problem — where technology fails and no one knows what to do about it — can happen to any company.
About this story
I asked Microsoft to confirm or deny 35 different facts contained in this story. Instead, I received a formal statement from a Microsoft spokesperson, saying the company had already acknowledged an “unacceptable number of repairs” to Xbox 360 consoles and responded to the hardware failures with a free replacement program. The statement also said, “This topic has already been covered extensively in the media. This new story repeats old information, and contains rumors and innuendo from anonymous sources, attempting to create a new sensational angle, and is highly irresponsible.”
I don’t think this story is sensational. I have tried to verify the facts over several years. I view this story as the last chapter for my book on the making of the Xbox 360, “The Xbox 360 Uncloaked: The Real Story Behind Microsoft’s Next-Generation Video Game Console.”
[aditude-amp id="medium3" targeting='{"env":"staging","page_type":"article","post_id":92872,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"games,mobile,social,","session":"A"}']
The facts revealed themselves slowly, emerging from the day-to-day stories that I wrote about the game business. Some people might consider this post mortem to be ancient history. But the reverberations are still playing out today. They help explain why Microsoft isn’t being aggressive with its price cuts and why gamers aren’t getting bargains on hardware as they did the last generation. While I talked to many people for this story, few were willing to let me use their names. As you will see, not every source is anonymous, and we have included the viewpoint of Microsoft executives from past interviews.
The details are interesting because they offer a deeper look into how the console business runs than is otherwise available. Microsoft, for instance, still hasn’t perfected its Xbox 360 manufacturing process. In the absence of a precise chronology from Microsoft, some anonymous sources have tried to describe what happened. But the history of the decision making and inside story of what happened on the RROD has never been told, until now. (Please check out our GamesBeat 09 conference on March 24).
[aditude-amp id="medium4" targeting='{"env":"staging","page_type":"article","post_id":92872,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"games,mobile,social,","session":"A"}']
Microsoft’s mea culpa
In many ways, the Xbox 360 was a big achievement. Microsoft had beaten its rivals Sony and Nintendo to market by a year. Sony had outsold Microsoft five-to-one in the previous generation of consoles, but Microsoft has outsold Sony’s PlayStation 3 with the Xbox 360. Microsoft’s to game executive Bach (left) had repeatedly promised that Microsoft would be profitable in his Entertainment & Devices group in the fiscal year ending June 30, 2008. The company met that target. Big losses have turned into profits.
But when the company launched its Xbox 360 video game console in November, 2005, it didn’t have a handle on product quality and it was not prepared to systematically analyze its product returns and debug bad consoles.
Microsoft has admitted those mistakes. On July 5, 2007, the company said it would take a billion-dollar write-off to pay for free replacement of Xbox 360s by up to three years from the date of purchase. Peter Moore, who at the time was head of the games business at Microsoft, said in an interview the decision to take the write-off was due to an “unacceptable number” of returns. The company made improvements to the console’s quality, but Moore said it was unclear how many units would fail in the field. Microsoft then followed up its new policy with numerous public apologies, including a statement from Bach, its game executive.
[aditude-amp id="medium5" targeting='{"env":"staging","page_type":"article","post_id":92872,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"games,mobile,social,","session":"A"}']
Microsoft never disclosed its actual return rates. But according to data obtained by VentureBeat, the total number climbed above 1.2 million consoles in early 2007. That is a huge amount, considering Microsoft had only shipped 11.6 million into stores by the time of the announcement in mid-2007.
The company was dragged kicking and screaming to its admission of widespread defects — with gamers and the press doing the dragging. In interviews in April and May of 2007, Microsoft executives denied that quality problems were haunting them. Todd Holmdahl, (pictured left) general manager in charge of the Xbox platform for Microsoft in Redmond, Wash., said in an interview with me on May 9, 2007, while I was at the San Jose Mercury News, that the “vast majority” of Microsoft’s Xbox 360 customers were having a great experience with their consoles. He declined, however, to say what percentage of consoles were being returned for defect reasons. In doing so, he was as coy as any of the console makers are about such sensitive data. But everyone wanted an answer to that question.
It’s not clear why it took Microsoft so long to announce its warranty extension last year, almost 19 months from the time when the first complaints arose at the outset of the launch. As it was investigating the cause of the problems, the company stayed quiet, acknowledging problems only little by little. The company delayed its free-replacement announcement until it understood the scope of the problem.
In July, game executive Bach said that the number of returns was not measurably big until a year after the launch. It was a problem that became evident only over time, he said.
“It wasn’t related to things we were seeing in testing or some judgment call we had to make about whether the product was ready,” Bach said at the dinner. “We were confident the product was ready. We did a lot of testing. The problem that shows up with the three red lights on the console is a complex interaction with some very complex parts.”
Ignoring warnings
But the evidence for the quality debacle was there to see even before Microsoft shipped any machines. In August, 2005, as Microsoft was gearing up production, an engineer raised a hand and said, “Stop. You have to shut down the line.” This wasn’t just a brief moment. The engineer spoke up repeatedly.
That engineer, who asked not to be identified, had deep experience in manufacturing. When production results were really off kilter, stopping a line and tracing a problem back to its roots was the answer. But the higher-ranking engineers, managers and executives chose to risk going forward. There wasn’t a universal backlash from the engineering ranks, according to one engineering source.
Nobody listened to that engineer — who spoke on condition of anonymity — apparently because console launches are always hurried affairs. Yields — the percentage of working products in a given batch of total products produced — generally start low. As the manufacturers conduct statistical analysis and tight controls on every step in assembly, they learn how to drive the yields up.
Still, the picture wasn’t pretty. The defect rate for the machines was an abysmal 68 percent at that point, according to several sources. That meant for every 100 machines that Microsoft’s contract manufacturers, Flextronics and Wistron, made at their factories in China, 68 didn’t work. At the recent dinner, Bach denied that there was a big concern about defects at that point in time.
At that point, it is likely true that the engineers weren’t raising enough red flags for the executives to pay attention. Early yields on electronic goods are almost always lousy. Those veteran engineers figured that they would be able to debug the problems and bring the yields up quickly. But the expected rapid improvement in quality just didn’t happen. The communication between upper management and the engineers wasn’t clear. Nor was the strategy aligned between marketing and customer support.
There were plenty of warning signs. Early reports on the problems were myriad. In an Aug. 30, 2005 memo, the team reported overheating graphics chip, cracking heat sinks, cosmetic issues with the hard disk drive and the front of the box, under-performing graphics memory chips from Infineon (now Qimonda), a problem with the DVD drive, and other things. At that point, the contract manufacturers were behind schedule and had only built hundreds of units. They were supposed to have been in high gear, making thousands every week.
Yet around that same time, in mid-August 2005, Microsoft gave me a tour of the hardware test center in Mountain View, Calif. Hundreds of boxes were being tested in heat chambers and test labs. Leslie Leland, an engineering manager, said all was on schedule. Microsoft wanted to convey its own confidence to its partners, game developers, and gamers. To my untrained eye, it looked like a smooth operation.
But on the inside, it was a frenetic time. The initial yield on the most critical chip, the three-core microprocessor designed and manufactured by IBM, was only 16 percent. For every 100 produced, only 16 worked. This low yield was surprising because a plaque on the wall in Microsoft’s campus in Mountain View proclaimed that IBM had “taped out,” or completed the design, of its first microprocessor prototype on Dec. 8, 2004. The timing of the tape out was right on schedule, which was good because it is normal for six months to a year of debugging to follow a tape out. IBM had an easier time than the maker of the graphics chip.
ATI Technologies (a graphics chip maker that has since been acquired by Advanced Micro Devices) was late to deliver working graphics chips, or GPUs. Like IBM, it had promised to deliver one of the most complex chips it had ever made in about half the time it usually took to create such chips. But ATI only had half the number of the 400 engineers IBM had on the project.
The fact that both companies had designs done at all was the result of a Herculean effort. Microsoft’s engineers started working on the Xbox 360 at least a year after Sony’s engineers began work on the PlayStation 3, yet Microsoft wound up shipping a year ahead of Sony. Everything Microsoft did was under time pressure, including the creation of the IBM microprocessor and the ATI graphics chip. The only way to pull it all off was through a virtual organization, where Microsoft outsourced many of the tasks to companies that specialized in particular tasks. Microsoft’s hardware engineers in 
Considering all of the work, Microsoft had too few hardware people. Some of the designers of the Xbox 360, including engineering chief Greg Gibson, were stretched thin. Gibson and J Allard (pictured left), who led the console design effort, had begun work on Zune, Microsoft’s portable media player. Top brass had approved the project to dethrone Apple’s iPod, but Microsoft kept Zune secret from the outside world until much later. Some engineers were pulled of the Xbox 360 at a critical moment to join Allard’s effort to create a music player. Those who were left to work on the test team worked around the clock, traveling to China to work in the factory.
Outsiders frequently slam Microsoft’s hardware engineers as having a software (fix it later) mentality. But Microsoft’s engineers had been trained to put quality first. Most of them had years of training. Holmdahl, for instance, had been with Microsoft in the early days of its hardware division. By the time he joined the Xbox project, he was a veteran, having worked with contractors such as Flextronics to make 20 million mice per year. Making consoles was an altogether different task, but the team had expert help through its Web TV acquisition, which brought veteran console hardware engineers aboard in 1998. One of the best at debugging was Nick Baker, one of the architects of the Xbox 360’s chips.
In 2001, the team went through the paces making sure that the original Xbox had good quality. On that launch, Microsoft was 20 months behind Sony’s launch of the PlayStation 2. Microsoft’s original Xbox team moved as fast as they could, but they took a long time defining the box and picking vendors.
By the time they hired Nvidia in March 2000 to make the graphics chip, they were hopelessly behind. Nvidia combined two different chips into one in half the time it took to design a chip, but it fell behind schedule in part because of a bug. Someone else had specified a sub-par power supply for the Xbox, and that made the graphics chip hiccup now and then. Once the team fixed the problem, they raced ahead.
But Flextronics, the contract manufacturer, wasn’t ready. It had to build a bigger manufacturing database to accommodate the design. Microsoft had to postpone its launch by a week. It also decided to launch only in the U.S. market and would wait until the next spring to launch in Europe and Japan. No console maker had ever pulled off a worldwide launch before, and now Microsoft understood why. The delays in other territories put it even further behind Sony. Sony eventually outsold Microsoft by more than five to one — scarring into memory a tough lesson about market timing.
But the Microsoft hardware team had come out shining. Aside from a problem with a supplier’s DVD drive, the number of defective Xboxes was low. After the DVD drive issue was fixed, yields rose above 90 percent and stayed there for the duration of the manufacturing. Microsoft thus had a seasoned team to work on the Xbox 360.
Despite the good quality manufacturing, Microsoft’s game business was deeply flawed in the Xbox generation. In the second round, Microsoft vowed to learn from its mistakes.
In the first generation, the company lost $3.7 billion over four years. Most of that was because the costs of the box — particularly its hard disk drive — were too high. Microsoft’s planners thought the hard drive would give them an edge over Sony’s PS 2. It didn’t.
Bill Gates didn’t really care about the losses in the first generation. That was simply the ante for getting into an exciting new business. Microsoft has always viewed its ambitions in the game business as strategic. It isn’t in the business just to dominate video games, but to own the living room. The game console is a kind of gateway to the Internet and all of the future entertainment services that can be piped into the living room.
But Steve Ballmer, who took over from Gates as CEO during the first generation, really wanted the Xbox business to be profitable second time around. He wanted to keep the foothold in the living room, but not at the expense of draining Microsoft’s cash, which was needed to fight Google and other rivals.
This time, Gates said he felt that Microsoft would have a fundamental cost advantage over Sony. One of the key ways Bach hoped to outwit his rivals was with speed: getting a machine to market in four years instead of five. Secretly, Microsoft had planned on selling 50 million Xbox 360s at that point.
The Xbox team met for a series of retreats. One such retreat took place in early 2004 at the elegant Salish Lodge beside the Snoqualmie Falls made famous in the Twin Peaks TV show. After that meeting, Bach, who had headed the Xbox business as its senior executive from the time of its inception, decided that company would aim for profitability with the Xbox 360. As much as possible, the machines would not be sold for a loss.
Bach believed the hardware itself should be profitable over the life cycle of the system. His planners counted on good revenues from accessories and games to ensure profits the second time around. And, in the three-page paper that outlined the grand strategy, he said that Microsoft wanted to hit the market at the same time as its rivals.
Microsoft’s best guess was that Sony would launch the PS 3 in 2005. That turned out to be a year off, since Sony ran into its own development problems related to its decision to put Blu-ray drives instead of DVD drives into the PS 3.
There was more time to design the box than the first-generation team had, but it was still a race. Different teams sped into planning. The different groups included chip designers, hardware planners, game designers, software operating system engineers, marketers and industrial designers.
With the Xbox 360, Microsoft was guilty of packing a little too much into its box. The company sent out its scouts in late 2002 to ask gamers what they wanted. The first Xbox came off as “The Incredible Hulk.” The second Xbox, Microsoft had mandated, had to be smaller, if only to fit in the smaller homes in Japan, a key target market. The designers thought it had to be more like Bruce Lee. Microsoft’s own user-experience team ran the show, while outside industrial design firms competed in a bake-off to define the soul of the machine.
Hers Experimental Laboratory of Japan and Astro Studios in San Francisco won the bake-off. They wanted to create something that looked elegant and iconic, qualities that were sometimes at odds with the computer that it was. Those firms gave the box its signature look — curved sides that looked as if the box were inhaling air and was about to explode with energy (that was the marketing image, anyway). There was a constant tug of war between the industrial designers, who wanted something small, and the engineers, who wanted to pack a lot of performance into the box. The more powerful the system was, the bigger the box had to be to provide air flow for heat-sensitive components.
Microsoft decided late to add a hard disk drive to most of the machines. It also came up late with a plan to add wireless controllers; all of the previous consoles shipped with wired controllers. The hard drive blocked a lot of the air flow on one side of the machine. And the wireless modules had to have enough of their own space to ensure that there was no electrical interference. In the end, the machine was a series of compromises. The gigantic power supply needed to run the machine sat outside in the power cord, while the console shell was poked full of holes on the sides to ensure air flow.
“It turned out in the end that this was all going too far, too fast,” said one source. “They were adding too many features after things were locked down. That incremental feature adding just made it fragile.”
The marketers also decided the box had to be backward compatible with the older games, including “Halo” and “Halo 2.” And they also felt the box had to have enough horsepower to make games look beautiful on high-definition TVs.
Those parameters, in turn, gave direction to the chip and hardware designers. This time, hardware chief Todd Holmdahl wanted Microsoft to have much more control over its fate. And he believed he had built up a team that could act as the “system integrator.”
Microsoft had a team of 100 chip engineers and 100 more hardware engineers between Redmond and Mountain View. They were the overseers of the integration of the parts into a working console. It was a relatively small team for such a large project compared to a hardware company such as Sony, but Microsoft had plenty of help from its partners.
In contracts with IBM and ATI, Microsoft set early deadlines for delivery of the first working chips. It also decided that it wanted to own the chip designs. That way, if it wanted to, it could take the designs and have them made by a factory of its choosing and wasn’t locked into using IBM and ATI (or ATI’s chosen partner) chip factories. The reason for this change was that Microsoft wanted to avoid dependence on its chip partners. It had learned the hard way after getting into a bad dispute with Nvidia over the prices charged for the graphics chip in the original Xbox. The case had to go to arbitration and Microsoft ultimately lost that battle. However, this was a mistake: it was now responsible for failures related to the chip.
Contract manufacturing had matured since the last generation. Flextronics was willing to make each console for just $10 each, while another manufacturer, Wistron, would do the same for $7, according to sources. They hoped to make profits on large volumes of machines sold. They were also practiced in the art of reducing the costs of the components they put into their machines. Overall, Microsoft had a much more formidable hardware operation, when you consider all of its virtual resources, on the second time around. But it still wasn’t enough to do a perfect job.
Leading up to the launch in the fall of 2005, the number of defective units would soon grow to tens of thousands. Any other consumer electronics company would likely have postponed a launch with such low yields. But Microsoft had more money in the bank than anyone else. The decision this time would fall to Bach and Moore (pictured left). The costs of launching with low yields — where you take big losses on every product sold — could bankrupt other companies. But Microsoft could afford to do so. Microsoft did delay the launch date from October until November. But some inside the company still believed returns would be out of control.
Some of the defects were “latent.” That means they may not show up for some time after the consumers starts playing with the machine. Latent defects can be as high as 50 percent of all defects. That means if you have a low yield to start with, you expect to have a lot of returns when the latent defects kick in later on. Bach said that the real problems with the Xbox 360s didn’t really show up until a year after the launch. That’s when returns were mounting, when they should have been declining.
The yield problem was only discussed internally, and so the public at large was left wondering whether Microsoft was intentionally creating a shortage of consoles by making just a small number of machines. The truth was that Microsoft had to produce a lot of units — many of which failed — to get working consoles that it could ship. It was trying to get as many machines to the market as it could.
A bad decision on testing machines?
The problems began before the Xbox 360s got to the market. The testing machines were not ready, and the battery of tests that they ran were not fully developed. That meant that the testing machines would inspect the Xbox 360s coming off the line and approve them for shipment, even though there were likely flaws.
The test machines were not properly debugged, due to an ill-advised cost-cutting initiative that shaved $2 million from $25 million paid to Cimtek, a test machine maker in Canada. The Microsoft team decided not to pay the consulting fee to Cimtek to build, manage and debug the test machines. Sources familiar with the matter said there were only about 500 test machines at the time of launch, a third of the 1,500 needed.
“There were so many problems, you didn’t know what was wrong,” said one source of the machines. “The [test engineers] didn’t have enough time to get up and running.”
The Xbox 360’s yield climbed somewhat throughout the fall of 2005, as Microsoft readied the worldwide launch in November. But the “first pass yield” (before machines were taken off the line to be reworked inside the factory) was never over 70 percent. Microsoft was counting on its virtual organization of chip designers, assembly houses, and Chinese factory workers to solve the problems. It considered this low-cost infrastructure to be one of its main advantages over Sony, which had high-cost operations under its own roof. But the Chinese manufacturing machine wasn’t as honed as it is today.
If it had shut down the line until it properly debugged everything, Microsoft might have had to delay the launch in Europe or Japan. The Japanese market was considered important to crack, but the games being developed for that market were late anyway. But Robbie Bach had repeatedly said that it was strategically important for Microsoft to fulfill its promises.
If Microsoft had an outstanding test process, then the problems with the console might have been more glaring and Bach might have had more reason to delay.
On Nov. 22, 2005, Microsoft started selling the consoles. It invited 3,000 gamers to come to a hangar in the middle of the Mojave Desert. Gamers rushed into the hangar doors like Greeks rushing into Troy. There, they played the first games for 30 hours and were rewarded with the chance to be the first to buy the machines. Across the country, gamers lined up at stores and waited for midnight. They took them home and obsessively played games like Activision’s World War II shooting game, “Call of Duty 2.”
The hype around the launch was so strong that gamers snapped up every machine. Microsoft had hoped to sell millions, but it was running short in Europe and the U.S., while machines in Japan were barely moving off the shelves. By the end of 2005, Microsoft had only shipped 1.5 million consoles worldwide. That compared to 1.4 million shipped in 2001 with the original Xbox, when it launched in only the U.S. There was now much more demand. Consumers were understandably upset with the relative shortage, since an equal number of units was now shared across three major territories. In Japan, the lack of home-grown Japanese games to go with the Xbox 360 consoles meant that demand was tepid; machines sat unsold in stores even as consumers in the U.S. were paying hundreds more beyond retail on auction sites such as eBay.
The consoles started failing almost immediately. Consumers were excited about being the first on their block to own the new system. They could run graphically beautiful games on high-definition flat-screen TVs, which were just becoming affordable. But their disappointment upon encountering the red rings of death was evident in posts on the Internet. Some customers swore off Microsoft. At the time, Microsoft said that it had received “isolated reports” of console failures and that returns were within the normal range.
Many consumers complained that Microsoft’s customer support people often tried to talk them out of doing a return. (The support people could come across as condescending. They would caution users to avoid putting the console on a plush carpet, or inside a home entertainment center with no air flow). It would be many months before the company set up an easy process where customers could track their returns and replacement units on the web.
According to the Consumer Electronics Association, the average return rate for products where the consumer gets their money back is about 2 percent. That is based on a study conducted in 2005 which showed the return rate for any consumer electronic product is fairly constant, said Steve Koenig, a senior analyst at the CEA.
According to Microsoft’s internal data, the assumption for the long term was that 6 percent or 7 percent of the consoles would be defective before shipping into the market, meaning the yield would be about 94 percent or 93 percent. The rate of return was expected to be low as well. But even after the holidays, Microsoft continued to struggle to ship enough units.
“A significant factor that determines rate of return is how complex a product is to build,” Souri said.
The Xbox 360 was in fact more difficult to build than anything the team had built before. It had 1,700 parts, many of them hard to make. There were more than 200 suppliers. The chips and the box (insides pictured left) were designed to balance performance, aesthetics and power dissipation. As Holmdahl pointed out, a problem with any one of the parts or any one of the vendors could result in a problem that affected the whole box.
In 2006, it only got worse
Despite the problems, some results were turning out better than expected, such as how quickly Microsoft would sign gamers up for its Xbox Live online games service. (That figure has now hit 12 million subscriptions for both paid and unpaid subscriptions, more than originally forecast).
But some things, Microsoft clearly didn’t plan for. The complaints of shortages and problems with consoles continued into the spring of 2006. By the end of March 31, Microsoft said it had shipped more than 3.3 million consoles to retailers. There was a growing “bone pile” in a warehouse at Wistron and at a repair center in Texas.
Microsoft had more than 500,000 defective consoles that sat in warehouses. They were either duds coming out of the factory or they were returned boxes, according to inside sources. The yield was climbing, but far too slowly. The company stood by its statement that returns were within “normal rates for consumer electronics products.”
In the spring of 2006, Moore said with some relief that Microsoft was finally readying a surge of new consoles for the market. Celestica, a third manufacturing partner, was bringing its factory online. It seemed that Microsoft was finally turning a corner on production.
Because Sony and Nintendo didn’t have their new machines ready yet, Microsoft had a wide-open field in 2006. It was racing to snare the most customers before the rivals could compete. But the cheaper priced and older PlayStation 2 continued to outsell the Xbox 360 throughout 2006.
Customer complaints gather steam
Szarek, the previously mentioned resident of Chicopee, Mass. gained some notoriety when he said that he suffered three failed consoles by May, 2006. His first console had lasted only 28 days after he picked it up. Then he became even more frustrated with the returns process.
Under the glare of publicity, Moore personally apologized to Szarek and made sure he got a free game and wasn’t charged for the repairs. Moore’s comments were conciliatory and he managed to calm some of the critics. One of the problems: though the return rate was predictably high, Microsoft had not made big investments in its return infrastructure to enable a smooth return process. One PR woman said she had to do shifts doing support calls.
“When you have tens of millions of people who own something, you’ll have people using the Internet as a forum,” Moore said at the time. “I’m not hiding behind the fact that people have had problems with the console and at times we have been deficient with the way we’ve handled them. We are improving our customer service. There were times when we did not handle complaints the way we should have. I take accountability for that. I have not seen since that any further issues.”
The denial of widespread problems infuriated Szarek and others who felt Microsoft was stonewalling. As the inventory surge materialized, Microsoft was able to send replacement units out more quickly and deal with those who were unhappy about the shortage. Sales topped 5 million by June 30, 2006. Moore had personally set a goal of selling 10 million machines before his rivals sold their first machines. That was going to be tough, since Sony and Nintendo had set their launches for the fall of 2006.
The internal response
During the spring of 2006, the engineers were working on a transition to a new motherboard, code-named Zephyr. But they postponed that transition to work on fixing the “bone pile” issues and “maximize the yield,” according to an email circulated by engineering manager Harjit Singh to the hardware team on March 10, 2006. The yield at that point was an abysmal 50 percent on the first pass. When the bad machines were reworked within the factory, the yield went up to 75 percent –- hardly acceptable. Singh had said the process in the factories was “not repeatable” at a time when they were scheduled to triple production in the coming months. That meant that something could go slightly wrong and Microsoft would have no idea how to fix the problem.
Microsoft finally started throwing more engineers at the problem. It took engineers off projects such as reducing the costs of its wireless controllers because, Singh wrote, “if we don’t have the consoles, we don’t need the peripherals.” Marc Whitten was temporarily assigned to work on evaluations for adding new heat sinks to address thermal issues on the big chips. It was essentially “all hands on deck” for engineers, who were expected to devote 75 percent of their time to the yield/bone pile issues. But, again, those engineers didn’t achieve magical fixes. And it was late in the game.
The company pushed to stir up sales. Because Sony appeared to be falling off schedule a second time, the Microsoft executive team challenged the engineers to improve shipment targets by 25 percent through June, 2006. The team decided to postpone the launch of Zephyr (a board with HDMI connector for better video quality) and concentrate on shipping Xenon boards. The target was to hit 80 percent first-pass yield, but that wasn’t reaslistic.
During the fall, Microsoft offered bundles, or packages of games and consoles together, which effectively amounted to a price cut. Some analysts accused Microsoft of “stuffing the channel,” or shipping more consoles to retailers than they wanted or needed for the holidays. Microsoft denied those claims.
“We’re a very responsible company,” Moore said to me in the May, 2007 interview. “How do you determine who stuffed the channel? What is the optimal number of units? We’re not in a position to force the product upon the retailers. It wouldn’t be the console wars without conspiracy theories.”
But problems with manufacturing continued. The yield on the consoles was only 85 percent at the beginning of 2008, not at the 90-percent plus level that was expected at this point in the console cycle. That meant there was an extra billion dollars in cost that Microsoft hadn’t anticipated. But, again, Moore at the time said that the manufacturing problems Microsoft had were not enough to ruin the business model.
Dead Rising
“If you coordinated the music player with the dashboard, you could get almost every 360 to lock up,” he said in an interview. “I did it first on a combo DVD/audio disk. With NBA 2K6, you would select the music. The screen went black.”
The NBA 2K6 flaw resulted in the Red Rings of Death. Delaware, who as of this writing works at Microsoft as a game tester, agreed to go on the record for this story because he said he believes passionately in his work, which involved painstakingly playing games over and over again in order to uncover bugs. But he also believed passionately in the Xbox 360 gamer community. He is one of those thousands of employees who was good at his job, loved the games, and sought to push Microsoft to excellence. But he said he found it difficult to stay silent on problems that caused a number of his friends a great deal of consternation. He said he realized that speaking out could cost him his job and he plans on asking Microsoft to forgive him for it. He was courageous in coming forward, and he was not the source of emails cited in this story.
Delaware was convinced that at least some of the problems reported about Dead Rising were related to the “2858 dashboard update” for Xbox Live, which he believed was embedded in the game. Another explanation was that games such as Dead Rising demanded more from the consoles, which gave out under stress. Delaware believed that the practice of embedding console and dashboard updates in game disks is responsible for some hardware failures.
In October, 2006 Delaware was concerned about how much information Microsoft was sharing with its testing partner and offered a warning about the Xbox Live updates. “With the upcoming Wii and PS 3 launches, MS cannot afford the bad PR that might result from Update related issues,” he said. “Asking these questions from the start might help to prevent possible criticism of our testing process, in the event of update-related problems. The last thing VMC needs is to take the blame for problems that Microsoft has (known) about from the start.”
So what exactly was wrong with the machines? As time would reveal, there was no single reason for the failures, though many of the problems could be blamed on the ATI graphics chip, which could overheat so much it warped the motherboard. This put stress on bad solder joints, causing them to fail early in the machine’s life. Sometimes the heat sinks on top of the GPU were put on the wrong way, resulting in heat problems. Finally, games would sometimes crash because of sub-par memory. Infineon had been brought aboard as the second supplier behind Samsung for the GDDR3 memory used in the Xbox 360. This new kind of memory chip was specified for 700 megahertz, but the Infineon parts were falling short of that target. Microsoft had to set up a line for sorting through the good parts and the bad parts, contributing to a shortage of consoles.
Problems with the DVD drive also lasted longer than expected. And the console was also one of the first products that had to meet new environmental standards in Europe, which prohibited the use of lead in solder (which, when melted, fuses electronic components together). Paul Wang, a Microsoft test engineer on the Xbox 360, said in a speech in 2007 before a Silicon Valley engineering group that the lead-free solder created a lot of problems.
Perhaps Microsoft had too many balls in the air. As it launched the Xbox 360, it was already planning other big related projects. Cobalt was a next-generation HD-DVD movie player (pictured above) that Microsoft would launch in the holidays of 2006 as a $199 add-on. A project code-named Zephyr (later renamed the Xbox 360 Elite) would add a more expensive hard disk drive with 120 gigabytes of storage and a new HDMI connector. And the company’s engineers were hard at work with suppliers on a project code-named Falcon, which was an effort to create a motherboard, or main circuit board inside the box, to reduce the costs of the console itself during 2007.
“That was really annoying, that they had us working on so many things,” an engineering source said.
As the engineers wound down one task, they had to move to the next. There was no team set aside to deal with the low yields, in part because the poor yields hadn’t been anticipated.
Fessing up
In September, 2006 that the company made a rare admission. It said in a statement that the quality of the consoles it made during 2005 wasn’t as high as it expected and therefore it would extend the policy of free replacement for consoles made during 2005, even though the warranties had expired.
While some industry observers expected Microsoft to cut the costs on the console and offer consumers a price cut during the second holiday season, that was never in the plans. Instead, the company was ramping up to offer a new accessory, an HD-DVD drive that could play next-generation DVD movies for the low price of $199. That was meant to parry Sony’s PlayStation 3, which had a Blu-ray next-generation DVD drive built into the system.
Fortunately, Sony was stumbling. Besides being a year late, the costs on the Blu-ray drive were so high that Sony had to price its consoles at $499 and $599, far above Microsoft’s prices of $299 and $399. Market analysts estimated the cost of Sony’s box was around $800 or more. Microsoft didn’t need to offer a price cut after all, particularly since Sony was struggling to build enough consoles for its fall launch.
It was looking like, even with the manufacturing problems, Microsoft was going to do just fine. In mid-November 2006, about 645,079 units, or 8.8 percent of the total units shipped, had been returned for repair. Sony had been more conscientious about delaying its launch when the product wasn’t ready. JackTretton, the president of Sony’s U.S. games division, said he was proud that Sony waited. But it was going to pay for that cautiousness.
The three Microsoft contract manufacturers had now ramped up to full production. They produced more than 12 million consoles. In early 2007, Peter Moore announced that Microsoft had hit its revised target and had shipped more than 10.4 million consoles into the market before the end of 2006. It only sold through 9 million consoles. There were a lot of leftover consoles sitting on shelves.
At about the same time, in December, 2006, Microsoft took another step to appease consumers who had failed machines. It said that it was extending the warranty of free repair for all consoles made for customers in the U.S. and Canada. Now the machines would carry a one-year warranty, not just 90 days.
A drastic measure
Since it had an oversupply, Microsoft had a cushion. It could afford to suspend manufacturing as it sold off its inventory of consoles. The company stopped promoting the console heavily and shaved a million units off the goal for the fiscal year ended June 30, 2007. The problem was that Nintendo’s Wii was starting to catch fire. Nintendo had come up with a cheap and innovative console, which was proving extremely popular at $250, about $50 to $150 cheaper than Microsoft’s Xbox 360 models. During the six-month period ending June 30, 2007, Microsoft would sell only 1.2 million consoles. It was a disastrous slowdown, compared to the Wii’s sell-out success.
In January, 2007 at the Consumer Electronics Show, Bill Gates announced that the Xbox 360 would be capable of serving as an IPTV set-top box, meaning that telephone companies could use it to deliver video programming and other services that could compete with cable TV providers. Gates also said that because the Xbox 360 had beaten the PS 3 to market and didn’t include a hard drive on every machine, Microsoft would be able to ride a silicon cost-reduction curve faster than Sony, which saddled every unit with Blu-ray and hard drive costs. At any given point, the Xbox 360’s costs will be lower and more easily reduced, giving Microsoft a fundamental advantage over Sony in pricing, Gates said.
The Cassinghams stir discontent
They used several in their retail gaming center, where kids came in to play against each other. That prompted critics to blame the Cassinghams for over-using the machines. But several three of the machines that broke down were actually personal machines that the Cassinghams used in their own homes. Consumer outrage flared, and Moore again offered apologies to customers. One game magazine, 360 Gamer, conducted a poll that found many console owners reported more than one console failure.
Fixing bugs in the trenches
Microsoft quietly acknowledged that it was no longer using Wistron as a manufacturing partner. In April, 2007, Larry Yang, the general manager of Xbox division hardware, said in a group email that he was resigning that job and would take another position within Microsoft. He would still stay and help Microsoft get out its most important redesign, dubbed Falcon.
Falcon was the code name for the cost-reduced Xbox 360 that included a major redesign of its CPU. Scheduled for once every couple of years, the transition to new chips was a big deal. Microsoft’s partners had been making their chips with 90-nanometer technology since 2005. Now they would make the CPUs with 65-nanometer technology, meaning the widths between the circuits were smaller. With smaller circuits, the same chip design can fit on a smaller amount of space. With less space, the chip can be manufactured with less material, resulting in lower costs. Quality also gets better since it’s easier to make smaller chips than big chips. Shifting to a new generation of chips was thus the best way to reduce the cost of a game console and improve its quality.
An email sent out by S. Srini, director of Xbox manufacturing, on May 27, 2007, said that the overall yield of units in production was 85 percent. But the email said that the IBM microprocessors were still exhibiting “excessive failures” of 4 percent to 8 percent because of heat problems. There was still no “fresh start” manufacturing going on.
The hardware engineers worked mainly to address the overheating of the graphics chip from ATI. The chip was overheating and causing both the chip and the circuit board to warp, cracking the joints that were held together by the lead-free solder. Given enough time, many units will suffer this problem. To fix the problem, Microsoft put epoxy on both the IBM microprocessor and the ATI graphics chip. They also took out a 50 cent extruded aluminum heat sink (which sits on top of a chip and dissipates heat away from it) from the graphics chip and replaced it with a $5 heat sink that could do the job more efficiently. They also used a pipe to move heat in front of a fan.
Beyond the temporary solution of the heat sink change, Microsoft made more changes. Microsoft tried to get Falcon out as fast as it could. While the system box would remain the same size, there would be fewer components inside. That, in turn, would leave more room for air flow and to reduce the use of fans in the system, which meant it wouldn’t be as noisy as before. When rivals such as Sony made this kind of switch before, they had big shortages because it was always difficult to make the transition.
“Hopefully consumers will recognize we’re trying to do the right thing,” Moore said in an interview that day. “It’s a courageous step because it is not an inconsequential step. We are not burying our heads in the sand.”
The announcement put a dent in the cottage industry for repairing Xbox 360s (such as the book pictured at left). Even so, in early 2008, companies such as game retailer GameStop were making a fortune repairing thousands of Xbox 360s a week and then returning them as refurbished units to store shelves at a discounted price. Most of GameStop’s repairs were successful and involved re-soldering the graphics chip to the main board; unsuccessful repairs were passed on to Microsoft.
Peer Schneider, vice president of content publishing at gamer web site IGN, said, “The extended warranty and the reimbursement policies are important steps to win back consumer trust. The only question from Xbox fans is: what took Microsoft so long?”
There has been a lot of executive turnover in Microsoft’s game division, but none of it has been laid directly at the feet of the defect problem. Moore left Microsoft a year ago to become head of the sports label at Electronic Arts, while Don Mattrick, the former head of EA’s worldwide game studios, took the top games job under Bach at Microsoft. Bach said that Moore’s departure had nothing to do with the RROD problem, and Moore said the reason for his departure was that he and his family wanted to return to the San Francisco Bay Area. Bach remains one of the top executives at Microsoft, running the Entertainment & Devices group, (including the IPTV, Games for Windows, Xbox, and Zune businesses) which reported its first-ever profit in the fiscal year that ended June 30, 2008. Allard is chief technology officer of the E&D group, while Todd Holmdahl is head of games hardware.
Oddly enough, demand for Microsoft’s console continued, partly because gamers loved playing games so much that they kept on buying replacement consoles. By the fall of 2007, Microsoft had its Falcon-based machines ready. It was able to cut the price of the console, but not so much that it hurt its bottom line. The launch of “Halo 3” in the fall helped spur console sales, helping Microsoft stay ahead of Sony and at least stay competitive with the Wii in terms of game sales. In a development that was reminiscent of the “Dead Rising” failures, a number of Halo 3 buyers said the game appeared to fry their consoles.
Microsoft never announced how many machines had been returns. Based on an estimated $60 repair price, the company’s reserve can cover lots of machines. In early 2008, SquareTrade, which sells warranties for electronics, reported that there was a 16.4 percent failure rate for Xbox 360 systems. Michael Pachter, an analyst at Wedbush Morgan, estimated that the number is 3 million, or about 15 percent of the total. It isn’t clear if the numbers should count consoles that fail and aren’t repaired for free (only those with the three red lights are replaced for free); also retailers such as GameStop have begun repairing many of the units themselves. Bach said that Microsoft’s own research shows that fans continued to love the console and buy games, in spite of the problems with the defective units. Pachter believes that is true.
“In the ordinary course of something like this, you would expect it to show up in the customer reaction data,” Bach said. “We just haven’t seen that. It speaks to the fact that they love their games and Xbox Live. Does it frustrate them? Yes. On the other hand, they know we’re taking care of them. People have a certain amount of respect for that. If it had happened on a product that had less baseline customer satisfaction, it would have had a bigger impact. We really haven’t seen that.”
Was it worth it to go first, despite the concerns of the engineers and the inevitable consumer return problems?
“If you take the question of whether it was the right thing to try to be first, the answer to that is definitely yes,” Bach said in July 2008. “It has given us a leg up in a number of places that are super important. It has given us a leg up with game developers. It has given us a leg up from an economics perspective. It helped us expand Xbox Live quickly. At a strategy level, if you asked if we wanted to be first again, I would say yes. It’s easy for me to go back and say, if I knew what would transpire over the next two years, would I go back and do something different, I think that’s an obvious answer. But the fact is, based on the data we had at the time and all the hard work we put into it, there was no way to see what actually happened.”
The problem for Microsoft was that it sacrificed something to rush into the market and establish the Xbox 360 as the premiere platform of the living room. What it sacrificed was the good will of consumers, who are actually critical in terms of establishing a lasting platform. As for profits, Microsoft has finally turned a fiscal-year profit in the Entertainment & Devices group. It did so because it wrote off the costs of the defective units and warranty program in the previous fiscal year. (The $1.15 billion write-off covered several years of repair costs, some of which would have otherwise been incurred during the profitable fiscal year.) But Bach said that the business model for the Xbox 360 isn’t permanently ruined.
“It doesn’t make it good,” he said. “Would I like the billion dollars back? Of course, the answer is yes. But there is still plenty of economics in the business for us to be successful.”
Microsoft hasn’t been as aggressive at cutting prices this generation, but Sony has a more expensive box and so it hasn’t been putting price pressure on Microsoft. Had Sony been more aggressive, the consequences could have been much worse in terms of customer defections due to the RROD. Even with the $50 to $79 price cuts announced by Microsoft on Wednesday, it is taking a lot longer for prices to fall in this generation.
Jerry Knoben, a former Intel manufacturing chief, joined Microsoft in January this year as head of Xbox 360 manufacturing. He recently sent out a memo to the staff saying that the yield on Xbox 360 production had improved from the mid-80-percent-range to 92 percent. But he said that there were still some surprises in manufacturing that Microsoft had to fix.
By next year at the earliest, Microsoft is scheduled to launch a new iteration of the Xbox 360 with a board dubbed Valhalla. That board will have a single chip that combines both the graphics chip and microprocessor on a single device. This is the kind of technical tour de force that Sony achieved with the chips for the PlayStation 2, a move that allowed the company to significantly reduce manufacturing costs. Microsoft still has an advantage over the PS 3 in manufacturing costs, since it doesn’t have an expensive Blu-ray player.
Shane Kim, (pictured left) until recently head of Microsoft Game Studios and now head of the game division’s business development efforts, recently said that it was too early to call the console war in favor of Nintendo. We won’t know the winner, he says, until somebody sells 100 million units. Microsoft’s plan is to sell 75 million Xbox 360s. The only way they could ever do that is to be more aggressive on their pricing, which means driving down the costs of the console as low as possible. And by doing the right thing for their customers from now on.
Sony and Nintendo aren’t safe just yet. Bach has always talked about how Microsoft has a 10-year to 15-year plan for success. Maybe Xbox 3.0, just like Windows 3.1, will do the trick.
To conclude, the video game industry has never seen a consumer problem as bad as the “red rings of death” and the size of the $1.15 billion charge stands as one of the biggest liability glitches in consumer electronics history. How Microsoft handled the flaw may provide a lesson for all modern electronics companies; that is, if you are going to promote the hell out of something, it better work the way you say it does and you better have a strong customer support and engineering debugging team to back it up.
Amazon and eBay have their periodic outages. Sony had its problems with batteries, Intel had flawed Pentium chips that were bad at math, Nvidia took a $200 million charge to deal with graphics failures on laptops, and Apple has had its share of glitches on the iPhone. As electronics become and more complicated, it becomes a practical impossibility to test for every single flaw. But with the force of the Internet behind them, angry customers can exact a revenge that can be extremely damaging to a company’s brand image.
One thing is clear. Microsoft has to move beyond its mentality of being a software company that can launch fast and fix later. With global markets and global launches, the consequences of such a cavalier approach to hardware quality can start to pile up. The company clearly has the cash and the size to compete among the best hardware companies out there. It cannot afford to tarnish its brand when competitors on various fronts — such as Google on the web side and Apple, Sony, and Nintendo on the hardware side — are there to exploit its stumbles.
At this writing, it seems clear Microsoft’s hegemony in operating systems and productivity software won’t last forever. The company has limited time to get the rest of its house in order. It has to diversify into fields such as video games and make them pay off with profits. Microsoft has demonstrated an incredible tenacity in a lot of markets. It took almost a decade for the company to march to victory with its Office productivity software. I have no doubt the company can execute. It just has to decide that quality is something that matters.
If Bach is correct and Xbox 360 gamers are loyal, then Microsoft has been lucky. In the future, I suspect consumers won’t be as forgiving as fans such as Chris Szarek, who doesn’t own any other console and has bought three dozen Xbox 360 games.
Szarek said, “They got me. There was a time when I was furious with them.”
So why did he stay with the Xbox 360?
“What can I say?,” he said. “I like the games.”
If you liked this story, check out our interview with Spore creator Will Wright.