The Halo of Xbox 2


There were 50 games under way at Microsoft’s growing game studios, now at 1,200 people. But it wasn’t clear which ones should be created for the Xbox and which ones should become launch titles for the Xbox 360. Fries still needed better games. In February 2003, he went to an industry conference called D.I.C.E. in Las Vegas. By chance, he met with Cliff “Cliffyb” Bleszinski, who pitched Fries on making the “Halo of Xbox 2.” Fries replied, “I might give you that chance.” The game would later be called Gears of War.

One of Fries’ problems was that Halo 2 had run off schedule. It had to be redone and Bungie decided late in the process to chop the game in two, leaving the second half as Halo 3. Other problems forced Fries to lay off hundreds of developers, even as he was still responsible for making games for the PC, Xbox, and Xbox 360. The rest of the executive team pressured Fries to ship Halo 2 in the spring of 2004 so that Bungie could be ready for a 2005 launch of Halo 3. Fries refused and got them to relent.

But it cost him a lot of political capital. Bach asked him to submit his game contracts to approval from finance executive, Bryan Lee. Fries again refused, seeing that as a loss of the independence and authority that he had always had. It put a finance executive with no knowledge of game development in charge of Fries’ budget.

After some soul searching, Fries decided to resign in early 2004. He had a hand in creating 18 titles that sold more than million copies in his game career. But it was time to move on. He couldn’t get the broader authority he wanted to keep the games group independent. A. J. Redmer said of Fries’ departure, “You could call it a Greek tragedy.”

His replacement, Shane Kim, took a different approach. He cut back on the studios and organized the launch titles. Gears of War slipped out of the launch window to 2006. Halo 2 shipped in the fall of 2004, as Fries wanted, and became the best-selling game in Xbox history.

Kim lined up what he could for the Xbox 360 launch, but he didn’t have as much firepower this time. There was a risk that the Xbox 360 wouldn’t have a ton of great titles at launch. But fortunately, Microsoft now had so much credibility that the third party game publishers were more than willing to step into the breach. Among the most promising titles came from Infinity Ward, an Activision studio which was making Call of Duty 2, a spectacular first-person shooter World War II action game. Full told, Microsoft was now planning for 15 third-party titles at the launch and 40 before the end of the first holiday season.

IBM finishes up

Amazingly, IBM’s engineers finished Waternoose in 11 months, partly because Sony had commissioned work on the core years earlier. In fact, IBM finished both the Sony and Microsoft chips in September 2004. Sony’s game developers were at a huge disadvantage, as they couldn’t start game design because there was no similar chip on the market already.

But Microsoft’s game developers could start earlier on their game designs because its chip resembled the IBM PowerPC 970 processors that were being made for Apple. It was a weird day when a bunch of Macs showed up at Microsoft. But those machines enabled an army of developers to go to work with plenty of time to finish top-notch games in 2005.

While working primarily with IBM’s manufacturing, Microsoft exercised a right to have a second-source manufacturer with Chartered Semiconductor of Singapore. In January 2005, IBM got the first chips back and debugged them over the phone with Microsoft operating system expert Dinarte Morais, who was watching the Super Bowl at the time. IBM’s Dave Shippy (pictured right) got to play the first game on the Xbox 360 prototype hardware.

One day, as he stepped out of the elevator, Shippy realized he was in the elevator with representatives of Sony, Toshiba, Microsoft and IBM. A sign said, “Do not discuss confidential information in this area.”

After Sony’s chips arrived in February, 2005, the Sony team got to work on games. But delays in the PS 3’s other big component, the Blu-ray drive, pushed the PS 3 launch into 2006. Sony also had to enlist Nvidia to provide a graphics chip at the last minute, because Sony’s own engineers had failed to create one. By accident, Microsoft was going to go first in this round.

ATI’s rush to the finish line

ATI’s team was also hard at work. The company had to put solid walls in place, just as IBM had, because it was building a graphics chip for Nintendo as well as Microsoft. Microsoft’s Nick Baker and Masoud Foudeh kept in close touch with the progress. Their boss, Larry Yang, kept in touch with Greg Gibson and Todd Holmdahl. To meet the schedule, ATI had to finish its first prototype by the fall of 2004 to stay on the production schedule for the summer of 2005. Chips rarely work the first time, and debugging can take months.

Microsoft still entertained the idea of launching worldwide in three markets at once. To do that, it needed to have 3 million consoles on hand in the fall of 2005. No console maker had every launched with such a plentiful supply, though they always tried. By 2004, ATI had 180 engineers working on the chip, or about half the size of IBM’s team. IBM’s first prototype emerged from a factory in August, 2004, while ATI’s first prototype was ready in November, 2004. That was late enough to make everybody a little nervous.

IBM started cranking chips out of its factory in December, 2004. That enabled more Xenon development kits to be built for the game developers, who then had close-to-finished hardware to build games upon. Game publishers such as Brian Farrell, CEO of THQ, were stunned to learn that Microsoft was hitting its targets for getting developer tools out.

Meanwhile, six weeks went by and ATI’s graphics chip kept crashing while running code. Because it took 12 weeks to get a chip through a factory, the engineers had to wait and hold their breath to see if their hardware bug fixes would resolve the problem. A large part of the engineering team was now focused on bug hunting. IBM’s final Waternoose chip was ready by May. No such luck for ATI.

By February, 2005, ATI’s engineers figured out one of the major problems and submitted it for a new round of production tests. But the final bugs weren’t exorcised until July. Three months after that, working graphics chips started coming out of the factory. It was late. Microsoft had multiple box assembly factories in place, but the lateness meant that Microsoft would not get its supply of 3 million consoles to ship by November. It was going to be lucky to get half of that. And it would later learn the graphics chip would be the cause of one of the Xbox 360’s fatal flaws.

The first to get to 10 million wins

From the completion of the design to the launch of the console, a whirlwind of events happened. At E3 2004, Peter Moore committed Microsoft to the Halo 2 launch in the fall. Electronic Arts came back into the fold and agreed to make games for Xbox Live. Nintendo and Sony took the wraps off their designs for new portable game systems, but they had no consoles to show. By the spring of 2005, Sony’s game chief, Ken Kutaragi, was demoted. He was no longer in the running to be CEO.

In May 2005, Microsoft scored a marketing coup as MTV aired a special half-hour TV show to unveil the Xbox 360. Microsoft began showing the machine off to the press and Moore expressed his excitement about Microsoft being first. Usually, the first console in a generation to get to 10 million units sold wins, he said. And Microsoft planned to get to 10 million units before the holidays of 2006, when its rivals were launching. 

At E3 that year, Microsoft was alone in preparing to launch a new console. Sony said its PlayStation 3 would come in the spring of 2006 (it would later push it back further for the U.S.). Sony also tantalized the crowd with a beautiful demo of Killzone 2, a next-generation game which was amped up to look like game play but was really a pre-rendered animation. Nintendo, meanwhile, took the wraps off something it called The Revolution, coming in 2006.

Microsoft’s approach was much more mainstream than in the past, in recognition of the broader goals. Square pledged support for the Xbox 360.

“Here we are, at the end of the beginning,” Bach announced.

Just before the launch, Microsoft held major events in Amsterdam and at a giant aircraft hangar in Palmdale, Calif., in the Mojave Desert. Fans drove across the country to make it to the Zero Hour event, which culminated in the Nov. 22 launch of the Xbox 360. Ominously, some fans reported their consoles were overheating. But Call of Duty 2 turned out to be the smash hit for the season.

By early 2006, Microsoft announced it had sold 1.5 million units in the holiday season. It was short of its goal of 3 million, but it was selling just as well as the first Xbox had done.

The red rings of death

Sales for the Xbox 360 climbed throughout 2006, but a growing number of complaints were holding it back. Gamers like Chris Szarek went through four failed consoles, suffering problems known as the Red Rings of Death, or RROD, for the red lights shining from around the power button when the system crashed due to overheating.

By the summer of 2007, Microsoft said it would replace machines with the RROD for free. By the spring of 2008, there were at least a million or two other people like Szarek. For the Xbox team, it was its worst crisis.

Bach’s rush to market may have led to the disaster, though there was no exact connection between trying to get out early and the product’s high rate of malfunctions. The Japanese, ever wary of quality problems, steered clear of the Xbox 360. Once again, despite a heavy investment in Japanese games, Microsoft’s console landed on the Japanese market with a thud.

Microsoft knew it had flawed machines, but it did not delay its launch because it believed the quality problems would subside over time. With each new machine, the company figured it would ride the “learning curve,” continuously improving its production. Even though Microsoft’s leaders knew their quality wasn’t top notch, they did not ensure that resources were in place to handle returns and quickly debug bad consoles. There were plenty of warning signs, but the company chose to ignore them. The different parts of the business weren’t aligned. Microsoft had a culture where you shipped your software and fixed bugs later. But with hardware, that strategy was a recipe for consumer outrage.

In August, 2005, as Microsoft was gearing up production, an engineer raised a hand and said, “Stop. You have to shut down the line.” This wasn’t just a brief moment. The engineer spoke up repeatedly, as we reported in our Xbox 360 defects story.

That engineer, who asked not to be identified, had deep experience in manufacturing. When production results were really off kilter, stopping a line and tracing a problem back to its roots was the answer. But the higher-ranking engineers, managers and executives chose to risk going forward. There wasn’t a universal backlash from the engineering ranks, according to one engineering source.

Nobody listened to that engineer — who spoke on condition of anonymity — apparently because console launches are always hurried affairs. At that time, the defect rate was 68 percent, meaning that for every 100 machines made, 68 didn’t work.

It reminded me of the German war machine just before World War I, as chronicled by Barbara Tuchman in the classic history book, “The Guns of August.” The German generals were intent on keeping their trains on time, but the leaders overlooked their chances for stopping the war altogether. The Schlieffen plan called for them to strike first. Once the Russians and French mobilized, the Germans had to move into action. They marched off blindly into tragedy.

Microsoft’s effort to jointly create the graphics chip with ATI Technologies ran into a physics problem. They had a huge chip and it was mounted on a circuit board through a ball grid array package. That grid of tiny balls held the chip to the board, but it was a rigid structure. When the chip became hot and that heat was distributed unevenly, it caused uneven expansion. That put a lot of stress on the solder joints, which led to hairline cracks and failure when the graphics chip became unglued from the board.

“This should have never happened and it’s fairly basic chip engineering, but this was Microsoft’s first attempt at making a huge chip,” said Rob Wyatt, a former Xbox architect, looking back on the problem many years later.

Millions of machines were defective, suffering from the Red Ring of Death problem that we chronicled in a post on VentureBeat. There was a growing “bone pile” of failed machines in a repair center in Texas. Microsoft had to take a $1.05 billion to $1.15 billion write-off to pay for the cost of returns. The damage to Microsoft’s reputation was pretty severe, as one survey noted that 24 percent of the consoles failed after two years. The normal defect rates for consumer electronics products are about 2 percent.

For six months in the first part of 2007, Microsoft shut down Xbox 360 production in order to figure out the source of its quality problems. It was only when Microsoft shipped a redesigned version of the Xbox 360 that the quality problems ceased. In 2009, Michael Pachter, an analyst at Wedbush Morgan, estimated that the number of failed machines was 3 million, or about 15 percent of the total at the time.

Seamus Blackley himself went through three failed Xbox 360s.

“It’s hard to ship a new hardware device,” Blackley said. “People can say they released it too early. When you engineer something as complicated as that, it is really hard. I don’t know if it is true that extra time would have helped with that problem.”

Robbie Bach made the right decision in turning the tables on Sony by going first. He put Sony behind Microsoft, and it never caught up during the generation. But the early rush was only a 90 percent success, like in the film “A Bridge Too Far,” about the battle of Operation Market Garden in World War II, when a whole British airborne division was lost at the last bridge that the allies couldn’t take. While Microsoft had spent billions generating a great Xbox brand, the Red Rings of Death — while largely forgotten now — did more to damage that brand than anything else that happened to Microsoft in the console wars.

Microsoft could have captured more gamers during this product generation, yet the RROD problem held it back. It delayed Microsoft’s plan to aggressively slash prices and market the machine heavily, stopping it from reaching a broader mass market. It forced Microsoft to lower its ambitions to merely beating Sony. The Xbox 360′s defect problem will go down as one of the worst snafus in consumer electronics history.