Investigation confirms Apple Macbook Pros have Nvidia bad bump material

Bumpgate: We break out the electron microscope

Nvidia world iconEditors Note: From time to time, SemiAccurate will be republishing some older articles by its authors, some with additional commentary, updates and information.  We are mainly reprinting some of the oft referenced articles that originally appeared on the Inquirer. Some will have added content, but all will be re-edited from the originals as per contractual obligations. You may see some slight differences between the two versions.

This article has had some of the original links removed, and was published on Tuesday, December 9, 2008 at 11:20AM.

WHEN THE NEW Macbooks came out a few weeks ago Nvidia stated that the chips they provided to Apple did not contain the proverbial ‘bad bumps’. Unfortunately, an investigation lead by Charlie Demerjian proves that not to be the case.

Background:

If you recall, Nvidia has been in the spotlight all summer for failing chips due to bad materials and thermal stress. The end result is that bumps, the tiny balls of solder that hold a chip to the green printed circuit board it sits on, crack, and the computer it is in, dies. If you want the full technical analysis, read this article (Parts 2 and 3).

Nvidia took a $200 million charge over the problem in July, but they refuse to support their customers by saying which parts are defective, and which computers they were sold in. You can get some clue from message boards, with Dell, HP, and Apple being prominent victims. Nvidia says that the problem only affects notebooks, HP says otherwise. Nvidia assures manufacturers that their cards won’t have problems, manufacturers say otherwise.

It looks like a massive cover-up that keeps affected customers in the dark. Doing right by them would cost a lot of money, which says a lot about the reason for a cover up. Fixed parts with a new ‘material set’, basically new bumps and underfill, were phased into production starting in mid-summer, and the old, defective bumps are being sold off slowly along side the new.

The question of the season is whether or not the brand new Macbook was designed and sold with ‘bad bumps’. Nvidia told us directly that the chips were not using the ‘bad bumps’, and we took their word for it even though internal Nvidia sources were telling us that this was not the case.

One thing to keep in mind however is that these bumps are so small that you almost cannot see them with the naked eye. The bumps in question are about 100 micrometers in diameter, near the diameter of a human hair. To complicate things, they are permanently sandwiched between the chip die and the green fiberglass carrier, the bumps literally solder the two together. They are then covered with an epoxy-like material called underfill.

Nvidia could have shipped chips with bumps made of peanut butter and said that they were gold. As long as the chips function there is almost no way of knowing exactly what materials were used in their design and manufacture.

It is a pretty safe bet for Nvidia to call the parts “good” publicly, even Apple might not bother to check up on them. Again.

To say definitively what the bumps are made of, you would need to buy a Macbook off the shelf, disassemble it, de-solder the chips, saw them in half, encase them in lucite, and run them through a scanning electron microscope equipped with an X-ray microanalysis system like this.

That is exactly what we did.

The Science:

G96 die cut up

Half of an Nvidia G96/9600 fom a MacBook Pro

Yes, you read that right, a brand new 15″ Macbook Pro was purchased in California as soon as they went on sale. This was an off the shelf part, not a review sample, not a gift, but a normal model that hundreds of thousands of you bought. It was then secreted to a small lab of mad scientists who do not wish to be named fearing repercussions from Nvidia and Apple.

These well meaning mad scientists took it apart, de-soldered the parts, and cut the defenseless notebook into many pieces. With meticulous care, they then ran it through multi-million dollar tools that would tell them exactly what materials the bumps were made of. Exactly.

MacBook Pro Mobo

The three main chips on a MacBook Pro motherboard

The motherboard of the new 15″ Macbook Pro looks like this with the heat pipes removed. There are basically three chips on it, the Intel CPU on the bottom, the Nvidia MCP79 chipset, and the Nvidia G96 CPU. The MCP79 is marketed under the name 9400M, and the G96 is called a 9600M GT GPU, but we may refer to them as the 9400 and 9600 in this article.

The bumps have two possibilities, new and old, good and bad respectively. According to Nvidia documentation,

http://www.theinquirer.net/gb/inquirer/news/2008/08/28/nvidia-55nm-parts-bad

the ‘bad bumps’ consist of mostly lead, 95% lead (Pb) in fact, with the remainder being tin (5% Sn). That is why they are called high lead bumps. The newer ‘good bumps’ are called eutectic, and what they do differently is explained in great detail in the technical links at the top of this (****this or that article?  the one I’m reading or the one you just referenced?********) article. As far as composition goes, they are about 2/3 tin (63% Sn) and 1/3 lead (37% Pb).

Bump from MCP79 and analysis

On the right a closeup of the bump from an MCP79/9400, the left shows the materials analysis

Here is a closeup of the bump itself from an electron microscope. Note the even graining pattern, that is indicitive of eutectic bumps. When they cool, it grains evenly.

MCP79 Bump

Electron Micrograph of an MCP79/9400 bump

Take a closer look at a portion of the graph that contains the elements in question

MCP79 materials analysis

MCP79/9400 bump materials analysis

Even if you don’t have a degree in material science, you can see that there are two big clumps in the graph. The two pronged one on the right is tin, and there is notably more of it than there is lead, the spike on the left. This means the bumps on the MCP79/9400 are made of eutectic material (63% Sn, 37% Pb), and they are ‘good’. Nvidia’s story checks out so far.

Take a look at the same data for the 9600.

G96 die cut up

Nvidia 9600 bump closeup and materials analysis from a MacBook Pro

And again, a closeup of the bump.

G96 bump

9600 bump electron micrograph

And once again a closeup of the graph. (Please note that the original was black and white, we filled it in with red for clarity)

9600 bump materials analysis

9600 bump materials analysis

Even a communications major can tell that there is one big spike at lead (Pb) and a very small one at tin (Sn). This fits the profile of high lead (95% Pb, 5% Sn) and is radically different from the ‘good bumps’ of the 9400. The 9600 is unquestionably using ‘bad bumps’, directly contradicting the statements from Nvidia.

If you want more evidence, look at the surface of the bumps in the pictures above. Eutectic solder has a melting point that is the same for all components. When it cools, you should get an even physical structure with a fairly consistent grain. That is what can be seen on the 9400 picture. With non-eutectic solder, the component that cools first should clump, and that is what can be seen on the surface of the 9600 picture.

More Problems:

You will notice that we stated in the beginning of the article that Nvidia said the bumps in the chips were good, and you can see from the above data, this is definitely not the case. The computer that ‘donated’ it’s guts for the above analysis was a 15″ Macbook Pro, purchased off the shelf in California. There was nothing special about it, not a press sample, not even a pre-production version. This is what Nvidia said was good.

How did they say that? Below is the last mail in an email chain between the author and Mike Hara, Vice President of Investor Relations and Communications at Nvidia. Phone numbers and email addresses were removed, and only the formatting was slightly changed for readability. It was sent at 1:52PM CST on October 15, 2008, titled “RE: 9300/9400 materials sets”.

[Begin Email]

oops, sorry.

Michael Hara – NVIDIA Corporation – Vice President of Investor Relations – (408) xxx-xxxx – Fax: (408) xxx-xxxx

=======================================================

This email message is for the sole use of the intended recipient(s) and

may contain confidential and privileged information. Any unauthorized

review, use, disclosure or distribution is prohibited. If you are not

the intended recipient, please contact the sender by reply email and

destroy all copies of the original message.

—–Original Message—–

From: Charlie Demerjian [mailto:charlie@xxxxx]

Sent: Wednesday, October 15, 2008 11:44 AM

To: Michael W Hara

Subject: Re: 9300/9400 materials sets

I assume you mean 9400, not 6400.

Michael W Hara wrote:

> Charlie,

>

> The 9300/6400 and 9600 discrete all use the new material set.

>

>

> Michael Hara – NVIDIA Corporation – Vice President of Investor

Relations

> – (408) xxx-xxxx – Fax: (408) xxx-xxxx

>

> =======================================================

> This email message is for the sole use of the intended recipient(s) and

> may contain confidential and privileged information. Any unauthorized

> review, use, disclosure or distribution is prohibited. If you are not

> the intended recipient, please contact the sender by reply email and

> destroy all copies of the original message.

>

> —–Original Message—–

> From: Charlie Demerjian [mailto:charlie@xxxxx]

> Sent: Wednesday, October 15, 2008 9:27 AM

> To: Derek Perez; Michael W Hara

> Subject: 9300/9400 materials sets

>

> Guys,

> The obvious question of the day is, what materials set is used on

> the 9300/9400 and the discrete 9600 used in the macbooks?

>

> -Charlie

[End Email]

As you can see, the question was asked the day after the new Macbooks came out, and it is quite clear in naming all three potential parts, 9300, 9400, and discrete 9600. There is no nuance, and the then brand new Macbooks are directly named. As we have proven above, the statement in that email is simply not true.

One problem that journalists run into every so often is that PR people don’t always tell the truth. They usually do, but every once in a while, they don’t. Sometimes this is found out, and that inevitably leads to a very embarrassing story, sometimes it doesn’t. The only down side to this for the PR person is their getting caught with their corporate pants down, and the inevitable hit to their reputation.

The problem this time is that Mike Hara is not PR, he is IR, Investor Relations. His main job is to deal with stockholders and analysts, and is consequently under a very different set of rules when speaking to such people. People buying and selling stock expect, and are required to get honest answers to the questions that they ask.

An IR person, like a PR person, has three answers they can give, yes, no, or no comment, but the consequences for not telling the truth have legal teeth behind them. Talking to lawyers who specialize in corporate affairs, the legal boundaries for IR are the same if IR is speaking to analysts, stockholders, the milkman or the press. Unlike PR, if IR speaks on behalf of the company, they are obliged to tell the truth.

Analysis:

So, what does this all mean? It means that there are 15″ Macbook Pros being sold with ‘bad bumps’, the same materials that brought down so many HP, Dell and Apple parts, both laptop and desktop. For some odd reason, Nvidia really does not want you to know this.

The first and most obvious question is, does Apple know? Repeated calls to Apple PR were not returned prior to this story, and while that looks pretty damning, it isn’t. Apple will not talk to journalists unless they are assured the response will be fawning, and we do not fit that mold. That said, given the history between Apple and Nvidia, it could go either way.

Nvidia may try to explain it away by saying that the bumps did not need to be changed, or that the Macbooks are engineered to keep the thermals under the critical temperature for the bumps. These explanations all have some very serious problems.

The first problem is that if the bumps were not a problem at all, why were they changed on the 9400? The 9400 is a much cooler running chip than the 9600, so why change the part that is less likely to die? If the 9600 with ‘good bumps’ is being phased in, why bother with the qualification costs, time, and inventory hassles if it is not a problem?

The other problem comes down to heat. The new Macbooks run hot, very hot. The net is filled with reports of them overheating and hanging. This is most often seen when gaming, a task that stresses the GPU hard and results in a ‘black screen of death’. These beasties run extremely hot.

On the surface, the explanation about the Macbooks not getting hot enough to crack the bumps is quite laughable. If it is hot enough to sear the flesh off your thigh, it is likely more than able to reach an internal temperature of 60-80C, the point where the underfill softens. If the chips get hot enough to crash, it is unlikely they are running within prescribed thermal boundaries.

If you assume the engineering work was done, and done correctly to keep the 9400 and 9600 in the correct thermal range, why didn’t Nvidia simply say so? Barring a total failure of their lot tracking system, they had to have known the Macbooks shipped with ‘bad bumps’, so the explanation from NV of “not knowing” is unrealistic.

Instead of IR making a definitive statement, if they were not sure, they should have said so, then researched the question. Even a “no comment” would have sufficed, but instead, they gave a definitive and quite incorrect answer. Coincidentally, it was also the answer that reflected best on the company, preempting several embarrassing articles, and a likely drop in stock price.

If they knew, and it wouldn’t be a problem, Nvidia could simply have answered the questions I asked with a “Yes, they ship with the old material set, but the chassis has been re-engineered to keep the bumps from ever experiencing the stress that will crack them.” This would have been an entirely satisfactory answer, and preempted any bad press unless the Macbooks did die in the field.

That leaves one option, the bumps are bad, Nvidia knows it, and it will be a problem. Anything else comes with a simple explanation and no cover up needed. If there was any explanation other than the bumps being bad and it will be a problem, Nvidia would have put that forward in a very public manner.

The Future:

What do you do from here? The answer is simple, don’t buy a 15″ Macbook Pro. While there is no assurance that the high lead bumps will cause a failure, given their history, we can not recommend that you take the chance.

Apple and Nvidia need to clearly mark which machines have the ‘bad bumps’ so consumers can decide for themselves. Given that Nvidia claims to be transitioning from high lead to eutectic bumps, it is only a matter of time until the high lead inventory is depleted, and the Macbooks are safe to buy.

Until that time, you would be well advised to avoid these potentially problematic notebooks.S|A

Note: In as statement before publication, Nvidia’s Mike Hara had the following comment on the situation. “The GeForce 9600 GPU in the MacBook Pro does not have bad bumps. The material set (combination of underfill and bump) that is being used is similar to the material set that has been shipped in 100’s of millions of chipsets by the world’s largest semiconductor company.”

Author’s Update: Two years later, a few things come to mind. First is how badly Nvidia wanted to avoid this issue. If you notice Hara’s response, he goes out of his way to avoid saying that the bumps are still high lead. He said, “The 9300/6400[sic] and 9600 discrete all use the new material set.” which is not true.  Material set usually refers to the combination of bump material and underfill material ie high lead bumps plus Hitachi underfill is one kind of material set, another would be Eutectic bumps with Hitachi underfill for another kind of material set. 

Interpreting this you can read it different ways.  The ‘new material set’ could be anything form a new box that the materials came in to actually new materials. He is playing a game in saying that changing the less relevant component of the two, the underfill, and specifically not changing the material that fails, is a new material set. In some ways it is, but since the bumps in question aren’t changed, it is at the very least disingenuous. Very disingenuous.

If you are playing semantic games as Nvidia so loves to do, even when it is not in their best interest to do so, thing become funnier. “The 9300/6400 and 9600 discrete all use the new material set.” Note that “new material set” is singular, and it refers to both the bumps and underfill. As you can see from the graphs above, while they both MAY use the same underfill, they most definitely do not use the same bumps.

Next up are a few pictures from the original investigation that I never published. Nothing horribly revealing, just some background info.

Macbook mobo chips desoldered

MacBook Pro with chips desoldered

A lot of people said that we really didn’t get a MacBook Pro and really desolder the chips. The conspiracy theory was that we made it all up. The mobo with the chips removed is above, and below is one of the chips encased in lucite. That is basically what you do to prep it for electron microscopy.

Chip in Lucite

One of the chips in Lucite

You can clearly see the layers of the package here, and some of the vias. You can also see a few bumps, and the underfill. It is a nice shot, but not horribly revealing.

Xray pics of the materials

XRay pics of the materials

Here is a nice one of the materials used in the 9600/G96. You can see by the false color pics that the tin is clumped in large chunks, and the majority of the bump is lead. If you compare it to the black and white 9600 bump picture above, you can see how visibly different high lead bumps are from eutectic.S|A

The following two tabs change content below.

Charlie Demerjian

Roving engine of chaos and snide remarks at SemiAccurate
Charlie Demerjian is the founder of Stone Arch Networking Services and SemiAccurate.com. SemiAccurate.com is a technology news site; addressing hardware design, software selection, customization, securing and maintenance, with over one million views per month. He is a technologist and analyst specializing in semiconductors, system and network architecture. As head writer of SemiAccurate.com, he regularly advises writers, analysts, and industry executives on technical matters and long lead industry trends. Charlie is also available through Guidepoint and Mosaic. FullyAccurate