Qualcomm is cheating on the Snapdragon X Plus/Elite benchmarks given to OEMs and the press. SemiAccurate doesn’t use these words lightly but there is no denying what multiple sources are telling us.
Lets start out with some disclaimers, the most obvious being how much we are blurring out a lot of the details. We are doing this for several reasons, the main one is that we facilitated some questions between OEMs and Qualcomm engineers which may point to a source if we aren’t careful. Anonymity of our sources will always come first.
Second is the briefings or lack thereof. At the Hawaii conference last fall, Qualcomm basically dodged every technical question asked. Both SemiAccurate and many others were promised that before launch there would be deep technical briefings to answer these questions and more. Today is the launch. Qualcomm lied. At least they lied to us, we haven’t had the chance to talk to others wo got the same line but, well, we doubt they kept their word at all.
Then there were the actual ‘briefings’ for the X Pro SoC. To call them pathetic is giving them more than their due. The deck was 11 slides, three of which were empty/fluff, five ‘benchmark’ slides with woefully inadequate disclosure, and two infographic summary slides. The last was the slide below with the ‘deep technical’ stats, much of which we told you about last week. And more.
It is really this bad
The rest of the ‘disclosure’ for Snapdragon X Pro was a list of features that all fall under the guise of exactly what you would expect. The rest was filled with deep ‘details’ like the GPU capabilities of 3.8TFLOPS. Thats it. No specs, no capabilities, no nothing. It was truly pathetic. But wait there is more, or less really, with statements like it having AV1 encode and decode. Trivialities like frame rates and resolutions were seemingly not needed for such technical briefs. See what we mean by pathetic? Those 10 cores are arranged how again? That 42MB of cache is what level? Shall I go on about the bare minimum basics or do you get the point now?
SemiAccurate was planning to ask Qualcomm about their cheating on benchmarks at the promised briefing but, well, they lied to us and cut us out of the pathetic bits they did brief on. We honestly would have liked to know why they were cheating but we kind of think they will do their usual response to bad news and pretend it never happened like last time. If they actually do explain things we will of course update this article as we always do.
So what are they cheating on? The short version is that the numbers that they are showing to the press and are not achievable with
the settings they claim. Qualcomm is showing a different set of numbers to OEMs and these also are not achievable with the settings they claim. This information comes from two tier 1 OEMs and other sources. (Note to Qualcomm: No it wasn’t him, really, we knew long before last week) SemiAccurate is 100% confident in saying that some of the numbers Qualcomm was showing off can not be reproduced with the settings they claim. More on it in a bit but first a little background.
Last fall when Qualcomm was briefing on the Snapdragon X Elite SoC, their benchmarks and claims lacked the bare minimum disclosure to verify the numbers. We brought this up with Qualcomm and to their credit they took big steps to rectify the issue. They did the right thing for the right reasons, not perfect but vastly improved. Well done. Then for MWC they regressed badly. This time around they put up five benchmark slides with claims of vast superiority over Apple, AMD, and Intel CPUs but without the minimum disclosure needed to check those claims. Any guesses why?
The why is because if you knew what was being tested and could recreate the tests, you could get vastly lower numbers. In Hawaii last fall they showed benchmarks running and according to our sources those were relatively clean. That said given the woeful state of WART (Windows on ARM) they were far from final. The silicon was either final or very close to it, the software not so much. That said no journalists were allowed to run anything, test anything, or even check the settings, it was presented as a black box. Again Qualcomm said we would have ample time before the launch to get briefs, test, and all that. Once again they lied.
Step forward a little in time. After OEMs got initial samples and made something close to the final designs, SemiAccurate got reports of poor performance. By poor we mean far sub-50% of the numbers Qualcomm was telling them in the technical docs and presentations. Trying to help we told some Qualcomm engineers about the findings and asked if there were any known issues with the silicon that would cause this. They repeated the claim that the silicon was clean, something we still believe to be true, but the state of WART was horrific. We also believe this to be true. They asked us to pass back to the OEMs that likely culprits were cooling and having the right benchmark build that was ARM native. Both are fair comments and we passed that back. Then silence for a while.
Later, with more Snapdragon X Elite samples in the wild and many more revisions of WART, we got similar reports from OEMs and another Tier 1. Both reported numbers that were nowhere close to what Qualcomm promised. How not close? Above 50% this time but one used the term ‘Celeron’ to describe performance. The claims of better than Apple’s Rosetta 2 x86 emulation are clearly not real on what is probably the release hardware and software. Actually the silicon emulation may be better but everything else is unquestionably not.
Two major OEMs with serious engineering capabilities are strong evidence but not proof. Here is where we will blur things out more than we like, sorry about that. A while back we were digging on performance or lack thereof in preparation for the promised briefings. A deep source at Qualcomm told us that the benchmarks were cheats, told us how they were cooked, and told us that Qualcomm was well aware of it. This same technique meant the numbers looked far better than they could be on a non-trivial set of tests. Ironically some other benchmarks could have looked much better than those presented if Qualcomm adequately disclosed their testing details. Lose some, lose some, but no winning here.
We realize it is anecdotal but Qualcomm won’t let anyone do independent testing on these parts. Windows on ARM is still said to be in a painfully awful state, or perfect by Microsoft standards. We are weeks away from the official announcement on May 20th at Build and the laptops are going on sale in June. If you take build time, shipping, and all the rest into account, what software and hardware we have now is what you are going to see in the stores, or really really really close to it. There is no excuse for not letting independent sites test the hardware other than the obvious bits this story is about. Qualcomm wants to control the message and has a host of social media influencers primed and ready to go. Curious that. Or not.
In the end the problem is simple. You have a product with promised performance that is not achievable with the claimed settings. Because the claims can’t be independently verified, the influencers will parrot back how ‘good’ the product is based on something we know is not true. Between the initial publicity blitz and the time things go on sale, the cheating values will become ‘truth’ and few if any will question them months down the road. Cynical hardly begins to describe this mechanism. The most interesting part is whether OEMs will parrot back numbers they know they can’t replicate or release their own. Time will tell.
What more is there to say? Qualcomm has been making promises about disclosure and testing for about six months now. They have broken every one of those and at least to SemiAccurate, repeatedly lied. Because of this the digging we have been doing since before the Hawaii disclosure, some of which lead to this older article, has kept us focused on how these benchmark claims were cheating. While we don’t use this term lightly, we are absolutely sure that Qualcomm is doing exactly that with their numbers. Worse yet they know the reasons why the numbers are tainted but continue to present them to OEMs and the press. The Qualcomm Snapdragon X Elite/Pro SoCs still seem to be very good silicon, everything above that is the polar opposite.S|A
Charlie Demerjian
Latest posts by Charlie Demerjian (see all)
- Qualcomm Is Cheating On Their Snapdragon X Elite/Pro Benchmarks - Apr 24, 2024
- What is Qualcomm’s Purwa/X Pro SoC? - Apr 19, 2024
- Intel Announces their NXE: 5000 High NA EUV Tool - Apr 18, 2024
- AMD outs MI300 plans… sort of - Apr 11, 2024
- Qualcomm is planning a lot of Nuvia/X-Elite announcements - Mar 25, 2024
