More often than not, companies have to play dirty in the race to the top. We have seen plenty of cases where a company was busted doing something really stupid just to get an edge over its competitor. Things like false advertisements, false specs by even established companies are not unheard of, but today let’s talk about something that Huawei was caught doing — benchmark cheating aka benchmark optimisation.

Huawei’s Benchmark Cheating Scandal

Huawei was recently caught by AnandTech optimising some of their smartphones like Huawei P20 to over perform in a specific benchmarking app, 3DMark. We have already talked about this here but the story in short is that Huawei P20 and a few other top flagships from the company were identifying benchmarking apps to extensively increase their performance with the primary goal of scoring way higher than in a traditional use case scenario.

In this age of AI, it’s not uncommon for smartphones to identify specific type of apps to boost performance. For example, new age phones can often detect gaming apps and switch to peak performance mode for the best user experience. However, what’s really the big issue here is that Huawei had tweaked its newer flagships to detect specific benchmarking apps (3DMark in this case) to run the phone at an unrealistic performance level. Apart from issues like overheating, such a level of performance decreases efficiency significantly, making the whole experience impractical for daily usage.

• RELATED: Our Honor Play Review Here!

Another issue with this type of performance optimisation is the motive. The big reason why companies like to optimise their smartphones for benchmark is to score higher than the competition. That’s not really a good enough reason to push your smartphones beyond its limits.

Our Obsession With Benchmark Scores

That being said, I can’t personally crucify Huawei for trying to score high on benchmarks. History has shown us that almost every manufacturer has been found tweaking their smartphones for better benchmark scores. This list includes the big names such as Samsung, LG, Asus, HTC, Meizu, and even OnePlus. So, the only reason why this was made such a big deal is because Huawei didn’t seem to get the memo that this was unacceptable, even in 2018.

Given the number of companies that have tried tweaking their scores for benchmark, the fundamental question is why they are doing this, even now.

The answer directly points to our needless obsession with benchmarks.

I get why benchmarks are so popular. It’s really hard to quantify the performance of a smartphone in any other way. Benchmarks give us a figure which we can compare to determine which one’s better than the other. It kind of makes sense. But then, it doesn’t. Because, getting this figure involves the app running a few resource intensive tasks to get a final score. The process isn’t fool proof and now you know that the results aren’t either.

We must start giving less importance to benchmarks and more importance to the phone’s actual user experience.

The Right Way To Evaluate A Phone’s Performance With A Benchmark App

Whilst a higher benchmark score in general doesn’t necessarily mean that the phone is better than others, users need to start understanding the scores in the right way.

Instead of obsessing over a particular score, we should start to evaluate a smartphone on the basis of levels of performance.

For example, take a very simple benchmarking app like AnTuTu. If a smartphone scores above a certain figure, for example 250K, its already established that the phone has a very powerful processor inside that can handle any task that you throw at it. Understanding that once a phone crosses a particular threshold, it doesn’t really matter whether the phone scores 290K or 261K on the app, is very important. What we need to realise is the simple fact that a score of 291K doesn’t necessarily mean that its better than a phone that scored 273K. In real life, the lower scoring phone could offer a much better experience than the other.

I know as enthusiasts, it’s really hard to ignore specific benchmark figures, but once we learn to stop this practice, we will start seeing the real picture. A SD835 smartphone with great optimisation is hundred times better than a SD845 phone with a bad optimisation. After all, would you be happy with a phone that scores 280K on AnTuTu but has a crappy UI or a phone which scores 240K on the same app, but handles everything with butter smoothness?

• RELATED: Top 10 UI with best performance released by Master Lu Benchmark: OnePlus wins

Let’s start giving less importance to individual benchmark figures and more importance to their real life performance and we would no longer have to worry companies optimising / cheating on these apps. After all, figures do lie!

I know, its easier said than done. It’s really tempting to see higher and better benchmark figures on flagship devices every year, but as consumers, if we start giving importance to the right things, in the long term the industry will benefit as a whole.

So the next time you see a company marketing a very high benchmark score for its flagship, just say — that’s pretty cool but how does it perform in real life?

The post Don’t You Judge A Smartphone By Its Benchmark, Be Better appeared first on Gizmochina.

What is Huawei Benchmark Controversy ?

We all are familiar with the benchmarks, it allows us to judge a devices performance. But results can be manipulated and various manufacturers have done this time and again to attain high scores. Benchmark manipulation has been in the industry for some time and many companies including OnePlus, Samsung and few other OEMs in the past have been found to tweak their benchmark scores for higher than normal results.

Couple of days ago, we got to know that Huawei devices were automatically shifting to the performance mode after detecting the benchmarking applications, which resulted in enhanced performance and better scores. Huawei and Honor devices such as P20, P20 Pro and Honor Play were using the “performance mode” to tweak the benchmarks. The results were attained by drawing more power from the Kirin 960 and Kirin 970 processors. Later in a report by UL, it is found that the power drawn by both the processors were more than sustainable levels for their architecture design.

Folks over at AnandTech exposed the whole scenario forcing Huawei to let users access the so called “performance mode”. Further, old devices such as Huawei Mate 9 and Huawei P10 are not affected. The devices prone to benchmark manipulation are primarily running on EMUI 8.0, the latest version of the company’s customised UI.

Affected Huawei Devices Delisted

Yesterday, after an uproar from various media houses and concerned authorities, UL delisted the affected Huawei and Honor devices from the benchmarks. Devices those were removed are Huawei P20 Pro, Nova 3, and Honor Play. UL stated that the devices were scoring 47% higher in performance on 3DMark app. It further added that “A device must run the benchmark as if it were any other application.”

Update: Official Huawei and UL joint Statement 

Huawei has just revealed a statement saying that benchmark throttling is executed using ethical ways and there was nothing wrong in it. They emphasized that the AI technology is optimising the performance of their devices by utilizing the CPU, GPU and NPU at right time. Here’s the official statement:

Huawei always prioritizes the user experience rather than pursuing high benchmark scores – especially since there isn’t a direct connection between smartphone benchmarks and user experiences. Huawei smartphones use advanced technologies such as AI to optimize the performance of hardware, including the CPU, GPU and NPU.

When someone launches a photography app or plays a graphically-intensive game, Huawei’s intelligent software creates a smooth and stable user experience by applying the full capabilities of the hardware, while simultaneously managing the device’s temperature and power efficiency. For applications that aren’t as power intensive like browsing the web, it will only allocate the resources necessary to deliver the performance that’s needed.

In normal benchmarking scenarios, once Huawei’s software recognizes a benchmarking application, it intelligently adapts to “Performance Mode” and delivers optimum performance. Huawei is planning to provide users with access to “Performance Mode” so they can use the maximum power of their device when they need to.

Huawei – as the industry leader – is willing to work with partners to find the best benchmarking standards that can accurately evaluate the user experience.

In short, Huawei will make the hidden performance mode available for all to end the Huawei Benchmark Controversy.

So, what do you think? Is it a big deal for you that companies manipulate their devices to perform best on benchmarking apps?

(Via)

The post Benchmark Controversy Update: Huawei provides open access to “performance mode” appeared first on Gizmochina.