Intel Binary Optimisation and Geekbench warning.

Jimmyjames

Jimmyjames

Elite Member
Joined
Jul 13, 2022
Posts
1,506
Geekbench just posted a tweet and a blog post warning people that scores from Intel cpus where Intel’s Binary Optimisation tool have been used, are not comparable. Only 6 applications are listed which use this tool, Geekbench being one.

This feels like Intel are acting a little shady. Is that unfair?

Geekbench 6 and Intel's Binary Optimization Tool - Geekbench Blog

www.geekbench.com www.geekbench.com

1774387713393.png
 
Certainly feels scammy to make an optimization tool that works on only 6 applications, 1 of which is the most important benchmark.
 
KingOfPain said:
It wouldn't be the first time that Intel cheated on benchmarks (I believe one of their compilers once recognized a benchmark and made optimizations that would not work in any other context) and it unfortunately won't be the last either.
Click to expand...

All companies selling components based on benchmark results cheat. Not a long time ago AMD would enable unsafe CPU optimizations when they detected that a game is running. This resulted in occasional crashes, but hey, games crash, right?
 
I'm just pretty curious what the "Binary Optimisation Package" even does. - The fact it only works on a super limited set of programs is suspicious. I'd be OK with it if it, for example, statically analysed all applications for specific op-code patterns and replaced them with patterns that are better on a specific CPU but functionally identical. Like recognising something was compiled for generic x86 but being on a chip that supported SSE/AVX a specific pattern could safely and consistently be vectorised or whatever
 
casperes1996 said:
I'm just pretty curious what the "Binary Optimisation Package" even does. - The fact it only works on a super limited set of programs is suspicious. I'd be OK with it if it, for example, statically analysed all applications for specific op-code patterns and replaced them with patterns that are better on a specific CPU but functionally identical. Like recognising something was compiled for generic x86 but being on a chip that supported SSE/AVX a specific pattern could safely and consistently be vectorised or whatever
Click to expand...
From what PCGamer says, it is not automatic. So think of it like how AMD/NVidia does graphics driver updates (optimizations) for specific games. Apparently they have GB6 listed as a proof of concept, it appears that they are mainly focused on using this for games. From what I can tell it isn't terribly useful if you are not CPU bound (so I guess those folks with high end cards running at 1080p low would be good candidates).
 
leman said:
All companies selling components based on benchmark results cheat. Not a long time ago AMD would enable unsafe CPU optimizations when they detected that a game is running. This resulted in occasional crashes, but hey, games crash, right?
Click to expand...
That is a lot less scammy than hardcoding optimizations that only apply to a benchmark though. Some people could genuinely want to do that, and have better FPS at the expense of an increased risk of crashes.
 
Odd that Intel optimizes GB and yet AMD pantses them in the averages. It looks like AMD still uses HT, and the Intel Ultras are looking not so great without it. But its value is dubious: a 32-core/64-thread Threadripper has a MC score less than 10% higher than an 18-core M5 Max (and still only 30% better than an 18-core Snapdragon).
 
Yoused said:
Odd that Intel optimizes GB and yet AMD pantses them in the averages. It looks like AMD still uses HT, and the Intel Ultras are looking not so great without it. But its value is dubious: a 32-core/64-thread Threadripper has a MC score less than 10% higher than an 18-core M5 Max (and still only 30% better than an 18-core Snapdragon).
Click to expand...

That’s expected given GB 6’s design. It’s great for giving most consumers a sense of what they’re buying, but if you’re considering buying a Threadripper (or an Ultra or any other super high core count CPU), then you are no longer the target audience for GB - and that’s according to the people making GB. It’s designed for consumer devices.
 
Follow up post.
TLDR: Intel not only checks the application, but checksums the version of Geekbench. 6.3 in this case. For matching apps and versions, it takes a full forty seconds to launch the app. Subsequent launches with this tool enabled take two seconds. We see improvements of ~5%-6% overall for both single and multi core tests. Some subtests see bigger increases. Up to 30%. When running the as yet unreleased Geekbench 6.7, BOT yields no increase in performance but delays launch by two seconds. Inspecting the tool shows it is vectorizing code which was meant to be run as scalar code.

Team Intel on Twitter taking this about as well as expected.
1774980586826.png


Analyzing Geekbench 6 under Intel's BOT - Geekbench Blog

www.geekbench.com www.geekbench.com
 
Jimmyjames said:
Inspecting the tool shows it is vectorizing code which was meant to be run as scalar code.
Click to expand...
If they can define a general pattern for automatically converting binary chunks safely into equivalent but vectorized versions of the same logic in the general case, why not just always do it?
 
casperes1996 said:
If they can define a general pattern for automatically converting binary chunks safely into equivalent but vectorized versions of the same logic in the general case, why not just always do it?
Click to expand...
Not an expert but I’d assume that they can’t? Perhaps they can but if so why choose Geekbench as one of the few?
 
Jimmyjames said:
Not an expert but I’d assume that they can’t? Perhaps they can but if so why choose Geekbench as one of the few?
Click to expand...
It seemed like a proof of concept usage. They are doing it in gaming workloads which geekbench doesn't track for anyways. Has Intel talked about doing this on Adobe apps?
 
diamond.g said:
It seemed like a proof of concept usage. They are doing it in gaming workloads which geekbench doesn't track for anyways. Has Intel talked about doing this on Adobe apps?
Click to expand...
It’s a proof of concept to mislead people on performance by choosing one of the most, if not the most popular benchmarks and having to essentially hand optimize scalar code into vector code?
 
Jimmyjames said:
It’s a proof of concept to mislead people on performance by choosing one of the most, if not the most popular benchmarks and having to essentially hand optimize scalar code into vector code?
Click to expand...
I guess we will see if Intel updates the list for the newest version of GB?

I mean if they drop GB from the list would this be a nothing burger, or would folks still be upset that they are making changes to how say Cyberpunk 2077 runs?
 
diamond.g said:
I guess we will see if Intel updates the list for the newest version of GB?

I mean if they drop GB from the list would this be a nothing burger, or would folks still be upset that they are making changes to how say Cyberpunk 2077 runs?
Click to expand...
I’m doubt it. It’s the fact that they chose a benchmark that’s concerning.

Having said that, they are changing code that devs chose to ship. What are the consequences of doing this? Are there stability or security issues that could arise?
 
Jimmyjames said:
I’m doubt it. It’s the fact that they chose a benchmark that’s concerning.

Having said that, they are changing code that devs chose to ship. What are the consequences of doing this? Are there stability or security issues that could arise?
Click to expand...
I guess I am willing to take them at their word that they used GB as a proof that there are gains to be had, more so than them trying to game the system. Games with anti cheat/anti tamper are safe from this tool. I guess it could make an app unstable.

Since they seemingly have to hand tune the changes this (BOTs) probably won't be supported for all that long. The real question is if they could bake this functionality into the platform drivers or talk Microsoft into making it part of the OS/kernel.
 
I guess what I am trying to point out is the folks here (or that run benchmarks on the regular) aren't the target audience for the tool. It is something they are trying to shill to gamers.
 
I dunno. They change scalar code which Geekbench chose to use as a test, into vector code which only Panther Lake can benefit from. It stinks of manipulation to me.
 
diamond.g said:
I guess what I am trying to point out is the folks here (or that run benchmarks on the regular) aren't the target audience for the tool. It is something they are trying to shill to gamers.
Click to expand...
But Geekbench 6.3 managed to attract their attention before 99% of games? A benchmark whose scores just so happens to garner attention in the press when new chips are announced.

Hmmm.
 
Back
Top