FP8/FP4 tensors in Metal 4.1

[leman]

The documentation for new frameworks is not out yet, but I've been looking at headers, and the main change I see is support for more limited-precision formats. In particular, we have:

/// An 8-bit floating point format data type with 8 exponent bits.
MTLTensorDataTypeFloat8UE8M0 API_AVAILABLE(macos(27.0), ios(27.0)) = 145,

/// A 2-bit unsigned integer format data type.
MTLTensorDataTypeUInt2 API_AVAILABLE(macos(27.0), ios(27.0)) = 149,

/// A 2-bit signed integer format data type.
MTLTensorDataTypeInt2 API_AVAILABLE(macos(27.0), ios(27.0)) = 150,

/// An 8-bit floating point format data type with 5 exponent bits, 2 mantissa bits and 1 sign bit.
MTLTensorDataTypeFloat8E5M2 API_AVAILABLE(macos(27.0), ios(27.0)) = 141,

/// An 8-bit floating point format data type with 4 exponent bits, 3 mantissa bits and 1 sign bit.
MTLTensorDataTypeFloat8E4M3 API_AVAILABLE(macos(27.0), ios(27.0)) = 142,

/// A 4-bit floating point format data type with 2 exponent bits, 1 mantissa bit and 1 sign bit.
MTLTensorDataTypeFloat4E2M1 API_AVAILABLE(macos(27.0), ios(27.0)) = 148,

The API also mentions additional attachment points to store scale factors required for these formats. INT2 is particularly interesting since I don't think anybody really supports it currently? Is this an indication that M6 will support all these natively? Who knows? Who cares?

 

[Cmaier]

The documentation for new frameworks is not out yet, but I've been looking at headers, and the main change I see is support for more limited-precision formats. In particular, we have:



The API also mentions additional attachment points to store scale factors required for these formats. INT2 is particularly interesting since I don't think anybody really supports it currently? Is this an indication that M6 will support all these natively? Who knows? Who cares?

2 bit signed integer? So -1, 0, 1, and 2? For what?

 

[leman]

2 bit signed integer? So -1, 0, 1, and 2?

Both signed and signed.

For what?

AI!

I suppose it's to support extreme quantization? Not really my area of expertise. They could be using it as an index into a weight table, or maybe combining it with the scaling factor just like other extreme low-precision formats. Maybe some ML folks could chime in.

 

[Yoused]

2 bit signed integer? So -1, 0, 1, and 2? For what?

Most likely ternary state evaluation. The 2 value would probably reflect invalid/error.

 

[Nycturne]

AI!

I suppose it's to support extreme quantization?

This. I'm a little fuzzy, but one of the challenges is trying to deal with large models. IIRC, going to this level doesn't impact the quality of output of an LLM as much as someone might expect, and can save noticeable chunks of memory. And fitting a model into less memory means cheaper token costs.

 

[The Hardcard]

Is this software support or does Apple10 GPUs have hardware support for FP8/FP4? In neural accelerators?

 

[casperes1996]

2 bit signed integer? So -1, 0, 1, and 2? For what?

Correct me if I'm wrong, but shouldn't that be
-2, -1, 0, 1?

Evaluations in 2's complement:
00 => 0
01 => 1
10 => -2
11 => -1

Nice that there's only 2 bits making it super fast to write that out!

 

[Cmaier]

Correct me if I'm wrong, but shouldn't that be
-2, -1, 0, 1?

Evaluations in 2's complement:
00 => 0
01 => 1
10 => -2
11 => -1

Nice that there's only 2 bits making it super fast to write that out!

if it’s 2’s complement, then yes. But for 2 bits, you could easily encode it either way with no hardware penalty.

 

[Jimmyjames]

Is this saying the Neural Accelerators have fp4/8 support?

 

[leman]

Is this saying the Neural Accelerators have fp4/8 support?
View attachment 39182

I don’t know who this person is or what it is they are saying. To know whether m5 natively supports these one would need to test it.

 

[Jimmyjames]

I don’t know who this person is or what it is they are saying. To know whether m5 natively supports these one would need to test it.

They work for Apple. Originated Electron but now work on MLX and the Cuda backend I believe.

 

[casperes1996]

if it’s 2’s complement, then yes. But for 2 bits, you could easily encode it either way with no hardware penalty.

I mean sure, you can always flip the meaning of the sign I guess but is there any reason to do so and not follow the normal convention here?


I don't know all too much about the math inside neural networks - I didn't take any ML courses and don't have as deep insight into the subject as I do others, but I could see a usecase for the unsigned version of UInt2 as a selection index into an n-ary decision tree.
Cannot think of all that many uses for the signed version. I guess in a weight evaluation thing it can act as a multiplier; Negate a part of the decision graph with -1, nullify it with 0, keep it as is with 1 or double its meaning with 2. (or if using the standard 2's complement convention, just keep everything in the negative domain and *-1 at the end to get the equivalent positive)

 

[mr_roboto]

I mean sure, you can always flip the meaning of the sign I guess but is there any reason to do so and not follow the normal convention here?

I think @Cmaier means that when you decide that 2 bits is "enough" information for a value, and you're designing custom hardware around 2-bit number formats, you get to pick whatever mapping between bit patterns and numeric values you like. Arbitrary functions of 2 bits cost very few gates, so the hardware overhead of translating between the in-memory representation and the in-ALU values is so low that it can be disregarded.

So, if you decide that it's more important to represent +2 than it is to represent -2, you can do that. The in-memory representation won't be 2's complement any more, and the ALU will have to be (at least) 3 bits, but nobody will really care.

 

[leman]

I just did a very quick check in new Xcode, and these new formats can be combined with a E8M0 (8 bit exponent, no mantissa) scale for a block of 32 values. The API errors out when creating a tensor with any other value combinations.

 

[The Hardcard]

I just did a very quick check in new Xcode, and these new formats can be combined with a E8M0 (8 bit exponent, no mantissa) scale for a block of 32 values. The API errors out when creating a tensor with any other value combinations.

It looks exactly like Open Compute Platform’s mxfp8 and mxfp4.

 

[leman]

It looks exactly like Open Compute Platform’s mxfp8 and mxfp4.

Yes, except the 2-bit variants which are odd.

 

[Yoused]

Yes, except the 2-bit variants which are odd.

Bear in mind that IEEE 754 defines an implied in the highest-order bit of a non-zero exponent: a 2-bit mantissa is actually a 3-bit mantissa.

 

[leman]

Bear in mind that IEEE 754 defines an implied in the highest-order bit of a non-zero exponent: a 2-bit mantissa is actually a 3-bit mantissa.

How would that work in practice? IEEE 754 has special rules for zero and denormals but here you just have two bits with a shared exponent per block. Would be quite interesting to see how calculations with these numbers work.

 

[Yoused]

How would that work in practice? IEEE 754 has special rules for zero and denormals but here you just have two bits with a shared exponent per block. Would be quite interesting to see how calculations with these numbers work.

Denormals are just a zero exponent. All non-zero exponents have an implied 1 in the exponential position. For instance, the number 2 is an exponent of 0x81 with a mantissa of zero. So the two-bit mantissa is just the sub-exponential part of the number.

For unsigned two-bit numbers, they are just the two bits below the implicit 1, and the number is always positive, within its context. For signed two-bit numbers, the upper-order bit is just the sign bit of the FP value (which would be missing from the 8-bit exponent) and the lower-order bit is appended below the implicit 1.

These rules would also apply to larger mantissa field sets. A 4-bit signed mantissa would just append its low-order 3 bits to the implicit 1 and use its high-order bit for S. The 8-bit exponent is the standard for FP32 (real) so the implementation is really low-impact WRT existing FPU hardware, and its range is practically quite large.

 

[leman]

Denormals are just a zero exponent. All non-zero exponents have an implied 1 in the exponential position. For instance, the number 2 is an exponent of 0x81 with a mantissa of zero. So the two-bit mantissa is just the sub-exponential part of the number.

For unsigned two-bit numbers, they are just the two bits below the implicit 1, and the number is always positive, within its context. For signed two-bit numbers, the upper-order bit is just the sign bit of the FP value (which would be missing from the 8-bit exponent) and the lower-order bit is appended below the implicit 1.

These rules would also apply to larger mantissa field sets. A 4-bit signed mantissa would just append its low-order 3 bits to the implicit 1 and use its high-order bit for S. The 8-bit exponent is the standard for FP32 (real) so the implementation is really low-impact WRT existing FPU hardware, and its range is practically quite large.

The downside I see is that you can't have individual zeros within a block, since the 00 bit pattern would be interpreted as 1.0. And its not quite clear how signed stuff would work — does that leave you with a single mantissa bit, for effective ±1.0 and ±1.5? Using two's complement would give you -2, -1, 0, and 1 before scaling, which IMO might be more useful?