Apple SSDs and “Special Sauce”.

[Jimmyjames]

Since the launch of the Mac Studio and more recently the M4 Mac Mini, there has been interest in replacing the supplied ssds with more cost effective replacements. Both for the purpose of capacity increases and as a way to replace faulty drives. As of late there have been a few people advertising the fact that they will be able to supply compatible, user-serviceable ssds.

This week, the ATP (Accidental Tech Podcast) discussed the feasibility of these replacements. As part of this discussion, someone seemingly “in the know” pointed out how difficult an endeavour this will be. I was unaware, or had forgotten that in 2011, Apple had purchased an Israeli company called Anobit. Their products and technology provided a very large increase in nand reliability through their technology called a “Memory Signal Processor”. For example, they were able to increase the number of program/erase (write cycles) by a factor of 10-20. So instead of using expensive SLC nand, a drive manufacturer could use MLC nand, and achieve the write cycles of SLC and. Typically SLC nand gets upwards of 50000 p/e cycles, MLC gets 3000-10000. This is obviously appealing to manufacturers. Allowing them to use cheaper MLC nand, and get SLC reliability. Now as part of Apple, these people continued to make and innovate with these Memory Signal Processors (MSP). It’s not clear for how long, but certainly for a number of years, these MSPs have been part of Apple supplied ssds.

The point of the discussion on the podcast was to say that making third party compatible ssds that work with modern Macs is going to be very difficult. Makers will need to source these MSPs, which Apple doesn’t sell. Therefore, these MSPs will need to sourced from dodgy supply chain leaks, or old Macs. It’s hard to believe enough of these chips will be available to quench a large amount of customers, so beware if you are thinking of buying one!

To me the more interesting part is the tech itself. I had always thought Apple used standard SSDs and overcharged for them. Given this increase in endurance enabled through the MSP, it seems it’s not so simple.

If anyone is interested in knowing more about Anobit, the wayback machine still has some pages saved, including some pdfs. I’d be interested in thoughts. Here is the saved page.

https://web.archive.org/web/20110904032612/http://www.anobit.com/default.asp?PageID=3

 

[cbum]

This MSP sounds like a stunning tech advance - how come no one has been discussing this in the last 13 years?

 

[Nycturne]

To me the more interesting part is the tech itself. I had always thought Apple used standard SSDs and overcharged for them. Given this increase in endurance enabled through the MSP, it seems it’s not so simple.

The question I am left with is where does the MSP live? Is this something that needs to live on the SSD boards, or is it embedded in the SoC like the rest of the controller?

Looking at the SSD board from the M4 Mini, there’s only 4 chips on the board, 3 of them straight-forward to identify. The last one is a custom Apple chip, so it isn’t impossible that it was misidentified, but it seems like this claim from ATP needs some evidence to back it up. https://www.ifixit.com/Guide/Mac+Mini+(2024)+Chip+ID/178986#s381190

 

[Jimmyjames]

The question I am left with is where does the MSP live? Is this something that needs to live on the SSD boards, or is it embedded in the SoC like the rest of the controller?

Looking at the SSD board from the M4 Mini, there’s only 4 chips on the board, 3 of them straight-forward to identify. The last one is a custom Apple chip, so it isn’t impossible that it was misidentified, but it seems like this claim from ATP needs some evidence to back it up. https://www.ifixit.com/Guide/Mac+Mini+(2024)+Chip+ID/178986#s381190

Iirc, it’s not part of the soc, but part of the ssd board.

Edit. This is the quote from the podcast.

“Apple SSDs utilize multi-chip packages, each of which contains a single MSP or memory signal processing die, along with 2-16 NAND flash memory dies.”

 

[Jimmyjames]

This MSP sounds like a stunning tech advance - how come no one has been discussing this in the last 13 years?

Yeah it’s a good question. I don’t know the answer. I think partly the solution is that manufacturers have found other ways to increase reliability. Sandforce was known for introducing a proprietary compression method which reduced writes and therefore increased reliability. The problem here is that not all data is compressible. Others over-provision their nand. This costs more. Some do both I think. I’m sure they all work but Anobit’s tech seems the most interesting to me.

 

[mr_roboto]

I've read up about Anobit MSP before and have always been a little skeptical. It was a startup looking for funding and customers; you expect startups to promote themselves as energetically as they can.

There's a frustrating lack of specificty to their public claims about MSP. They try very hard to imply lots of things without ever providing enough detail to actually understand. Nor is it clear why nobody else could replicate the basic idea (applying DSP to get more out of a noisy communications channel was not a new concept). Their MSP-ish patents don't provide much clarity either. Lots of them discuss things that seem impossible to do outside of the NAND flash die, meaning the only way Anobit could have gotten value out of those patents was licensing them to companies which actually built flash. (Maybe early on, they had dreams of owning their own flash fabs? I dunno.)

This is not to say Anobit wasn't a good acquisition for Apple. They'd built SSD controller IP too, and I think had previously licensed it to Apple, and it was Apple's strategy to try to fully own as much of their chip tech stack as they could.

Getting back to SSD upgrades, whether or not these proprietary Apple chips do MSP (and how important MSP might be) doesn't really matter. The main thing is that the chips do exist, and Apple's SoCs have no way of talking to flash memory without them - the chips bridge between NAND flash interfaces (ONFI, Toggle) and PCIe.

It's known that there are signed firmware images for these bridge ICs, and early on in the system's bootup, one step in bringing the SSD online is to download the appropriate firmware image to the bridge chips and get them running. Apple's use of signed firmware blobs is likely an insurmountable problem for anyone looking to make a compatible bridge chip.

 

[mr_roboto]

The question I am left with is where does the MSP live? Is this something that needs to live on the SSD boards, or is it embedded in the SoC like the rest of the controller?

Looking at the SSD board from the M4 Mini, there’s only 4 chips on the board, 3 of them straight-forward to identify. The last one is a custom Apple chip, so it isn’t impossible that it was misidentified, but it seems like this claim from ATP needs some evidence to back it up. https://www.ifixit.com/Guide/Mac+Mini+(2024)+Chip+ID/178986#s381190

The custom Apple chip in those photos is buried in the SanDisk SDMVGKLK2 device. High denstiy NAND and DRAM contain multiple die per package. Apple gets their flash suppliers to build proprietary Apple-only packages with several standard flash die and an Apple interface die all integrated into one package.

 

[Jimmyjames]

I've read up about Anobit MSP before and have always been a little skeptical. It was a startup looking for funding and customers; you expect startups to promote themselves as energetically as they can.

There's a frustrating lack of specificty to their public claims about MSP. They try very hard to imply lots of things without ever providing enough detail to actually understand. Nor is it clear why nobody else could replicate the basic idea (applying DSP to get more out of a noisy communications channel was not a new concept). Their MSP-ish patents don't provide much clarity either. Lots of them discuss things that seem impossible to do outside of the NAND flash die, meaning the only way Anobit could have gotten value out of those patents was licensing them to companies which actually built flash. (Maybe early on, they had dreams of owning their own flash fabs? I dunno.)

This is not to say Anobit wasn't a good acquisition for Apple. They'd built SSD controller IP too, and I think had previously licensed it to Apple, and it was Apple's strategy to try to fully own as much of their chip tech stack as they could.

Getting back to SSD upgrades, whether or not these proprietary Apple chips do MSP (and how important MSP might be) doesn't really matter. The main thing is that the chips do exist, and Apple's SoCs have no way of talking to flash memory without them - the chips bridge between NAND flash interfaces (ONFI, Toggle) and PCIe.

It's known that there are signed firmware images for these bridge ICs, and early on in the system's bootup, one step in bringing the SSD online is to download the appropriate firmware image to the bridge chips and get them running. Apple's use of signed firmware blobs is likely an insurmountable problem for anyone looking to make a compatible bridge chip.

Yeah it’d be very interesting to know if claims of higher p/e cycles hold up in practice. I would assume Apple would find out before buying them, but perhaps there is other stuff going on as you say. It seems unlikely we’ll find out any time soon given Apple’s secrecy.

 

[dada_dave]

For the majority of Apple devices where the SSD is soldered onto the motherboard, this chip must be present there, yes?

Also, for the Studio modules that Polysoft is creating, they seem to be able to do it and sell at lower prices than Apple. They’re able to do that while also needing to harvest chips from dead Apple parts?

I’m a little confused.

 

[Nycturne]

Iirc, it’s not part of the soc, but part of the ssd board.

Edit. This is the quote from the podcast.

“Apple SSDs utilize multi-chip packages, each of which contains a single MSP or memory signal processing die, along with 2-16 NAND flash memory dies.”

Doesn’t really provide the evidence. Because for this to work, they are seemingly saying that Apple is having SanDisk make custom packages under a SanDisk part number? The board itself is a couple of NAND packages, a power management chip, and a buck converter. There's not a lot of space to hypothesize here.

It doesn’t really add up unless they come forward with more details that can be corroborated, IMO.

Yeah it’d be very interesting to know if claims of higher p/e cycles hold up in practice. I would assume Apple would find out before buying them, but perhaps there is other stuff going on as you say. It seems unlikely we’ll find out any time soon given Apple’s secrecy.

The idea itself doesn't surprise me since this is something that SSD manufacturers would want to chase. Maybe not Anobit's specific implementation, but anything to improve wear behaviors of MLC is a competitive advantage for someone who makes the NAND (or controllers). SLC is costly. It's more that ATP made a specific claim that I don't see corroborating evidence for at this point in time.

Getting back to SSD upgrades, whether or not these proprietary Apple chips do MSP (and how important MSP might be) doesn't really matter. The main thing is that the chips do exist, and Apple's SoCs have no way of talking to flash memory without them - the chips bridge between NAND flash interfaces (ONFI, Toggle) and PCIe.

It's known that there are signed firmware images for these bridge ICs, and early on in the system's bootup, one step in bringing the SSD online is to download the appropriate firmware image to the bridge chips and get them running. Apple's use of signed firmware blobs is likely an insurmountable problem for anyone looking to make a compatible bridge chip.

A couple key things:

1) The bridge "IC" is baked into the SoC die. It's the SSD controller.
2) The bridge is between the Apple Fabric and the NAND interface, not PCIe. It just looks a lot like a PCIe NVMe device in the hardware registers (Asahi Linux has pointed out a lot of this already in their reverse engineering work).
3) For modern Mac/iOS device security, the SSD controller lives on the SoC, to avoid sending keys from the Secure Enclave / AES subsystem across external traces. So you will not see this sort of translation happening on these subassemblies.

That isn't to say that you can just "make an SSD board" for Apple hardware, without potentially duplicating Apple's power chip or otherwise replacing its functionality. Now, that's not hard, just tedious, but I wouldn't be surprised if the economics just doesn't make sense.

 

[mr_roboto]

1) The bridge "IC" is baked into the SoC die. It's the SSD controller.
2) The bridge is between the Apple Fabric and the NAND interface, not PCIe. It just looks a lot like a PCIe NVMe device in the hardware registers (Asahi Linux has pointed out a lot of this already in their reverse engineering work).

There is PCIe involved here, it's just not in the place you'd expect. I made this block diagram for a post at the other place about a year ago to contrast Apple's SSD architecture with a conventional M.2 SSD. (BTW, I learned about this from Hector Martin's (Asahi lead dev) social media posts.)

Why'd Apple do it this way? My best guess is that at a minimum, the proprietary bridge chips serve as pin expanders. You need a lot of ONFI or Toggle channels to make a fast SSD, and that chews up a lot of pins since ONFI and Toggle are parallel interfaces at relatively low clock rates. PCIe is narrow at high clock rates, so you get the same I/O throughput from a lot fewer pins.

This isn't an issue for single-function SSD controller ASICs since they get to dedicate the entire pincount of the package to SSD functions, but probably is an issue for Apple since M series chips are big, complicated SoCs with a lot of non-SSD related I/O.

 

[Nycturne]

There is PCIe involved here, it's just not in the place you'd expect. I made this block diagram for a post at the other place about a year ago to contrast Apple's SSD architecture with a conventional M.2 SSD. (BTW, I learned about this from Hector Martin's (Asahi lead dev) social media posts.)

<snip>

This isn't an issue for single-function SSD controller ASICs since they get to dedicate the entire pincount of the package to SSD functions, but probably is an issue for Apple since M series chips are big, complicated SoCs with a lot of non-SSD related I/O.

So are you saying that SanDisk is making custom modules for Apple here?

 

[mr_roboto]

So are you saying that SanDisk is making custom modules for Apple here?

Yes.

Their NAND fab doesn't have to make anything custom, though. Should be the same flash die that they can put in a different package (sans Apple's bridge die) and sell to anyone else. That's important - memory fabs hate the idea of customer-specific wafers.

 

[dada_dave]

Yes.

Their NAND fab doesn't have to make anything custom, though. Should be the same flash die that they can put in a different package (sans Apple's bridge die) and sell to anyone else. That's important - memory fabs hate the idea of customer-specific wafers.

So where do these bridge chips live on the devices without SSD modules (where raw NANDs are soldered to the board and people can add blank NANDs to the motherboard), the motherboard I presume? And that raises the question how are Polysoft doing what they are doing?

 

[mr_roboto]

So where do these bridge chips live on the devices without SSD modules (where raw NANDs are soldered to the board and people can add blank NANDs to the motherboard), the motherboard I presume? And that raises the question how are Polysoft doing what they are doing?

The 'raw NANDs' on motherboards are not actually raw, at least not in the sense I think you're using the word. Regardless of whether the NAND is soldered to a Mac's motherboard or a carrier card, it's always a BGA package containing one bridge die and one or more NAND die. (Modern chip packages can be tricky; both NAND and DRAM are multiple die per package more often than not, even when it doesn't look like it from the outside.)

These BGA packages can be swapped between any type of device in Apple's product line - iPhones, Mac Studio/Mac Pro modules, Mac motherboards, whatever. There are some restrictions, but they're only that the BGA pinout must be compatible (Apple has used a couple different BGA package sizes and pinouts), it must be a NAND+SoC pairing Apple has used in their own products, and a NAND installed in channel 0 must be properly "blanked" for use in channel 0. (Not truly blank, but preformatted. Some implementation detail that nobody has reverse engineered the reason for.)

As for sourcing, hobbyists like dosdude1 usually buy NAND from vendors on Ali Express. Those companies don't seem to be big operations and are probably harvesting it from broken Apple devices.

I haven't seen anything about how Polysoft is getting its supply, but I suspect that whatever it is, it won't scale. More approachable and appealing to the general public than DIY hot air solder rework, but they might not be able to build enough modules to meet public demand. The only options I know of are the recycling supply chain, just like the hobbyists, or gray market product that 'fell off the back of the truck'.

 

[dada_dave]

The 'raw NANDs' on motherboards are not actually raw, at least not in the sense I think you're using the word. Regardless of whether the NAND is soldered to a Mac's motherboard or a carrier card, it's always a BGA package containing one bridge die and one or more NAND die. (Modern chip packages can be tricky; both NAND and DRAM are multiple die per package more often than not, even when it doesn't look like it from the outside.)

These BGA packages can be swapped between any type of device in Apple's product line - iPhones, Mac Studio/Mac Pro modules, Mac motherboards, whatever. There are some restrictions, but they're only that the BGA pinout must be compatible (Apple has used a couple different BGA package sizes and pinouts), it must be a NAND+SoC pairing Apple has used in their own products, and a NAND installed in channel 0 must be properly "blanked" for use in channel 0. (Not truly blank, but preformatted. Some implementation detail that nobody has reverse engineered the reason for.)

As for sourcing, hobbyists like dosdude1 usually buy NAND from vendors on Ali Express. Those companies don't seem to be big operations and are probably harvesting it from broken Apple devices.

I haven't seen anything about how Polysoft is getting its supply, but I suspect that whatever it is, it won't scale. More approachable and appealing to the general public than DIY hot air solder rework, but they might not be able to build enough modules to meet public demand. The only options I know of are the recycling supply chain, just like the hobbyists, or gray market product that 'fell off the back of the truck'.

I’m still confused on one point, those who do this sort of work insist that the NANDs must not have stuff written on them in order to work. So surely these SSD modules can’t be coming from recycled devices unless there is a way to “blank” them not available to most people? “Off the back of the truck” I can believe. I thought originally these were standard off the shelf NAND packages with these standard designations, just that they aren’t normally for sale to consumers (ie B2B only). Forgive me as I’m still wrapping my head around the additional complexity of this.

Apple Silicon Soldered SSD Upgrade Thread

once a nand has been written upon, there is no going back to blank....and in that case the nand is most likely unusable unless you have the exact dump files needed for that specific chip model n installation config. I only have the dump files for KICM223 Nands. (which is a BGA110 chip-1tb x2...

forums.macrumors.com

This user here talks about dump files, are those the same firmware you’re referring to? But he seems to be saying that’s only necessary to worry about if you’ve accidentally written something to the NAND and then if you use the dump file, you’re then restricted to using it in a particular slot. Otherwise with a “blank” you aren’t.

 

[mr_roboto]

I’m still confused on one point, those who do this sort of work insist that the NANDs must not have stuff written on them in order to work. So surely these SSD modules can’t be coming from recycled devices unless there is a way to “blank” them not available to most people?

Standalone programmers exist; in one of dosdude1's videos he uses one to write a blank dump image to a NAND. Here's one that supports Apple NAND packages as far back as those used in the iPhone 4.

P15 Integrated Nand Programmer For BGA60 BGA70 BGA110 BGA315 IOS Nand-JCID_JCID

“Off the back of the truck” I can believe. I thought originally these were standard off the shelf NAND packages with these standard designations, just that they aren’t normally for sale to consumers (ie B2B only).

Standard NAND isn't always limited to B2B. Direct-to-consumer companies like DigiKey can sell it, if they like. Here's an example 8 terabit Micron part:

https://www.digikey.com/en/products/detail/micron-technology-inc/MT29F8T08EWLEEM5-R-E-TR/17632018

Admittedly, this one's only technically not B2B. While anyone can order stuff through DigiKey, not everything they list has the option to buy qty 1. This is such a case; you have to buy a whole reel of 1500 parts. Their price per unit isn't very competitive, so the reel will cost you $343K before tax. Oof.

Apple Silicon Soldered SSD Upgrade Thread

once a nand has been written upon, there is no going back to blank....and in that case the nand is most likely unusable unless you have the exact dump files needed for that specific chip model n installation config. I only have the dump files for KICM223 Nands. (which is a BGA110 chip-1tb x2...

forums.macrumors.com

This user here talks about dump files, are those the same firmware you’re referring to? But he seems to be saying that’s only necessary to worry about if you’ve accidentally written something to the NAND and then if you use the dump file, you’re then restricted to using it in a particular slot. Otherwise with a “blank” you aren’t.

My interpretation of that and other posts (mostly dosdude1's, I tend to ignore the rest) is that "blank" is not truly blank, it's a pattern that's required to be there. If it isn't, Apple Configurator cannot work with the SSD. Furthermore, this pattern may vary by which SoC PCIe lane the BGA is connected to.

The dump files are images of NAND devices in this "blank" state made with one of those standalone programmers (they're R/W devices). Some of dosdude1's posts have links to dump files he's collected.

Something you have to keep in mind is that we're in the place where the full capacity of each page (the NAND equivalent of a sector) is visible. NAND pages are deliberately larger than a simple power of 2; the expectation is that the flash translation layer (the FTL is usually entirely contained in the SSD controller) will utilize the power-of-2 sized portion of a page to store user data, and the remaining capacity for error correction codes and FTL data structures.

The FTL has to do a lot of complex stuff behind the scenes to make flash look like a simple block device to the host computer. Imagine a scenario where a program is frequently modifying one or two bytes in a file (something which happens all the time). If you're using a FS where this rewrites that logical block address in-place, and a naive FTL with a fixed mapping between LBAs and NAND page numbers, you'll burn out that NAND page very quickly! TLC flash commonly has a write/erase limit of only about 1000 cycles.

So instead, the FTL maintains an any-to-any mapping between LBAs and physical locations in flash, and it's constantly rewriting this map to spread write wear evenly throughout the media. Writes are never performed in-place; they always (as a side effect) migrate that LBA to a different physical location.

That means the FTL has to store and maintain complex data structures which exist outside of your user data, and they have to be coherent for the SSD's FTL to work. Apple seems to have some level of initial FTL state they need to be present before their low level formatting tools (such as Configurator) can make sense of the SSD.

I wouldn't get too hung up on trying to understand all the details. We basically can't know them, short of someone doing a comprehensive reverse engineering of Apple's SSD architecture. It's enough to know that this kind of thing makes sense in a SSD; there is a lot of complex stuff lurking under that simple block device abstraction of "read a LBA, get the same data that was last written to that LBA".