Apple M6 rumors/discussion

[NotEntirelyConfused]

Do some non-cursory reading on the event mechanism. It is brutally confusing exactly how it works.[...]

Yeah, no thanks.

Those details are not necessary for my point. The feature simply has to exist and, more or less, work.

 

[NotEntirelyConfused]

BTW, if you haven't read up on what Apple's been doing with dexts to let them run in user space, try to find a decent summary. It's pretty amazing what they've already done. Your app can do I/O without ever talking to the kernel and without ever hitting a context switch!

 

[mr_roboto]

Hm. Thinking about this more... I just realized something significant. This model means that ONLY the OS cluster is writing the TLB! That means that all invalidations are generated only on that cluster, and can be pushed cheaply to all other clusters. This drastically simplifies cache design, and reduces latency.

So in fact I think this design would find cache sync much cheaper than typical designs.

I don't think that follows at all.

There's two things to keep in mind. One is that a TLB is a specialized cache which stores the results of page table walks. The other is that every processor needs to be able to generate cache invalidations (and, more generally, fully participate in cache coherency). If it couldn't, it could not safely write memory.

It sounds like you're assuming there are special instructions for writing TLB entries, and TLB writes generate specialized TLB-only coherency traffic. But that isn't how this works. (*) Instead, the update process is simply a matter of using ordinary CPU store instructions to modify page table entries.

Because they're caches, TLBs use cache tags and match them against coherency broadcasts. When a TLB sees invalidations hit any of its tags, it invalidates and frees all TLB entries that matched. The next attempt to access any affected pages will cause the TLB to perform a full page table walk, then cache the result in a new TLB entry.

* - well, not in sane CPUs anyways. There have been ISAs where TLBs were managed by software; these did have privileged mode instructions for manipulating the TLB. Whenever a TLB miss occurred, the CPU would fire an exception and the OS handler would manually perform the page table walk and generate a TLB entry for it. This proved to be a terrible idea, as software managed caches almost always are.

 

[NotEntirelyConfused]

I don't think that follows at all.

There's two things to keep in mind. One is that a TLB is a specialized cache which stores the results of page table walks. The other is that every processor needs to be able to generate cache invalidations (and, more generally, fully participate in cache coherency). If it couldn't, it could not safely write memory.

I don't understand this claim. You don't need to be able to modify the TLB unless you are paging, freeing memory, or doing a few other things that are all OS tasks (page promotion to huge pages, defragging, etc.). If the OS never runs on a core cluster, that cluster will never need to invalidate TLB entries. I think I remember that Apple did some funky thing to make JITs able to flip a page state from writable to executable without involving the OS; if so that should continue to work fine with this design.

It sounds like you're assuming there are special instructions for writing TLB entries, and TLB writes generate specialized TLB-only coherency traffic. But that isn't how this works. (*) Instead, the update process is simply a matter of using ordinary CPU store instructions to modify page table entries.

Because they're caches, TLBs use cache tags and match them against coherency broadcasts. When a TLB sees invalidations hit any of its tags, it invalidates and frees all TLB entries that matched. The next attempt to access any affected pages will cause the TLB to perform a full page table walk, then cache the result in a new TLB entry.

I am not assuming that. Maybe I explained this badly.

I referred to two cases, TLB misses and real faults. The first case is handled by the hw page walker, which is essentially the same thing that happens today, as you described. For faults, an IPI (except not really, see below) is generated, and an OS core does the necessary thing before sending a message back indicating completion. In the meantime, OoO execution can continue to the extent that that page isn't needed by everything yet.

There are a pile of other cases that can require page table changes, but they all are significant long-running events, compared to which the O(100ns) delay going from the user core to the OS core is insignificant.

Note that a few of those needs diminish or go away entirely in this architecture. For example, KPTI is a non-issue, because user cores will never have kernel pages in their cache. So all those invalidations don't have to happen.

Now, about IPIs for faults: This seems like another great place to replace interrupts with polling. I imagine building some specialized hardware inside the NoC. It will support things like CoW and zero-fill intrinsically without ever touching the OS cores. But for major faults, when you have to go to the OS, there's a LOT of things you can do.

My first thought was make a simple ring buffer into which all faults go. All OS cores draw from that buffer whenever they're idle. The standard NoC backpressure mechanisms prevent fault storms, and you can also have forward pressure where a filling ring causes OS cores to give faults somewhat higher priority. (Except, the buffer isn't totally simple, you need multi-reader/updater capability... but that's a solved problem.)

Of course, that's very naive. You can do better in terms of locality by instead having a queue per OS core, and affinities from user to OS cores. But to distribute the work better, a trivial hash of the address space ID would better distribute the work. And adding in some upper bits from the page address would help distribute a single process's faults across multiple OS cores without losing most locality benefits. Ideally you'd want a hysteresis mechanism to switch between those two schemes depending on the fault load from an individual address space ID.

You know what else you get almost for free? Batching/coalescing. Since you get a queue of faults that can be inspected by the OS core (potentially aided by some specialty hardware that can sort queued faults by address?), it's much much easier to handle a bunch of (near-)contiguous-space faults all in one operation, AND pass them as one IO to the disk, than it is with current OSes where you mostly have to speculate about locality and IO sizes.

Well, that was several rabbit holes I hadn't planned on going down. It's been interesting!

 

[leman]

I think this mostly doesn't need to happen... though I'm waving my hands quite furiously here. The theory is that most data-heavy calls talk to the OS core for control but it in turn passes pointers to a userspace process which is running back on the same user CPU for actual data movement. Thus my reference to a split driver and zero-copy. My vague idea was to have the NoC have design support for syscalls, such that a limited amount of arguments can be passed directly over the NoC rather than being referred to as memory addresses which are in the cache. It seems that this exists already, give or take (inter-core fifos for registers).

That was also what I was thinking about it — the control bus just sends a skeleton message and the bulk of the data is passed via in-memory buffers. But here precisely is why we need to care about synchronization — the client will be writing these buffers to their caches, and the OS cores will be writing the buffers to theirs. You need to ensure that these are coherent.

First sentence, exactly right. Second, I don't think so. Perhaps every syscall is in fact a sleep instruction that indicates "wait for a message from the OS core with a new address to start executing from" (I mentioned this earlier; it would also specify which register types would be restored, and from where). Sometimes, the response from the OS means "resume now without context switch" and it does instead of yielding. And of course some (probably more and more over time, if this idea is successful) return immediately without sleeping, as they're totally async. This is entirely implemented in hw, there's no OS overseeing it.

Sure, that would be the most straightforward approach. But it's also potentially wasteful. This way, the client core is parked until the OS server gets to processing the request. Now, you can of course split things — like have the thread scheduling done on the client core, and other operations done on the OS core, but then you still need privileged execution on the client and so the main argument (hard memory isolation) for this approach is getting weaker. Resuming execution from a given address would require privilege escalation anyway, since you'd need to restore state from a different thread/process.

For a TLB miss, that's the potential performance problem, going across the NoC to the OS Cores. But... why would you have to do that? The HW page walker exists in every core. Let it do its thing, and return, while holding the thread suspended (or not even, as it can execute out of order).

For a real fault, the latency between clusters, on the order of 100ns, will be entirely hidden by the latency to disk which is more than two orders of magnitude larger. (Or, for a minor fault, by the latency of the page copy/zero-fill/whatever.)

Page traps are used to implement things like copy on write, so this has to be very fast. I'd wager that handing off copying memory pages to a different cluster would be a massive performance disaster.

Yes, balance seems like a question you can only answer with massive telemetry. Fortunately, Apple should have that, so they can probably know how many cores you'd need for this to work well (and thus whether it would be cost-effective). However there's some literature saying 5-10% of compute resources are typically sufficient, assuming all the heavy driver lifting is done in userspace. On the high end of that, or maybe a little bit more, for some IO-heavy tasks like RDBMSes. That would make 2 cores plenty for even the M5 max.

The nice thing about the classical approach is that it's auto-balancing. On codes that uses few sys calls, your CPU cores spend more time executing the client code, and on codes that requires heavy OS involvement, the larger proportion of the CPU resources is spend running kernel code. I wonder whether statically partitioning kernel resources might lead to the situation where everything works well — until it doesn't, because some specialized software executes more syscalls than the system designer has anticipated. Of course, this is purely speculative, I have no idea whether any software would do this.

 

[mr_roboto]

I don't understand this claim. You don't need to be able to modify the TLB unless you are paging, freeing memory, or doing a few other things that are all OS tasks (page promotion to huge pages, defragging, etc.). If the OS never runs on a core cluster, that cluster will never need to invalidate TLB entries.

But that cluster's TLBs will still need to self-invalidate their own entries.

In arm64, TLBs are automatic, hardware-managed caches. Cores don't provide a special path for operating systems to directly modify TLB entries. The OS simply writes to memory addresses that happen to be page table entries, and like any other write, this generates cache coherency broadcasts in the NoC to notify interested parties (caches) that the contents of some memory have changed. Any caches (TLBs) which have one or more entries that cache the modified PTE will wake up and invalidate them.

That's why I don't see any way to get complexity reduction in application-only clusters out of this, let alone the significant optimizations you're hoping for. The hardware used to modify TLB entries is the memory store path. Can't get rid of that! And on the other end, the app cluster TLBs still need to be able to monitor coherency traffic to catch PTE updates made by OS clusters.

Note that a few of those needs diminish or go away entirely in this architecture. For example, KPTI is a non-issue, because user cores will never have kernel pages in their cache. So all those invalidations don't have to happen.

But once again, there are no hardware optimizations to be found here. The mechanism responsible for broadcasting both kernel and user page table TLB changes is just cache coherency, which all cores unconditionally require.

Not sure I want to go down the other rabbit holes, but I will say that so far I don't think there's much merit in those ideas either. (Sorry.)

 

[NotEntirelyConfused]

That was also what I was thinking about it — the control bus just sends a skeleton message and the bulk of the data is passed via in-memory buffers. But here precisely is why we need to care about synchronization — the client will be writing these buffers to their caches, and the OS cores will be writing the buffers to theirs. You need to ensure that these are coherent.

There are no buffers for many syscalls. Everything goes over the NoC. For large data passing (IOs, generally) you use zero-copy or zero-kernel calls to the driver, and everything stays on the core so there's no coherency issue in the first place. There certainly will be some calls that have arguments that are memory pointers, and then your issue is real. But I think it's not that much of the overall syscall load.

Sure, that would be the most straightforward approach. But it's also potentially wasteful. This way, the client core is parked until the OS server gets to processing the request. Now, you can of course split things — like have the thread scheduling done on the client core, and other operations done on the OS core, but then you still need privileged execution on the client and so the main argument (hard memory isolation) for this approach is getting weaker. Resuming execution from a given address would require privilege escalation anyway, since you'd need to restore state from a different thread/process.

The client core isn't parked because it does OoO. Syscalls aren't by nature an execution barrier. And you really don't want to do a context switch for a syscall that might take a fraction of a microsecond.

I posited hardware on the core that can, upon receipt of notification over the NoC from an OS core, do a task switch. With that in place, you do not need any EL1+ execution on the user core.

Page traps are used to implement things like copy on write, so this has to be very fast. I'd wager that handing off copying memory pages to a different cluster would be a massive performance disaster.

I specifically mentioned that you can implement zero-fills and CoW at the NoC layer. I think I read somewhere that Apple already has a zero-fill engine? This is more of the same.

The nice thing about the classical approach is that it's auto-balancing. On codes that uses few sys calls, your CPU cores spend more time executing the client code, and on codes that requires heavy OS involvement, the larger proportion of the CPU resources is spend running kernel code. I wonder whether statically partitioning kernel resources might lead to the situation where everything works well — until it doesn't, because some specialized software executes more syscalls than the system designer has anticipated. Of course, this is purely speculative, I have no idea whether any software would do this.

Yes, I agree this is a reasonable concern- it was one of the first things I thought about. My somewhat lazy research suggests that this won't be an issue, in that no real-world code needs so much OS interaction (again, assuming the split driver model). If someone knows this is wrong, tell me.

 

[NotEntirelyConfused]

There are no buffers for many syscalls. Everything goes over the NoC. For large data passing (IOs, generally) you use zero-copy or zero-kernel calls to the driver, and everything stays on the core so there's no coherency issue in the first place. There certainly will be some calls that have arguments that are memory pointers, and then your issue is real. But I think it's not that much of the overall syscall load.

Some extremely lazy research suggests that typically, about 70% of syscall use involves memory (as opposed to registers). But of that, 90% is driver calls that are irrelevant (due to zero-copy, and general driver latency). So maybe 7% of calls are actually a potential problem.

For many of those, the arguments can go directly over the NoC. For the rest, they get pushed to the SLC, so at least the os cores won't trigger cache snooping to get them.

 

[NotEntirelyConfused]

But that cluster's TLBs will still need to self-invalidate their own entries.

In arm64, TLBs are automatic, hardware-managed caches. Cores don't provide a special path for operating systems to directly modify TLB entries. The OS simply writes to memory addresses that happen to be page table entries, and like any other write, this generates cache coherency broadcasts in the NoC to notify interested parties (caches) that the contents of some memory have changed. Any caches (TLBs) which have one or more entries that cache the modified PTE will wake up and invalidate them.

That's why I don't see any way to get complexity reduction in application-only clusters out of this, let alone the significant optimizations you're hoping for. The hardware used to modify TLB entries is the memory store path. Can't get rid of that! And on the other end, the app cluster TLBs still need to be able to monitor coherency traffic to catch PTE updates made by OS clusters.

It's true that local invalidations will be generated. But won't all those apply to pages that are only mapped by that core anyway? In that case, those invalidations don't need to be seen by the other clusters, and my argument holds.

I can think of a few sources of invalidations. Speculation mispredictions, drivers, context switches that reuse ASIDs... they're all purely local.

There is one case I didn't consider, and ... it actually really sucks, and it makes me realize I missed something important. If you're running traditional OS VMs, they will generate invalidations, and I'm not sure all of those will be local to that cluster. I'm also not sure they're not. But what this also reminds me is that the big win in eliminating EL1+ support from user cores means they won't be able to run traditional OSes in VMs. You'd have to give up that win to run Linux. Of course, you could still run a virtualized MacOS, since it would know how to live in this environment, but that's not enough for many applications. (It would be enough for typical Mac users who aren't devs, though. So this might seem like an acceptable trade for Apple. I would hate it though.)

And on the other end, the app cluster TLBs still need to be able to monitor coherency traffic to catch PTE updates made by OS clusters.

There's only one source of PT updates, the OS cluster. Handling that should be a lot more streamlined.

[About KPTI] But once again, there are no hardware optimizations to be found here. The mechanism responsible for broadcasting both kernel and user page table TLB changes is just cache coherency, which all cores unconditionally require.

That wasn't about a hardware optimization. I was saying that you can eliminate all that code from the OS. That's a performance win.

I will say that so far I don't think there's much merit in those ideas either. (Sorry.)

No need to be sorry, I'm here to learn. But so far aside from the VM issue I just thought of (very sad, but it's not a deal-breaker), I don't see any huge problems with the idea. Unless I'm still missing something in the TLB discussion.

 

[NotEntirelyConfused]

I saw earlier that @mr_roboto had posted a response to my latest message but now I can't see it. :-(

I don't recall everything in it, but there were two things I remember, so I'll answer briefly (and more later if the post reappears).

1) About removing KPTI code from the kernel: Yes, I'm pretty sure you can delete it, and no, I'm not expecting the hardware to do the same job. The whole point of KPTI is to protect the kernel against various speculation attacks. If the kernel and userspace never execute on the same cores, no kernel page will ever be present in the user cores' TLBs. The page tables themselves should never be accessible to userspace, and only the hw table walker will be reading them. Side channels should be completely avoidable.

This is a really gigantic win for I/O and should compensate for some performance loss in other areas, if necessary.

2) Skepticism of my claim that invalidations would never have to be broadcast from a userspace cluster outwards.

Um... that may be justified. My idea works, I think, up until you need a single address space to run across multiple userspace clusters. At that point, I'm not sure it's doable any more. Maybe you could add an aging timer to each entry, and just let them expire naturally, but then you have to randomize the timer to block a new side-channel, and you're trading against extra memory pressure. I'm not at all sure that that is worth doing.

Notably, I think my original scheme would still work in a smaller chip where there's just one OS cluster and one userspace cluster (not realistic) or if Apple decided that on the base Mx chips, no address space could exist simultaneously on the different core clusters. I don't know if that's plausible or not, in terms of performance tradeoffs, and I don't know if the benefit of doing that just for base M chips is worth it. But, hm, all the A chips would presumably use it too. Maybe?

 

[dada_dave]

I saw earlier that @mr_roboto had posted a response to my latest message but now I can't see it. :-(

I don't recall everything in it, but there were two things I remember, so I'll answer briefly (and more later if the post reappears).

1) About removing KPTI code from the kernel: Yes, I'm pretty sure you can delete it, and no, I'm not expecting the hardware to do the same job. The whole point of KPTI is to protect the kernel against various speculation attacks. If the kernel and userspace never execute on the same cores, no kernel page will ever be present in the user cores' TLBs. The page tables themselves should never be accessible to userspace, and only the hw table walker will be reading them. Side channels should be completely avoidable.

This is a really gigantic win for I/O and should compensate for some performance loss in other areas, if necessary.

2) Skepticism of my claim that invalidations would never have to be broadcast from a userspace cluster outwards.

Um... that may be justified. My idea works, I think, up until you need a single address space to run across multiple userspace clusters. At that point, I'm not sure it's doable any more. Maybe you could add an aging timer to each entry, and just let them expire naturally, but then you have to randomize the timer to block a new side-channel, and you're trading against extra memory pressure. I'm not at all sure that that is worth doing.

Notably, I think my original scheme would still work in a smaller chip where there's just one OS cluster and one userspace cluster (not realistic) or if Apple decided that on the base Mx chips, no address space could exist simultaneously on the different core clusters. I don't know if that's plausible or not, in terms of performance tradeoffs, and I don't know if the benefit of doing that just for base M chips is worth it. But, hm, all the A chips would presumably use it too. Maybe?

https://techboards.net/threads/down-for-a-while.5723/#post-201360

Unfortunately we lost a few posts