Exclaves, XNU and the future.

[Jimmyjames]

Continuing my series of very niche interests, I was recently considering if anything significant has changed with regard to XNU.

For anyone who doesn’t know, XNU is the name of the kernel in pretty much all of Apple’s products. It’s comprised of the Mach microkernel and quite a bit of FreeBSD’s code. Unlike Mach, XNU isn’t run as a microkernel, but as a more traditional monolithic kernel with loadable kernel modules. Pretty much the standard for mainstream os’s these days. That has its good points and bad points. They are too numerous to go into here, and indeed, my neophyte understanding may be unaware of them, but essentially I would say the current situation gives pretty good performance, all essential low level services are within the kernel, bypassing the need for slower message passing that early iterations of Mach incurred. The main downsides are complexity: the kernel is a very thorny place to venture into, and any small error can wreak havoc. Secondly there is the issue of security. That is, once you are in the kernel, you can pretty much do anything. This can lead to big security and stability problems.

Over time, Apple has locked down kernel access even on macOS to developers. Kexts are increasing difficult or impossible to use. Drivers must be developed in userland using various frameworks Apple has developed. Even useful utilities like Dtrace (an instrumentation and debugging tool) have been restricted. I began to wonder if this restriction was in service of something more ambitious. Not just control, but the ability to slowly migrate from XNU to something more secure, efficient and stable.

There were rumours about an academic project called L4, a very efficient and small microkernel. Indeed, the Secure Enclave uses L4 at its heart. Nothing seemed to happen for while though.

As luck would have it, I also recently saw an article https://www.df-f.com/blog/ios17 which goes into some new features of iOS 17. These are low level features for Apple themself, rather than devs. They are designed primarily to increase security. One thing stood out in the article however:

This is in line with the few mentions of "exclaves" spotted elsewhere in strings. It seems Apple's new design is to transition away from the PPL to this new, micro-kernel like architecture, in which XNU's security functionality is refactored and isolated into exclave domains. Those are kept physically separate from XNU proper, so that even a hypothetical kernel compromise would be unable to further jeopardize the integrity of the other exclave components.

Very interesting! Could this be a new direction? These “exclave domains” which allow partial compromise of a kernel without all parts being effected. It’s an intriguing and exciting idea to me. Could they spread to other kernel areas?

I’d be interested in any thoughts anyone might have on this, even though I do realise this is not a mainstream area of interest.

 

[Cmaier]

Could be. Very typical of Apple to make these low-level advancements without anyone noticing

 

[Yoused]

I have been reading about microkernels for many years. L4 was started about a quarter century ago and evolved into a stable version called seL4 that gets used for stuff. Unlike Mach, L4 just hands messages between domains without examining them, and leaves it to the recipient to decide if it is within privilege bounds. Also, L4 leaves page table management and swapping and many other kernel-level functions to userland processes, where they are less likely to cause kernel panics/system failure. This places seL4 on the edge between microkernel and exokernel (a design where privileged code is at the absolute minimum). The GNU OS project is trying to create this kind of modular system but are using GNUMach instead of L4 due to porting difficulties.

 

[Nycturne]

Over time, Apple has locked down kernel access even on macOS to developers. Kexts are increasing difficult or impossible to use. Drivers must be developed in userland using various frameworks Apple has developed. Even useful utilities like Dtrace (an instrumentation and debugging tool) have been restricted. I began to wonder if this restriction was in service of something more ambitious. Not just control, but the ability to slowly migrate from XNU to something more secure, efficient and stable.

The driver situation isn't terribly surprising, as I don't think the idea of kicking drivers into userland is a new one. It's just been with the more recent security environment a more urgent issue. Windows is even pushing drivers into user space, just a little less forcefully than Apple (as they tend to do with all transitions).

I would be surprised if they fully migrated away from XNU, but we could see changes large enough that the architecture would be a shock to anyone who's been asleep for the last 10-15 years.

Very interesting! Could this be a new direction? These “exclave domains” which allow partial compromise of a kernel without all parts being effected. It’s an intriguing and exciting idea to me. Could they spread to other kernel areas?

Certainly looks that way. Interesting that PPL was only introduced in A12 and already being retired in favor of this new GXF model. But I think the interesting bit is that some of the first components moved into exclaves are things that allow XNU to be effectively "demoted" in some ways. It's isolating core pieces of the security model away from the kernel, which seems to be a good place to start.

 

[Yoused]

The driver situation isn't terribly surprising, as I don't think the idea of kicking drivers into userland is a new one.

There used to be a reason for having drivers in privileged space. They often handled the hardware aspects of device communication, or critical timing-related protocols. There is enough standardization these days that drivers really only handle high-level communication, such as can be done in userland, with no impact on performance. If you design hardware that uses unique protocols that must be done at the supervisory level, I should expect it would be a specialized thing requiring special support from the OS vendor, not a broadly accessible consumer-level product.

 

[Jimmyjames]

Interesting...

News about seL4 and the seL4 Foundation | seL4

sel4.systems

 

[Yoused]

According to the wikithingy,

Apple A series processors beginning with the A7 contain a Secure Enclave coprocessor running an L4 operating system called sepOS (Secure Enclave Processor OS) based on the L4-embedded kernel developed at NICTA in 2006. As a result, L4 ships on all modern Apple devices including Macs with Apple silicon.​

 

[mr_roboto]

Yoused beat me to it - Apple's been shipping a customized version of L4 for a while. The only oddity here is why they weren't already a member of this seL4 consortium.

 

[Jimmyjames]

While it’s true that the sep uses L4, in fact I mention it in the OP, I think what’s being missed here is firstly that seL4 was recently formally proved correct for AArch64, while the sep is an ARMv7a “Kingfisher” core, which is 32 bit afaik. and the link I provided states there are mentions of CL4 (Apple’s modified L4) within iOS itself, which were not present in versions prior to iOS 17.

I believe there is evidence to indicate a slow move towards a more secure microkernel architecture.

 

[mr_roboto]

Are you sure the SEP in Apple Silicon Macs is a 32-bit core? The references I can find which mention "Kingfisher" date from when A9 was Apple's most current SoC, ~9 years ago. I'd give good odds that as of A14/M1 the SEP was already 64-bit, because all the other coprocessors in that generation I've heard of are Apple-designed "Chinook" 64-bit cores.

Many of the security hardening features being introduced into the Arm ISA are 64-bit only. Pointer authentication is only possible in 64-bit. It takes advantage of practical physical and virtual address sizes being much smaller than 2^64, which lets it borrow some bits for encoding authentication info instead of address bits. Not possible on 32-bit. So it would make a lot of sense for Apple to have moved the SEP to 64-bit quite a long time ago, just to align with all the security features they've been pushing into Arm v8-A.

Also...

I believe there is evidence to indicate a slow move towards a more secure microkernel architecture.

Are you talking about replacing Darwin with seL4? Because I don't think that's on the table. Maybe they've got a skunkworks project, but I doubt they have concrete plans.

I don't base this on any evidence, it's just that in general, formal correctness proofs of software don't scale. It's a very attractive idea for sepOS, where the feature set is small and the interfaces narrow, but it is probably impossible to attempt formal proofs on something with the set of features any Darwin replacement must have.

 

[Jimmyjames]

Are you sure the SEP in Apple Silicon Macs is a 32-bit core? The references I can find which mention "Kingfisher" date from when A9 was Apple's most current SoC, ~9 years ago. I'd give good odds that as of A14/M1 the SEP was already 64-bit, because all the other coprocessors in that generation I've heard of are Apple-designed "Chinook" 64-bit cores.

It’s difficult to find information on this. I used this as a resource:

https://www.blackhat.com/docs/us-16/materials/us-16-Mandt-Demystifying-The-Secure-Enclave-Processor.pdf

They mention the SEP using a ARMv7a “Kingfisher” core on page 10. On page 17 they mention SEP is 32-bit.

Many of the security hardening features being introduced into the Arm ISA are 64-bit only. Pointer authentication is only possible in 64-bit. It takes advantage of practical physical and virtual address sizes being much smaller than 2^64, which lets it borrow some bits for encoding authentication info instead of address bits. Not possible on 32-bit. So it would make a lot of sense for Apple to have moved the SEP to 64-bit quite a long time ago, just to align with all the security features they've been pushing into Arm v8-A.

Also...

Are you talking about replacing Darwin with seL4? Because I don't think that's on the table. Maybe they've got a skunkworks project, but I doubt they have concrete plans.

Replace in this context is difficult to say. I am not saying they are going to just remove xnu and put seL4 in its place. If you read the original article, they mention “exclaves”. They are breaking out security related components into these exclave pieces which are unaffected by a compromise of the kernel, aka xnu. So my theory is they won’t remove things initially. They will augment xnu with these exclaves, for security related components and move on from there while preventing any further things going into the kernel, as they have already started many years ago with DriverKit, no third party kexts etc. Eventually xnu becomes less and less vulnerable and important.

I don't base this on any evidence, it's just that in general, formal correctness proofs of software don't scale. It's a very attractive idea for sepOS, where the feature set is small and the interfaces narrow, but it is probably impossible to attempt formal proofs on something with the set of features any Darwin replacement must have.

Well tbh I don’t think any of us have any evidence. It’s small nuggets at best. The original article which mentions L4 within iOS 17 and exclaves, Mark Gurman’s live chat after the dissolution of the car project mentions “safetyOS".

 

[Yoused]

Secure Enclave is like a tiny server inside the system. It guards the most sensitive data – biometrics, passcodes, Apple Pay data – and does not allow the OS or apps to get a peek at it. It can be asked, "is this ok?" or "please pay the man", but the data it holds stays in its own box. Hence, a 32-bit processor, accompanied by an AES CP, is entirely adequate, because the box is really, really small. And it makes sense for it to run a different kernel and OS, because that makes it harder to attack. Not quite impossible, but wuch more difficult.

 

[mr_roboto]

It’s difficult to find information on this. I used this as a resource:

https://www.blackhat.com/docs/us-16/materials/us-16-Mandt-Demystifying-The-Secure-Enclave-Processor.pdf

They mention the SEP using a ARMv7a “Kingfisher” core on page 10. On page 17 they mention SEP is 32-bit.

Yeah, I found that same conference paper. It seems to be most of the information that's easily searched about the internals of the SEP, and it's the thing I was thinking of when I mentioned that the info out there is from the A9 era - that being the latest SEP the paper discusses.

The problem is, as this Apple page makes clear, the SEP is a moving target.

Secure Enclave

The Secure Enclave is a dedicated secure subsystem in the latest versions of iPhone, iPad, Mac, Apple TV, Apple Watch, and HomePod.

support.apple.com

It includes its own TL;DR section - at the end there's a "Secure Enclave feature summary" table. Lots of changes have been made on the road from A8 to M1. Now, the Apple page doesn't talk about which version of Arm ISA the SEP uses, but they don't talk about that (or the fact that it uses Arm ISA at all) because no one but Apple is supposed to have code execution on the SEP, so their docs treat its ISA as a black box.

Replace in this context is difficult to say. I am not saying they are going to just remove xnu and put seL4 in its place. If you read the original article, they mention “exclaves”. They are breaking out security related components into these exclave pieces which are unaffected by a compromise of the kernel, aka xnu. So my theory is they won’t remove things initially. They will augment xnu with these exclaves, for security related components and move on from there while preventing any further things going into the kernel, as they have already started many years ago with DriverKit, no third party kexts etc. Eventually xnu becomes less and less vulnerable and important.

Gotcha. I think you may be overestimating the extent to which XNU will become less important, though. Their approach here is very surgical and precise - not much of XNU's functionality is being taken over by external coprocessors not running XNU, just very small and targeted things.

Also, sometimes the SEP gets its own "exclave". Look at the stuff about the Secure Enclave Boot Monitor in the Apple article I linked; it sounds like the same sort of thing.

Way back when Apple started getting serious about security, I remember a WWDC slide where they presented their philosophy of "defense in depth". This means that they plan on various security layers failing, and try to limit the amount of damage by having something else backing that part of the system up. I'd say these things fit that theme - they're looking for key small and self-contained vulnerable points currently exposed to other pieces of the system at the same privilege layer, and breaking them out into their own independent layer that won't be compromised by the original thing getting broken. If that makes sense. Don't know that I've written out my thoughts all that clearly here.

Secure Enclave is like a tiny server inside the system. It guards the most sensitive data – biometrics, passcodes, Apple Pay data – and does not allow the OS or apps to get a peek at it. It can be asked, "is this ok?" or "please pay the man", but the data it holds stays in its own box. Hence, a 32-bit processor, accompanied by an AES CP, is entirely adequate, because the box is really, really small. And it makes sense for it to run a different kernel and OS, because that makes it harder to attack. Not quite impossible, but wuch more difficult.

Balanced against that is the fact that the SEP, like most / all the other coprocessors, shares the same physical address space as the main application processors. If you're a SoC architect at Apple, you'd probably like the SEP to understand 64-bit addresses so it has access to the whole address space of the system. This is true even though the SEP gets locked down to its own private encrypted memory region (see that same Apple document I linked above); flexibility in address space layout is a good thing.

Also balanced against it is the fact that, as I mentioned before, most or all of the exciting new security extensions in the Arm ISA are happening in the 64-bit ISA, not the old 32-bit ISA. sepOS is precisely the kind of place where Apple would want to adopt these features, so that'd be a very good argument for them to have already switched the SEP over to one of their 64-bit microcontroller cores. Which we know they have, as they're all over the place in A14/M1 and onwards.