Foundation models APIs

[leman]

So I finally had some time to look at the new ML frameworks and I think that Apple opening access to the foundational models is kind of a big deal that did not get enough attention. In a nutshell, what these APIs allow you to do is send prompts to the on-device LLM, with some nice features like boilerplate-less structured output and MCM-like functionality.

None of this is new of course — after all you could do this stuff for years with services like ChatGPT or locally installed LLMs (e.g. via llama.cpp). The difference is accessibility to developers. Previously, if I wanted to add some LLM-based features to my app, I'd either need an online connection with an API key (issues with security, privacy, extra costs, user experience), or I'd need to bundle a several-TB LLM with my app (almost never an option). But with this release I can just drop some LLM functionality in minutes and have it run entirely on device. Granted, these LLMs are not very good, but they are more than sufficient for many applications. I will certainly be playing around with them for automatic language annotation and similar research applications.

I think we are about to see a bunch of really interesting apps taking advantage of these features.

P.S. The way they implemented agent-based AI editing in Xcode is very cool as well.

 

[throAU]

If they on-device model can't handle it, does it ship the query off to the user's plug-in model (e.g., your ChatGPT account if you have one)?

If not, maybe that would be a good way of doing things locally if possible and transparently forwarding to the cloud if not (assuming you consent).

 

[leman]

If they on-device model can't handle it, does it ship the query off to the user's plug-in model (e.g., your ChatGPT account if you have one)?

If not, maybe that would be a good way of doing things locally if possible and transparently forwarding to the cloud if not (assuming you consent).

I did not see any mention of forwarding like you describe in the APIs, at the same time it should be easy enough to do with a custom wrapper.

 

[Cmaier]

I was playing around in shortcuts with this, and it’s very cool how you can choose whether to use on-device, off-device, or ChatGPT, and you can have it automatically fill a data structure with the results. This will open this stuff up to lots of people.

 

[ArgoDuck]

Yep i was pretty blown away with @Generable and tool calling in their presentation, great way to get more from a 3B LLM. Eric’s enthusiasm and ideas in the related lab was infectious!

 

[Jimmyjames]

So I finally had some time to look at the new ML frameworks and I think that Apple opening access to the foundational models is kind of a big deal that did not get enough attention. In a nutshell, what these APIs allow you to do is send prompts to the on-device LLM, with some nice features like boilerplate-less structured output and MCM-like functionality.

None of this is new of course — after all you could do this stuff for years with services like ChatGPT or locally installed LLMs (e.g. via llama.cpp). The difference is accessibility to developers. Previously, if I wanted to add some LLM-based features to my app, I'd either need an online connection with an API key (issues with security, privacy, extra costs, user experience), or I'd need to bundle a several-TB LLM with my app (almost never an option). But with this release I can just drop some LLM functionality in minutes and have it run entirely on device. Granted, these LLMs are not very good, but they are more than sufficient for many applications. I will certainly be playing around with them for automatic language annotation and similar research applications.

I think we are about to see a bunch of really interesting apps taking advantage of these features.

P.S. The way they implemented agent-based AI editing in Xcode is very cool as well.

Yes I’ve seen quite a few devs enthuse about the foundation models and echo your sentiments. While Siri holds the media’s attention, these models have the chance to substantially improve so many apps with AI that will go under the radar.