FULL RECAP: Apple Intelligence Group Lab, the short version for Local AI builders Sharing highlights from…
@pratikg
FULL RECAP: Apple Intelligence Group Lab, the short version for Local AI builders
Sharing highlights from this week's session hosted by @shshnkp (Apple AI/ML Evangelism Team) with @mgorbach and engineers from Foundation Models, Core AI, Siri and App Intents teams at Apple.
Apple’s app framework can use your open-weight model as its LLM. Apple’s new app framework can use Apple’s on-device model, Apple’s cloud model or a Core AI converted model, which is interchangeable with a one liner! And your model gets access to the same tool calling, structured output, profiling with Instruments, etc. as Apple’s models.
Don’t use Apple’s on-device model for coding, even according to Apple engineers! Use Core AI to bring in a model for coding. One engineer uses a setup where he can use MLX and Core AI models in different “coding harnesses” and reports good experiences.
Free local Chat Completions endpoint on any Apple silicon Mac with macOS 27! fm serve provides an OpenAI style Chat Completions endpoint backed by Apple’s on-device model on any Mac. Nothing to install! (shared by Shashank the day after the lab)
Apple’s new on-device model is 20B, but only has 8K context. AFM Core Advanced (20B) will be available on the latest iPhones, iPads and M3+ Macs. Can’t opt out of Apple Intelligence in 27, and the model will always be present. Tip: use typed units for fields and let constrained decoding handle it (instead of wasting tokens explaining in prompt).
Agentic Siri only works when online. Tool calls from Siri to apps are done using Private Cloud Compute. When offline, Siri will use the on-device model but won’t be able to call into apps.
Lessons Apple engineers follow when building agents apply to any agent:
- write down instructions to the model about how/when to use your tools in a markdown file (“skills” are the same idea as agent skills everywhere else!)
- skills are just context to the model, meaning no ordering guarantees. If something should always happen in an order, it should probably be a tool
- make tools accept lists of items to operate on to avoid agent having to call it in a loop
- unit test your tools (deterministic) before testing the whole agent (non-deterministic)
Curated Q&A relevant to Local AI builders in 🧵
FULL RECAP: Apple Intelligence Group Lab, the short version for Local AI builders
Sharing highlights from this week's session hosted by @shshnkp (Apple AI/ML Evangelism Team) with @mgorbach and engineers from Foundation Models, Core AI, Siri and App Intents teams at Apple.
Apple’s app framework can use your open-weight model as its LLM. Apple’s new app framework can use Apple’s on-device model, Apple’s cloud model or a Core AI converted model, which is interchangeable with a one liner! And your model gets access to the same tool calling, structured output, profiling with Instruments, etc. as Apple’s models.
Don’t use Apple’s on-device model for coding, even according to Apple engineers! Use Core AI to bring in a model for coding. One engineer uses a setup where he can use MLX and Core AI models in different “coding harnesses” and reports good experiences.
Free local Chat Completions endpoint on any Apple silicon Mac with macOS 27! fm serve provides an OpenAI style Chat Completions endpoint backed by Apple’s on-device model on any Mac. Nothing to install! (shared by Shashank the day after the lab)
Apple’s new on-device model is 20B, but only has 8K context. AFM Core Advanced (20B) will be available on the latest iPhones, iPads and M3+ Macs. Can’t opt out of Apple Intelligence in 27, and the model will always be present. Tip: use typed units for fields and let constrained decoding handle it (instead of wasting tokens explaining in prompt).
Agentic Siri only works when online. Tool calls from Siri to apps are done using Private Cloud Compute. When offline, Siri will use the on-device model but won’t be able to call into apps.
Lessons Apple engineers follow when building agents apply to any agent:
- write down instructions to the model about how/when to use your tools in a markdown file (“skills” are the same idea as agent skills everywhere else!)
- skills are just context to the model, meaning no ordering guarantees. If something should always happen in an order, it should probably be a tool
- make tools accept lists of items to operate on to avoid agent having to call it in a loop
- unit test your tools (deterministic) before testing the whole agent (non-deterministic)
Curated Q&A relevant to Local AI builders in 🧵
