This, plus metal acceleration works quite well. 7~8B parameter models quantized ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		bt1a on June 11, 2024 \| parent \| context \| favorite \| on: Apple's On-Device and Server Foundation Models This, plus metal acceleration works quite well. 7~8B parameter models quantized to 3bpw or so run with good tok/s on my iphone 15 pro

sanxiyn on June 11, 2024 [–]

It works quite well as long as you don't care about battery.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact