> or even a single socket Epyc system with >1TB of RAM How many tokens/second wo... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		gpm 6 months ago \| parent \| context \| favorite \| on: Kimi K2 is a state-of-the-art mixture-of-experts (... > or even a single socket Epyc system with >1TB of RAM How many tokens/second would this likely achieve?

chithanh 6 months ago | [–]

KTransformers now supports Kimi K2 for MoE offloading

They claim 14 tps for the 4-bit quant on a single socket system with 600 GB RAM and 14 GB GPU memory.

kachapopopow 6 months ago | | [–]

around 1 by the time you try to do anything useful with it (>10000 tokens)

neuroelectron 6 months ago | [–]

1

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact