8k context is tiny compared to what’s out there. They promise much larger contex... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		pointlessone on April 21, 2024 \| parent \| context \| favorite \| on: Has Llama-3 just killed proprietary AI models? 8k context is tiny compared to what’s out there. They promise much larger contexts but until then it can’t even reliably summarize every web page out there.

tarruda on April 22, 2024 [–]

There are techniques to extend the context window via fine tuning.

The authors claim this method was used to extend Llama 2 to 128k: https://github.com/jquesnelle/yarn

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact