at a certain point you're gonna need to change your benchmark because this will ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

holtkam2 69 days ago | parent | context | favorite | on: Gemini 3.5 Flash

at a certain point you're gonna need to change your benchmark because this will end up in the model's training set

simonw 69 days ago | [–]

Gemini were the team most likely to have this in their training set - see https://x.com/JeffDean/status/2024525132266688757 - and yet their latest model still messes up the bicycle frame!

recursive 69 days ago | | [–]

I'm sure that certain point came and went many releases ago.

kzrdude 69 days ago | [–]

As mentioned in another recent thread, that time is now.

Consider applying for YC's Fall 2026 batch! Applications are open till July 27.
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact