But you could return the 1000 objects (or less? 1000 records sounds like a lot for any UI to show at once) of 1kb size and allow the clients to request specific pages with a request parameter. There may be applications where you need to ship the full 1M records I guess, but that seems like very much an edge case as far as web apps go.
True, you would not return 1000 objects at once to the frontend.
I first thought it's just a backend use-case, where processing 1000 records in a paginated result is common, but the parent mentions "rails", so it sounds like a frontend use-case.
1,000 records is absolutely not a lot on modern computers or connections. On a business LAN, this request should take well under a second full latency.
On an average mobile connection, it’s maybe a second or so.
You’re right. It’s not a lot for a machine. The point isn’t the speed capability. It’s why? What UI has 1,000 rows in, e.g., a table all at once (much less 1M)?
Many plots contain thousands of data points. Eg: 10 x 100 heatmap which supports sorting by various metadata. This is a common visualization for biological data where your data matrix is samples x proteins, so potentially much larger than 1000 data points.
My entire point was that Rails is not designed for this.