I feel like they mostly don't playtest the rulebook very much, and that doing so would get it in great condition.
Watching someone try to learn the rules will tell you exactly where the rulebook is too complicated, doesn't introduce something at the right time, doesn't reiterate something that needs reiterating, doesn't emphasize something enough, etc.
It's hard to playtest a rulebook, because you need to get fresh players every time, if you want to optimize for the out of box experience (which I think you should!). But also, I think selection bias for playtesters is going to give you testers who are willing to dig through complex rule interactions, and that might not be everyone.
I've had the best luck with explaining games that have short rule sets. Anything more than 4 pages and you really need the multiple sets of rules paradigm, and a good tech editor to make sure the playthrough instructions are accurate with the reference instructions. It's my personal choice, but I also don't care for the games where there's enough stuff going on that it's easy to skip things that are important to balance the game; but there's certainly a balance --- I want a game where there's enough chance that anyone could win, but enough skill that I feel like I can still do well even if chance is against me, and I strongly prefer games where everyone is engaged until the end.
In my experience, it is not that hard to get a stream of fresh playtesters. Maybe I'm just fortunate.
I also don't think the playtester bias to be particularly bad for this. You'll miss some stuff because your playtesters will usually be a lot better than average at reading rules. But nearly everyone stumbles in visible ways on rulebooks, and the fixes for an experienced playtester stumbling tend to be simplifications / clarifications that are even more useful to inexperienced players.
Fresh players are easy, just go any college and put up some posters. $minimum wage to try a new game - it isn't a good deal if you are into money, but if you ensure they have fun and have some beverages they will do it anyway. If someone is really good at something you know who to make an offer to.
Usability studied are not about finding solutions, they are about finding where your solution works/doesn't work. There are too many places where creative solutions are not usable, and so you need to send your creative people back to the drawing board.
This is my top comment. Some designers I've worked with are so obsessed about playtesting and tuning individual mechanics or systems that the never actually playtest the full game flow with no-knowledge players also _running_ the game.
From a technical standpoint, too many technical writers (much less game designers writing rulebooks) lack the tools or understanding to single-source content and consistently reuse it in multiple places. A lot of rulebooks are written and designed as if repeating content is the greatest of sins, but for a game in play, _the last_ thing you want to do is make someone flip to referenced pages. You know what the text should be, just reprint it consistently where it's relevant.
But then this reveals that the problem is often inherent in a game's physical design. If a game has several discrete rules that must be frequently cross-referenced or repeated, those rules should move out of a physical book and onto cards or discrete handout items. And indeed, in my experience doing that has been the effective endgame of the "rulebooks are reference documents that lack the searchability of a reference document" complaint from this essay's writer: rather than having a comprehensive, organized, searchable reference document, find ways to move the rules into the game. Even better: find ways to move the actions allowed by the rules into the physical actions of interacting with the game, such that you don't need words to describe them.
The essay instead suggests borrowing or informing structures from textbooks and lesson plans, but the pragmatic answer isn't to make a new, better kind of game-teaching book, it's to make the rules more accessible _within the medium of the game itself_, of which the rulebook rarely or never is. The best reference guide is the one that doesn't exist because the designers recognized that it shouldn't and designed the game appropriately. The second-best is one that's integrated into the game or board as a component.[1]
In other words, and against the essay's conclusion, the necessary text to run a board game _should_ fit on the inside of the box lid: who can play, how to set it up, and how the components embody the rules. The board and components themselves should be the best, and ideally only, required documentation after that.
But that's also a fundamental design problem in games. This essay proposes a way to make a better band-aid for that self-inflicted wound, but it would be best if designers thought about how to avoid it, and that can only happen when you have know-nothing playtesters crack the box open for the first time. Alas, much of modern game design is so dense and maximalist that if you shipped a game that didn't require a dense rulebook to play, people would knee-jerk react as if the game was too simple or just awful.
1: One of my favorite recent board games is That Time You Killed Me, a time-travel-themed game with a complex conceit. It incorporates the manual as part of the game's progression, and it's flavorful, tells a story, is aesthetically pretty, and builds its rules upon itself at each step. It does few of the things advised in this essay, and if it did more of them, I think I might like the game less.
https://gamers-hq.de/media/pdf/aa/31/ef/TTYKM_Rules-min.pdf is the reference guide and core tutorial; it's 12 pages long, most of it art. The rest of the rules are in the spoiler-filled chapters not part of that PDF, each of which introduces new rules alongside a separate sealed box of new components.
Just about the only innovations in board games per se that I've seen, that've made them easier to play, are increased focus on helpful iconography, and more widespread use of "cheat sheets" (cards, little cardboard bits with print on them) for each player. Not every game uses them, and not every game uses them well (the iconography, especially) but damn is it ever nice to sit down to a game you haven't touched in a year and find that just glancing over the player cheat-sheet and icons on the games' various bits is enough to refresh your memory of the rules well enough to present it to newbies again, leaving maybe just a few details of initial set-up to be looked up in the rulebook (and sometimes the really good ones sneak those onto the cheat sheets or graphic design of e.g. the board!)
Watching someone try to learn the rules will tell you exactly where the rulebook is too complicated, doesn't introduce something at the right time, doesn't reiterate something that needs reiterating, doesn't emphasize something enough, etc.