I actually find that the opposite works for most cases.
The best board games are the ones that are either a take on "common social games" or have a fairly simple ruleset.
The problem with complex rulesets is that they tend to make rounds very long which when you combine it with 4-6 players makes games drag for considerably longer than they need to be.
The worse games are the ones that try to mitigate that with multiple victory paths often partially RNG based or silly "sudden death" rounds after an arbitrary round count to force games to last no longer than 60-90min.
Sure you can point to games or systems like D20 but even then the core ruleset is fairly simple and uniform and the rest is just ad hoc that can be used rarely or not at all.
The one thing you need for a good board game is to have decent strategic depth with mechanics that induce interesting social dynamics, a lot of games try to implement the former by making it complicated by design and forget the latter.
The rulebook for chess and go is zillions of published books on pattern matching and book openings and book responses. To play at a high level you have to memorize and perfectly apply uncountable zillions of rules and patterns or you'll just be destroyed by someone who's memorized more rules. If you don't know all the chess book openings and you play against someone who does know them, you may as well just save everyone some time and immediately resign. Much like MtG, how much time and money are you willing to spend to win? This turned me off from the whole genre of "memorize and pattern match" games, I just don't find that to be recreational.
The pinnacle of large rulesets is RPG games, anyway. With the possible exception of some 80s classics like squad leader and starfleet battles there are no rulebooks that compare in size to the foot or so linear shelf space of a full set of Pathfinder rulebooks.
Strategy/tactics != rules. The rules for chess and go are simple. The gameplaying that stems from those simple rules is obviously quite complex--and, as you say, it has been deeply studied so anyone wanting to play even somewhat competently needs to have a passing understanding of quite a bit. But the rules themselves (especially with go) are quite straightforward compared to most modern adult boardgames.
So what is it you're really asking for? A game where even a beginner can beat a long-time fan, yet one that is interesting? I think those goals are conflicting unless we're talking pure gambling.
A game that was basically an IQ test might work as well. Like having to solve those abstract pattern matching or 3D-to-2D geometry questions. Maybe. I'd guess there's some slight advantage to having answered many of such questions, but I doubt it is nearly as big a difference as, say, studying Chess or Go.
Do you find the replayability of smallworld is largely due to the randomisation of which (race, attribute) pairs are available per game?
Arguably Dominion has good replayability for similar reasons: you only play with a subset of the cards every game, and a lot of the strategy revolves around exploiting interactions between different cards. On the other hand Dominion is one of those games where there is very limited interaction between players, which perhaps doesn't make for great replayability.
I also play Hearthstone, the Blizzard card game, and that's a pretty awful example, sometimes, of bad randomization with all those effect that can range from harmful to do nothing to win the game on the spot.
The randomization in Smallworld is good because you always have a certain amount of control over it and you can plan for it/around it. On top of that the power swings are not really extreme (some race - attribute combos are stronger than others but the overall power range is reasonably small).
I love Smallworld, and yeah, I never get tired of it. Another one whose rules are pretty straightforward is Power Grid, been playing it for years and it hasn't gotten old yet (except for the sometimes pretty sudden and ruthless endgame).
Go/baduk would be the most obvious counter-example here. It has something like 9 rules in total (depending on how you count them) and has been replayed for more than 5000 years.
There are several games that are not too complicated, that also has a charm, but necessarily, more complicated mechanism makes for more variable play.