Its still more secure. Rubber hose cryptanalysis applies to both equally, but that doesn't mean there aren't other attacks that apply to totp which don't to yubikeys.
With a phone you need my passcode to accept to 2FA request (assuming lock screen notifications are disabled). I think yubikeys can work without a passcode as long you plug it in right?
Right, but presumably the site is already asking for a password, and if the attacker can bypass one password, im not sure its a safe assumption that they cant bypass two. However fair enough. Some yubikeys do involve fingerprint scans too though.
The main security benefit is unphishability. With yubikey/webauth crypto is used so you can't give the code to the wrong website. Phishing is a pretty major cause of account hacks generally, so pragmatically that is a very big win.
With a Yubikey, you need to use your password to log in to your computer, and then need to auth using Yubikey.
With OTP app, you need to use your password to log into your computer, passcode for phone, and then auth.
In both cases, it's something you know, and something you have. You could argue that the app based is a bit more secure in that you need two passwords. On the flipside, if your phone gets pwned, someone can access completely remote.
More secure != perfectly secure.