And what would happen if anyone steals that key? And that "anyone" has physical access to your terminal...
With most of the services Yubikeys are used with, it's used with FIDO U2F, which is a second factor authentication, not the whole authentication.
You can compare it to RSA tokens that spit out random-looking 6 or 8 digit numbers that change over time, and you use it together with a password.
Compared to that though, the Yubikey is using stronger cryptography, and you can use a single key with multiple services, without the different services knowing it's the same token. It's also stronger in that it signed specifically for the site you're at, providing significant protection against phishing.
Now, you're right that theft would be an issue, but no more with the Yubikey than an RSA token, and theft of the token alone gets you nowhere.
It's arguably somewhat safer from theft, because by using the same token with multiple services, you'd be a lot more likely to notice if the token went missing.
For things like SSH and PGP keys, you have password protection, including an attempt-counter, so the token can wipe itself if too many incorrect attempts are made.
If an attacker has physical access to terminal, and sufficient skills and resources, I'd be f*ed no matter what, and that's a bit outside the scope that the Yubikey tries to solve.
There's an old cliché that "security isn't a product", but it holds true, and it very much applies here. You can't just buy a yubikey and be secure, but you can use it to significantly upgrade some of the authentication-specific parts of a larger security plan for example.