Change default port 22 and use key authentication.
LF_DISTATTACK = ON
LF_DISTATTACK_UNIQ = 2
Block certain countries, depending on your required 'market' eg. CN,TH,TW,RU,IL,BR,AG,SG,IN,PK,HK
(Still a lot from the USA though.)
Block AWS, Contabo, DO and GCloud.
Block all other lusers on your provider's LAN.
Other than that nearly nothing - it's a consequence of being on the 'net.
You could limit ssh access to a particular VPN but is of limited use, in terms of actual port scanning.
(My record for setting up a server and getting attacks, is 1400 failed logins, in the time it takes to do an apt update and change ssh to key authentication!)