You can, for sure run a mail-only server. Probably the easiest thing would be to only allow the mail ports with CSF -- 25,465,587,110,143,993,995, as well as SSH and CWP ports. Don't allow any other standard ports (FTP, HTTP, HTTPS, etc.). And you could disable other services permanently so nothing auto-starts with the server.
And as you said, configure DNS appropriately and the MX and SPF records and you should be good.