But then it would still be an issue when I removed ClamAV etc.
It turns out the issue was the SPF, I found it by sending an email to Yahoo and checked the header, which gave an error that the SPF error and couldn't be checked, which for Google this would give a low reputation.