How big deny list can be before affecting performance?

I understand deny list default limit is about 200 IPs. I find the need to increase more and more everyday and the list may get really large over time. I have about 400 right now. I want to know when do I have to start worrying about server performance. Most of the ip in the list are a block of many ip such as 111.222.0.0/16. So technically it is hundred or thousands of IPs every line. i assume at some point the server would crash or slow down processing large deny list correct? What is the best way to manage it?

Let it grow but make sure you remove older then 60 days entries. Block subnet IPs and clean individual botnet IPs.
My suggestion put DENY_IP_LIMIT = "5000" and run this script daily. I have created it exactly for that.

https://forum.centos-webpanel.com/csf-firewall/csf-analyzer/

DENY_IP_LIMIT will check first 5000 entries from deny list and this script will take care of your deny list to stay clean and lean.
If your list still grow over 5000 entries then enlarge your DENY_IP_LIMIT.