SSH Port change

Ok, so I've successfully changed my ssh port and it is working fine.  However, I really wanted to change it to something completely unguessable - like 23456 (not actually that, but like that).  However, anytime I change it to anything that does not start with 22 and is only 4 characters long, it doesn't work.

Suggestions? 

It shouldnt happen like that. What port you are using for example and how you are changing it?

http://wiki.centos-webpanel.com/how-to-change-ssh-port

I just changed yesterday to a 4 digit one (starting with 3) with no problem.

I want to change it to a 5-digit port.  I've successfully changed it to a 4-digit port, but can't get a 5-digit to work. 

I've tried 3269, 8123, 32691 and none of these work.  That is after following the http://wiki.centos-webpanel.com/how-to-change-ssh-port.  My current port - 2222 - does work.

Have you checked this guide? It has a 5 digit port in the example.
http://wiki.centos-webpanel.com/csflfd-firewall-configuration

Looked at it.  Tried this:

# Allow incoming TCP ports
TCP_IN = "20,21,29950,25,53,80,110,143,443,465,587,993,995,2030,2031,2082,2083,2086,2087,2095,2096,2222,3269,8123,32691"

# Allow outgoing TCP ports
TCP_OUT = "20,21,29950,25,53,80,110,113,443,2030,2031,2082,2083,2086,2087,2095,2096,587,993,995,2222,3269,8123,32691"

And this in my sshd_config:
#
Port 3269
Port 8123
Port 32691
Port 2222
Port 29950

Only 2222 works.  Everything else fails.