CONNECTION_TIMED_OUT if I access the site on PC and Android (from same IP)

I'm migrating my company's site to a vps (as the hosting we were using was to limiting for us). I set it up, and it seemingly works just fine... However if you access the site from an Android device that's connected to the same network as other PCs, the PCs will get CONNECTION_TIMED_OUT error for some time. That makes the site unreachable but also CWP. It's as if the Android get priority or something as the mobile device won't get the error and the site works fine there...
This does not happen if the device uses mobile network to access the internet
I can have the site accessed on many PCs at the same time and it doesn't happen...