Cloudflare returns 522 timeout errors when trying to connect to Apache PHP-FPM

I have a CWP Pro virtual server running AlmaLinux 8.10 and CWP 1.14

The server is never under any appreciable load - max 33% of processor capacity and 4GB of RAM from 16GB total. Checking the Apache Server Status via the CWP control panel, everything looks sane - 12 spawned servers, utilises 1.5GB RAM, 272 tasks.

All sites on the server use PHP-FPM, some stuck on legacy PHP7.4 but most running PHP8.3

Cloudflare returns a timeout error of 522 on some sites, seemingly at random but rather often. If I disable Cloudflare, I get the same timeouts but with a vanilla error message in the browser. So Cloudflare is not at fault here.

I have tried various tactics to prevent this:

a) Apache was reporting it had run out of RequestWorker processes, so I included the following snippet via conf/extra/httpd-mpm.conf (Apache runs with the MPM Evnt mod on my server).

Code: [Select]

<IfModule mpm_event_module>
    StartServers            25
    ServerLimit             32
    MinSpareThreads         75
    MaxSpareThreads        250
    ThreadsPerChild         25
    MaxRequestWorkers    800
    MaxConnectionsPerChild   0
</IfModule>

b) When that didn't solve the problem, I also increased the number of max PHP-FPM processes in the default template and for individual sites. I increased from the default of MaxChildProcceses 4 to 25.

c) I enabled KeepAlive which I don't think is enabled by default, via self-authored snippet included in conf/extra/httpd-keepalive.conf

Code: [Select]

KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5

None of the above has resolved the problem. I am at a loss. Furthermore, some of the sites show extremely poor performance. Yet the server has loads of spare capacity.

Anyone got any insight here?

I have checked if disabling the firewall made any difference (in case it was blocking Cloudflare IP addresses). It did not and it is not.

1. Check how many processes are running under the user account of the problematic website when the error occurs.

2. Make sure the number of processes is not close to the Process Limit specified for the account in the CWP admin panel (check the account properties).

3. Check the package limits assigned to the problematic account. Pay particular attention to nproc, apache_nproc, and nofile.

4. Is mod_security enabled ?

5. Do you use a VPS or a dedicated server? If it’s a VPS, what type of virtualization is used?

1. Check how many processes are running under the user account of the problematic website when the error occurs.

2. Make sure the number of processes is not close to the Process Limit specified for the account in the CWP admin panel (check the account properties).

3. Check the package limits assigned to the problematic account. Pay particular attention to nproc, apache_nproc, and nofile.

Thanks - this is a great tip that had not occurred to me. I have increased all those limits now, will monitor and see if it improves things.

4. Is mod_security enabled ?

Yes

5. Do you use a VPS or a dedicated server? If it’s a VPS, what type of virtualization is used?

It's a VPS, running on Hetzner Cloud (KVM)

The fact that you still get the timeout with Cloudflare disabled is useful because it largely removes Cloudflare from the equation. I also wouldn’t increase MaxRequestWorkers or PHP-FPM children any further until you identify which layer is actually saturating.

When the timeout happens, I’d capture the state at that exact moment rather than looking only at overall CPU/RAM. Check the affected account’s PHP-FPM processes, its CWP process/package limits (nproc, apache_nproc, nofile), Apache scoreboard, and the corresponding PHP-FPM slow/error log. Also compare pm.max_children against the number of active/queued requests for that pool.

One particularly useful test would be to run several simultaneous requests directly against the origin while watching the affected PHP-FPM pool. If requests begin queueing while the machine still has free CPU/RAM, you’ve found a concurrency/limit bottleneck rather than a server-capacity problem.

I’d also temporarily correlate the timeout timestamps with ModSecurity audit entries. That should help distinguish PHP-FPM saturation, account limits, Apache worker exhaustion and request filtering instead of changing several limits at once.

I develop CWP7 Turbo Manager, specifically around diagnosing and tuning this sort of CWP web-stack performance problem. Full disclosure: I’m the developer. It may be relevant here, but I’d first capture the PHP-FPM/account state during one of the 522 events—the numbers from that moment should tell us which direction to go.

Thank you, this is very useful.

Actually when I checked the PHP-FPM slow logs (which I had not thought to do previously) I quickly discovered a site with a simple hack. It was hanging on the homepage due to a hack on the index.php executing various encrypted commands with curl.

Cleaning up the hack immediately improved performance to what I would expect!