apache stop responding then crashes

337
September 06, 2017, at 4:25 PM

Apache (2.4.25 FPM/FastCGI) stop responding and crashes after a while on my wordpress multisite website.

For some reason it used to run fine several days until I changed the wordpress(mu) domain using this script github.com/interconnectit/Search-Replace-DB (probably not the reason).

When restarted it works for about an hour then choked again.

I have added a mysql slow query log but its empty.

Apache error_log has multiple errors like this one:

[Wed Sep 06 08:50:27.941819 2017] [proxy_fcgi:error] [pid 25444:tid 140610719610624] (70007)The timeout specified has expired: [client x.x.x.x:53398] AH01075: Error dispatching request to : (polling)

and occasionally this error appears too:

[Wed Sep 06 09:13:33.296777 2017] [core:notice] [pid 10710:tid 140611211331392] AH00051: child pid 25582 exit signal Segmentation fault (11), possible coredump in /opt/bitnami/apache2

another occasional error (expected due to the multiple AH01075 timeout errors):

[Mon Sep 04 20:18:58.758718 2017] [mpm_event:error] [pid 19928:tid 140675798046528] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

Stopping mysql server kills my site but seems to keep apache working fine, at least for an hour or two that i`ve been testing.

Is 457452 value is normal for mysqld.bin in the following process list? (ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS)

2128 /sbin/init
2400 php-fpm: master process (/opt/bitnami/php/etc/php-fpm.conf)
3840 -bash
4244 sshd: ubuntu [priv]
6928 /usr/bin/gonit
11036 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
11320 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
11364 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
11592 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
20592 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
57668 php-fpm: pool wordpress
57716 php-fpm: pool wordpress
65252 php-fpm: pool wordpress
67608 php-fpm: pool wordpress
68776 php-fpm: pool wordpress
457452 /opt/bitnami/mysql/bin/mysqld.bin --defaults-file=/opt/bitnami/mysql/my.cnf --basedir=/opt/bi

PHP Version 5.6.30

mysql.bin Ver 14.14 Distrib 5.6.36

Ubuntu 14.04.5 (bitnami image on AWS)

Memory usage while apache not responding: used:883mb of 992mb

php.ini script memory_limit = 128M

Answer 1

Was a bit long to put it in a comment, so replying as an answer.

Are you logging any php errors? From what you presented so far, my theory would be that:

  1. You used that script to do a search & replace of db but it executed poorly causing malformed or corrupted data.
  2. That caused your php scripts that rely on mysql to go haywire due to malformed or corrupted data.
  3. Because your php on fcgi isn't responding, your apache waits for it and holds onto the resources.
  4. All the requests are hanging and now you run out of resources and start seg-faulting.
  5. And then stupid apache, seeing that you're out of child count asks you to raise it not realizing you're already exhausted on physical resource beyond reason. Typical apache...
  6. Because apache is now out of free children, it can't take anymore requests and it appears as if all of apache is broken even though it's merely clogged up by bad fcgi processes.

If my theory is correct, you should.

  1. Check that none of the mysql data is corrupted

  2. Check that the data that was replaced in mysql was as what you expected

  3. Check for PHP error logs as they should be logging something if they can.

  4. You should isolate the PHP processes that are actually the cause of clogging up and remove it if cleaning up data alone wasn't enough. Ideally, no matter how bad the data, the php program should've died cleanly.

Good luck hunting.

Rent Charter Buses Company
READ ALSO
Fetch record from table with duplicate entries

Fetch record from table with duplicate entries

I have my table structure like below in desc order

180
Is it possible call stored procedure in php fetch dynamic columns and results?

Is it possible call stored procedure in php fetch dynamic columns and results?

I create pivot using stored procedure, but i don't know how to automatically fetch all columns and results in PHP

189
Another… Query fail: Commands out of sync; you can't run this command now

Another… Query fail: Commands out of sync; you can't run this command now

I am new here as a poster so please excuse me becoming familiar with protocol and use of the siteI am not a software engineer but have written quite a bit over the years

280