We are having issues with a server in the following configuration:
CentOS linux PHP FPM 7.0.21 Nginx 1.12.0 Opcache v7.0.2 Redis 3.2.3
The website is using Amazon Cloud (CDN) also.
Journalctl reports the following error:
Aug 01 20:42:34 server.domain.com kernel: php-fpm70[57528]: segfault at 7ffcf961ffe8 ip 00007fc6219abab5 sp 00007ffcf961ffc0 error 6 in libpcre.so.0.0.1[7fc621995000+49000]
Aug 01 21:16:28 server.domain.com kernel: php-fpm70[57251]: segfault at 7ffcf961ffe8 ip 00007fc6219abab5 sp 00007ffcf961ffc0 error 6 in libpcre.so.0.0.1[7fc621995000+49000]
Aug 01 21:17:42 server.domain.com kernel: php-fpm70[59998]: segfault at 7ffcf961ffe8 ip 00007fc6219abab5 sp 00007ffcf961ffc0 error 6 in libpcre.so.0.0.1[7fc621995000+49000]
In PHP FPM log we are seeing errors like
[01-Aug-2017 20:38:06] WARNING: [pool admin] child 45894 exited on signal 11 (SIGSEGV) after 1322.654261 seconds from start
[01-Aug-2017 20:38:06] NOTICE: [pool admin] child 57528 started
[01-Aug-2017 20:38:59] NOTICE: [pool admin] child 22389 exited with code 0 after 3958.433189 seconds from start
[01-Aug-2017 20:38:59] NOTICE: [pool admin] child 58022 started
[01-Aug-2017 20:39:25] WARNING: [pool admin] child 22629 exited on signal 11 (SIGSEGV) after 3952.381104 seconds from start
[01-Aug-2017 20:39:25] NOTICE: [pool admin] child 58283 started
[01-Aug-2017 20:39:53] NOTICE: [pool admin] child 22628 exited with code 0 after 3982.730214 seconds from start
[01-Aug-2017 20:39:53] NOTICE: [pool admin] child 58542 started
[01-Aug-2017 20:41:13] WARNING: [pool admin] child 58283 exited on signal 11 (SIGSEGV) after 107.402522 seconds from start
The website dies and then after a restart of the PHP-FPM starts working again but sometimes it looses CSS/other styling for a minute or two. We also suspected this may be due to some Magento cache thats stored in redis.
We tried to create a coredump of the process when it died and this is the result:
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-94.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/php70/sbin/php-fpm70...done.
[New LWP 42619]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `php-fpm: pool admin '.
Program terminated with signal 11, Segmentation fault.
#0 0x00007fcbbf6a0b9f in match () from /usr/local/lib/libpcre.so.0
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 elfutils-libelf-0.166-2.el7.x86_64 elfutils-libs-0.166-2.el7.x86_64 glibc-2.17-157.el7_3.5.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-27.el7_3.x86_64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-8.el7.x86_64 libcom_err-1.42.9-9.el7.x86_64 libcurl-7.29.0-35.el7.centos.x86_64 libgcc-4.8.5-11.el7.x86_64 libgcrypt-1.5.3-13.el7_3.1.x86_64 libgpg-error-1.12-3.el7.x86_64 libidn-1.28-4.el7.x86_64 libselinux-2.5-6.el7.x86_64 libssh2-1.4.3-10.el7_2.1.x86_64 libstdc++-4.8.5-11.el7.x86_64 nspr-4.13.1-1.0.el7_3.x86_64 nss-3.28.4-1.2.el7_3.x86_64 nss-softokn-freebl-3.16.2.3-14.4.el7.x86_64 nss-util-3.28.4-1.0.el7_3.x86_64 openldap-2.4.40-13.el7.x86_64 openssl-libs-1.0.1e-60.el7_3.1.x86_64 pcre-8.32-15.el7_2.1.x86_64 systemd-libs-219-30.el7_3.9.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0 0x00007fcbbf6a0b9f in match () from /usr/local/lib/libpcre.so.0
#1 0x00007fcbbf69f52d in match () from /usr/local/lib/libpcre.so.0
#2 0x00007fcbbf69f52d in match () from /usr/local/lib/libpcre.so.0
#3 0x00007fcbbf6a0bbd in match () from /usr/local/lib/libpcre.so.0
#4 0x00007fcbbf6b707e in match () from /usr/local/lib/libpcre.so.0
#5 0x00007fcbbf6b849a in pcre_exec () from /usr/local/lib/libpcre.so.0
#6 0x000000000048af93 in php_pcre_replace_impl (pce=pce@entry=0x24cc050,
subject_str=subject_str@entry=0x7fcb599bb000,
subject=subject@entry=0x7fcb599bb018 "SELECT `main_table`.* FROM `sales_flat_quote_address` AS `main_table` WHERE (`quote_id` = '700576')", subject_len=subject_len@entry=99,
replace_val=replace_val@entry=0x7fcb59dce900, is_callable_replace=is_callable_replace@entry=0,
limit=<optimized out>, limit@entry=-1, replace_count=replace_count@entry=0x7ffe49bd256c)
at /usr/local/directadmin/custombuild/php-7.0.21/ext/pcre/php_pcre.c:1234
#7 0x000000000048bd86 in php_pcre_replace (regex=<optimized out>,
subject_str=subject_str@entry=0x7fcb599bb000,
subject=subject@entry=0x7fcb599bb018 "SELECT `main_table`.* FROM `sales_flat_quote_address` AS `main_table` WHERE (`quote_id` = '700576')", subject_len=99, replace_val=replace_val@entry=0x7fcb59dce900,
is_callable_replace=is_callable_replace@entry=0, limit=limit@entry=-1,
replace_count=replace_count@entry=0x7ffe49bd256c)
at /usr/local/directadmin/custombuild/php-7.0.21/ext/pcre/php_pcre.c:1136
#8 0x000000000048be3e in php_replace_in_subject (regex=regex@entry=0x7fcb59dce8f0,
replace=replace@entry=0x7fcb59dce900, subject=subject@entry=0x7fcb59dce910, limit=limit@entry=-1,
is_callable_replace=is_callable_replace@entry=0, replace_count=replace_count@entry=0x7ffe49bd256c)
at /usr/local/directadmin/custombuild/php-7.0.21/ext/pcre/php_pcre.c:1495
#9 0x000000000048c20e in preg_replace_impl (return_value=return_value@entry=0x7fcb59dce830,
regex=regex@entry=0x7fcb59dce8f0, replace=0x7fcb59dce900, subject=0x7fcb59dce910, limit_val=-1,
is_callable_replace=is_callable_replace@entry=0, is_filter=is_filter@entry=0)
at /usr/local/directadmin/custombuild/php-7.0.21/ext/pcre/php_pcre.c:1554
#10 0x000000000048ccfe in zif_preg_replace (execute_data=0x7fcb59dce890, return_value=0x7fcb59dce830)
at /usr/local/directadmin/custombuild/php-7.0.21/ext/pcre/php_pcre.c:1593
#11 0x000000000085a57d in ZEND_DO_ICALL_SPEC_HANDLER ()
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:586
#12 0x000000000084c8fb in execute_ex (ex=<optimized out>)
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:414
... then about 15000 lines of this and ends with:
#142522 0x000000000084c8fb in execute_ex (ex=<optimized out>)
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:414
#142523 0x00007fcbb111ef88 in ?? () from /usr/local/lib/ioncube/ioncube_loader_lin_7.0.so
#142524 0x000000000088a580 in ZEND_DO_FCALL_SPEC_HANDLER ()
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:800
#142525 0x000000000084c8fb in execute_ex (ex=<optimized out>)
---Type <return> to continue, or q <return> to quit---
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:414
#142526 0x00007fcbb111ef88 in ?? () from /usr/local/lib/ioncube/ioncube_loader_lin_7.0.so
#142527 0x000000000088a580 in ZEND_DO_FCALL_SPEC_HANDLER ()
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:800
#142528 0x000000000084c8fb in execute_ex (ex=<optimized out>)
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:414
#142529 0x00007fcbb111ef88 in ?? () from /usr/local/lib/ioncube/ioncube_loader_lin_7.0.so
#142530 0x000000000088a580 in ZEND_DO_FCALL_SPEC_HANDLER ()
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:800
#142531 0x000000000084c8fb in execute_ex (ex=<optimized out>)
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:414
#142532 0x00007fcbb111ef88 in ?? () from /usr/local/lib/ioncube/ioncube_loader_lin_7.0.so
#142533 0x000000000088a580 in ZEND_DO_FCALL_SPEC_HANDLER ()
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:800
#142534 0x000000000084c8fb in execute_ex (ex=<optimized out>)
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:414
#142535 0x00007fcbb111ef88 in ?? () from /usr/local/lib/ioncube/ioncube_loader_lin_7.0.so
#142536 0x000000000088a580 in ZEND_DO_FCALL_SPEC_HANDLER ()
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:800
#142537 0x000000000084c8fb in execute_ex (ex=<optimized out>)
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:414
#142538 0x00007fcbb111ef88 in ?? () from /usr/local/lib/ioncube/ioncube_loader_lin_7.0.so
#142539 0x000000000088a580 in ZEND_DO_FCALL_SPEC_HANDLER ()
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:800
#142540 0x000000000084c8fb in execute_ex (ex=<optimized out>)
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:414
#142541 0x00007fcbb111ef88 in ?? () from /usr/local/lib/ioncube/ioncube_loader_lin_7.0.so
#142542 0x0000000000892c86 in ZEND_INCLUDE_OR_EVAL_SPEC_CV_HANDLER ()
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:29565
#142543 0x000000000084c8fb in execute_ex (ex=<optimized out>)
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:414
#142544 0x00007fcbb111ef88 in ?? () from /usr/local/lib/ioncube/ioncube_loader_lin_7.0.so
#142545 0x000000000088a580 in ZEND_DO_FCALL_SPEC_HANDLER ()
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:800
---Type <return> to continue, or q <return> to quit---
#142546 0x000000000084c8fb in execute_ex (ex=<optimized out>)
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:414
#142547 0x00007fcbb111ef88 in ?? () from /usr/local/lib/ioncube/ioncube_loader_lin_7.0.so
#142548 0x000000000088a580 in ZEND_DO_FCALL_SPEC_HANDLER ()
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:800
#142549 0x000000000084c8fb in execute_ex (ex=<optimized out>)
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:414
#142550 0x00007fcbb111ef88 in ?? () from /usr/local/lib/ioncube/ioncube_loader_lin_7.0.so
#142551 0x000000000088a580 in ZEND_DO_FCALL_SPEC_HANDLER ()
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:800
#142552 0x000000000084c8fb in execute_ex (ex=<optimized out>)
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:414
#142553 0x00007fcbb111ef88 in ?? () from /usr/local/lib/ioncube/ioncube_loader_lin_7.0.so
#142554 0x000000000088a580 in ZEND_DO_FCALL_SPEC_HANDLER ()
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:800
#142555 0x000000000084c8fb in execute_ex (ex=<optimized out>)
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:414
#142556 0x00007fcbb111ef88 in ?? () from /usr/local/lib/ioncube/ioncube_loader_lin_7.0.so
#142557 0x000000000088a580 in ZEND_DO_FCALL_SPEC_HANDLER ()
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:800
#142558 0x000000000084c8fb in execute_ex (ex=<optimized out>)
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:414
#142559 0x00007fcbb111ef88 in ?? () from /usr/local/lib/ioncube/ioncube_loader_lin_7.0.so
#142560 0x000000000088a580 in ZEND_DO_FCALL_SPEC_HANDLER ()
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:800
#142561 0x000000000084c8fb in execute_ex (ex=<optimized out>)
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:414
#142562 0x00007fcbb111ef88 in ?? () from /usr/local/lib/ioncube/ioncube_loader_lin_7.0.so
#142563 0x000000000088a580 in ZEND_DO_FCALL_SPEC_HANDLER ()
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:800
#142564 0x000000000084c8fb in execute_ex (ex=<optimized out>)
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:414
#142565 0x00007fcbb111ef88 in ?? () from /usr/local/lib/ioncube/ioncube_loader_lin_7.0.so
#142566 0x000000000088a580 in ZEND_DO_FCALL_SPEC_HANDLER ()
---Type <return> to continue, or q <return> to quit---
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:800
#142567 0x000000000084c8fb in execute_ex (ex=<optimized out>)
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:414
#142568 0x00007fcbb111ef88 in ?? () from /usr/local/lib/ioncube/ioncube_loader_lin_7.0.so
#142569 0x000000000088a580 in ZEND_DO_FCALL_SPEC_HANDLER ()
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:800
#142570 0x000000000084c8fb in execute_ex (ex=<optimized out>)
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:414
#142571 0x00007fcbb111ef88 in ?? () from /usr/local/lib/ioncube/ioncube_loader_lin_7.0.so
#142572 0x000000000088a580 in ZEND_DO_FCALL_SPEC_HANDLER ()
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:800
#142573 0x000000000084c8fb in execute_ex (ex=<optimized out>)
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:414
#142574 0x00007fcbb111ef88 in ?? () from /usr/local/lib/ioncube/ioncube_loader_lin_7.0.so
#142575 0x000000000088a580 in ZEND_DO_FCALL_SPEC_HANDLER ()
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:800
#142576 0x000000000084c8fb in execute_ex (ex=<optimized out>)
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:414
#142577 0x00007fcbb111ef88 in ?? () from /usr/local/lib/ioncube/ioncube_loader_lin_7.0.so
#142578 0x000000000088a580 in ZEND_DO_FCALL_SPEC_HANDLER ()
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:800
#142579 0x000000000084c8fb in execute_ex (ex=<optimized out>)
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:414
#142580 0x00007fcbb111ef88 in ?? () from /usr/local/lib/ioncube/ioncube_loader_lin_7.0.so
#142581 0x0000000000896407 in zend_execute (op_array=0x7fcbb786b000, op_array@entry=0x7fcb70573960,
return_value=return_value@entry=0x7fcb59dcded0)
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend_vm_execute.h:458
#142582 0x0000000000810244 in zend_execute_scripts (type=type@entry=8, retval=0x7fcb59dcded0,
retval@entry=0x0, file_count=file_count@entry=3)
at /usr/local/directadmin/custombuild/php-7.0.21/Zend/zend.c:1443
#142583 0x00000000007b3cf0 in php_execute_script (primary_file=primary_file@entry=0x7ffe4a3ced60)
at /usr/local/directadmin/custombuild/php-7.0.21/main/main.c:2492
#142584 0x000000000044af9c in main (argc=<optimized out>, argv=<optimized out>)
at /usr/local/directadmin/custombuild/php-7.0.21/sapi/fpm/fpm/fpm_main.c:1967
Could it be due to some pcre limits? ulimit? We're kind of stuck here with whats going on. We suspect it could be some code thats causing this but unsure what? Or maybe a bug in one of the applications?
Thanks
