Mostly posting this as a rubber duck method and for my future self.
We have a RHEL 7.4 server running Apache 2.4 (httpd24-httpd-2.4.27-8.el7.x86_64) with mod_php (no php-fpm). Today, all of the sudden about 20 of our servers started failing in our PHP app with php_network_getaddresses: getaddrinfo failed:. Doing a systemctl restart httpd24-httpd
resolves the issue, (reload does not (opcache). If I monitor DNS resolution with tcpdump -i eth0 -vvv -nn udp port 53
I don't even see the DNS lookups being made, however I do see others. It appears that Apache isn't even making the DNS requests that PHP wants. These servers reboot daily and had all turned on around 7am or later so they weren't running that long (not some kind of buildup) and had been operating fine for months.
Our main concern now is that our production servers are running the same stack and although production has been fine I want to avoid this issue reaching production.
Why would Apache all of the sudden stop making DNS lookups? (It doesn't even make it to /etc/hosts)