A bit of a weird problem that has popped up out of nowhere and will little to no information. I work in support of a PHP (Laravel) application that stores asset files for our users. The application runs on across a few EC2 instances, and the assets are stored in EFS mounts.
The other day, one of our core routes which is responsible for triggering the server to copy a directory from A to B began to take a significant amount of time (from 5s to 90s) with no changes to the contents or application code.
I was able to isolate the delay to an xcopy call that moves 1 folder on the EFS to another folder in the same EFS. When I reached out to Amazon for assistance looking at the EFS metrics in CloudWatch, they indicated that everything looked strong, but they noted the fall off. Throughput is provisioned at 10Mb/s, and I've only spiked at 4-5Mb/s in the past few days.
Using NetData, I'm able to observe performance of RAM, CPU, NFS Client calls, and much more. From what I can tell, the NFS client is taking its sweet time. When testing locally without the EFS, the call is very speedy. I tried to modify the app code so that rather than a PHP copy, it would perform a system call to cp. This didn't seem to work.
Has anyone else experienced random latency issues like this before when using EFS? Does anyone have any recommendations on debugging techniques to see why this might be happening?
Thank you for reading and information you can offer up.
