Updating, recompiling, VirtualHost templates, customizations, php safemode …Using strace to debug what apache/php are doing.

If you Trying to find a slowdown in your website and not sure where it is?
Example: WordPress loads slowly and you do not know why.
Use strace!

The idea is comes from that strace that can dump all binary calls to a log, so we will dump them to disk and sift through them later.
You can also use all this method for other processes, just swap “httpd” with some other process name in the command below.

Note: this is a fairly high-level debug method for advanced administrators. Root ssh access is requiring.

1. Run strace to log:

   cd /root
   mkdir straces
   cd straces
   ps ax | grep httpd | grep -v grep | awk ‘{ print “strace -f -s500 -o strace.”$1”.log -p “$1″&” }’ | sh

2. Trigger the slow case.  Load the website that is slowly, however you need to duplicate this issue.  Note, it might be best to highlight it in the URL bar and hit enter, instead of using F5. The reason is we only intend to load the site, and not the including images/css/js, etc.. and F5 or ctrl-F5 may reload everything, causing the logs to fill with more info, making the cause harder to find.
3. End the strace, open a 2nd console and run:
   

   killall -9 strace

   You will now have several strace.*.log files, one for each httpd process that was running.  Note, if you’re debugging a slowdown (or possibly socket timeout), hit ctrl-c before the timeout happens, but enough time where you can let it chew on things for a bit.  This way, is the last “chunks” of code that causes the slowdown and will be near the end of the log, making it easier to track.

4. You will need to find which PID is handling your request, so grep out your IP from the logs to figure that out. Change 1.2.3.4 with your actual IP address:
   

   grep 1.2.3.4 *.log

   This will dump some messy code, but on the far left should be which files that code came from.  Let’s say it shows the output in strace.29622.log (your PID number will be different from 29622). Also there could be multiple files, depending on how many requests were made and how apache handles them.  We are only concerning with the “slow” code, and not the other things like images, so you will need to go through each one to figure out which is which, to get at the important bits you are after.

5. From there, you can go through the logs, looking at what was happening, eg:
   

   less strace.29622.log

   This is going to be show you quite a lot of code, but you can also search with the / character.

   You can alternatively “grep” for things, eg:

   grep somethingspecific strace.29622.log

   which only shows the lines you’re looking for.

From this point, what you are looking for, how you find it, and what you do with it would be up to you.