Sometimes, you just have to dig in!
I’ve always been happiest when I can dig all the way down to the very bottom of a software problem. For the last couple of years I’ve been writing lots of PHP, where I can remain more or less ignorant of what’s happening internally. However, it is still nice to be able to fire up a low-level debugger and take a look around.
Case in point, I have a daemon (background process) running on the Syndic8 server. It reads URLs from a System V message queue, fetches and parses the page, and looks for interesting items on the page. 99% of the time this process runs fine. Sometimes, however, it would get stuck and I didn’t know why.
It happened again today, and I had a few spare minutes to dig in. So I used ps to get the Process Id, then I invoked GDB like this:
gdb /usr/local/bin/php 10942
I used the **where **command to get a stack trace. The top entries on the stack looked like this:
`#0 0x00b0fc32 in dlsysinfo_int80 () from /lib/ld-linux.so.2</p>
1 0x00bfbaed in __newselectnocancel () from /lib/tls/libc.so.6`
The word “select” in the second entry caught my eye, since it is used to wait for an event to occur on a file descriptor. If the select is used without an accompanying timeout, it may very well wait forever.
Up a few more levels on the stack, and I found this:
#5 0x081337ec in php<em>stream</em>url<em>wrap</em>http<em>ex (wrapper=0x9c08280,
path=0x9ef11fc "http://www.amaamy.com/blog", mode=0xa1f990c "r",
options=4, opened</em>path=0x0, context=0x0, redirect<em>max=19, header</em>init=1)
at /home/downloads/Apachetoolbox-1.5.70/src/php-4.3.6/ext/standard/http<em>fopen</em>wrapper.c:492
Hmmm, fopen, that’s interesting. I never use fopen to connect to web sites; I prefer cURL and the fine grained control that comes along with it.
So now I had to know what code was making this call. I went up the stack a few more levels until I was inside of the PHP interpreter. From previous experience, this is a function named execute. I got there and then took a look at the local variables:
(gdb) inf loc
calling<em>symbol</em>table = (HashTable *) 0x9c6e4bc
original_return_value = (zval **) 0xbff36684
execute_data = {opline = 0x9c71d1c, function_state = {
function_symbol_table = 0x9c809e4, function = 0x9c78278, reserved = {
0x8184c05, 0x837ec4c, 0x9c6fba4, 0xb}}, fbc = 0x9c78278, ce = 0x0,
object = {ptr = 0x0}, Ts = 0xbff34710, original_in_execution = 1 '
After a bit more digging I found the data structure containing the name of the PHP function, and I was all set. It turns out that the fopen call is made from a third party HTML parsing library that I use. Since it is open, I can fix it or work around it.
Now I could have tried to solve this problem any number of ways. I could have added a bunch of print statements, or I could have tried to install a PHP debugger. But I have been using GDB for a long, long time and it was the tool at hand. I spent about 3 minutes debugging and about 20 blogging about it.