Symbolizing backtraces

If a release build of mod_pagespeed crashes, the .so file is stripped and so not a lot of information is available in the backtrace:

(gdb) bt
#0  0x00007f54cfd95625 in raise () from /lib64/libc.so.6
#1  0x00007f54cfd96e05 in abort () from /lib64/libc.so.6
#2  0x00007f54cad1fa5d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
#3  0x00007f54cad1dbe6 in ?? () from /usr/lib64/libstdc++.so.6
#4  0x00007f54cad1dc13 in std::terminate() () from /usr/lib64/libstdc++.so.6
#5  0x00007f54cad1dd0e in __cxa_throw () from /usr/lib64/libstdc++.so.6
#6  0x00007f54cacc2837 in std::__throw_logic_error(char const*) () from /usr/lib64/libstdc++.so.6
#7  0x00007f54cacfde59 in ?? () from /usr/lib64/libstdc++.so.6
#8  0x00007f54cacfdf33 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) () from /usr/lib64/libstdc++.so.6
#9  0x00007f54c9fce073 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#10 0x00007f54c9effb9a in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#11 0x00007f54c9f01e34 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#12 0x00007f54c9f0b3a4 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#13 0x00007f54d1860a58 in ap_send_error_response ()
#14 0x00007f54d1848506 in ap_read_request ()
#15 0x00007f54d18600e0 in ?? ()
#16 0x00007f54d185c148 in ap_run_process_connection ()
#17 0x00007f54d1869052 in ?? ()
#18 0x00007f54d00fe9d1 in start_thread () from /lib64/libpthread.so.0
#19 0x00007f54cfe4b8fd in clone () from /lib64/libc.so.6

Notice the relevant lines:

#9  0x00007f54c9fce073 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#10 0x00007f54c9effb9a in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#11 0x00007f54c9f01e34 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so
#12 0x00007f54c9f0b3a4 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed.so

Obtaining an unstripped library

In order to symbolize, you will need an unstripped copy of the .so. Look for the mod_pagespeed release number in the crash reporter and download the corresponding unstripped file from releases on GitHub. If the file isn't there for your release, you'll need to ask a Googler to retrieve it from the build machine.

The .tar.gz contains 4 files, for each combination of 32- vs 64-bit and Centos (rpm) or Ubuntu(deb). If you don't know, the crash reporter should contain information to help you pick the right one.

Finding the base address

You will also need the base address of the library. The simplest way to retrieve the base address in gdb is:

info proc mappings

This will display where the libraries are mapped (showing the base address), however it does require that /proc to be available.

If /proc is missing

If /proc is not mounted, then we can use info sharedlibrary. Unlike info proc mappings, the "from" address in info sharedlibrary is the start of the .text section. We must do a little math to compute the offset between the "from" address and the base address. We can use readelf or rabin2.

$ readelf -S  ./libmod_pagespeed.so | grep .text
  [12] .text             PROGBITS         0000000000078840  00078840

$ rabin2 -S ./libmod_pagespeed.so | grep .text
idx=11 vaddr=0x00078840 paddr=0x00078840 sz=4311238 vsz=4311238 perm=--r-x name=.text

The base address is be the "from" address minus the address of the .text section. In this case, the base address is 7f54c9e80000:

[1] pry(main)> 0x00007f54c9ef8840 - 0x00078840
=> 140002141405184
[2] pry(main)> _.to_s 16
=> "7f54c9e80000"

Symbolize!

Now that you have the base address, an unstripped binary, and the addresses in the backtrace, it's time to symbolize. Either addr2line or radare2 can do the work, we just need to feed them the addresses from our stack trace minus the base address. Below are a couple of short ruby scripts to do just that.

addr2line

Create a bt.rb script like this, substituting addresses from your stack trace and base_address with what you calculated above:

addresses = [0x00007f54c9fce073, 0x00007f54c9effb9a,
             0x00007f54c9f01e34, 0x00007f54c9f0b3a4]

base_address = 0x7f54c9e80000

addresses.each{|address|
  offset = "0x#{(address - base_address).to_s 16}" # note must be hex
  exec = "addr2line -f -e ./libmod_pagespeed.so #{offset} --demangle"
  puts `#{exec}`
}

Now run it:

$ ruby ./bt.rb
net_instaweb::GoogleUrl::GoogleUrl(char const*)
:?
net_instaweb::InstawebContext::MakeRequestUrl(net_instaweb::RewriteOptions const&, request_rec*)
:?
net_instaweb::InstawebHandler::InstawebHandler(request_rec*)
:?
net_instaweb::(anonymous namespace)::instaweb_out_filter(ap_filter_t*, apr_bucket_brigade*)
mod_instaweb.cc:?

radare2

Here's the same thing using radare2. Again, you must change addresses and base_address to match your own results.

require 'r2pipe'

addresses = [0x00007f54c9fce073, 0x00007f54c9effb9a,
             0x00007f54c9f01e34, 0x00007f54c9f0b3a4]

base_address = 0x7f54c9e80000

r2p = R2Pipe.new "./libmod_pagespeed.so"

addresses.each{|address|
  offset = address - base_address
  puts r2p.cmd "fd #{offset}"
}

And the result:

$ ruby ./bt.rb
sym.net_instaweb::GoogleUrl::Relativize + 851
sym.net_instaweb::InstawebContext::MakeRequestUrl + 282
sym.net_instaweb::InstawebHandler::InstawebHandler + 436
sym.net_instaweb::_anonymousnamespace_::instaweb_out_filter + 1012

the output from radare2 seems a bit better than from addr2line.