Debugging tombstones with ndk-stack and addr2line

android

I really like to work with several components in a system including linux kernel or keep my emotions on userspace.

If you work with android and your are a real engineer, it is very difficult to resist the native support using Android NDK. If your software requires performance using graphic API like in OpenGL or if you need to access some specific information provided by some native library, the NDK fits for you.

However, bugs in native side sometimes takes time and usually it is not an easy and fast task. If you have the device and an easy way to reproduce the issue, it is ok but suppose you need to collect logs from remote users and the scenario is very difficult to reproduce. Some cases the users even see the crash visually but the system contains the logs blocking the approval of your software that must be completed to some costumer.. at this point the program/project managers team are “talking” in your ears… “fix it!”.

Working with Android, every time a process that runs on native side crashes, we have some small pieces of your stack in files called tombstones.

The tombstones are located at /data/tombstones as isolated files (one files represents one crash) or you can see them in your logcat Take a look in the adb shell:

root@android:/ # find . |grep tombs

./data/tombstones
./data/tombstones/tombstone_00
./data/tombstones/tombstone_01
root@android:/ #

The tombstone inform you about:

  1. Build fingerprint
  2. Crashed process and PIDs
  3. Terminated signal and fault address
  4. CPU registers
  5. Call stack
  6. Stack content of each call

I will not post a full tombstone here. Check the /data on your device and you will observe the 7 sessions mentioned above . Let’s go straight to the point.. how to debug the stack in tombstones files!

Debugging the stack…  easier way

But sometimes looking to raw tombstones it is a little difficult to realize where and what the problem really is.

The intention of this post is to introduce two tools that comes with android NDK that really helps: ndk-stack and addr2line

Follow the steps below:

1) Install the NDK (if you do not have yet)

If you do not have the android ndk package you need to install in order to have access to the tool chains. Please download the package from this link and install on your computer.

I recommend you include your NDK directory in your path. If you are using linux here goes an example that you need to include in your .bashrc:

export ANDROID_NDK_PATH=/home/bytesthink/tools/android-ndk-r8d/

PATH= $ANDROID_NDK_PATH:$PATH

2) Execute ndk-stack tool

If you install everything, you should be able to run the tool ndk-stack

ndk-stack -sym <path> [-dump <path>]

   -sym  Contains full path to the root directory for symbols.
   -dump Contains full path to the file containing the crash dump.
         This is an optional parameter. If ommited, ndk-stack will
         read input data from stdin

See docs/NDK-STACK.html in your NDK installation tree for more details.

Very simple right ? Let’s run an example.. considering I pull a tombstone file from my system.

bytesthink@ubuntu:~/tools/android-ndk-r8d$ ./ndk-stack -sym ~/myrelease/symbols -dump ~/win/bytesthink/Documents/test/perform_releases/tombstone_01
********** Crash dump: **********
Build fingerprint: 'version1239210121'
pid: 19135, tid: 19135, name: erviceinterface >>> com.test.powermeterserver.serviceinterface <<<
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr deadd00d
Stack frame #00 pc 00072c9a /system/lib/libdvm.so (dvmAbort+170)
Stack frame #01 pc 00050f6b /system/lib/libdvm.so (ScopedCheck::checkInstanceFieldID(_jobject*, _jfieldID*)+379)
Stack frame #02 pc 0005db56 /system/lib/libdvm.so (Check_SetBooleanField(_JNIEnv*, _jobject*, _jfieldID*, unsigned char)+150)
Stack frame #03 pc 00003a93 /system/lib/libmytestt.so (Java_com_test_jni_MyLibWrapper_getMonitorSample+259)
Stack frame #04 pc 0002d850 /system/lib/libdvm.so (dvmPlatformInvoke+80)
Stack frame #05 pc 00085b48 /system/lib/libdvm.so (dvmCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*)+648)
Stack frame #06 pc 000505db /system/lib/libdvm.so (dvmCheckCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*)+59)
Stack frame #07 pc 0008b96d /system/lib/libdvm.so (dvmResolveNativeMethod(unsigned int const*, JValue*, Method const*, Thread*)+317)
Stack frame #08 pc 00170ff1 /system/lib/libdvm.so
Stack frame #09 pc 000402a6 /system/lib/libdvm.so (dvmMterpStd(Thread*)+70)
Stack frame #10 pc 0003dc14 /system/lib/libdvm.so (dvmInterpret(Thread*, Method const*, JValue*)+212)
Stack frame #11 pc 000b69db /system/lib/libdvm.so (dvmInvokeMethod(Object*, Method const*, ArrayObject*, ArrayObject*, ClassObject*, bool)+2011)
Stack frame #12 pc 000c9f07 /system/lib/libdvm.so (Dalvik_java_lang_reflect_Method_invokeNative(unsigned int const*, JValue*)+295)
Stack frame #13 pc 00170ff1 /system/lib/libdvm.so
Stack frame #14 pc 000402a6 /system/lib/libdvm.so (dvmMterpStd(Thread*)+70)
Stack frame #15 pc 0003dc14 /system/lib/libdvm.so (dvmInterpret(Thread*, Method const*, JValue*)+212)
Stack frame #16 pc 000b575c /system/lib/libdvm.so (dvmCallMethodV(Thread*, Method const*, Object*, bool, JValue*, char*)+700)
Stack frame #17 pc 00079b8e /system/lib/libdvm.so (CallStaticVoidMethodV(_JNIEnv*, _jclass*, _jmethodID*, char*)+78)
Stack frame #18 pc 0005a020 /system/lib/libdvm.so (Check_CallStaticVoidMethodV(_JNIEnv*, _jclass*, _jmethodID*, char*)+432)
Stack frame #19 pc 00057c5a /system/lib/libandroid_runtime.so (_JNIEnv::CallStaticVoidMethod(_jclass*, _jmethodID*, ...)+42)
Stack frame #20 pc 00059351 /system/lib/libandroid_runtime.so (android::AndroidRuntime::start(char const*, char const*)+945)
Stack frame #21 pc 00000fd0 /system/bin/app_process (main+528)
Stack frame #22 pc 0001a579 /system/lib/libc.so (__libc_init+89)
Stack frame #00 pc 0001ca31 /system/lib/libc.so
Stack frame #01 pc 000e66e8 /system/lib/libdvm.so (gcDaemonThread(void*)+216)
Stack frame #02 pc 00097ee0 /system/lib/libdvm.so (internalThreadStart(void*)+528)
Stack frame #03 pc 0000deb8 /system/lib/libc.so (__thread_entry+248)
Stack frame #04 pc 0001c951 /system/lib/libc.so
Stack frame #05 pc 000021e0 /dev/ashmem/dalvik-LinearAlloc (deleted)
Stack frame #06 pc 0000af42 /data/dalvik-cache/system@app@PowerManagementService.apk@classes.dex
Stack frame #00 pc 0001c05b /system/lib/libc.so
Stack frame #01 pc 0009163c /system/lib/libdvm.so (signalCatcherThreadStart(void*)+92)
Stack frame #02 pc 00097ee0 /system/lib/libdvm.so (internalThreadStart(void*)+528)
Stack frame #03 pc 0000deb8 /system/lib/libc.so (__thread_entry+248)
Stack frame #04 pc 0001c951 /system/lib/libc.so
Stack frame #05 pc 00002ff4 /system/lib/libdvm.so
Stack frame #00 pc 0001c317 /system/lib/libc.so
Stack frame #01 pc 000c012b /system/lib/libdvm.so (jdwpThreadStart(void*)+123)

Cool! Checking the stack trace we can see that my libmytestt.so crashed and the memory position is 00003a93

So, what do we have on this memory address ? Let’s go to step 3!

Step 3) Using the addr2line

This is another tool that comes with your android ndk toolchain but the name is not only addr2line as we have the standard linux distributions. If it is a special one and comes with your android ndk. For example, in my installation is:

i686-linux-android-addr2line

located at:

android-ndk-r8d/toolchains/x86-4.6/prebuilt/linux-x86/bin/

Feel free to create an alias to addr2line with whole path and correct name

alias addr2line='~/tools/android-ndk-r8d/toolchains/x86-4.6/prebuilt/linux-x86/bin/i686-linux-android-addr2line'

To run this command is quite simple. You can type “addr2line -?” in your linux.

Usage: /home/mcramon/tools/android-ndk-r8d/toolchains/x86-4.6/prebuilt/linux-x86/bin/i686-linux-android-addr2line [option(s)] [addr(s)]
 Convert addresses into line number/file name pairs.
 If no addresses are specified on the command line, they will be read from stdin
 The options are:
  @                Read options from 
  -a --addresses         Show addresses
  -b --target=  Set the binary file format
  -e --exe=  Set the input file name (default is a.out)
  -i --inlines           Unwind inlined functions
  -j --section=    Read section-relative offsets instead of addresses
  -p --pretty-print      Make the output easier to read for humans
  -s --basenames         Strip directory names
  -f --functions         Show function names
  -C --demangle[=style]  Demangle function names
  -h --help              Display this information
  -v --version           Display the program's version

In my case I will run the following command line specifying the lib that crashed and address:

bytesthink@ubuntu:~/tools/android-ndk-r8d$ addr2line -f -e ~/myrelease/symbols/system/lib/libmytestt.so 00003a93

the my output will be:

Java_com_test_jni_MyLibWrapper_getMonitorSample
.../device/test/PRIVATE/log_infra/myservice/jni/com_test_jni_MyLibWrapper.c:189

So, now I know the file and line that your software crashed!

Good debuggings!!

113 thoughts on “Debugging tombstones with ndk-stack and addr2line

  1. Hi, what if I’m not using NDK in my project but 2 of the libs I use collide and there’s a crash? How do I debug that so that I can open an issue on github providing useful information?

    Thanks!

    • hi Maik

      These tools is part of NDK, so they are supposedly to be used with NDK 🙂 But anyway if you have android device, tombstone will be generated during your crash and these tools are applicable for your external libs as well.. but keep in mind, use need to have the same binaries (images with debug info) used when you had the crash. Otherwise the symbols will not match and the tools will not work. Try it! it will work.

      Regards!
      Manoel Ramon

  2. Very nice blog on TOMBSTONE. i am very much interested on ndk-stack trace.
    i downloaded NDK and set the path in .bashrc ; in the execution “./ndk-stack -sym ~/myrelease/symbols” i didnt understand the symbols — as we are having images (boot.img bootloader.img , system.img etc. and vmlinux images) and we are having a local copy of full code base. So my doubt is for the -sym symbols either to give images path or source code.???. tombsotne file we are having that , i understood.

    Please help me in this to use ndk-stack trace tool.

  3. Hi,

    Well explained. i’m new to Android Framework development. In Tombstone i’m getting list of .so occurance. here how can i conclude the particular .so file having this issue. kindly tell me. please correct me if am wrong here.

    how you are pointing libmytestt.so is infected,
    pc 00003a93 /system/lib/libmytestt.so

    thank you.

  4. Pingback: d rose 7 bleached lilac uk trainers

  5. Pingback: uk trainers womens nike lebron soldier 11 blue white

  6. Pingback: noir rose nike air zoom spiridon

  7. Pingback: y3 slippers hyper jade uk trainers

  8. Pingback: nike flyknit cortez date red

  9. Pingback: violet marine nike star runner tdv kids

  10. Pingback: adidas joggers khaki youtube.com

  11. Pingback: volt r酶d kvinders air jordan rcvr

  12. Pingback: 褋懈薪懈泄 lebron 9

  13. Pingback: kobe 10 sneaker

  14. Pingback: nike huarache purple and black glitter swoosh

  15. Pingback: nike sb dunk low premium qs mia skate shop 10th year anniversary brv blue mtlc gld str mnrl bl 504750 474

  16. Pingback: uk trainers hyperdunk x plum

  17. Pingback: wholesale mens denver broncos royal retro full zip hoodie

  18. Pingback: cheap real nike air max sequent 852465 002 online for sale

  19. Pingback: nike zoom shoes nike zoom pegasus turbo

  20. Pingback: adidas f50

  21. Pingback: juicy couture jumpsuit new black xs velour

  22. Pingback: h by hudson black suede ankle boots peak woven

  23. Pingback: concepts x reebok insta pump fury arriving at additonal retailers

  24. Pingback: winter mens faux leather cap warm hat baseball cap with ear flaps flat top caps men big size 61cm brown

  25. Pingback: nike lebron 8 p.s. home

  26. Pingback: fila fila long sleeve zip front leotard side logo deta

  27. Pingback: nike boys air jordan 9 retro bp kobe bryant blanco tour amarillo

  28. Pingback: 639 footwear mens shoes sneakers

  29. Pingback: reebok black ventilator shoes

  30. Pingback: chaussure nike air max 97 se pour femme

  31. Pingback: real madrid vs leganes live stream watch online tv channel

  32. Pingback: air jordan 8 retro 3peat

  33. Pingback: rose gold canada nike sportswear shoes kids air force 1 se trainers metallic

  34. Pingback: nike team matchfit core otc sock white black

  35. Pingback: lsu snapback fanatics

  36. Pingback: adidas w cc oasis lite ii womens golf spikes shoes white blue 737188 s 5 nwob

  37. Pingback: santa cruz otter keychain tim ward

  38. Pingback: womens grey dressing gown uk pinterest.com

  39. Pingback: womens freedom high waisted lattice capri leggings c9 champion grapevine purple xs

  40. Pingback: performance plastics chemware pfa wide mouth wash bottles saint gobain performance plastics chemware pfa wide mouth wash bottles

  41. Pingback: on polarized lens sunglasses

  42. Pingback: asics gel saga x mas dark greenburgundy

  43. Pingback: 30mm rose gold round flat split key rings

  44. Pingback: air force chica

  45. Pingback: crystal skull mixed studs flip flop black wicked wonders vip bling

  46. Pingback: palace tee shirt homme t shirt t shirt femme tops

  47. Pingback: supreme x the north face blue mountain logo sweatshirt

  48. Pingback: rayban 7142 prescription eyeglasses

  49. Pingback: nike air max 90 hyp prm mens shoes 2015 light gray green black hot

  50. Pingback: cheap ralph lauren outlet online

  51. Pingback: caqui naranja asics whizzer lo

  52. Pingback: succinct chuck taylor united kingdom women converse womens double tongue shoes size b m us spring summer shoes

  53. Pingback: asics gel saga smoke us9.5 gr眉n

  54. Pingback: nike air force 1 mcdonalds dames

  55. Pingback: adidas d rose 5 price philippines

  56. Pingback: jovani ivory ruched bodice long sleeve scuba cocktail dress 50160

  57. Pingback: ravelry project gallery for gift wrap romper sweater pattern by carina spencer

  58. Pingback: vpfashion mermaid c017 ombr茅 cool anna ms annamello photo beautylish

  59. Pingback: vans full patch ii grey brown snapback hat

  60. Pingback: inva 200 globatt inverter battery

  61. Pingback: sandal puma original aqua cat ii evospeed full hitam

  62. Pingback: uk sale women nike air max 2013 shoes dark blue nx84082

  63. Pingback: nike air force 1 07 se lx w white black 1

  64. Pingback: chaussures nike sportswear pas cher pour femme enfant blanc noir 749986 100

  65. Pingback: rood the north face donsjack jassen voor mannen la paz mannen

  66. Pingback: shopping guide to bangkok sports goods and shoes adidas nike

  67. Pingback: nike low top shoes

  68. Pingback: nextadidas predator lz trx

  69. Pingback: Yeezy Boost

  70. Pingback: nike roshe one trainers womens

  71. Pingback: 2013 nike air red jordan flight club 90s sz 10.5 black red air grey 602661 004 df477d

  72. Pingback: womens new balance charcoal running shoes 574 uk

  73. Pingback: cheap air jordan 4 gs pure money white metallic silver pure platinum 308497

  74. Pingback: zapatillas new balance running course mujer cargando zoom

  75. Pingback: all black leather slip on vans

  76. Pingback: adidas superstar super color light pink for sale

  77. Pingback: adidas eqt cushion adv noir rose baskets ah2231

  78. Pingback: image is loading under armour ysm youth small purple short sleeve

  79. Pingback: mens nike 2014 grey white running shoes flyknit air max and

  80. Pingback: iphone 4 case unique iphone 4s case and cute iphone 4 cover image

  81. Pingback: nike hypervenom phantom iii df tf men鈥瞫 nike football shoes red gery

  82. Pingback: anwar carrots x puma suede collection

  83. Pingback: ty peek a boo phone holder with screen cleaner bottom walmart.com

  84. Pingback: 2nd place south africa

  85. Pingback: olive green hoodie saving adidas womens

  86. Pingback: the flash kids adjustable snapback cap

  87. Pingback: new style online sales womens shoes vans unisex old skool suede canvas skate shoe pewter vn 0a2z5wot4on salehigh end

  88. Pingback: nike shortsleeve legend v neck t shirt 2.0 womens at champs sports

  89. Pingback: reebok crossfit unisex crew socks black training wicking sports fitness socks

  90. Pingback: nike air force 1 high team red white

  91. Pingback: odd future black of donut beanie hbx

  92. Pingback: samsung galaxy tab s2 8.0 full tablet specifications

  93. Pingback: 36 raglan sleeve stripe trim skew neck t shirt

  94. Pingback: lyst bottega veneta key wallet in brown for men

  95. Pingback: nike mercurial superfly camo pack best

  96. Pingback: exclusive sneakeroftheday 4 8 galaxy foamposite db aj 7s baron

  97. Pingback: adidas yeezy 350 boost v2 bred running shoes 6 for sale

  98. Pingback: mens short nike t shirt clothing dri fit cotton sleeved

  99. Pingback: manteaux vestes homme bomber reversible burberry comparez et achetez

  100. Pingback: womens cable knit winter knitted beanie faux fur detachable pom pom

  101. Pingback: chaussures bateau en cuir pour homme tbs phenix jeans

  102. Pingback: 銉偆銉淬偅銉堛兂 ouis vuitton 銉濄儷銉?銉撱偍 銈炽兂銉戙偗銉?浜屻仱鎶樸倞 璨″竷 銈ㄣ償

  103. Pingback: fendi black patent leather cut out platform pumps

  104. Pingback: wei脽 un眉bertroffen aldo traunna schuhe stoffschuhe herren gr枚脽e 40

  105. Pingback: nike lunar air force 1 lo

  106. Pingback: details about new balance m420 v3 mens premium running shoes fitness gym trainers grey

  107. Pingback: mothercare 胁 谐褉邪薪写 泻邪薪褜芯薪械

  108. Pingback: nike shorts dri fit air jordan basketball poshmark

  109. Pingback: uk trainers adidas y 3 zazu triple black cg3158

  110. Pingback: miunana 3 abiti vestiti

Leave a Reply