I really like to work with several components in a system including linux kernel or keep my emotions on userspace.
If you work with android and your are a real engineer, it is very difficult to resist the native support using Android NDK. If your software requires performance using graphic API like in OpenGL or if you need to access some specific information provided by some native library, the NDK fits for you.
However, bugs in native side sometimes takes time and usually it is not an easy and fast task. If you have the device and an easy way to reproduce the issue, it is ok but suppose you need to collect logs from remote users and the scenario is very difficult to reproduce. Some cases the users even see the crash visually but the system contains the logs blocking the approval of your software that must be completed to some costumer.. at this point the program/project managers team are “talking” in your ears… “fix it!”.
Working with Android, every time a process that runs on native side crashes, we have some small pieces of your stack in files called tombstones.
The tombstones are located at /data/tombstones as isolated files (one files represents one crash) or you can see them in your logcat Take a look in the adb shell:
root@android:/ # find . |grep tombs ./data/tombstones ./data/tombstones/tombstone_00 ./data/tombstones/tombstone_01 root@android:/ #
The tombstone inform you about:
- Build fingerprint
- Crashed process and PIDs
- Terminated signal and fault address
- CPU registers
- Call stack
- Stack content of each call
I will not post a full tombstone here. Check the /data on your device and you will observe the 7 sessions mentioned above . Let’s go straight to the point.. how to debug the stack in tombstones files!
Debugging the stack… easier way
But sometimes looking to raw tombstones it is a little difficult to realize where and what the problem really is.
The intention of this post is to introduce two tools that comes with android NDK that really helps: ndk-stack and addr2line
Follow the steps below:
1) Install the NDK (if you do not have yet)
If you do not have the android ndk package you need to install in order to have access to the tool chains. Please download the package from this link and install on your computer.
I recommend you include your NDK directory in your path. If you are using linux here goes an example that you need to include in your .bashrc:
export ANDROID_NDK_PATH=/home/bytesthink/tools/android-ndk-r8d/ PATH= $ANDROID_NDK_PATH:$PATH
2) Execute ndk-stack tool
If you install everything, you should be able to run the tool ndk-stack
ndk-stack -sym <path> [-dump <path>] -sym Contains full path to the root directory for symbols. -dump Contains full path to the file containing the crash dump. This is an optional parameter. If ommited, ndk-stack will read input data from stdin See docs/NDK-STACK.html in your NDK installation tree for more details.
Very simple right ? Let’s run an example.. considering I pull a tombstone file from my system.
bytesthink@ubuntu:~/tools/android-ndk-r8d$ ./ndk-stack -sym ~/myrelease/symbols -dump ~/win/bytesthink/Documents/test/perform_releases/tombstone_01 ********** Crash dump: ********** Build fingerprint: 'version1239210121' pid: 19135, tid: 19135, name: erviceinterface >>> com.test.powermeterserver.serviceinterface <<< signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr deadd00d Stack frame #00 pc 00072c9a /system/lib/libdvm.so (dvmAbort+170) Stack frame #01 pc 00050f6b /system/lib/libdvm.so (ScopedCheck::checkInstanceFieldID(_jobject*, _jfieldID*)+379) Stack frame #02 pc 0005db56 /system/lib/libdvm.so (Check_SetBooleanField(_JNIEnv*, _jobject*, _jfieldID*, unsigned char)+150) Stack frame #03 pc 00003a93 /system/lib/libmytestt.so (Java_com_test_jni_MyLibWrapper_getMonitorSample+259) Stack frame #04 pc 0002d850 /system/lib/libdvm.so (dvmPlatformInvoke+80) Stack frame #05 pc 00085b48 /system/lib/libdvm.so (dvmCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*)+648) Stack frame #06 pc 000505db /system/lib/libdvm.so (dvmCheckCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*)+59) Stack frame #07 pc 0008b96d /system/lib/libdvm.so (dvmResolveNativeMethod(unsigned int const*, JValue*, Method const*, Thread*)+317) Stack frame #08 pc 00170ff1 /system/lib/libdvm.so Stack frame #09 pc 000402a6 /system/lib/libdvm.so (dvmMterpStd(Thread*)+70) Stack frame #10 pc 0003dc14 /system/lib/libdvm.so (dvmInterpret(Thread*, Method const*, JValue*)+212) Stack frame #11 pc 000b69db /system/lib/libdvm.so (dvmInvokeMethod(Object*, Method const*, ArrayObject*, ArrayObject*, ClassObject*, bool)+2011) Stack frame #12 pc 000c9f07 /system/lib/libdvm.so (Dalvik_java_lang_reflect_Method_invokeNative(unsigned int const*, JValue*)+295) Stack frame #13 pc 00170ff1 /system/lib/libdvm.so Stack frame #14 pc 000402a6 /system/lib/libdvm.so (dvmMterpStd(Thread*)+70) Stack frame #15 pc 0003dc14 /system/lib/libdvm.so (dvmInterpret(Thread*, Method const*, JValue*)+212) Stack frame #16 pc 000b575c /system/lib/libdvm.so (dvmCallMethodV(Thread*, Method const*, Object*, bool, JValue*, char*)+700) Stack frame #17 pc 00079b8e /system/lib/libdvm.so (CallStaticVoidMethodV(_JNIEnv*, _jclass*, _jmethodID*, char*)+78) Stack frame #18 pc 0005a020 /system/lib/libdvm.so (Check_CallStaticVoidMethodV(_JNIEnv*, _jclass*, _jmethodID*, char*)+432) Stack frame #19 pc 00057c5a /system/lib/libandroid_runtime.so (_JNIEnv::CallStaticVoidMethod(_jclass*, _jmethodID*, ...)+42) Stack frame #20 pc 00059351 /system/lib/libandroid_runtime.so (android::AndroidRuntime::start(char const*, char const*)+945) Stack frame #21 pc 00000fd0 /system/bin/app_process (main+528) Stack frame #22 pc 0001a579 /system/lib/libc.so (__libc_init+89) Stack frame #00 pc 0001ca31 /system/lib/libc.so Stack frame #01 pc 000e66e8 /system/lib/libdvm.so (gcDaemonThread(void*)+216) Stack frame #02 pc 00097ee0 /system/lib/libdvm.so (internalThreadStart(void*)+528) Stack frame #03 pc 0000deb8 /system/lib/libc.so (__thread_entry+248) Stack frame #04 pc 0001c951 /system/lib/libc.so Stack frame #05 pc 000021e0 /dev/ashmem/dalvik-LinearAlloc (deleted) Stack frame #06 pc 0000af42 /data/dalvik-cache/system@app@PowerManagementService.apk@classes.dex Stack frame #00 pc 0001c05b /system/lib/libc.so Stack frame #01 pc 0009163c /system/lib/libdvm.so (signalCatcherThreadStart(void*)+92) Stack frame #02 pc 00097ee0 /system/lib/libdvm.so (internalThreadStart(void*)+528) Stack frame #03 pc 0000deb8 /system/lib/libc.so (__thread_entry+248) Stack frame #04 pc 0001c951 /system/lib/libc.so Stack frame #05 pc 00002ff4 /system/lib/libdvm.so Stack frame #00 pc 0001c317 /system/lib/libc.so Stack frame #01 pc 000c012b /system/lib/libdvm.so (jdwpThreadStart(void*)+123)
Cool! Checking the stack trace we can see that my libmytestt.so crashed and the memory position is 00003a93
So, what do we have on this memory address ? Let’s go to step 3!
Step 3) Using the addr2line
This is another tool that comes with your android ndk toolchain but the name is not only addr2line as we have the standard linux distributions. If it is a special one and comes with your android ndk. For example, in my installation is:
i686-linux-android-addr2line
located at:
android-ndk-r8d/toolchains/x86-4.6/prebuilt/linux-x86/bin/
Feel free to create an alias to addr2line with whole path and correct name
alias addr2line='~/tools/android-ndk-r8d/toolchains/x86-4.6/prebuilt/linux-x86/bin/i686-linux-android-addr2line'
To run this command is quite simple. You can type “addr2line -?” in your linux.
Usage: /home/mcramon/tools/android-ndk-r8d/toolchains/x86-4.6/prebuilt/linux-x86/bin/i686-linux-android-addr2line [option(s)] [addr(s)] Convert addresses into line number/file name pairs. If no addresses are specified on the command line, they will be read from stdin The options are: @ Read options from -a --addresses Show addresses -b --target= Set the binary file format -e --exe= Set the input file name (default is a.out) -i --inlines Unwind inlined functions -j --section= Read section-relative offsets instead of addresses -p --pretty-print Make the output easier to read for humans -s --basenames Strip directory names -f --functions Show function names -C --demangle[=style] Demangle function names -h --help Display this information -v --version Display the program's version
In my case I will run the following command line specifying the lib that crashed and address:
bytesthink@ubuntu:~/tools/android-ndk-r8d$ addr2line -f -e ~/myrelease/symbols/system/lib/libmytestt.so 00003a93
the my output will be:
Java_com_test_jni_MyLibWrapper_getMonitorSample .../device/test/PRIVATE/log_infra/myservice/jni/com_test_jni_MyLibWrapper.c:189
So, now I know the file and line that your software crashed!
Good debuggings!!
Hi, what if I’m not using NDK in my project but 2 of the libs I use collide and there’s a crash? How do I debug that so that I can open an issue on github providing useful information?
Thanks!
hi Maik
These tools is part of NDK, so they are supposedly to be used with NDK 🙂 But anyway if you have android device, tombstone will be generated during your crash and these tools are applicable for your external libs as well.. but keep in mind, use need to have the same binaries (images with debug info) used when you had the crash. Otherwise the symbols will not match and the tools will not work. Try it! it will work.
Regards!
Manoel Ramon
Hi
-sym Contains full path to the root directory for symbols.
Can you please tell where the symbols directory is located.?
when you compile your kernel, usually you have an “out” directory. Your symbols must be there.
Very nice blog on TOMBSTONE. i am very much interested on ndk-stack trace.
i downloaded NDK and set the path in .bashrc ; in the execution “./ndk-stack -sym ~/myrelease/symbols” i didnt understand the symbols — as we are having images (boot.img bootloader.img , system.img etc. and vmlinux images) and we are having a local copy of full code base. So my doubt is for the -sym symbols either to give images path or source code.???. tombsotne file we are having that , i understood.
Please help me in this to use ndk-stack trace tool.
Hi,
Well explained. i’m new to Android Framework development. In Tombstone i’m getting list of .so occurance. here how can i conclude the particular .so file having this issue. kindly tell me. please correct me if am wrong here.
how you are pointing libmytestt.so is infected,
pc 00003a93 /system/lib/libmytestt.so
thank you.
Pingback: d rose 7 bleached lilac uk trainers
Pingback: uk trainers womens nike lebron soldier 11 blue white
Pingback: noir rose nike air zoom spiridon
Pingback: y3 slippers hyper jade uk trainers
Pingback: nike flyknit cortez date red
Pingback: violet marine nike star runner tdv kids
Pingback: adidas joggers khaki youtube.com
Pingback: volt r酶d kvinders air jordan rcvr
Pingback: 褋懈薪懈泄 lebron 9
Pingback: kobe 10 sneaker
Pingback: nike huarache purple and black glitter swoosh
Pingback: nike sb dunk low premium qs mia skate shop 10th year anniversary brv blue mtlc gld str mnrl bl 504750 474
Pingback: uk trainers hyperdunk x plum
Pingback: wholesale mens denver broncos royal retro full zip hoodie
Pingback: cheap real nike air max sequent 852465 002 online for sale
Pingback: nike zoom shoes nike zoom pegasus turbo
Pingback: adidas f50
Pingback: juicy couture jumpsuit new black xs velour
Pingback: h by hudson black suede ankle boots peak woven
Pingback: concepts x reebok insta pump fury arriving at additonal retailers
Pingback: winter mens faux leather cap warm hat baseball cap with ear flaps flat top caps men big size 61cm brown
Pingback: nike lebron 8 p.s. home
Pingback: fila fila long sleeve zip front leotard side logo deta
Pingback: nike boys air jordan 9 retro bp kobe bryant blanco tour amarillo
Pingback: 639 footwear mens shoes sneakers
Pingback: reebok black ventilator shoes
Pingback: chaussure nike air max 97 se pour femme
Pingback: real madrid vs leganes live stream watch online tv channel
Pingback: air jordan 8 retro 3peat
Pingback: rose gold canada nike sportswear shoes kids air force 1 se trainers metallic
Pingback: nike team matchfit core otc sock white black
Pingback: lsu snapback fanatics
Pingback: adidas w cc oasis lite ii womens golf spikes shoes white blue 737188 s 5 nwob
Pingback: santa cruz otter keychain tim ward
Pingback: womens grey dressing gown uk pinterest.com
Pingback: womens freedom high waisted lattice capri leggings c9 champion grapevine purple xs
Pingback: performance plastics chemware pfa wide mouth wash bottles saint gobain performance plastics chemware pfa wide mouth wash bottles
Pingback: on polarized lens sunglasses
Pingback: asics gel saga x mas dark greenburgundy
Pingback: 30mm rose gold round flat split key rings
Pingback: air force chica
Pingback: crystal skull mixed studs flip flop black wicked wonders vip bling
Pingback: palace tee shirt homme t shirt t shirt femme tops
Pingback: supreme x the north face blue mountain logo sweatshirt
Pingback: rayban 7142 prescription eyeglasses
Pingback: nike air max 90 hyp prm mens shoes 2015 light gray green black hot
Pingback: cheap ralph lauren outlet online
Pingback: caqui naranja asics whizzer lo
Pingback: succinct chuck taylor united kingdom women converse womens double tongue shoes size b m us spring summer shoes
Pingback: asics gel saga smoke us9.5 gr眉n
Pingback: nike air force 1 mcdonalds dames
Pingback: adidas d rose 5 price philippines
Pingback: jovani ivory ruched bodice long sleeve scuba cocktail dress 50160
Pingback: ravelry project gallery for gift wrap romper sweater pattern by carina spencer
Pingback: vpfashion mermaid c017 ombr茅 cool anna ms annamello photo beautylish
Pingback: vans full patch ii grey brown snapback hat
Pingback: inva 200 globatt inverter battery
Pingback: sandal puma original aqua cat ii evospeed full hitam
Pingback: uk sale women nike air max 2013 shoes dark blue nx84082
Pingback: nike air force 1 07 se lx w white black 1
Pingback: chaussures nike sportswear pas cher pour femme enfant blanc noir 749986 100
Pingback: rood the north face donsjack jassen voor mannen la paz mannen
Pingback: shopping guide to bangkok sports goods and shoes adidas nike
Pingback: nike low top shoes
Pingback: nextadidas predator lz trx
Pingback: Yeezy Boost
Pingback: nike roshe one trainers womens
Pingback: 2013 nike air red jordan flight club 90s sz 10.5 black red air grey 602661 004 df477d
Pingback: womens new balance charcoal running shoes 574 uk
Pingback: cheap air jordan 4 gs pure money white metallic silver pure platinum 308497
Pingback: zapatillas new balance running course mujer cargando zoom
Pingback: all black leather slip on vans
Pingback: adidas superstar super color light pink for sale
Pingback: adidas eqt cushion adv noir rose baskets ah2231
Pingback: image is loading under armour ysm youth small purple short sleeve
Pingback: mens nike 2014 grey white running shoes flyknit air max and
Pingback: iphone 4 case unique iphone 4s case and cute iphone 4 cover image
Pingback: nike hypervenom phantom iii df tf men鈥瞫 nike football shoes red gery
Pingback: anwar carrots x puma suede collection
Pingback: ty peek a boo phone holder with screen cleaner bottom walmart.com
Pingback: 2nd place south africa
Pingback: olive green hoodie saving adidas womens
Pingback: the flash kids adjustable snapback cap
Pingback: new style online sales womens shoes vans unisex old skool suede canvas skate shoe pewter vn 0a2z5wot4on salehigh end
Pingback: nike shortsleeve legend v neck t shirt 2.0 womens at champs sports
Pingback: reebok crossfit unisex crew socks black training wicking sports fitness socks
Pingback: nike air force 1 high team red white
Pingback: odd future black of donut beanie hbx
Pingback: samsung galaxy tab s2 8.0 full tablet specifications
Pingback: 36 raglan sleeve stripe trim skew neck t shirt
Pingback: lyst bottega veneta key wallet in brown for men
Pingback: nike mercurial superfly camo pack best
Pingback: exclusive sneakeroftheday 4 8 galaxy foamposite db aj 7s baron
Pingback: adidas yeezy 350 boost v2 bred running shoes 6 for sale
Pingback: mens short nike t shirt clothing dri fit cotton sleeved
Pingback: manteaux vestes homme bomber reversible burberry comparez et achetez
Pingback: womens cable knit winter knitted beanie faux fur detachable pom pom
Pingback: chaussures bateau en cuir pour homme tbs phenix jeans
Pingback: 銉偆銉淬偅銉堛兂 ouis vuitton 銉濄儷銉?銉撱偍 銈炽兂銉戙偗銉?浜屻仱鎶樸倞 璨″竷 銈ㄣ償
Pingback: fendi black patent leather cut out platform pumps
Pingback: wei脽 un眉bertroffen aldo traunna schuhe stoffschuhe herren gr枚脽e 40
Pingback: nike lunar air force 1 lo
Pingback: details about new balance m420 v3 mens premium running shoes fitness gym trainers grey
Pingback: mothercare 胁 谐褉邪薪写 泻邪薪褜芯薪械
Pingback: nike shorts dri fit air jordan basketball poshmark
Pingback: uk trainers adidas y 3 zazu triple black cg3158
Pingback: miunana 3 abiti vestiti