Debugging tombstones with ndk-stack and addr2line

android

I really like to work with several components in a system including linux kernel or keep my emotions on userspace.

If you work with android and your are a real engineer, it is very difficult to resist the native support using Android NDK. If your software requires performance using graphic API like in OpenGL or if you need to access some specific information provided by some native library, the NDK fits for you.

However, bugs in native side sometimes takes time and usually it is not an easy and fast task. If you have the device and an easy way to reproduce the issue, it is ok but suppose you need to collect logs from remote users and the scenario is very difficult to reproduce. Some cases the users even see the crash visually but the system contains the logs blocking the approval of your software that must be completed to some costumer.. at this point the program/project managers team are “talking” in your ears… “fix it!”.

Working with Android, every time a process that runs on native side crashes, we have some small pieces of your stack in files called tombstones.

The tombstones are located at /data/tombstones as isolated files (one files represents one crash) or you can see them in your logcat Take a look in the adb shell:

root@android:/ # find . |grep tombs

./data/tombstones
./data/tombstones/tombstone_00
./data/tombstones/tombstone_01
root@android:/ #

The tombstone inform you about:

  1. Build fingerprint
  2. Crashed process and PIDs
  3. Terminated signal and fault address
  4. CPU registers
  5. Call stack
  6. Stack content of each call

I will not post a full tombstone here. Check the /data on your device and you will observe the 7 sessions mentioned above . Let’s go straight to the point.. how to debug the stack in tombstones files!

Debugging the stack…  easier way

But sometimes looking to raw tombstones it is a little difficult to realize where and what the problem really is.

The intention of this post is to introduce two tools that comes with android NDK that really helps: ndk-stack and addr2line

Follow the steps below:

1) Install the NDK (if you do not have yet)

If you do not have the android ndk package you need to install in order to have access to the tool chains. Please download the package from this link and install on your computer.

I recommend you include your NDK directory in your path. If you are using linux here goes an example that you need to include in your .bashrc:

export ANDROID_NDK_PATH=/home/bytesthink/tools/android-ndk-r8d/

PATH= $ANDROID_NDK_PATH:$PATH

2) Execute ndk-stack tool

If you install everything, you should be able to run the tool ndk-stack

ndk-stack -sym <path> [-dump <path>]

   -sym  Contains full path to the root directory for symbols.
   -dump Contains full path to the file containing the crash dump.
         This is an optional parameter. If ommited, ndk-stack will
         read input data from stdin

See docs/NDK-STACK.html in your NDK installation tree for more details.

Very simple right ? Let’s run an example.. considering I pull a tombstone file from my system.

bytesthink@ubuntu:~/tools/android-ndk-r8d$ ./ndk-stack -sym ~/myrelease/symbols -dump ~/win/bytesthink/Documents/test/perform_releases/tombstone_01
********** Crash dump: **********
Build fingerprint: 'version1239210121'
pid: 19135, tid: 19135, name: erviceinterface >>> com.test.powermeterserver.serviceinterface <<<
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr deadd00d
Stack frame #00 pc 00072c9a /system/lib/libdvm.so (dvmAbort+170)
Stack frame #01 pc 00050f6b /system/lib/libdvm.so (ScopedCheck::checkInstanceFieldID(_jobject*, _jfieldID*)+379)
Stack frame #02 pc 0005db56 /system/lib/libdvm.so (Check_SetBooleanField(_JNIEnv*, _jobject*, _jfieldID*, unsigned char)+150)
Stack frame #03 pc 00003a93 /system/lib/libmytestt.so (Java_com_test_jni_MyLibWrapper_getMonitorSample+259)
Stack frame #04 pc 0002d850 /system/lib/libdvm.so (dvmPlatformInvoke+80)
Stack frame #05 pc 00085b48 /system/lib/libdvm.so (dvmCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*)+648)
Stack frame #06 pc 000505db /system/lib/libdvm.so (dvmCheckCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*)+59)
Stack frame #07 pc 0008b96d /system/lib/libdvm.so (dvmResolveNativeMethod(unsigned int const*, JValue*, Method const*, Thread*)+317)
Stack frame #08 pc 00170ff1 /system/lib/libdvm.so
Stack frame #09 pc 000402a6 /system/lib/libdvm.so (dvmMterpStd(Thread*)+70)
Stack frame #10 pc 0003dc14 /system/lib/libdvm.so (dvmInterpret(Thread*, Method const*, JValue*)+212)
Stack frame #11 pc 000b69db /system/lib/libdvm.so (dvmInvokeMethod(Object*, Method const*, ArrayObject*, ArrayObject*, ClassObject*, bool)+2011)
Stack frame #12 pc 000c9f07 /system/lib/libdvm.so (Dalvik_java_lang_reflect_Method_invokeNative(unsigned int const*, JValue*)+295)
Stack frame #13 pc 00170ff1 /system/lib/libdvm.so
Stack frame #14 pc 000402a6 /system/lib/libdvm.so (dvmMterpStd(Thread*)+70)
Stack frame #15 pc 0003dc14 /system/lib/libdvm.so (dvmInterpret(Thread*, Method const*, JValue*)+212)
Stack frame #16 pc 000b575c /system/lib/libdvm.so (dvmCallMethodV(Thread*, Method const*, Object*, bool, JValue*, char*)+700)
Stack frame #17 pc 00079b8e /system/lib/libdvm.so (CallStaticVoidMethodV(_JNIEnv*, _jclass*, _jmethodID*, char*)+78)
Stack frame #18 pc 0005a020 /system/lib/libdvm.so (Check_CallStaticVoidMethodV(_JNIEnv*, _jclass*, _jmethodID*, char*)+432)
Stack frame #19 pc 00057c5a /system/lib/libandroid_runtime.so (_JNIEnv::CallStaticVoidMethod(_jclass*, _jmethodID*, ...)+42)
Stack frame #20 pc 00059351 /system/lib/libandroid_runtime.so (android::AndroidRuntime::start(char const*, char const*)+945)
Stack frame #21 pc 00000fd0 /system/bin/app_process (main+528)
Stack frame #22 pc 0001a579 /system/lib/libc.so (__libc_init+89)
Stack frame #00 pc 0001ca31 /system/lib/libc.so
Stack frame #01 pc 000e66e8 /system/lib/libdvm.so (gcDaemonThread(void*)+216)
Stack frame #02 pc 00097ee0 /system/lib/libdvm.so (internalThreadStart(void*)+528)
Stack frame #03 pc 0000deb8 /system/lib/libc.so (__thread_entry+248)
Stack frame #04 pc 0001c951 /system/lib/libc.so
Stack frame #05 pc 000021e0 /dev/ashmem/dalvik-LinearAlloc (deleted)
Stack frame #06 pc 0000af42 /data/dalvik-cache/system@app@PowerManagementService.apk@classes.dex
Stack frame #00 pc 0001c05b /system/lib/libc.so
Stack frame #01 pc 0009163c /system/lib/libdvm.so (signalCatcherThreadStart(void*)+92)
Stack frame #02 pc 00097ee0 /system/lib/libdvm.so (internalThreadStart(void*)+528)
Stack frame #03 pc 0000deb8 /system/lib/libc.so (__thread_entry+248)
Stack frame #04 pc 0001c951 /system/lib/libc.so
Stack frame #05 pc 00002ff4 /system/lib/libdvm.so
Stack frame #00 pc 0001c317 /system/lib/libc.so
Stack frame #01 pc 000c012b /system/lib/libdvm.so (jdwpThreadStart(void*)+123)

Cool! Checking the stack trace we can see that my libmytestt.so crashed and the memory position is 00003a93

So, what do we have on this memory address ? Let’s go to step 3!

Step 3) Using the addr2line

This is another tool that comes with your android ndk toolchain but the name is not only addr2line as we have the standard linux distributions. If it is a special one and comes with your android ndk. For example, in my installation is:

i686-linux-android-addr2line

located at:

android-ndk-r8d/toolchains/x86-4.6/prebuilt/linux-x86/bin/

Feel free to create an alias to addr2line with whole path and correct name

alias addr2line='~/tools/android-ndk-r8d/toolchains/x86-4.6/prebuilt/linux-x86/bin/i686-linux-android-addr2line'

To run this command is quite simple. You can type “addr2line -?” in your linux.

Usage: /home/mcramon/tools/android-ndk-r8d/toolchains/x86-4.6/prebuilt/linux-x86/bin/i686-linux-android-addr2line [option(s)] [addr(s)]
 Convert addresses into line number/file name pairs.
 If no addresses are specified on the command line, they will be read from stdin
 The options are:
  @                Read options from 
  -a --addresses         Show addresses
  -b --target=  Set the binary file format
  -e --exe=  Set the input file name (default is a.out)
  -i --inlines           Unwind inlined functions
  -j --section=    Read section-relative offsets instead of addresses
  -p --pretty-print      Make the output easier to read for humans
  -s --basenames         Strip directory names
  -f --functions         Show function names
  -C --demangle[=style]  Demangle function names
  -h --help              Display this information
  -v --version           Display the program's version

In my case I will run the following command line specifying the lib that crashed and address:

bytesthink@ubuntu:~/tools/android-ndk-r8d$ addr2line -f -e ~/myrelease/symbols/system/lib/libmytestt.so 00003a93

the my output will be:

Java_com_test_jni_MyLibWrapper_getMonitorSample
.../device/test/PRIVATE/log_infra/myservice/jni/com_test_jni_MyLibWrapper.c:189

So, now I know the file and line that your software crashed!

Good debuggings!!

232 thoughts on “Debugging tombstones with ndk-stack and addr2line

  1. Hi, what if I’m not using NDK in my project but 2 of the libs I use collide and there’s a crash? How do I debug that so that I can open an issue on github providing useful information?

    Thanks!

    • hi Maik

      These tools is part of NDK, so they are supposedly to be used with NDK 🙂 But anyway if you have android device, tombstone will be generated during your crash and these tools are applicable for your external libs as well.. but keep in mind, use need to have the same binaries (images with debug info) used when you had the crash. Otherwise the symbols will not match and the tools will not work. Try it! it will work.

      Regards!
      Manoel Ramon

  2. Very nice blog on TOMBSTONE. i am very much interested on ndk-stack trace.
    i downloaded NDK and set the path in .bashrc ; in the execution “./ndk-stack -sym ~/myrelease/symbols” i didnt understand the symbols — as we are having images (boot.img bootloader.img , system.img etc. and vmlinux images) and we are having a local copy of full code base. So my doubt is for the -sym symbols either to give images path or source code.???. tombsotne file we are having that , i understood.

    Please help me in this to use ndk-stack trace tool.

  3. Hi,

    Well explained. i’m new to Android Framework development. In Tombstone i’m getting list of .so occurance. here how can i conclude the particular .so file having this issue. kindly tell me. please correct me if am wrong here.

    how you are pointing libmytestt.so is infected,
    pc 00003a93 /system/lib/libmytestt.so

    thank you.

  4. Pingback: d rose 7 bleached lilac uk trainers

  5. Pingback: uk trainers womens nike lebron soldier 11 blue white

  6. Pingback: noir rose nike air zoom spiridon

  7. Pingback: y3 slippers hyper jade uk trainers

  8. Pingback: nike flyknit cortez date red

  9. Pingback: violet marine nike star runner tdv kids

  10. Pingback: adidas joggers khaki youtube.com

  11. Pingback: volt r酶d kvinders air jordan rcvr

  12. Pingback: 褋懈薪懈泄 lebron 9

  13. Pingback: kobe 10 sneaker

  14. Pingback: nike huarache purple and black glitter swoosh

  15. Pingback: nike sb dunk low premium qs mia skate shop 10th year anniversary brv blue mtlc gld str mnrl bl 504750 474

  16. Pingback: uk trainers hyperdunk x plum

  17. Pingback: wholesale mens denver broncos royal retro full zip hoodie

  18. Pingback: cheap real nike air max sequent 852465 002 online for sale

  19. Pingback: nike zoom shoes nike zoom pegasus turbo

  20. Pingback: adidas f50

  21. Pingback: juicy couture jumpsuit new black xs velour

  22. Pingback: h by hudson black suede ankle boots peak woven

  23. Pingback: concepts x reebok insta pump fury arriving at additonal retailers

  24. Pingback: winter mens faux leather cap warm hat baseball cap with ear flaps flat top caps men big size 61cm brown

  25. Pingback: nike lebron 8 p.s. home

  26. Pingback: fila fila long sleeve zip front leotard side logo deta

  27. Pingback: nike boys air jordan 9 retro bp kobe bryant blanco tour amarillo

  28. Pingback: 639 footwear mens shoes sneakers

  29. Pingback: reebok black ventilator shoes

  30. Pingback: chaussure nike air max 97 se pour femme

  31. Pingback: real madrid vs leganes live stream watch online tv channel

  32. Pingback: air jordan 8 retro 3peat

  33. Pingback: rose gold canada nike sportswear shoes kids air force 1 se trainers metallic

  34. Pingback: nike team matchfit core otc sock white black

  35. Pingback: lsu snapback fanatics

  36. Pingback: adidas w cc oasis lite ii womens golf spikes shoes white blue 737188 s 5 nwob

  37. Pingback: santa cruz otter keychain tim ward

  38. Pingback: womens grey dressing gown uk pinterest.com

  39. Pingback: womens freedom high waisted lattice capri leggings c9 champion grapevine purple xs

  40. Pingback: performance plastics chemware pfa wide mouth wash bottles saint gobain performance plastics chemware pfa wide mouth wash bottles

  41. Pingback: on polarized lens sunglasses

  42. Pingback: asics gel saga x mas dark greenburgundy

  43. Pingback: 30mm rose gold round flat split key rings

  44. Pingback: air force chica

  45. Pingback: crystal skull mixed studs flip flop black wicked wonders vip bling

  46. Pingback: palace tee shirt homme t shirt t shirt femme tops

  47. Pingback: supreme x the north face blue mountain logo sweatshirt

  48. Pingback: rayban 7142 prescription eyeglasses

  49. Pingback: nike air max 90 hyp prm mens shoes 2015 light gray green black hot

  50. Pingback: cheap ralph lauren outlet online

  51. Pingback: caqui naranja asics whizzer lo

  52. Pingback: succinct chuck taylor united kingdom women converse womens double tongue shoes size b m us spring summer shoes

  53. Pingback: asics gel saga smoke us9.5 gr眉n

  54. Pingback: nike air force 1 mcdonalds dames

  55. Pingback: adidas d rose 5 price philippines

  56. Pingback: jovani ivory ruched bodice long sleeve scuba cocktail dress 50160

  57. Pingback: ravelry project gallery for gift wrap romper sweater pattern by carina spencer

  58. Pingback: vpfashion mermaid c017 ombr茅 cool anna ms annamello photo beautylish

  59. Pingback: vans full patch ii grey brown snapback hat

  60. Pingback: inva 200 globatt inverter battery

  61. Pingback: sandal puma original aqua cat ii evospeed full hitam

  62. Pingback: uk sale women nike air max 2013 shoes dark blue nx84082

  63. Pingback: nike air force 1 07 se lx w white black 1

  64. Pingback: chaussures nike sportswear pas cher pour femme enfant blanc noir 749986 100

  65. Pingback: rood the north face donsjack jassen voor mannen la paz mannen

  66. Pingback: shopping guide to bangkok sports goods and shoes adidas nike

  67. Pingback: nike low top shoes

  68. Pingback: nextadidas predator lz trx

  69. Pingback: Yeezy Boost

  70. Pingback: nike roshe one trainers womens

  71. Pingback: 2013 nike air red jordan flight club 90s sz 10.5 black red air grey 602661 004 df477d

  72. Pingback: womens new balance charcoal running shoes 574 uk

  73. Pingback: cheap air jordan 4 gs pure money white metallic silver pure platinum 308497

  74. Pingback: zapatillas new balance running course mujer cargando zoom

  75. Pingback: all black leather slip on vans

  76. Pingback: adidas superstar super color light pink for sale

  77. Pingback: adidas eqt cushion adv noir rose baskets ah2231

  78. Pingback: image is loading under armour ysm youth small purple short sleeve

  79. Pingback: mens nike 2014 grey white running shoes flyknit air max and

  80. Pingback: iphone 4 case unique iphone 4s case and cute iphone 4 cover image

  81. Pingback: nike hypervenom phantom iii df tf men鈥瞫 nike football shoes red gery

  82. Pingback: anwar carrots x puma suede collection

  83. Pingback: ty peek a boo phone holder with screen cleaner bottom walmart.com

  84. Pingback: 2nd place south africa

  85. Pingback: olive green hoodie saving adidas womens

  86. Pingback: the flash kids adjustable snapback cap

  87. Pingback: new style online sales womens shoes vans unisex old skool suede canvas skate shoe pewter vn 0a2z5wot4on salehigh end

  88. Pingback: nike shortsleeve legend v neck t shirt 2.0 womens at champs sports

  89. Pingback: reebok crossfit unisex crew socks black training wicking sports fitness socks

  90. Pingback: nike air force 1 high team red white

  91. Pingback: odd future black of donut beanie hbx

  92. Pingback: samsung galaxy tab s2 8.0 full tablet specifications

  93. Pingback: 36 raglan sleeve stripe trim skew neck t shirt

  94. Pingback: lyst bottega veneta key wallet in brown for men

  95. Pingback: nike mercurial superfly camo pack best

  96. Pingback: exclusive sneakeroftheday 4 8 galaxy foamposite db aj 7s baron

  97. Pingback: adidas yeezy 350 boost v2 bred running shoes 6 for sale

  98. Pingback: mens short nike t shirt clothing dri fit cotton sleeved

  99. Pingback: manteaux vestes homme bomber reversible burberry comparez et achetez

  100. Pingback: womens cable knit winter knitted beanie faux fur detachable pom pom

  101. Pingback: chaussures bateau en cuir pour homme tbs phenix jeans

  102. Pingback: 銉偆銉淬偅銉堛兂 ouis vuitton 銉濄儷銉?銉撱偍 銈炽兂銉戙偗銉?浜屻仱鎶樸倞 璨″竷 銈ㄣ償

  103. Pingback: fendi black patent leather cut out platform pumps

  104. Pingback: wei脽 un眉bertroffen aldo traunna schuhe stoffschuhe herren gr枚脽e 40

  105. Pingback: nike lunar air force 1 lo

  106. Pingback: details about new balance m420 v3 mens premium running shoes fitness gym trainers grey

  107. Pingback: mothercare 胁 谐褉邪薪写 泻邪薪褜芯薪械

  108. Pingback: nike shorts dri fit air jordan basketball poshmark

  109. Pingback: uk trainers adidas y 3 zazu triple black cg3158

  110. Pingback: miunana 3 abiti vestiti

  111. Pingback: cheap jerseys from china

  112. Pingback: sneaker de ville 脿 scratch walking homme spa 226 park avenue

  113. Pingback: xzp men and women anti fog outdoor sports glasses equipped with myopia football glasses frames

  114. Pingback: mc marine as motorcity honda mc indian mc victory mc

  115. Pingback: kit 1 t锚nis mascu ino yzy ca 莽ados mascu inos 1 chine o r 69

  116. Pingback: avis pesn petite echarpe sans noeud jpmbb. portage physiologique

  117. Pingback: sandalias puma rosa ropa

  118. Pingback: rybaczki spodenki damskie du偶e rozmiary zdj臋cie na imged

  119. Pingback: exotic versace mens black eyeglasses versace show

  120. Pingback: lunettes de soleil homme ecailles tortue ray ban comparez et achetez

  121. Pingback: Yeezy Shoes

  122. Pingback: 贸culos de sol von zipper plimpton black satin rust gradient

  123. Pingback: camiseta hulk adidas bogot谩

  124. Pingback: ceinture armani jeans a vendre

  125. Pingback: camisa polo lacoste piquet fit paris masculina compre agora netshoes

  126. Pingback: sand谩 ia mascu ina freeway agave 4 couro pinh茫o

  127. Pingback: grey superdry orange label moody mens joggers new zealand

  128. Pingback: casio watch g shock large dial double display sports men watch ga 110lpa 1a ga 110lpa 4a ga 110lp 1a

  129. Pingback: 閫佹枡鐒℃枡 鏈潻 銉偠銉笺儛銉冦偘 銉儑銈c兗銈?銉忋兂銉夈儛銉冦偘 鐗涢潻 銈枫儳銉儉銉笺儛銉冦偘 銈枫兂銉椼儷 鏂溿倎銇屻亼

  130. Pingback: new ifiniti pro hot air spin hair styler roller spin air brush

  131. Pingback: creazy the new home use ear nose and facial hair trimmer shaver trimmer

  132. Pingback: expresso cal莽ado. kyaia d谩 plataforma de vendas online ao sector

  133. Pingback: nye stilarter techno fabric dame nike sneakers textile fibres

  134. Pingback: manufacturer of online latest horsehair leather women backpack bag

  135. Pingback: mens philadelphia eagles nfl pro line by fanatics branded heather black super bowl lii champions trophy

  136. Pingback: 5sos s style guide calum hood original hoodrat total loser beanie

  137. Pingback: speedo zanpa erkek deniz ayakkab谋s谋

  138. Pingback: sapatos delicados combinam muito bem com produ莽玫es rom芒nticas

  139. Pingback: nike hypervenom 3 phelon df mens astro turf

  140. Pingback: reebok train fast xt womens fitness shoes sports outdoor indoor court reebok bags

  141. Pingback: denali veste en molleton the north face femmes rose grissac the north face

  142. Pingback: choosing the right shoes for any type of workout is very importantshoes provide support and using the wrong pair can cause not only discomfort but even

  143. Pingback: nike dual fusion run 3 womens running shoes magnet grey hyper punch volt black

  144. Pingback: chuteira society finta chuteiras de society para adultos no

  145. Pingback: air jordan 1 flight 2 black green pink women shoes jordan shoes jordan for

  146. Pingback: 拧koln铆 aktovka emipo ergo one pupies gumovac铆 pero pilot frixion doprava zdarma

  147. Pingback: under armour overwear hoodie favorite fleece sportstyle in grey women under armour sports direct for sale entire

  148. Pingback: flexfit american flag hat olive

  149. Pingback: jordan jumpman snapback cap black white

  150. Pingback: swimsuits plus size 2018 amalhantashfitness

  151. Pingback: roupa feminina academia gin谩stica fitness bermuda lupo 71311

  152. Pingback: jaeger lecoultre square m茅canique mechanical leather bracelet steel case

  153. Pingback: nike zoom strike aj0189 300 mens running shoes cargo khaki dark stucco

  154. Pingback: longhair bengal eligible for championship competition in tica

  155. Pingback: womens nike air zoom elite

  156. Pingback: to coincide with todays festivities adidas has released a tribute to the fire monkey by way of the chinese new year with 3 new offerings

  157. Pingback: nike mens lunartempo 2 10 orange

  158. Pingback: adidas y3 qasa high canada lightning green warrior tightness au black

  159. Pingback: nike air max 270 in black and white confirmed for ph drop

  160. Pingback: vestido 丿賷 賮賷爻鬲丕 賰賵乇鬲賵 賲孬賷乇 卮賷乇 賰賲 胤賵賷賱 毓賳丕亘賷 賯氐賷乇丞 丕賱毓賵丿丞 賱賱賵胤賳 賮爻丕鬲賷賳 乇禺賷氐丞 賷夭賷賳 丕賱丿丕賳鬲賷賱

  161. Pingback: nike air max 1 fb woven leopard black patent http

  162. Pingback: adidas originals basket nizza

  163. Pingback: bridal shoe clips london uk freya rose

  164. Pingback: nike air max 1 mens lucky green white red

  165. Pingback: midi jumper dress dish blue

  166. Pingback: nike jordan mens jordan flight time 14.5 wolf grey white dark grey electric

  167. Pingback: womens starbury price

  168. Pingback: basket homme nike air max 87 baskets chaussures noir vert

  169. Pingback: converse chuck taylor all star ox womens iconic low cut silhouette genuine leather upper pebbled finish

  170. Pingback: bota hypervenom phantom iii acc fg glacier blue gamma blue obsidian white

  171. Pingback: nike men 039 s premier fleece hoodie sweatshirt

  172. Pingback: simple style long sleeve solid color bodycon t shirt dress for women black xl

  173. Pingback: 氐賵乇 賵氐賵賱 兀禺乇 丿賮毓丞 賮爻丕鬲賷賳 丕賵賳 賱丕賷賳 賲囟乇賵亘丞 賲賳賵毓丕鬲

  174. Pingback: nfl miami dolphins bob griese 1972 legacy jersey

  175. Pingback: nike air jordan future gg black liquid lime

  176. Pingback: tevise hombres marca reloj reloj de pulsera de lujo de moda impermeable semiautom谩tico reloj mec谩nico luminoso

  177. Pingback: unicorn headband women

  178. Pingback: ver seu time de futebol perder piora seu desempenho no trabalho

  179. Pingback: shopping scarleton top brands or our brands 2 stars up

  180. Pingback: top noir solde

  181. Pingback: a guide to authenticating the louis vuitton monogram alma sizes pmmmand gm authenticating louis vuitton book 16 kindle edition by resale republic

  182. Pingback: lyst ralph lauren safari hat in green

  183. Pingback: converse shoes converse all star chuck taylor in dubai uae sss

  184. Pingback: 薪邪褉褍褔薪懈泄 谐芯写懈薪薪懈泻 casio 卸褨薪芯褔褨 褨 褔芯谢芯胁褨褔褨 屑芯写械谢褨

  185. Pingback: suede leather childrens martin boots plush girls boots fringe girls boots kids snow boots size 27

  186. Pingback: led belysning og spots med ops忙tning og installation

  187. Pingback: arbejdst酶j til diskotek sorte sko disco kostumer

  188. Pingback: handbag small bag stall super low price manufacturer

  189. Pingback: black and pink saucony shoes

  190. Pingback: mochi as under armour mascu inas netshoes

  191. Pingback: air zoom pegasus 35 homme noir et blanche

  192. Pingback: hawaii boogie boarding fun

  193. Pingback: touca ninja balaclava 3 furos para frio inverno motoqueiro mercado aventura

  194. Pingback: naisten puvut pc851811 eritt盲in halpa adidas puvut verkkaripuku musta

  195. Pingback: levn茅 d臎tsk谩 trekov谩 obuv adidas performance ax2 mid i barvy modr谩 zna膷ky trekov茅 boty

  196. Pingback: tretorn tretorn viken ii ka osze damskie 473239 10 buty na mivo p

  197. Pingback: vans x starter flat brim av white snapback cap shop online at

  198. Pingback: nike jordan hydro 2 premier ii white university blue crimson 456524

  199. Pingback: adidas originals gazelle mujer grises blancas doradas met谩lico by2852

  200. Pingback: graduation gift pencil case pencil pouch pencil holder pencil bag zipper pouch cotton canvas pencil pouch small

  201. Pingback: image is loading balenciaga giant 21 city white w gold tone

  202. Pingback: inov 8 roclite 315 trail runners

  203. Pingback: novelty hats arvada army navy surplus arvada denver colorado

  204. Pingback: arsip sisir ion sisir pelurus rambut instant tanpa catok lagi

  205. Pingback: jordan future shoes footaction mobile

  206. Pingback: nike vapor keystone low graphite white mens baseball cleats us 8 m eu 41

  207. Pingback: kiabi soutien gorge coton post op茅ratoire

  208. Pingback: marchesa wedding gown bateau hand made flower floor length chapel train dress tulle elastic net tiered champagne wedding dresses

  209. Pingback: sneakers pepe jeans pls30695 325 portobello w

  210. Pingback: buy reebok classic nylon womens brown off64 discounted

  211. Pingback: air jordan 1 royal 17 youtube

  212. Pingback: uk trainers silver sequin sneakers glittershoeco.com wanelo.co

  213. Pingback: west see herren m盲nner hip hop t shirt basic tee tops langarm daumen loch handschellen

  214. Pingback: flavio castellani scarpe sandali fashion blog gladiatos tacchi alti sandali blog di scarpe shoeblog high heels golden sandals golden gladiators dress legs

  215. Pingback: new balance 868 phantom with grey red sko

  216. Pingback: dr martens original 1461 black 3 eyelet lace up shoes unisex

  217. Pingback: t锚nis nike air max excellerate 3 rn masculino 2 reviews

  218. Pingback: mode adidas originals cuff chalk wei脽 hosen damen bekleidung sale

  219. Pingback: uk trainers jordan xvii crocus

  220. Pingback: womens long sleeve v neck cotton nightshirt

  221. Pingback: encontre boss hugo boss pelo menor pre莽o para铆so feminino

  222. Pingback: barbour jacket sale

  223. Pingback: szpi ki z ozdobnym obcasem 36 br膮zowe etnie buty 7443258302 oficja ne archiwum a egro

  224. Pingback: perfume masculino beleza perfumaria magazine luiza

  225. Pingback: mashur fab gowns for women women gowns for girls latest collection 2018 dress for girls designer gawn gown dark pink amazon.in clothing accessories

  226. Pingback: cappello tedesco ww2 da parata o battaglia a cassola kijiji

  227. Pingback: pochette ceinture en cuir elephant dor la redoute

  228. Pingback: under armour speedform 3 laufschuhe rot damen

  229. Pingback: cloud nine touch emilio de la morena limited edition stylers

Leave a Reply