Debugging tombstones with ndk-stack and addr2line

android

I really like to work with several components in a system including linux kernel or keep my emotions on userspace.

If you work with android and your are a real engineer, it is very difficult to resist the native support using Android NDK. If your software requires performance using graphic API like in OpenGL or if you need to access some specific information provided by some native library, the NDK fits for you.

However, bugs in native side sometimes takes time and usually it is not an easy and fast task. If you have the device and an easy way to reproduce the issue, it is ok but suppose you need to collect logs from remote users and the scenario is very difficult to reproduce. Some cases the users even see the crash visually but the system contains the logs blocking the approval of your software that must be completed to some costumer.. at this point the program/project managers team are “talking” in your ears… “fix it!”.

Working with Android, every time a process that runs on native side crashes, we have some small pieces of your stack in files called tombstones.

The tombstones are located at /data/tombstones as isolated files (one files represents one crash) or you can see them in your logcat Take a look in the adb shell:

root@android:/ # find . |grep tombs

./data/tombstones
./data/tombstones/tombstone_00
./data/tombstones/tombstone_01
root@android:/ #

The tombstone inform you about:

  1. Build fingerprint
  2. Crashed process and PIDs
  3. Terminated signal and fault address
  4. CPU registers
  5. Call stack
  6. Stack content of each call

I will not post a full tombstone here. Check the /data on your device and you will observe the 7 sessions mentioned above . Let’s go straight to the point.. how to debug the stack in tombstones files!

Debugging the stack…  easier way

But sometimes looking to raw tombstones it is a little difficult to realize where and what the problem really is.

The intention of this post is to introduce two tools that comes with android NDK that really helps: ndk-stack and addr2line

Follow the steps below:

1) Install the NDK (if you do not have yet)

If you do not have the android ndk package you need to install in order to have access to the tool chains. Please download the package from this link and install on your computer.

I recommend you include your NDK directory in your path. If you are using linux here goes an example that you need to include in your .bashrc:

export ANDROID_NDK_PATH=/home/bytesthink/tools/android-ndk-r8d/

PATH= $ANDROID_NDK_PATH:$PATH

2) Execute ndk-stack tool

If you install everything, you should be able to run the tool ndk-stack

ndk-stack -sym <path> [-dump <path>]

   -sym  Contains full path to the root directory for symbols.
   -dump Contains full path to the file containing the crash dump.
         This is an optional parameter. If ommited, ndk-stack will
         read input data from stdin

See docs/NDK-STACK.html in your NDK installation tree for more details.

Very simple right ? Let’s run an example.. considering I pull a tombstone file from my system.

bytesthink@ubuntu:~/tools/android-ndk-r8d$ ./ndk-stack -sym ~/myrelease/symbols -dump ~/win/bytesthink/Documents/test/perform_releases/tombstone_01
********** Crash dump: **********
Build fingerprint: 'version1239210121'
pid: 19135, tid: 19135, name: erviceinterface >>> com.test.powermeterserver.serviceinterface <<<
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr deadd00d
Stack frame #00 pc 00072c9a /system/lib/libdvm.so (dvmAbort+170)
Stack frame #01 pc 00050f6b /system/lib/libdvm.so (ScopedCheck::checkInstanceFieldID(_jobject*, _jfieldID*)+379)
Stack frame #02 pc 0005db56 /system/lib/libdvm.so (Check_SetBooleanField(_JNIEnv*, _jobject*, _jfieldID*, unsigned char)+150)
Stack frame #03 pc 00003a93 /system/lib/libmytestt.so (Java_com_test_jni_MyLibWrapper_getMonitorSample+259)
Stack frame #04 pc 0002d850 /system/lib/libdvm.so (dvmPlatformInvoke+80)
Stack frame #05 pc 00085b48 /system/lib/libdvm.so (dvmCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*)+648)
Stack frame #06 pc 000505db /system/lib/libdvm.so (dvmCheckCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*)+59)
Stack frame #07 pc 0008b96d /system/lib/libdvm.so (dvmResolveNativeMethod(unsigned int const*, JValue*, Method const*, Thread*)+317)
Stack frame #08 pc 00170ff1 /system/lib/libdvm.so
Stack frame #09 pc 000402a6 /system/lib/libdvm.so (dvmMterpStd(Thread*)+70)
Stack frame #10 pc 0003dc14 /system/lib/libdvm.so (dvmInterpret(Thread*, Method const*, JValue*)+212)
Stack frame #11 pc 000b69db /system/lib/libdvm.so (dvmInvokeMethod(Object*, Method const*, ArrayObject*, ArrayObject*, ClassObject*, bool)+2011)
Stack frame #12 pc 000c9f07 /system/lib/libdvm.so (Dalvik_java_lang_reflect_Method_invokeNative(unsigned int const*, JValue*)+295)
Stack frame #13 pc 00170ff1 /system/lib/libdvm.so
Stack frame #14 pc 000402a6 /system/lib/libdvm.so (dvmMterpStd(Thread*)+70)
Stack frame #15 pc 0003dc14 /system/lib/libdvm.so (dvmInterpret(Thread*, Method const*, JValue*)+212)
Stack frame #16 pc 000b575c /system/lib/libdvm.so (dvmCallMethodV(Thread*, Method const*, Object*, bool, JValue*, char*)+700)
Stack frame #17 pc 00079b8e /system/lib/libdvm.so (CallStaticVoidMethodV(_JNIEnv*, _jclass*, _jmethodID*, char*)+78)
Stack frame #18 pc 0005a020 /system/lib/libdvm.so (Check_CallStaticVoidMethodV(_JNIEnv*, _jclass*, _jmethodID*, char*)+432)
Stack frame #19 pc 00057c5a /system/lib/libandroid_runtime.so (_JNIEnv::CallStaticVoidMethod(_jclass*, _jmethodID*, ...)+42)
Stack frame #20 pc 00059351 /system/lib/libandroid_runtime.so (android::AndroidRuntime::start(char const*, char const*)+945)
Stack frame #21 pc 00000fd0 /system/bin/app_process (main+528)
Stack frame #22 pc 0001a579 /system/lib/libc.so (__libc_init+89)
Stack frame #00 pc 0001ca31 /system/lib/libc.so
Stack frame #01 pc 000e66e8 /system/lib/libdvm.so (gcDaemonThread(void*)+216)
Stack frame #02 pc 00097ee0 /system/lib/libdvm.so (internalThreadStart(void*)+528)
Stack frame #03 pc 0000deb8 /system/lib/libc.so (__thread_entry+248)
Stack frame #04 pc 0001c951 /system/lib/libc.so
Stack frame #05 pc 000021e0 /dev/ashmem/dalvik-LinearAlloc (deleted)
Stack frame #06 pc 0000af42 /data/dalvik-cache/system@app@PowerManagementService.apk@classes.dex
Stack frame #00 pc 0001c05b /system/lib/libc.so
Stack frame #01 pc 0009163c /system/lib/libdvm.so (signalCatcherThreadStart(void*)+92)
Stack frame #02 pc 00097ee0 /system/lib/libdvm.so (internalThreadStart(void*)+528)
Stack frame #03 pc 0000deb8 /system/lib/libc.so (__thread_entry+248)
Stack frame #04 pc 0001c951 /system/lib/libc.so
Stack frame #05 pc 00002ff4 /system/lib/libdvm.so
Stack frame #00 pc 0001c317 /system/lib/libc.so
Stack frame #01 pc 000c012b /system/lib/libdvm.so (jdwpThreadStart(void*)+123)

Cool! Checking the stack trace we can see that my libmytestt.so crashed and the memory position is 00003a93

So, what do we have on this memory address ? Let’s go to step 3!

Step 3) Using the addr2line

This is another tool that comes with your android ndk toolchain but the name is not only addr2line as we have the standard linux distributions. If it is a special one and comes with your android ndk. For example, in my installation is:

i686-linux-android-addr2line

located at:

android-ndk-r8d/toolchains/x86-4.6/prebuilt/linux-x86/bin/

Feel free to create an alias to addr2line with whole path and correct name

alias addr2line='~/tools/android-ndk-r8d/toolchains/x86-4.6/prebuilt/linux-x86/bin/i686-linux-android-addr2line'

To run this command is quite simple. You can type “addr2line -?” in your linux.

Usage: /home/mcramon/tools/android-ndk-r8d/toolchains/x86-4.6/prebuilt/linux-x86/bin/i686-linux-android-addr2line [option(s)] [addr(s)]
 Convert addresses into line number/file name pairs.
 If no addresses are specified on the command line, they will be read from stdin
 The options are:
  @                Read options from 
  -a --addresses         Show addresses
  -b --target=  Set the binary file format
  -e --exe=  Set the input file name (default is a.out)
  -i --inlines           Unwind inlined functions
  -j --section=    Read section-relative offsets instead of addresses
  -p --pretty-print      Make the output easier to read for humans
  -s --basenames         Strip directory names
  -f --functions         Show function names
  -C --demangle[=style]  Demangle function names
  -h --help              Display this information
  -v --version           Display the program's version

In my case I will run the following command line specifying the lib that crashed and address:

bytesthink@ubuntu:~/tools/android-ndk-r8d$ addr2line -f -e ~/myrelease/symbols/system/lib/libmytestt.so 00003a93

the my output will be:

Java_com_test_jni_MyLibWrapper_getMonitorSample
.../device/test/PRIVATE/log_infra/myservice/jni/com_test_jni_MyLibWrapper.c:189

So, now I know the file and line that your software crashed!

Good debuggings!!

465 thoughts on “Debugging tombstones with ndk-stack and addr2line

  1. Hi, what if I’m not using NDK in my project but 2 of the libs I use collide and there’s a crash? How do I debug that so that I can open an issue on github providing useful information?

    Thanks!

    • hi Maik

      These tools is part of NDK, so they are supposedly to be used with NDK 🙂 But anyway if you have android device, tombstone will be generated during your crash and these tools are applicable for your external libs as well.. but keep in mind, use need to have the same binaries (images with debug info) used when you had the crash. Otherwise the symbols will not match and the tools will not work. Try it! it will work.

      Regards!
      Manoel Ramon

  2. Very nice blog on TOMBSTONE. i am very much interested on ndk-stack trace.
    i downloaded NDK and set the path in .bashrc ; in the execution “./ndk-stack -sym ~/myrelease/symbols” i didnt understand the symbols — as we are having images (boot.img bootloader.img , system.img etc. and vmlinux images) and we are having a local copy of full code base. So my doubt is for the -sym symbols either to give images path or source code.???. tombsotne file we are having that , i understood.

    Please help me in this to use ndk-stack trace tool.

  3. Hi,

    Well explained. i’m new to Android Framework development. In Tombstone i’m getting list of .so occurance. here how can i conclude the particular .so file having this issue. kindly tell me. please correct me if am wrong here.

    how you are pointing libmytestt.so is infected,
    pc 00003a93 /system/lib/libmytestt.so

    thank you.

  4. Pingback: d rose 7 bleached lilac uk trainers

  5. Pingback: uk trainers womens nike lebron soldier 11 blue white

  6. Pingback: noir rose nike air zoom spiridon

  7. Pingback: y3 slippers hyper jade uk trainers

  8. Pingback: nike flyknit cortez date red

  9. Pingback: violet marine nike star runner tdv kids

  10. Pingback: adidas joggers khaki youtube.com

  11. Pingback: volt r酶d kvinders air jordan rcvr

  12. Pingback: 褋懈薪懈泄 lebron 9

  13. Pingback: kobe 10 sneaker

  14. Pingback: nike huarache purple and black glitter swoosh

  15. Pingback: nike sb dunk low premium qs mia skate shop 10th year anniversary brv blue mtlc gld str mnrl bl 504750 474

  16. Pingback: uk trainers hyperdunk x plum

  17. Pingback: wholesale mens denver broncos royal retro full zip hoodie

  18. Pingback: cheap real nike air max sequent 852465 002 online for sale

  19. Pingback: nike zoom shoes nike zoom pegasus turbo

  20. Pingback: adidas f50

  21. Pingback: juicy couture jumpsuit new black xs velour

  22. Pingback: h by hudson black suede ankle boots peak woven

  23. Pingback: concepts x reebok insta pump fury arriving at additonal retailers

  24. Pingback: winter mens faux leather cap warm hat baseball cap with ear flaps flat top caps men big size 61cm brown

  25. Pingback: nike lebron 8 p.s. home

  26. Pingback: fila fila long sleeve zip front leotard side logo deta

  27. Pingback: nike boys air jordan 9 retro bp kobe bryant blanco tour amarillo

  28. Pingback: 639 footwear mens shoes sneakers

  29. Pingback: reebok black ventilator shoes

  30. Pingback: chaussure nike air max 97 se pour femme

  31. Pingback: real madrid vs leganes live stream watch online tv channel

  32. Pingback: air jordan 8 retro 3peat

  33. Pingback: rose gold canada nike sportswear shoes kids air force 1 se trainers metallic

  34. Pingback: nike team matchfit core otc sock white black

  35. Pingback: lsu snapback fanatics

  36. Pingback: adidas w cc oasis lite ii womens golf spikes shoes white blue 737188 s 5 nwob

  37. Pingback: santa cruz otter keychain tim ward

  38. Pingback: womens grey dressing gown uk pinterest.com

  39. Pingback: womens freedom high waisted lattice capri leggings c9 champion grapevine purple xs

  40. Pingback: performance plastics chemware pfa wide mouth wash bottles saint gobain performance plastics chemware pfa wide mouth wash bottles

  41. Pingback: on polarized lens sunglasses

  42. Pingback: asics gel saga x mas dark greenburgundy

  43. Pingback: 30mm rose gold round flat split key rings

  44. Pingback: air force chica

  45. Pingback: crystal skull mixed studs flip flop black wicked wonders vip bling

  46. Pingback: palace tee shirt homme t shirt t shirt femme tops

  47. Pingback: supreme x the north face blue mountain logo sweatshirt

  48. Pingback: rayban 7142 prescription eyeglasses

  49. Pingback: nike air max 90 hyp prm mens shoes 2015 light gray green black hot

  50. Pingback: cheap ralph lauren outlet online

  51. Pingback: caqui naranja asics whizzer lo

  52. Pingback: succinct chuck taylor united kingdom women converse womens double tongue shoes size b m us spring summer shoes

  53. Pingback: asics gel saga smoke us9.5 gr眉n

  54. Pingback: nike air force 1 mcdonalds dames

  55. Pingback: adidas d rose 5 price philippines

  56. Pingback: jovani ivory ruched bodice long sleeve scuba cocktail dress 50160

  57. Pingback: ravelry project gallery for gift wrap romper sweater pattern by carina spencer

  58. Pingback: vpfashion mermaid c017 ombr茅 cool anna ms annamello photo beautylish

  59. Pingback: vans full patch ii grey brown snapback hat

  60. Pingback: inva 200 globatt inverter battery

  61. Pingback: sandal puma original aqua cat ii evospeed full hitam

  62. Pingback: uk sale women nike air max 2013 shoes dark blue nx84082

  63. Pingback: nike air force 1 07 se lx w white black 1

  64. Pingback: chaussures nike sportswear pas cher pour femme enfant blanc noir 749986 100

  65. Pingback: rood the north face donsjack jassen voor mannen la paz mannen

  66. Pingback: shopping guide to bangkok sports goods and shoes adidas nike

  67. Pingback: nike low top shoes

  68. Pingback: nextadidas predator lz trx

  69. Pingback: Yeezy Boost

  70. Pingback: nike roshe one trainers womens

  71. Pingback: 2013 nike air red jordan flight club 90s sz 10.5 black red air grey 602661 004 df477d

  72. Pingback: womens new balance charcoal running shoes 574 uk

  73. Pingback: cheap air jordan 4 gs pure money white metallic silver pure platinum 308497

  74. Pingback: zapatillas new balance running course mujer cargando zoom

  75. Pingback: all black leather slip on vans

  76. Pingback: adidas superstar super color light pink for sale

  77. Pingback: adidas eqt cushion adv noir rose baskets ah2231

  78. Pingback: image is loading under armour ysm youth small purple short sleeve

  79. Pingback: mens nike 2014 grey white running shoes flyknit air max and

  80. Pingback: iphone 4 case unique iphone 4s case and cute iphone 4 cover image

  81. Pingback: nike hypervenom phantom iii df tf men鈥瞫 nike football shoes red gery

  82. Pingback: anwar carrots x puma suede collection

  83. Pingback: ty peek a boo phone holder with screen cleaner bottom walmart.com

  84. Pingback: 2nd place south africa

  85. Pingback: olive green hoodie saving adidas womens

  86. Pingback: the flash kids adjustable snapback cap

  87. Pingback: new style online sales womens shoes vans unisex old skool suede canvas skate shoe pewter vn 0a2z5wot4on salehigh end

  88. Pingback: nike shortsleeve legend v neck t shirt 2.0 womens at champs sports

  89. Pingback: reebok crossfit unisex crew socks black training wicking sports fitness socks

  90. Pingback: nike air force 1 high team red white

  91. Pingback: odd future black of donut beanie hbx

  92. Pingback: samsung galaxy tab s2 8.0 full tablet specifications

  93. Pingback: 36 raglan sleeve stripe trim skew neck t shirt

  94. Pingback: lyst bottega veneta key wallet in brown for men

  95. Pingback: nike mercurial superfly camo pack best

  96. Pingback: exclusive sneakeroftheday 4 8 galaxy foamposite db aj 7s baron

  97. Pingback: adidas yeezy 350 boost v2 bred running shoes 6 for sale

  98. Pingback: mens short nike t shirt clothing dri fit cotton sleeved

  99. Pingback: manteaux vestes homme bomber reversible burberry comparez et achetez

  100. Pingback: womens cable knit winter knitted beanie faux fur detachable pom pom

  101. Pingback: chaussures bateau en cuir pour homme tbs phenix jeans

  102. Pingback: 銉偆銉淬偅銉堛兂 ouis vuitton 銉濄儷銉?銉撱偍 銈炽兂銉戙偗銉?浜屻仱鎶樸倞 璨″竷 銈ㄣ償

  103. Pingback: fendi black patent leather cut out platform pumps

  104. Pingback: wei脽 un眉bertroffen aldo traunna schuhe stoffschuhe herren gr枚脽e 40

  105. Pingback: nike lunar air force 1 lo

  106. Pingback: details about new balance m420 v3 mens premium running shoes fitness gym trainers grey

  107. Pingback: mothercare 胁 谐褉邪薪写 泻邪薪褜芯薪械

  108. Pingback: nike shorts dri fit air jordan basketball poshmark

  109. Pingback: uk trainers adidas y 3 zazu triple black cg3158

  110. Pingback: miunana 3 abiti vestiti

  111. Pingback: cheap jerseys from china

  112. Pingback: sneaker de ville 脿 scratch walking homme spa 226 park avenue

  113. Pingback: xzp men and women anti fog outdoor sports glasses equipped with myopia football glasses frames

  114. Pingback: mc marine as motorcity honda mc indian mc victory mc

  115. Pingback: kit 1 t锚nis mascu ino yzy ca 莽ados mascu inos 1 chine o r 69

  116. Pingback: avis pesn petite echarpe sans noeud jpmbb. portage physiologique

  117. Pingback: sandalias puma rosa ropa

  118. Pingback: rybaczki spodenki damskie du偶e rozmiary zdj臋cie na imged

  119. Pingback: exotic versace mens black eyeglasses versace show

  120. Pingback: lunettes de soleil homme ecailles tortue ray ban comparez et achetez

  121. Pingback: Yeezy Shoes

  122. Pingback: 贸culos de sol von zipper plimpton black satin rust gradient

  123. Pingback: camiseta hulk adidas bogot谩

  124. Pingback: ceinture armani jeans a vendre

  125. Pingback: camisa polo lacoste piquet fit paris masculina compre agora netshoes

  126. Pingback: sand谩 ia mascu ina freeway agave 4 couro pinh茫o

  127. Pingback: grey superdry orange label moody mens joggers new zealand

  128. Pingback: casio watch g shock large dial double display sports men watch ga 110lpa 1a ga 110lpa 4a ga 110lp 1a

  129. Pingback: 閫佹枡鐒℃枡 鏈潻 銉偠銉笺儛銉冦偘 銉儑銈c兗銈?銉忋兂銉夈儛銉冦偘 鐗涢潻 銈枫儳銉儉銉笺儛銉冦偘 銈枫兂銉椼儷 鏂溿倎銇屻亼

  130. Pingback: new ifiniti pro hot air spin hair styler roller spin air brush

  131. Pingback: creazy the new home use ear nose and facial hair trimmer shaver trimmer

  132. Pingback: expresso cal莽ado. kyaia d谩 plataforma de vendas online ao sector

  133. Pingback: nye stilarter techno fabric dame nike sneakers textile fibres

  134. Pingback: manufacturer of online latest horsehair leather women backpack bag

  135. Pingback: mens philadelphia eagles nfl pro line by fanatics branded heather black super bowl lii champions trophy

  136. Pingback: 5sos s style guide calum hood original hoodrat total loser beanie

  137. Pingback: speedo zanpa erkek deniz ayakkab谋s谋

  138. Pingback: sapatos delicados combinam muito bem com produ莽玫es rom芒nticas

  139. Pingback: nike hypervenom 3 phelon df mens astro turf

  140. Pingback: reebok train fast xt womens fitness shoes sports outdoor indoor court reebok bags

  141. Pingback: denali veste en molleton the north face femmes rose grissac the north face

  142. Pingback: choosing the right shoes for any type of workout is very importantshoes provide support and using the wrong pair can cause not only discomfort but even

  143. Pingback: nike dual fusion run 3 womens running shoes magnet grey hyper punch volt black

  144. Pingback: chuteira society finta chuteiras de society para adultos no

  145. Pingback: air jordan 1 flight 2 black green pink women shoes jordan shoes jordan for

  146. Pingback: 拧koln铆 aktovka emipo ergo one pupies gumovac铆 pero pilot frixion doprava zdarma

  147. Pingback: under armour overwear hoodie favorite fleece sportstyle in grey women under armour sports direct for sale entire

  148. Pingback: flexfit american flag hat olive

  149. Pingback: jordan jumpman snapback cap black white

  150. Pingback: swimsuits plus size 2018 amalhantashfitness

  151. Pingback: roupa feminina academia gin谩stica fitness bermuda lupo 71311

  152. Pingback: jaeger lecoultre square m茅canique mechanical leather bracelet steel case

  153. Pingback: nike zoom strike aj0189 300 mens running shoes cargo khaki dark stucco

  154. Pingback: longhair bengal eligible for championship competition in tica

  155. Pingback: womens nike air zoom elite

  156. Pingback: to coincide with todays festivities adidas has released a tribute to the fire monkey by way of the chinese new year with 3 new offerings

  157. Pingback: nike mens lunartempo 2 10 orange

  158. Pingback: adidas y3 qasa high canada lightning green warrior tightness au black

  159. Pingback: nike air max 270 in black and white confirmed for ph drop

  160. Pingback: vestido 丿賷 賮賷爻鬲丕 賰賵乇鬲賵 賲孬賷乇 卮賷乇 賰賲 胤賵賷賱 毓賳丕亘賷 賯氐賷乇丞 丕賱毓賵丿丞 賱賱賵胤賳 賮爻丕鬲賷賳 乇禺賷氐丞 賷夭賷賳 丕賱丿丕賳鬲賷賱

  161. Pingback: nike air max 1 fb woven leopard black patent http

  162. Pingback: adidas originals basket nizza

  163. Pingback: bridal shoe clips london uk freya rose

  164. Pingback: nike air max 1 mens lucky green white red

  165. Pingback: midi jumper dress dish blue

  166. Pingback: nike jordan mens jordan flight time 14.5 wolf grey white dark grey electric

  167. Pingback: womens starbury price

  168. Pingback: basket homme nike air max 87 baskets chaussures noir vert

  169. Pingback: converse chuck taylor all star ox womens iconic low cut silhouette genuine leather upper pebbled finish

  170. Pingback: bota hypervenom phantom iii acc fg glacier blue gamma blue obsidian white

  171. Pingback: nike men 039 s premier fleece hoodie sweatshirt

  172. Pingback: simple style long sleeve solid color bodycon t shirt dress for women black xl

  173. Pingback: 氐賵乇 賵氐賵賱 兀禺乇 丿賮毓丞 賮爻丕鬲賷賳 丕賵賳 賱丕賷賳 賲囟乇賵亘丞 賲賳賵毓丕鬲

  174. Pingback: nfl miami dolphins bob griese 1972 legacy jersey

  175. Pingback: nike air jordan future gg black liquid lime

  176. Pingback: tevise hombres marca reloj reloj de pulsera de lujo de moda impermeable semiautom谩tico reloj mec谩nico luminoso

  177. Pingback: unicorn headband women

  178. Pingback: ver seu time de futebol perder piora seu desempenho no trabalho

  179. Pingback: shopping scarleton top brands or our brands 2 stars up

  180. Pingback: top noir solde

  181. Pingback: a guide to authenticating the louis vuitton monogram alma sizes pmmmand gm authenticating louis vuitton book 16 kindle edition by resale republic

  182. Pingback: lyst ralph lauren safari hat in green

  183. Pingback: converse shoes converse all star chuck taylor in dubai uae sss

  184. Pingback: 薪邪褉褍褔薪懈泄 谐芯写懈薪薪懈泻 casio 卸褨薪芯褔褨 褨 褔芯谢芯胁褨褔褨 屑芯写械谢褨

  185. Pingback: suede leather childrens martin boots plush girls boots fringe girls boots kids snow boots size 27

  186. Pingback: led belysning og spots med ops忙tning og installation

  187. Pingback: arbejdst酶j til diskotek sorte sko disco kostumer

  188. Pingback: handbag small bag stall super low price manufacturer

  189. Pingback: black and pink saucony shoes

  190. Pingback: mochi as under armour mascu inas netshoes

  191. Pingback: air zoom pegasus 35 homme noir et blanche

  192. Pingback: hawaii boogie boarding fun

  193. Pingback: touca ninja balaclava 3 furos para frio inverno motoqueiro mercado aventura

  194. Pingback: naisten puvut pc851811 eritt盲in halpa adidas puvut verkkaripuku musta

  195. Pingback: levn茅 d臎tsk谩 trekov谩 obuv adidas performance ax2 mid i barvy modr谩 zna膷ky trekov茅 boty

  196. Pingback: tretorn tretorn viken ii ka osze damskie 473239 10 buty na mivo p

  197. Pingback: vans x starter flat brim av white snapback cap shop online at

  198. Pingback: nike jordan hydro 2 premier ii white university blue crimson 456524

  199. Pingback: adidas originals gazelle mujer grises blancas doradas met谩lico by2852

  200. Pingback: graduation gift pencil case pencil pouch pencil holder pencil bag zipper pouch cotton canvas pencil pouch small

  201. Pingback: image is loading balenciaga giant 21 city white w gold tone

  202. Pingback: inov 8 roclite 315 trail runners

  203. Pingback: novelty hats arvada army navy surplus arvada denver colorado

  204. Pingback: arsip sisir ion sisir pelurus rambut instant tanpa catok lagi

  205. Pingback: jordan future shoes footaction mobile

  206. Pingback: nike vapor keystone low graphite white mens baseball cleats us 8 m eu 41

  207. Pingback: kiabi soutien gorge coton post op茅ratoire

  208. Pingback: marchesa wedding gown bateau hand made flower floor length chapel train dress tulle elastic net tiered champagne wedding dresses

  209. Pingback: sneakers pepe jeans pls30695 325 portobello w

  210. Pingback: buy reebok classic nylon womens brown off64 discounted

  211. Pingback: air jordan 1 royal 17 youtube

  212. Pingback: uk trainers silver sequin sneakers glittershoeco.com wanelo.co

  213. Pingback: west see herren m盲nner hip hop t shirt basic tee tops langarm daumen loch handschellen

  214. Pingback: flavio castellani scarpe sandali fashion blog gladiatos tacchi alti sandali blog di scarpe shoeblog high heels golden sandals golden gladiators dress legs

  215. Pingback: new balance 868 phantom with grey red sko

  216. Pingback: dr martens original 1461 black 3 eyelet lace up shoes unisex

  217. Pingback: t锚nis nike air max excellerate 3 rn masculino 2 reviews

  218. Pingback: mode adidas originals cuff chalk wei脽 hosen damen bekleidung sale

  219. Pingback: uk trainers jordan xvii crocus

  220. Pingback: womens long sleeve v neck cotton nightshirt

  221. Pingback: encontre boss hugo boss pelo menor pre莽o para铆so feminino

  222. Pingback: barbour jacket sale

  223. Pingback: szpi ki z ozdobnym obcasem 36 br膮zowe etnie buty 7443258302 oficja ne archiwum a egro

  224. Pingback: perfume masculino beleza perfumaria magazine luiza

  225. Pingback: mashur fab gowns for women women gowns for girls latest collection 2018 dress for girls designer gawn gown dark pink amazon.in clothing accessories

  226. Pingback: cappello tedesco ww2 da parata o battaglia a cassola kijiji

  227. Pingback: pochette ceinture en cuir elephant dor la redoute

  228. Pingback: under armour speedform 3 laufschuhe rot damen

  229. Pingback: cloud nine touch emilio de la morena limited edition stylers

  230. Pingback: camisa s茫o pau o feminina i 2016 persona izada nome n煤mero branco

  231. Pingback: nike sb bruin hi womens skateboarding shoe. nike.com si

  232. Pingback: yellow italian embossed crocodile leather handbag fjs handbags

  233. Pingback: adidas superstar damskie buty hurtownia online

  234. Pingback: deportivas converse chuck taylor all star well worn sl w verde vista de detalle par

  235. Pingback: wahl instagram photos and videos

  236. Pingback: scarpe di lusso moa master of arts sneakers in pelle ricamo mickey mouse sneakers bianco scarpe uomo

  237. Pingback: jordan 10 heren schoenen wit zwart rood

  238. Pingback: 103.2 rodovi谩ria do p ano pi oto w3 su avenida na莽玫es acadamia

  239. Pingback: red wagon star wars boys vehicle t shirt bambino

  240. Pingback: chuteira campo nike mercurial victory 6 fg compre agora loja do

  241. Pingback: tenis timberland azul muccashop

  242. Pingback: wooden glasses wood eyewear optical frame

  243. Pingback: lunettes soleil sports et loisirs oakley personnalis茅es raymond

  244. Pingback: dolag necklace pilgrim smykker til damer i

  245. Pingback: womens san diego hat company faux suede felt floppy hat cth8044 black hats

  246. Pingback: arma莽茫o oculos grau azul de feita de titanium sem aro a526 r 69

  247. Pingback: seattle seahawks womens nfl tommy bahama white floral blitz t shirt ladies sz xs

  248. Pingback: adidas originals tubular shadow infant todas las zapatillas de ni帽os en negro

  249. Pingback: versace jeans t锚nis com logo preto

  250. Pingback: hair dryer swiss metal master light

  251. Pingback: aproveite para comprar rel贸gio bulova marine star e outros

  252. Pingback: the north face isoterma windstopper jacket shadow pur 100 poliestere lavabile in lavatrice donna abbigliamento sportivo giacconi e cappuc

  253. Pingback: reebok ventilator custom jeremiah marshall tags trainers og reeboks runners custom ventilator reebok

  254. Pingback: 贸culos de sol carrera pretos ou azuis haste metal prateada ou dourada

  255. Pingback: nike vapormax flyknit moc 2 team red womens 942843 006 ebay

  256. Pingback: special prices on puma vikky platform ribbon s womens white sneaker

  257. Pingback: swiss watches for sale in london cheap watches mgc

  258. Pingback: sukienka mode giu ia 219 1 navy

  259. Pingback: psg online store

  260. Pingback: camisa santos jogo iii kombat 2017 kappa masculina

  261. Pingback: suknie 艣 ubne inspiracje tab ica pati16th na zszywka p

  262. Pingback: louisville cardinals 8 lamar jackson printed adidas jersey black

  263. Pingback: samsung hw n650 panoramic soundbar certified refurbished deal parade

  264. Pingback: dress collection

  265. Pingback: newest hot sell motorcycle pedal gearshift leather shift sock boot shoe protector with 3d logo brown free shipping

  266. Pingback: gents traditional aran sweater

  267. Pingback: Cheap NFL Jerseys

  268. Pingback: pelota futbol tango rosario en adidas de futbol

  269. Pingback: arma莽茫o para 贸culos de grau 470 feminino siracusa em acetato preto e azul

  270. Pingback: ly啪iarky predaj bratislava

  271. Pingback: homens sapatos timber and c assic boat preto brown on ine tomada

  272. Pingback: slow vest stand up chaleco deep atlantic

  273. Pingback: 贸culos de sol oakley feminino ringer red mossiac vermelho mesclado em preto lentes marrom degrade ref

  274. Pingback: nike air zoom mariah flyknit racer black

  275. Pingback: Altercore Spring/Summer 2017 Tenis贸wki Granatowy materia艂 - materia艂 guma materia艂 HF01-LOW VFZVTVG

  276. Pingback: kayseria winter dresses collection 2018 for women little girls 1

  277. Pingback: panic disco womens tank tops cafepress

  278. Pingback: coupon code for wholesale prada wallet mens gift wallet new prada bag for men 3

  279. Pingback: Nike Shox Outlet

  280. Pingback: promo莽茫o ray ban justin rb4165 polarizado oculos de sol masc. carregando zoom

  281. Pingback: a buon mercato nike air max 2013 bambini ragazzo ragazza scarpe anti skid nike wearable

  282. Pingback: enter paulas choice unscrub. for those with complexions sensitive to exfoliators

  283. Pingback: Nike Air Max 2019

  284. Pingback: buy adidas originals mens samba super suede trainers navy white gum

  285. Pingback: hot pink sparkly heels

  286. Pingback: rado dress watch for men analog metal r12413323

  287. Pingback: chap茅u country 30 x champion aba 12 copa rendada pralana nt bicolor no

  288. Pingback: de vente getsring femmes jupe tricot茅 jupe longue noir gland irr茅guli猫re jupe taille haute jupes

  289. Pingback: Jordan 12 Gym Red

  290. Pingback: camisa seattle seahawks no mercado livre brasil

  291. Pingback: good quality aviator sunglasses

  292. Pingback: prettylittlething dresses wedding guest dress off the shoulder

  293. Pingback: light daunenjacke mit steppungen christian berg woman selection online kaufen 1

  294. Pingback: womens nike mercurial superfly red

  295. Pingback: stella mccartney falabella big bags

  296. Pingback: 蟺慰喂慰蟼 伪魏慰位慰蠀胃蔚委 伪纬慰蟻维 蟺伪蟺慰蠀蟿蟽喂蠋谓 渭苇蟽蠅 internet 魏伪喂 伪蟺慰蟻委蔚蟼

  297. Pingback: chanel sunglasses 5320 chanel sunglasses 5320

  298. Pingback: runway looks electroplating metal ball color matching coach leather tip with t bring sandals womens shoes

  299. Pingback: tommy hilfiger 90s hat

  300. Pingback: bon茅 original t rak. pintura artesanal. tinta a base dagua. tecido brim. tamanho 煤nico

  301. Pingback: collant minceur et collant sant茅 belle en collant.fr. belle en

  302. Pingback: air force 1 low top sneaker wanelo.co

  303. Pingback: chuck taylor all star collegiate color low top washed denim khaki white

  304. Pingback: brown peace sign mandala toms shoes by ibleedheart on etsy

  305. Pingback: pandora charms

  306. Pingback: michael kors elegantn谩 hned谩 ko啪en谩 kabelka studio mercer large zip

  307. Pingback: pokemon snapback hat

  308. Pingback: liquidation sepatu basket reebok termahal

  309. Pingback: where to buy french beret armyuniverse

  310. Pingback: 斜械蟹锌褉芯胁褨写薪懈泄 屑褨泻褉芯褎芯薪

  311. Pingback: nike air force 1 07 se womens casual shoes black wheat gold white

  312. Pingback: bnwot ladies boohoo black pu leather biker jacket size 18

  313. Pingback: 10 strass forcine fermagli acconciatura capelli sposa

  314. Pingback: oukai 904 affordable classic shape stylish eyeglass frame eyewear glasses

  315. Pingback: standard price 33.00 sale price 15.97. 4.8 out of 5 stars. read reviews. 42

  316. Pingback: tenis new balance 1500

  317. Pingback: violet over the knee boots endource

  318. Pingback: puma beach casual sports sandals leadcat ylm true blue puma white slippers 365630 03 new

  319. Pingback: tiny chou pure color water resistant premium nylon tote handbag green cross body shoulder bag crossbody bags amazon partner link

  320. Pingback: nike puma adidas backpacks sweaters buy nike puma adidas backpacks

  321. Pingback: p谩nske modr茅 tenisky nike air jordan 1 retro high og 41

  322. Pingback: sparkly high top nikes etsy.com

  323. Pingback: calvin klein modern cotton bikini femme sous v锚tements slips black

  324. Pingback: mens summer caps online

  325. Pingback: 15 蟿蟻蠈蟺慰喂 纬喂伪 谓伪 蠁慰蟻苇蟽蔚喂蟼 蟿喂蟼 蟻委纬蔚蟼 蟿畏谓 维谓慰喂尉畏 fays book

  326. Pingback: tenis adidas komet mesh vermelho feminino star t锚nis casuais no

  327. Pingback: heavy duty with roller wheels polyester waterproof duffel

  328. Pingback: black silk robe white a and red bathrobe mens . black collar silk robe

  329. Pingback: caudatus vintage corsets and bustiers plus size flower print bridal bustier corset push up victorian

  330. Pingback: amazon.it pantaloncini adidas donna pantaloncini donna

  331. Pingback: converse sneaker alta all star hi suede shearling fucsia nero eu 30 amazon

  332. Pingback: lightweight safety shoes for womens

  333. Pingback: 蟹邪屑褕械胁褨 褋褍屑泻懈 泻褍锌懈褌懈 胁 泻懈褦胁褨

  334. Pingback: new england patriots vintage snapback sports specialties script hat nfl rare

  335. Pingback: sold out frida kahlo flower tee plus sizes

  336. Pingback: vans shoes vans off the wall hawaiian floral pink tropical

  337. Pingback: a bathing ape bape wing logo brown camo

  338. Pingback: hugo boss hugo boss crew neck logo tracksuit navy blue 50378238

  339. Pingback: basket noire paillet茅e

  340. Pingback: basket nba pantaloncini poco prezzo minnesota timberwolves nike marino maglie

  341. Pingback: womens mens sunglasses spy cyrus cyrus soft matte bla frame cyrus soft matte black

  342. Pingback: armani jeans backpack 100 authentic bought from choice depop

  343. Pingback: marvin the martian warner brothers vintage snapback official 1991 90s adj vtg

  344. Pingback: panama jack venta de los zapatos outlet store

  345. Pingback: jordan holiday release dates 2018 october november december

  346. Pingback: drejeb忙nk med digital styring

  347. Pingback: Pandora Jewelry Official Site

  348. Pingback: only farkkumekko lori sininen

  349. Pingback: tiny bikini extreme micro bikini sexy beachwear micro mini

  350. Pingback: pred谩m michael kors d谩mska pe艌a啪enka

  351. Pingback: boater hat style

  352. Pingback: 泻褍锌懈褌懈 邪泻褋械褋褍邪褉懈 芯泻褍谢褟褉懈 褋芯薪褑械蟹邪褏懈褋薪褨 polaroid

  353. Pingback: cody james cowboy hat size m wool leather band great deal unisex

  354. Pingback: sabres 11 gilbert perreault black ccm throwback stitched nhl jersey

  355. Pingback: hamur sepatu sandal kulit harley davidson sandal kulit pria sandal kulit ban 2

  356. Pingback: patons his and her aran cardigans knitting pattern 6065 1960s

  357. Pingback: buy air jordan 12 playoffs

  358. Pingback: hot girl school bags pu leather bag korean children school bowknot backpacks 1 6 grade girl backpack free shipping cool baby

  359. Pingback: purchase price nike roshe casual ejf

  360. Pingback: givenchy nightingale micro light pink leather satchel bag goxip

  361. Pingback: all the best nfl jerseys atlanta falcons custom youth red nfl jersey canada on sale

  362. Pingback: borse accessori da donna in promozione

  363. Pingback: deal alert wacoal lace impressions racerback contour bra 853257

  364. Pingback: voldsom hovedpine ved h氓rd tr忙ning. konditionstr忙ning motion

  365. Pingback: luna womens flatform peep toe espadrille sliders on onbuy

  366. Pingback: winters hottest sales on womens 1.state tie waist pencil skirt

  367. Pingback: marvel heroes guys pajama pants

  368. Pingback: amy glass end table

  369. Pingback: looking to buy adidas y3 qasa low uk 9 let me know if youre depop

  370. Pingback: sigrid olsen purse black leather cross body bag tradesy

  371. Pingback: gucci 8600m chronograph stainless steel mens watch ya086308 amazon.ca watches

  372. Pingback: lot de v锚tements fille sport 1 short vert rocky 1 body corps

  373. Pingback: bonds zip romper wondersuit pink white bxnma

  374. Pingback: la cesta online en espa帽a creci贸 un 11

  375. Pingback: best day hiking backpack black diamond nitro pack

  376. Pingback: boeing remove before flight keychain

  377. Pingback: modcloth through the bluebells lace dress in black

  378. Pingback: asus mx32vq 80 cm 32 zoll

  379. Pingback: buy puma phenom shimmer whisper white training shoes for women at

  380. Pingback: licras deportivas para el gym al mayor y al detal 2017 en venezuela

  381. Pingback: Travis Scott Jordan 1

  382. Pingback: dell backpacks buy dell backpacks online at best prices in india

  383. Pingback: Nike Air VaporMax Hombre

  384. Pingback: genuine rabbit fur brown ushanka winter hat

  385. Pingback: dresses skirts flawless short romper

  386. Pingback: en la tienda hay mallas

  387. Pingback: havaianas bridal collection fancy flip flops but maybe black

  388. Pingback: stylov茅 d啪铆ny svitavy

  389. Pingback: nike factory outlet

  390. Pingback: chemise femme avec quoi porter

  391. Pingback: abito a portafoglio in velluto devor茅 bordeaux

  392. Pingback: organizador ducha de ba帽o cromado tres estantes colgante 699

  393. Pingback: lyst converse all star hi aruba blue excl in blue for men

  394. Pingback: vistazo zapatillas vans mujer vans ultrarange rapidweld zapatillas

  395. Pingback: acogedor tenis reebok astroride reebok negro blanco zapatos de los hombres tu mejor elecci贸n

  396. Pingback: red high neck wide leg jumpsuit. previous next

  397. Pingback: fossil blue mens watch am 3598

  398. Pingback: ceinture en cuir avec boucle serpent gucci casual femme

  399. Pingback: toddler girl fallwinter outfit kortenstein toddler girl outfits toddler girl fashionista

  400. Pingback: 邪褉褏懈胁 gucci guilty 芯褉懈谐褨薪邪谢 褑褨薪邪 胁 duty free 100 1 450 谐褉薪

  401. Pingback: echarpe burberry beige en cachemire

  402. Pingback: belldini plus size lurex cowl neck sweater sweaters plus sizes

  403. Pingback: adult skeleton halloween black bodysuit women 3d skulls roses rompers womens jumpsuit tight sexy nightclubs bodysuit

  404. Pingback: preowned nike air max acg goaterra black leather ankle boots depop

  405. Pingback: 賮爻丕鬲賷賳 夭賮丕賮 賲賳 睾賷乇 賲亘丕賱睾丞

  406. Pingback: new beautifull simple wool short sweater designs 2018 latest fashion

  407. Pingback: Cheap NFL Jerseys

  408. Pingback: 2017 new tom fashion brand designer cat eye women sunglasses female

  409. Pingback: high neck one piece swimsuit with laced details bleu rod beattie

  410. Pingback: Adidas ZX Flux Hombre Rojas

  411. Pingback: elena wedge sandals tommy hilfiger

  412. Pingback: converse star player skate ox shoes in stock at spot skate shop

  413. Pingback: nike air force 1 white white metallic gold bei

  414. Pingback: tenis tacos para mujer deportes y fitness en mercado libre m茅xico

  415. Pingback: NFL Jerseys 2019

  416. Pingback: glitter converse crystal converse swarovski converse

  417. Pingback: champion womens shoes champion shoes champion clover skimmers champion womens shoes payless

  418. Pingback: DC Shoes HAVEN - Snowboard jacket beige Zip fastening Hood Zip Camouflage Sleeve pocket zip pockets flap pockets inside pocket 100% polyester DC142F00A-B11 UJBYTQP

  419. Pingback: Nike Outlet Store

  420. Pingback: woodland camo marpat operator operators tactical cap hat w import it all

  421. Pingback: backless saree blouse south india fashion

  422. Pingback: new york yankees x gucci accessories are here l o w k e

  423. Pingback: pins needles molly black cupro culotte jumpsuit

  424. Pingback: fatal crash school van 16 years old delhi news times of india

  425. Pingback: oculos gatinho oculos quadrado

  426. Pingback: air force 1 original bling pinterest.com

  427. Pingback: pink baby jumpers sweaters compare prices and buy online

  428. Pingback: reclaimed vintage inspired vestito estivo a righe con bottoni bianco asos grigio righe

  429. Pingback: 9781469360096 nfl dallas cowboys 2019 wall calendar turner licensing calendar

  430. Pingback: white strapless skater dress

  431. Pingback: grigio chiaro uomo maglie e felpe pullover polo ralph lauren

  432. Pingback: lacoste plimsolls sale

  433. Pingback: reef grom boys leather sandals boys from charles clinkard uk

  434. Pingback: new mens nike air jordan 1 hi high strap padded pack black 342132 004 sz

  435. Pingback: big travel bag

  436. Pingback: spy discord prescription sunglasses

  437. Pingback: modern muslim dress plus size floral printed womens long sleeve style muslim dubai arab middle east ladies casual dresses

  438. Pingback: qupid patrol 26 faux suede vegan wedge sneakers black suede pu

  439. Pingback: black black girls snow problemjacket view 0

  440. Pingback: intelligent attractive middle aged caucasian woman in silk blouse and pencil skirt

  441. Pingback: doudoune jott femme cloe taupe

  442. Pingback: sapatos mulher sapatilhas lacoste carnaby evo 119 3 rosa

  443. Pingback: gorra new era 3930 nfl denver broncos onf. naranja y blanco

  444. Pingback: hip trendy maternity hospital gowns animal print and hearts

  445. Pingback: bakham k swiss male slippers beckham flip sandals summer flip flops shoes trend

  446. Pingback: pijama tipo osito enterito h m en argentina rebajas mayo

  447. Pingback: nike air max shoes

  448. Pingback: gosch shoes stylt damen stiefeletten chelsea boots 7105 200 schuhe

  449. Pingback: amazon.com hjl womens light blue turn down collar sleeveless denim

  450. Pingback: Pandora Charms Sale Clearance

  451. Pingback: luv betsey johnson cutie bot robot micro mini backpack

  452. Pingback: next lace knickers brazilian online

  453. Pingback: MOSCHINO Stivaletti Verde militare logo contiene parti non tessili di origine animale interno in pelle punta stretta 11535830JS

  454. Pingback: authentic reebok princess white shoes

  455. Pingback: the authenticity of this vintage gucci business bag briefcase is guaranteed by lxrandco. this exquisite laptop bag was crafted in gg supreme coated canvas

  456. Pingback: nike air vapormax plus silver

  457. Pingback: alexandra taupe jeweled prom dress bridesmaid dresses

  458. Pingback: retro jordans

  459. Pingback: nike free run 5.0 2 women nike men green gray men shoes nike free

  460. Pingback: audrey hepburn posters at

  461. Pingback: Yeezy UK

  462. Pingback: pull fin ajust茅 col v 脿 boutons noir morgan femme

Leave a Reply