系统运行环境分析
1. 系統運行環境
客觀的反應系統的執行環境,通常包括如CPU 利用率,Memory 使用情況, Storage 剩余情況等。這些資料也非常重要,比如可以快速的知道,當時是否有Process 在瘋狂的執行,當時是不是處于嚴重的low memory 情況, Storage 是否有耗盡的情況發生等。 2. CPU Usage
追查CPU 利用率可大體的知道,當時機器是否有Process 在瘋狂的運行,? 當時系統運行是否繁忙。通常死機分析,只需要抓取基本的使用情況即可。下面說一下一般的抓取方式 2.1 top
top 可以簡單的查詢Cpu 的基本使用情況
Usage: top [ -m max_procs ] [ -n iterations ] [ -d delay ] [ -s sort_column ] [ -t ] [ -h ]
??? -m num? Maximum number of processes to display.
??? -n num? Updates to show before exiting.
??? -d num? Seconds to wait between updates.
??? -s col? Column to sort by (cpu,vss,rss,thr).
??? -t????? Show threads instead of processes.
??? -h????? Display this help screen.
?
注意的是top 的CPU% 是按全部CPU 來計算的,如果以單線程來計算,比如當時有開啟4個核心,那么最多吃到25%.?
?
個人常見的操作方式如:? top -t -m 5 -n 2 I.E 正常情況 User 2%, System 12%, IOW 0%, IRQ 0%
User 14 + Nice 0 + Sys 67 + Idle 471 + IOW 0 + IRQ 0 + SIRQ 2 = 554 PID?? TID PR CPU% S???? VSS???? RSS PCY UID????? Thread????????? Proc
?2423? 2423? 1?? 8% R?? 2316K?? 1128K???? root???? top???????????? top
? 270?? 270? 0?? 1% S?? 2160K??? 924K???? root???? aee_resmon????? /system/bin/aee_resmon
? 159?? 159? 0?? 0% D????? 0K????? 0K???? root???? bat_thread_kthr
??? 3???? 3? 0?? 0% S????? 0K????? 0K???? root???? ksoftirqd/0
?? 57??? 57? 0?? 0% D????? 0K????? 0K???? root???? hps_main User 1%, System 7%, IOW 0%, IRQ 0%
User 10 + Nice 0 + Sys 41 + Idle 494 + IOW 0 + IRQ 0 + SIRQ 0 = 545 PID?? TID PR CPU% S???? VSS???? RSS PCY UID????? Thread????????? Proc
?2423? 2423? 1?? 8% R?? 2324K?? 1152K???? root???? top???????????? top
?? 57??? 57? 0?? 0% D????? 0K????? 0K???? root???? hps_main
? 242?? 419? 0?? 0% S?? 8600K?? 4540K???? shell??? mobile_log_d??? /system/bin/mobile_log_d
? 982?? 991? 0?? 0% S?? 4364K?? 1156K???? media_rw sdcard????????? /system/bin/sdcard
? 272?? 272? 0?? 0% S? 30680K?? 9048K???? root???? em_svr????????? /system/bin/em_svr
??
? 從上面可以看出, 系統基本運行正常,沒有很吃CPU 的進程. I.E 異常情況 User 59%, System 4%, IOW 2%, IRQ 0%
User 1428 + Nice 0 + Sys 110 + Idle 811 + IOW 67 + IRQ 0 + SIRQ 1 = 2417 PID?? TID PR CPU% S???? VSS???? RSS PCY UID????? Thread????????? Proc
16132 32195? 3? 14% R 997100K? 53492K? bg u0_a60?? Thread-1401???? com.android.mms
16132 32190? 1? 14% R 997100K? 53492K? bg u0_a60?? Thread-1400???? com.android.mms
16132 32188? 2? 14% R 997100K? 53492K? bg u0_a60?? Thread-1399???? com.android.mms
16132 32187? 0? 14% R 997100K? 53492K? bg u0_a60?? Thread-1398???? com.android.mms
18793 18793? 4?? 1% R?? 2068K?? 1020K???? shell??? top???????????? top User 67%, System 3%, IOW 7%, IRQ 0%
User 1391 + Nice 0 + Sys 75 + Idle 435 + IOW 146 + IRQ 0 + SIRQ 1 = 2048 PID?? TID PR CPU% S???? VSS???? RSS PCY UID????? Thread????????? Proc
16132 32195? 3? 16% R 997100K? 53492K? bg u0_a60?? Thread-1401???? com.android.mms
16132 32188? 2? 16% R 997100K? 53492K? bg u0_a60?? Thread-1399???? com.android.mms
16132 32190? 0? 16% R 997100K? 53492K? bg u0_a60?? Thread-1400???? com.android.mms
16132 32187? 1? 16% R 997100K? 53492K? bg u0_a60?? Thread-1398???? com.android.mms
18793 18793? 4?? 2% R?? 2196K?? 1284K???? shell??? top???????????? top 可以明顯的看到,貴司的mms 的4個thread 都有進入了deadloop, 分別占用了一個cpu core.? 同時可以快速抓取他們的java trace,? 進一步可以看到當時MMS 的四個backtrace, 以便快速分析. "Thread-1401" prio=5 tid=32 SUSPENDED JIT
? | group="main" sCount=1 dsCount=0 obj=0x4264f860 self=0x7b183558
? | sysTid=32195 nice=0 sched=0/0 cgrp=apps/bg_non_interactive handle=2078705952
? | state=S schedstat=( 3284456714198 104216273858 383002 ) utm=324720 stm=3725 core=5
? at com.yulong.android.mms.c.f.d(MmsChatDataServer.java:~1095)
? at com.yulong.android.mms.ui.MmsChatActivity$37.run(MmsChatActivity.java:7582)
? at java.lang.Thread.run(Thread.java:841) "Thread-1400" prio=5 tid=31 SUSPENDED JIT
? | group="main" sCount=1 dsCount=0 obj=0x41f5d8f0 self=0x7be2a8e8
? | sysTid=32190 nice=0 sched=0/0 cgrp=apps/bg_non_interactive handle=2078029504
? | state=S schedstat=( 3284905134412 105526230562 382946 ) utm=324805 stm=3685 core=5
? at com.yulong.android.mms.ui.MmsChatActivity$37.run(MmsChatActivity.java:~7586)
? at java.lang.Thread.run(Thread.java:841) "Thread-1399" prio=5 tid=30 SUSPENDED JIT
? | group="main" sCount=1 dsCount=0 obj=0x42564d28 self=0x7b0e6838
? | sysTid=32188 nice=0 sched=0/0 cgrp=apps/bg_non_interactive handle=2077662640
? | state=S schedstat=( 3288042313685 103203810616 375959 ) utm=325143 stm=3661 core=7
? at com.yulong.android.mms.ui.MmsChatActivity$37.run(MmsChatActivity.java:~7586)
? at java.lang.Thread.run(Thread.java:841) "Thread-1398" prio=5 tid=29 SUSPENDED
? | group="main" sCount=1 dsCount=0 obj=0x4248e5a8 self=0x7be0d128
? | sysTid=32187 nice=0 sched=0/0 cgrp=apps/bg_non_interactive handle=2079251904
? | state=S schedstat=( 3287248372432 105116936413 379634 ) utm=325055 stm=3669 core=6
? at com.yulong.android.mms.ui.MmsChatActivity$37.run(MmsChatActivity.java:~7586)
? at java.lang.Thread.run(Thread.java:841) 當時處于suspend, 即意味著當時這四個thread 正在執行java code, 而抓取backtrace 時強制將thread suspend 。看起來客戶改動所致,并且客戶有Proguard, 麻煩客戶自己review 代碼。 2.2 cputime
通常用cputime 來打印一段時間內, CPU 的利用率的統計情況, 資訊比top 更加詳細.
cputime -h
Usage: cputime? [-start/-stop]? [-n iterations] [-d delay] [-e time] [ -m max_count ]? [-p] [-s sort_column] [-i id] [-h]
?-start?????? Start cpu time monitor.
?-stop??????? Stop cpu time monitor.
?-n? num????? Updates to show before exiting.
?-d? num????? Seconds to wait between updates.
?-m? num????? Maximum number of information to display.
?-e? num????? Enable CPU time monitor and stop monitor after num seconds. If no this parameter will
????????????? show last cputime monitor data.
?-p?????????? Show process instead of thread. If no this parameter default will show thread information.
?-s? col????? Column to sort by time/user/kernel/id/isr_c/isr_t(cputime/user time/kernel time/id/isr_count/
????????????? isr_time). If no this parameter default will sort by cputime.
?-i? id?????? show isr information of thread id.
?-h?????????? Display this help screen. *?????? Example???? *
cputime -e 100: Enable cputime monitor and stop after 100 seconds. Then show thread cputime, and sort by cputime.
cputime -e 200? -s user: Enable cputime monitor and stop after 200 seconds. Then show thread cputime, and sort by user time.
cputime: Show thread cputime, and sort by cputime.
cputime -p -s id: Show process cputime, and sort by process id. 2.3 ftrace
ftrace 可以紀錄CPU 最為詳細的執行情況, 即linux scheduler 的執行情況. 通常默認只開啟 sched_switch.?
如何抓取ftrace 可以查詢相關的FAQ. 2.4 Kernel core status
有的時候我們需要追查一下,當時Kernel 的基本調度情況,以及接收中斷的情況,以判斷當前CPU 執行的基本情況是否異常。比如有時候如果某個中斷上來太過頻繁,就容易導致系統運行緩慢,甚至死機。
* CPU Sched status
???? adb shell cat proc/sched_debug
???? Use sysrq-trigger * CPU interrupts
???? adb shell cat proc/interrupts
???? adb shell cat proc/irq/irq_id/spurious 3. Memory Usage
Memory Usage, 我們通常會審查,系統當時memory 是否足夠, 是否處于low memory 狀態, 是否可能出現因無法申請到memory 而卡死的情況.
常見的一些基本信息如下:
* meminfo: basic memory status
adb shell cat proc/meminfo
adb shell cat proc/pid/maps
adb shell cat proc/pid/smaps
* procrank info: all process memory status
adb shell procrank
adb shell procmem pid
adb shell dumpsys meminfo pid
* zoneinfo:
adb shell cat proc/zoneinfo
* buddyinfo:
adb shell cat /proc/buddyinfo 4. Storage Usage
查看Storage 的情況,通常主要是查詢data 分區是否已經刷滿, sdcard 是否已經刷滿, 剩余的空間是否足夠。以及是否有產生超大文件等。
通常使用的命令如 df
df
Filesystem?????????????? Size???? Used???? Free?? Blksize
/dev?????????????????? 446.0M?? 128.0K?? 445.8M?? 4096
/sys/fs/cgroup???????? 446.0M??? 12.0K?? 445.9M?? 4096
/mnt/secure??????????? 446.0M???? 0.0K?? 446.0M?? 4096
/mnt/asec????????????? 446.0M???? 0.0K?? 446.0M?? 4096
/mnt/obb?????????????? 446.0M???? 0.0K?? 446.0M?? 4096
/system????????????????? 1.2G?? 915.3M?? 355.5M?? 4096
/data??????????????????? 1.1G?? 136.7M? 1010.1M?? 4096
/cache???????????????? 106.2M??? 48.0K?? 106.2M?? 4096
/protect_f?????????????? 4.8M??? 52.0K???? 4.8M?? 4096
/protect_s?????????????? 4.8M??? 48.0K???? 4.8M?? 4096
/mnt/cd-rom????????????? 1.2M???? 1.2M???? 0.0K?? 2048
/mnt/media_rw/sdcard0???? 4.6G???? 1.1G???? 3.4G?? 32768
/mnt/secure/asec???????? 4.6G???? 1.1G???? 3.4G?? 32768
/storage/sdcard0???????? 4.6G???? 1.1G???? 3.4G?? 32768 以及ls, du 等命令,如du
du -help
usage: du [-H | -L | -P] [-a | -d depth | -s] [-cgkmrx] [file ...]
du -LP -d 1
8?????? ./lost+found
88????? ./local
384???? ./misc
48????? ./nativebenchmark
912???? ./nativetest
8?????? ./dontpanic
13376?? ./data
8?????? ./app-private
8?????? ./app-asec
129424? ./app-lib
8?????? ./app
136???? ./property
16????? ./ssh
116312? ./dalvik-cache
8?????? ./resource-cache
48????? ./drm
8?????? ./mediadrm
8?????? ./security
3888??? ./nvram
8?????? ./amit
8?????? ./acdapi
88??????./@btmtk
32????? ./sec
8?????? ./user
16????? ./media
16????? ./agps_supl
8?????? ./anr
8?????? ./gps_mnl
8?????? ./nfc_socket
16????? ./ccci_cfg
32????? ./mdlog
1312??? ./system
176???? ./recovery
32????? ./backup
274688? .
客觀的反應系統的執行環境,通常包括如CPU 利用率,Memory 使用情況, Storage 剩余情況等。這些資料也非常重要,比如可以快速的知道,當時是否有Process 在瘋狂的執行,當時是不是處于嚴重的low memory 情況, Storage 是否有耗盡的情況發生等。 2. CPU Usage
追查CPU 利用率可大體的知道,當時機器是否有Process 在瘋狂的運行,? 當時系統運行是否繁忙。通常死機分析,只需要抓取基本的使用情況即可。下面說一下一般的抓取方式 2.1 top
top 可以簡單的查詢Cpu 的基本使用情況
Usage: top [ -m max_procs ] [ -n iterations ] [ -d delay ] [ -s sort_column ] [ -t ] [ -h ]
??? -m num? Maximum number of processes to display.
??? -n num? Updates to show before exiting.
??? -d num? Seconds to wait between updates.
??? -s col? Column to sort by (cpu,vss,rss,thr).
??? -t????? Show threads instead of processes.
??? -h????? Display this help screen.
?
注意的是top 的CPU% 是按全部CPU 來計算的,如果以單線程來計算,比如當時有開啟4個核心,那么最多吃到25%.?
?
個人常見的操作方式如:? top -t -m 5 -n 2 I.E 正常情況 User 2%, System 12%, IOW 0%, IRQ 0%
User 14 + Nice 0 + Sys 67 + Idle 471 + IOW 0 + IRQ 0 + SIRQ 2 = 554 PID?? TID PR CPU% S???? VSS???? RSS PCY UID????? Thread????????? Proc
?2423? 2423? 1?? 8% R?? 2316K?? 1128K???? root???? top???????????? top
? 270?? 270? 0?? 1% S?? 2160K??? 924K???? root???? aee_resmon????? /system/bin/aee_resmon
? 159?? 159? 0?? 0% D????? 0K????? 0K???? root???? bat_thread_kthr
??? 3???? 3? 0?? 0% S????? 0K????? 0K???? root???? ksoftirqd/0
?? 57??? 57? 0?? 0% D????? 0K????? 0K???? root???? hps_main User 1%, System 7%, IOW 0%, IRQ 0%
User 10 + Nice 0 + Sys 41 + Idle 494 + IOW 0 + IRQ 0 + SIRQ 0 = 545 PID?? TID PR CPU% S???? VSS???? RSS PCY UID????? Thread????????? Proc
?2423? 2423? 1?? 8% R?? 2324K?? 1152K???? root???? top???????????? top
?? 57??? 57? 0?? 0% D????? 0K????? 0K???? root???? hps_main
? 242?? 419? 0?? 0% S?? 8600K?? 4540K???? shell??? mobile_log_d??? /system/bin/mobile_log_d
? 982?? 991? 0?? 0% S?? 4364K?? 1156K???? media_rw sdcard????????? /system/bin/sdcard
? 272?? 272? 0?? 0% S? 30680K?? 9048K???? root???? em_svr????????? /system/bin/em_svr
??
? 從上面可以看出, 系統基本運行正常,沒有很吃CPU 的進程. I.E 異常情況 User 59%, System 4%, IOW 2%, IRQ 0%
User 1428 + Nice 0 + Sys 110 + Idle 811 + IOW 67 + IRQ 0 + SIRQ 1 = 2417 PID?? TID PR CPU% S???? VSS???? RSS PCY UID????? Thread????????? Proc
16132 32195? 3? 14% R 997100K? 53492K? bg u0_a60?? Thread-1401???? com.android.mms
16132 32190? 1? 14% R 997100K? 53492K? bg u0_a60?? Thread-1400???? com.android.mms
16132 32188? 2? 14% R 997100K? 53492K? bg u0_a60?? Thread-1399???? com.android.mms
16132 32187? 0? 14% R 997100K? 53492K? bg u0_a60?? Thread-1398???? com.android.mms
18793 18793? 4?? 1% R?? 2068K?? 1020K???? shell??? top???????????? top User 67%, System 3%, IOW 7%, IRQ 0%
User 1391 + Nice 0 + Sys 75 + Idle 435 + IOW 146 + IRQ 0 + SIRQ 1 = 2048 PID?? TID PR CPU% S???? VSS???? RSS PCY UID????? Thread????????? Proc
16132 32195? 3? 16% R 997100K? 53492K? bg u0_a60?? Thread-1401???? com.android.mms
16132 32188? 2? 16% R 997100K? 53492K? bg u0_a60?? Thread-1399???? com.android.mms
16132 32190? 0? 16% R 997100K? 53492K? bg u0_a60?? Thread-1400???? com.android.mms
16132 32187? 1? 16% R 997100K? 53492K? bg u0_a60?? Thread-1398???? com.android.mms
18793 18793? 4?? 2% R?? 2196K?? 1284K???? shell??? top???????????? top 可以明顯的看到,貴司的mms 的4個thread 都有進入了deadloop, 分別占用了一個cpu core.? 同時可以快速抓取他們的java trace,? 進一步可以看到當時MMS 的四個backtrace, 以便快速分析. "Thread-1401" prio=5 tid=32 SUSPENDED JIT
? | group="main" sCount=1 dsCount=0 obj=0x4264f860 self=0x7b183558
? | sysTid=32195 nice=0 sched=0/0 cgrp=apps/bg_non_interactive handle=2078705952
? | state=S schedstat=( 3284456714198 104216273858 383002 ) utm=324720 stm=3725 core=5
? at com.yulong.android.mms.c.f.d(MmsChatDataServer.java:~1095)
? at com.yulong.android.mms.ui.MmsChatActivity$37.run(MmsChatActivity.java:7582)
? at java.lang.Thread.run(Thread.java:841) "Thread-1400" prio=5 tid=31 SUSPENDED JIT
? | group="main" sCount=1 dsCount=0 obj=0x41f5d8f0 self=0x7be2a8e8
? | sysTid=32190 nice=0 sched=0/0 cgrp=apps/bg_non_interactive handle=2078029504
? | state=S schedstat=( 3284905134412 105526230562 382946 ) utm=324805 stm=3685 core=5
? at com.yulong.android.mms.ui.MmsChatActivity$37.run(MmsChatActivity.java:~7586)
? at java.lang.Thread.run(Thread.java:841) "Thread-1399" prio=5 tid=30 SUSPENDED JIT
? | group="main" sCount=1 dsCount=0 obj=0x42564d28 self=0x7b0e6838
? | sysTid=32188 nice=0 sched=0/0 cgrp=apps/bg_non_interactive handle=2077662640
? | state=S schedstat=( 3288042313685 103203810616 375959 ) utm=325143 stm=3661 core=7
? at com.yulong.android.mms.ui.MmsChatActivity$37.run(MmsChatActivity.java:~7586)
? at java.lang.Thread.run(Thread.java:841) "Thread-1398" prio=5 tid=29 SUSPENDED
? | group="main" sCount=1 dsCount=0 obj=0x4248e5a8 self=0x7be0d128
? | sysTid=32187 nice=0 sched=0/0 cgrp=apps/bg_non_interactive handle=2079251904
? | state=S schedstat=( 3287248372432 105116936413 379634 ) utm=325055 stm=3669 core=6
? at com.yulong.android.mms.ui.MmsChatActivity$37.run(MmsChatActivity.java:~7586)
? at java.lang.Thread.run(Thread.java:841) 當時處于suspend, 即意味著當時這四個thread 正在執行java code, 而抓取backtrace 時強制將thread suspend 。看起來客戶改動所致,并且客戶有Proguard, 麻煩客戶自己review 代碼。 2.2 cputime
通常用cputime 來打印一段時間內, CPU 的利用率的統計情況, 資訊比top 更加詳細.
cputime -h
Usage: cputime? [-start/-stop]? [-n iterations] [-d delay] [-e time] [ -m max_count ]? [-p] [-s sort_column] [-i id] [-h]
?-start?????? Start cpu time monitor.
?-stop??????? Stop cpu time monitor.
?-n? num????? Updates to show before exiting.
?-d? num????? Seconds to wait between updates.
?-m? num????? Maximum number of information to display.
?-e? num????? Enable CPU time monitor and stop monitor after num seconds. If no this parameter will
????????????? show last cputime monitor data.
?-p?????????? Show process instead of thread. If no this parameter default will show thread information.
?-s? col????? Column to sort by time/user/kernel/id/isr_c/isr_t(cputime/user time/kernel time/id/isr_count/
????????????? isr_time). If no this parameter default will sort by cputime.
?-i? id?????? show isr information of thread id.
?-h?????????? Display this help screen. *?????? Example???? *
cputime -e 100: Enable cputime monitor and stop after 100 seconds. Then show thread cputime, and sort by cputime.
cputime -e 200? -s user: Enable cputime monitor and stop after 200 seconds. Then show thread cputime, and sort by user time.
cputime: Show thread cputime, and sort by cputime.
cputime -p -s id: Show process cputime, and sort by process id. 2.3 ftrace
ftrace 可以紀錄CPU 最為詳細的執行情況, 即linux scheduler 的執行情況. 通常默認只開啟 sched_switch.?
如何抓取ftrace 可以查詢相關的FAQ. 2.4 Kernel core status
有的時候我們需要追查一下,當時Kernel 的基本調度情況,以及接收中斷的情況,以判斷當前CPU 執行的基本情況是否異常。比如有時候如果某個中斷上來太過頻繁,就容易導致系統運行緩慢,甚至死機。
* CPU Sched status
???? adb shell cat proc/sched_debug
???? Use sysrq-trigger * CPU interrupts
???? adb shell cat proc/interrupts
???? adb shell cat proc/irq/irq_id/spurious 3. Memory Usage
Memory Usage, 我們通常會審查,系統當時memory 是否足夠, 是否處于low memory 狀態, 是否可能出現因無法申請到memory 而卡死的情況.
常見的一些基本信息如下:
* meminfo: basic memory status
adb shell cat proc/meminfo
adb shell cat proc/pid/maps
adb shell cat proc/pid/smaps
* procrank info: all process memory status
adb shell procrank
adb shell procmem pid
adb shell dumpsys meminfo pid
* zoneinfo:
adb shell cat proc/zoneinfo
* buddyinfo:
adb shell cat /proc/buddyinfo 4. Storage Usage
查看Storage 的情況,通常主要是查詢data 分區是否已經刷滿, sdcard 是否已經刷滿, 剩余的空間是否足夠。以及是否有產生超大文件等。
通常使用的命令如 df
df
Filesystem?????????????? Size???? Used???? Free?? Blksize
/dev?????????????????? 446.0M?? 128.0K?? 445.8M?? 4096
/sys/fs/cgroup???????? 446.0M??? 12.0K?? 445.9M?? 4096
/mnt/secure??????????? 446.0M???? 0.0K?? 446.0M?? 4096
/mnt/asec????????????? 446.0M???? 0.0K?? 446.0M?? 4096
/mnt/obb?????????????? 446.0M???? 0.0K?? 446.0M?? 4096
/system????????????????? 1.2G?? 915.3M?? 355.5M?? 4096
/data??????????????????? 1.1G?? 136.7M? 1010.1M?? 4096
/cache???????????????? 106.2M??? 48.0K?? 106.2M?? 4096
/protect_f?????????????? 4.8M??? 52.0K???? 4.8M?? 4096
/protect_s?????????????? 4.8M??? 48.0K???? 4.8M?? 4096
/mnt/cd-rom????????????? 1.2M???? 1.2M???? 0.0K?? 2048
/mnt/media_rw/sdcard0???? 4.6G???? 1.1G???? 3.4G?? 32768
/mnt/secure/asec???????? 4.6G???? 1.1G???? 3.4G?? 32768
/storage/sdcard0???????? 4.6G???? 1.1G???? 3.4G?? 32768 以及ls, du 等命令,如du
du -help
usage: du [-H | -L | -P] [-a | -d depth | -s] [-cgkmrx] [file ...]
du -LP -d 1
8?????? ./lost+found
88????? ./local
384???? ./misc
48????? ./nativebenchmark
912???? ./nativetest
8?????? ./dontpanic
13376?? ./data
8?????? ./app-private
8?????? ./app-asec
129424? ./app-lib
8?????? ./app
136???? ./property
16????? ./ssh
116312? ./dalvik-cache
8?????? ./resource-cache
48????? ./drm
8?????? ./mediadrm
8?????? ./security
3888??? ./nvram
8?????? ./amit
8?????? ./acdapi
88??????./@btmtk
32????? ./sec
8?????? ./user
16????? ./media
16????? ./agps_supl
8?????? ./anr
8?????? ./gps_mnl
8?????? ./nfc_socket
16????? ./ccci_cfg
32????? ./mdlog
1312??? ./system
176???? ./recovery
32????? ./backup
274688? .
總結
- 上一篇: 【精华】PB函数大全
- 下一篇: 【系统分析师之路】系统分析师知识地图