Crash工具介绍和常见命令使用_crash工具使用-程序员宅基地
技术标签： linux Linux debug
1. 介绍
本文主要介绍crash工具的使用以及常用的命令。crash工具，常用来分析内核的coredump以及应用的coredump，功能非常强大。
使用crash分析内核crash情况，需要准备以下内容：
- 内核crash时生成的coredump文件或者raw data
- 运行内核对应的符号表vmlinux
- crash 工具
2. crash基本使用
2.1 解释命令
使用crash工具解析内核crash产生的raw data，命令如下：
crash cluster-vmlinux ramdump@0x57400000
解析成功后，会进入crash工具cmdline模式，如下图所示：
使用crash工具解析内核crash产生的coredump文件，命令如下：
crash vmlinux vmcore
解析成功后，也会进入crash工具的cmdline模式
2.2 crash工具基本输出
KERNEL: cluster-vmlinux [TAINTED]
DUMPFILES: /var/tmp/ramdump_elf_Tp0C6d [temporary ELF header]
ramdump
CPUS: 6
DATE: Mon Apr 18 11:33:58 CST 2022 //panic时间
UPTIME: 01:28:40 //panic时，已运行时间
LOAD AVERAGE: 1.31, 1.32, 1.29
TASKS: 134 //panic时，进程总数
NODENAME: x9h_ms //硬件信息
RELEASE: 4.14.61 //内核版本信息
VERSION: #1 SMP PREEMPT Fri Apr 15 14:25:47 UTC 2022
MACHINE: aarch64 (unknown Mhz)
MEMORY: 544 MB //Linux系统内存数
PANIC: "sysrq: SysRq : Trigger a crash" //panic原因
PID: 2496 //panic进程号
COMMAND: "sh" //panic进程名词
TASK: ffff800023c5b800 [THREAD_INFO: ffff800023c5b800] //panic进程task和thread_info信息
CPU: 5 //系统cpu核数
STATE: TASK_RUNNING (SYSRQ) //panic时任务状态
2.2.1 crash常用命令
- help
该命令是查看crash工具的帮助信息，譬如支持的命令、具体某个命令的使用方法等。
crash> help
* extend log rd task
alias files mach repeat timer
ascii foreach mod runq tree
bpf fuser mount search union
bt gdb net set vm
btop help p sig vtop
dev ipcs ps struct waitq
dis irq pte swap whatis
eval kmem ptob sym wr
exit list ptov sys q
crash version: 8.0.0++ gdb version: 10.2
For help on any command above, enter "help <command>".
For help on input options, enter "help input".
For help on output options, enter "help output".
//查看某个命令使用方法
crash> help ps
NAME
ps - display process status information
SYNOPSIS
ps [-k|-u|-G|-y policy] [-s] [-p|-c|-t|-[l|m][-C cpu]|-a|-g|-r|-S|-A]
[pid | task | command] ...
DESCRIPTION
This command displays process status for selected, or all, processes
in the system. If no arguments are entered, the process data is
is displayed for all processes. Specific processes may be selected
by using the following identifier formats:
pid a process PID.
task a hexadecimal task_struct pointer.
command a command name. If a command name is made up of letters that
are all numerical values, precede the name string with a "\".
If the command string is enclosed within "'" characters, then
the encompassed string must be a POSIX extended regular expression
that will be used to match task names.
The process list may be further restricted by the following options:
-k restrict the output to only kernel threads.
-u restrict the output to only user tasks.
-G display only the thread group leader in a thread group.
-y policy restrict the output to tasks having a specified scheduling policy
expressed by its integer value or by its (case-insensitive) name;
multiple policies may be entered in a comma-separated list:
0 or NORMAL
1 or FIFO
2 or RR
3 or BATCH
4 or ISO
5 or IDLE
6 or DEADLINE
The process identifier types may be mixed. For each task, the following
items are displayed:
1. the process PID.
2. the parent process PID.
3. the CPU number that the task ran on last.
4. the task_struct address or the kernel stack pointer of the process.
(see -s option below)
5. the task state (RU, IN, UN, ZO, ST, TR, DE, SW, WA, PA, ID, NE).
6. the percentage of physical memory being used by this task.
7. the virtual address size of this task in kilobytes.
8. the resident set size of this task in kilobytes.
9. the command name.
The default output shows the task_struct address of each process under a
column titled "TASK". This can be changed to show the kernel stack
pointer under a column titled "KSTACKP".
-s replace the TASK column with the KSTACKP column.
On SMP machines, the active task on each CPU will be highlighted by an
angle bracket (">") preceding its information. If the crash variable
"offline" is set to "hide", the active task on an offline CPU will
be highlighted by a "-" preceding its information.
Alternatively, information regarding parent-child relationships,
per-task time usage data, argument/environment data, thread groups,
or resource limits may be displayed:
-p display the parental hierarchy of selected, or all, tasks.
-c display the children of selected, or all, tasks.
-t display the task run time, start time, and cumulative user
and system times.
-l display the task's last-run timestamp value, using either the
task_struct's last_run value, the task_struct's timestamp value
or the task_struct's sched_entity last_arrival value, whichever
applies, of selected, or all, tasks; the list is sorted with the
most recently-run task (with the largest timestamp) shown first,
followed by the task's current state.
-m similar to -l, but the timestamp value is translated into days,
hours, minutes, seconds, and milliseconds since the task was
last run on a cpu.
-C cpus only usable with the -l or -m options, dump the timestamp data
in per-cpu blocks, where the cpu[s] can be specified as "1,3,5",
"1-3", "1,3,5-7,10", "all", or "a" (shortcut for "all").
-a display the command line arguments and environment strings of
selected, or all, user-mode tasks.
-g display tasks by thread group, of selected, or all, tasks.
-r display resource limits (rlimits) of selected, or all, tasks.
-S display a summary consisting of the number of tasks in a task state.
-A display only the active task on each cpu.
EXAMPLES
Show the process status of all current tasks:
crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
> 0 0 3 c024c000 RU 0.0 0 0 [swapper]
> 0 0 0 c0dce000 RU 0.0 0 0 [swapper]
0 0 1 c0fa8000 RU 0.0 0 0 [swapper]
> 0 0 2 c009a000 RU 0.0 0 0 [swapper]
1 0 1 c0098000 IN 0.0 1096 476 init
2 1 1 c0090000 IN 0.0 0 0 [kflushd]
3 1 1 c000e000 IN 0.0 0 0 [kpiod]
4 1 3 c000c000 IN 0.0 0 0 [kswapd]
5 1 1 c0008000 IN 0.0 0 0 [mdrecoveryd]
253 1 2 fbc4c000 IN 0.0 1088 376 portmap
- bt命令
该命令是查看进程栈信息的，可以通过pid/cpu等选项指定相关信息
crash> bt
PID: 2496 TASK: ffff800023c5b800 CPU: 5 COMMAND: "sh"
#0 [ffff000013573800] __crash_kexec at ffff00000817cee8
#1 [ffff000013573890] (null) at 108142390
#2 [ffff000013573920] psci_sys_reset at ffff000008965ae0
#3 [ffff000013573940] machine_restart at ffff000008085b94
#4 [ffff000013573960] emergency_restart at ffff0000080fddc4
#5 [ffff000013573970] panic at ffff0000080d8c04
#6 [ffff000013573a50] die at ffff00000808ae10
#7 [ffff000013573a90] __do_kernel_fault at ffff00000809feb8
#8 [ffff000013573ac0] do_page_fault at ffff00000809ffdc
#9 [ffff000013573b30] do_translation_fault at ffff0000080a0390
#10 [ffff000013573b40] do_mem_abort at ffff00000808130c
#11 [ffff000013573d20] el1_ia at ffff000008083050
PC: ffff0000086095b0 [sysrq_handle_crash+32]
LR: ffff00000860959c [sysrq_handle_crash+12]
SP: ffff000013573d30 PSTATE: 60400145
X29: ffff000013573d30 X28: ffff800023c5b800 X27: ffff000008cc1000
X26: 0000000000000040 X25: 0000000000000124 X24: 0000000000000000
X23: 0000000000000004 X22: ffff000009217000 X21: ffff000009217f10
X20: 0000000000000063 X19: ffff00000919e000 X18: 0000000000000010
X17: 000000000049b4d8 X16: ffff000008277b80 X15: ffffffffffffffff
X14: ffff0000892fc7a7 X13: ffff0000092fc7b5 X12: ffff00000919e000
X11: ffff00000917ae78 X10: ffff000008434f78 X9: 00000000ffffffd0
X8: 0000000000000015 X7: 6767697254203a20 X6: 00000000000004a3
X5: 0000000000000000 X4: 0000000000000000 X3: 0000000000000000
X2: ffff80002731efb8 X1: 0000000000000000 X0: 0000000000000001
#12 [ffff000013573d30] sysrq_handle_crash at ffff0000086095ac
#13 [ffff000013573d40] __handle_sysrq at ffff000008609b5c
#14 [ffff000013573d80] write_sysrq_trigger at ffff00000860a100
#15 [ffff000013573da0] proc_reg_write at ffff0000082ea22c
#16 [ffff000013573dc0] __vfs_write at ffff00000827760c
#17 [ffff000013573e40] vfs_write at ffff000008277900
#18 [ffff000013573e80] sys_write at ffff000008277bc4
#19 [ffff000013573ff0] el0_svc_naked at ffff000008083abc
PC: 0000ffff9e5aaeac LR: 0000000000408cc4 SP: 0000ffffeea390d0
X29: 0000ffffeea390d0 X28: 0000000000000000 X27: 0000000000000000
X26: 00000000071a3cc5 X25: 000000000049b000 X24: 00000000071a6372
X23: 0000000000000000 X22: 0000000000000001 X21: 00000000071a6370
X20: 0000000000000002 X19: 0000000000000001 X18: 00000000000002c8
X17: 000000000049b4d8 X16: 0000ffff9e5aae80 X15: 0000ffff9e4ebde0
X14: 0000ffff9e4f92c8 X13: 000000000000270f X12: 0101010101010101
X11: 0000000000000000 X10: 0101010101010101 X9: fffffffffffffff0
X8: 0000000000000040 X7: 7f7f7f7f7f7f7f7f X6: 0080000080808080
X5: 0000000000000000 X4: 00000000071a0063 X3: 0000ffff9e63f190
X2: 0000000000000002 X1: 00000000071a6370 X0: 0000000000000001
ORIG_X0: 0000000000000001 SYSCALLNO: 40 PSTATE: 80000000
- set
该命令获取crash的进程信息
crash> set
PID: 2496
COMMAND: "sh"
TASK: ffff800023c5b800 [THREAD_INFO: ffff800023c5b800]
CPU: 5
STATE: TASK_RUNNING (SYSRQ)
- sym
该命令查看符号和符号对应的地址信息，可以根据符号给出对应的地址，也可以根据给出的地址给出对应的符号
crash> sym proc_reg_write
ffff0000082ea1d0 (t) proc_reg_write /linux/fs/proc/inode.c: 224
crash> sym ffff0000082ea1d0
ffff0000082ea1d0 (t) proc_reg_write /linux/fs/proc/inode.c: 224
crash> sym __log_buf
ffff0000092ff020 (b) __log_buf
crash> sym ffff0000092ff020
ffff0000092ff020 (b) __log_buf
- ps
查看系统的进程信息，类似linux的ps
crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
> 0 0 0 ffff000009186300 RU 0.0 0 0 [swapper/0]
> 0 0 1 ffff800025ba9c00 RU 0.0 0 0 [swapper/1]
> 0 0 2 ffff800025baaa00 RU 0.0 0 0 [swapper/2]
> 0 0 3 ffff800025bab800 RU 0.0 0 0 [swapper/3]
> 0 0 4 ffff800025bac600 RU 0.0 0 0 [swapper/4]
0 0 5 ffff800025bad400 RU 0.0 0 0 [swapper/5]
1 0 2 ffff800025b70000 IN 0.1 1824 1276 init
2 0 2 ffff800025b70e00 IN 0.0 0 0 [kthreadd]
4 2 0 ffff800025b72a00 ID 0.0 0 0 [kworker/0:0H]
6 2 0 ffff800025b74600 ID 0.0 0 0 [mm_percpu_wq]
7 2 0 ffff800025b75400 IN 0.0 0 0 [ksoftirqd/0]
8 2 5 ffff800025b76200 ID 0.0 0 0 [rcu_preempt]
9 2 1 ffff800025b77000 ID 0.0 0 0 [rcu_sched]
10 2 0 ffff800025ba8000 ID 0.0 0 0 [rcu_bh]
11 2 0 ffff800025ba8e00 IN 0.0 0 0 [migration/0]
12 2 0 ffff800025bae200 IN 0.0 0 0 [cpuhp/0]
13 2 1 ffff800025baf000 IN 0.0 0 0 [cpuhp/1]
14 2 1 ffff800025be8000 IN 0.0 0 0 [migration/1]
...
- vtop
根据一个虚拟地址转换成对应的物理地址
crash> vtop ffff800025065400
VIRTUAL PHYSICAL
ffff800025065400 65065400
PAGE DIRECTORY: ffff00000938b000
PGD: ffff00000938b800 => 673f8803
PUD: ffff8000273f8000 => 673f7803
PMD: ffff8000273f7940 => f8000065000f11
PAGE: 65000000 (2MB)
PTE PHYSICAL FLAGS
f8000065000f11 65000000 (VALID|SHARED|AF|NG|PXN|UXN|DIRTY)
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
ffff7e0000941940 65065000 dead000000000400 0 0 fffc00000000000
- ptov
根据一个物理地址，转存成对应的内核态虚拟地址
crash> ptov 65065400
VIRTUAL PHYSICAL
ffff800025065400 65065400
- struct
查看结构体信息，如成员偏移，某个结构体变量内存中的值等
crash> struct thread_info -o
struct thread_info {
[0] unsigned long flags;
[8] mm_segment_t addr_limit;
[16] int preempt_count;
}
SIZE: 24
crash> struct thread_info -o -x
struct thread_info {
[0x0] unsigned long flags;
[0x8] mm_segment_t addr_limit;
[0x10] int preempt_count;
}
SIZE: 0x18
crash> struct thread_info 0xffff000009186300
struct thread_info {
flags = 32,
addr_limit = 281474976710655,
preempt_count = 65537
}
crash> task_struct.thread_info 0xffff000009186300 -p
thread_info = {
flags = 32,
addr_limit = 281474976710655,
preempt_count = 65537
},
- dis
反汇编，根据给出的PC地址，反编译出汇编源码，一般结合bt提供的栈信息和寄存器信息和反编译的汇编源码信息分析。
crash> bt
PID: 2496 TASK: ffff800023c5b800 CPU: 5 COMMAND: "sh"
#0 [ffff000013573800] __crash_kexec at ffff00000817cee8
#1 [ffff000013573890] (null) at 108142390
#2 [ffff000013573920] psci_sys_reset at ffff000008965ae0
#3 [ffff000013573940] machine_restart at ffff000008085b94
#4 [ffff000013573960] emergency_restart at ffff0000080fddc4
#5 [ffff000013573970] panic at ffff0000080d8c04
#6 [ffff000013573a50] die at ffff00000808ae10
#7 [ffff000013573a90] __do_kernel_fault at ffff00000809feb8
#8 [ffff000013573ac0] do_page_fault at ffff00000809ffdc
#9 [ffff000013573b30] do_translation_fault at ffff0000080a0390
#10 [ffff000013573b40] do_mem_abort at ffff00000808130c
#11 [ffff000013573d20] el1_ia at ffff000008083050
PC: ffff0000086095b0 [sysrq_handle_crash+32]
LR: ffff00000860959c [sysrq_handle_crash+12]
SP: ffff000013573d30 PSTATE: 60400145
X29: ffff000013573d30 X28: ffff800023c5b800 X27: ffff000008cc1000
X26: 0000000000000040 X25: 0000000000000124 X24: 0000000000000000
X23: 0000000000000004 X22: ffff000009217000 X21: ffff000009217f10
X20: 0000000000000063 X19: ffff00000919e000 X18: 0000000000000010
X17: 000000000049b4d8 X16: ffff000008277b80 X15: ffffffffffffffff
X14: ffff0000892fc7a7 X13: ffff0000092fc7b5 X12: ffff00000919e000
X11: ffff00000917ae78 X10: ffff000008434f78 X9: 00000000ffffffd0
X8: 0000000000000015 X7: 6767697254203a20 X6: 00000000000004a3
X5: 0000000000000000 X4: 0000000000000000 X3: 0000000000000000
X2: ffff80002731efb8 X1: 0000000000000000 X0: 0000000000000001
#12 [ffff000013573d30] sysrq_handle_crash at ffff0000086095ac
#13 [ffff000013573d40] __handle_sysrq at ffff000008609b5c
#14 [ffff000013573d80] write_sysrq_trigger at ffff00000860a100
#15 [ffff000013573da0] proc_reg_write at ffff0000082ea22c
#16 [ffff000013573dc0] __vfs_write at ffff00000827760c
#17 [ffff000013573e40] vfs_write at ffff000008277900
#18 [ffff000013573e80] sys_write at ffff000008277bc4
#19 [ffff000013573ff0] el0_svc_naked at ffff000008083abc
PC: 0000ffff9e5aaeac LR: 0000000000408cc4 SP: 0000ffffeea390d0
X29: 0000ffffeea390d0 X28: 0000000000000000 X27: 0000000000000000
X26: 00000000071a3cc5 X25: 000000000049b000 X24: 00000000071a6372
X23: 0000000000000000 X22: 0000000000000001 X21: 00000000071a6370
X20: 0000000000000002 X19: 0000000000000001 X18: 00000000000002c8
X17: 000000000049b4d8 X16: 0000ffff9e5aae80 X15: 0000ffff9e4ebde0
X14: 0000ffff9e4f92c8 X13: 000000000000270f X12: 0101010101010101
X11: 0000000000000000 X10: 0101010101010101 X9: fffffffffffffff0
X8: 0000000000000040 X7: 7f7f7f7f7f7f7f7f X6: 0080000080808080
X5: 0000000000000000 X4: 00000000071a0063 X3: 0000ffff9e63f190
X2: 0000000000000002 X1: 00000000071a6370 X0: 0000000000000001
ORIG_X0: 0000000000000001 SYSCALLNO: 40 PSTATE: 80000000
//查看栈数据
crash> bt -f 2
PID: 2 TASK: ffff800179d30000 CPU: 1 COMMAND: "kthreadd"
#0 [ffff00000a56bd90] __switch_to at ffff000008086690
ffff00000a56bd90: ffff00000a56bdb0 ffff000008ea5fe8
ffff00000a56bda0: ffff80017feae280 ffff800179d6e580
#1 [ffff00000a56bdb0] __schedule at ffff000008ea5fe4
ffff00000a56bdb0: ffff00000a56be40 ffff000008ea6708
ffff00000a56bdc0: ffff800179d30000 ffff000009a2bda0
ffff00000a56bdd0: ffff000009b8df60 ffff8000f4276600
ffff00000a56bde0: 0000000000000000 ffff000008108b68
ffff00000a56bdf0: 0000000000000001 ffff000008108000
ffff00000a56be00: 0000000000000000 0000000000000000
ffff00000a56be10: ffff000008109f54 ffff00000a56be60
ffff00000a56be20: ffff000008109e80 80838e611e66f600
ffff00000a56be30: ffff000000000004 80838e611e66f600
#2 [ffff00000a56be40] schedule at ffff000008ea6704
ffff00000a56be40: ffff00000a56be60 ffff000008109f54
ffff00000a56be50: ffff8000f4276628 ffff000008109e80
#3 [ffff00000a56be60] kthreadd at ffff000008109f50
//查看异常栈
crash> bt -e
PID: 1803 TASK: ffff8001794fab80 CPU: 5 COMMAND: "pvr_defer_free"
KERNEL-MODE EXCEPTION FRAME AT: ffff00000e7dbbc0
PC: ffff00000876d444 [DeviceMemSet+140]
LR: ffff0000087609c0 [_ZeroPageArray+120]
SP: ffff00000e7dbd00 PSTATE: 20c00145
X29: ffff00000e7dbd00 X28: ffff80015bc9fb00 X27: 00000000000186a0
X26: 0000000000000000 X25: 00e8000000000f0f X24: 0000000000000080
X23: ffff80012d0db000 X22: ffff00001fb7c000 X21: 0000000000000000
X20: 0000000000080000 X19: ffff00001fbaf000 X18: 0000ffffac000bcc
X17: 0000ffff97c22ec8 X16: ffff00000818ded0 X15: 0000ffffac000bc8
X14: 0140000000000000 X13: ffff00001fbfc000 X12: 0000000000000000
X11: 0000000000000000 X10: 0000000000000040 X9: 0040000000000041
X8: 0040000000000001 X7: 0000000000000001 X6: 000000017fffd7e8
X5: ffff8001398feb98 X4: ffff8001398feb98 X3: ffff00001fbfbfff
X2: 0000000000000000 X1: 0000000000000000 X0: ffff00001fbfc000
- irq
查看系统的中断信息
crash> irq
IRQ IRQ_DESC/_DATA IRQACTION NAME
0 (unused) (unused)
1 ffff800025807400 (unused)
2 ffff800025807800 (unused)
3 ffff800025807a00 (unused)
4 ffff800025807c00 ffff80002588df00 "arch_timer"
5 ffff800025807e00 (unused)
6 ffff800024c7e400 (unused)
7 ffff800024c7e600 ffff800024fcc100 "ttyS0"
8 ffff800024c7e800 (unused)
9 ffff800024c7ea00 ffff800024b10680 "30b00000.i2c"
10 ffff800024c7ec00 ffff800024efc200 "30b10000.i2c"
11 ffff800024c7ee00 ffff800024c72f00 "30b30000.i2c"
...
- files
查看进程打开的文件
crash> files
PID: 2496 TASK: ffff800023c5b800 CPU: 5 COMMAND: "sh"
ROOT: / CWD: /home/root
FD FILE DENTRY INODE TYPE PATH
0 ffff800023772500 ffff8000246089c0 ffff8000240b2f28 CHR /dev/ttyS0
1 ffff800024de9500 ffff8000239b29c0 ffff80002399e508 REG /proc/sysrq-trigger
2 ffff800023772500 ffff8000246089c0 ffff8000240b2f28 CHR /dev/ttyS0
10 ffff8000236a3b00 ffff8000245ff300 ffff800025095668 CHR /dev/tty
11 ffff800023772500 ffff8000246089c0 ffff8000240b2f28 CHR /dev/ttyS0
crash> foreach files -R /dev/console //结合foreach命令搜索打开/dev/console文件的进程
PID: 2331 TASK: ffff800024a1b800 CPU: 2 COMMAND: "messagecenter"
ROOT: / CWD: /
FD FILE DENTRY INODE TYPE PATH
1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
PID: 2460 TASK: ffff800024f8aa00 CPU: 5 COMMAND: "adbd"
ROOT: / CWD: /
FD FILE DENTRY INODE TYPE PATH
1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
PID: 2462 TASK: ffff80002421f000 CPU: 2 COMMAND: "usb ffs open"
ROOT: / CWD: /
FD FILE DENTRY INODE TYPE PATH
1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
PID: 2463 TASK: ffff80002421aa00 CPU: 5 COMMAND: "server socket"
ROOT: / CWD: /
FD FILE DENTRY INODE TYPE PATH
1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
PID: 2464 TASK: ffff800024a1aa00 CPU: 5 COMMAND: "->transport"
ROOT: / CWD: /
FD FILE DENTRY INODE TYPE PATH
1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
PID: 2465 TASK: ffff800024a1c600 CPU: 5 COMMAND: "<-transport"
ROOT: / CWD: /
FD FILE DENTRY INODE TYPE PATH
1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
PID: 2478 TASK: ffff800024a1d400 CPU: 1 COMMAND: "start_getty"
ROOT: / CWD: /
FD FILE DENTRY INODE TYPE PATH
0 ffff800023772c00 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
1 ffff800023772c00 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
2 ffff800023772c00 ffff8000245ff3c0 ffff800025095928 CHR /dev/console
- dev
查看系统注册的设备信息
crash> dev
CHRDEV NAME CDEV OPERATIONS
1 mem ffff8000253f5b00 memory_fops
2 pty ffff800024a6ed80 tty_fops
3 ttyp ffff8000249e3280 tty_fops
4 /dev/vc/0 ffff00000934b7a8 console_fops
4 tty ffff80002487e180 tty_fops
4 ttyS ffff800024f02e80 tty_fops
5 /dev/tty ffff00000934a2d0 tty_fops
5 /dev/console ffff00000934a338 console_fops
5 /dev/ptmx ffff00000934a4b0 ptmx_fops
...
BLKDEV NAME GENDISK OPERATIONS
259 blkext (none)
7 loop ffff800024e85000 lo_fops
8 sd (none)
9 md ffff800023db8800 md_fops
31 mtdblock (none)
...
- mount
查看系统挂载的文件系统
crash> mount
MOUNT SUPERBLK TYPE DEVNAME DIRNAME
ffff800025b4c000 ffff800025812000 rootfs rootfs /
ffff800023d0c000 ffff800023db9000 ext4 /dev/root /
ffff800023d0c1c0 ffff8000250b8000 devtmpfs devtmpfs /dev
ffff800023e0c000 ffff800025815000 proc proc /proc
ffff800023e0c1c0 ffff800023e60000 sysfs sysfs /sys
ffff800023e0c380 ffff800025817000 debugfs debugfs /sys/kernel/debug
ffff800023e0c540 ffff80002403f800 configfs configfs /sys/kernel/config
ffff800023e0c700 ffff800023e60800 tmpfs tmpfs /run
ffff800023e0c8c0 ffff800023e61000 tmpfs tmpfs /var/volatile
ffff8000250a8540 ffff8000234e4000 devpts devpts /dev/pts
ffff8000250a8700 ffff800023e61000 tmpfs tmpfs /var/lib
ffff800023e0ca80 ffff80002403f800 configfs none /config
ffff800023e0cc40 ffff800023e62000 functionfs adb /dev/usb-ffs/adb
ffff800023d0c380 ffff800024e19800 tracefs tracefs /sys/kernel/debug/tracing
ffff8000250a88c0 ffff8000234e6800 ext4 /dev/block/by-name/userdata /data
- kmem
查看内核态下内存的使用情况，如slab信息、vmalloc信息等
crash> kmem -i
PAGES TOTAL PERCENTAGE
TOTAL MEM 120690 471.4 MB ----
FREE 76395 298.4 MB 63% of TOTAL MEM
USED 44295 173 MB 36% of TOTAL MEM
SHARED 7202 28.1 MB 5% of TOTAL MEM
BUFFERS 649 2.5 MB 0% of TOTAL MEM
CACHED 10890 42.5 MB 9% of TOTAL MEM
SLAB 7869 30.7 MB 6% of TOTAL MEM
TOTAL HUGE 0 0 ----
HUGE FREE 0 0 0% of TOTAL HUGE
TOTAL SWAP 0 0 ----
SWAP USED 0 0 0% of TOTAL SWAP
SWAP FREE 0 0 0% of TOTAL SWAP
COMMIT LIMIT 60345 235.7 MB ----
COMMITTED 43807 171.1 MB 72% of TOTAL LIMIT
crash> kmem -s //查看kmalloc信息
CACHE OBJSIZE ALLOCATED TOTAL SLABS SSIZE NAME
ffff800024ada180 136 3052 3060 102 4k ext4_groupinfo_1k
ffff800023c96000 144 3 28 1 4k ext4_groupinfo_4k
ffff800024164480 1408 2 46 2 32k UDPv6
ffff800024164300 232 0 0 0 4k tw_sock_TCPv6
ffff800024164180 304 0 0 0 8k request_sock_TCPv6
ffff800024164000 2112 5 75 5 32k TCPv6
ffff8000241c7e00 224 233 882 49 4k nf_conntrack_expect
ffff8000241c7c80 256 0 0 0 8k nf_conntrack
ffff8000241c7b00 312 0 0 0 8k ashmem_area_cache
ffff8000241c7980 3312 0 0 0 32k kcopyd_job
ffff8000241c7800 120 0 0 0 4k dm_rq_target_io
ffff8000241c7680 72 0 0 0 4k isp1760_qtd
ffff8000241c7500 112 13 216 6 4k cfq_io_cq
ffff8000241c7380 240 14 102 6 4k cfq_queue
ffff8000241c7200 216 0 0 0 4k bsg_cmd
ffff8000241c7080 896 1 17 1 16k mqueue_inode_cache
ffff8000241c6f00 656 0 0 0 16k v9fs_inode_cache
...
crash> kmem -S //查看所有slab对象信息
CACHE OBJSIZE ALLOCATED TOTAL SLABS SSIZE NAME
ffff800024ada180 136 3052 3060 102 4k ext4_groupinfo_1k
CPU 0 KMEM_CACHE_CPU:
ffff7dffbfecbfb0
CPU 0 SLAB:
(empty)
CPU 0 PARTIAL:
(empty)
CPU 1 KMEM_CACHE_CPU:
ffff7dffbfee4fb0
CPU 1 SLAB:
SLAB MEMORY NODE TOTAL ALLOCATED FREE
ffff7e0000721f80 ffff80001c87e000 0 30 22 8
FREE / [ALLOCATED]
[ffff80001c87e000]
[ffff80001c87e088]
...
crash> kmem -p //查看page信息
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
ffff7e0000000000 40000000 0 0 0 0
ffff7e0000000040 40001000 0 0 0 0
ffff7e0000000080 40002000 0 0 0 0
ffff7e00000000c0 40003000 0 0 0 0
ffff7e0000000100 40004000 0 0 0 0
ffff7e0000000140 40005000 0 0 0 0
...
- foreach
遍历系统上task的信息，如查看系统上处于UNINTERRUPT状态的task的栈信息
crash> foreach UN bt
PID: 561 TASK: ffff8000e3d33a00 CPU: 0 COMMAND: "kworker/u12:1"
#0 [ffff00002038b5f0] __switch_to at ffff000008086690
#1 [ffff00002038b610] __schedule at ffff000008ea5fe4
#2 [ffff00002038b6a0] schedule at ffff000008ea6704
#3 [ffff00002038b6c0] wait_transaction_locked at ffff0000083b5d8c
#4 [ffff00002038b730] add_transaction_credits at ffff0000083b602c
#5 [ffff00002038b7a0] start_this_handle at ffff0000083b624c
#6 [ffff00002038b840] jbd2__journal_start at ffff0000083b6a50
#7 [ffff00002038b8a0] __ext4_journal_start_sb at ffff0000083564d8
#8 [ffff00002038b8f0] ext4_writepages at ffff0000083760cc
#9 [ffff00002038ba40] do_writepages at ffff000008225400
#10 [ffff00002038bac0] __writeback_single_inode at ffff0000082ee774
#11 [ffff00002038bb10] writeback_sb_inodes at ffff0000082ef180
#12 [ffff00002038bc10] __writeback_inodes_wb at ffff0000082ef4b0
#13 [ffff00002038bc60] wb_writeback at ffff0000082ef828
#14 [ffff00002038bd10] wb_workfn at ffff0000082f0400
#15 [ffff00002038bdd0] process_one_work at ffff000008101950
#16 [ffff00002038be20] worker_thread at ffff000008101c6c
#17 [ffff00002038be70] kthread at ffff000008108c9c
PID: 814 TASK: ffff8001793f8000 CPU: 0 COMMAND: "saftey_heart_be"
#0 [ffff00000c053c10] __switch_to at ffff000008086690
#1 [ffff00000c053c30] __schedule at ffff000008ea5fe4
#2 [ffff00000c053cc0] schedule at ffff000008ea6704
#3 [ffff00000c053ce0] schedule_timeout at ffff000008eaa428
#4 [ffff00000c053d80] msleep at ffff000008178d64
#5 [ffff00000c053da0] saftey_hb_thread at ffff000008ad925c
#6 [ffff00000c053e70] kthread at ffff000008108c9c
PID: 1767 TASK: ffff800020776580 CPU: 0 COMMAND: "AVDemuxer::Read"
#0 [ffff000015dc3780] __switch_to at ffff000008086690
#1 [ffff000015dc37a0] __schedule at ffff000008ea5fe4
#2 [ffff000015dc3830] schedule at ffff000008ea6704
#3 [ffff000015dc3850] wait_transaction_locked at ffff0000083b5d8c
#4 [ffff000015dc38c0] add_transaction_credits at ffff0000083b602c
#5 [ffff000015dc3930] start_this_handle at ffff0000083b624c
#6 [ffff000015dc39d0] jbd2__journal_start at ffff0000083b6a50
#7 [ffff000015dc3a30] __ext4_journal_start_sb at ffff0000083564d8
#8 [ffff000015dc3a80] ext4_da_write_begin at ffff000008378448
#9 [ffff000015dc3c10] generic_perform_write at ffff00000821263c
#10 [ffff000015dc3ca0] __generic_file_write_iter at ffff000008215884
#11 [ffff000015dc3cf0] ext4_file_write_iter at ffff000008362760
#12 [ffff000015dc3d80] __vfs_write at ffff0000082b5180
#13 [ffff000015dc3e30] vfs_write at ffff0000082b53e0
#14 [ffff000015dc3e70] sys_write at ffff0000082b56f8
#15 [ffff000015dc3ff0] el0_svc_naked at ffff000008083af4
PC: 0000ffffa8eaee60 LR: 0000ffffa8eaee48 SP: 0000ffff93ffcdc0
X29: 0000ffff93ffcdc0 X28: 0000000000008000 X27: 0000000002faf080
X26: 0000000000000000 X25: 0000000000000000 X24: 0000000000001000
X23: 0000ffff84004de0 X22: 0000000000001000 X21: 0000ffff84004de0
X20: 0000000000001000 X19: 000000000000001f X18: 0000ffffa8f5ba70
X17: 0000ffffa8e50398 X16: 0000ffffa9b00420 X15: 0000074895555910
X14: 0000000000000000 X13: 52e6647d9bae90b9 X12: 936c08cc82e6992c
X11: 2de7fcffe5393767 X10: 41a928c963510c66 X9: 44377654e95c11b2
X8: 0000000000000040 X7: 042d1f3b698dddd1 X6: 0000000000000000
X5: 0000ffff93ffede0 X4: 00000000ffffffbb X3: 0000000000000000
X2: 0000000000001000 X1: 0000ffff84004de0 X0: 000000000000001f
ORIG_X0: 000000000000001f SYSCALLNO: 40 PSTATE: 80000000
...
- p
打印变量或者表达式的值
crash> p jiffies
jiffies = $1 = 4298283231
crash> p vm_event_states
PER-CPU DATA TYPE:
struct vm_event_state vm_event_states;
PER-CPU ADDRESSES:
[0]: ffff80017fe91ab8
[1]: ffff80017feaaab8
[2]: ffff80017fec3ab8
[3]: ffff80017fedcab8
[4]: ffff80017fef5ab8
[5]: ffff80017ff0eab8
//查看cpu1对应的变量的值
crash> p vm_event_states:1
per_cpu(vm_event_states, 1) = $3 = {
event = {318552, 739584, 0, 0, 558108, 2889268, 0, 0, 0, 0, 0, 0, 0, 4189877, 38713, 21935, 255, 2454551, 259, 0, 27971, 34455, 0, 63883, 0, 0, 0, 0, 62080, 233, 2, 1, 14, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4576, 0, 0, 0, 46844, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 622, 34, 48, 0, 0, 0, 0, 0, 2119, 18, 0, 0, 0, 0, 0, 0, 0, 0, 0}
}
//查看数组元素
crash> p &page_wait_table
$4 = (wait_queue_head_t (*)[256]) 0xffff000009a07d80 <page_wait_table>
crash> p page_wait_table[4]
$5 = {
lock = {
{
rlock = {
raw_lock = {
owner = 6378,
next = 6378
}
}
}
},
head = {
next = 0xffff000009a07de8 <page_wait_table+104>,
prev = 0xffff000009a07de8 <page_wait_table+104>
}
}
- list
遍历由指针或者链表连接起来的内容
//查看挂在panic_notifier_list的notify
crash> p &panic_notifier_list
$3 = (struct atomic_notifier_head *) 0xffff000009b8a670 <panic_notifier_list>
crash> struct atomic_notifier_head
struct atomic_notifier_head {
spinlock_t lock;
struct notifier_block *head;
}
SIZE: 16
crash> struct notifier_block
struct notifier_block {
notifier_fn_t notifier_call;
struct notifier_block *next;
int priority;
}
SIZE: 24
crash> list atomic_notifier_head.head 0xffff000009b8a670 -s notifier_block.next
ffff000009b8a670
next = 0xffff000009afaad0 <sdrv_wdt_panic_event_nb>,
ffff000009afaad0
next = 0xffff000009a3b540 <trace_panic_notifier>,
ffff000009a3b540
next = 0xffff000009a322a0 <rcu_panic_block>,
ffff000009a322a0
next = 0xffff000009a3ad50 <panic_block>,
ffff000009a3ad50
next = 0xffff000009aa5b80 <crashdump_panic_event_nb>,
ffff000009aa5b80
next = 0xffff000009a1b190 <kernel_offset_notifier>,
ffff000009a1b190
next = 0xffff000009a1b588 <cpu_hwcaps_notifier>,
ffff000009a1b588
next = 0xffff000009a1d3b0 <mem_limit_notifier>,
ffff000009a1d3b0
next = 0xffff000009afd460 <heartbeat_panic_nb>,
ffff000009afd460
next = 0x0,
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
智能推荐
while循环&CPU占用率高问题深入分析与解决方案_main函数使用while(1)循环cpu占用99-程序员宅基地
文章浏览阅读3.8k次，点赞9次，收藏28次。直接上一个工作中碰到的问题，另外一个系统开启多线程调用我这边的接口，然后我这边会开启多线程批量查询第三方接口并且返回给调用方。使用的是两三年前别人遗留下来的方法，放到线上后发现确实是可以正常取到结果，但是一旦调用，CPU占用就直接100%(部署环境是win server服务器)。因此查看了下相关的老代码并使用JProfiler查看发现是在某个while循环的时候有问题。具体项目代码就不贴了，类似于下面这段代码。while(flag) {//your code;}这里的flag._main函数使用while(1)循环cpu占用99
【无标题】jetbrains idea shift f6不生效_idea shift +f6快捷键不生效-程序员宅基地
文章浏览阅读347次。idea shift f6 快捷键无效_idea shift +f6快捷键不生效
node.js学习笔记之Node中的核心模块_node模块中有很多核心模块,以下不属于核心模块,使用时需下载的是-程序员宅基地
文章浏览阅读135次。Ecmacript 中没有DOM 和 BOM核心模块Node为JavaScript提供了很多服务器级别，这些API绝大多数都被包装到了一个具名和核心模块中了，例如文件操作的 fs 核心模块 ，http服务构建的http 模块 path 路径操作模块 os 操作系统信息模块// 用来获取机器信息的var os = require('os')// 用来操作路径的var path = require('path')// 获取当前机器的 CPU 信息console.log(os.cpus._node模块中有很多核心模块,以下不属于核心模块,使用时需下载的是
数学建模【SPSS 下载-安装、方差分析与回归分析的SPSS实现(软件概述、方差分析、回归分析)】_化工数学模型数据回归软件-程序员宅基地
文章浏览阅读10w+次，点赞435次，收藏3.4k次。SPSS 22 下载安装过程7.6 方差分析与回归分析的SPSS实现7.6.1 SPSS软件概述1 SPSS版本与安装2 SPSS界面3 SPSS特点4 SPSS数据7.6.2 SPSS与方差分析1 单因素方差分析2 双因素方差分析7.6.3 SPSS与回归分析SPSS回归分析过程牙膏价格问题的回归分析_化工数学模型数据回归软件
利用hutool实现邮件发送功能_hutool发送邮件-程序员宅基地
文章浏览阅读7.5k次。如何利用hutool工具包实现邮件发送功能呢？1、首先引入hutool依赖<dependency> <groupId>cn.hutool</groupId> <artifactId>hutool-all</artifactId> <version>5.7.19</version></dependency>2、编写邮件发送工具类package com.pc.c..._hutool发送邮件
docker安装elasticsearch，elasticsearch-head，kibana，ik分词器_docker安装kibana连接elasticsearch并且elasticsearch有密码-程序员宅基地
文章浏览阅读867次，点赞2次，收藏2次。docker安装elasticsearch，elasticsearch-head，kibana，ik分词器安装方式基本有两种，一种是pull的方式，一种是Dockerfile的方式，由于pull的方式pull下来后还需配置许多东西且不便于复用，个人比较喜欢使用Dockerfile的方式所有docker支持的镜像基本都在https://hub.docker.com/docker的官网上能找到合..._docker安装kibana连接elasticsearch并且elasticsearch有密码
随便推点
Python 攻克移动开发失败！_beeware-程序员宅基地
文章浏览阅读1.3w次，点赞57次，收藏92次。整理 | 郑丽媛出品 | CSDN（ID：CSDNnews）近年来，随着机器学习的兴起，有一门编程语言逐渐变得火热——Python。得益于其针对机器学习提供了大量开源框架和第三方模块，内置..._beeware
Swift4.0_Timer 的基本使用_swift timer 暂停-程序员宅基地
文章浏览阅读7.9k次。//// ViewController.swift// Day_10_Timer//// Created by dongqiangfei on 2018/10/15.// Copyright 2018年 飞飞. All rights reserved.//import UIKitclass ViewController: UIViewController { ..._swift timer 暂停
元素三大等待-程序员宅基地
文章浏览阅读986次，点赞2次，收藏2次。1.硬性等待让当前线程暂停执行，应用场景：代码执行速度太快了，但是UI元素没有立马加载出来，造成两者不同步，这时候就可以让代码等待一下，再去执行找元素的动作线程休眠，强制等待 Thread.sleep(long mills)package com.example.demo;import org.junit.jupiter.api.Test;import org.openqa.selenium.By;import org.openqa.selenium.firefox.Firefox.._元素三大等待
Java软件工程师职位分析_java岗位分析-程序员宅基地
文章浏览阅读3k次，点赞4次，收藏14次。Java软件工程师职位分析_java岗位分析
Java：Unreachable code的解决方法_java unreachable code-程序员宅基地
文章浏览阅读2k次。Java：Unreachable code的解决方法_java unreachable code
标签data-*自定义属性值和根据data属性值查找对应标签_如何根据data-*属性获取对应的标签对象-程序员宅基地
文章浏览阅读1w次。1、html中设置标签data-*的值 标题 11111 222222、点击获取当前标签的data-url的值$('dd').on('click', function() { var urlVal = $(this).data('ur_如何根据data-*属性获取对应的标签对象