forked from openbsd/www
-
Notifications
You must be signed in to change notification settings - Fork 0
/
ddb.html
225 lines (186 loc) · 6.59 KB
/
ddb.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
<!doctype html>
<html lang=en>
<meta charset=utf-8>
<title>OpenBSD: Crash Reports</title>
<meta name="description" content="How to report an OpenBSD kernel crash">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" type="text/css" href="openbsd.css">
<link rel="canonical" href="https://www.openbsd.org/ddb.html">
<style>
h3, h4 {
color: var(--blue);
}
</style>
<h2 id=OpenBSD>
<a href="index.html">
<i>Open</i><b>BSD</b></a>
Crash Reports
</h2>
<hr>
<h3>Minimum information for kernel problems</h3>
<p>
Familiarize yourself with
<a href="report.html">the general bug reporting procedures</a>
first.
All of that will apply.
When reporting a kernel panic or crash, please remember:
<ul>
<li><em>We need the console output on the screen</em>.
Capture it and save it.
Serial consoles are best, but if you are on a VGA console you can
<a href="faq/faq7.html">scroll the console back</a>
and take readable pictures with a phone or camera.
<li><em>If the kernel panicked we need the panic message and the traceback.</em>
It may be displayed on the screen.
If you are at a
<samp><a href="https://man.openbsd.org/ddb.4">ddb</a>></samp>
prompt, type <kbd>show panic</kbd> and <kbd>trace</kbd>.
If you are running SMP, use the <kbd>mach ddbcpu N</kbd> command for each
of the <var>N</var> processors you have and repeat the <kbd>trace</kbd>
command for each processor.
<li><em>We need the process list.</em>
Use the command <kbd>ps</kbd> to get that.
</ul>
<p>
<em>
Reports without the above information are useless.
This is the minimum we need to be able to track down the issue.
</em>
<h3>Additional information you can send</h3>
<p>
In some situations more information is desirable.
Below are outlined some additional steps you can take in certain situations:
<ul>
<li><i>If your crash appears to involve filesystems.</i>
The following additional things would be helpful
<ul>
<li>The output of the
<samp><a href="https://man.openbsd.org/ddb.4">ddb</a>></samp> command
<kbd>show uvm</kbd>
<li>The output of the
<samp><a href="https://man.openbsd.org/ddb.4">ddb</a>></samp>
command <kbd>show bcstats</kbd>
<li>The output of the <kbd>mount</kbd> command from your running machine, so
we know what filesystems are mounted and how.
</ul>
<li> ... XXX boot crash? XXX
<li> ... XXX show regs? XXX
</ul>
<h3>Lost the panic message?</h3>
<p>
Under some circumstances, you may lose the very first message of a panic,
stating the reason for the panic.
<pre class="cmdbox">
ddb> <b>show panic</b>
0: kernel: page fault trap, code=0
ddb>
</pre>
<h3>Note for SMP systems</h3>
<p>
You should get a trace from each processor as part of your report:
<pre class="cmdbox">
ddb{0}> <b>trace</b>
pool_get(d05e7c20,0,dab19ef8,d0169414,80) at pool_get+0x226
fxp_add_rfabuf(d0a62000,d3c12b00,dab19f10,dab19f10) at fxp_add_rfabuf+0xa5
fxp_intr(d0a62000) at fxp_intr+0x1e7
Xintr_ioapic0() at Xintr_ioapic0+0x6d
--- interrupt ---
idle_loop+0x21:
ddb{0}> <b>machine ddbcpu 1</b>
Stopped at Debugger+0x4: leave
ddb{1}> <b>trace</b>
Debugger(d0319e28,d05ff5a0,dab1bee8,d031cc6e,d0a61800) at Debugger+0x4
i386_ipi_db(d0a61800,d05ff5a0,dab1bef8,d01eb997) at i386_ipi_db+0xb
i386_ipi_handler(b0,d05f0058,dab10010,d01d0010,dab10010) at i386_ipi_handler+0x
4a
Xintripi() at Xintripi+0x47
--- interrupt ---
i386_softintlock(0,58,dab10010,dab10010,d01e0010) at i386_softintlock+0x37
Xintrltimer() at Xintrltimer+0x47
--- interrupt ---
idle_loop+0x21:
ddb{1}>
</pre>
<p>
Repeat the <code>machine ddbcpu x</code> followed by <code>trace</code> for each
processor in your machine.
<h3>How do I gather further information from a kernel crash?</h3><p>
<p>
A typical kernel crash on OpenBSD might look like this:
<pre class="cmdbox">
kernel: page fault trap, code=0
Stopped at <b>pf_route+0x263</b>: mov 0x40(%edi),%edx
ddb>
</pre>
<p>
This crash happened at offset <code>0x263</code> in the function <code>pf_route</code>.
<p>
The first command to run from the
<a href="https://man.openbsd.org/ddb">ddb(4)</a> prompt is <code>trace</code>:
<pre class="cmdbox">
ddb> <b>trace</b>
<b>pf_route</b>(e28cb7e4,e28bc978,2,1fad,d0b8b120) at <b>pf_route+0x263</b>
pf_test(2,1f4ad,e28cb7e4,b4c1) at pf_test+0x706
pf_route(e28cbb00,e28bc978,2,d0a65440,d0b8b120) at pf_route+0x207
pf_test(2,d0a65440,e28cbb00,d023c282) at pf_test+0x706
ip_output(d0b6a200,0,0,0,0) at ip_output+0xb67
icmp_send(d0b6a200,0,1,a012) at icmp_send+0x57
icmp_reflect(d0b6a200,0,1,0,3) at icmp_reflect+0x26b
icmp_input(d0b6a200,14,0,0,d0b6a200) at icmp_input+0x42c
ipv4_input(d0b6a200,e289f140,d0a489e0,e289f140) at ipv4_input+0x6eb
ipintr(10,10,e289f140,e289f140,e28cbd38) at ipintr+0x8d
Bad frame pointer: 0xe28cbcac
ddb>
</pre>
<p>
This tells us what function calls lead to the crash.
<p>
To find out the particular line of C code that caused the crash, you can
do the following:
<p>
Find the source file where the crashing function is defined.
In this example, that would be <code>pf_route()</code> in <code>/sys/net/pf.c</code>.
Use <a href="https://man.openbsd.org/objdump">objdump(1)</a> to get the
disassembly:
<pre class="cmdbox">
$ <b>cd /sys/arch/$(uname -m)/compile/GENERIC</b>
$ <b>objdump -dlr obj/pf.o >/tmp/pf.dis</b>
</pre>
<p>
In the output, grep for the function name:
<pre class="cmdbox">
$ <b>grep "<pf_route>:" /tmp/pf.dis</b>
0000<b>7d88</b> <pf_route>:
</pre>
<p>
Take this first hex number <code>7d88</code> and add the offset <code>0x263</code> from
the <code>Stopped at</code> line:
<pre class="cmdbox">
$ <b>printf '%x\n' $((0x7d88 + 0x263))</b>
7feb
</pre>
<p>
Scroll down to the line <code>7feb</code>.
The assembler instruction should match the one quoted in the <code>Stopped at</code>
line.
Then scroll up to the nearest C line number:
<pre class="cmdbox">
$ <b>more /tmp/pf.dis</b>
/sys/net/pf.c:<b>3872</b>
7fe7: 0f b7 43 02 movzwl 0x2(%ebx),%eax
<b>7feb</b>: 8b 57 40 <b>mov 0x40(%edi),%edx</b>
7fee: 39 d0 cmp %edx,%eax
7ff0: 0f 87 92 00 00 00 ja 8088 <pf_route+0x300>
</pre>
<p>
So, it's precisely line <code>3872</code> of <code>pf.c</code> that crashes:
<pre class="cmdbox">
$ <b>nl -ba /sys/net/pf.c | sed -n 3872p</b>
3872 if ((u_int16_t)ip->ip_len <= ifp->if_mtu) {
</pre>
<p>
The kernel that produced the crash output and the object file for objdump must
be compiled from the exact same source file, otherwise the offsets won't match.
<p>
If you provide both the ddb trace output and the relevant objdump section,
that's very helpful.