Skip to content
This repository has been archived by the owner on Sep 15, 2022. It is now read-only.

Analyzing coredumps archived by the buildbot

Jump to bottom
Rico Wind edited this page Feb 18, 2016 · 4 revisions

Archived coredumps

The intel-based linux and mac builders of our Dartino buildbot are archiving now coredumps of any crash that happens. In case a step in a build caused a crash, it's coredump will be under Buildbot -> Build -> Step -> "coredumps" logfile. The "coredumps" logfile looks like this:

gs://fletch-buildbot-coredumps/36fe9a99-1c2b-4dc9-8238-00bef9ee2760/out/ReleaseX64Clang/fletch-vm (https://storage.cloud.google.com/fletch-buildbot-coredumps/36fe9a99-1c2b-4dc9-8238-00bef9ee2760/out/ReleaseX64Clang/fletch-vm)
gs://fletch-buildbot-coredumps/36fe9a99-1c2b-4dc9-8238-00bef9ee2760/core.6948 (https://storage.cloud.google.com/fletch-buildbot-coredumps/36fe9a99-1c2b-4dc9-8238-00bef9ee2760/core.6948)
gs://fletch-buildbot-coredumps/36fe9a99-1c2b-4dc9-8238-00bef9ee2760/core.7024 (https://storage.cloud.google.com/fletch-buildbot-coredumps/36fe9a99-1c2b-4dc9-8238-00bef9ee2760/core.7024)

For each uploaded binary a gs:// and a https:// URI is given how one can obtain the binaries. Usually it is the fletch-vm binary and one or more core.PID files.

Apple crash reports

The mac bots, in addition to storing the coredumps, also store mac os specific crash reports. These are located in ~/Library/Logs/DiagnosticReports on the bots, ask somebody with access to the bots to get it if you need it.

Download coredumps

An easy way of obtaining the files is via gsutil:

$ mkdir files
$ gsutil -m cp -R 'gs://fletch-buildbot-coredumps/36fe9a99-1c2b-4dc9-8238-00bef9ee2760/*' files
$ tree files
files
├── core.6948
├── core.7024
└── out
    └── ReleaseX64Clang
        └── fletch-vm
$ cd files

Identify binary from which the coredump was created

A coredump (i.e. core.PID file) can come from any binary (e.g. dart, fletch-vm, or even fletch). Though it will be almost always the fletch-vm binary.

The coredump contains environment variables, command line arguments. This can be used to verify the name of the binary:

$ strings core.PID | grep fletch-vm
/mnt/data/b/build/slave/fletch-linux-release-x86/build/fletch/out/ReleaseX64Clang/fletch-vm
...

We also set special environment variables to provide more context:

$ strings core.PID | grep BOT_RUN_PROCESS_CMD
BOT_RUN_PROCESS_CMD=python tools/test.py -mrelease -ax64 --time --report -pbuildbot --step_name=test_ReleaseX64Clang --kill-persistent-process=0 --run-gclient-hooks=0 --build-before-testing=0 --host-checked --clang

$ strings core.PID | grep FEATURE_TEST_TESTNAME
FEATURE_TEST_TESTNAME=compile_time_error_002

Getting dart binaries with symbols.

The dart binaries that we pull down in gclient runhooks are stripped. The sdk builders will, however, also archive unstripped versions of the binary. To get these, find the commit hash of the dart version we use by checking the DEPS file. Double check that the pulled binaries was update in the same commit - look at the history of third_party/bin/linux/dart.sha1.

Go to the sdk builder on the Dartino waterfall, e.g., for linux. Click the build for the commit hash you found above, there is a download link for the unstripped binary.

Analyze coredump

To analyze the crash, one needs to know the binary and the coredump, then gdb can be used:

$ gdb out/DevelopX64/fletch-vm
...
(gdb) core core.PID
[Thread debugging using libthread_db enabled]
...
Program terminated with signal SIGABRT, Aborted.
#0  0x00007faed9951cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56      ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007faed9951cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007faed99550d8 in __GI_abort () at abort.c:89
#2  0x0000000000418871 in fletch::DynamicAssertionHelper::Fail (this=0x7faed89177f0, format=0x480fe2 "expected: %s")
    at ../../src/shared/assert.cc:35
#3  0x0000000000427033 in fletch::Array::get (this=0x2c00829, index=4) at ../../src/vm/object.h:1461
#4  0x000000000042879c in fletch::Engine::Interpret (this=0x7faed8917ce0, target_yield_result=0x7faed8917dc0)
    at ../../src/vm/interpreter.cc:258
#5  0x000000000042e328 in fletch::Interpreter::Run (this=0x7faed8917db0) at ../../src/vm/interpreter.cc:1054
#6  0x000000000044fdaf in fletch::Scheduler::InterpretProcess (this=0x7ffd1cab65c0, process=0x2c5a000, immutable_heap=0x2be0450,
    thread_state=0x2be03f0, allocation_failure=0x7faed8917df7) at ../../src/vm/scheduler.cc:580
#7  0x000000000044fb20 in fletch::Scheduler::RunInterpreterLoop (this=0x7ffd1cab65c0, thread_state=0x2be03f0)
    at ../../src/vm/scheduler.cc:526
#8  0x000000000044f9f9 in fletch::Scheduler::RunInThread (this=0x7ffd1cab65c0) at ../../src/vm/scheduler.cc:489
#9  0x0000000000450eec in fletch::Scheduler::RunThread (data=0x7ffd1cab65c0) at ../../src/vm/scheduler.cc:886
#10 0x0000000000460243 in fletch::ThreadPool::RunThread (arg=0x2be0300) at ../../src/vm/thread_pool.cc:71
#11 0x00007faeda70c182 in start_thread (arg=0x7faed8918700) at pthread_create.c:312
#12 0x00007faed9a1547d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Clone this wiki locally