debugging.md

Stacktraces with maybe some additional information are pretty useful.

This document is also related to profiling.

Python

• My own project: better_exchook
• My own proof-of-concept project: pydbattach - attach to running Python process. More references on this topic in the README.
• LLDB CPython script
• LLDB Python scripting, LLDB Python reference
• Background ZMQ IPython/Jupyter kernel
• faulthandler module (for crashes, like segfault)
• py-spy dump --pid 12345 (via py-spy)
• PyStack (Python). show stack, even including locals.
• See also C below. Esp. if your Python project involves heavy C libs. E.g. load libSegFault.so (example)
• It can be useful to install a signal handler on SIGUSR1 and then print all stacktraces from all threads (via better_exchook).

C/C++ and other native

• LLDB and GDB.
• Otherwise backtrace_symbols_fd() etc.
• Google Breakpad.
• libSegFault library (on Linux)

From CppCon 2018: Greg Law “Debugging Linux C++”:

• GDB
  • Ctrl X-A: curses interface
  • GDB has Python
  • ~/.gdbinit (set history save on, set print pretty on)
  • Breakpoints / watchpoints (read/writes on var, conditional)
  • dprintf (add printf at runtime)
  • Catchpoints
  • Remote debugging, gdbserver
  • Multiprocess debuggin (set follow-fork-mode child|parent, set detach-on-fork off)
  • Temporary breakpoint (tbreak), reg-ex breakpoint (rbreak)
• Valgrind & Sanitizers
  • memcheck
  • valgrind --vgdb ...
  • clang -g -fsanitize=address foo.c
• strace (syscall tracing)
• ltrace (dynamic library call tracing)
• ftrace
  • /sys/kernel/debug/tracing/events/signal/enable etc
  • trace-cmd
• perf trace (similar as strace)
• GDB + rr/Undo
• GDB + Valgrind
• GDB + Asan
• Fortify, compile with -D_FORTIFY_SOURCE=1