Gain insights into debugging Linux process and kernel failures using GDB and crash utility. Delve into memory dump analysis, identifying issues like memory leaks, CPU spikes, and deadlocks.

Accelerated Linux Core Dump Analysis Second Edition.png

support.tar.gz

Challenge

Exercise

Linux_core_dump_analysis_notebook

Live Linux_core_dump_analysis_notebook

Final_Linux_core_dump_analysis_notebook-copy

It’s essential to examine core dumps to diagnose any non-trivial hangs or crashes due to a user process or the kernel faults. Most people have no training in dealing with core dumps, so the causes of most crashes remain undiagnosed.

This course starts with debugging core dumps. You will learn how to debug the Linux process and kernel failures (using the GDB debugger and the crash utility), browse core memory dumps, identify corruption, memory leaks, CPU spikes, halted threads, deadlocks and wait chains, among other things.

Instead of taking a theoretical route, you will focus on hands-on lessons on a variety of memory analysis patterns found in 64-bit core memory dumps from the x64 architecture. You will follow the novel pattern-oriented diagnostic analysis approach that considerably helps with the analysis. Moreover, the course will provide you with problematic applications and the associated core dumps, along with brief descriptions of relevant patterns and some frequently asked questions.

Accelerated Linux Core Dump Analysis

# Congratulations!

Congratulations on completing this course! We hope your new debugging skills help you with the next core dump you encounter and give you an excellent foundational grasp of what's possible with core dump analysis. We hope it got you thinking about the pattern-oriented diagnostic approach.





# Key highlights

Here are some of the key highlights of this course:

* We learned about virtual memory setup in Linux and the storage of memory dumps and explored various dump types such as process, kernel, and fiber bundle memory dumps.
* We developed an understanding of thread stacks and traces, enabling us to analyze and debug thread-related issues.
* We learned how to identify and handle access violation exceptions and runtime exceptions.
* We learned how to collect core dumps in multiple ways.
* We reviewed basic x64 disassembly, including the role of various CPU registers, memory addressing, and various instructions for memory manipulation, flow control, function calls, and how to do stack trace reconstruction.
* We learned how to use GDB, including loading core dump files, printing stack traces, retrieving information, examining memory contents, disassembling functions, and the use of some other frequently used commands.
* We detected issues related to analyzing multi-threaded applications using GDB, enabling us to navigate stack traces, disassemble functions, and manage threads within a multi-threaded environment.
* We gained expertise in identifying `NULL` pointer problems that arise both due to data and code.
* We identified issues arising due to spiking threads, enabling analysis of non-waiting threads and identification of infinite loops.
* We identified issues arising due to dynamic memory corruption, enabling the identification of heap corruption and memory access violations.
* We identified issues that arose due to dividing by zero, enabling the detection of resulting undefined behavior in core dumps.
* We studied how to make use of execution residues and coincidental symbolic information and how to identify C++ exceptions, both handled and unhandled, enabling effective detection and resolution of complex issues.
* We learned how to do critical region analysis, enabling us to identify and examine code regions sandwiched by contending function calls, leading to the identification of synchronization issues.

* We detected issues related to deadlocks and synchronization wait chains, enabling us to identify these problems by analyzing core dumps, thread stacks, specific threads, and disassembled processes.
* We detected issues related to memory dumping for post-processing, function and variable list retrieval, symbol file loading, and variable inspection.
* We learned how to navigate both normal and problematic kernel dumps, identify issues with kernel threads and spiking kernel threads, and identify kernel stack overflows and boundaries using the crash utility, enabling us to analyze and debug kernel-related issues effectively.

Let’s wrap up this course with a final note.

Getting Started

Fundamentals of Core Dump Analysis

Using GDB With Multi-Threaded Applications

Null Pointer Patterns and External Debugging Information

Spiking Threads

Heap Corruption

Stack Corruption and Overflow

Active Threads

Runtime Exceptions and Execution Residues

Heap Errors

Deadlocks

Post-processing

Kernel Dump Analysis with Crash

Wrapping Up

Appendix

Conclusion

Congratulations!

Key highlights