Troubleshooting process hangs using pstack

In software development and operations, process freezes are frequently encountered, leading to system performance degradation or service unavailability. This article introduces the use of the pstack tool to troubleshoot frozen processes by analyzing stack information to identify and resolve root causes.

The risk control system’s sub-service experienced a freeze, rendering the risk control service unavailable. Due to a lack of service availability monitoring, the frozen process was not detected promptly, leading to system unavailability.

Main body

A hung process is a state where the process stops responding but does not exit. This can be caused by various factors, such as deadlocks, resource exhaustion, or exceptions. To resolve these issues, we can use the pstack tool to analyze the process’s stack information and identify the root cause.

Steps

pstack is a commonly used tool, typically provided with gdb (the GNU debugger). You can install it using the following command:

sudo apt-get install gdb

To obtain the process ID: First, we need to get the process ID (PID) of the frozen process. You can use the ps command to list all processes and find the PID you want to troubleshoot. Use the pstack tool to analyze process stacks. Once you have obtained the process ID, you can use pstack to retrieve stack information for that process. Run the following command:

pstack <PID>

This will output the process’s stack information, displaying the current sequence of function calls. Analyzing this information can reveal where the process is stalled and help locate the issue.

Analyzing stack information can help identify the cause of process freezes. This may reveal deadlocks, infinite loops, or other anomalies. Take appropriate measures based on the specific situation, such as releasing locks or fixing code logic.

Case study

A simple demo where, after the main function starts, a new sub-thread is created. The actual execution function enters an infinite loop, causing the program to fail to terminate normally and enter a frozen state.

cmake_minimum_required(VERSION 3.0.0)
project(pstack_main VERSION 0.1.0 LANGUAGES C CXX)

include(CTest)
enable_testing()

# 查找线程库
find_package(Threads REQUIRED)

add_executable(pstack_main main.cpp)

# 链接线程库
target_link_libraries(pstack_main PRIVATE Threads::Threads)

set(CPACK_PROJECT_NAME ${PROJECT_NAME})
set(CPACK_PROJECT_VERSION ${PROJECT_VERSION})
include(CPack)

#include <iostream>
#include <thread>
#include <chrono>

void infiniteLoop() {
    while (true) {
        // 主线程进入死循环
    }
}

int main() {
    std::thread thread(infiniteLoop); // 创建一个线程，执行死循环函数
    thread.join(); // 等待线程结束
    return 0;
}

Starting program, pstack result:

Thread 2 (Thread 0x7eff3619b700 (LWP 1315017)):
#0  infiniteLoop () at /root/pstack/main.cpp:6
#1  0x0000000000402ca9 in std::__invoke_impl<void, void (*)()> (__f=@0x2260eb8: 0x4029a6 <infiniteLoop()>) at /usr/include/c++/8/bits/invoke.h:60
#2  0x0000000000402b02 in std::__invoke<void (*)()> (__fn=@0x2260eb8: 0x4029a6 <infiniteLoop()>) at /usr/include/c++/8/bits/invoke.h:95
#3  0x0000000000403150 in std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul> (this=0x2260eb8) at /usr/include/c++/8/thread:244
#4  0x0000000000403126 in std::thread::_Invoker<std::tuple<void (*)()> >::operator() (this=0x2260eb8) at /usr/include/c++/8/thread:253
#5  0x000000000040310a in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)()> > >::_M_run (this=0x2260eb0) at /usr/include/c++/8/thread:196
#6  0x00007eff36bceb23 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#7  0x00007eff36ea91ca in start_thread () from /lib64/libpthread.so.0
#8  0x00007eff361d58d3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7eff372e1740 (LWP 1315016)):
#0  0x00007eff36eaa6cd in __pthread_timedjoin_ex () from /lib64/libpthread.so.0
#1  0x00007eff36bceda7 in std::thread::join() () from /lib64/libstdc++.so.6
#2  0x00000000004029d2 in main () at /root/pstack/main.cpp:13

The process froze because of a deadlock loop. The main thread entered a deadlock loop, preventing the child threads from exiting, which resulted in the process freeze.