The existing communication protocol within the group uses steady_clock as a timestamp to calculate the latency for each individual node. In a specific scenario, a message packet’s built-in timestamp was used, and this timestamp originated from another machine – resulting in an anomalous latency calculation.

As a side note: Gemini2.5 Pro shows promise of completely surpassing GPT-4.

Troubleshooting

I didn’t pay attention to the issue with the base layer timestamp calculation at the beginning, so I just shut down all services and accessed locally to analyze the logs. I found that one service kept failing to stop, continuously sending business data, and there was nothing I could do. I used packet capture on the communication port to locate the machine’s position.

sudo tcpdump -nni any -B 4096 -s 0 -w tmp.pcap port 13100

The internal network situation was complex, and messages were being forwarded through proxies. First, I captured packets on port 13100 from the service’s local machine using tcpdump. Then, I switched to capturing packets on the proxy server on port 13100.

Analysis revealed that requests with excessive latency originated from the Shenzhen office. When troubleshooting the affected services, they were deployed in Shanghai.

Steady Clock and System Clock Differences

std::steady_clock and std::system_clock are the two primary clocks in C++ for time handling. They have the following key differences:

std::system_clock

• Represents “Wall Clock Time”: It represents the actual, real-world time within the system. This time is consistent with the time displayed by the operating system.
• Can be Adjusted: The time of this clock can be adjusted by a user or a system service (such as NTP Network Time Protocol) forward or backward. For example, if you manually modify the system time or the system synchronizes with a time server, the value of system_clock will jump.
• Not Suitable for Measuring Time Intervals: Because it can jump backward, using it to calculate the difference between two time points may result in a negative number or inaccurate results.
• Primary Use: To obtain the current calendar time, used in scenarios where it needs to correspond to real-world time (e.g., logging timestamp).

std::steady_clock

• Monotonic Clock: This clock starts from a certain point and will only steadily move forward, never decreasing. Its rate may be fixed or not (although it’s usually fixed).
• Unadjustable: steady_clock is unaffected by system time changes. Even if the user modifies the system time, it will continue to advance steadily.
• Best for Measuring Time Intervals: Due to its monotonicity, it’s the best choice for measuring code execution time, timeout waits, etc., ensuring accuracy.
• Uncertain Startpoint: Its starting point (epoch) is typically when the system starts, but this isn’t guaranteed by the standard.

Are steady_clock values the same across different machines?

No. The value of steady_clock is not comparable between different machines. Even between two different startup instances on the same machine, its value is not comparable. This is because it was designed to precisely measure a time interval during a single program execution, rather than representing an absolute point in time. Its epoch (start) is undefined and is almost always different across systems or different startup sessions.

Summary

Summary

Summary

Summary

Summary

In simple terms:

• To know “What time is it now?”, use system_clock.
• To know “How long did this code run for?”, use steady_clock.