The US stock market has three trading sessions: pre-market, live market, and post-market. The logic for pushing data – whether it’s full data or numerical increments – is optimized to conserve bandwidth (sending as little data as possible). Initially, only the full dataset is sent in the first transmission; subsequent transmissions are incremental updates of all fields.
Why not use the optimal solution? This involves multiple project teams, some of which have been live for many years. As we’re a new integration, we can only strive for compatibility.
A Series of Issues
Just from the summary, it might seem like there aren’t any problems, but once the system architecture is brought in, a series of issues arise. Immediately after resolving the previous issue, a new one emerged – this problem was caused by the prior one.
Unable to Identify Trading Intervals
The market phase is defined in protobuf as 0, but due to incremental push data delivery, the business side cannot effectively identify whether this ‘0’ represents the default value or a genuine business value.
In simpler terms: Each time a ‘0’ is received, it’s impossible to determine if it’s the new market phase setting or the protobuf default value.
Introducing Optional
Since protobuf release 3.15, proto3 supports using the optional keyword (just as in proto2) to provide presence information for a scalar field.
The group’s communication protocol is based on protobuf, but due to historical reasons, the version selected was older and did not support the optional keyword. As you know, because we introduced protobuf from the ground up, publishing the project as a static library, this resulted in needing to upgrade the entire build chain, which was a very high cost.
GCC Version Issues
After painstakingly devising a solution, we planned to release two different underlying versions to control the propagation of dependencies for the new protobuf version as much as possible. However, during compilation, we discovered that the gcc version was too low and did not support the new features of protobuf.
The commonly used server types in our team are CentOS7 and CentOS8. The default gcc version on CentOS7 is 4.8, while the default gcc version on CentOS8 is 8.3. Since the new features of protobuf require a gcc version of 7.4 or higher, CentOS7 could not support it.
Ultimately, after a lot of troubleshooting, we moved the deployments of related services and the compilation server to CentOS8, resolving this issue.
Reasonable Enumeration
Reviewing the entire problem, there’s a simpler and more efficient solution: adjust the enumeration definition to start numbering from 1 instead of 0. This effectively distinguishes between default values and business values, avoiding all the aforementioned issues.
Why Starting from 1 is More Reasonable?
In protobuf, enum types have a default value fixed to 0. If we define meaningful business values as 0 (e.g., “Market Open”), in incremental pushes, the business side cannot determine whether the received 0 is a business value or an unset default value. However, if the enum starts from 1, 0 can be reserved for a meaningless default value or “Unknown” state, solving the problem directly.
Recommended Practice:
When designing protobuf enums, always define 0 as a meaningless default value (e.g., UNKNOWN or RESERVED).
Assign actual business values starting from 1 to ensure they are distinguished from the default value of 0.
With this small adjustment, we not only resolved the issue of identifying market hours but also provided a valuable lesson for future protocol design.