Debugging Across Two Runtimes and Two Different Languages | Alex Drum (Hytale) | GameDev Day 2026
In a Nutshell
Cross-runtime debugging requires treating the network protocol as the single source of truth, not the client or server code. Set breakpoints on both sides using a shared Correlation ID (tick + sequence) so you can immediately compare what each side believes about the same packet. When they disagree, the protocol definition—not transmitted data—reveals whether the fault is on the client, server, or in mismatched padding/enum values.
These notes were generated by AI and may contain inaccuracies.
Alex Drum, engine developer and architect at Hytale, begins with a live demonstration of a customer walking across the screen and hitting a box, triggering an immediate error. The inventory displays two items received when only one should have been picked up. The question of whether the error lies on the client or server side is posed before any introduction, establishing the core challenge of debugging across two different operating environments and languages.
When the client and server are two different programs in different languages connected only by a communication protocol, determining where to begin debugging becomes the fundamental question. The visible defect shows on the client with the wrong number in the inventory, but placing breakpoints in the display code wastes an hour because the fault does not lie with the customer. The answer lies in the protocol itself - comparing what each side thinks about the same thing pinpoints where the mistake lies.
The protocol serves as a clear line where the answer should never change across the middle. By asking each side separately what they think about a given value, if they disagree, the search area is immediately halved without opening any source files. Placing trust in the protocol or communication between sides provides a clear distinction about which side is telling the truth and which is lying.
Four outcomes should result from this discussion: first, how to pinpoint where the error lies on which side of the connection before reading any code; second, how to run two debuggers simultaneously without it being a terrible experience while extracting enough information; third, how to read the communication when both debuggers are lying; and fourth, failure patterns unique to each operating environment, where knowing the language's common problems allows earlier guesses based on where the problem occurs.
Returning to the inventory error where the player approaches a box and receives two items instead of one, the initial instinct leads to examining client inventory code. The inventory only displays total inputs received, so the command must have come from the server. However, this back-and-forth wastes significant time. The demonstration shows running debugging with a specific breakpoint on the client side, where the inventory shows zero inputs initially, then receives server and inbox information.
When debugging client-side, the client becomes unresponsive because heartbeats end, reaching a timeout. Scaling through the debugger provides no information when debugging across both sides because it stops working due to the timeout. The client displays two items because the inventory returns any number it's given as a total, meaning the inventory total cannot be the problem - it was given that number. This indicates the problem is actually on the server side.
Once the server is identified as needing fixing, examination reveals the problem in inventory logic that sends information. When an item is given, it goes through code that gives the item and sends an add-on package to the client. The inventory has an assertion listening for the client's response confirming receipt, then a return loop. The demonstration runs a demo server showing the inventory sends a quantity set to one, but the client receives two.
The send and add function sends only the ID, item, and count of one, but the client receives two. The problem lies in the resend logic, which sends information again if not confirmed as received. Lines 41 to 44 contain a resend loop verifying the packet reached the client without information leak, checking only after sending. The issue is confirming receipt after the resend rather than before.
Once the client is determined to only reread and display given information, the problem is confirmed on the server side. The client thinks there are two items while the server thinks there is one, revealing the client was actually telling the truth. The network is the truth, not at fault. Both client and server must be assumed to be lying, requiring proof of which side is at fault.
The second demonstration addresses using two debuggers simultaneously when an error appears on only one side. A client experiencing stutter regresses with almost every beat in the loop. The solution involves making code easier to read across the network by setting breakpoints on each packet's arrival order and monitoring that side when it stops.
Protocol changes include headers written for each packet sent across the network containing the game tick and sequence value on lines 33 and 34. These convert into a Correlation ID tracking packets on both sender and arrival sides, ensuring the correct packet is examined every time. This prevents confusion when one packet arrives a pulse later, causing jitter. Movement on an authoritative server is inherently pulse-dependent with a reconciliation loop.
Tick skew occurs when the server's response arrives slightly late but technically on time. Conditional breakpoints prove this by setting conditions where snapshot.tick equals 400 on both client and server sides. The power of cross-platform debugging in Rider and CLion allows placing snapshots on both sides to pinpoint the issue source. The problem lies with heartbeats, where a client running at 20 Hz experiences timeout at 60 seconds or 3 seconds without receiving heartbeats.
An execution hang set to "none" allows the server to keep running while still receiving breakpoint information without heartbeat crashes. Rider can dump stack traces when problems occur with particular debuggers. Thread-based debugging allows suspending particular threads, with the simulation running in its own separate thread from the main thread and network thread, ensuring heartbeats never stop transmitting.
Controlling conditions allows identifying when pulse 193 arrives at the client but the server sends only pulse 191, indicating a server-side issue with wrong order. The alignment loop causes the server to send information one cycle late. Finding the problem in about 10 minutes saves hours of repetitive searching.
When wolves behave like boxes and boxes behave like wolves, this indicates a data problem. The client clearly shows boxes as wolves and two wolves as boxes. The connection code reads information correctly, and the protocol specifies entity types as Player, Box, Wolf, Other Box, and Torch in numerical format, all appearing correct.
The server simulator sets up a box, wolf, cabinet, and wolf correctly, but they appear in reverse order. The protocol built in two languages (C# and C++) has a bug on the C++ side sending information slightly backward. The C++ protocol shows Box then Wolf, while the C# side shows Wolf then Cabinet because the Box type was added to the server without rewriting the client protocol.
Building both sides using a single central protocol type then building for any language needed provides the solution. Tools like Protobuf allow writing the protocol in one language then creating binding types. One end becomes the respondent while the other listens. The client thinks wolves are boxes because the server sends correct enum values, but number two is a box on the client while a wolf is one, converting the box to a torch (number four) because that enum doesn't exist.
The protocol build problem is discovered by looking at what both sides think about the entity before chasing either side. Both sides think they have correct information and receive correct enumeration values, but they interpret them differently. When the protocol isn't correct, reviewing it finds the problem. Looking at the protocol first, rather than chasing one side first, prevents wasting time.
The simulation is responsible for all information being shared. When examining the information sent via the protocol, the protocol serves as the source of truth. It contains what should be sent, the types of information, how it should be implemented, and a list of messages. In the simulation, the header is written with correct information, and the entity is sent with ID, type, X and Y coordinates, and health. Only X and Y values are being sent, and the network entity appears to be sent correctly.
When switching to the client and examining network code at reception, the entity implementation takes X and Y values. Using console.write line to print all values received from the server reveals the problem: the client receives illogical values such as 3.6×10^-41 and numbers like -13 billion. The server records show it is sending correct X and Y values, confirming the problem lies in how the client reads the data.
The problem is identified in the write loop side where a padding value is written to ensure the information being sent is padded. The packet is supposed to be 106 bytes in size, containing type, ID, X value, Y value, and health. The packet appears to be a simple write to the header, but padding is actually occurring. When reading, X and Y values become incorrect because the padding is overwriting the value, and the client reads over that padded value.
The network entity is defined as an entity record with ID, type, two decimal places, health, and raw data. On the server side, the entity includes ID, type, X and Y padding, and health. The discrepancy shows padding on the server side but not on the client side. This error hides in places where nothing is written in the protocol. The problem is difficult to prove because it is not in the protocol itself, not in the server, and not in the client. Both sides read and understand exactly what they are supposed to understand. The real problem is something not defined in the protocol but defined in the transmitted data.
Defining more in the protocol and not in the transmitted data solves this problem. As long as focus remains on writing everything in the protocol, it becomes a protocol problem. Two-sided debugging works when the method is applied correctly. The problem arises when writing things that aren't in the protocol and sending them over the network.
Every language has a set of failures that look very similar in pattern. The runtime environment can be identified as the culprit by examining the presentation.
Tasks are very good at swallowing up exceptions. If a task is not waited on, the exception simply disappears, released from memory and never heard from again. Garbage collectors like to pause, and this is often interpreted as network lag because the garbage collector can slow things down. When this happens, the network seems to be lagging, as if the server is sending packets slowly. Resumes on the wrong thread create contextual issues that are hard to pinpoint. Rider's debugging feature allows examination of threads and variables, enabling quick review of each thread individually to determine if something is resuming on the wrong thread by examining its state and identifying the point at which the problem occurred.
Debugging C++ and native code is really hard. Padding issues occur across the network. Buffer reuse issues across pulses lead to very strange padding problems where information gets corrupted. Sometimes padded information doesn't cause problems because it is zeroed out, but if the buffer is reused, corrupted information may be written, creating real problems. Use after free occurs when an object is released in a loop and another thread uses it, appearing as data corruption that is really hard to spot. Tracking malloc and free usage is recommended to avoid these problems.
Set a breakpoint before reading any code. Connect the two ends with a common ID so both sides have some information to build on. Then read the bytes. Go to read the padding or network information when the debuggers disagree. Boundaries are checked last, not first. Checking protocol, network, and boundaries first means spending more time debugging protocol boundaries than fixing one side or the other. Make boundaries the source of truth and consider them the answer to the problem, then proceed from there on each side based on which side tells the truth.
Read both sides to see where the truth lies when placing the breakpoint. When reaching the boundaries, they should be the last thing looked at because breakpoints have been placed on both sides, and both contain the answers needed. This method has always worked. Checking boundaries last, making sure they're the last place to look, putting code in, binding each packet to its corresponding packet, reading bytes when debuggers disagree, and putting all of this together is a method that helps significantly.
A shared library is the least favorite way of dealing with the problem because the problem itself is usually on the client. If the protocol is placed on the client, the server must match it and comply with that protocol. This is preferred because the client is doing things like rendering, handling reconciliation, and dealing with all these little things. Having the client enforce the protocol results in fewer runtime errors on the client itself. The server will be much happier to comply with that issue. When the server fails instead of the client, the problem can be quickly detected with better logs on the server side. Steering toward the client is preferred, and shared libraries should be avoided as they represent an added burden.
Generated protocols like Protobuf do some of the heavy lifting. Protobuf is much more widely used than writing protocols on one side or the other. Writing the protocol once in the language being used is preferred. Writing the protocol for the client in C# and generating links for the C++ server to read the same protocol provides greater compatibility and proximity. Only one compilation is needed. Protobuf has links for all existing languages, but it represents an additional place where problems can arise. It is an additional language to deal with and an additional runtime environment to debug. If the protocol is written in C# and links are generated for C++, debugging in C# means debugging the protocol in C#, and running the protocol in C++ means debugging it in C++. If both fail, Protobuf must be consulted, creating a kind of disconnect. Keeping it in one language or the other is preferred in most cases, though Protobuf has advantages.
Live services must deal with clients and servers with slightly different versions and update them over time so that not everyone is up to date at the same time. This differs from environments where users can be forced to update or both sides can be controlled. Protobuf has difficulties here because a protocol can be issued on the client side and a server can run that takes that version into account, connecting only to a version packet it sends. If the client is old, it still has the version it needs to know. A server can be built to the same specifications and kept running within reasonable timeframes.
Keep JetBrains in your library
Save the videos and channels worth coming back to, and find them again in one place.
