SurviveTh3Dead

Arma 2 DayZ mod, survival servers, and the people who ran them

Server Monitoring Tools Saw Nothing. The .RPT File Named All 43 of Them.

On 2026-08-09, the DayZ server on CT 161 gamelab crashed hard enough to write a crash file but not hard enough to stop. The file sitting next to the server binary, serverfiles/crash_2026-08-09_16-58-20.log, was 89 bytes. I already knew from three log estates on this server that the small file beside the binary is usually a decoy; the useful log hides in the XDG profile directory.

Unhandled exception

Program: /home/dayzserver/serverfiles/DayZServer
Reason: Unknown

The file worth reading, ~/.local/share/DayZ/crash_2026-08-09_17-07-40.log, was 26,174 bytes. It held 43 Virtual Machine Exception blocks. Every one was a vehicle.

The server had been running the mod set I measured earlier — mods/@CF;mods/@Dabs_Framework;mods/@Community-Online-Tools;mods/@VPPAdminTools;mods/@Trader.

Reading one block properly

The block below is from the reproduction, not the original incident, but it is the pattern that filled 26,174 bytes.

22:23:40 !!! Scripted variables corrupted upon "CivilianSedan_Wine"
22:23:40  INVENTORY    : StreamLoad entity CivilianSedan_Wine:21598
22:23:40   SCRIPT    (E): Virtual Machine Exception

Reason: [EntityAI::OnStoreLoad] :: [WARNING] :: Scripted variables corrupted upon "CivilianSedan_Wine".
Entity will not be loaded correctly.
One exception block showing StreamLoad entity CivilianSedan_Wine:21598 and EntityAI::OnStoreLoad scripted variables corrupted
The StreamLoad line is the clue: this is an entity being read back out of persistence.

Three things matter. EntityAI::OnStoreLoad is the persistence load path. StreamLoad entity CivilianSedan_Wine:21598 names a stored entity by id. And “scripted variables” is modded data attached to a base-game entity.

Because each block writes the phrase Scripted variables corrupted twice, grepping the file returns 86 matching lines. The block count is still 43.

What mods write into persistence

The persistence store at mpmissions/dayzOffline.chernarusplus/storage_1/ contains mod-named subdirectories: expansion/, last written 2026-08-09 17:26, and communityframework/, last written 2026-08-13 22:11, alongside the base game’s data/, backup/, players.db, and spawnpoints.bin.

The persistence store's mod subdirectories expansion and communityframework, next to the mod list with Expansion removed
The store has a directory named after the mod. Then I started the server without it.

Those directory names are the clue. Mods do not keep all state in memory; they write into the same persistence store the base game reads.

The hypothesis

An entity saved while a mod was loaded carries that mod’s scripted variables. Load the same entity without the mod, and EntityAI::OnStoreLoad cannot deserialise them. The error appears as corrupted scripted variables, and the entity will not be loaded correctly.

Before the experiment, one limitation has to be stated directly: the persistence store from 2026-08-09 no longer exists. Later runs overwrote it. I could not replay the original event. What I could test was the mechanism.

The experiment

On 2026-08-13, the server had most recently run mods/@CF;mods/@Dabs_Framework;mods/@DayZ-Expansion-Bundle. That run wrote to persistence; storage_1/expansion/ existed because of it.

Prediction: restart with Expansion removed, against that same store, and the vehicles it touched will fail to load with the same error.

Method: the persistence store was copied aside first — 6.2 MB. The LinuxGSM mods= line was changed to mods/@CF;mods/@Dabs_Framework;mods/@Community-Online-Tools;mods/@VPPAdminTools;mods/@Trader, removing Expansion. The server started cold. The new .RPT was polled every 4 seconds for the string Scripted variables corrupted.

Result: reproduced at roughly 68 seconds.

Metric2026-08-09 (original)2026-08-13 (reproduced)
Virtual Machine Exception blocks4343
crash log size26,174 B26,174 B
distinct vehicle classes1515
Side by side counts: 43 exception blocks and a 26,174 byte crash log in both the original and the reproduction
Same block count, same crash log size, four days and one deliberate change apart.

The distribution matched too. The reproduction logged 86 Scripted variables corrupted lines across those 43 blocks. Per class, the reproduction line counts were: Truck_01_Covered_Orange 10, Sedan_02 10, OffroadHatchback_White 8, CivilianSedan_Wine 8, CivilianSedan_Black 8, Offroad_02 6, OffroadHatchback 6, Hatchback_02_Black 6, Hatchback_02 6, Truck_01_Covered 4, Sedan_02_Red 4, Hatchback_02_Blue 4, Truck_01_Covered_Blue 2, Sedan_02_Grey 2, OffroadHatchback_Blue 2. The original’s block counts were 5, 5, 4, 4, 4, 3, 3, 3, 3, 2, 2, 2, 1, 1, 1 — the same 15 classes in the same proportions.

That result proves the mechanism, not the exact original event. The original persistence state was gone, and the mod removed on 2026-08-13 was not in the 2026-08-09 mod list. What the reproduction shows is that removing a mod that has written into persistence produces an indistinguishable result: 43 blocks, 26,174 bytes, 15 classes.

What would have caught it

The server itself did nothing visible. pgrep -x enfMain showed the process still running after all 43 failures. It wrote a crash log and carried on. The vehicles were simply not loaded correctly.

  • A process check: no. The process lived.
  • A port check: no. The port stayed bound.
  • A query check: no. This never stopped the server answering.
  • The 89-byte crash file: no. It says Reason: Unknown.

Only the game’s own crash log, or a collector explicitly told to watch that hidden XDG directory, contains the answer. Nothing external saw it. What server monitoring tools saw was a running process, a bound port, and an answering query. Log management software would have seen nothing either unless the collection path pointed at that hidden file — which is why the collector configuration I wrote had to be explicit.