Disappearing Ground Units

[Argoniur]

It's hard to know exactly when it happened, but I'm pretty sure it was after a reload since that's when I noticed my colony was starting to have unrest problems and my construction teams were no longer retrieving alien installations. 

I didn't save before reloading, I had saved about one ingame year before and decided to close and not save because I forgot about a bunch of lifepods and let them die and I wanted to replay it, so after reloading and trying to play the same way is when I noticed they were missing.

[Froggiest1982]

BTW can I just clarify if the units disappear after save, or after the game is reloaded?

quite sure reloaded

[Doren]

Maybe some form of save event log for ground troops could be introduced? Would contain the troops that were saved so that people who are trying to reproduce this could look at them and see which ones went missing? Could be hidden by default

Not sure if worth the trouble though as I guess people could copy the Total force text and compare it to after save reload and even against previous save

[SpikeTheHobbitMage]

Maybe some form of save event log for ground troops could be introduced? Would contain the troops that were saved so that people who are trying to reproduce this could look at them and see which ones went missing? Could be hidden by default

Not sure if worth the trouble though as I guess people could copy the Total force text and compare it to after save reload and even against previous save

One of the things that has been suggested is a more verbose error log, possibly making it general across all tables.

Another possible event chain that complicates diagnosis is a failed save followed by a reload and then creating more units before noticing the problem.

One of the fun possibilities is that more people might be affected than we know about, because if the units are still in memory after the failed save and something happens that clears the cause of the error, then the next save will restore the table with no evidence that there was ever a problem.

[Omnivore]

Caution: pure speculation follows -

Or it might be that the save/load mechanisms have nothing to do with the bug.  Being that the items (game objects)  are being stored in a list and that list is basically a table with records as entries, somewhere there is a mechanism (dictionary/hash table) for translating record ids into list indexes or references to the items themselves perhaps.  In any case, what if the AI is, when marking an item as destroyed (or deleting), it is somehow getting the wrong id?   

Or it may be something else completely different, I'm going to force myself to stop posting on this issue because I'm down to pure guesswork past the above.

[SpikeTheHobbitMage]

Or it may be something else completely different, I'm going to force myself to stop posting on this issue because I'm down to pure guesswork past the above.

I'll second this.  We've helped as much as we reasonably can with the information available.  The ball is in Steve's court now.  All we can do now from our end is to wait and see.  Cheers.

[db48x]

For this type of problem I personally use RR to debug (rr-project.org). It records the entire execution of the program (with minimal overhead), and lets you replay the program backwards and forwards. If you can capture the bug in a recording even once, it becomes very easy to track down where the data was lost or corrupted. You can literally set a watchpoint on a variable or memory location that has the wrong value, then run the program in reverse (as weird as that sounds) until the watchpoint indicates that the value was changed. The program will stop right where this happens, so you can see what was going on. It's basically a superpower.

The downside in this case is that RR only works in Linux, and only on Intel CPUs (it uses the performance counters that Intel added a few years back to do the tracing without adding insane overhead).

However, Microsoft has announced that they're working on this same kind of feature for MSVC, and it's already becoming available for some use cases. I think you have to pay extra for it, and you have to run your program in Azure (Microsoft's computer rental service), and probably jump through some other hoops, but I really recommend looking into it. It makes these kinds of bugs so easy to track down that it's definitely worth it.

[Steve Walmsley (OP)]

Thanks everyone for their help in trying to track this problem and the help in improving the save code.

I'm going to release v1.9 soon, which won't have the new save code, because I don't want to make too many changes at once. Then I will work on the save code change and error logging for the next version.

[AlitarSemiramis]

For this type of problem I personally use RR to debug (rr-project.org). It records the entire execution of the program (with minimal overhead), and lets you replay the program backwards and forwards. If you can capture the bug in a recording even once, it becomes very easy to track down where the data was lost or corrupted. You can literally set a watchpoint on a variable or memory location that has the wrong value, then run the program in reverse (as weird as that sounds) until the watchpoint indicates that the value was changed. The program will stop right where this happens, so you can see what was going on. It's basically a superpower.

The downside in this case is that RR only works in Linux, and only on Intel CPUs (it uses the performance counters that Intel added a few years back to do the tracing without adding insane overhead).

However, Microsoft has announced that they're working on this same kind of feature for MSVC, and it's already becoming available for some use cases. I think you have to pay extra for it, and you have to run your program in Azure (Microsoft's computer rental service), and probably jump through some other hoops, but I really recommend looking into it. It makes these kinds of bugs so easy to track down that it's definitely worth it.

You can do similar things already in C# with several different Microsoft tools: Intellitrace, ETLs, WinDBG/SoS. Though It's hard to do this type of postmortem profiling/investigation unless you have repro steps.

[Second Foundationer]

In case it is still of use and not already solved in 1. 9.  While rummaging around a bit in my aurora manual backup folder, recycle and backup bins in an unsuccessful search for a buried replica of the 1. 8 Antarctic db described in my earlier post, I found that the new, less setup-intensive game I started after that, shows the same issue caught at exactly the right moment, and this time, without any obvious Microsoft in the head disturbance to Aurora saving or loading.  In the db, ground unit formations seem to be there: names, IDs, parent formations, locations and all; and the formation elements, too, as far as I can tell; but from this db, none of them load into Aurora, and with any subsequent save (including load/do nothing/save/quit), the units vanish completely from the records.  – At least the player units, I don't know about NPRs.

One thing I notice in the db is that some NPR formations seem to have both PopulationID and ShipID set to 0.  I don't know what that means and if that's intended, but it feels like an error.  Maybe some bugged NPR unit transport? Or a save issue after all? Or it may be a different disappearing units bug from the main one as someone suggested.

[db48x]

You can do similar things already in C# with several different Microsoft tools: Intellitrace, ETLs, WinDBG/SoS. Though It's hard to do this type of postmortem profiling/investigation unless you have repro steps.

I've not used Intellitrace, but I've heard that it's pretty useful. From what I understand though, it only records things that you've actually seen in the debugger as you stepped through. That does let you step back to any point you've already been. RR lets you record the whole program from start to finish, and then you can debug any point in the recording.

Steps to reproduce are very important; it's no good having any kind of recording if you can't reproduce the bug in one. Once you do catch the bug in a recording, you can replay it as many times as is necessary. One useful strategy has been to record every execution of the program, and only keep those recordings that happened to catch bugs. Another is to record all your automated tests. When a test fails you save the recording for examination.

If Steve had a Linux machine, I could record Aurora every time I ran it then send him the recording once I saw the bug happen. He'd then be able to debug the recording to track down the problem. Second Foundationer seems to have found some very important information. He's found a database where all the ground formations seem to have been saved to the database, but their location information is invalid. That tells us a lot about the error, but it still isn't the smoking gun. If we had a recording of the program as it saved that database, Steve would be able to follow that 0 ID back to where it came from. Maybe the real PopulationID was overwritten, or this copy of the data wasn't completely initialized. Either way, the recording would perfectly preserve that information for Steve to discover.

[Steve Walmsley (OP)]

In case it is still of use and not already solved in 1. 9.  While rummaging around a bit in my aurora manual backup folder, recycle and backup bins in an unsuccessful search for a buried replica of the 1. 8 Antarctic db described in my earlier post, I found that the new, less setup-intensive game I started after that, shows the same issue caught at exactly the right moment, and this time, without any obvious Microsoft in the head disturbance to Aurora saving or loading.  In the db, ground unit formations seem to be there: names, IDs, parent formations, locations and all; and the formation elements, too, as far as I can tell; but from this db, none of them load into Aurora, and with any subsequent save (including load/do nothing/save/quit), the units vanish completely from the records.  – At least the player units, I don't know about NPRs.

One thing I notice in the db is that some NPR formations seem to have both PopulationID and ShipID set to 0.  I don't know what that means and if that's intended, but it feels like an error.  Maybe some bugged NPR unit transport? Or a save issue after all? Or it may be a different disappearing units bug from the main one as someone suggested.

Found it! Well, 95% of it

Thanks for the db. There are four units out of 872 that have 0 for both PopulationID and ShipID. When formations are loaded, any that do not have either a PopulationID and ShipID or skipped so they don't cause errors.

Except, I didn't type 'continue' in the foreach, I typed 'return'. Oops!

I still don't know why those NPR units lost the IDs, but that is a minor problem compared to the effect it had. Also, this only affects ground units, not ships, so any ship-related issue is a separate problem.

Fixed now. I will put out a v1.9.1 with the fix, but it will probably be tomorrow as we are about to open a bottle of wine

[skoormit]

...
Except, I didn't type 'continue' in the foreach, I typed 'return'. Oops!
...

Ahhhhhhhhh...that satisfying feeling when you finally find it.

It's like banging your head against a wall for a few hours, because it feels so good when you stop.

[Bags77]

Hey Guys
I just had all my ground unit dissapear in my latest game.  It seems to be version 1. 51 (at least this is what the rar file was called, the install thread still describes it as version 1. 31)
I see that this bugs is supposed to be fixed in version 1. 9. 1 almost a year ago.
So my question: is the download link in this forum not updated anymore? If so: where can one get the latest version?

[Garfunkel]

Hey Guys
I just had all my ground unit dissapear in my latest game.  It seems to be version 1. 51 (at least this is what the rar file was called, the install thread still describes it as version 1. 31)
I see that this bugs is supposed to be fixed in version 1. 9. 1 almost a year ago.
So my question: is the download link in this forum not updated anymore? If so: where can one get the latest version?

First of all, you should register your account so that you can post normally. Until you register, a moderator has to manually approve each post you make.

Secondly, the latest version is now 1.13.0, Steve just released it this weekend. You can download it here:
http://aurora2.pentarch.org/index.php?topic=10635.0

Just delete your current Aurora folder completely and download that one. Welcome!