Author Topic: Disappearing Ground Units  (Read 15861 times)

0 Members and 1 Guest are viewing this topic.

Offline Argoniur

  • Leading Rate
  • *
  • A
  • Posts: 12
  • Thanked: 4 times
Re: Disappearing Ground Units
« Reply #30 on: April 27, 2020, 02:27:33 PM »
I was 78 years into the campaign, and it did happen after not saving for about 3 hours and leaving the game running without playing for one hour and a half

Edit: also idk if it's the same for everyone but it happened on two different places, in my second colony and in a planet in a distant system where I was excavating some alien installations, only thing I can think of that they have in common is that they were the latest ground units I had trained, about 2-5 years before the bug happened
« Last Edit: April 27, 2020, 02:34:47 PM by Argoniur »
 

Offline Bughunter

  • Bug Moderators
  • Rear Admiral
  • ***
  • Posts: 929
  • Thanked: 132 times
  • Discord Username: Bughunter
Re: Disappearing Ground Units
« Reply #31 on: April 27, 2020, 02:32:10 PM »
I don't know anything about your data structures, but sounds like most logical explanation would be the save loop sometimes don't go over all objects rather than a problem with the inserts. Maybe a problem going through a hierarchical structure like command chain? An early return somewhere? A loop variable not going all the way like i++ not reaching last item in list. Objects moved to some separate/temporary list not included in the save operation.
 

Offline Steve Walmsley (OP)

  • Aurora Designer
  • Star Marshal
  • S
  • Posts: 11675
  • Thanked: 20461 times
Re: Disappearing Ground Units
« Reply #32 on: April 27, 2020, 02:51:19 PM »
I don't know anything about your data structures, but sounds like most logical explanation would be the save loop sometimes don't go over all objects rather than a problem with the inserts. Maybe a problem going through a hierarchical structure like command chain? An early return somewhere? A loop variable not going all the way like i++ not reaching last item in list. Objects moved to some separate/temporary list not included in the save operation.

I'm not using loops that way for saving. I use LINQ to retrieve a collection of objects then cycle through using foreach. If that was a problem, it would always be a problem.

BTW do ALL ground units vanish or only some? If only some, are they different in some ways to the survivors?
 

Offline Kristover

  • Gold Supporter
  • Lt. Commander
  • *****
  • K
  • Posts: 259
  • Thanked: 135 times
  • Gold Supporter Gold Supporter : Support the forums with a Gold subscription
    2021 Supporter 2021 Supporter : Donate for 2021
    2022 Supporter 2022 Supporter : Donate for 2022
    2023 Supporter 2023 Supporter : Donate for 2023
Re: Disappearing Ground Units
« Reply #33 on: April 27, 2020, 02:57:29 PM »
I don't know if this helps or not but I did had another recurrence of the 'disappearing' bug and have the db attached - in this particular case I lost all my ships and strangely enough, the most recent CMC militia unit (it was the only new ground unit constructed since the previous save cycle a year earlier).  This case, I saved the game and watched the hourglass disappear indicating save complete (I think I even opened up a window and adjusted one ships error right before leaving it untended so the save was complete) and then left it unattended for an hour while I did a work Zoom conference.  Came back and saw the hourglass had come back and after waiting near an hour for the hourglass to go away while I did other things I forced closed it.  When I reloaded the game and took inventory, all the ships I had constructed since the beginning were gone - you can still see their fleets they were part of in the ship tab and their designs but no ship entries - and although I saw all of my ground troops were there, the civilian CMC unit was gone.  Just as an additional oddity, all the commanders of the ships are still there in the commander tabbed but registering as unassigned - BUT I had assigned one of my spare officers, Captain Keith Derouen, to command the civilian force unit (Broker Mineral Group - Cybele Attachment) and he is there but still registered as assigned to the unit as if it still exists which is different than the ship commanders who register as unassigned.  If I hit any increment of time to progress, I get an unending amount of Function #787 Sequence Contains no Elements errors.  The attached DB is immediately after reloading the first time so it is RIGHT after the error occurred.

The game is version 1.7.3 (I haven't updated to 1.8 yet) and it is 18 years in. 
« Last Edit: April 27, 2020, 03:05:22 PM by Kristover »
 

Offline skoormit

  • Rear Admiral
  • **********
  • Posts: 804
  • Thanked: 324 times
Re: Disappearing Ground Units
« Reply #34 on: April 27, 2020, 03:09:11 PM »
Apart from system bodies, the game deletes the existing table and inserts the data from the program.

Does the delete for each table happen immediately prior to the insert, or do you do all the deletes at once and then all of the inserts?

I know you are opening and closing a connection for each insert. Are you reusing the same connection, or creating a new connection each time?

What would happen if you tried to insert a duplicate primary key? Would that possibly result in a silent failure that either ignores the duplicate row or overwrites the original with the duplicate?
 

Offline Argoniur

  • Leading Rate
  • *
  • A
  • Posts: 12
  • Thanked: 4 times
Re: Disappearing Ground Units
« Reply #35 on: April 27, 2020, 03:10:44 PM »
Quote from: Argoniur link=topic=11094. msg128629#msg128629 date=1588015653
I was 78 years into the campaign, and it did happen after not saving for about 3 hours and leaving the game running without playing for one hour and a half

Edit: also idk if it's the same for everyone but it happened on two different places, in my second colony and in a planet in a distant system where I was excavating some alien installations, only thing I can think of that they have in common is that they were the latest ground units I had trained, about 2-5 years before the bug happened

Quote from: Steve Walmsley link=topic=11094. msg128635#msg128635 date=1588017079
Quote from: Bughunter link=topic=11094. msg128631#msg128631 date=1588015930
I don't know anything about your data structures, but sounds like most logical explanation would be the save loop sometimes don't go over all objects rather than a problem with the inserts.  Maybe a problem going through a hierarchical structure like command chain? An early return somewhere? A loop variable not going all the way like i++ not reaching last item in list.  Objects moved to some separate/temporary list not included in the save operation.

I'm not using loops that way for saving.  I use LINQ to retrieve a collection of objects then cycle through using foreach.  If that was a problem, it would always be a problem.

BTW do ALL ground units vanish or only some? If only some, are they different in some ways to the survivors?

In my case all the units in both locations vanished
 

Offline Steve Walmsley (OP)

  • Aurora Designer
  • Star Marshal
  • S
  • Posts: 11675
  • Thanked: 20461 times
Re: Disappearing Ground Units
« Reply #36 on: April 27, 2020, 03:12:06 PM »
Apart from system bodies, the game deletes the existing table and inserts the data from the program.

Does the delete for each table happen immediately prior to the insert, or do you do all the deletes at once and then all of the inserts?

I know you are opening and closing a connection for each insert. Are you reusing the same connection, or creating a new connection each time?

What would happen if you tried to insert a duplicate primary key? Would that possibly result in a silent failure that either ignores the duplicate row or overwrites the original with the duplicate?

Each table is a completely separate function. Open connection, delete table, open transaction, create all inserts, commit transaction, close connection. Move to next function.

Here is some example code

        public void SaveSystems()
        {
            try
            {
                // establish a database connection
                SQLiteConnection AuroraDB = new SQLiteConnection(GlobalValues.AuroraConnectionString);
                AuroraDB.Open();

                // delete the existing records
                SQLiteCommand del = new SQLiteCommand("DELETE FROM FCT_System WHERE GameID = " + GameID, AuroraDB);
                del.ExecuteNonQuery();

                using (var cmd = new SQLiteCommand(AuroraDB))
                {
                    using (SQLiteTransaction transaction = AuroraDB.BeginTransaction())
                    {
                        // insert new records
                        foreach (StarSystem obj in SystemList.Values)
                        {
                            cmd.CommandText =
                                @"INSERT INTO FCT_System (SystemID, SystemNumber, Age, AbundanceModifier, Stars, GameID, JumpPointSurveyPoints, SystemTypeID, DustDensity, SolSystem, NoSensorChecks )
                                VALUES ( @SystemID, @SystemNumber, @Age, @AbundanceModifier, @Stars, @GameID, @JumpPointSurveyPoints, @SystemTypeID, @DustDensity, @SolSystem, @NoSensorChecks)";

                            cmd.Parameters.AddWithValue("@SystemID", obj.SystemID);
                            cmd.Parameters.AddWithValue("@SystemNumber", obj.SystemNumber);
                            cmd.Parameters.AddWithValue("@Age", obj.Age);
                            cmd.Parameters.AddWithValue("@AbundanceModifier", obj.AbundanceModifier);
                            cmd.Parameters.AddWithValue("@Stars", obj.Stars);
                            cmd.Parameters.AddWithValue("@GameID", GameID);
                            cmd.Parameters.AddWithValue("@JumpPointSurveyPoints", obj.JumpPointSurveyPoints);
                            cmd.Parameters.AddWithValue("@SystemTypeID", obj.SystemTypeID);
                            cmd.Parameters.AddWithValue("@DustDensity", obj.DustDensity);
                            cmd.Parameters.AddWithValue("@SolSystem", obj.SolSystem);
                            cmd.Parameters.AddWithValue("@NoSensorChecks", obj.NoSensorChecks);
                            cmd.ExecuteNonQuery();
                        }

                        transaction.Commit();
                    }
                }

                AuroraDB.Close();
            }
            catch (Exception error) { GlobalValues.ErrorHandler(error, 1430); return; }
        }
 
The following users thanked this post: db48x

Offline Bughunter

  • Bug Moderators
  • Rear Admiral
  • ***
  • Posts: 929
  • Thanked: 132 times
  • Discord Username: Bughunter
Re: Disappearing Ground Units
« Reply #37 on: April 27, 2020, 03:14:26 PM »
What Kristover describes points towards an endless loop somewhere in the save function, that is technically possible to achieve with LINQ expressions, example: https://stackoverflow.com/questions/31993443/why-does-this-method-result-in-an-infinite-loop
 

Offline Steve Walmsley (OP)

  • Aurora Designer
  • Star Marshal
  • S
  • Posts: 11675
  • Thanked: 20461 times
Re: Disappearing Ground Units
« Reply #38 on: April 27, 2020, 03:17:05 PM »
What Kristover describes points towards an endless loop somewhere in the save function, that is technically possible to achieve with LINQ expressions, example: https://stackoverflow.com/questions/31993443/why-does-this-method-result-in-an-infinite-loop

I've never had an infinite loop error in a foreach - its is usually while loops that are the issue. The above just looks like a recursive function.
 

Offline Bughunter

  • Bug Moderators
  • Rear Admiral
  • ***
  • Posts: 929
  • Thanked: 132 times
  • Discord Username: Bughunter
Re: Disappearing Ground Units
« Reply #39 on: April 27, 2020, 03:20:15 PM »
No, with the above code I cannot see how that could happen. Just assumed you might be using some more complex LINQ selection somewhere to pull data into your save function.
 

Offline JuJo

  • Able Ordinary Rate
  • J
  • Posts: 4
Re: Disappearing Ground Units
« Reply #40 on: April 27, 2020, 03:21:25 PM »
It happened to me about 12 years into a TN start.  All my ground units were affected.  I normally save every time Im traveling through an unexplored JP.  I think at the time I saved and closed the game maybe an hour(at most) passed before the previous save.  I didn’t had any STO units, just infantry, armor and GEO units.  Some units were in my 2 other colonies.  They all disappeared.
 

Offline Steve Walmsley (OP)

  • Aurora Designer
  • Star Marshal
  • S
  • Posts: 11675
  • Thanked: 20461 times
Re: Disappearing Ground Units
« Reply #41 on: April 27, 2020, 03:23:12 PM »
No, with the above code I cannot see how that could happen. Just assumed you might be using some more complex LINQ selection somewhere to pull data into your save function.

The collection there already exists so I just iterate. Occasionally I will use some simple LINQ prior to the loop to get collections from within a collection, such as:

List<MoveOrder> Orders = FleetsList.Values.SelectMany(x => x.MoveOrderList.Values).ToList();

and then foreach that to save.

Most major things like fleets, ships, systems, ground formations, are held in collections at game level. I use LINQ in functions to pull out the ones I need for a particular purpose, but the saves are almost all collections that exists for the whole game.
« Last Edit: April 27, 2020, 03:24:55 PM by Steve Walmsley »
 

Offline skoormit

  • Rear Admiral
  • **********
  • Posts: 804
  • Thanked: 324 times
Re: Disappearing Ground Units
« Reply #42 on: April 27, 2020, 03:24:38 PM »
Here is some example code
...

First thing that stands out is that the connection itself is not wrapped in a using statement.
So, you are closing the connection, but not disposing it.
That can lead to funky problems that don't show up in basic unit testing.

The other thing: if an insert throws an exception, the catch block is after the connection close statement.
Which means that the connection will be left open.
But, if the exception were being caught, we'd be seeing an error message, right?
Well, how sure are you about that? If you just put a "throw new Exception();" line before the insert, do you in fact get the expected error messge?
 

Offline Steve Walmsley (OP)

  • Aurora Designer
  • Star Marshal
  • S
  • Posts: 11675
  • Thanked: 20461 times
Re: Disappearing Ground Units
« Reply #43 on: April 27, 2020, 03:27:01 PM »
Here is some example code
...

First thing that stands out is that the connection itself is not wrapped in a using statement.
So, you are closing the connection, but not disposing it.
That can lead to funky problems that don't show up in basic unit testing.

The other thing: if an insert throws an exception, the catch block is after the connection close statement.
Which means that the connection will be left open.
But, if the exception were being caught, we'd be seeing an error message, right?
Well, how sure are you about that? If you just put a "throw new Exception();" line before the insert, do you in fact get the expected error messge?

I've never seen a save problem in my own games without an error message.

I have started to change some save functions to have a second try/catch for each record rather than the whole insert, so maybe I need to complete that for every save function.

So you suggest changing to:
using (SQLiteConnection AuroraDB = new SQLiteConnection(GlobalValues.AuroraConnectionString))
{
}
 

Offline Steve Walmsley (OP)

  • Aurora Designer
  • Star Marshal
  • S
  • Posts: 11675
  • Thanked: 20461 times
Re: Disappearing Ground Units
« Reply #44 on: April 27, 2020, 03:30:53 PM »
How about updating the code to...

        public void SaveSystems()
        {
            try
            {
                // establish a database connection
                using (SQLiteConnection AuroraDB = new SQLiteConnection(GlobalValues.AuroraConnectionString))
                {
                    AuroraDB.Open();

                    // delete the existing records
                    SQLiteCommand del = new SQLiteCommand("DELETE FROM FCT_System WHERE GameID = " + GameID, AuroraDB);
                    del.ExecuteNonQuery();

                    using (var cmd = new SQLiteCommand(AuroraDB))
                    {
                        using (SQLiteTransaction transaction = AuroraDB.BeginTransaction())
                        {
                            // insert new records
                            foreach (StarSystem obj in SystemList.Values)
                            {
                                try
                                {
                                    cmd.CommandText =
                                        @"INSERT INTO FCT_System (SystemID, SystemNumber, Age, AbundanceModifier, Stars, GameID, JumpPointSurveyPoints, SystemTypeID, DustDensity, SolSystem, NoSensorChecks )
                                VALUES ( @SystemID, @SystemNumber, @Age, @AbundanceModifier, @Stars, @GameID, @JumpPointSurveyPoints, @SystemTypeID, @DustDensity, @SolSystem, @NoSensorChecks)";

                                    cmd.Parameters.AddWithValue("@SystemID", obj.SystemID);
                                    cmd.Parameters.AddWithValue("@SystemNumber", obj.SystemNumber);
                                    cmd.Parameters.AddWithValue("@Age", obj.Age);
                                    cmd.Parameters.AddWithValue("@AbundanceModifier", obj.AbundanceModifier);
                                    cmd.Parameters.AddWithValue("@Stars", obj.Stars);
                                    cmd.Parameters.AddWithValue("@GameID", GameID);
                                    cmd.Parameters.AddWithValue("@JumpPointSurveyPoints", obj.JumpPointSurveyPoints);
                                    cmd.Parameters.AddWithValue("@SystemTypeID", obj.SystemTypeID);
                                    cmd.Parameters.AddWithValue("@DustDensity", obj.DustDensity);
                                    cmd.Parameters.AddWithValue("@SolSystem", obj.SolSystem);
                                    cmd.Parameters.AddWithValue("@NoSensorChecks", obj.NoSensorChecks);
                                    cmd.ExecuteNonQuery();
                                }
                                catch (Exception error) { GlobalValues.ErrorHandler(error, 3249); return; }
                            }

                            transaction.Commit();
                        }
                    }

                    AuroraDB.Close();
                }
            }
            catch (Exception error) { GlobalValues.ErrorHandler(error, 1430); return; }
        }