It's possible it is related to not saving for a extended period of time. The three times I have had it happen, the game had been open and unsaved for many hours. I had a theory that maybe GU complexity was to blame, but I tested that and no go. I did notice once when it happened that the only units that disappeared were new units. All the ones created and organized during that play period were the ones to disappear.
Only thing I can think is that, at one point, I did hit the "Clear hierarchy" button when I had made a mess of the OOB but I never checked what the button does.
I haven't had this happen to me but I just saw something occur when I saved a game that had been running all night and thought it might be pertinent.
After I clicked save there was a point in time where I could close the game window yet the save hadn't completed yet. This in and of itself may not be a problem with the way SQLite works in C#, but I have to ask, are you using multiple transactions? If so, is the ground unit data being saved last?
If the answer to both of those questions is true, then the following might be relevant:
"In SQLite, only one transaction is allowed to have changes pending in the database at a time. Because of this, calls to BeginTransaction and the Execute methods on SqliteCommand may time out if another transaction takes too long to complete. (Microsoft C# docs)"
Hope this helps,
PS: If this reply is unwelcome, please accept my apologies.
I haven't had this happen to me but I just saw something occur when I saved a game that had been running all night and thought it might be pertinent.
After I clicked save there was a point in time where I could close the game window yet the save hadn't completed yet. This in and of itself may not be a problem with the way SQLite works in C#, but I have to ask, are you using multiple transactions? If so, is the ground unit data being saved last?
If the answer to both of those questions is true, then the following might be relevant:
"In SQLite, only one transaction is allowed to have changes pending in the database at a time. Because of this, calls to BeginTransaction and the Execute methods on SqliteCommand may time out if another transaction takes too long to complete. (Microsoft C# docs)"
Hope this helps,
PS: If this reply is unwelcome, please accept my apologies.
I am using one transaction for each of seventy-eight different table inserts. The ground formations aren't last but they are in the last ten so you might be on to something.
I open a connection and then close for each table as well. Am I better using a single open/close and a single transaction for all inserts?
I haven't had this happen to me but I just saw something occur when I saved a game that had been running all night and thought it might be pertinent.
After I clicked save there was a point in time where I could close the game window yet the save hadn't completed yet. This in and of itself may not be a problem with the way SQLite works in C#, but I have to ask, are you using multiple transactions? If so, is the ground unit data being saved last?
If the answer to both of those questions is true, then the following might be relevant:
"In SQLite, only one transaction is allowed to have changes pending in the database at a time. Because of this, calls to BeginTransaction and the Execute methods on SqliteCommand may time out if another transaction takes too long to complete. (Microsoft C# docs)"
Hope this helps,
PS: If this reply is unwelcome, please accept my apologies.
I am using one transaction for each of seventy-eight different table inserts. The ground formations aren't last but they are in the last ten so you might be on to something.
I open a connection and then close for each table as well. Am I better using a single open/close and a single transaction for all inserts?
I haven't had this happen to me but I just saw something occur when I saved a game that had been running all night and thought it might be pertinent.
After I clicked save there was a point in time where I could close the game window yet the save hadn't completed yet. This in and of itself may not be a problem with the way SQLite works in C#, but I have to ask, are you using multiple transactions? If so, is the ground unit data being saved last?
If the answer to both of those questions is true, then the following might be relevant:
"In SQLite, only one transaction is allowed to have changes pending in the database at a time. Because of this, calls to BeginTransaction and the Execute methods on SqliteCommand may time out if another transaction takes too long to complete. (Microsoft C# docs)"
Hope this helps,
PS: If this reply is unwelcome, please accept my apologies.
I am using one transaction for each of seventy-eight different table inserts. The ground formations aren't last but they are in the last ten so you might be on to something.
I open a connection and then close for each table as well. Am I better using a single open/close and a single transaction for all inserts?
If these 78 inserts are occurring serially (not in parallel), you aren't facing the problem described here, which occurs when two different transactions want to modify the database at the same time (one transaction has to wait on the other to finish, and it will only wait so long before giving up).
Using a single open/close for a single transaction comprising all 78 inserts would reduce connection overhead (possible significantly), and would make the 78 inserts succeed or fail as a unit (which may not be desired--do you want to rollback all of the changes whenever one of them fails?).
It is very odd, but the only consistent factor seems to be saves after leaving the game open a long time.
I haven't had this happen to me but I just saw something occur when I saved a game that had been running all night and thought it might be pertinent.
After I clicked save there was a point in time where I could close the game window yet the save hadn't completed yet. This in and of itself may not be a problem with the way SQLite works in C#, but I have to ask, are you using multiple transactions? If so, is the ground unit data being saved last?
If the answer to both of those questions is true, then the following might be relevant:
"In SQLite, only one transaction is allowed to have changes pending in the database at a time. Because of this, calls to BeginTransaction and the Execute methods on SqliteCommand may time out if another transaction takes too long to complete. (Microsoft C# docs)"
Hope this helps,
PS: If this reply is unwelcome, please accept my apologies.
I am using one transaction for each of seventy-eight different table inserts. The ground formations aren't last but they are in the last ten so you might be on to something.
I open a connection and then close for each table as well. Am I better using a single open/close and a single transaction for all inserts?
If these 78 inserts are occurring serially (not in parallel), you aren't facing the problem described here, which occurs when two different transactions want to modify the database at the same time (one transaction has to wait on the other to finish, and it will only wait so long before giving up).
Using a single open/close for a single transaction comprising all 78 inserts would reduce connection overhead (possible significantly), and would make the 78 inserts succeed or fail as a unit (which may not be desired--do you want to rollback all of the changes whenever one of them fails?).
Each insert is a separate function and the functions are called in sequence. Unless there is some asynchronous function in SQLite that continues to execute the transaction after I close the DB connection, they will be happening in sequence.
However, the lack of an error is concerning me. I have had database insert failure errors in the past, but never one without an error popup. Also, given there are so many inserts, why is only this one affected? Plus, formations will not be a particularly large table compared to some others. It is very odd, but the only consistent factor seems to be saves after leaving the game open a long time.
...the lack of an error is concerning me.
...the only consistent factor seems to be saves after leaving the game open a long time.
Which makes me think that in these cases you are sending an empty data set to insert.
Er. Just a shot in the dark but what about STO and P?Which makes me think that in these cases you are sending an empty data set to insert.
Or maybe you aren't sending the insert at all?
At a former place of employment, there was a tale of a legendary bug that for quite a long time had flummoxed a dev team using a dynamic SQL tool to build and execute very long SQL statements.
Very occasionally, the SQL statements would only partially complete.
Eventually they discovered that, for some obscure reason, a piece of string concatenation logic would, in rare circumstances, fail to insert a space between something aliased as "s" and the keyword "top". As a result, the word "stop" would be inserted instead, and the parsing engine, naturally enough, concluded that it should stop parsing this particular statement.
Please can everyone who has encountered this bug tell me how long their campaign had been running (in game years) when the bug happened.
In my game where the ships disappeared, I was about 55 years in and was playing pretty loose with ships all over the place. In the game where my ground units disappeared, I was about 21 years in BUT I was also taking a lot of time to build up a massive ground force and had about 4 divisions of troops with about 12-15 subordinate battalions (my smallest unit) attached. I had JUST created my first set of STO units.
I don't know anything about your data structures, but sounds like most logical explanation would be the save loop sometimes don't go over all objects rather than a problem with the inserts. Maybe a problem going through a hierarchical structure like command chain? An early return somewhere? A loop variable not going all the way like i++ not reaching last item in list. Objects moved to some separate/temporary list not included in the save operation.
Apart from system bodies, the game deletes the existing table and inserts the data from the program.
I was 78 years into the campaign, and it did happen after not saving for about 3 hours and leaving the game running without playing for one hour and a half
Edit: also idk if it's the same for everyone but it happened on two different places, in my second colony and in a planet in a distant system where I was excavating some alien installations, only thing I can think of that they have in common is that they were the latest ground units I had trained, about 2-5 years before the bug happened
Quote from: Bughunter link=topic=11094. msg128631#msg128631 date=1588015930I don't know anything about your data structures, but sounds like most logical explanation would be the save loop sometimes don't go over all objects rather than a problem with the inserts. Maybe a problem going through a hierarchical structure like command chain? An early return somewhere? A loop variable not going all the way like i++ not reaching last item in list. Objects moved to some separate/temporary list not included in the save operation.
I'm not using loops that way for saving. I use LINQ to retrieve a collection of objects then cycle through using foreach. If that was a problem, it would always be a problem.
BTW do ALL ground units vanish or only some? If only some, are they different in some ways to the survivors?
Apart from system bodies, the game deletes the existing table and inserts the data from the program.
Does the delete for each table happen immediately prior to the insert, or do you do all the deletes at once and then all of the inserts?
I know you are opening and closing a connection for each insert. Are you reusing the same connection, or creating a new connection each time?
What would happen if you tried to insert a duplicate primary key? Would that possibly result in a silent failure that either ignores the duplicate row or overwrites the original with the duplicate?
What Kristover describes points towards an endless loop somewhere in the save function, that is technically possible to achieve with LINQ expressions, example: https://stackoverflow.com/questions/31993443/why-does-this-method-result-in-an-infinite-loop (https://stackoverflow.com/questions/31993443/why-does-this-method-result-in-an-infinite-loop)
No, with the above code I cannot see how that could happen. Just assumed you might be using some more complex LINQ selection somewhere to pull data into your save function.
Here is some example code
...
Here is some example code
...
First thing that stands out is that the connection itself is not wrapped in a using statement.
So, you are closing the connection, but not disposing it.
That can lead to funky problems that don't show up in basic unit testing.
The other thing: if an insert throws an exception, the catch block is after the connection close statement.
Which means that the connection will be left open.
But, if the exception were being caught, we'd be seeing an error message, right?
Well, how sure are you about that? If you just put a "throw new Exception();" line before the insert, do you in fact get the expected error messge?
I've never seen a save problem in my own games without an error message.
I have started to change some save functions to have a second try/catch for each record rather than the whole insert, so maybe I need to complete that for every save function.
How about updating the code to...I'd move the AuroraDB.Close(); to the finally section of the try/catch
public void SaveSystems()
{
try
{
// establish a database connection
using (SQLiteConnection AuroraDB = new SQLiteConnection(GlobalValues.AuroraConnectionString))
{
AuroraDB.Open();
// delete the existing records
SQLiteCommand del = new SQLiteCommand("DELETE FROM FCT_System WHERE GameID = " + GameID, AuroraDB);
del.ExecuteNonQuery();
using (var cmd = new SQLiteCommand(AuroraDB))
{
using (SQLiteTransaction transaction = AuroraDB.BeginTransaction())
{
// insert new records
foreach (StarSystem obj in SystemList.Values)
{
try
{
cmd.CommandText =
@"INSERT INTO FCT_System (SystemID, SystemNumber, Age, AbundanceModifier, Stars, GameID, JumpPointSurveyPoints, SystemTypeID, DustDensity, SolSystem, NoSensorChecks )
VALUES ( @SystemID, @SystemNumber, @Age, @AbundanceModifier, @Stars, @GameID, @JumpPointSurveyPoints, @SystemTypeID, @DustDensity, @SolSystem, @NoSensorChecks)";
cmd.Parameters.AddWithValue("@SystemID", obj.SystemID);
cmd.Parameters.AddWithValue("@SystemNumber", obj.SystemNumber);
cmd.Parameters.AddWithValue("@Age", obj.Age);
cmd.Parameters.AddWithValue("@AbundanceModifier", obj.AbundanceModifier);
cmd.Parameters.AddWithValue("@Stars", obj.Stars);
cmd.Parameters.AddWithValue("@GameID", GameID);
cmd.Parameters.AddWithValue("@JumpPointSurveyPoints", obj.JumpPointSurveyPoints);
cmd.Parameters.AddWithValue("@SystemTypeID", obj.SystemTypeID);
cmd.Parameters.AddWithValue("@DustDensity", obj.DustDensity);
cmd.Parameters.AddWithValue("@SolSystem", obj.SolSystem);
cmd.Parameters.AddWithValue("@NoSensorChecks", obj.NoSensorChecks);
cmd.ExecuteNonQuery();
}
catch (Exception error) { GlobalValues.ErrorHandler(error, 3249); return; }
}
transaction.Commit();
}
}
AuroraDB.Close();
}
}
catch (Exception error) { GlobalValues.ErrorHandler(error, 1430); return; }
}
How about updating the code
...
I'd move the AuroraDB.Close(); to the finally section of the try/catch
How about updating the code
...
That looks exactly right.
I'd move the AuroraDB.Close(); to the finally section of the try/catchShouldn't the 'using' clause take care of that? Edit: ninjas everywhere.
One thing I'd like to point out (and I'm sorry if I'm getting annoying) is that the delete operation is outside of the transaction. This means that if the insert is interrupted and gets rolled back, the delete still gets committed.I'd move the AuroraDB.Close(); to the finally section of the try/catchShouldn't the 'using' clause take care of that? Edit: ninjas everywhere.
One thing I'd like to point out (and I'm sorry if I'm getting annoying) is that the delete operation is outside of the transaction. This means that if the insert is interrupted and gets rolled back, the delete still gets committed.I'd move the AuroraDB.Close(); to the finally section of the try/catchShouldn't the 'using' clause take care of that?
I'd move the AuroraDB.Close(); to the finally section of the try/catch
If I prevent the delete then I could end up with a table that is out of sync with all the others, which might cause harder to find bugs. With the try/catch on the record level, would I not be better deleting and then only losing any individual records that fail?
...
As other people have reported, I too was AFK for a couple of hours inbetween the two saves, though the game was not advancing time.
...
Quote from: Earthrise link=topic=11094. msg128677#msg128677 date=1588020626. . .
As other people have reported, I too was AFK for a couple of hours inbetween the two saves, though the game was not advancing time.
. . .
I'm curious: did any game-time pass between the two saves?
Well, add this to the pile of clues: my fuel keeps disappearing from Luna.
I moved all my shipyards to Luna, as well as a refuelling station.
Then I used a tanker to transport several million units of fuel to Luna.
Sometime later, I built a ship and was told it had no fuel.
I looked at the Luna colony summary. Zero fuel available.
This has happened several times now. I can't figure out what precedes it, though I do often go a long time between saves, with long stretches of idle time.
I'm in year 12.
I haven't noticed anything else missing.
if you took out the return from the inner catch block, then it would have the advantage that if it fails to insert any single record, then only that one record is lost. If you logged the information about the failing record(s), you could track down the bug more easily.
Well, add this to the pile of clues: my fuel keeps disappearing from Luna.
I moved all my shipyards to Luna, as well as a refuelling station.
Then I used a tanker to transport several million units of fuel to Luna.
Sometime later, I built a ship and was told it had no fuel.
I looked at the Luna colony summary. Zero fuel available.
This has happened several times now. I can't figure out what precedes it, though I do often go a long time between saves, with long stretches of idle time.
I'm in year 12.
I haven't noticed anything else missing.
I saw a fuel bug on the bug posts related to missing MSP to replenish ships that causes the lost of fuel. Might be that? Steve fixed it already for 1.9 release
Then I shall endeavor to give good advice. :)One thing I'd like to point out (and I'm sorry if I'm getting annoying) is that the delete operation is outside of the transaction. This means that if the insert is interrupted and gets rolled back, the delete still gets committed.Good advice is never annoying :)I'd move the AuroraDB.Close(); to the finally section of the try/catchShouldn't the 'using' clause take care of that?
Now I am torn. If I prevent the delete then I could end up with a table that is out of sync with all the others, which might cause harder to find bugs. With the try/catch on the record level, would I not be better deleting and then only losing any individual records that fail?
Hi Steve, I am most definitely not a C# or Sqlite or linq expert, but what is the intention of opening and closing a connection every time? In other languages/DBs it is common to use connection pooling. Second, is your intention that each of those 78 inserts are independent from each other? If one fails should they all fail or not? I would group those 78 into groups that must all succeed or fail together. Otherwise you can leave a DB in an inconsistent state. In a regular DB setup, that would be much faster as well. I assume you are doing an in-memory DB or at least only your aurora process is updating it. If that is the case, using connection pooling and optimizing your transaction might not have much performance impact. I'd profile it.Quote from: Omnivore link=topic=11094. msg128536#msg128536 date=1587999296I haven't had this happen to me but I just saw something occur when I saved a game that had been running all night and thought it might be pertinent.
After I clicked save there was a point in time where I could close the game window yet the save hadn't completed yet. This in and of itself may not be a problem with the way SQLite works in C#, but I have to ask, are you using multiple transactions? If so, is the ground unit data being saved last?
If the answer to both of those questions is true, then the following might be relevant:
"In SQLite, only one transaction is allowed to have changes pending in the database at a time. Because of this, calls to BeginTransaction and the Execute methods on SqliteCommand may time out if another transaction takes too long to complete. (Microsoft C# docs)"
Hope this helps,
PS: If this reply is unwelcome, please accept my apologies.
I am using one transaction for each of seventy-eight different table inserts. The ground formations aren't last but they are in the last ten so you might be on to something.
I open a connection and then close for each table as well. Am I better using a single open/close and a single transaction for all inserts?
You will need to fully suppress any row-level exceptions unless you intend to abort the whole save, because the using clause on the transaction will rollback if it sees one. Returning from inside the loop has the same effect. Maybe log those errors to a text file instead? Remember to open in append mode.
Another issue is that if a non-recoverable error does occur or Aurora crashes during the save, only the current transaction gets rolled back. That is guaranteed to leave the database in an inconsistent state with the current setup. The solution to that problem is to start the transaction in the function that calls the save functions. Just pass the connection in as an argument so they can all share it. This has the added bonus of speeding up saves by a factor of about 7x.
Hi Steve, I am most definitely not a C# or Sqlite or linq expert, but what is the intention of opening and closing a connection every time? In other languages/DBs it is common to use connection pooling. Second, is your intention that each of those 78 inserts are independent from each other? If one fails should they all fail or not? I would group those 78 into groups that must all succeed or fail together. Otherwise you can leave a DB in an inconsistent state. In a regular DB setup, that would be much faster as well. I assume you are doing an in-memory DB or at least only your aurora process is updating it. If that is the case, using connection pooling and optimizing your transaction might not have much performance impact. I'd profile it.
Let me give a simple example, if you have a function that saves fleets and another that save ships and there are links between the 2, i. e. , the ships have a fleet id and the fleet have ship ids and you have separate transactions, then if saved the fleet successfully but the ships save fails, it can cause problems. In that case, you could end up with your fleets pointing to ships that don't exist or vice versa. If they were all in the same transaction, you avoid that problem and they are all successful or fail as a group.
Exactly correct. A transaction should always leave the database in a consistent state. The other more complicated option is to try to recover failed rows, but again with the same goal. Edit: One thing to check here is that C# isn't issuing a rollback if an insert fails. If it does, the solution is to wrap each insert in a savepoint, but I'm not quite certain how to do that in C#. I'll need to get back to you if that is case.You will need to fully suppress any row-level exceptions unless you intend to abort the whole save, because the using clause on the transaction will rollback if it sees one. Returning from inside the loop has the same effect. Maybe log those errors to a text file instead? Remember to open in append mode.
Another issue is that if a non-recoverable error does occur or Aurora crashes during the save, only the current transaction gets rolled back. That is guaranteed to leave the database in an inconsistent state with the current setup. The solution to that problem is to start the transaction in the function that calls the save functions. Just pass the connection in as an argument so they can all share it. This has the added bonus of speeding up saves by a factor of about 7x.
OK, I need to clarify my understanding here :)
Let's assume that as suggested the transaction is setup in the function that calls all the separate save function. BeginTransaction needs the DB connection, so that would also need to be in the 'master save' function. The various SQLiteCommands for the deletes and the insert also need the DB connection but that could be passed to each function.
So it would look like
Using Connection
{
Using Transaction
{
SaveFunction #1 (Connection)
{
Delete Table
SQLiteCommand
{
Insert Rows & log errors
}
}
SaveFunction #2 (Connection)
{
Delete Table
SQLiteCommand
{
Insert Rows & log errors
}
}
}
}
My understanding from what you described is that in this scenario, either the whole save is successful or the save doesn't happen at all. Any error for any record in any table prevent the save taking place, although I would retain the error log of the failed save. Is that correct?
One other thing to check: Does that popup in the middle of the insert loop actually work, or does it hang? If it hangs, that could explain the problem people reported of saving, leaving, and coming back to find Aurora frozen with a broken database.
Fair enough, though I'm still concerned about whatever is causing that hang. Would Aurora be able to recover if the bad rows were stripped out without wiping the whole table?One other thing to check: Does that popup in the middle of the insert loop actually work, or does it hang? If it hangs, that could explain the problem people reported of saving, leaving, and coming back to find Aurora frozen with a broken database.
Based on the current code, not the proposed code, if there is an error in an object such as a null value, an error pops up and the transaction is aborted. Because the transaction is at the table level, the data in the target table is deleted and no records are inserted. The program carries on to the next table.
I've always had that happen for every insert error over several years of dev work.
The one thing that seems clear about this issue is that you need more information to track down the root cause. Would you consider adding error logging to the next release so that you would have a concise and accurate report on what occurred around the time of the error?So much this. If nothing else, logging the bad rows would at least tell you why they were rejected.
Part of the problem in a bug of this nature is human perception and rather coarse capabilities, a key clue might easily go unnoticed or be forgotten, the granularity of the snapshots you are able to look at (before and after databases) is evidently insufficient in this case and could possibly be likewise lacking in tracking later issues.
I'm retired so not very active but I spent some time talking to a few friends who are still on the job and there was a unanimous consensus on the need for logging in cases like these.
Hope this helps
public static void WriteLog(string message, LogFile logType, LogLevel level, string moduleName = "", string gameName = "", string raceName = "")
{
string logFile;
string header;
DateTimeFormatInfo dtfi = new CultureInfo(currCultureInfo.ToString(), false).DateTimeFormat;
dtfi.LongTimePattern = "HH:mm:ss";
header = "[" + DateTime.Today.ToLongDateString() + " " + DateTime.Now.ToLongTimeString() + "] - ";
switch (level)
{
case LogLevel.errorLevel:
header = header + "[ERROR]: " + moduleName;
break;
case LogLevel.debugLevel:
header = header + "[DEBUG]: " + moduleName;
break;
default:
header = header + "[INFO]: " + moduleName;
break;
}
message = header + message + "\r\n";
switch (logType)
{
case LogFile.errorLog:
logFile = LogPath + @"\Error.log";
break;
case LogFile.debugLog:
logFile = LogPath + @"\Debug" + DateTime.Today.ToString("yyyymmdd") + ".log";
break;
case LogFile.gameLog:
logFile = LogPath + @"Game-" + gameName + ".log";
break;
case LogFile.raceLog:
logFile = LogPath + @"Race-" + raceName + ".log";
break;
default:
logFile = LogPath + @"\General.log";
break;
}
if (!File.Exists(logFile))
{
using (StreamWriter file = new StreamWriter(logFile))
{
file.Write(message);
}
}
else
{
using (StreamWriter file = new StreamWriter(logFile, true))
{
file.Write(message);
}
}
}
Then I shall endeavor to give good advice. :)
You will need to fully suppress any row-level exceptions unless you intend to abort the whole save, because the using clause on the transaction will rollback if it sees one. Returning from inside the loop has the same effect.Maybe log those errors to a text file instead? Remember to open in append mode.Edit: Actually, just remove the return after showing the popup so the save can continue.
Another issue is that if a non-recoverable error does occur or Aurora crashes during the save, only the current table gets rolled back. That is guaranteed to leave the database in an inconsistent state with the current setup. The solution to that problem is to start the transaction in the function that calls the save functions. Just pass the connection in as an argument so they can all share it. This has the added bonus of speeding up saves by a factor of about 7x.
Quote from: SpikeTheHobbitMage link=topic=11094. msg128701#msg128701 date=1588026706
Then I shall endeavor to give good advice. :)
You will need to fully suppress any row-level exceptions unless you intend to abort the whole save, because the using clause on the transaction will rollback if it sees one. Returning from inside the loop has the same effect.Maybe log those errors to a text file instead? Remember to open in append mode.Edit: Actually, just remove the return after showing the popup so the save can continue.
Another issue is that if a non-recoverable error does occur or Aurora crashes during the save, only the current table gets rolled back. That is guaranteed to leave the database in an inconsistent state with the current setup. The solution to that problem is to start the transaction in the function that calls the save functions. Just pass the connection in as an argument so they can all share it. This has the added bonus of speeding up saves by a factor of about 7x.
Instead of passing the connection already opened as a parameter in each function, Steve can put it in a Singleton class. It might be less work refactoring.
Instead of passing the connection already opened as a parameter in each function, Steve can put it in a Singleton class. It might be less work refactoring.That seems to me to be overkill for something that could just as easily be tied to the game-state or some other long lived object if he wants to keep the connection open across saves.
Once I got past my instinctive NO GLOBALS ARE EVIL reaction... ok I'm still not over it, but if it was deemed absolutely necessary you'd have to make it compatible with using statements and make sure it threw exceptions if accessed outside of such a scope, otherwise asking for trouble.Globals are evil and also unnecessary. There is also no need to use a 'using' statement with a persistent connection as it will only be opened once and is crash-safe. The transactions and prepared statements are the parts that need reliable cleanup.
As a non-programmer, I have only one probably unhelpful suggestion:Based on his sample code, Steve has learned the Sacred Art of the Prepared Statement. Input sanitization is assured.
Did you sanitize your database inputs?
Is this caused by people mis-naming their drop pod troops as drop table troops?
(relevant xkcd (https://xkcd.com/327/))
Is there a place where the ground force data structure is copied and the data structure used to display diverges from the one that is used to save?There's a bunch of places where the UI will not refresh unless the window is closed and reopened again, or the refresh button is clicked, this may explain the above.
I have not triggered this bug jet, but i have followed this and the other bug reports.
Reading through the last paged i came to the conclusion that the function that saves
the ground forces is not the root of the problem.
Steve uses a transaction around the inserts. So all or none of the ground forces should be in the database.
Most reports say that some but not all ground forces are missing and there are no error messages shown.
This leads to my conclusions
- that there is no error inserting the ground forces in the database
- that the missing ground forces where already missing in the data structure
So how do the ground forces go missing? What do we know?
- in most cases it was a group of ground forces: e.g. all STOs, all recent created troops, all initial created troops
- the action/actions didn't remove the troops from the display immediately, because nobody clearly remembers what they have done with the troops before
- the action/actions are not that common or we would have more reports
Is there a place where the ground force data structure is copied and the data structure used to display diverges from the one that is used to save?
Don't want to create a side discussion; but seeing the potential problems with saving, I was wondering if using your own save format might solve the issues. Using a DB now derives from using it in the past because easier to handle when not all data is in RAM; but now it is... . So might be worth while to think about this.
I know. Did that myself as I had written you with the 1.7. to 1.8 DB - worked fine by the way. No errors since :DDon't want to create a side discussion; but seeing the potential problems with saving, I was wondering if using your own save format might solve the issues. Using a DB now derives from using it in the past because easier to handle when not all data is in RAM; but now it is... . So might be worth while to think about this.
The database is also really useful for viewing and modifying data outside the program.
BTW can I just clarify if the units disappear after save, or after the game is reloaded?
Maybe some form of save event log for ground troops could be introduced? Would contain the troops that were saved so that people who are trying to reproduce this could look at them and see which ones went missing? Could be hidden by defaultOne of the things that has been suggested is a more verbose error log, possibly making it general across all tables.
Not sure if worth the trouble though as I guess people could copy the Total force text and compare it to after save reload and even against previous save
Or it may be something else completely different, I'm going to force myself to stop posting on this issue because I'm down to pure guesswork past the above.I'll second this. We've helped as much as we reasonably can with the information available. The ball is in Steve's court now. All we can do now from our end is to wait and see. Cheers.
For this type of problem I personally use RR to debug (rr-project.org). It records the entire execution of the program (with minimal overhead), and lets you replay the program backwards and forwards. If you can capture the bug in a recording even once, it becomes very easy to track down where the data was lost or corrupted. You can literally set a watchpoint on a variable or memory location that has the wrong value, then run the program in reverse (as weird as that sounds) until the watchpoint indicates that the value was changed. The program will stop right where this happens, so you can see what was going on. It's basically a superpower.
The downside in this case is that RR only works in Linux, and only on Intel CPUs (it uses the performance counters that Intel added a few years back to do the tracing without adding insane overhead).
However, Microsoft has announced that they're working on this same kind of feature for MSVC, and it's already becoming available for some use cases. I think you have to pay extra for it, and you have to run your program in Azure (Microsoft's computer rental service), and probably jump through some other hoops, but I really recommend looking into it. It makes these kinds of bugs so easy to track down that it's definitely worth it.
You can do similar things already in C# with several different Microsoft tools: Intellitrace, ETLs, WinDBG/SoS. Though It's hard to do this type of postmortem profiling/investigation unless you have repro steps.
In case it is still of use and not already solved in 1. 9. While rummaging around a bit in my aurora manual backup folder, recycle and backup bins in an unsuccessful search for a buried replica of the 1. 8 Antarctic db described in my earlier post, I found that the new, less setup-intensive game I started after that, shows the same issue caught at exactly the right moment, and this time, without any obvious Microsoft in the head disturbance to Aurora saving or loading. In the db, ground unit formations seem to be there: names, IDs, parent formations, locations and all; and the formation elements, too, as far as I can tell; but from this db, none of them load into Aurora, and with any subsequent save (including load/do nothing/save/quit), the units vanish completely from the records. – At least the player units, I don't know about NPRs.
One thing I notice in the db is that some NPR formations seem to have both PopulationID and ShipID set to 0. I don't know what that means and if that's intended, but it feels like an error. Maybe some bugged NPR unit transport? Or a save issue after all? Or it may be a different disappearing units bug from the main one as someone suggested.
...
Except, I didn't type 'continue' in the foreach, I typed 'return'. Oops!
...
Hey GuysFirst of all, you should register your account so that you can post normally. Until you register, a moderator has to manually approve each post you make.
I just had all my ground unit dissapear in my latest game. It seems to be version 1. 51 (at least this is what the rar file was called, the install thread still describes it as version 1. 31)
I see that this bugs is supposed to be fixed in version 1. 9. 1 almost a year ago.
So my question: is the download link in this forum not updated anymore? If so: where can one get the latest version?
Hmmm... I don't know if it's a related issue or a desired game mechanic: a small troop transport with troops in refit (I forgot to unload them...); when the refit is complete the troops are gone...
v.1.13, no mods.
J.
Hmmm... I don't know if it's a related issue or a desired game mechanic: a small troop transport with troops in refit (I forgot to unload them...); when the refit is complete the troops are gone...
v.1.13, no mods.
J.
You should repost this in the bug thread