Hi slonjh
Thanks for your response. I was not aware that the substeps was already being cached, since I could not find a way to detect this in the game, nor any outside way to tel this, so I assumed it was working the same way as the rest of the game. I will try to go over you different points.
1)
My "From what I can tell" is based on a combination different sources of information and some reasoning. The primary information source being the effect it appears to have on the game, especially concerning shut downs. From my experiance, it seems as if every actual action you take in the game is recorded in the database immidiately, since whenever the game shuts down to some random bug (like clicking the system name in the colony summery) every thing seems to be just as I left it, never missing a turn or some other modification. The second information source is small bits from different discussions of the game over different forums, where my conclussions seemed to be reinforced. Finally I have run some minor analysis of CPU use during some several second turns, where it only amounted to 11-12% on a 4 core CPU, and on closer analysis, it was one of the CPU's firing up at the end of the turn calculations, and not through the majority of the turn processing, which one would have expected if CPU was the major bottleneck. I could not get usable readings from my hard drive, due to not being able to get propor information of exact what type it is (I use an external hard drive for this, due to certain practicalities), and even if I did, without such knowledge they would be of limmited use. The next rational was a sort of possibility elimination, which goes as follows: Some of the problems with performance that people reported would not actually be that bad on the CPU, unless the algorithms and compiler/language was terribly slow. Here I am partially talking about the detection of other ships in a system. Here it was mentioned that when hundreds of ships had to detect each other, it would require the calculation of a large matrix, in other words it would be O(N*M) (where N and M are the ships from 2 factions). However, if we even assume that each faction has 1000 ships (way more than otherwise mentioned) it would only require 10^6 calculations, but almost every (mordern) CPU is capable of several 10^9 calculations per second per core, so this should at most take a few miliseconds, unless each calculation would take hundreds or thousand clock cycles, which I assumed to be unlikely. Even if they had to check on themselves too, it would only be O((N+M)^2) which would only be a factor 4 larger, and still nowhere near a problem range. Since the CPU could not account for the performance, even when the problem was increase significantly, there would have to be some other form of bottleneck. There simply were not that many other options that would make sense to give such problems, other data load problems. Considering that memory and caching just as often can be bottlenecks, even when a program only uses RAM, it seemed to be screaming at me that there would be a bottleneck if the data objects manipulated during turn processing had to be loaded and saved to a the database on a secondary hard drive.
Concerning the use of solid state drives (SSD), they would actually be expected perform relatively poorly on aurora, since solid state drives are much slower at overwriting data than it is reading (technically it takes a long time to delete data). Since aurora have to write the new state to the database, which for consistency writes it to the harddrive, it will be overwriting a lot of data during each update, and as such one would loose most of the gain from reducing the reading speed.
As I do not have access to the source code, and my destribution seems to be in a compiled version, it would be quite hard to use to examine break points. I would also rather not have to reverse engineer from the assembler code, if I tried to do it with that.
2)
You have every right to be sceptical about the 100 000 factor, since it in practice could be sevaral orders of magnitude smaller or larger (mostly smaller). This is partly due to the complexity of the general problem, and exactly how one should measure it. Since I was looking speedups using a database, I searched for some article that would describe this proporty. What I found was: hxxp: www. directionsmag. com/articles/ram-is-100-thousand-times-faster-than-disk-for-database-access/123964 (be aware that I cannot get hyperlinks to work in the preview). My main point here is actually more that the scale of potential (theoretical) speed up is on a massive scale, not just a factor 2-10, which one can otherwise often wave away far more easily, if the costs are substantual. The numbers you qouted are concerned with sequential reads, which is generally the optimal scenario, where certain overhead costs can be ignored. Even if we could use those numbers, it would still amount to some time, since we would have to read and write, say 10Mb, which would still take hundres of miliseconds. This is not exactly enough to cover multisecond long turn times, but it is fairly unlikely that we will actually get to do long sequential read and writes on the hardisk. The reason for this is 2 fold, first off our data is not large enough that we can ignore the seek overhead, which is in the order of miliseconds, secondly we would actually expect the reads to happen over several orders. This is due to each SQL request usually being processed atomically by the database, so they would have to be in seperate reads. Also, if he is using more complicated SQL queries, he would have to read from different tables, which would each constitute some seek time. This means, that every time there is a SQL query we can expect it to take several miliseconds to actually happen compared to the nanoseconds it would take to perform if the data was primarly located in the RAM.
It will agree upon it being very hard to get good performance numbers on caching schemes, and if there is some more delicate caching scheme running, then the numbers will end up very differently. The problem here, is that persisteny seems to be reinforced, which means that everything have to be put down on the harddrive and up from it again (otherwise we would not experiance everthing being the same after a crash). Since caches are voided when such a crash happens (the operation system will take over those resources, and they are as such dirtied), we are effectively working in a situation where caching is not really working for us, and it can therefor be ignored.
Your point about Amdahl's law was actually already considred, since I mentioned earlier, that other bottlenecks would probably rise, and I therefor guessed that the actual performance increase would problably be in the order of 100-10 000. Part of my point is actually that since the harddisk read/write times constitute such a huge proportion of the porformance (according my rational), then there would not be much point in improving performance in other areas, that was not related to this, since they would end up miniscules, since this bottleneck would be dominating the performance. This is precisely the point of Amdahl's law.
3)
I will admidt that since I have no access to the code base, nor any experiance with Visual Basic, I found it wiser to not give actual recommendations for exactly how one would move data to the RAM. There are several approaches to this, amoung the steps you mentione Steve have already taken, and some of them will have quite severe develpment costs. I neither have any idea how flexible the code base is with regard to this topic, and how costly it will be to rewrite it into different ways (since this depends on the former). Amoung the possible ways to go about it, would be things like more bulk like loading of the database, and then do the actual sorting in the RAM, as opposed to several SQL queries, another way would be to simply use the constructs he uses for substeps over longer periodes, and then only do back ups to the database more rarely, and finally one could consider using a database which is entirely loaded into the RAM, which would then only load and save to the disk less often. I suppose that the method that would require the least develpment cost is the later (a RAM based database), since it would mainly require converting the SQL queries to a slightly different database version.
Consistency would have to go down with this, because this is what we are paying such a huge price for in performance. While a database could still be used for long term consistency, one could reduce the amount the requirement of action to be included, to say only the last turn on a X day schedule or such would send data to the database for long term storage, or ofcause the propor termination of the program.
I will agree with you, that it is probably quite unrealistic to expect a full rewrite. It will however make it even more appealing to do, since there will also automatically come better performance by eliminating this kind of problem.
Also, thanks for the link the blog. I do not usually follow blogs, so I was not aware of it.
