Aurora 4x

C# Aurora => C# Bug Reports => Topic started by: Steve Walmsley on May 02, 2020, 04:39:14 PM

Title: Potential vs Confirmed Bugs
Post by: Steve Walmsley on May 02, 2020, 04:39:14 PM
This is a point for discussion, rather than anything set in stone.

I'm reaching the point where a significant number of bug posts aren't bugs, but rather working as intended or decimal separator related or other known issues, plus reports on what could be real bugs but have very little information in the post. While I am working through these and providing feedback, it is taking time that could be spent on 'real' bugs or adding new functions, plus generating some frustration on my part :)

I am wondering whether to setup two separate bug threads; a 'Potential Bug' thread and a 'Confirmed Bug' thread. Confirmed bugs would be those that need me to focus on finding and fixing them, so I would only work through the 'Confirmed' thread.

The tricky part would be the mechanics of moving bugs from Potential to Confirmed. One option would be locking the Confirmed thread except for a set of 'Bug Moderators', which would be volunteers. Their role would be filtering out WAI, separators, asking for more detail/clarification, etc. and only passing on bugs that I need to work on. Another option is trusting people to only post in Confirmed after they posted in Potential and received confirmation from others that is was a bug. Option 2 is more flexible, but based on content of the current bug thread I can see a lot of people not reading the header post and simply posting everything in Confirmed.

I am open to other options for Potential vs Confirmed or other completely different options that solve the same problem.
Title: Re: Potential vs Confirmed Bugs
Post by: Caplin on May 02, 2020, 05:33:55 PM
For your sanity, I favor option 1. :)

As a QA person, I definitely recognize the problem with bug reports which don't provide sufficient information. Some kind of triage to ensure that bugs you work on have undergone a minimum of vetting sounds like a good idea.
Title: Re: Potential vs Confirmed Bugs
Post by: Lightning on May 02, 2020, 05:35:20 PM
I agree with option 1 as well
Title: Re: Potential vs Confirmed Bugs
Post by: Black on May 02, 2020, 05:46:25 PM
I myself got confused with mechanics that was changed from how it worked in earlier version of C# Aurora and reported it as a bug. I didn't notice that it changed and I am frequenting the forum a lot.

So in current situation I believe you should go for option 1 and select some people that can filter through bug reports and try to replicate reported bugs.
Title: Re: Potential vs Confirmed Bugs
Post by: Zincat on May 02, 2020, 05:54:18 PM
Absolutely option 1 Steve.
You cannot possibly expect people on the internet to be disciplined with bug reporting. While a large part will be, just as many will not and that can be disastrous for your time AND motivation.

Make two threads, appoint some volunteers, and never look into the potential bug reports thread ever again. Let the volunteers post verified bug reports into the confirmed bugs thread.

I say this as a person who has beta tested a lot for an open souce project. We used to do something like this in that project, and the developers' sanity was saved.
This way you can see and work only on confirmed and/or reproductible bugs, and adding new features. And of course, play the game for yourself! You said time and time again how much that's important to you. We don't want you to burn out you know  ;D

If someone reports a "bug" but cannot provide information, or guarantee that he's on the latest version etc, it's better if you don't see it. If it's a real bug someone else will report it in more detail sooner or later. If not, a volunteer can deal with the likely mistaken report.

EDIT: I would also forbid users to make new threads in the bug reporting sub-forum. Else some people will just make new threads for their bugs, which once again you don't need. Let those two threads created by you be the only ones used.
Title: Re: Potential vs Confirmed Bugs
Post by: Froggiest1982 on May 02, 2020, 06:13:38 PM
One option would be locking the Confirmed thread except for a set of 'Bug Moderators', which would be volunteers. Their role would be filtering out WAI, separators, asking for more detail/clarification, etc. and only passing on bugs that I need to work on.

+1

that could be spent on 'real' bugs or adding new functions

I would't spend time on more features until the known bugs are fixed, to be honest. A couple of them are really annoying.
Title: Re: Potential vs Confirmed Bugs
Post by: pwhk on May 02, 2020, 06:58:28 PM
Agreed to do option 1. As a software developer myself I know how frustrating non-bug reports are  ;)
Title: Re: Potential vs Confirmed Bugs
Post by: Omnivore on May 02, 2020, 07:20:12 PM
Number one please.  Otherwise current trend will probably worsen as Aurora is received by a wider audience.

Thanks!
Title: Re: Potential vs Confirmed Bugs
Post by: Iceranger on May 02, 2020, 07:37:09 PM
I personally think the first is the better approach to streamline the bug fixing process.
Title: Re: Potential vs Confirmed Bugs
Post by: Steve Walmsley on May 02, 2020, 07:46:15 PM
Thanks for the unanimous feedback. I will implement this for the next release onward to avoid confusion with the current release.

If you would like to be a bug-thread moderator please PM me. As a minimum you would need plenty of experience with the game and to have read the changes log all the way through :)  If you are an experienced tester that would be ideal. I appreciate the good intentions behind all bug reports, but if I am often asking for more information when you post reports, or I often reply that it isn't a bug, please wait until you have more experience before applying to be a moderator.

The role of the bug moderators will be to review posts in the 'potential thread', ask for more information if needed, seek confirmatory reports from other players if required and then make a post in the confirmed thread.

I will concentrate on working my way through the confirmed thread.

Erik, if you are watching, I'm not sure how to restrict posters to a thread so please let me know :)
Title: Re: Potential vs Confirmed Bugs
Post by: Erik L on May 02, 2020, 09:37:56 PM
I can create a Bug Moderator group and play around with the permissions.

I don't believe the permissions can be locked to a specific thread or not, but rather to a forum. If that is the case, I can create a subforum for confirmed bugs.
Title: Re: Potential vs Confirmed Bugs
Post by: Erik L on May 02, 2020, 09:51:51 PM
There is a new group. And a new board for Confirmed bugs. At this moment, it is only visible to me, Steve, moderators, and the as-yet-unpopulated Bug Moderator group. So bug reports would continue as normal, and the bug moderators will copy confirmed bugs into the confirmed bug forum, whether this is a single thread per version or one thread per bug. I'd suggest one per bug, then it can be locked and put into a changelog thread (by either Steve or a Bug Moderator).
Title: Re: Potential vs Confirmed Bugs
Post by: Yezarul on May 03, 2020, 12:09:51 AM
It is a good idea.  By having approved people move things from potential to confirmed, they can also conform it to a standard template that makes it easier for you to look through so things are faster/tidier.  This would not only improve the likelihood of fixing things but your own mental state as well.

I'm not sure how hard it would be to implement via this forum but there could also be a template that people type the information into that is already set-up, rather than relying on people to type it in their forum post.

I would suggest too that the approved moderators have access to delete from the potential bug threads, that way duplicates could be removed.  (IE Say I don't have the time to verify the bug myself in game but it has been posted multiple times.  I could then remove it and someone else that has the time could look into it).

Spreading out the workload of verifying or gathering additional data benefits everyone.

v/r
Yez
Title: Re: Potential vs Confirmed Bugs
Post by: Cosinus on May 03, 2020, 06:01:06 AM
May I suggest using a bug tracker? There are some open source versions out there and they have multiple advantages:
For example Dwarf Fortress uses the Mantis Bug tracker https://www.mantisbt.org/ (https://www.mantisbt.org/), which is amazing and has many features a simple forum thread does not have:
(see their bug tracker in action here: http://www.bay12games.com/dwarves/mantisbt/my_view_page.php (http://www.bay12games.com/dwarves/mantisbt/my_view_page.php))
This simplifies reporting and fixing bugs immensely:
Everybody wins.
Title: Re: Potential vs Confirmed Bugs
Post by: Froggiest1982 on May 03, 2020, 06:13:27 AM
I would suggest too that the approved moderators have access to delete from the potential bug threads, that way duplicates could be removed.  (IE Say I don't have the time to verify the bug myself in game but it has been posted multiple times.  I could then remove it and someone else that has the time could look into it).

I agree with this; actually, they could start from the reports from 1.0 at today as I am sure few bugs have not been sorted or may gone unnoticed. I am also sure Steve is keeping a list but while we are at it better do a full and complete report.
Title: Re: Potential vs Confirmed Bugs
Post by: Steve Walmsley on May 03, 2020, 06:59:21 AM
May I suggest using a bug tracker? There are some open source versions out there and they have multiple advantages:
For example Dwarf Fortress uses the Mantis Bug tracker https://www.mantisbt.org/ (https://www.mantisbt.org/), which is amazing and has many features a simple forum thread does not have:
  • Ability to search for bugs easily.
  • Ability to mark bugs as confirmed, solved, duplicate, WAI, etc.
  • Ability to easily categorise bugs into Typos, crashes, display bugs, gameplay bugs, unintuitive behaviour, missing features, etc.
  • They provide multiple fields that need to be filled when reporting the bug, such as version number, reproducible status, etc. This standardises bug reports.
  • Ability to have comments on a specific bug, without having to search for them across multiple pages.
(see their bug tracker in action here: http://www.bay12games.com/dwarves/mantisbt/my_view_page.php (http://www.bay12games.com/dwarves/mantisbt/my_view_page.php))
This simplifies reporting and fixing bugs immensely:
  • Bug reporters can search for existing duplicate bug reports without looking though dozens of pages.
  • Moderators can close WAI bug reports, duplicates, stuff that is caused by decimal separator, so steve doesn't have to deal with it.
  • Steve can concentrate on bugs that are flagged as confirmed.
  • Players can see persisting bugs easily and can work around them.
Everybody wins.

Thanks for the suggestion, but all of the above is assuming that players follow those rules. As many players don't follow the request at the start of the bugs thread, or read known issues or read the change log, then I cannot be confident they would use the above software as intended. I need the simplest solution that works. Bugs threads have worked fine for many years and the only issue is the amount of non-bug posts in recent days. If moderators can filter those, it would work fine.
Title: Re: Potential vs Confirmed Bugs
Post by: SpaceMarine on May 03, 2020, 07:07:21 AM
I have already sent my PM to you for the application of helping moderate the threads required, hopefully you got it as I cannot see it in my sent messages, Anyway I feel the system suggested previously with moderators filtering out the non bugs/not confirmed into one pile and the confirmed into others and being able to move those around accordingly, I think this will allow steve to focus on the more important bugs across the board.
Title: Re: Potential vs Confirmed Bugs
Post by: Steve Walmsley on May 03, 2020, 07:21:07 AM
I have already sent my PM to you for the application of helping moderate the threads required, hopefully you got it as I cannot see it in my sent messages, Anyway I feel the system suggested previously with moderators filtering out the non bugs/not confirmed into one pile and the confirmed into others and being able to move those around accordingly, I think this will allow steve to focus on the more important bugs across the board.

Yes, received thanks. I'll set this up later today.
Title: Re: Potential vs Confirmed Bugs
Post by: Migi on May 03, 2020, 07:25:45 AM
May I suggest using a bug tracker? There are some open source versions out there and they have multiple advantages:
For example Dwarf Fortress uses the Mantis Bug tracker https://www.mantisbt.org/ (https://www.mantisbt.org/), which is amazing and has many features a simple forum thread does not have:
  • Ability to search for bugs easily.
  • Ability to mark bugs as confirmed, solved, duplicate, WAI, etc.
  • Ability to easily categorise bugs into Typos, crashes, display bugs, gameplay bugs, unintuitive behaviour, missing features, etc.
  • They provide multiple fields that need to be filled when reporting the bug, such as version number, reproducible status, etc. This standardises bug reports.
  • Ability to have comments on a specific bug, without having to search for them across multiple pages.
(see their bug tracker in action here: http://www.bay12games.com/dwarves/mantisbt/my_view_page.php (http://www.bay12games.com/dwarves/mantisbt/my_view_page.php))
This simplifies reporting and fixing bugs immensely:
  • Bug reporters can search for existing duplicate bug reports without looking though dozens of pages.
  • Moderators can close WAI bug reports, duplicates, stuff that is caused by decimal separator, so steve doesn't have to deal with it.
  • Steve can concentrate on bugs that are flagged as confirmed.
  • Players can see persisting bugs easily and can work around them.
Everybody wins.

Thanks for the suggestion, but all of the above is assuming that players follow those rules. As many players don't follow the request at the start of the bugs thread, or read known issues or read the change log, then I cannot be confident they would use the above software as intended. I need the simplest solution that works. Bugs threads have worked fine for many years and the only issue is the amount of non-bug posts in recent days. If moderators can filter those, it would work fine.
I think the intention of the bug tracker would be for approved bug posters.

I do have a question, are bug thread moderators expected to reproduce the bugs?
Title: Re: Potential vs Confirmed Bugs
Post by: Cosinus on May 03, 2020, 07:40:32 AM
Thanks for the suggestion, but all of the above is assuming that players follow those rules. As many players don't follow the request at the start of the bugs thread, or read known issues or read the change log, then I cannot be confident they would use the above software as intended. I need the simplest solution that works. Bugs threads have worked fine for many years and the only issue is the amount of non-bug posts in recent days. If moderators can filter those, it would work fine.

Even if people do not follow the rules, the moderators can easily delete or close those reports and you can simply look at confirmed ones.
In a forum setting there are no required fields when submitting a bug report, so bug reports will be even less uniform.

I agree that it is more work to set it up, but this work only needs to be done once and can probably be assigned to the volunteer bug moderators. Dwarf Fortress worked with a bug report forum just like this one for years, but now I don't think they would ever go back to that. The advantages are a benefit for everyone.
Title: Re: Potential vs Confirmed Bugs
Post by: Steve Walmsley on May 03, 2020, 08:51:21 AM
I agree that it is more work to set it up, but this work only needs to be done once and can probably be assigned to the volunteer bug moderators. Dwarf Fortress worked with a bug report forum just like this one for years, but now I don't think they would ever go back to that. The advantages are a benefit for everyone.

At work, several teams in my department uses JIRA for analytics requests and we also use Confluence for IT projects, so I am familiar with the type of software involved. I avoid them personally because my experience has been that they create a bureaucratic wall between the stakeholders and the delivery department, but if the various teams believe they can make it work I am not going to stand in their way. I always deal direct with people by email or instant chat because it works better for me and that personal connection helps a lot I need their help. My philosophy is always the least amount of process necessary to get the job done.

I am hoping that the moderator concept will keep communication flowing in an easy way but enable me to spend my own time more effectively. I'm not trying to make Aurora players into professional testers - I just need a low process approach that can handle the fact they are not.
Title: Re: Potential vs Confirmed Bugs
Post by: Randy on May 03, 2020, 12:20:41 PM
Only suggestion to add is that the confirmed bugs thread(s) be readable by all, but only editable by the moderators. That way people can easily see what is a known bug - and either provide additional details, or at least avoid reposting the same thing without knowledge of the existing condition. (Well at least for those that do try to reponsibly report bugs...)
Title: Re: Potential vs Confirmed Bugs
Post by: Steve Walmsley on May 03, 2020, 12:29:01 PM
Only suggestion to add is that the confirmed bugs thread(s) be readable by all, but only editable by the moderators. That way people can easily see what is a known bug - and either provide additional details, or at least avoid reposting the same thing without knowledge of the existing condition. (Well at least for those that do try to reponsibly report bugs...)

Erik, is that possible?
Title: Re: Potential vs Confirmed Bugs
Post by: Yezarul on May 03, 2020, 12:34:12 PM
Quote from: Steve Walmsley link=topic=11215. msg130313#msg130313 date=1588513881

At work, several teams in my department uses JIRA for analytics requests

I know when I was doing QA work for 5th Planet Games back in the day that we operated on JIRA as well.  I only know the QA portion of it with submitting the JIRA request, I'm not sure how it works on the other end, but I'll trust your judgement on not operating it.
The good thing about doing it on the forums is that it is easy to setup and will be easy to see how beneficial it is.

I think what you are mainly looking for is people to just clean up the clutter of what is already a known bug and WAI and to save you the headache of going through it yourself yeah? As Migi said, having the mods test things is a question, as they can't help troubleshoot things in the DB from dev point of view.

The way I'm thinking is you just want the clutter removed so you can focus on what is important to you rather than having to respond to what is essentially constantly yeah?
I think just a clear structure will sort people out as well.  By having a format (that you already request) they can easily transcribe things after they look at them, but some things might not be apparent.  Perhaps a discord channel for the bug team would be beneficial so they can seek clarity from yourself/other mods that might be in the know as well.

v/r

Title: Re: Potential vs Confirmed Bugs
Post by: Resand on May 03, 2020, 02:26:55 PM
I can't be the first to have thought of this, but just to make sure.
Now that Steve isn't "spamming" :P the bug thread with "fixed" posts, we're going to need a more proper list of fixed bugs with upcoming releases.
If for no other reason then being able to test the result.
Title: Re: Potential vs Confirmed Bugs
Post by: Erik L on May 03, 2020, 03:25:56 PM
Only suggestion to add is that the confirmed bugs thread(s) be readable by all, but only editable by the moderators. That way people can easily see what is a known bug - and either provide additional details, or at least avoid reposting the same thing without knowledge of the existing condition. (Well at least for those that do try to reponsibly report bugs...)

Erik, is that possible?
I can make it globally visible, yes.
Title: Re: Potential vs Confirmed Bugs
Post by: Tikigod on May 03, 2020, 11:47:44 PM
One of the things I hope anyone considering offering to be a 'Bug Mod' takes into account is to put aside familiarity with VB when assuming how Aurora should work when deciding what is a confirmed bug or not.... though not really sure how this would actually be done properly without actually requiring open sanity checking discussion on the forum covering how C# should be behaving with input from Steve where needed to clear up uncertainty..... which ends up just taking up more time from Steve short term until we're all up to speed on the specifics of C# with VB assumptions put aside.

There is already a fair bit of assumption 'answers' going around to newcomers to Aurora who are picking up C# as their first ever version of Aurora, and there have been occasions of newcomers have been given incorrect or partially inaccurate information because information from over familiarity with VB is taking priority over actually checking out specific C# differences, would suck to see some of that carrying over to deciding if something is working as intended or not.
Title: Re: Potential vs Confirmed Bugs
Post by: thashepherd on May 04, 2020, 12:44:24 AM
FWIW I run a software team in my day job that works with Jira, and I'm happy to help out if that is selected - but I'd recommend something more lightweight. Maybe a shared Trello. As Steve sort of implied, the tool is irrelevant. Ideally something with an API is picked since we'd love to support automated bug reports via AuroraLoader.
Title: Re: Potential vs Confirmed Bugs
Post by: Zincat on May 04, 2020, 02:10:41 AM
I can make it globally visible, yes.

That would really help us players, so we can see what is beign fixed from one version to the next
Title: Re: Potential vs Confirmed Bugs
Post by: Bughunter on May 04, 2020, 04:54:07 AM
About WAI or not. In cases where this is not certain I think we should pass it on to Steve as long as the bug report meet other criteria. He is the ultimate and as I see it only authority on such things. If we can filter out the ones where he already answered in the past we still save a lot of his time here compared to before.
Title: Re: Potential vs Confirmed Bugs
Post by: SpikeTheHobbitMage on May 04, 2020, 07:34:55 AM
I agree that it is more work to set it up, but this work only needs to be done once and can probably be assigned to the volunteer bug moderators. Dwarf Fortress worked with a bug report forum just like this one for years, but now I don't think they would ever go back to that. The advantages are a benefit for everyone.

At work, several teams in my department uses JIRA for analytics requests and we also use Confluence for IT projects, so I am familiar with the type of software involved. I avoid them personally because my experience has been that they create a bureaucratic wall between the stakeholders and the delivery department, but if the various teams believe they can make it work I am not going to stand in their way. I always deal direct with people by email or instant chat because it works better for me and that personal connection helps a lot I need their help. My philosophy is always the least amount of process necessary to get the job done.

I am hoping that the moderator concept will keep communication flowing in an easy way but enable me to spend my own time more effectively. I'm not trying to make Aurora players into professional testers - I just need a low process approach that can handle the fact they are not.
With all due respect, Steve, and having suffered under such idiocy myself at work, the moderator system itself is that very 'bureaucratic wall', not whichever software tool happens to be in use.  The problems happen when the lines of communication are cut off entirely, which in my experience is invariably due to incompetence and petty turf wars among management, and frankly I haven't seen any of that from your team.  As long as there is open communication between yourself, your mods, and us players, so that systemic problems can be brought to light and dealt with, then the system will continue to work as intended.  As for what tool is used, whether the old bug thread, the new bug forum, or a more specialized system, that really comes down to workload, and time will tell.

Cheers.
Title: Re: Potential vs Confirmed Bugs
Post by: Migi on May 04, 2020, 02:55:23 PM
About WAI or not. In cases where this is not certain I think we should pass it on to Steve as long as the bug report meet other criteria. He is the ultimate and as I see it only authority on such things. If we can filter out the ones where he already answered in the past we still save a lot of his time here compared to before.
I think it would be worth feeding most or some of the results into the FAQ thread.
Title: Re: Potential vs Confirmed Bugs
Post by: Inglonias on May 05, 2020, 11:03:39 AM
I wanted to note that the confirmed bug thread is still invisible, and that's kind of annoying. I want to know what bugs have been fixed so far, basically.
Title: Re: Potential vs Confirmed Bugs
Post by: Steve Walmsley on May 05, 2020, 11:16:30 AM
I wanted to note that the confirmed bug thread is still invisible, and that's kind of annoying. I want to know what bugs have been fixed so far, basically.

I've created a changes post with the bug fixes so far.
Title: Re: Potential vs Confirmed Bugs
Post by: Inglonias on May 05, 2020, 02:03:00 PM
I wanted to note that the confirmed bug thread is still invisible, and that's kind of annoying. I want to know what bugs have been fixed so far, basically.

I've created a changes post with the bug fixes so far.

That works great, yeah. Thanks.
Title: Re: Potential vs Confirmed Bugs
Post by: Kyle on May 05, 2020, 02:30:22 PM
It would still be nice to be able to see the list of confirmed bugs, as a sort of more up-to-date list of known issues
Title: Re: Potential vs Confirmed Bugs
Post by: Yonder on May 05, 2020, 05:34:24 PM
Yeah, for example I took a break after a couple weeks, and every few days I've been scanning through the reported bugs to decide if it's time for me to start up a longer campaign. Now it's not really easy to do that, because the "potential" bug thread is even harder to read through because there is a lot more back and forth in it.
Title: Re: Potential vs Confirmed Bugs
Post by: Bughunter on May 07, 2020, 04:22:07 PM
Agreed, I stopped replying to everything now and marking them Confirmed instead.

After doing bug preprocessing on and off for just a few days now I'm even more impressed how Steve could do this and fixing them and play the game and all the other stuff he does.