QA:Forgejo evaluation

QA evaluation summary (2024-11-27)

The Fedora Quality team has evaluated Forgejo as a potential distgit+Bugzilla replacement. In the limited time we had, we found the majority of QA use cases to be either working or likely working (meaning even though the Forgejo configuration and deployment details are not completely finalized at the moment, we see a path forward for satisfying our needs). However, we also identified several concerning shortcomings. None of them seem to be a complete blocker, at least from our QA perspective, but they are highly-visible regressions and other parties might be affected by them as well, so we want to highlight them here.

Major concerns:

1. It’s not possible to create private issues (or individual private comments in issues) in Forgejo.
   • Currently QA uses Bugzilla’s ability to both create private issues and private comments to hide info. Bugzilla can switch a ticket/comment to private even after it has been created (and this is also used regularly).
   • Forgejo can only create a private repo (all issues private), but can’t create a private issue in a public repo.
   • This affects use cases like reporting CVEs and security issues, reporting bugs with sensitive information (manually or through ABRT, which uses private issues heavily), manually hiding sensitive info after posting, private e.g. Council discussions, etc.
2. It’s not possible to search in all issues in distgit combined, only in individual repos one by one.
   • In Forgejo, you have to open the particular repo (meaning a package repo, in our distgit case) to search in its issues. There’s no “global” search. (Logged in users can do a combined search in their own repos and in organizations that they’re a member of, but that’s not an actual global search, like in Bugzilla).
   • This means that people can’t search if a problem was already reported, unless they know the right component from the beginning (and assuming that the ticket was really reported under the correct component). This will lead to lower visibility of important issues and increased bug reports duplication.
3. It’s not possible to move issues between repos (i.e. distgit packages).
   • In Bugzilla, it’s very common to switch a bug between different components (packages), even several times, as we narrow down the source of the problem. The bug report is independent of the package itself.
   • In Forgejo, all issues exist in a repo of a certain package. If the issue should be moved to a different package, Forgejo can only create a completely new ticket elsewhere, and copy over a single comment from the old issue, and reference the old issue. If this is done several times, it creates a sequence of interlinked issues, which are much harder to follow and read the problem description as a whole. Furthermore, collaborators in the old issue must manually subscribe to the new issue, otherwise they no longer receive any new comments. (Gitlab, for comparison, can clone the whole ticket including all contributors, into the new repo).
   • This new approach will likely make debugging complex issues much harder, because it will be harder to follow the trail of interlinked issues and it will be easy for involved people to unintentionally stop receiving notifications for future comments.

Medium concerns:

1. Figuring out how to report a bug is harder in Forgejo compared to Bugzilla.
   • Bugzilla’s home page is quite simple with big buttons which guide the users to file a new bug (including a wizard, and a duplicate bug search detection) or search in existing bugs.
   • In Forgejo, the home page isn’t useful for bug reporting at all. For logged in users, it shows their repos and their activity, and for the general public, it doesn’t show anything useful.
   • We will need to create a reasonable way to guide community contributors to reporting a new bug, right from the home page. Otherwise we might lose a lot of bug reports that we previously used to receive.

Please note that the issues above are likely not unsolvable. There might be workarounds (but not likely simple and obvious ones), which might have different pros and cons, requiring further investigation, or we can decide to create custom patches for Forgejo. The focus of this summary was to document the current state as is.

If you’re interested in details, or other minor issues that we identified, you can find them in:
https://fedoraproject.org/wiki/QA:Forgejo_evaluation

Meeting 2024-11-14

Attendees: Akashdeep Dhar, Frantisek Zatloukal, Josef Skladanka, Kamil Paral, Sumantro Mukherjee, Tomas Hrcka, Lukas Ruzicka, Lukas Brabec

Notes

• Closing bugs on EOL branches
  • Probably a good use-case for an internal worker that gets triggered at a certain time or manually
• “Mass edits”
  • Post a comment to “all bugs that are about to be auto-closed”
  • Not run by FQA (Aoife Moloney takes care of this atm?)
• Big/batched queries
  • The API should be able to handle whatever we throw at it, whenever something specific gets slow, we can optimize/throw resources at the problem
  • Note: API should be able to limit the field-set for queries
• Renaming UI elements
  • UI elements could be renamed only in the scope of an organization (e.g. renaming “Projects” to “RPMS” within the /rpms org UI)
• Moving data (pre)processing into actions API
  • Some of the QA tools query “all” the data from BZ to later filter/group them. This could be done by an action/worker that will (periodically/on demand?) publish the collated data in machine parse-able format (or as a static web page, …)
• Looks like we can’t query issues according to a single branch (in UI)
  • Refers to a proposal for using git-branch to reference a fedora release inside a ticket (e.g. this).
  • Both in a particular repo (rpms/grub2), and also globally in all organization repos.
  • We found no UI nor API to query according to a branch.
  • Might be easier/better to solve using a standard tag/label
• Private issues/comments (requested in Forgejo):
  • Currently QA uses Bugzilla’s ability to both create private issues and private comments to hide info. Bugzilla can switch a ticket/comment to private even after the fact (and this is also used regularly).
  • Forgejo can only create a private repo (all issues private), but can’t create a private issue in a public repo.
  • This affects use cases like reporting CVEs and security issues, reporting bugs with sensitive information (manually or through ABRT, which uses private issues heavily), manually hiding sensitive info after posting, private e.g. Council discussions, etc.
• Moving tickets to other components (repos) (requested in Forgejo):
  • Moving tickets between components (now repos in Forgejo) is a common workflow for us
  • Ticket references copy just a single comment, instead of the whole ticket (gitlab does this)
  • Could be solved by a “copy bot”/worker/action that creates a new issue in the target tracker, meaningfully copies the data (comments, references, tags, …), and closes the “old” issue referencing the new one.
    • Ideally, the worker/bot would be able to post the comments in the newly created ticket as the original poster, not as “copy bot”. Not necessary for MVP
• Close issue as duplicate - probably is in the API, would be a useful addition to the UI
  • Could be worked around by referencing the “prime” bug, and then closing, but seems clumsy for daily use
• There’s no needinfo as in Bugzilla.
  • Needinfo can be replaced by assigning a person to the ticket (because there can be multiple assignees). It will not post nag-mails like Bugzilla, but at least it should show up in some dashboard.
• We probably can’t define a section listing (multiple) links to external trackers. We use it quite a lot.
  • A possible workaround is to edit comment 0 and store it there (but that will require high privileges).

Additional notes post-meeting

• End-user experience when reporting a new bug will be a downgrade compared to Bugzilla, unless we make custom improvements.
  • Bugzilla’s home page is quite simple with several big buttons in the middle - to file a new bug, search in existing bugs, create a new account, and read documentation.
    • Filing a new bug opens a wizard that follows the user through product etc selection. It even searches for similar bugs and shows them to the user, to avoid filing duplicates.
    • Searching allows the user to search in all issues in a particular product (or all), optionally applying different search criteria.
  • Forgejo’s home page without login just has “A painless, self-hosted Git service” ad blurb without any instructions.
    • With login, the homepage is your gitforge dashboard, where you’re expected to know your way around. There’s no easy guide for newcomers.
  • Of course the scope and goal of Forgejo is completely different to Bugzilla, they serve different needs. No surprise their homepage differs a lot.
  • But we will need to create a reasonable way to guide community contributors to reporting a new bug, right from the home page. Otherwise we might lose a lot of bug reports that we previously used to receive.
• Issues can’t be filtered by org labels in API, only repo labels.
  • E.g. try to search in rpms/grub2 for issues labeled “release/40”.
  • This would preclude us from using org labels, universal across the whole distgit.
  • It should be fixed in Forgejo v9 (says Lukas Brabec, but without any testing, just peeking into the source code).
• Issues don’t support any states (NEW, MODIFIED, ON_QA, etc).
  • A workaround is to configure exclusive labels (only one can be active) on distgit repos and use those. Or add a new widget to the right bar of an issue view.
• It’s not possible to search in all issues in an organization (rpms, i.e. distgit). This means people can’t search if a problem was already reported, unless they know the right component from the beginning (and assuming that the ticket was really reported under the correct component).
  • There’s something that looks like global search, but that only lists issues from your own repos and from organizations which you’re already a part of. So it’s a “personal search” instead. It also doesn’t support any search filters.
  • There’s an organization-level search, but it’s hard to find (it doesn’t show up among the organization tabs, you have to go to the “personal search” and switch the drop-down), and again, you need to be part of the organization. If you’re not, you can’t search there. The only filter that can be applied is the label.
  • There’s no anonymous search.
  • The overall effect is that community won’t be able to search for reported issues across distgit, only in a specific repo. And maybe not even QA people (depending on their “membership” and what it means).
• It looks like we can’t upload attachments with any or no extension (syslog, user.journal, etc).
  • This is a very frequent use case when reporting a bug.
  • Tomas claims that this might be just a problem in our deployment.