Revise community feedback

[NotMyFault]

Problem

The usability of neutral and negative ratings on /changelog/ and /changelog-stable/ decreased significantly over the past quarters. People tend to click buttons rather than filing a bug report or engaging in a chat.
The described scenario can be faced on various releases: 2.399 (2 issues, 6 rollback), 2.394 (12 issues, 10 rollbacks) or 2.388 (10 issues, 9 rollbacks).

Analysis

Accumulating the count of 3 (!!) weekly releases results in 49 potential bugs. This reveals two bigger issues:

1. Lack of classification

You can't classify the reports. Often, it's impossible to tell the impact an issue falls under. Did the user miss a step on the upgrade guide? Did they fail to update a plugin or did they run into a regression; we can't know. The number of people entering a useful Jira ID are much surveyable.
Given the lack of Jira issues or threads on the community forums pointing to a Jira issue makes it impossible for maintainers to get insights about the issue, unless you ask the submitter, which gets us to point 2.

2. Lack of addressability & verifiability

All votes are anonymous. Taking the figure of 49 potential issues, it's impossible to ask the submitter about details or point them to Jira to file a reproducible bug report. Additionally, we can't verify the results. Picking up the point from 1.), we don't know what the user did or what their environment looks like.

Solution

Remove neutral and negative votes and put a banner in place where to report issues and ask for support. Based on the analysis above, it's healthier for maintainers to have a detailed bug report or a thread containing the information needed, to determine the severity of an issue, work on a fix or point people to the documentation.
Positive ratings have been kept because they don't require the amount of triage, classification, and maintenance attention potential issues do.

Remarks

Jira, GitHub issues or discourse threads have a voting functionality. If several people ran into an issue, they can cast their vote still, just by voting on a valid issue, instead of casting their vote into the void.
Nonetheless, I would like to thank the few people using the buttons and link a valid Jira issue :)

[MarkEWaite]

As far as I can tell, this change removes the ratings system completely. I think that is a mistake. Even with the flaws noted in the pull request description, I find the ratings system helpful to provide a rough idea of the health of the weekly releases. Replacing the simplicity of the ratings system with text to invite the user to submit an issue report will sacrifice one of the forms of feedback that we've used successfully in the past.

I'd like to have opinions from @Wadeck , @timja , @dduportal , @kmartens27 , and others that use the ratings system for feedback.

[timja]

I’m very conflicted with it. The ratings are nice to get a pulse of the community.

but it’s very spammy and I feel the value is quite low due to the poor data quality.

I’m not a huge fan of the info call-out as it draws more attention than I think it should.

[kmartens27]

I can understand the concerns regarding value and ability to properly address the issues that users are having.

Instead of removing the neutral or negative options from the changelog, what about utilizing them in a different way? Currently, the icons ask users for a JIRA id, but would it be possible to have them redirect users to the issue tracker/create an issue link? I recognize that could result in more issues than necessary, and don't want to create more work.

However, if the repeated issues are valid, there would at least be some trail to follow and work from or reach out for contact. Alternatively, if there is already a JIRA for an issue that is seeing multiple reports, they could be closed and linked to the one that is being worked on. This could also provide insight as to how impactful the issues could be and unique configurations that users may have.

I worry that removing the negative & neutral ratings would appear as reducing or removing transparency/info for users. Obviously that is not the intent.

[MarkEWaite]

Docs Office Hours (Asia) noted that the end users can also use the ratings entries to guide their choice to install or not install a version. It was also noted that if a list of issues were presented, in addition to the empty field prompting for a bug number, it might significantly increase the value of the responses - titles of the most recent 7 bug reports and titles of bug reports already mentioned.

A link that would open the bug tracker might also improve the quality of submissions.

Suggested if we could extend the ratings database to accept general text, we could let them express their opinion. Then we need to read that with the usual indelicate and inappropriate content that arrives as in the Docs feedback sheet.

[basil]

I regularly look at the ratings as a mechanism for assessing the quality of the delivered software, so I am against the removal of the ratings. However, the ratings system could surely be improved. I was recently impressed with the user experience when reporting a Mozilla Firefox bug at https://bugzilla.mozilla.org — I was presented with a clean 2-3 page wizard to describe my problem in a few words, which showed me the results of a search of existing issues and (after I clicked through that) the form to report a new issue. Perhaps most of the issues with the current ratings system could be resolved by directing the user to the issue tracking system (ITS) after they have rated a release.

[MarkEWaite]

I propose that we close this because I use the ratings system as one assessment of general release quality.