Knowledge:Community health initiative on English Knowledge/Research about Administrators' Noticeboard Incidents

3128:"too often I've seen meticulously documented reports being dismissed as TL;DR or just not acted upon because apparently no one wants to take the time to read the diffs and make a judgment call. AN/I also has its regulars who seem to delight in the drama (and often foment it), and the mischief they create sometimes complicates a thread so much that the core issues are lost. I also am a little tired of seeing complainants (myself included) criticized for choosing to file an AN/I report rather than pursing some other avenue. So, sometimes even if the report has its intended effect, the process of getting there is fraught with unpleasantness. For volunteer editors, this may not be worth it." 114:(AN/I), a noticeboard that is a key part of the conflict resolution process on the English Knowledge. This survey is intended to understand community sentiments around AN/I, and will not lead to immediate or imposed changes to AN/I from the Foundation. Rather the purpose of the survey is to fill in gaps in data that could lead to on wiki discussions about possible improvements to how AN/I cases are managed. Any changes would need to be backed by the volunteer community on the English Knowledge. 149:

representative of the general Wikimedia user base. Furthermore, the findings presented in this report are based on the respondents' own assessment and understanding of their direct or secondhand experience with AN/I and, as such, they are limited in capturing the experiences only as perceived by the respondents. Other methods need to be explored in order to validate the information. We have gathered more information on the AN/I process in our

3237: 3021: 2717: 2412: 2133: 1805: 1661: 1534: 1289: 1146: 1013: 859: 732: 602: 452: 325: 198: 3134:"an ideal solution would address the thanklessness of both the Admin's and the involved editor's role. Find a way to motivate Admins to spend more time, not less, per case. Redistribute the workload load and review Admin decisions for signs that they are overworked and becoming burnt out." Other suggestions for increasing participation include, have a rotating "watch officer" to review inputs and disposition some of them." 3169:"There is a tendency to issue "boomerangs" against the filer of a complaint -- mostly justified, but sometimes not so. Because of this, I try to make certain that the report is fully justified and of a serious-enough nature that it's worth the risk of an unjustified sanction being levied against me. I'd say in general, the AN/I community is overly fond of finding reasons for issuing boomerangs." 72: 2359:"Complex interpersonal disputes, content disputes that have been ongoing for long enough or involve enough editors to have turned into conduct problems, complaints related to "incivility", any complaint whose specific circumstances touch on broader wiki-political issues and which therefore encourages a lot of soapboxing and off-topic personal commentary from observers." 2332:

problems. A complex problem may be something where the rules are ambiguous or contradictory, making solving a problem difficult, whereas a complicated problem could be a long running dispute between editors. Examples of these problems were described as cases that involved issues such as controversial topics, or issues involving civility problems and personal threats.

3340:"How ANI is currently built is extremely conducive for agenda driven editors to watch the page all the time and continue drama and other "power maneuvers" against "other factions" I think having some fundamental changes in design could incentives people to be civil and/or look for peaceful resolution, rather than always try to escalate." 150: 52: 47: 3480:

This research was prepared as part of a larger project to give Wikimedia communities more data when making decisions about their policies, and the processes used to enforce them. Support and Safety and the Anti-Harassment Tools team are committed to providing resources for community-supported change,

3476:

The results of this survey, similar to some community discussions around improvements to AN/I, do not point to a single point of improvement. However, the findings do strongly point to several areas where contributors see opportunities for improvements. Some would involve technical solutions, such as

2068:

Generally, the descriptions and suggestions were that AN/I handled cases best that were "objective and obvious cases" and "clear cut ones." Specifically, vandalism, bots, and sockpuppeting were mentioned frequently. In their responses, survey participants did not always include specific problems that

3374:

Respondents also highlighted issues with uninvolved participants on the noticeboard; some contributions from participants uninvolved with the cases were seen as unhelpful or distracting. Some free text survey comments suggest to approach this problem by developing methods to sort or filter comments

130:

The survey asked the participants to complete 23 questions and included six with write-in answers. Answers from the six write-in questions were sorted and grouped, and then labeled to organize the feedback into cohesive ideas. Some answers received multiple labels as they provided multiple kinds of

2331:

This question had 99 write-in responses. Some of the responses have been grouped into multiple buckets since the answers fit multiple kinds of labels. The more complex, nuanced, complicated, or personal the issues, the more participants seemed to stress AN/I is not the right space to handle those

148:

A total of 136 people filled out the survey. It is important to note that the following statistical analysis draws upon a survey with a small sample size of self selected experienced editors. As the survey was a voluntary opt-in survey, the sample of people who opted to respond to it might not be

2347:

The more ambiguous and/or more investigative work required, the harder it is to solve. Longer term problems that require more back and forth were viewed by admins as harder to solve, and something that AN/I didn't solve well. One user suggests that longer term problems would be better solved on

122:

The Wikimedia Foundation Support and Safety team and the Anti-Harassment Tools team recruited English Knowledge contributors to participate in a survey about AN/I in order to identify possible areas of improvement for handling cases by learning about user experiences and satisfactions levels.

127:

participation of a more diverse group of contributors, invitations to take the survey were sent to mailing lists of Wikimedia affiliates and some other known organized groups of English Knowledge editors. The survey opened on November 28, 2017, and closed on December 8, 2017, at 23:00 UTC.

126:

Invitations to take the survey were made on English Knowledge noticeboards and invitations were placed on the user talk pages of 400 randomly selected active editors with recent edits to AN/I (defined as taking place within the previous 12 months, October 2016–October 2017). To involve the

3447:

I like its open nature: everyone with relevant information has the opportunity to provide it. Also, when it works, it works well. And sometimes it works perfectly: a succinct, informative report is made, an admin reads it, investigates, and takes action. That's relatively rare, however...

3485: 2052:

AN/I seems to work best for more 'obvious' or less complex problems like sock puppetry, "short term" problems (problems that can quickly or easily be resolved), copyright violations, and 'bot or automated script' problems. This seems to be reflected in the write-in answers, as well.

2048:

When asked "for which type of problems does AN/I work well," there were 98 write in responses. Responses were focused on specific kinds of problems, which we sorted into four major groups: the complexity of the problems, technical issues, rules, or problems involving editor conduct.

3466:

The instructions, form and process is too unstructured to be effective, reporters need the free time to keep responding, experienced users know how to manipulate the process in their favor, and administrators do not seem to have the time to be involved to investigate.

1481:"There is the boomerang factor. Who knows what I have in my history that people will comb through, so I put up with harassment or bullying or vandalism because the general rule is that boomerangs are allowed since there is no scope rule with ANI reports" 3346:"create an optional submission form that would (1) automatically generate links to mentioned users' contributions, (2) automatically notify those users, and (3) eliminate edit conflicts when initially filing a report." 3173:

Additionally, the survey results show that some respondents want improved policies or enforcement of existing policy in order to positively change the tone and substance of the discussions on the AN/I during cases.

2343:"Ambiguous issues, issues where the topic is ideologically divisive, or where one of the users is new and the other one is a 'old hat' with plenty of friends (rarely is the discussion unbiased in these cases)." 3387:

Some respondents want to provide an opportunity for anonymous, or privacy-shielded reporting that protects personal information about gender and other personal information that might be used for retaliation.

2069:

AN/I is able to handle well; rather, they spoke about more general concepts. For example, lots of feedback mentioned 'straightforward cases' without specifying what makes a case straightforward.

1488:"It depends on who shows up; bias comes into play, e.g. people take sides based on whether they personally agree with the actions of people, not if people's actions follow policies and guidelines" 3155:"a greater commitment to fairness. Newcomers may not know all the ins and outs of Knowledge but that doesn't automatically make them wrong in a dispute." Another comment suggests that the rule " 2969: 3153:

of AN/I discussions that might be addressed with new policy. In particular, there were suggestions to make the discussions on AN/I fairer for new users. One respondent wants,

2353:"Anything where looking at large amounts of evidence is needed, where the outcome isn't obvious or where one or more editors are long-term so have supporters and opposers." 89: 17: 2080:"relatively simple issues that just need one admin or don't really involve disputes; disputes where one editor is clearly the disruptive one and can be sanctioned." 3488:, and will be following and participating as they develop. If you have ideas about how to further research processes like AN/I, we would be happy to discuss them. 3192:

While there was not concurrence about a specific type of change, there was strong feedback suggesting the benefit of modifying the structure of reports to AN/I.

3513:

Some responses were sorted into more than one "bucket", which means that the number of actual responses may be less than the total number of bucketed responses.

3187: 962:

There are mixed opinions among respondents about how well the general process works to resolve AN/I cases. There is a stronger dissatisfaction expressed about

3159:

should be a rule for everybody at AN/I, not just admins. Otherwise 'popular' users often avoid heavy sanctions for issues that would get new editors banned."

3363:

While there is not agreement about specific ideas, there is feedback that technical solutions could improve the function of AN/I. Some suggestions include:

2337:"What AN/I doesn't do well is grey areas, long-term abuse, and taking action against "competence is required" editors whose problems are less clear cut." 3477:

structural changes like form-based submission. Some would need social and procedural changes, such as the suggestions around clerking and moderation.

957: 3358: 111: 1113:

Have you avoided reporting one or more incidents to AN/I in the last 12 months, because you did not think it would be handled appropriately there?

3326:

Around a quarter of respondents thought that AN/I was fine the way it was, but almost 60% wanted some form of more structured reporting.

107: 2974:

A reoccurring theme in the free text comments was for there to be more moderators or clerks to enforce the rules. One person said,

2988:

How often, in general, would you say the following statements apply to discussion on AN/I? — Discussion is focused and neutral.

1628:

On a scale of 1–5, do you agree with the general process of how AN/I reports work? (e.g report, discussion, and then decision?)

936:

Almost three-quarters of survey participants reported being involved in an incident reported on AN/I in the last 12 months.

3435:

Several comments by survey participants concisely summarize opinions about AN/I that were commonly stated in the survey.

1756: 3179:"restrictions on non-administrators participating in discussions, limiting them to a certain number of edits per month. 1232:

52.99% have specifically avoided making a report on ANI because they were afraid it would not be handled appropriately.

1611:

More than six in ten participants reported "sometimes" or "frequently" disagreeing with the outcomes of cases at AN/I.

3120:

Other responses indicate a general need for more administrators to monitor AN/I. A user suggested for there to be

560: 3107:

Almost half of respondents said that discussions on AN/I are "almost never" or "rarely" focused and neutral.

3216: 3000: 2689: 2384: 2105: 1777: 1640: 1513: 1261: 1125: 992: 838: 711: 581: 431: 304: 177: 3144: 2058:"AN/I works great for clear-cut rule violations, bot problems, etc. It does black-and-white very well." 1249:

If you did not select "no" above, why did you think those incidents would not be handled appropriately?

2363: 809:

More than three-quarters of participants had been an uninvolved participant at AN/I in the past year.

3247: 3031: 2727: 2422: 2143: 1815: 1671: 1544: 1299: 1156: 1023: 869: 742: 612: 462: 335: 208: 145:

The goal was to gauge what AN/I does well, what it doesn't do well, and what needs to be improved.

156: 1744:

More than 50% of users agree or strongly agree with the general process of how AN/I reports work.

103: 1096:

Only 27% of respondents indicated they were satisfied with the way that AN/I cases are handled.

699:

How often have you been an uninvolved participant in discussions on AN/I in the last 12 months?

3350:"Standardize reporting, to avoid walls of text", "Structured reporting (with opt-out allowed)" 679:

Almost one-third of survey participants had not reported an incident to AN/I in the past year.

3122:"more of the admin corpus active on any given day. It's the same old faces time after time." 2064:"housekeeping and basic admin stuff, like IP vandalism/sockpuppetry, blatant trolling, etc." 275:

Three-quarters of survey participants reported being a contributor for more than five years.

8: 3164: 2976:"discussions need to be clerked to keep them from raising more problems than they solve." 153:; this data supplements the results of the survey from a more metrics-based perspective. 3141:

The survey indicates dissatisfaction with the way that AN/I cases are currently closed.

1501:

Have you ever disagreed with an AN/I outcome, such as the wrong person being sanctioned?

3156: 826:

How often have you been involved in an incident reported on AN/I in the last 12 months?

78: 3405: 3243: 3027: 2723: 2418: 2139: 2072:

Examples of answers that were labeled as 'straightforward cases': A user wrote in,

1811: 1667: 1540: 1295: 1152: 1019: 980:

On a scale of 1–5, how satisfied are you with the way reports are handled on AN/I?

865: 738: 608: 458: 331: 204: 2677:

What are some other changes you would like to see in the AN/I reporting process?

3291:

Multiple options for reporting (e.g. public reporting and private reporting)

3199:

In your opinion, what changes should be made to how reports are made on AN/I?

117: 3481:

be that technical development, further research, or coordination support.

110:

surveyed English-language Knowledge contributors about their perception of

3163:

There are mixed opinions among survey participants about the use of the

547:

More than 85% of participants are from North America or Western Europe.

3242: