353:
276:
288:
316:
378:
found that going through that process means drastically less collaboration than creating an article directly in the main namespace. Is that beneficial to
Knowledge? If the community decides that article creation should be restricted, is autoconfirmed status a good threshold? One example where that restriction hinders contributions is if an experienced contributor comes in from another Knowledge. Where they previously could create an article (e.g. a translation of one of theirs), it would now have to go through AfC or be created as a draft in the user namespace and later moved, both reducing the opportunity for collaboration and improvement.
168:
72:
184:
256:, which when ACTRIAL started contained more than 14,000 articles; the issue of keeping up with the influx of articles being created was arguably one of the motivations for the trial. During the trial, this backlog has been reduced considerably, mainly accomplished during a backlog drive from mid-December through January. Prior to and after that drive, there were long periods of general stability in the backlog. Typically, patrolling work is in balance with the influx of articles being reviewed, something we also find is the case for the first part of the trial (related measure of Hypothesis 9).
304:
404:
uses it, and who reviews the drafts? What are the key social and technological parts of it, and can these be improved? The focus during ACTRIAL has perhaps been on NPP, a process which the WMF previously devoted some development resources. With our findings that AfC struggles to keep up with the influx of review requests, it is time to devote some resources there too.
408:
pages created in the Draft namespace is incredibly low (about 1.2%), and that a substantial proportion of articles created by fairly old accounts get deleted. These both suggest that article creation is challenging. How can that process be improved? Are there social aspects of it that become blockers, and how does technology fit into the picture?
365:
increase in permanent deletions, an increase in the rate of deletions, and that some of this increase comes through deletion of unencyclopedic content (e.g. advertisements). This increase in deletions is not commensurate with the increase in draft creations, meaning that we see a lot of created drafts that appear to not warrant deletion.
237:(AfC), which reviews drafts. We find a shift in content creation from articles to drafts, meaning we see a shift in workload from NPP to AfC. Following that shift, we see indications that AfC struggles to keep up with the increased number of submissions, while NPP gets some breathing room and substantially reduces their backlog.
205:
registration, which is impossible during ACTRIAL. Secondly, the trial puts in place an increased threshold, both time- and activity-based, for those who want to create articles, which previous research indicates is associated with increased retention. This makes a comparison of retention for article creators less interesting.
412:
articles got deleted. Who were the creators of those? Did these creators show up to
Knowledge during ACTRIAL but left when they were unable to create the article? Maybe these new contributors were creating articles that otherwise would not be created, e.g. articles in areas that are underrepresented on Knowledge. With the
596:
407:
Creating an article that does not get deleted is a difficult task. Can the WMF help design the process in such a way that someone who wants to create a legitimate article is more likely to succeed? While not the primary focus of our analysis, during our work we discovered that the publication rate of
403:
We have found a shift in content creation during ACTRIAL from the article namespace to the Draft namespace, and submissions of drafts for review at AfC. Prior to our analysis, little was known about AfC. While we now know more about AfC, there are many open questions. How good is the AfC process? Who
335:
We used several indicators of content quality. First, we measured the proportion of pages that are permanently deleted. If a page is permanently deleted, it obviously contained content that is unfit for the encyclopedia (e.g. copyright infringement). Secondly, we measured the proportion of pages that
268:
This shift of content creation from the article to the Draft namespace is worrisome. Previous research has shown that the AfC process is not as collaborative as creating content in the article namespace. It’s problematic if
Knowledge derails contributions by new users into a space where collaboration
212:
in retention compared to 2015–2016 from 5.2% to 4.6%. However, as we will discuss below, there is a shift of content creation from articles to drafts during ACTRIAL, meaning that a direct comparison is problematic. If we instead hypothesize that draft creators during ACTRIAL are a group consisting of
204:
an article or draft, we find an increase in retention similarly as described above. When it comes to accounts creating an article or draft, we focus on those creating drafts. We do this partly because we have found that those accounts would make those creations usually within the first 24 hours after
141:
Content creation has shifted from the article namespace to the Draft namespace, with a subsequent shift in reviewing from New Page Patrol to
Articles for Creation. Where New Page Patrol previously reported issues with their backlog, Articles for Creation appears to struggle with similar issues during
130:
The trial started on
September 14, 2017 (at approximately 22:30 UTC) and ran until March 14, 2018. This report is based on an analysis of the first two months of the trial, comparing statistics for that period against historical data. We will also make observations about later developments during the
331:
Multiple indicators point to a significantly lower influx of unencyclopedic content in the article (Main) namespace. Some of this content appears to be created in the Draft namespace. For articles passing our "encyclopedic content" criteria, there is not a significant change in quality. We also find
360:
Using these methods to measure article quality, we find a significant reduction for the indicators associated with unencyclopedic content: permanently deleted articles, and articles not flagged as "OK" by the draft quality model (Hypothesis 20). This finding is echoed in our analysis of reasons for
364:
We also made a similar analysis of pages created in the Draft namespace (Hypothesis 19). Given the shift in creations to that namespace, it is important to know whether the content created there is different from what we previously saw in the article namespace. Here we find a small but significant
179:
Secondly, we also find that the trial has no effect on activity levels of new accounts in the first 30 days after registration. There is no difference in the proportion of accounts making at least one edit (Hypothesis 2 ), the average number of edits made (Hypothesis 7), the proportion of accounts
163:
Our first set of hypotheses focuses on how newly registered users are affected during ACTRIAL. There are a variety of ways to measure activity on
Knowledge, and these hypotheses cover most angles. Two key measurements of concern are user retention (surviving editors, Hypothesis 5) and overall user
377:
A key question for the community following the trial is: what should
Knowledge’s publishing model be? The Wiki Way is to publish instantly, but make it easy to undo. The restrictions on article creation made by ACTRIAL shifts the model to review-then-publish for many accounts. Research on AfC has
399:
Our suggestions to the
Wikimedia Foundation all relate to a key question: how can Knowledge’s article creation process be improved? We discuss three perspectives on the process, and these perspectives seek to determine where the pain points in article creation are, and how they affect whether we
390:
mentions that there were no
Requests for Adminship in January 2018. Being an admin is performing maintenance work. Studying ACTRIAL, we see some of the challenges with sustaining these types of communities, e.g. being able to keep up with the influx of articles or drafts needing review. It seems
368:
In summary, we see a significant reduction in deletion of unencyclopedic content during ACTRIAL, but otherwise no change in content quality. Some of the unencyclopedic content appears to have moved to the draft namespace. To what extent ACTRIAL discourages disruptive contributions is unknown and
83:
The
Autoconfirmed article creation trial (also known as "ACTRIAL") is a six-month trial that ran on the English Knowledge from September 14, 2017 to March 14, 2018. During the trial, article creation was limited to users with autoconfirmed status, meaning they had made at least ten edits and the
411:
A key question that our study does not answer is: how are we losing promising contributors? At what point in their time on the site do promising contributors decide to leave, and why do they do so? Prior to ACTRIAL, we saw a lot of articles created by newly registered accounts. Not all of those
196:
in retention to a daily average of 3.3% (compared to 2.8% across similar periods of the years 2014–2016; Hypothesis 5). One thing to keep in mind is that during this period, September to November, we typically see an increase in retention, likely due to the school cycle. It is therefore unclear
40:
The Autoconfirmed article creation trial (also known as "ACTRIAL") is a six-month trial that ran on the English Knowledge from September 14, 2017 to March 14, 2018. During the trial, article creation was limited to users with autoconfirmed status (at least ten edits and at least four days since
287:
150:
Our study is organized into three themes corresponding to the main findings above: new user activity and retention, Knowledge's quality control processes, and content quality. We go into our findings for these three areas in more detail below, before providing some suggestions for the English
175:
We find that ACTRIAL has no measurable effect on new user registrations and activity levels. During the trial, the number of monthly registrations holds steady (Hypothesis 1). The average number of registrations for Sept–Nov 2017 is 151.0k, or about 5,000 accounts per day. This is a moderate
105:
123:. After discussing this report with the community, the WMF agreed to implement the trial in collaboration with the community and study the results. The WMF devoted resources and hired a researcher to work on designing the study and perform the analysis of the effects of the trial.
151:
Knowledge community and the Wikimedia Foundation aiming to spur a fruitful discussion of how the community should proceed and how the WMF can help improve content creation processes on Knowledge. Readers who seek more detailed results and analysis can find more information on our
419:
There is a great opportunity for the WMF to make substantial contributions to both the research literature on online communities as well as the Knowledge community by devoting resources to studying the article creation process and how new contributors are welcomed to Knowledge.
48:
A shift in content creation from the article namespace to the Draft namespace, with a subsequent shift in review workload from New Pages Patrol to Articles for Creation. This shift leads to the latter reviewing process struggling to keep up with an increasing backlog of review
508:
486:
391:
quite clear that switching off article creation for a group of potential contributors does not solve this problem, meaning that the community should carefully consider how it values and rewards maintenance work, and how it can maintain a healthy community of maintainers.
220:
In summary, our results suggest that ACTRIAL has little to no effect on new user participation. We do find an increase in new user retention among users that do not create an article or draft, but are uncertain whether that is caused by ACTRIAL or just random variation.
248:
indicated that a large proportion of non-autoconfirmed article creations were deleted, and during the trial these are no longer created. We also investigated the deletion rate of article creations by autoconfirmed users, and again find no change (Hypothesis 14).
101:
145:
Changes in content quality during the trial are mainly found in a reduction in creation (and subsequent deletion) of unencyclopedic content in the article namespace. There is also a small increase in deletions of similar types of content from the Draft
275:
475:
191:
Where we do see changes during the first two months of ACTRIAL is new user retention, measured by the rate of surviving new editors (accounts who make an edit in their first and fifth week since registration). Looking at all new accounts, we find an
439:
530:
497:
453:
563:
164:
activity level (average number of edits, Hypothesis 7). We would be particularly concerned if newly registered accounts were less likely to stick around during ACTRIAL, or if they tend to make fewer contributions to the encyclopedia.
259:
For the Draft namespace, we see a significant increase in page creations together with an increase in submissions to AfC (Hypothesis 16). We see indications that people reviewing Articles for Creation struggle with keeping up; their
332:
that articles do not gain quality more rapidly during their first 30 days since creation. In other words, ACTRIAL appears to be successful in reducing the creation of low-quality articles that would typically have been deleted.
88:
offering links to the user's sandbox, a task recommendation page, and using the Article Wizard to create the page in the Draft namespace. Prior to the trial, article creation was available to anyone with a registered account.
264:
increases more rapidly during the first two months of ACTRIAL than any previous period in our dataset (Hypothesis 17). They continue to be “severely backlogged”, per the description of their backlog on the AfC project page.
574:
315:
464:
180:
reaching autoconfirmed status (Hypothesis 3), median time to autoconfirmed status (for accounts reaching it, Hypothesis 4), and the diversity of work done (average number of namespaces or pages edited, Hypothesis 6).
585:
381:
How can the community encourage and reward maintenance work such as reviews at New Page Patrol and Articles for Creation? There is no doubt that work on encyclopedic content is important, but maintenance work is
552:
344:
in order to maximize prediction performance. Lastly, for articles that were labelled "OK", we measure their quality using ORES' WP 1.0 quality model. That model predicts which of the English Knowledge's
519:
341:
361:
why articles get deleted (Hypothesis 18). During ACTRIAL, we see a significant decrease in the average number of deletions per day, and this reduction mainly comes through speedy deletion criteria.
541:
303:
200:
Since there is a restriction on article creation in place during the trial, we further segment accounts into those that created an article/draft and those that did not. For accounts that
240:
Following the start of ACTRIAL, the rate of article creation is reduced in concordance with the removal of non-autoconfirmed article creations. This reduction does not affect overall
97:
297:
showing the age of articles in the New Page Patrol backlog as of March 12, 2018. Green signifies articles younger than one month, orange articles are one to three months old.
672:
56:
We expand on these findings and provide key questions for follow-up discussions both amongst members of the English Knowledge community as well as the Wikimedia Foundation.
119:
The trial resurfaced again in 2017, at which time an ad-hoc consortium of employees of the Wikimedia Foundation did a study of the backlog of articles needing review and
387:
127:
was designed in collaboration with the community on both the English Knowledge and Meta-wiki in order to determine how to best study the effects of the trial.
152:
124:
60:
597:
H19: The reasons for deleting non-article pages will change towards those previously used for deletion of articles created by non-autoconfirmed users.
281:
Graph of the number of articles in the New Pages Patrol backlog during the trial. Dotted line indicates the beginning of ACTRIAL on Sept 14th, 2017.
158:
245:
120:
93:
92:
This trial has a long history. Restrictions on who could create articles on the English Knowledge were first put in place in December 2005, when
17:
229:
Our second set of hypotheses focuses on the English Knowledge’s quality assurance processes, in particular two processes that review content:
372:
340:
draft quality model. The draft quality model is trained on linguistic features to identify spam, vandalism, and attack pages, and we
413:
321:
Graph of AfC submissions per day from the Draft namespace. Dotted line indicates the beginning of ACTRIAL on Sept 14th, 2017.
416:
focus on “knowledge equity”, knowing if content goes missing when article creation is further restricted is of the essence.
326:
187:
Graph of the monthly proportion of surviving new editors. Dotted line indicates the beginning of ACTRIAL on Sept 14th, 2017.
352:
346:
394:
509:
H6: The diversity of participation done by accounts that reach autoconfirmed status in the first 30 days is unchanged.
487:
H3: Proportion of accounts reaching autoconfirmed status within the first 30 days since account creation is unchanged.
261:
224:
171:
Graph of number of accounts registered monthly. Dotted line indicates the beginning of ACTRIAL on Sept 14th, 2017.
234:
85:
76:
689:"Accept, decline, postpone: How newcomer productivity is reduced in English Knowledge by pre-publication review"
84:
account was at least four days old. When new editors tried to create a new page, they were redirected to a
252:
Backlogs of work needing to be done is a consistent challenge in Knowledge. One of them is the backlog of
108:
determined that the trial should run for six months. The Wikimedia Foundation declined to implement this (
294:
213:
those who would have created drafts and those who otherwise would have created an article, then we find
230:
176:
reduction of 1.7% from 2016's average of 153.6k for Sept–Nov, and well within forecasted expectations.
167:
71:
183:
476:
H2: Proportion of newly registered accounts with non-zero edits in the first 30 days is reduced.
309:
Screenshot of the AfC project page on March 13, 2018, showing the "severely backlogged" status.
440:
H5: The proportion of surviving new editors who make an edit in their fifth week is unchanged.
531:
H14: The survival rate of newly created articles by autoconfirmed users will remain stable.
253:
8:
498:
H4: The median time to reach autoconfirmed status within the first 30 days is unchanged.
52:
There is a reduction in unencyclopedic content being created in the article namespace.
35:
113:
696:
655:
628:
138:
The trial has no apparent impact on the activity levels and retention of new users.
454:
H7: The average number of edits in the first 30 days since registering is reduced.
564:
H17: The backlog of articles in the AfC queue will increase faster than expected.
386:
important. They are a vital part of Knowledge’s quality assurance processes. The
356:
Proportion of permanently deleted articles created from January 1, 2016 onwards.
109:
102:
proposal to further restrict article creation to users with autoconfirmed status
688:
647:
620:
269:
does not happen, leaving the newcomer to figure everything out on their own.
41:
registration). Studying the first two months of the trial, our findings are:
700:
659:
632:
31:
27:
648:"Wikipedians Are Born, Not Made: A Study of Power Editors on Knowledge"
575:
H20: The quality of articles entering the NPP queue will increase.
337:
66:
621:"Crafting the initial user experience to achieve community goals"
100:) that users without accounts could no longer create articles. A
465:
H1: Number of accounts registered per day will not be affected.
605:
423:
673:
Page Creation dashboard of non-redirect pages created per day
646:
Panciera, Katherine; Halfaker, Aaron; Terveen, Loren (2009).
116:
extension to make it easier to patrol new article creations.
45:
No apparent effect on new user activity levels and retention.
687:
Schneider, Jodi; Gelley, Bluma S.; Halfaker, Aaron (2014).
586:
H18: The reasons for deleting articles will remain stable.
686:
645:
59:
For a more detailed analysis and breakdown, please see
553:
H16: The rate of new submissions at AfC will increase.
159:
New user activity and retention is largely unaffected
233:(NPP), which reviews articles that are created, and
619:Drenner, Sara; Sen, Shilad; Terveen, Loren (2008).
618:
682:
680:
520:H15: The rate of article growth will be reduced.
246:analysis by WMF staff prior to ACTRIAL starting
197:whether ACTRIAL is the cause of this increase.
373:Suggestions to the English Knowledge Community
244:, where we find no change (Hypothesis 15). An
18:Knowledge:Autoconfirmed article creation trial
677:
61:Research:Autoconfirmed article creation trial
542:H9: Number of patrol actions will decrease.
208:For accounts that create drafts we find a
388:February 20, 2018 Signpost news and notes
327:Less low-quality content in article space
351:
182:
166:
70:
395:Suggestions to the Wikimedia Foundation
14:
449:
447:
435:
433:
225:Shift in content creation and review
104:was put forth and passed in 2011. A
400:retain promising new contributors.
23:
134:In summary, our key findings are:
112:), and instead helped develop the
24:
717:
444:
430:
314:
302:
286:
274:
666:
590:
579:
568:
557:
546:
535:
524:
513:
639:
612:
502:
491:
480:
469:
458:
342:chose an appropriate threshold
13:
1:
369:would require further study.
7:
10:
722:
347:quality assessment classes
262:backlog of pending reviews
153:research page on Meta-wiki
131:trial where appropriate.
414:2017 movement strategy’s
336:are labelled as "OK" by
701:10.1145/2641580.2641614
660:10.1145/1531674.1531682
633:10.1145/1454008.1454039
349:an article belongs to.
254:articles needing review
693:Proceedings of OpenSym
357:
295:Insertcleverphrasehere
188:
172:
80:
625:Proceedings of RecSys
355:
235:Articles for Creation
215:no significant change
186:
170:
94:Jimmy Wales announced
77:New User Landing Page
74:
652:Proceedings of GROUP
28:Morten Warncke-Wang
26:Report written by:
358:
189:
173:
121:published a report
81:
75:Screenshot of the
293:Graph created by
98:Signpost coverage
713:
705:
704:
684:
675:
670:
664:
663:
643:
637:
636:
616:
599:
594:
588:
583:
577:
572:
566:
561:
555:
550:
544:
539:
533:
528:
522:
517:
511:
506:
500:
495:
489:
484:
478:
473:
467:
462:
456:
451:
442:
437:
318:
306:
290:
278:
231:New Pages Patrol
721:
720:
716:
715:
714:
712:
711:
710:
709:
708:
685:
678:
671:
667:
644:
640:
617:
613:
608:
603:
602:
595:
591:
584:
580:
573:
569:
562:
558:
551:
547:
540:
536:
529:
525:
518:
514:
507:
503:
496:
492:
485:
481:
474:
470:
463:
459:
452:
445:
438:
431:
426:
397:
375:
329:
322:
319:
310:
307:
298:
291:
282:
279:
227:
161:
106:second proposal
69:
22:
21:
20:
12:
11:
5:
719:
707:
706:
676:
665:
638:
610:
609:
607:
604:
601:
600:
589:
578:
567:
556:
545:
534:
523:
512:
501:
490:
479:
468:
457:
443:
428:
427:
425:
422:
396:
393:
374:
371:
328:
325:
324:
323:
320:
313:
311:
308:
301:
299:
292:
285:
283:
280:
273:
242:article growth
226:
223:
202:did not create
160:
157:
148:
147:
143:
139:
68:
65:
54:
53:
50:
46:
15:
9:
6:
4:
3:
2:
718:
702:
698:
694:
690:
683:
681:
674:
669:
661:
657:
653:
649:
642:
634:
630:
626:
622:
615:
611:
598:
593:
587:
582:
576:
571:
565:
560:
554:
549:
543:
538:
532:
527:
521:
516:
510:
505:
499:
494:
488:
483:
477:
472:
466:
461:
455:
450:
448:
441:
436:
434:
429:
421:
417:
415:
409:
405:
401:
392:
389:
385:
379:
370:
366:
362:
354:
350:
348:
343:
339:
333:
317:
312:
305:
300:
296:
289:
284:
277:
272:
271:
270:
266:
263:
257:
255:
250:
247:
243:
238:
236:
232:
222:
218:
216:
211:
206:
203:
198:
195:
185:
181:
177:
169:
165:
156:
154:
144:
140:
137:
136:
135:
132:
128:
126:
122:
117:
115:
114:Page Curation
111:
107:
103:
99:
95:
90:
87:
78:
73:
64:
62:
57:
51:
47:
44:
43:
42:
38:
37:
33:
29:
19:
692:
668:
651:
641:
624:
614:
592:
581:
570:
559:
548:
537:
526:
515:
504:
493:
482:
471:
460:
418:
410:
406:
402:
398:
383:
380:
376:
367:
363:
359:
334:
330:
267:
258:
251:
241:
239:
228:
219:
214:
209:
207:
201:
199:
193:
190:
178:
174:
162:
149:
133:
129:
118:
91:
86:landing page
82:
67:Introduction
58:
55:
39:
32:Ryan Kaldari
25:
606:References
424:Hypotheses
146:namespace.
142:the trial.
96:(see also
36:Danny Horn
125:The study
49:requests.
210:decrease
194:increase
30:, with
110:T32208
338:ORES'
16:<
384:also
34:and
697:doi
656:doi
629:doi
695:.
691:.
679:^
654:.
650:.
627:.
623:.
446:^
432:^
217:.
155:.
63:.
703:.
699::
662:.
658::
635:.
631::
79:.
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.
