By Georgia Bullen, Executive Director, Simply Secure; Daniela Saderi, Co-Founder and Director, PREreview; Katie Wilson, Design Researcher, Simply Secure
At PREreview we are building a new, more equitable way of evaluating scholarly work. We build community. We co-create knowledge. We create equity.
PREreview is a platform, resource center and convener that encourages feedback to preprints to be done openly, rapidly, constructively and by a global community of peers. By building a tool and a space for this paradigm of scholarship, PREreview seeks to build a more equitable, inclusive and diverse culture of knowledge evaluation within academia and beyond.
The challenges facing equity, diversity and inclusion (EDI) efforts in scholarship are numerous; problematic norms and biases are deeply entrenched, and inequities – the lack of fairness and justice that would lead everyone to have truly equal opportunities to thrive and participate – are baked into the very fabric of knowledge production as we know it. These inequities are not only reflected in existing policy and programmatic work, but are also embedded in the design of tools and technology that enable processes of knowledge dissemination and evaluation in the digital world. Ultimately, any conversation around access, user experience and power shifting can and should be about addressing the symptoms of systemic inequities.
This perspective is what motivates us at PREreview to apply an equity lens to the work we do, including the design and implementation of tools to engage research communities across the globe in the constructive, collaborative and unbiased review of research manuscripts made readily accessible to the community as preprints. To us, applying an equity lens to tool design is about listening to, centering and prioritizing the needs of the communities we want to support and empower.
Here we summarize some key challenges our team has identified through the last few years of working with the user experience and design organization Simply Secure and in close collaboration with diverse and global research communities. Challenges are paired with design choices implemented in response to those challenges in the latest version of our preprint review platform.
Challenge: There is an overreliance on name recognition, institutional affiliation, network connections and other signifiers of prestige as proxies for expertise, ability, trustworthiness, quality, reputation, validity and visibility of researchers in general and reviewers in particular.
Design response: Intentionally represent and highlight people on the PREreview platform based on their contributions to preprint and community peer review rather than their affiliations, credentials, or career level.
- User Profiles: PREreview users, aka PREreviewers, are displayed on the platform listing user-written bios, their contributions to preprint reviews, any PREreview community they are part of, and badges they may have received as recognition of their contributions. Career level and affiliation are de-emphasized or not readily visible, and are not available as a way to filter reviews. This model enables peer support and recognition of contributions and highlights the “human as a human” rather than identifying contributors by their academic prestige. Future ideas include implementing searching tools to find and connect users based on self-defined expertise and their offerings to the community (e.g., “open to mentor”; “open to review”).
- Community Pages act as organizing spaces for groups based on subject, initiative, or affinity. The site can offer different types of templates. Subject-affinity-based groups, such as the Outbreak Science community, act as a place for preprints by subject matter, whereas cohort-affinity-based community pages, such as the Open Reviewers, can act as a place to aggregate content generated by a specific group of individuals who share a training experience – in this case our peer review mentoring program – or belong to the same journal club/reviewing group.
- Badges are tags assigned to PREreviewers to highlight the role they may have played in a particular context – e.g., “Reviewer Trainee” and “Reviewer Mentor” to signal participation in our Open Reviewer program as stage 1 trainees and stage 4 mentors, respectively – or recognition for their contributions to preprint reviews as endorsed by the community via integration with Plaudit.pub.
Challenge: The peer-review process is inaccessible to many individuals and groups of individuals, as well as entire nations. Research is a global enterprise and its evaluation should involve all researchers, including early-career researchers (ECRs), who can provide valuable insights and feedback while also building their network and skills in peer review.
Design response: Lower the barrier to access, and engage and let contributors be recognized for their contributions to an open peer-review process.
- Profiles via ORCID: Login and signup to PREreview is done via the ORCID Public API. We share ORCID’s vision of a world in which all who participate in research, scholarship and innovation are uniquely identified and connected to their contributions across disciplines, borders and time. Via the API, user information publicly available on the ORCID profile, such as full name, ORCID iD, biography, and list of publications, is imported to the PREreview profile. Future ideas include offering the option to PREreviewers to have their preprint reviews listed on their ORCID profile as a way to enhance their chances to be recognized, and better connect scholars to their contributions to the scholarly record.
- Rapid PREreviews were developed through a partnership with the organization Outbreak Science as a way to enable rapid and structured feedback to a preprint in the context of an outbreak (or even a pandemic). In the new platform, rapid PREreviews are the first output of the review process for any topic preprint. Reviewers then have the option of writing a longer and free-formed review – the full PREreview. Rapid PREreviews offer the whole community an easy and standardized way to provide feedback, lowering the entry access to contributing to peer review. Additionally, the aggregated responses can help indicate to a preprint reader the community perceptions of the preprint’s credibility and value.
- Templates are text-based guidelines that can be loaded directly into the full PREreview editor to guide PREreviewers in the composition of the free-form review. The lack of formal and standardized peer review training leaves many ECRs wondering how they should go about writing a review report. Templates were designed to guide the writing process without imposing a prescriptive structure. Community managers have the ability to create and add more templates, customizing them to their community needs. Future ideas include organizing a design sprint inviting communities to co-create templates that can be useful to others.
- Requests for PREreview API: PREreview has added functionality that allows other sites and platforms to enable authors or users to request PREreview feedback on preprints through our API. Soon, preprint authors who submit to preprint servers based on Open Preprint Systems such as SciELO and RINarXiv will be able to submit requests for feedback from PREreview communities and have that feedback displayed near the preprint. This workflow enables the community to be the intermediary between authors and feedback, whereas traditionally, those channels are completely in the control of editors and journals. By disassociating the community feedback process from the editorial feedback and publishing process, the API broadens access to participation. Additionally, anyone on PREreview can request a review on a published preprint, which allows the community to drive interest in preprints rather than journals and editors through the “request a PREreview” button. Future ideas include enabling the PREreviews to be displayed next to preprints in a language that matches the language used by the preprint authors.
Challenge: The feedback process can be unnecessarily aggressive and unconstructive, leaving some scholars (particularly from underrepresented groups) vulnerable to destructive reviews and treatment.
Design response: Protect our users and create a safe space for discussion.
- Onboarding and Code of Conduct: The site also has an explicitly emphasized Code of Conduct (CoC). New users are engaged interactively in this CoC as part of onboarding, and all PREreviewers are asked to review and reflect on competing interests and community guidelines from the CoC upon submissions of the PREreviews. The CoC is not in place to prevent PREreviewers from being honest with their feedback. It is there to prevent them from being unconstructive with it. We believe one can be honest and respectful at the same time, and that is the tone we not only want to encourage, but that we expect of ourselves and our community members.
- Anonymous profiles: Our work evolves in the context of a global movement that advocates for openness and transparency. However, we recognize that communities traditionally excluded from the peer-review process are particularly vulnerable to potential, albeit unethical, retribution from critical reviews. By adding a (pseudo)anonymity profile option for each user to choose as an alternative to signing their PREreviews with their public profile, PREreviewers are able to protect their identities while still contributing to a peer-review process and being kept accountable for any violation of our CoC.
- Reporting: Users have the ability to flag inappropriate reviews, which are then queued for review and moderation by PREreview administrators. Depending on the severity of the violation, administrators can remove the inappropriate content and then follow up with the user to address any violations of the CoC – which may include blocking the user on the platform permanently.
Our commitment to our mission and values means we are fully invested in applying an equity lens to the process of re-imagining peer review and building technology to support the culture change we want to see in this space. To us, committing to equity and inclusion means incorporating these frameworks into every aspect of our work, which starts with the design of the technologies and platforms we use.
Our work in response to these challenges is by no means perfect and exhaustive – in fact, it is far from it. As we work against these formidable challenges that are systemic in nature, we continue to ask ourselves, and we encourage you to ask yourself, how are our systems – our tools, our technologies, our contributions, our governance – reinforcing inequalities, exclusion, racism, and bias?
Call to Action:
- Join us on PREreview: If you are ready to review preprints and build your profile as a PREreviewer, join us now! Find out more about the work we do on our website.
- Start your own community on PREreview: Ready to take your reviewing skills to the next level and bring others along? Request to start your own community on PREreview.
- Collaborate with us: To fulfill our mission we seek partnerships with like-minded individuals and organizations who can help us broaden our perspectives and impact, and who can benefit from the tools, resources and opportunities we offer. Get in touch with us and let’s build an equitable collaboration!
- Give feedback: We are here to listen and reflect on any constructive feedback you have for us. You can use Hypothesis to publicly annotate this piece or you can email us at contact@prereview.org. If you are curious about the source code for the new platform, you can find it on our organization’s GitHub, and you are welcome to leave questions/suggestions to our open issues or make a pull request.
- Demand and advocate for CoCs: We love codes of conduct as we believe they are a way to prioritize the safety of our community. We invite you to join us in advocating for codes of conduct (or participation guidelines – they have many names!) in every space you decide to interact and participate in.
- Stay connected: Follow us on Twitter @PREreview_ and sign up to our quarterly newsletter.
Acknowledgments
The design and development of the PREreview preprint review platform has received funding from the Sloan P. Foundation, from Mozilla, and the Wellcome Trust.
We thank PREreview Project Manager Katrina Murphy for her dedication to leading the development of the platform through this past year, working in close collaboration with the PREreview Leadership Team, which includes Drs. Monica Granados and Samantha Hindle, and the developing team composed of Josh King, Rae Gaines, and Harum Helmy, open-source developers at Throneless Tech.
We thank Simply Secure’s intern Carrisa Yao for her dedication in ensuring our values are communicated through the platform’s design.
We thank the team of open-source developers at eScire, in particular Maria Elena Herrera and Pedro López Casique, for their invaluable work in building the open integration between PREreview and Open Preprint System. We also thank Alex Medonça and Dasapta Erwin Irawan for guiding us in understanding the needs of their communities.
Last but not least, we thank all the people across research and open communities who have given their time to work with us in understanding needs, challenges and barriers, and have provided us with the constructive feedback necessary to move forward in our mission.
#
We welcome comments, questions and feedback. Please annotate publicly on the article or contact us at innovation [at] elifesciences [dot] org.
Do you have an idea or innovation to share? Send a short outline for a Labs blogpost to innovation [at] elifesciences [dot] org.
For the latest in innovation, eLife Labs and new open-source tools, sign up for our technology and innovation newsletter. You can also follow @eLifeInnovation on Twitter.
