ATRIUM Workshop - Open Peer Review in Action
- Date
- January 18, 2027
- Location
- Berlin, Germany
Testing Transformations Evaluation Criteria
Venue: Centre Marc Bloch, Berlin, Germany
This workshop is designed as part of the ATRIUM project - 7.4 task on “Implementing a Peer Review Framework” (see version 1, https://zenodo.org/records/17787916 ), whose main goal is to develop a qualitative, criteria-based assessment model, in particular for Transformations. A DARIAH Journal to assess and recognise, within the research evaluation landscape, critical research contributions such as datasets, models, workflows or software. This workshop is complementing a DARIAH Campus course on improving peer review skills with APRF (to take place in Berlin, June 2027) and will enable the Transformations Editorial Board to improve their current evaluation grids and get feedback from the pool of Transformations’ reviewers for the new grids to be launched in Autumn 2026. It will provide participants (internal reviewers from the Scientific Committee board and external reviewers) with an opportunity to discuss and share their reviewing methodology collaboratively and discover innovative evaluation processes for non-traditional outputs.
Invited Reviewers: Georgios Artopoulos, Vicky Dritsou, Tugce Karatas, Sarah Middle, Walter Scholger
Organisers from Transformations Editorial Board: Toma Tasovac, Françoise Gouzi, Anne Baillot, Megan Black
Detailed program
The workshop will be organised into two interactive sessions. For the first session, we ask guest reviewers to prepare in advance their 10-minute presentation in order to stick to the timeline and run the workshop smoothly.
1:00 – 2:00 Lunch
2.00 - 2.30 General introduction and presentation of the current grids
Welcome, introduction to the context of the ATRIUM evaluation framework, overview of the Transformations assessment grids for research paper, data paper, workflow paper.
2.30 - 3.30 Sharing your reviewing experience
The reviewers will present their reviewing experience and examples on how they used the current grids with Transformations published articles from Issue 2 (Research paper, Data paper, workflow paper).
- 2.30 - 2.50 Feedback on reviewing a Research paper
- 2.50 - 3.10 Feedback on reviewing a Data paper
- 3.10 - 3.30 Feedback on reviewing a Workflow paper
Each reviewer will explain how they reviewed the paper(s) they were assigned (10 minutes per paper type + 10 minutes discussion):
- How did you use the evaluation criteria grid?
- What difficulties did you encounter when doing the review? (form/ content criteria, platform)?
- How did you structure your argument for the authors, given the open-review system?
3.30 - 4.00 Coffee Break
4.00 - 5.45 Critical discussion on the new grids
Discussion, sharing views with all participants on the new grids
4.00 - 4.30 The Editorial Board will present the new 2 or 3 grids (software paper, Digital Scholarly Editions - DSE, training paper).
4.30 - 5.30 Ask participants to analyse the new grids and share their feedback.
Work in 3 small groups simultaneously (30 mn by group - analyse the grid and share feedback with other participants, one group per grid, one member of EB to contribute to each group).
4.30 - 5.00 Each analysing one grid: Software paper, DSE paper, Training material
5.00 - 5.45 Each group sharing opinion and experience on their grid (15 minutes per group)