ReproGen Shared task
What to submit
- Max 4-page report of the reproduction study including:
- description of how the study was conducted
- link to where resources used in reproduction study can be obtained
- overview of respects in which the reproduction was the same, respects in which it was different, and respects in which choices had to made without knowing what was done in the original study (i.e. whether or not it was the same or different is unknown)
- side by side tabular presentation of the original and the reproduction scores
- some amount of analysis and comparison between the original and reproduction results - Completed HEDS datasheet (please fill in here: https://forms.gle/MgWiKVu7i5UHeMNQ9)
- Any supplementary material
Format and author guidelines
Submissions to the ReproGen Shared Task 2021 should follow the same instructions as INLG 2021 submissions, and the ACL Author Guidelines and policies for submission, review and citation.
Please use ACL 2021 style files for which LaTeX style files and Microsoft Word templates are available at https://2021.aclweb.org/downloads/acl-ijcnlp2021-templates.zip. Alternatively you can use the Overleaf template at https://www.overleaf.com/latex/templates/instructions-for-acl-ijcnlp-2021-proceedings/mhxffkjdwymb.
Page limit and content
ReproGen papers have a page limit of 4 pages plus any number of pages with references. Supplementary materials can be submitted alongside a paper submission. Participants are also required to fill in the Human Evaluation Datasheet for each evaluation. More instructions here.
Please note that the submissions need not be anonymised for reviewing.
Deadline and where to submit
Submissions are due on Aug 15, 2021 (11.59 pm UTC-12) and should be emailed to reprogen.task@gmail.com.
For more information, please follow the CFP.