Registered reports – A first for the Gateway

Registered reports are a two-part publication, where the study proposal is first reviewed before the research takes place. Once the study is complete, the publication is then updated to include the results and discussion. Unlike more traditional publications, acceptance of a registered report does not depend on the results obtained during the study, only on the rigour of the methodology. Reviewing and provisionally accepting a report before results are collected minimises publication bias and helps to ensure results, positive or negative, are published and accessible to a research field.

We spoke to Professor Alistair Lawrence and Dr Vincent Bombail, authors of the first registered report on the NC3Rs Gateway, to hear their experiences and advice for future authors. The publication is also the first registered report for an in vivo study on the F1000Research platform and one of only a handful published worldwide. Alistair and Vincent are part of a research team awarded an NC3Rs Project grant, led by Alistair, to maximise the positive welfare benefits of rat tickling.
 

Author bios

Vincent Bombail (VB) is a Research Scientist at SRUC whose work focuses on animal welfare, specifically investigating behavioural and physiological correlates of mood and affective states.

Alistair Lawrence (AL) is Professor of Animal Behaviour and Welfare at Scotland’s Rural College (SRUC) and the Roslin Institute at the University of Edinburgh. Alistair’s research focuses on positive animal welfare and more recently he has begun work on the concept of brain health and the relevance of this to animal health and welfare.

Why did you feel it was important to write a registered report?

VB: There have been many articles promoting registered reports as a tool to improve reproducibility, which is particularly important for in vivo studies. The whole process is transparent and having the protocol peer-reviewed gives you confidence and almost an indication that you’re on the right track before the experimental work is conducted. We all know it can be challenging to report experiments with unexpected outcomes or negative data and publishing the study as a registered report removes that concern. You will be able to publish the data that comes from this study if you adhere to the protocol, regardless of whether your hypothesis turns out to be true or not. I’ve always thought a registered report is like a marker, where you have formally put your research ideas out in the world. And of course, the initial report calls for the publication of the second one, where you describe the outcome of your experiment, so the first part is only the beginning of the story.

AL: I first heard the concept of registered reports from a young researcher in France. To begin with I didn’t understand, why would you give away your ideas before you start a study? Especially with some aspects of my research focusing on welfare in large animals such as pigs and cows, it can take a long time for these experiments to take place. But in the years since that visit, speaking to colleagues and listening to the NC3Rs speak about registered reports at their introductory meeting for new grant holders I came to realise the idea of “mustn’t give my ideas away so keep everything close to my chest” is an old-fashioned view. There has been a considerable shift towards open research since the start of my career and I think that’s a positive thing.

Given the differences to a traditional research article, what approach did you take to writing the registered report?

AL: In many ways it didn’t differ from what you would normally do when setting up for an experiment, but this process did make us really focus in on certain points ahead of collecting data. We had quite a tight deadline for publishing the registered report due to breeding schedules and the age of the rats we would be using, and we were encroaching on that deadline. We set aside time for a meeting, and I remember very clearly the discussion about what our primary hypothesis would be and what we could predict as our primary outcome measure. Writing the registered report made us, as a team, come to these decisions earlier on in the research process and more formally agree on these points.

VB: We also had to be prepared about what could happen during experiments and be aware of the possible scenarios, such as confounding factors, so we could build in contingencies into studies ahead of time. For example, sometimes you might notice during an experiment that your animals are showing signs of anxiety and so you would organise a test to confirm this observation. Instead of noting this during an experiment we had to consider these possibilities ahead of time.

How did writing a registered report benefit your research?

AL: As well as considering the experimental design and planned statistical analyses in much more depth at the beginning of the study, it also meant we sought the opinion of collaborators earlier in the process. The writing team for our registered report is very broad, many authors had a different expertise to our own and some had differing opinions to our group. Receiving input from such a wide range of people so early in the process allows you to act on any feedback or comments ahead of gathering data. For example as a group, we debated and agreed on the specifics of the different treatments weighing up what was required to address our research question against the need for consistency with previous work.

VB: I think it’s quite possible to get caught in your own world of ideas and concepts and it was extremely helpful sharing views before writing the report at a point when we were able to adjust experiments, rather than hearing these perspectives after data had already been collected. Unexpectedly, as a publication, this research report was successfully used to promote our research and attract project students as well as scientific collaborators, for further data and sample analysis.

What benefits did the open peer review process provide?

VB: I found the process really reassuring, I was concerned that by sharing these ideas ahead of time we’d be opening ourselves up to criticism or to significant changes to the experiments. This wasn’t the case at all and by the time we came to start the experiments it gave us confidence because we had support from the peer reviewers. If experimental outcomes are negative, it is very easy to blame yourself and question whether we have done something wrong. By writing the registered report, I knew this wouldn’t be the case, we would be able to publish the data and there would be a story to share even if the data showed our hypothesis was false. It doesn’t matter if the hypothesis is shown to be “wrong”. In fact, it is okay to be “wrong” and that is still a narrative that should be published, and I think that’s an important message for researchers to hear.

What would your advice be to other researchers who are interested in writing a registered report?

VB: It’s important to note the shift in time investment. While you’d already be planning the experimental design of studies, writing it into a registered report means a lot of this thinking must be shifted to before the experiment. I say shift because it does save you time later in the process, for example you have already done your literature review and identified the most appropriate way to analyse data, but there is a larger time investment at the beginning. Our experiments did benefit from this it was just a change to how we were used to working.

AL: I would suggest building this work into the beginning of your grant. There can be downtime at the start before research really kicks off, that would be a perfect time to be working towards publishing a registered report. You could also build it into the workplans and/or objectives of research proposals, Gantt charts would be a good way to illustrate this and could be a great way for a new postdoctoral researcher to become familiar with the research they’d be working on.

VB: We had good knowledge of the behavioural studies to investigate rat tickling, which was helpful for deciding statistical power needed for experiments and calculating the required sample size. If these are new models for your group, then definitely build in time for searching the literature to help inform these calculations. The NC3Rs Experimental Design Assistant also has guidance to help with statistical calculations if there are no estimates of variability available.

AL:  And at some point, you must pass the report over to the peer review process. It’s very easy to take a section or a sentence and keep analysing it, keep trying to provide clarification and sometimes it feels like that could be never ending. At the end of the day, it is up to the peer reviewers to confirm the report has enough detail surrounding the rationale, study design, hypotheses, and analysis strategy so once you’ve provided as much clarification as you and your team think is needed, then don’t be afraid to submit.

Read Vincent and Alistair’s publication Stage 1 Registered Report: Refinement of tickling protocols to improve positive animal welfare in laboratory rats on the NC3Rs Gateway.

Learn more about registered reports and benefits of publishing study protocols and hypotheses ahead of data collection on the F1000Research website. And if you are an NC3Rs grant holder wanting to publish a registered report, get in touch with our funding team.