Checklist: 9 tough questions to help sharpen your paper

Dan Moran
October 2018, Trondheim

Some critical questions and selections from my peer reviews to help ensure your paper is sharp.

Strongly recommended: Write a short sentence answering each of the 9 questions for your paper.

Also recommended is this list: 8 reasons I accepted your article

  1. It provides insight into an important issue – for example, by shedding light on an unsolved problem that affects a lot of people.
  2. The insight is useful to people who make decisions, particularly long-term organizational decisions or, in our particular field, family decisions.
  3. The insight is used to develop a framework or theory, either a new theory or advancing an existing one.
  4. The insight stimulates new, important questions.
  5. The methods used to explore the issue are appropriate (for example, data collection and analysis of data).
  6. The methods used are applied rigorously and explain why and how the data support the conclusions.
  7. Connections to prior work in the field or from other fields are made and serve to make the article's arguments clear.
  8. The article tells a good story, meaning it is well written and easy to understand, the arguments are logical and not internally contradictory.

Background:
A new reader of your paper doesn’t care how much detail and time you’ve put into the analysis. They need to be convinced it makes a valuable contribution. Don’t be the one to get the comment: “This paper represents nothing more than a nice modelling exercise by the authors”. Take a few moments to think about how outside reviewers and the editor will see the paper.

These questions set a high bar. They will certainly be asked if you submit to Nature, PNAS, etc. If you’re not submitting there, keep these questions in mind (the journal editor likely will) but don’t get paralyzed.


(Imagine you are answering a BBC reporter)

    1. What's the issue or problem?
    2. What did you find?
    3. So what? Why does it matter?
    4. Who benefits from the results?
    5. How did you find what you did?

Suggestion: start writing the Introduction as if you are explaining the topic to your smart grandmother. Can she understand the abstract or introduction?

    6. In one sentence, summarize the contribution of this study.

Reviewers will ask:

    7. What is the clear research question(s) or testable hypothesis?
    8. Are your results statistically significant?
    9. We normally expect a paper to present one of (a) a novel finding, (b) an advance in methods, (c) a new resource or (d) a new idea. Which of these do you do?

If you go for a methods advance, is it really an advance, or just an obvious way to combine a few existing well-known techniques? Does it really give us a better way to do research? Did you motivate why those new methods are needed and effective?

Another good way to frame research is:

  1. Once upon a time, researchers believed that...
  2. But I thought maybe...
  3. So I went and did....
  4. And I found that....
  5. Which will change the way that we...

Several typical kinds of media stories are:

  1. New/unusual/not already known
  2. Current
  3. Important/affects many people
  4. Conflict /vs other work
  5. Current
  6. Mystery and oddities
  7. Close to home /my back yard
  8. Useful solutions /how to


Also recommended: How to write a thorough peer review



Here are some selected passages here from my past peer review reports and advice emails.

SAMPLE REVIEWS

Recommendation: Accept
This is a really nice paper. It’s simple, gets straight to the point, and is well executed and well written.

Recommendation: Accept
The paper is excellent. The writing is clear, the model is well-explained, the research questions are clearly stated and the results are well presented.


SAMPLE REVIEW

Recommendation: Reject
This paper offers a fine analysis for the case study of XXX. I do not see any serious flaws in the work. The paper is nicely written. But I also do not see any big novel contribution. It uses existing methods. A and B are defined so narrowly/strangely it’s impossible to really understand the comparison. I also struggle to see how the information from this study could inform better decisions. And I fail to see that the chosen case study is especially significant in the bigger picture. So I can say the paper is well done, but I am not convinced it offers a valuable contribution to the peer-reviewed scientific literature. The discussion of Limitations is quite short. This paper is fine as a student paper, but offers almost no contribution to the peer-reviewed literature. There is no methods advance and no novel findings. Additionally, I cannot imagine how to use the results of the paper to make better decisions. Reject.


ADVICE TO A PHD STUDENT

Written as advice to a paper that was launched by “Oh, I can combine these two datasets!” without asking Why?

In the Introduction you do a good job of situating this paper in the stream of research on the topic. But, I think you could still make the paper sharper. Usually a paper should do one of these three:
1. Pose a specific research question, and answer it
2. Propose a hypothesis, and see if the data invalidate or support it
3. Make an argument ("we should do X" or "Y is true") and support it.

My suggestion would be to change the title, so you're not making an argument like “We should do X” but rather answering a research question, and then in the Intro try to get your research question clearly stated in a single sentence. Example of RQs could be:
- Do group A and B have different carbon footprints due to factor X?
- Can changes in X lower the carbon footprint?
- Is the carbon footprint changing over time due to changes in factor X?

You just need one, or maximum two, such RQs. Then just focus the Results on those. My personal preference is for shorter papers with very clear questions & finding.


SAMPLE REVIEW

Recommendation: Reject
The modelling is good, but I still don’t see the purpose of it. The paper falls short of motivating how and why we need this new more detailed model. Regarding modelling methods, certainly the added spatial and temporal detail are elegant and show great technical competence. But intellectually there is no great step forward showing us how to do our research in a new way. As for the more detailed results offered, still this paper has to argue that the previous results are wrong or insufficient, and that the added detail provides insights or actionable information that previous studies do not.

Figure 6 is too clever for me. I can’t understand what is the once-sentence main takeaway from this figure. The caption gives a technical description but what I am supposed to learn or understand from the figure?

I confess I am also a bit off-put by the dismissive attitude the authors seem to take regarding Smith 2016 and Jones 2017. They seem more eager to dismiss these studies rather than focus on the novel contribution they make over that large body of work. How about a table with the main studies in the columns, and a list of attributes along the rows (“spatial detail”, ”temporal detail”, ”country coverage”, etc.) and filling in as charitably as possible how each study treats each aspect. Then a reader could get a fair view of what has been done (hint: it’s a lot) and clearly see the step(s) forward that this this study offers. I think this attitude to dismiss earlier work, rather than think critically why we need this new study, is why I am recommending Decline.