CMG Journal Club procedure for institutional CWR reviews

On average once every 4-6 weeks the CERN CMS team is asked to perform an Institutional CWR review. The CMG group usually takes care of this, although help from the other EP-CMxx groups is welcomed, especially with more technical papers.

For recent reviews, we have adopted a Journal Club-style approach, including a presentation of the paper in the CMG group meeting to stimulate discussion, which has helped to increase the involvement of group members in the review and improves the quality of the feedback given to the paper authors.

The procedure is as follows:

  • The CMG group leader will look for two volunteers, typically one CMG staff member and a fellow, to collect comments from the CMG group, present the paper in a CMG meeting, and compile the review to be submitted.
  • The volunteers send an email to the ep-dep-cmg-all mailing list, inviting comments from group members, and collect those comments, eg on a Google Doc. The comments should be sorted by line number and separated in the usual "Type A" and "Type B" categories for a CWR review.
  • At a CMG group meeting one of the volunteers will give a presentation to present the paper draft and main comments, to stimulate discussion. Preferably in the 2nd week of the CWR period, to allow time to read the paper and collect feedback from the group beforehand.
  • During the "Journal Club" discussion the other volunteer acts as "note taker", keeping track of new suggestions and comments received during the CMG discussion.
  • After the CMG meeting, the volunteers compile the review and include a short pre-amble(*). It is important that the style and manner of presentation is constructive (**).
  • Before submitting the review on behalf of the CMG group, the group leader would like to do a final iteration on the review with the goal to approach a uniform style for the CMG institutional reviews.

Our experience with the Journal Club discussions has been very positive. As always, feel free to contact us if you have suggestions to further improve the way the institutional reviews are handled.

(*) Template for the pre-amble, to be edited as appropriate for each case:

CMS Institutional Review by the CERN group <– please keep this line at the start, so that it will appear on the CDS summary page

Dear colleagues;

Thank you for putting together this interesting analysis and the publication.

As is usually the case, in what follows we will concentrate on suggestions for clarifying some important aspects of the analysis, as well as for improving the presentation and the overall impact of our publication (Type B comments) placing less attention to Type A comments (for which one usually gets tons of comments, often overlapping and even conflicting with those from other groups).

We hope you find the following useful. And once again, thank you for carrying out this work.

With our best regards,
Your colleagues in the CERN group
===

Clearly, if we happen to submit extensive Type A comments as well, then we remove the corresponding statement.

(**) Please observe the following CMG “style guidelines” when producing our group comments:

1. Phrasing:

(a) We do not characterise parts of the paper as “bad” or “badly written”; simply refer to the need for improvement.

(b) We do not address the authors directly, e.g. “Have you considered the systematic uncertainty arising from red frogs?”. Two options here: “Have we considered the system. unc…?” Even better, “The readers would wonder whether we have considered…”. Worse: “Why did you include this”? Instead: “The reader is wondering why we are including this in the paper”. And so on.

(c) We precede our suggestions with a “we suggest” or with a simple “please”. Instead of “Change labels on the axes to …”: “We suggest we change the labels…” or “Please change the labels on the axes to…”.

Of course, for short Type A requests there is no need to overdo this (it would be overkill)…. Examples:
– “accumulated” -> “collected”
– After CERN LHC, add a comma or “and”
– “based on control regions” → “measured in data”

(d) We avoid all aggressive words (irrelevant/out-of-place/wrong…). If something is really wrong, please refer to it as “incorrect” (yes, it makes a difference) and include it in a longer explanatory sentence.

(e) We phrase comments so as to facilitate the editing of the paper, i.e. after an “L340-342 are quite confusing, as this is the first time this is mentioned…”. We must follow this up by “We suggest that a sentence or two introducing the issue be added…”. This is helpful (as opposed to just providing criticism).

(f) Last but not least: brevity, with clear statements, is a friend to all. Highly expansive commenting is welcome by no-one.

2. Above and beyond all: we should make sure that the main elements of each paper, namely (a) abstract (b) introduction (c) conclusion (d) plots and (e) tables are clear, to the point and properly captioned. If for whatever reason the CMG review is particularly short, we should make sure we offer our suggestions for the above set of five.

3. Finally, please keep in mind that we have sent out many papers for CWR to the collaboration. And we took that feedback much more lightly, and found it also much more useful, when it followed, more or less, the above “guidelines”.