Being part of a scientific community also means helping conferences or journals in the review process of scientific papers. In this lesson, we will see together how to do it. Let's start with an important premise, which I emphasize in particular for “rookies”: reading the paper from the beginning to the end is not necessarily the best strategy, particularly in the technical-scientific literature (which tends to be well organized and structured), although it seems the obvious thing to do. Let's see how to proceed instead. Start from the abstract, asking yourself these questions. What is the paper about? Naturally, the general theme or at least the sector should already be known because they are defined by the publication venue. We will hardly find a bioengineering article at an educational technology conference, of course. But having said that, it is important to identify the specific topic the paper is about. Secondly, we need to ask ourselves if the abstract is clear and entails all the main elements of the article. Because the abstract is meant to convey the marrow of the work: if that’s not the case, and it looks like and introduction instead or – even worse – a sort of meta-description, this is definitely one of the things to report to the authors. What do I mean by meta-description? I mean a sort of index. Something like: “Then, the method is explained, and later on, the conclusions and the important applications of the research are discussed…”. This is how the abstract should not be. Finally, we must ask ourselves whether the goal of the article is relevant. Is it worth reading it? You can find well-written articles that are fragile from the point of view of relevance. I remember an article on digital storytelling at school, a field I work on. The article asked the question: "Is digital storytelling at school an engaging activity?". Though the article was well-written, the question is not particularly relevant, for two reasons: one, the issue has already been amply demonstrated; two, since it is an issue of “educational technology”, the educational potential of the activity should rather be discussed. Engagement is fine, but are there substantial educational benefits as well? After looking at the abstract, I suggest you go directly to the conclusions (yes, I know, it's a nice leap, but it’s a good one). Let's start with the questions concerning the subject matter: are the results relevant? This is the fundamental question. A good research question is not enough, the results must also be important for the paper to be worth publishing. The next question is: are the results original? Do they advance the state of the art on the subject? And the last question concerns the "cohesion" of the paper, instead: are the results in line with the abstract? This is a question about the paper’s architecture, so to speak. I’ve noticed that even in an article you can find a technique that is typical of a completely different sector, that of screenwriting. This technique is called "set-up / pay-off". When an element is inserted (set-up), this must have its own "outcome": an explanation, a conclusion, a development. It must not hang loose, so to speak. If the article proposes two research questions, then the two research questions must be taken up in the conclusions and addressed properly. Sometimes conclusions entail other results and issues, which have been discovered on the way. This is fine of course, but the two research questions must not be forgotten. Anything else is welcome, but it is “more”. It seems obvious, but it is a very common mistake: to insert "set-ups" forgetting their “pay-off”. After having seen the abstracts and conclusions, it is necessary to identify the section where the goals of the work are expressed (in a more extended form than in the abstract, of course). Similar questions are asked again: are the goals relevant? Are they clear? And… are they consistent with the abstract? We then move on to evaluating the actual work. This is the heart of the paper and can be entailed into one or more paragraphs. The questions are: - Is the work sound from a scientific and technical point of view? - Was the method used appropriate? - Was the work done easy or difficult? - If there is data analysis, is it correct? The last section of the paper to look at, in my opinion, is the one on the state of the art. The amount of literature now available on any subject makes an accurate evaluation very difficult. Therefore, the main thing to look at is the focus: does the state of the art take the right focus on the theme? Is it too wide or too narrow? I’ve noticed recently that the flaw is to make state-of-the-art sections too large, given the ease of finding tons of literature on anything. A state of art that is too large is almost useless. Instead, it must be to the point. Of course, you must also verify that it is in line with the objectives. Still recalling the article I mentioned on digital storytelling: its theme was collaborative digital storytelling, but in the state of the art section there were no references on this specific organizational form, in spite of the fact that in literature, although not many, they are present. Last but not least: communication quality. Is the paper well structured? It is useful to read the sequence of paragraphs without reading the text, to see if they are coherent and convey the development of the reasoning. Is the text accurate? Are there any errors? Are the figures readable? Even if you print in black and white? We can now get to the final tips on how to review a paper. If abstracts and conclusions are excellent, then there is a strong possibility that the paper is excellent. The section on the work carried out should be verified so as not to be "cheated". If abstracts and conclusions are bad, you can generally avoid reading the rest. A quick check should suffice. If abstracts and conclusions are ... so-so, then roll up your sleeves. You have to read everything carefully. But in any case the paper is not very good We close with some notes on how to communicate the results of our analysis. What to say to the authors? In Italian there is a saying that says: "Every cockroach is beautiful for his mother". Remember to make constructive and not rough criticisms: do not offend the authors. At the same time, don't be too scrupulous: the reviewer's main role is to guarantee the scientific quality of what is published, so… no mercy. If a paper is not good, it's not good. What to say to the scientific committee? It is often a separate field in the review form, which the authors do not see. My advice? Tell the truth. If the article is not good, it's not good. If it's very good ... there's almost no need to tell. The publisher will see it from the review to the authors.
