This lecture is an introduction to a series that focuses on research evaluation. Evaluation is an essential aspect of research. It is ubiquitous and continuous on time. Its main goal is to ensure rigor and quality through objective assessment at all levels. It is the fundamental mechanism that regulates the highly critical and competitive research processes, fostering fairness and reducing bias and arbitrariness. As we saw, research findings are subject to an in-depth critical validation by researchers themselves as part of their research, before they may claim that results represent acceptable theories or valid research products. This may be called “internal evaluation”. Researchers evaluate their own results prior to submitting for publication, and peer review does a further evaluation before results are made public. The focus of this week is what can be called “external evaluation”: evaluation performed by others to assess research performance and quality. Research is evaluated when a proposal is submitted for funding, to ensure that money is spent in the best possible way. Research is evaluated when a researcher applies for an open position in a department or for promotion. Academic departments activate evaluation procedures to ensure that their limited resources are invested in productive researchers and to keep the level of their scientific reputation high. Research performance of a research institution can also be evaluated, for example to inform decisions that lead to resource allocations. Universities may do that for their departments, and even governments may do that for research institutions that receive public funding, to ensure accountability. Although the result of evaluation is often the return of a verdict (for example, regarding promotion to a tenured position), the goal is to sustain quality. Reviews should be constructive: they should identify possible weaknesses to compensate and possible areas of improvement. Evaluation is done by experts, who apply informed judgement. Judgement is largely based on qualitative analysis criteria. For example, when we discussed evaluation of research papers, I introduced three main qualitative criteria: originality, significance, and rigor. Because they are qualitative criteria, they are intrinsically subjective, open to unconscious bias, and the outcome of their assessment may be hard to compare across different experts who perform the evaluation and different subjects being evaluated. To mitigate subjectivity problems, normally evaluation is performed by a group of experts and suitable mechanisms are put in place to try to reconcile possible differences. In some cases, explicit consensus processes are put in place to expose the possibly different viewpoints to all evaluators, who then try to reach a shared assessment. Sometime evaluators perform repeated application of the consensus procedure to calibrate their individual assessments. We have seen an example of a simple evaluation process involving multiple reviewers in the case of journal paper peer review. In this case the synthesis of the various evaluations is largely in the hands of the editors, who interpret the reviewers’ reports and synthesize a decision. In the case of conferences, consensus is reached through discussions that may involve the entire program committee. In an attempt to reduce subjectivity that is intrinsic in qualitative analysis, researchers tried to develop quantitative analysis criteria. By adopting quantitative criteria, one would expect to achieve objective evaluations, which potentially eliminate bias and arbitrariness. If the result of a quantitative analysis may be expressed as a number, it can provide an indisputable way of comparing one subject against another. We will introduce quantitative evaluation criteria in a later lecture, but we will also show that unfortunately they do not solve the problem, and sometimes they even make matters worse. Before we discuss quantitative evaluation, we will first examine how research evaluation is performed traditionally in the main evaluation steps of a researcher’s career progress. This is the subject of another lecture, in which I will provide both factual information and some practical advices.