Fong, P. W. L. (2009). Reading a computer science research paper. ACM SIGCSE Bulletin, 41(2), 138. doi:10.1145/1595453.1595493;
Smith, a. J. (1990). The task of the referee. Computer, 23(4), 65–71. doi:10.1109/2.55470
The reading process can be divided into three tasks: comprehension, evaluation, and synthesis. The latter is dependent on the former and requires deeper understanding of the paper.
To fully comprehend paper in the shortest time, read the paper asking yourself four questions:
1. What is the research problem? Around this problem, motivation of the research may be addressed. Possible situations may include: there is some weakness in existing research approaches; there is some crisis in the current research field; or the paper challenges the existing research paradigm.
2. What are the claimed contribution of the paper? Try to find something new in the paper. The new thing may include:
- a new question,
- a new understanding of an existing research problem,
- a new methodology,
- a new algorithm,
- a new proof technique,
- a new formalism or notation,
- a new evidence to substantiate or disprove a previously published claim,
- a new evaluation method, or
- a new research area.
3. How the paper substantiates the claim? A paper becomes scientific only if it is strongly supported, or it becomes a mere opinion. To support the authors' claim, some methodology must be used, which may include:
- theorems
- experiments
- data analyses
- simulations
- user studies
- case studies
- examples
4. What are the conclusions? What are the lessons learnt from the paper?
Most often, all the four components can be found in the abstract and introduction sections. When writing an article ourselves, we should also make them explicit in the two sections.
Evaluation goes along with each component. Ask yourself the following questions:
1. Is the research problem significant? Does the work enable practical applications, deepen understanding, or explore new design space?
2. Are the contributions significant? Are the author simply repeating the state of the art? Are the authors aware of the relation of their work to existing literature? Are there any real surprises?
3. Are the claims valid? Has the right theorem been used? Any errors in proofs? What assumptions are made? Comparing apples and oranges? Experimental setup problems?
Synthesis requires to think beyond the paper. Some questions can be asked after reading the paper:
1. What are the alternative way to substantiate the claim?
2. Is there any good argument against the case made by the author (contention)?
3. Can the research results be strengthened?
4. Can the research results be extended to other contexts?
5. Is there any relationship between this paper with other literature?
