How to Read a Technical Paper
by Jason Eisner (2009)
Multi-pass reading
Skim the paper first, skipping over anything that would take much mental effort. Just get an idea of where the paper is going, why it was written, what's old hat and what's new to you. To force yourself to keep moving, give yourself a limited time budget per page or use the autoscroll feature of your PDF reader.
Now, assuming the paper still seems worthwhile, go back and read the whole thing more carefully.
Why not practice on this webpage? Go ahead, skim it first.
S. Keshav describes three-pass reading in detail: What are you trying to do on each pass?
Write as you read
Write as you read. This keeps your attention focused and makes you engage with the paper.
Low-level notes
Often it is easiest to scribble notes on the printed-out paper itself, responding in context to the formulas, figures, and text. In that case, file or scan your annotated copy for future reference.
(Or perhaps annotate the PDF file directly, without printing or scanning. A free alternative to Acrobat is PDF-XChange Viewer, a Windows program that can also be run on Linux via wine.)
You can use notes on the paper to
restate unclear points in your own words
fill in missing details (assumptions, algebraic steps, proofs, pseudocode) annotate mathematical objects with their types
come up with examples that illustrate the author's ideas, and examples that would be problematic for the author
draw connections to other methods and problems you know about ask questions about things that aren't stated or that don't make sense challenge the paper's claims or methods
dream up followup work that you (or someone) should do
Low-level notes aren't enough. Also keep high-level notes about papers. It's quite useful to distill the paper down: summarize the things that interested you, contrast with other papers, and record your own questions and ideas for future work. Writing this distillation gives you a goal while reading the paper, and the notes will be useful to you later.
Michael Mitzenmacher writes: "Read creatively. Reading a paper critically is easy, in that it is always easier to tear something down than to build it up. Reading creatively involves harder, more positive thinking. What are the good ideas in this paper? Do these ideas have other applications or extensions that the authors might not have thought of? Can they be generalized further? Are there possible improvements that might make important practical differences? If you were going to start doing research from this paper, what would be the next thing you would do?"
I suggest sorting your file of notes chronologically, by when you read the paper, since that may help you find vaguely remembered papers or remember what else you were reading at the time. Sometimes you'll want to search by author/title/etc., so start the notes for each paper with a rough citation. (See also How to Organize Your Files.)
If you had to put a lot of effort into really understanding some point, you can share that effort with others (and record it for your own future reference) by improving the discussion of that point on the relevant Wikipedia page.
When and where to read
Start early. Leave enough time that if your attention wanders, you can put the paper down and pick it up again when you're in a better reading mood. This is better than trying to force yourself through it on a deadline.
Some people find it easier to read at particular times of day, or while eating or walking or riding an exercise bike. Do you habitually pick up the closest thing to read when you're at the breakfast table or in the bathroom? Then leave papers there for yourself.
Try reading with a friend! Sit next to each other, looking at the same copy of the paper, and stay synchronized at the paragraph or sentence level. Read aloud at times. You'll keep each other moving and help each other through the hard parts. Discuss as you go along.
Set aside time
When you are starting out in a new area, it may take you hours to read a conference paper thoroughly. That's okay. It's worth spending that much time to really understand a good or foundational paper. It will pay off in your future reading and research.
But I'm already a third-year student. Why is this paper taking me so long? There is no shame in reading slowly. It still takes me several hours to absorb a paper on something that I genuinely don't know well. (Also, it takes me hours to review a paper even in my own area, because the burden is on me to spot all the problems or opportunities for improvement. 75% of submitted conference papers are rejected, and most of the remaining 25% also need improvement before publication.)
Which parts to focus on
So do you really have to read the whole paper carefully on your second pass? Sometimes, but not always. It depends on why you're reading the paper.
I do think that when you are learning a new area, you should read at least some papers extremely thoroughly. That means knowing what every sentence and every superscript is doing, so that you really learn all of the techniques used in the paper. And understanding why things were done as they were: ask yourself dumb questions and answer them. Practice the ability to decode the entire paper—as if you were reviewing it critically and trying to catch any errors, sloppy thinking, or incompleteness. This will sharpen your critical thinking. You will want to turn this practiced critical eye on yourself as you plan, execute, and write up your own research.
However, there will also be occasional papers where it is not worth reading all the details right now. Maybe the details are of limited interest, or you simply don't feel equipped to understand them yet. Consider the parts of a typical paper:
Motivation. You'll want to understand this fairly well, or there's no point in reading the paper at all. But part of the motivation may depend on things you don't know
(mathematical background or past work). If you don't want to chase those references down now, you could just raise their priority on your reading list.
Mathematics and algorithms. These parts are the technical heart of the paper. So don't make a habit of skimming them. (You can learn a lot from how the authors solved their problems.) Nonetheless, you might skim a technical section if
o It seems like an explanation of something you already know. In that case, just check that it really says what you think.
o While you probably would benefit right away from knowing the method in detail, this paper is just not a good place to learn it, or it is too advanced for you right now. Understand what you reasonably can, and then put it on your list of things to learn for real. Perhaps ask someone else to explain it to you or to recommend a reading.
o It seems like an ugly ad hoc solution that no one would ever want to use anyway. The only reason to understand it fully would be if you wanted to criticize it or improve upon it. (Still, even if you skip the ugly details, understand what the authors' intuitions were. Think about how to capture those intuitions more elegantly.)
But you should still achieve clarity now about what the method accomplishes (its interface). Also try to glean when it is applicable, how hard it would be to use, and what determines its runtime and accuracy. Then you'll remember the method when you need it.
What you might skip for now are the hard parts: the internal workings of the method (its implementation) and any proofs of correctness or efficiency.
Experiments. Many papers test their methods empirically. When you're new to a field, you should examine carefully how this is typically done (and whether you approve!). It can also be helpful to notice what datasets and code were used—as you may want to use them yourself in future.
But once you've learned the ropes, you may not always care so much about a paper's experiments. After all, sometimes you're only reading the paper to stoke your creativity with some new problems or techniques. I confess that I often pay less attention to the experimental details—though examples or error analysis do catch my attention because they often shed light.
If you do care about the conclusions of the paper ("did the method work?" "should I use it?"), then you should go back and carefully examine the experimental design, including the choice of data. Were the experiments fair? Do they support the claims? What's really going on? Are the conclusions likely to generalize beyond this experimental scenario?
In short, invest your time wisely. Focus on what is valuable to take away. If you can't figure out which parts of the paper are most "interesting" or "important," do ask someone who should know! If you don't know who to ask, find other papers that cite this one (via Google Scholar) and see what they say about this paper.
Delip Rao suggests: "Never read the original paper on X first. Instead read several later papers on what they say about X, get an idea of X and then read the original paper. Somehow the research community is much better in explaining ideas clearly than the original authors themselves."
What to read
do creative web search
o experiment with several searches
o put yourself in an author's shoes; what phrases might they have used? o become a power searcher! (read the help pages for your search engine) o specifically search at the ACL Anthology, Google Scholar, etc.
track down related work (once you've got a relevant paper)
o forward references: see who else has cited the work (via an interface such as Google Scholar)
has someone else already listed the right papers for you? o survey papers in journals (also called "review articles") o course syllabi
o reading group webpages o chapters in textbooks o online tutorials
o literature review chapters from dissertations
o direct recommendations from friends or professors (perhaps at other institutions)
breadth-first exploration
o read a lot of abstracts (and skim the papers as needed) before deciding which papers are best to read
o it's okay to read multiple related papers at once, flipping back and forth so that they clarify one another
o to get a feel for the research landscape in an area, flip through the proceedings of a relevant recent workshop, conference, or special-theme journal issue
when the going gets tough, switch to background reading o textbooks or tutorials
o review articles
o introductions and lit review chapters from dissertations o early papers that are heavily cited
o sometimes Wikipedia
This page online: http://cs.jhu.edu/~jason/advice/how-to-read-a-paper.html Jason Eisner - jason@cs.jhu.edu (suggestions welcome)
How to Read and Evaluate Technical Papers
Original Text by Bill Griswold
Reading research papers effectively is challenging. These papers are often written
in a very condensed style because of page limitations and the intended audience,
which is assumed to already know the area well. Moreover, the reasons for writing
the paper may be different than the reasons the paper has been assigned, meaning
you have to work harder to find the content that you are interested. Finally, your
time is very limited, so you may not have time to read every word of the paper or
read it several times to extract all the nuances. For all these reasons, reading a
research paper often requires a special approach.
The first thing to understand is that the research papers you will read have been
published in different kinds of places. Some papers have been published in the
proceedings of a conference. These papers have been peer-reviewed but are
typically strictly limited in length to about 10-12 pages. Other papers have been
published in archival journals. These papers have also been peer-reviewed but
there are typically not any length restrictions, so they generally include more detail
in every dimension. Some papers are technical reports and these are not
peer-reviewed. In areas related to Computer Science, you may find that there is first a
technical report version which is then later published in a conference or a journal.
If a paper appears both in conference and journal form, the journal version is
typically a later, expanded and revised version of the work.
To develop an effective reading style for research papers, it can help to know what
you should get out of the paper, and where that information is located in the paper.
Typically, the introduction will state not only the motivations behind the work, but
also outline the solution. Often this may be all the expert requires from the paper.
The body of the paper states the authors' solution to the problem in detail, and
should also describe a detailed evaluation of the solution in terms of arguments or
an empirical evaluation (case study, experiment, etc.). Finally, the paper will
conclude with a recap, including a discussion of the primary contributions. A paper
will also discuss related work to some degree. Related work sections put the paper
in perspective with other research being conducted in the area. Papers are often
repetitive because they present information at different levels of detail and from
different perspectives. As a result, it may be desirable to read the paper
out-of-order or to skip certain sections. More on this below.
The questions you want to have answered by reading a paper are the following:
the literature. This problem intrinsically has two parts. The first is often
unstated. Think of this as the broader impacts. This is the benefits that are
desired in the world at large; for example, some issue of quality of life, such
as saved time or increased safety. The other is the technical problem, which
is why the people problem does not have a trivial solution; that is, why a
new technological or engineering solution may be required. Implicitly there
is implication that previous solutions to the problem are inadequate.
Occasionally an author will fail to state either point, making your job much
more difficult.
What is the proposed solution? This is also called the hypothesis or idea.
There should also be an argument about why the solution solves the problem
better than previous solutions. There should also be a discussion about how
the solution is achieved (designed and implemented) or is at least
achievable.
What is the evaluation of the proposed solution? An idea alone is usually not
adequate for publication of a research paper.
What argument and/or experiment is made to make a case for the value of
the ideas?
What benefits or problems are identified?
What are the contributions? The contributions in a paper may be many and
varied. Ideas, software, experimental techniques, and area survey are a few
key possibilities.
What are future directions for this research? Not only what future directions
do the authors identify, but what ideas did you come up with while reading
the paper?
As you read or skim a paper, you should actively attempt to answer the above
questions. Presumably, the introduction should provide motivation.The
introduction and conclusion may discuss the solutions and evaluation at a high
level. Future work is likely in the concluding part of the paper. The details of the
solution and the evaluation should be in the body of the paper. You may find it
productive to try to answer each question in turn, writing your answer down. In
practice, you are not done reading a paper until you can answer all the questions.
William G. Griswold CSE, UC San Diego
Reading research papers effectively is challenging. These papers are written in a very condensed style because of page limitations and the intended audience, which is assumed to already know the area well. Moreover, the reasons for writing the paper may be different than the reasons the paper has been assigned, meaning you have to work harder to find the content that you are interested in. Finally, your time is very limited, so you may not have time to read every word of the paper or read it several times to extract all the nuances. For all these reasons, reading a research paper can require a special approach.
To develop an effective reading style for research papers, it can help to know two things: what you should get out of the paper, and where that information is located in the paper. First, I'll describe how a typical research paper is put together.
Despite a paper's condensed form, it is likely repetitive. The introduction will state not only the motivations behind the work, but also outline the solution. Often this may be all the expert requires from the paper. The body of the paper states the authors' solution to the problem in detail, and should also describe a detailed evaluation of the solution in terms of arugments or an experiment. Finally, the paper will conclude with a recap, including a discussion of the primary contributions. A paper will also discuss related work to some degree. Because of the repetition in these papers at different levels of detail and from different perspectives, it may be desirable, to read the paper ``out of order'' or to skip certain sections. More on this below.
The questions you want to have answered by reading a paper are the following:
1. What are motivations for this work? For a research paper, there is an expectation that a problem has been solved that no one else has published in the literature. This problem intrinsically has two parts. The first is often unstated, what I call the people problem. The people problem is the benefits that are desired in the world at large; for example some issue of quality of life, such as saved time or increased safety. The second part is the technical problem, which is: why doesn't the people problem have a trivial solution? There is also an implication that previous solutions to the problem are inadequate. What are the previous solutions and why are they inadequate? Finally, the motivation and statement of the problem are distilled into a research question that can be addressed within the confines of this particular paper. Oftentimes, one or more of these elements are not explicitly stated, making your job more difficult.
2. What is the proposed solution? This is also called the hypothesis or idea. There should also be an answer to the question why is it believed that this solution will work, and be better than previous solutions? There should also be a discussion about how the solution is achieved (designed and implemented) or is at least achievable.
4. What is your analysis of the identified problem, idea and evaluation? Is this a good idea? What flaws do you perceive in the work? What are the most interesting points made? What are the most controversial ideas or points made? For work that has practical implications, you also want to ask: Is this really going to work, who would want it, what it will take to give it to them, and when might it become a reality?
5. What are the contributions? The contributions in a paper may be many and varied. Beyond the insights on the research question, a few additional possibilities include: ideas, software, experimental techniques, or an area survey.
6. What are future directions for this research? Not only what future directions do the authors identify, but what ideas did you come up with while reading the paper?
Sometimes these may be identified as shortcomings or other critiques in the current work. 7. What questions are you left with? What questions would you like to raise in an open
discussion of the work? What do you find confusing or difficult to understand? By taking the time to list several, you will be forced to think more deeply about the work.
8. What is your take-away message from this paper? Sum up the main implication of the paper from your perspective. This is useful for very quick review and refreshing your memory. It also forces you to try to identify the essence of the work.
As you read or skim a paper, you should actively attempt to answer the above questions. Presumably, the introduction should provide motivation. The introduction and conclusion may discuss the solutions and evaluation at a high level. Future work is likely in the concluding part of the paper. The details of the solution and the evaluation should be in the body of the paper. You may find it productive to try to answer each question in turn, writing your answer down. I recommend that you keep a notebook on all the papers you read, or mark-up the papers
themselves. You could use my standard two-page form that you can fill out for each paper. In practice, you are not done reading a paper until you can answer all the questions.
Also, you should be aware of the context of the paper in relation to the other papers in the class. Often a paper will represent a generalization, new direction, or contradiction to earlier papers.
If you find that filling out this form doesn't work for you, you can try writing a 250 word abstract of the paper--not rewriting the abstract at the front of the paper, but your abstract, capturing the above five issues from your perspective. I often find it useful to write an abstract because it develops the logical connections between the above five issues.
Taking time to writing down questions you have about the paper will often surface thoughts that were not initially articulated. Perhaps the paper was vague on key issues, or ignored issues that you think are important. If you come to class with such questions, you are prepared to counter or preempt my own questions.
explaining the solution to a research problem. However, engineering books are invariably oriented towards problem solving of one kind or another.
I have a habit of writing on papers directly, less with books simply because they cost so much. A well-annotated paper is worth its weight in gold, as it not only contains the content of the paper, but your assessment of its value to you.
Advice on note taking. Although I have provided a form that can be filled out, I actually advocate annotating the paper directly. The paper is a rich canvas on which to layer your thoughts. Here is how I suggest approaching the reading and mark-up process:
Highlight important comments as you go. Using a highlighter, as opposed to underlining, can really help key senteneces "pop out" at you when you return to review the paper later. Mark the important paragraphs of the paper according to motivation/problem,
idea/solution, their evaluation, and contributions.
On the front of the paper, write down the take-away message.
On the front of the paper, or near the end, write down your key questions. Other questions may be written in the margins as you read.
Try to answer the questions for yourself, as best you can. Use Google or other sources as appropriate.
