In Favor of Lists: Some Thoughts on an Ancient Technology

In a recent post, I noted that annotated bibliographies are not the same as literature reviews.  An annotated bibliography is a list that describes each source you've read while a literature review is an argument that uses previous work in your field as evidence to establish why your research question needs to be asked.  But does that mean you shouldn't do an annotated bibliography? Actually, no -- you should.  Annotated bibliographies and other kinds of lists are an important tool you can use while you are drafting your literature review.  

Few people (at least amongst my students and mentees over the years) seem to realize this -- my suggestions to make and analyze lists in order to get a literature review off the ground often come as a bit of a surprise.  I think that is because lists are widely misunderstood. When we think of lists, we tend think of everyday ephemera -- the shopping list on the back of an envelope, the to-do lists on a scatter of post-its, the upcoming not-to-miss events list stuck to the fridge (which I always seem to miss anyway -- but that's a different story😉). So in this blog, I want to make the case for lists as a writing and drafting tool. 

I am going to be drawing on Jack Goody's The Domestication of the Savage Mind (1977, Cambridge: Cambridge University Press).  Here, Goody notes that one of our earliest uses of writing was not to set down our thoughts, or to memorialize myth, fable, story, or song -- it was to make lists, largely in relation to administrating agricultural societies and economies. 

Some of these lists were fairly pedestrian in purpose, what Goody calls an 'inventory of persons, objects or events' (80).  The annotated bibliography is such a list.  You can organize this list in all kinds of ways, but you will still be listing objects, things you have read.  This can be useful, because you do need to know what you have -- you need a record of what you have read and a basic description of what it's about. 

But this good and useful record keeping isn't the kind of list that will get you moving on your literature review.  For this, you need a different kind of list: 'an inventory of concepts' (80) -- a list of ideas.  And it is this kind of list, as Goody notes, that makes a different kind of thinking possible.  In addition to keeping track of what you have read, you need to list the key ideas from those sources -- the questions they ask, the methods they used, the findings they came up with, the interpretations put forward.  Notice here that you are no longer listing things you have read, and describing what they are about; you are listing the important concepts -- the ideas inside what you have read.  It is the ideas you have to analyze in order to work out what it is about previous work in your field that adds up to why your research question needs to be asked. 

And once you have the ideas in a list, you can re-order that list in all kinds of ways, and cross check it with other lists of ideas you've made, looking for connections and contrasts.  You can engage in 'systematic enquiry' (83), because you have made a list of ideas. And here is the really important point: you can organize your list into sub-sets of ideas by category (81). And you are the one who gets to decide what the name of that category is.  This is where your thinking really begins.  Let's say, for example, you have made a list 200 findings.  Do 30 of them all seem to point in the same direction, and make the same basic claim?  And what claim is that? When you come up with a label for that set of claims, you have moved your thinking to a higher level of abstraction.  You have taken a list of ideas, noticed that a number of those ideas seem to express a similar point, and you have named that point.  This is the beginning of your original synthesis. The observation that those 30 findings fit together, and how they fit together, was yours.  Now you are on your way to making a rock solid case for why your question should be asked.

 

But an Annotated Bibliography is Pretty Much the Same as the Literature Review, Isn't It?

 No.  It really, really isn't.  An annotated bibliography is an alphabetized list of all the sources you have read with a short summary given for each one.  It might take a long time and a lot of work to do your annotated bibliography.  You have to read all the sources, figure out what they say, and then summarize that in reasonable prose.  But this does not take much critical thinking and it does not require you to do anything original, which is why you will get comments like 'needs development' or 'not a literature review' or 'no critical voice here' if you just convert your annotated bibliography into paragraph form and think you are done.

In an annotated bibliography, you don't have to analyze what each source said and come to any sort of view about how the findings and conclusions from each source relate to each other and to your particular research question.  Neither does an annotated bibliography require you to formulate and structure an argument.  The only organizing principle behind an annotated bibliography is the alphabet, so you don't actually have to do any thinking to organize it -- all you have to do is know the alphabet and the surname of each author. You don't even have to know what the summary of each source says to organize your bibliography, because the alphabet has nothing to do with the summaries.  In addition, while the summaries might take some time and thought to formulate, they are not original.  Your annotated bibliography records what other people have already written, but does not do anything new or original with that information. You repeat existing knowledge in an annotated bibliography -- you do not transform it into something new.  

So the annotated bibliography is not the same as the literature review.  In the literature review, you need to synthesize the findings from each source as a body of literature.  If you start sorting the findings and putting like with like, what major patterns and trends emerge across all the findings? Why are they important?  If you put all the findings that seem similar (because they all point in the same direction, or all claim basically the same thing, or all flow from the same flawed assumption, for example) together in a column, what label would you give that column?  This label will be original to you, as it represents your idea of why that group of findings matters.  And you'll find that it will relate to your research question, as well (maybe not at first, but it will after you keep thinking about it and experimenting with different labels). After you sort all of the findings from previous sources into labelled columns, you will have identified the major patterns and trends in the existing body of work on your topic and how they add up to the need to ask your research question.  And this is what you need to establish in your literature review: why your research question needs to be asked.  

So is the annotated bibliography a waste of time? No. It is in fact a very good use of your time, and an essential step on the way to writing your literature review, which is why it is so often assigned as a task. You do have to read each source, and you do have to know, and really understand, what each one says -- what its main argument is, what particular perspective it represents, what it used for evidence, how it proceeded, and so on.  While listing all this out is a good way to get your head round what you've read and a necessary step, it does not result in a literature review. It is just the first step, and it does not require any critical thinking on your part.  Once you have a list of what each study you've read found (or argued or concluded), then your real work begins, as you have to determine how all these findings serve as evidence in the argument for why your research question needs to be asked.  And once you have determined that, you need to figure out what order to put this evidence in, that is, how to structure your argument.  This order will be based on your ideas, the patterns you have found in the previous findings and why you thought they were important.  This is what requires you to use your critical thinking skills, and it has nothing to do with the alphabet.  So honor the annotated bibliography.  Do the task with care and respect.  But don't confuse it with the actual literature review. 

    

My Literature Review: How Do I Know When I Am Done?

This is another question I often get, and it is a good one.  Gone are the days when you could locate all of the studies in your field and go through them as an individual. You could probably spend the rest of your life searching and reading and never be done. So knowing when you have read enough studies is one of many decisions you need to make as the author of your review.  So how do you make it?

There are two things you need to consider when you are deciding if your literature review is done.  As I have noted in a previous post, your literature review is an argument in which you make the case for why your research question needs to be asked. So have you made your case?  Do you take the reader, step by step, through the patterns and trends you have discerned in the body of work on your topic, establishing how these patterns and trends add up to the need to ask your research question? Have you instanced each of these patterns and trends with findings from the studies you've read? Sit down with your draft and read it honestly and critically.  Is the draft really just an annotated bibliography in paragraph form (this study said A; that study said B, etc.)? Or do you have a series of encyclopedia entries (descriptive accounts of studies on particular topics)? Or have you written a chronological account that has very little to do with your actual research question? If the answer to any of these questions is 'yes', you aren't done yet. This doesn't mean that what you have written so far is wrong or bad or was a waste of your time; rather, it simply means you have come to grips with the first step: getting your head round what each of the studies you have read established or claimed. You do need to know this in order to write your review.  But you also need to go on to the next step: understanding why all of these findings and claims are significant when it comes to establishing why your research question needs to be asked.  

You can think of the argument you are making in your literature review as a framework you are building.  Think of this prompt: My research question needs to be asked because . . . . .  How do findings from previous studies in your field allow you to complete this prompt?  For example, your research question might be about how students understand the concept of criticality. As part of your literature review, you might collect and synthesize a number of different ways that criticality has been defined in the academic literature. When you first do this, your argument will change every time you read a new paper, because it is still developing.  Every new paper you read gives you new information that you have to take into account, so you adjust your framework.  For example, you might have been arguing that your research question needs to be asked because we do not have good definitions of criticality.  But after reading a number of papers, you realise that actually we do have good definitions of criticality, they just stress different things, so you have to change your argument. Eventually, though, your argument will come together.  You will know why your research question needs to be asked, and you will have solid evidence for that from previous studies in your field. And you will read a paper that does not tell you anything new because it is just another example of a point you have already made and supported with numerous other examples from the literature. So now you have one more example you can add to an already robustly supported point.  And then you read another paper, and it is yet another example of this same point that is now even more robustly supported. And so on.  And this brings us to the second thing you need to consider when you are deciding if your literature review is done: saturation.  Saturation is another concept from qualitative research.  When all of your data fits satisfactorily into the framework you have built to explain it, you have reached saturation.  Applying this to your literature review, you reach saturation when all the new papers you read fit comfortably as yet other examples of points you have already robustly supported.  Now you know that your literature review is ready. Will someone else come along and read that one paper you didn't get to that contradicts one of your robustly supported points?  Maybe.  That can be the start of their dissertation.  Should you check your judgement with your supervisor? Certainly. But if you have a solid argument for why your research question should be asked and none of the additional papers you have been reading tell you anything new, you know you are in good shape.  

 

Reverse Engineering a Review

An exercise I often use with my students is to reverse engineer a literature review to work out how they are put together.  This helps to get students thinking about what authors actually have to do in order to turn reading notes into a review.  The analogy here is with creative writing: if you were a novice creative writer and you wanted to work out how to write great characters, you would not just read novels with great characters but study them carefully to see how the authors of those works created them. So as novice literature reviewers, I tell my students, train yourselves to learn from more experienced writers. 

As I guide my students through this process so that they can eventually do it more independently, one text I often use for this purpose is an article on English in multilingual advertising by An H. Kuppens (2010), which I have on my list of favorite reviews. Using the following short quote from the Introduction, I ask my students to speculate about what the author had to do to get this seemingly simple observation on the page.  

One notable exception is the research of English in advertising, which, after the publication of a few early studies (e.g. Haarman 1984, 1989; Masavisut et al. 1986; Bhatia 1987; Takashi 1990; Mueller 1992; Cheshire and Moser 1994; Myers 1994) has seen a striking expansion of research during the last decade (e.g. Griffin 1997, 2001; Martin 1998, 2002a, 2002b, 2007; Meinhof 1998; Gerritsen et al. 1999, 2000, 2007; Kelly-Holmes 2000, 2005; Piller 2000, 2001, 2003) . . . [8 more studies are listed, which I omit here for the sake of brevity] (Kuppens 20210, p. 115).

After a thoughtful silence, the students usually say something along the lines of Well, first the author would have had to read the articles.  Excellent, I say, but let's go back even further than that. Working together, we create a hypothetical to-do list. First, the author would have to have had the idea to work on multilingual advertising and would then have started reading on that topic.  At some point, that reading would have become more focused.  The author would then have had to locate studies on English in multilingual advertising, decide which ones were worth a closer read, and then start to make some sense of them as a body of work. At this point, I typically direct the students to the two blocks of references (the first beginning with Haarman 1984 and ending with Myers 1994, and the second beginning with Griffin 1997). Why are these references listed together like this?  What do they have in common?  There generally follows another thoughtful silence, longer this time, as the students look, really look -- perhaps for the first time -- at the references and consider why they are there.  There is no quote, so why are they given? And why so many?  Gradually, after several readings, the students notice that the first string of references are all examples of studies the author has decided are 'early', and the second, longer string represents what the author saw as the 'striking expansion' of studies that followed. The references support decisions the author has made about important trends in the body of previous work on multilingual advertising.  

When I first started doing this exercise, I was surprised that the students often found it a bit difficult.  I had designed it as a straightforward warm-up question, as the answers seemed obvious to me.  But the students did not find it straightforward or obvious -- it took some time for them to answer, and even then the responses were hesitant.  Why was this? I have noticed over the years that block references like this are typically absent from student work. Quotes from single sources are cited one at a time in support of claims made and the references are given, but I almost never see multiple sources in a block in student work, either at undergraduate or MA level.  

As a result of doing this exercise with multiple groups of students, I now have some insight as to why this might be.  Block references such as those Kuppens has given represent the results of synthesizing -- reading a body of work, coming to grips with what each study says, looking for patterns that hold across the whole body of work, and then naming those patterns in a relevant way -- deciding, that is, why they matter to the argument you are making.  It is in the noticing and naming of patterns in previous work that your original, critical voice as an academic author begins to emerge.  It is in and through this process that you offer your synthesis of how the multitude of previous papers fit together, what all the previous findings ultimately add up to.  The students, accustomed to finding quotes that confirmed what they already believed, were not practiced in this kind of synthesis, and so they did not notice syntheses in the articles they were reading, and because they did not notice them, what they read did not inform their own writing skills.  I realized that if I wanted my students to be able to write actual literature reviews in their literature review chapters, as opposed to serial descriptions of studies they had read, I would need to teach them how to synthesize -- which is why I started developing these reverse engineering exercises.  It is curious that the skill of synthesizing does not have more prominence in research methods curricula, at least as far as I am aware, and it remains mysterious in how-to guides.  Does anyone else find that?  Let me know in the comments 😊

Your Literature Review is Like a Qualitative Research Project

 No, seriously, it is.  Stay with me on this one for a bit, whether you are writing a literature view or teaching others how to write them.  You can think of reading notes as 'data' you have collected on an 'unknown culture' (the literature in your field).  Think of yourself as an ethnographer or anthropologist.  You have been a 'participant observer' in this culture (spending hours and hours in the library or on-line), 'talking' with the 'members' of this culture (reading and thinking about the articles, essays, and books) and seeking to inhabit their point of view by finding out what they say and why they say it (taking notes on the findings, methods, and methodological stances in previous research).  You've been asking questions about what they tell you (interrogating these findings, speculating about why, ultimately, they matter), and you have started to trace some connections and contrasts in what all the 'members' say; that is, you've been detecting what look like patterns in what you have found so far -- findings that tend to point in one direction but not another; an interest in some kinds of questions but not others; a preference for some methods over others.  You've also stumbled on some things that you intuitively feel are going to be important, but can't say why yet. You have, in other words, started to form some hypotheses about what this 'unknown culture', your field, is really all about.  With these hypotheses in mind, you continue your interactions with this new culture (locating additional sources, doing more reading and note-taking), asking if the patterns you have detected thus far hold and revising your model if they do not as you collect more evidence (qualitative researchers call this negative case analysis).

There is another way analogies with qualitative research can help us as we grapple with our literature review.  Qualitative researchers often use thematic coding to make sense of the mass of evidence they collect, and this too can be applied when you are doing your literature review.  Thematic coding is the process of assigning labels to the pieces of information you collect. Let's say, for example, that you are doing a study on attitudes towards wellington boots (well, why not?).  You have been reading studies on your topic and you now have masses of notes.  To start, you extract from your notes all the findings from these studies, and you put each finding on a separate card or slip of paper (that's what I do, anyway -- does anyone else proceed like this? Let me know!).  You start going through the findings from the literature in your field, looking for common themes, and you give each of these themes a name, a code.  For example, some of the findings seem to be about colour, so that is one of your codes: findings about colour.  Other findings seem to be about how long the boots last, how water-proof they are, how resistant to wear and tear they are. Durability, you think.  All of these findings are about durability.  You have just assigned your second code, durability. You proceed, many, many times -- the process is massively iterative and along the way you'll refine your codes, read additional studies, integrate findings from those studies, code and re-code again and so on.  Eventually, you will reach a stage when you can fit all the findings into workable codes, ordering an unruly mass of, for example, 50-odd disparate findings into 4 or 5 main themes that represent the major patterns or trends of study in your field.  How do these patterns add up to the need to ask your research question?  That is your next task in coming to grips with this 'new culture'. 

All of this labelling might strike you as pedestrian, especially when you start -- assigning your first few codes doesn't feel like you are doing anything significant.  It doesn't even feel like work.  But as you proceed, I bet you'll find yourself agreeing with Nowell and colleagues*: coding is thinking, and thinking quite deeply in a disciplined and structured way. It 'is a process of of reflection and a way of thinking about data' (pg. 5, italics mine).  When we code, we 'move from unstructured data to the development of ideas about what is going on in the data' (pg. 5). This coding process, in fact, is the beginning of your critical voice; that is, your original synthesis of why previous work in your field matters. You won't find the codes in the papers themselves, and there will be no paper that tells you what the codes are, because the codes come from your original assessment of why the work you are reading is important.  No one else will look at the mass of material in quite the same way. So if you haven't tried thematically coding your literature review notes yet, give it a try and let me know how you get on. 

* Nowell, L. S., Norris, J. M., White, D.E., and Moules, N.J. 2017. 'Thematic Analysis: Striving to Meet the Trustworthiness Criteria'. International Journal of Qualitative Methods 16: 1-13.

 

Should I Include Older Sources?

 This is a question I often get from my students, perhaps because they have heard things like Your literature review has to be up-to-date or You have to know what is current on your topic.  While both of these things are true, it does not follow that you should as a matter of course exclude older sources in your field. 

While this is one to discuss with your supervisor, in general terms my view is that there is no reason to ignore older sources just because they are old, and there can be many good reasons to include them.  You can be up-to-date and deal with older sources.  As the author of the literature review, you decide what gets covered and what doesn't, and that decision should be made on the basis of what the source in your hand contributes to the synthesis you are writing, not on some arbitrary cut-off date for publication. In any field, there will be the pioneering papers, the game-changing books, the edited collections from the conference that got it all started -- the sources that nearly everyone else in your field cites.  At the very least, you should read these and know their arguments thoroughly, since they are formative in your field.  

But there is another reason why these works are valuable.  Since much of the work in your field will include an account of them, you need to know what these sources said so that you can evaluate these later accounts.  You'll find that something very interesting tends to happen with the pioneering sources.  They are cited so often in so many places that an idea about what they said enters the public consciousness of the field and a consensus view about why they matter emerges. This consensus is often based on a particular aspect of the pioneering work, or a set of aspects, or even on just on a few quotes.  Other points of equal value in the original body of ideas may simply be forgotten.  Over time, the idea of what the pioneering source said can travel some distance from what it actually said, and the consensus about why it matters is taken for granted and no longer interrogated.  These differences between what the pioneering sources actually said and what current sources say they said may be just what you need to critically evaluate the work in your field in order to suggest a new direction. 

You might also find the opposite situation -- that much of the current work in your field has not travelled very far at all from the pioneering sources and that instead of progress there are just various re-statements of problems identified several decades ago.  Here too is an entrance for new work. 

And, of course, you could find that current work has taken a good account of the pioneering works and that there is a healthy debate on a wide range of issues and you can then place your work in this on-going debate.  The only way you will know this, however, is if you have read these pioneering works and come to grips with what they said. 

Do I Have to Have a Literature Review?

Yes. There is no way around it.  As Boote and Beile (2005) point out in their article Scholars before Researchers: On the Centrality of the Dissertation Literature Review in Research Preparation, the literature review can sometimes get short shrift in comparison to other elements of research that can seem more exciting or more significant, such as the findings. But without the literature review, your research can't really stand as a contribution to knowledge in your field.  If you haven't established why your research question needs to be asked, then what you did to answer it and the answers you ultimately got will come across as shallow and uninformed, little more than I wanted to know about A, so I did B, and found C. That will do if you are researching what phone to buy, but it won't be good enough for academic research.  Without a literature review to establish why your question is necessary, your answer will lose much of its potential power.  And if you have no deep and comprehensive sense of why your question is important to your field, you will have a difficult time articulating why the answer you got matters. So yes, you do have to have a literature review, and it does have to be taken seriously.  Your findings might crackle with originality, but you won't be able to see that, let alone say why, if you haven't done your literature review.  Do your excellent ideas justice -- invest time in your literature review.