Gamify Your Literature Review

Doing your literature review can be a real slog. Believe me, I know.  I like doing literature reviews and even I think it is a bit of a trudge sometimes.  But if you are a regular reader of this blog, you know that it simply must done -- there are no short cuts. You have to come to grips with the body of work that precedes your research. And that means reading many, many papers in order to make the case for why your research question needs to be asked. And, annoyingly, you will have to read papers that in the end turn out to be irrelevant. How else to know if they are relevant or not? Yes, a pain, I grant you.

 But can you have fun along the way, too? That is what I dare to suggest in today’s blog.

 

If you read literature reviews for long enough, you begin to see certain patterns in the way they are structured and written,  For example, many reviews begin, or state somewhere near their start, some version of There have been many studies of X (X being your particular topic).  Here are a few versions of this sentence from some papers I plucked randomly from one of my office shelves (What?! You don’t have a shelf full of papers yet? Acquire one immediately! 😊)

 

• ‘The importance of including counterarguments and rebuttals for making written argumentation persuasive has been underscored by much research’ (Liu & Stapleton 2014: 118, who block cite multiple papers from 1991-2007 in support of this claim).

• ‘Expressive writing studies are plentiful and the once anemic domain of letter writing as a vehicle for improving health has seen a resent surge of interest’ (Toepfer & Walker 2009: 182, block citing papers published between 2001-2009 in support). 

• ‘The concept of mindsets has received considerable attention in education in recent years’ (Irie et al. 2018: 576, with multiple papers cited in support as the review progresses). 

Not to be outdone, here is one from one of my papers.

 

• ‘The television series Buffy the Vampire Slayer (BTVS) has attracted a great deal of academic attention in recent years’ (Mandala 2007: 53, with a number of papers cited in support as the review unfolds).

And the ‘fun’ part? Set yourself a target for the day’s reading and see how many of these you can find. Gamify it further by making this a competition with your research chat group (What?! No chat group? See above 😊). Vote on the ones you like best. Look for the ones you think are most effective or creative or whatever. Get out your thesaurus and start coming up with your own versions.  See how many you can do in 10 minutes. Write some purposely outrageous ones you know you will not use, just for a chuckle.  And who knows? An outrageous version might lead you to a good one (a thought inspired by Edward deBono’s work on thinking).

 

There is much more to your literature review than your first sentence, as the pages of this blog amply demonstrate.  But starting is often the hardest part. So have some fun.

 

References

 

Irie, K., Ryan, S., and Mercer, S. 2018. ‘Using Q Methodology to Investigate Pre-Service EFT Teachers’ Mindsets about Teaching Competences’. Studies in Second Language Learning and Teaching. 8(3): 575-598.

Liu, F. and Stapleton, P. 2014. ‘Counterargumentation and the Cultivation of Critical Thinking in Argumentative Writing: Investigating Washback from a High-Stakes Test’. System 45: 117-128.

 

Mandala, S. 2007. Solidarity and the Scoobies: An Analysis of the -y Suffix in the Television Series Buffy the Vampire Slayer. Language and Literature 16 (1): 53-73.

 

Toepfer, S. and Walker, K. 2009. ‘Letters of Gratitude: Improving Well-Being through Expressive Writing’. Journal of Writing Research 1 (3): 181-198.

On Paraphrasing

 My students taught me something the other day. Paraphrasing can mean two different things. There's an easy version, and a hard version and -- you guessed it -- the easy version is what you might be most familiar with, but it is the hard version that you need for your literature review.

The easy version is when you take a quote, change some of the key words (perhaps with a trusty thesaurus), maybe alter the sentence structure a bit, and Voila! The quote is now in your own words, you give the author attribution, and you're good to go.  

But are you? Notice that in the easy version, you don't really have to fully understand the quote, or how it might or might not be evidence for why your research question needs to be asked. You could even do this by picking a quote at random from any source.  Paraphrasing this way is mostly a surface operation -- you don't have to think very hard about it. And why are you paraphrasing this way? Is it because you feel you have too many quotes already so you're going to paraphrase for a while? This too is surface-y. Deciding when to quote and when to paraphrase should be based on your engagement with your sources -- the dialogue you have with them when you are figuring out how patterns and trends in previous work add up to why your research question needs to be asked. Who agrees with you? Who doesn't? Whose work aligns with yours? Whose work gets close, but doesn't quite hit the mark? What will you 'say back' to these sources in your literature review as you establish why your research needs to be done? Where will you place them in those patterns and trends you have traced out?

Deciding when to quote and when to paraphrase on the basis of your on-going dialogue with previous work in your field requires the more complex version of paraphrasing.  This is not a surface operation, but an act of understanding. True paraphrasing emerges from a deep understanding of someone else's argument and a true appreciation for what their argument means for your research question. What is the crux of what someone is saying? Can you state that clearly in a few sentences, in words that a) show that you clearly understand; and b) would make sense to someone who had not read the source? You won't get to this by tinkering with surface features. You can only do this when you really understand the essence of what a previous piece of work sought to find out, and what they ultimately found (or didn't find, as the case may be).

And how do you get to this point? Practice. Take a piece of previous work from your pile of literature review sources -- a shorter piece, and one that you do have a good understanding of -- one that you read and said, 'Ah -- now that makes sense'.  Read it again. Several times. Now set it aside.  Give yourself a limit of 5 lines.  Try for something like this: The authors, interested in X, do a study on Y, and find Z. Remember that this is not a fill-in-the-blank exercise.  What you put for X, Y, and Z should be a fair assessment, in your own words, of what the authors were seeking to do. And remember that you must put the author attribution, as well -- a paraphrase is still someone else's thought. If it helps, try paraphrasing out loud to yourself first, as though you were telling a friend about this really interesting paper you had read, and then writing that down.  Or maybe tell a friend for real, if you have a particularly patient friend (😊), and then do your paraphrase.

As you get better and better at this -- and you will, if you practice -- graduate to more complex pieces. You'll find you are paraphrasing authentically, as part of your engagement with previous work and how it is evidence for why your research needs to be done, rather than tinkering with surface features for superficial reasons.

Picking Quotes and Your Literature Review

Now here is a topic that is more interesting than you might at first think. In any literature review, you'll be citing sources and quoting from them so, you pick the most relevant quotes and, well, quote them. What more is there to say? Quite a bit, as it happens. 

In some undergraduate work, the purpose is to determine your stance on a text or issue, and then to present evidence for that stance.  That evidence often comes from the literature and in many cases, the task goes like this: you dig in, do loads of reading, find work that supports your stance, pick the clearest or juiciest quotes, and cite them in support of your view. 

This work is not without value. You get practice in working out a clear stance, you get experience in searching for scholarly sources, and in reading them. You gain experience in working out how your work fits with previous work. Who agrees with you? Who does not? In order to answer these questions, you have to have a clear and robust sense of your own argument, you have to understand the arguments that other scholars have made, and then work out if they agree with you or not. And, of course, you get practice in citing source appropriately, using whatever citation conventions your institution requires. 

But when it comes to doing your literature review, more advanced work is necessary.  We need to move beyond picking quotes that confirm what we already believe. In a dissertation or research project, we dig into the literature with an open mind and then synthesize what we find there. The foray into the literature has two main phases (each of which, of course, will have many sub-phases).  When you are formulating your research question, you'll be digging into the literature to see what kinds of studies have been done before, what kinds of questions have been asked, and generally scoping out the current shape of your field. From this process, you'll finalize your research question.

Once you know that, you dig back into the literature, but you are no longer getting the general shape of what is there.  Now, you are looking at that shape, expanding your search, and then assessing the whole body of material to fully articulate why your research question needs to be asked. Picking quotes that agree with you will not help you here, because you also have to determine who does not agree with you, and who kind of agrees with you but not entirely and, in this latter case, where the agreement ends and the disagreement begins, and why that matters. You have to look at all the ways your topic has been studied, not just the way you are studying it, or the way you are most familiar with, or the way you secretly think is 'right'. And when you find sources that might threaten the need to ask your question, you have to deal with that, too, and carefully -- such sources can't simply be ignored. Do they make their points in convincing ways in robust studies? If so, your original question might need work. Or, you might assess such work, fairly and thoroughly, and come to the conclusion that the points made do not rest on thorough evidence.  But take care here -- there is a big difference between explaining previous work away because it is inconvenient, and dealing with it in principled and well-evidenced way. Are you sure you have been fair in your assessment? Are you sure you have been thorough? Can you demonstrate your view with clear evidence from your assessment? Quotes here will be very useful indeed, but notice how different this is from ferreting out only those voices that converge with yours and ignoring the rest.

So take the plunge in your literature review.  Listen to all the voices on your topic and decide why they matter to the question to you are asking.

Dissertation Deadline Looming? 5 Top Tips for Your Literature Review

 Have a dissertation deadline looming? Still wrestling with your literature review chapter? Or just generally stuck on what to do with that pile of notes and papers? Here are my top five tips for you, just in time for the start of 2024. 

1. Remember that your literature review is an argument. You are not describing the articles you read to show what you know.  You show what you know by using the findings, conclusions, and insights from previous work in your field as evidence in an argument that establishes why your research question should be asked.

2. Approach your literature review like an explorer.  Think of previous work in your field as the unknown terrain you have to make sense of. Invest time in the card-sorting technique up front as you venture into this terrain to save time later. Convert your notes into cards, with one thought, sentence, quote, finding, etc. per card, and then begin sorting.

3. As you sort, say to yourself, 'My research question needs to be asked because . . . . '.  You should have between 3 and 5 main reasons.  These are the steps in your argument. Carry on sorting to assemble the evidence for each step.   

4. You will inevitably be confused for a while, as your job in the literature review is to make sense of a chaotic jumble of publications. Don't fear confusion or run away from it. Embrace it, as it is a sign that you are moving out of your comfort zone towards new understanding. Each successive draft is a step closer to clarity, so give yourself time and space re-draft.  Remember that the card-sorting technique applies to your drafts as well as to your notes.

5. And finally, remember that your literature review is a key component of your work.  It is not simply something to get through so that you can move on to the more 'important' parts.  It is an important part. If you do your job right, the sense you make of previous work will be part of what makes your work original.  No one is asking your question in quite the way you are, so your argument for why it needs to be asked should be equally unique to you.  No one else can make your argument.   

So What's the Deal with 'Data' When I'm Writing My Literature Review?

Data can be confusing when it comes to your literature review.  Does your literature review have data? Do you deal with data when you are writing it? If your literature review has to be an analysis rather than a description -- which it does -- what are you analyzing, exactly?  Today's blog is about these questions. First, though, let's offer a definition of data.  Data, in a broad sense, is any information we collect systematically to answer a question.  In research, we usually think of data as the stuff we deal with in our analysis and results chapters.  That is, data is what we have elicited from our participants, be they texts or people, to answer our research question. This is sometimes called our primary data and it is of course crucial and essential. 

But collecting and analyzing data also applies when you are doing your literature review. While we tend to think of literature reviews as a very long exercise in reading, you are doing much more than that, so it can be useful to think about your literature review chapter as another place in your dissertation where you are collecting and dealing with data.  What you collect for your literature review is not your primary data, and it won't answer your research question.  But remember that in your literature review you are nevertheless answering a question: Why does my research need to be done? To answer this question, you go to the literature in your field, and maybe adjacent fields, and gather information.  This information is the data you collect, and then analyze, to figure out why your research needs to be done.  It is much more than just reading and taking notes.  You are are systematically searching for sources that relate to your project, not cherry picking studies that confirm what you believe.  You are then systematically capturing information from those studies. What did they ask? What did they find? How did they proceed? How do the findings relate to the research you want to do? This information, when you are doing your literature review, is your data. And if you've read my blog on how your literature review is like a qualitative research project, you know that you can corral this material (or analyze it, to use the formal term), by thematically coding it to identify the patterns and trends in previous work that add up to why your research needs to be done. It should be a purposeful endeavor, even if it takes you a bit of time to see the path through the data that leads to why your research should be asked. 

So the next time you look at that unwieldly stack of papers you need to read, or that bulging computer file (can a computer file bulge? I think you know what I mean 😊), and want to run away, remember: you are not just reading.  You are collecting data to figure out why your research question needs to be asked.

Should My Review Be Systematised? And What Is That, Anyway?

I came across an interesting example of a literature review the other day and thought it was worth sharing. The review was in a paper by Glowacki and colleagues (2020)*, who were reporting on a study to see if a series of grant writing workshops run with academic staff had actually increased the number of successful grant applications. 

What caught my eye was that the review was systematised. So what is that and why is it important if you are grappling with a literature review, or helping someone else grapple with one?  To answer that, I need to back up a few steps and explain systematic review studies.  If you are doing a qualitative systematic review study, or its quantitative counterpart, a meta-analysis, you are essentially doing a study of studies.  That is, you have a research question you want to answer, but instead of interviewing people or running an experiment or analysing a corpus of newspaper articles (or social media posts or whatever), you locate the relevant studies on your question and analyse the patterns and trends in the body of previous findings in order to find your answer.  An example is a recent-ish systematic review by Rogers and colleagues (2016)**, who wanted to know, now that we have considerable body of work on critical discourse analysis in educational research, what we have actually found out and what the implications of this are for future research. To answer these questions, they located 257 previous articles that undertook critical discourse anlayses in educational matters and plotted the major patterns and trends that emerged in that body of work. 

Systematic review studies are different from literature reviews.  While a systematic review study is a piece of research in its own right that sets out to answer a question, the literature review is not a piece of research in its own right; rather, it supports a piece of research by setting out why the research question needs to be asked.  That is why your dissertation literature review is a chapter in your dissertation or, if you are writing a journal article reporting on a research project, the literature review is section in that article. The literature review is not the research project itself -- it supports the research by establishing why the research question needs to be asked. 

Systematic reviews are very serious about the way they find the previous articles that will form their dataset, the body of articles that will be analysed in order to answer the research question. Protocols have been developed for how searches are to be done and how the final body of articles are to be delimited in order to a) be sure the answer to the question is based on the best available evidence to date; and b) avoid confirmation bias, cherry-picking articles that skew the answer in one direction or another. 

While your dissertation literature review is not a systematic review study, are the rigorous search protocols from these studies usefully applied?  This question is covered in an excellent book, Systematic Approaches to a Successful Literature Review, by Booth and colleagues***. They note that even if you are not doing a fully systematic review study, your literature review can still be systematised to various degrees.  For example, what databases did you search? Do you have a list of those? What search strings did you use to search them, and how did you develop and pilot them? From the potentially thousands of studies you might have located, how did you decide which of those studies would make the final cut for your dissertation literature review? If you also hand-searched for references, how did you do that?  You may or may not proceed this way -- that is a conversation for you and your supervisor.  And if you do proceed this way, you may or may not write all of your procedures up in your actual literature review, the way Glowacki and colleagues did (another question for your supervisor). You might instead put it in an appendix, or keep a record for yourself, in case an examiner asks you about your literature review. To return to our starting question, should your review be systematised? Maybe, maybe not. But it is worth thinking about. 

* Glowacki, S., Nims, J.K., & Liggit, P. 2020. 'Determining the Impact of Grant Writing Workshops on Faculty Learning'. The Journal of Research Administration. 51 (2): 58-83.

** Rogers, R., Schaenen, I., Schott, C., O'Brien, K., Trigos-Carillo, L., Starkey, K., and Chasteen, C.C. 2016. 'Critical Discourse Analysis in Education: A Review of the Literature, 2004-2012'. Review of Educational Research. 86 (4): 1192-1226.

*** Booth, A., Sutton, A., & Papaioannou, D., 2016. Systematic Approaches to a Successful Literature Review. 2nd ed. London: SAGE.

Do Guidebooks Work?

When you sit down to start your literature review and come to the realization that you have no idea what to do, there are any number of guidebooks out there you could consult. But how helpful are they? I recently started thinking about this when I happened to pull my copy of Phillips & Pugh's How to Get a Phd* off my book shelf.  I smiled to myself as I dusted it off and thumbed through the pages, catching sight of the notes I'd written in the margin. But what IS my position?! I had scrawled urgently on one page. 'Researchers examine data critically' I had emphatically underlined on another (48).

In many ways this was a useful book. It had a wealth of information on the nature of a PhD, what it was and wasn't; about the postgraduate and PhD system in the UK; about the supervision process; about the nature of research; and so on.  Another thing I remembered, however, was how confused I was about that thing called the literature review. I do not blame Phillips & Pugh's book for this, as I think this confusion had more to do with the fact that very little in my academic life up to that point had prepared me to understand the advice.  The same thing happened (and still happens) to me with technical manuals of any kind.  Perhaps that is why IKEA manuals dispensed with text and just used pictures. Alas, pictures alone will not help us figure out how to do a literature review. 

So what was it that confused me? I looked further at other things I had underlined.  I had to demonstrate that I was 'aware of the present state of the art: what developments, controversies, breakthroughs are currently exciting or engaging the leading practitioners and thus pushing forward thinking in the subject' (57).  And, I had to show all this to 'professional standard' (57). Excellent! I plunged into the stack of articles I had (it was an actual paper stack in those days). I recognized the breakthroughs, I could see what the consensus view was shaping up to be, and it soon became clear who the big names were and what their ideas were. I dutifully wrote all that down.  But what then? Crestfallen, I realized that all I had was a glorified list. Somehow, I didn't think that would count as 'professional standard'. It was description, and as Phillips and Pugh quite rightly and clearly said 'a mere encyclopaedic listing' (57) would not do. And yet, that was what I kept on producing as I drafted. I had by that time read countless other literature reviews (more good advice), but wasn't able to benefit from that because I didn't know what I was looking for.  

It wasn't until I read that 'key activities' (57) in doing my literature review would be 'organizing the material in an interesting and useful way, evaluating the contributions of others (and justifying the criticisms, of course), identifying trends in research activity, defining areas of theoretical and and empirical weakness' (57).  Ah -- now I was getting somewhere. Organize it all. I like organizing things. But organize it how? Eventually, after hitting on the idea of turning my notes -- my list --  into cards with one observation, thought, quote, etc. per card, I understood that I was making an argument, and my position was establishing why my research question should be asked -- why, as Phillips and Pugh point out, other people in my field would want to listen to what I had to say (57).  

Interestingly, Phillips and Pugh do make the point that the thesis needs to be an argument, but in a different section, on writing up (64). It took me a while to put the two things together because I had wrongly assumed that writing up was something I would do all at the end.  And once I had figured it out, then I could read other literature reviews and notice things like how they were structured and how they presented evidence and how they articulated their arguments. Once I knew what I was doing, the guidebook make perfect sense. Interesting, that. 

* Phillips, E.M. & Pugh, D.S. 1987. How to Get a PhD: A Handbook for Students and Their Supervisors. 2nd ed. Buckingham: Open University Press.