Should My Review Be Systematised? And What Is That, Anyway?

I came across an interesting example of a literature review the other day and thought it was worth sharing. The review was in a paper by Glowacki and colleagues (2020)*, who were reporting on a study to see if a series of grant writing workshops run with academic staff had actually increased the number of successful grant applications. 

What caught my eye was that the review was systematised. So what is that and why is it important if you are grappling with a literature review, or helping someone else grapple with one?  To answer that, I need to back up a few steps and explain systematic review studies.  If you are doing a qualitative systematic review study, or its quantitative counterpart, a meta-analysis, you are essentially doing a study of studies.  That is, you have a research question you want to answer, but instead of interviewing people or running an experiment or analysing a corpus of newspaper articles (or social media posts or whatever), you locate the relevant studies on your question and analyse the patterns and trends in the body of previous findings in order to find your answer.  An example is a recent-ish systematic review by Rogers and colleagues (2016)**, who wanted to know, now that we have considerable body of work on critical discourse analysis in educational research, what we have actually found out and what the implications of this are for future research. To answer these questions, they located 257 previous articles that undertook critical discourse anlayses in educational matters and plotted the major patterns and trends that emerged in that body of work. 

Systematic review studies are different from literature reviews.  While a systematic review study is a piece of research in its own right that sets out to answer a question, the literature review is not a piece of research in its own right; rather, it supports a piece of research by setting out why the research question needs to be asked.  That is why your dissertation literature review is a chapter in your dissertation or, if you are writing a journal article reporting on a research project, the literature review is section in that article. The literature review is not the research project itself -- it supports the research by establishing why the research question needs to be asked. 

Systematic reviews are very serious about the way they find the previous articles that will form their dataset, the body of articles that will be analysed in order to answer the research question. Protocols have been developed for how searches are to be done and how the final body of articles are to be delimited in order to a) be sure the answer to the question is based on the best available evidence to date; and b) avoid confirmation bias, cherry-picking articles that skew the answer in one direction or another. 

While your dissertation literature review is not a systematic review study, are the rigorous search protocols from these studies usefully applied?  This question is covered in an excellent book, Systematic Approaches to a Successful Literature Review, by Booth and colleagues***. They note that even if you are not doing a fully systematic review study, your literature review can still be systematised to various degrees.  For example, what databases did you search? Do you have a list of those? What search strings did you use to search them, and how did you develop and pilot them? From the potentially thousands of studies you might have located, how did you decide which of those studies would make the final cut for your dissertation literature review? If you also hand-searched for references, how did you do that?  You may or may not proceed this way -- that is a conversation for you and your supervisor.  And if you do proceed this way, you may or may not write all of your procedures up in your actual literature review, the way Glowacki and colleagues did (another question for your supervisor). You might instead put it in an appendix, or keep a record for yourself, in case an examiner asks you about your literature review. To return to our starting question, should your review be systematised? Maybe, maybe not. But it is worth thinking about. 

* Glowacki, S., Nims, J.K., & Liggit, P. 2020. 'Determining the Impact of Grant Writing Workshops on Faculty Learning'. The Journal of Research Administration. 51 (2): 58-83.

** Rogers, R., Schaenen, I., Schott, C., O'Brien, K., Trigos-Carillo, L., Starkey, K., and Chasteen, C.C. 2016. 'Critical Discourse Analysis in Education: A Review of the Literature, 2004-2012'. Review of Educational Research. 86 (4): 1192-1226.

*** Booth, A., Sutton, A., & Papaioannou, D., 2016. Systematic Approaches to a Successful Literature Review. 2nd ed. London: SAGE.

Do Guidebooks Work?

When you sit down to start your literature review and come to the realization that you have no idea what to do, there are any number of guidebooks out there you could consult. But how helpful are they? I recently started thinking about this when I happened to pull my copy of Phillips & Pugh's How to Get a Phd* off my book shelf.  I smiled to myself as I dusted it off and thumbed through the pages, catching sight of the notes I'd written in the margin. But what IS my position?! I had scrawled urgently on one page. 'Researchers examine data critically' I had emphatically underlined on another (48).

In many ways this was a useful book. It had a wealth of information on the nature of a PhD, what it was and wasn't; about the postgraduate and PhD system in the UK; about the supervision process; about the nature of research; and so on.  Another thing I remembered, however, was how confused I was about that thing called the literature review. I do not blame Phillips & Pugh's book for this, as I think this confusion had more to do with the fact that very little in my academic life up to that point had prepared me to understand the advice.  The same thing happened (and still happens) to me with technical manuals of any kind.  Perhaps that is why IKEA manuals dispensed with text and just used pictures. Alas, pictures alone will not help us figure out how to do a literature review. 

So what was it that confused me? I looked further at other things I had underlined.  I had to demonstrate that I was 'aware of the present state of the art: what developments, controversies, breakthroughs are currently exciting or engaging the leading practitioners and thus pushing forward thinking in the subject' (57).  And, I had to show all this to 'professional standard' (57). Excellent! I plunged into the stack of articles I had (it was an actual paper stack in those days). I recognized the breakthroughs, I could see what the consensus view was shaping up to be, and it soon became clear who the big names were and what their ideas were. I dutifully wrote all that down.  But what then? Crestfallen, I realized that all I had was a glorified list. Somehow, I didn't think that would count as 'professional standard'. It was description, and as Phillips and Pugh quite rightly and clearly said 'a mere encyclopaedic listing' (57) would not do. And yet, that was what I kept on producing as I drafted. I had by that time read countless other literature reviews (more good advice), but wasn't able to benefit from that because I didn't know what I was looking for.  

It wasn't until I read that 'key activities' (57) in doing my literature review would be 'organizing the material in an interesting and useful way, evaluating the contributions of others (and justifying the criticisms, of course), identifying trends in research activity, defining areas of theoretical and and empirical weakness' (57).  Ah -- now I was getting somewhere. Organize it all. I like organizing things. But organize it how? Eventually, after hitting on the idea of turning my notes -- my list --  into cards with one observation, thought, quote, etc. per card, I understood that I was making an argument, and my position was establishing why my research question should be asked -- why, as Phillips and Pugh point out, other people in my field would want to listen to what I had to say (57).  

Interestingly, Phillips and Pugh do make the point that the thesis needs to be an argument, but in a different section, on writing up (64). It took me a while to put the two things together because I had wrongly assumed that writing up was something I would do all at the end.  And once I had figured it out, then I could read other literature reviews and notice things like how they were structured and how they presented evidence and how they articulated their arguments. Once I knew what I was doing, the guidebook make perfect sense. Interesting, that. 

* Phillips, E.M. & Pugh, D.S. 1987. How to Get a PhD: A Handbook for Students and Their Supervisors. 2nd ed. Buckingham: Open University Press.