Tertiary review

In software engineering, a tertiary review is a systematic review of systematic reviews.^[1] It is also referred to as a tertiary study in the software engineering literature. However, Umbrella review is the term more commonly used in medicine.

Kitchenham et al.^[1] suggest that methodologically there is no difference between a systematic review and a tertiary review. However, as the software engineering community has started performing tertiary reviews new concerns unique to tertiary reviews have surfaced. These include the challenge of quality assessment of systematic reviews,^[2] search validation^[3] and the additional risk of double counting.^[4]

Examples of Tertiary reviews in software engineering literature

• Test quality ^[5]
• Machine Learning^[6]
• Test-driven development^[7]

References

1. Kitchenham, B. A. (2016). Evidence-based software engineering and systematic reviews. Boca Raton. ISBN 9781482228656.{{cite book}}: CS1 maint: location missing publisher (link)
2. Usman, Muhammad; Ali, Nauman Bin; Wohlin, Claes (2023). "A Quality Assessment Instrument for Systematic Literature Reviews in Software Engineering". E-Informatica Software Engineering Journal. 17: 230105. arXiv:2109.10134. doi:10.37190/e-inf230105. S2CID 237581262.
3. Börstler, Jürgen; bin Ali, Nauman; Unterkalmsteiner, Michael (2022). "How good are my search strings? Reflections on using an existing review as a quasi-gold standard". E-Informatica Software Engineering Journal. 16 (1): 220103. arXiv:2402.11041. doi:10.37190/e-inf220103. S2CID 245255682.
4. Börstler, Jürgen; bin Ali, Nauman; Petersen, Kai (2023-06-01). "Double-counting in software engineering tertiary studies — An overlooked threat to validity". Information and Software Technology. 158: 107174. doi:10.1016/j.infsof.2023.107174. ISSN 0950-5849. S2CID 257173845.
5. Tran, Huynh Khanh Vi; Unterkalmsteiner, Michael; Börstler, Jürgen; Ali, Nauman bin (November 2021). "Assessing test artifact quality—A tertiary study". Information and Software Technology. 139: 106620. arXiv:2402.09541. doi:10.1016/j.infsof.2021.106620.
6. Kotti, Zoe; Galanopoulou, Rafaila; Spinellis, Diomidis (2 March 2023). "Machine Learning for Software Engineering: A Tertiary Study". ACM Computing Surveys. 55 (12): 256:1–256:39. arXiv:2211.09425. doi:10.1145/3572905.
7. Munir, Hussan; Moayyed, Misagh; Petersen, Kai (1 April 2014). "Considering rigor and relevance when evaluating test driven development: A systematic review". Information and Software Technology. 56 (4): 375–394. doi:10.1016/j.infsof.2014.01.002.