3.2 Step 2: Set your inclusion and exclusion criteria.
Prior to commencing the actual search activity, the researchers need to develop a protocol to guide the selection of would-include and -exclude study materials. This is determined after the research question has been formulated and is used to set the delineations – in/out-scope of the study. Inclusion criteria are the factors that an extant study must have to answer the focused research question set. On the contrary, exclusion criteria are all the characteristics of extant study that do not in any way contribute to answering the research question. These criteria could be study objective or purpose, year study was carried out or published, peer-reviewed, study methodology, study population and sample size, age, language in which study was published, geographical location of study, theory-/non-theory-based review etc.
When the study is theory-based, inclusion and exclusion criteria should focus on IS interventions that have been evaluated and the use of theory of change. Over the last decade or so, IS has seen increased in implementation science, an equivalent of process evaluation to verify the fidelity of system implementation activities. For example, there is no doubt about the effectiveness of digital health. However, little is known about the unintended consequences of digital health from a system point of view, making it difficult to institutionalize eHealth into routine care as with the challenge of reforming health policies in this regard.
In IS, evaluation is the appraisal of the quality and value of the characteristics/functions of systems and their impact on end-user and on business operations in an organization. It is “the act of measuring or exploring properties of a health information system (in planning, development, implementation, or operation) (Ammenwerth et al., 2004, p.480). Conventionally, evaluations in IS has been impact evaluation, an assessment of how an IS intervention been assessed affects existing conditions to bring about needed change. For example, when evaluating the effectiveness of an IS, attention is focused on the impact of an implemented system on organizational/business operations in the context of the effectiveness of users and other outputs (Hamilton & Chervany, 1981). Again, this could have also been how the feature/functionality of a system impacted end-user perception. Furthermore, the evaluation of health information technologies for high-quality, safe and cost-effectiveness to improve quality of and increase access to healthcare. IS requires rigorous and effective evaluation with fidelity. “Interestingly, change theory has not been much explored in IS research.
A SLR protocol provides guides for undertaking systematic literature review study and help to minimise risks of bias whilst optimising standardization/uniformity resulting in increased transparency and fidelity. It is a mini-guide or reference within the broader guide for conducting SLR. In fact, the protocol services as a quick reference tool for the researcher(s). Components of the study protocol include but not limited to the research question to be answered, study selection criteria and procedures, eligibility, checklist for study quality assessment, search strategy, etc. (Kitchenham, 2004) and above all the methods for performing synthesis (Ryan, 2019). Examples of tools that help in developing protocol are PRISMA checklist, Theta Collaborative tool, Cochrane Review handbook, AMSTAR checklist.