Evaluating digital inclusion initiatives: How can we get better evidence for what works?
Executive summary
This document provides advice on options for supporting the evaluation of digital inclusion initiatives.
Evaluation is the systematic determination of value. It helps us to understand how well an initiative is working and how it could be better. When done and used well, evaluation is a key input into decisions about initiatives.
In a stocktake of New Zealand digital inclusion initiatives, we found that very few have been formally evaluated. Through the review and interviews with key people, we identified 8 evaluation challenges:
- Challenge 1 — Diversity among digital inclusion initiatives makes evaluation consistency very difficult.
- Challenge 2 — We lack a shared understanding of how to measure digital inclusion outcomes.
- Challenge 3 — For some digital inclusion initiatives, it’s very difficult to measure outcomes and to determine the initiative’s contribution. Difficulties include contacting and tracking participants; and isolating the contribution of the initiative from other contributions to outcomes.
- Challenge 4 — Providers of digital inclusion initiatives lack the resources to carry out evaluation.
- Challenge 5 — There may be insufficient support for scaling up successful digital inclusion initiatives, which can discourage evaluation.
- Challenge 6 — Conventional perceptions of how social outcomes are achieved do not consider the contribution of digital inclusion.
- Challenge 7 — The influence of evaluation on funding decisions has lacked transparency, reducing providers’ motivation to evaluate.
- Challenge 8 — There may be a lack of knowledge about evaluation.
It’s unlikely that these challenges will resolve themselves, and it’s important that we take action to address them. Without better evaluation, we will struggle to justify any increased funding for digital inclusion; we will have little evidence to inform decisions about what initiatives should be scaled up; and we will fail to identify opportunities for improvement.
We suggest the following actions to improve evaluation of digital inclusion initiatives
Embed incentives and support for evaluation into funding for digital inclusion initiatives:
- Embed evidence requirements into decisions about funding initiatives.
- Fund initiatives at a scale and for a duration that supports evaluation.
- Allocate funding specifically to evaluation.
Build evaluation skills and knowledge:
- Develop guidance on evaluation of digital inclusion.
- Facilitate access to tailored evaluation advice.
- Promote inter-organisational sharing of experiences in evaluating digital inclusion.
Consider using large-scale analytics to evaluate digital inclusion initiatives:
- Assess the feasibility and suitability of using large-scale analytics to evaluate the various types of digital inclusion initiatives.
- Where large-scale analytics are feasible and suitable, begin by ensuring that initiatives have the needed prerequisites in place (e.g. informed consent from participants and collection of appropriate data).
Promote measurement of digital inclusion alongside other outcomes:
- Work with other agencies to embed measurement of digital inclusion outcomes into their monitoring and evaluation, where appropriate.
Purpose and scope of this document
This document provides advice on options for supporting the evaluation of government and non-government digital inclusion initiatives.
As described in the 2019 Action Plan — Building the foundations, this contributes to the government’s role to ‘lead’ and comprises part of the next step to “…investigate how to measure the success of government digital inclusion initiatives”.
This document includes a discussion of scope (what evaluation is and what ‘digital inclusion initiatives’ are), and a review of the current state of digital inclusion initiatives. Challenges in evaluating digital inclusion initiatives are identified, and actions to address those challenges are suggested.
Findings and recommendations are based on a stocktake of digital inclusion initiatives, interviews with key people, and a limited review of New Zealand and international literature on evaluation of digital inclusion.
What is evaluation?
Evaluation tells us about the value of an initiative
Evaluation is the systematic determination of the value of something. We all evaluate things every day, and we use those value judgements to make decisions. In the discipline of formal evaluation, we combine evidence with explicit criteria for value, to understand:
- how well an initiative is working
- in what ways it is working well or not so well
- how it could be better.
Evaluation is not method-specific; many techniques, quantitative and qualitative, can be used to evaluate an initiative.
Evaluation helps us make good decisions
When done well and used constructively, evaluation forms a key input into decisions about the future of an initiative. Evaluation can:
- provide accountability to funders and stakeholders
- support arguments for more funding (or less)
- identify ways we can improve initiatives
- assist decisions about where to prioritise effort
- support our personal satisfaction and integrity by showing us whether we’re making a difference.
Further reading on evaluation
Several excellent resources with further information on evaluation are:
- Superu (2017). Making sense of evaluation: a handbook for everyone
User-friendly entry-level guidance that provides an overview of evaluation concepts and processes.[Footnote 1] - Davidson (2005).[Footnote 2]
Textbook providing guidance on how to evaluate. Describes the types of questions that evaluators need to answer, how to choose appropriate methods to answer the questions, and how to combine qualitative and quantitative data with relevant values to draw evaluative conclusions. - Better Evaluation
Comprehensive and searchable website providing descriptions and examples of many different evaluation approaches. Created by an international collaboration of evaluators. Very useful for finding out about specific evaluation methods and topics. - What works
Aotearoa New Zealand website providing advice, case studies, and links to resources on evaluation.
What are digital inclusion initiatives?
The Digital Inclusion Blueprint — Te Mahere mō te Whakaurunga Matihiko states that being digitally included currently means: “…having convenient access to, and the ability to confidently use, the internet through devices such as computers, smartphones and tablets”.
The Blueprint acknowledges that what is needed to be digitally included will change as technology and society evolve (for example, coding skills may become necessary in future), but it focuses our current effort on: “…enabling non-users and sporadic users of the internet to become users, rather than on upskilling people who already access and use the internet in their day-to-day lives.”
The Blueprint describes 4 elements that are needed for a person to be digitally included.
Becoming digitally included occurs when people have the motivation, access, skills and trust to conveniently and confidently use the internet.
Initiatives that are in scope
We define digital inclusion initiatives as services, projects or programmes that contribute to enabling everyone to conveniently and confidently use digital devices and the internet, via improving motivation, access, skills or trust.
Digital inclusion initiatives contribute to enabling everyone to conveniently and confidently use digital devices and the internet.
In-scope initiatives include:
- services that develop people’s motivation, access, skills or trust, and that are available to people who are not yet digitally included, such as:
- training in foundational digital skills
- arranging affordable access to devices and internet connections
- initiatives that improve online safety and trust, for example through improving people’s awareness of, and resilience to, online threats such as scams, privacy breaches, and through protecting Māori data sovereignty
- services, projects or programmes that make online content more accessible for disabled people.
Initiatives that are out of scope
There are other types of initiatives that touch on aspects of digital inclusion but are out of scope for the time being.
Out-of-scope initiatives include:
- initiatives that focus on improving motivation and digital skills among people who already access and use the internet in their daily lives, such as mentoring and training courses in coding and robotics (these initiatives do not fit the Blueprint’s current definition of digital inclusion)
- initiatives that support the wider digital inclusion system, for example through growing New Zealand’s understanding of digital inclusion, developing and implementing standards and frameworks to support digital inclusion or making connections between other initiatives (these activities are important, but are not a priority for the evaluation of digital inclusion).
Initiatives can focus on digital inclusion alongside other outcomes
Digital inclusion initiatives can (and usually do) focus on other outcomes alongside digital inclusion. For example, many digital inclusion initiatives also have education, employment or other social goals. This is appropriate, as digital inclusion is an enabler of outcomes in other areas, and because research shows that engagement in digital inclusion is better when people are ‘hooked in’ through a personal interest or when digital inclusion initiatives are embedded within other services.[Footnote 3][Footnote 4]
Digital inclusion policy, funding and evaluation must allow for this. Support for digital inclusion must be flexible enough to allow initiatives to work towards, report on, and evaluate digital inclusion alongside other key goals.
What is the current state of digital inclusion initiatives?
Four main types of initiatives can be distinguished based on the digital inclusion elements they address and the groups they reach.
The digital inclusion team is developing a stocktake of government and non-government digital inclusion initiatives. The stocktake attempts to list all digital inclusion initiatives in New Zealand, and gathers information on initiatives’ characteristics such as size, purpose and the groups they work with.
The stocktake has identified more than 60 currently active New Zealand services, projects or programmes that fit our definition for digital inclusion initiatives.
Of these, over 90% can be classified into 4 types, based on the digital inclusion elements they address and the groups of people they reach.
Most New Zealand digital inclusion initiatives haven’t been evaluated
Through the stocktake of digital inclusion initiatives, we found that around 20% of digital inclusion initiatives have been formally evaluated, or have a future evaluation planned. Some of the remainder have monitoring in place that may support future evaluation. Among the formal evaluations that we found, we saw very little consistency in the digital inclusion-related outcomes that have been measured.
Interviews with key people (Appendix 1) confirmed that little formal evaluation has been done, and that we lack a shared understanding about what digital inclusion outcomes we should measure and how we should measure them.
Eight main challenges with evaluating digital inclusion
Our stocktake of New Zealand digital inclusion initiatives showed that many different organisations are delivering digital inclusion initiatives in New Zealand, and that formal evaluation is rarely done and, when it is done, methods are not consistent across initiatives.
Drawing from the stocktake and 15 interviews with key people (Appendix 1), we identified 8 main challenges with evaluating digital inclusion initiatives.
Challenge 1: Diversity among initiatives makes evaluation consistency very difficult
There is a great deal of diversity across digital inclusion initiatives. For example, initiatives that facilitate connectivity for everyone are very different to skills training courses. Different types of initiatives require different evaluation methods and measures, and their evaluation findings will only rarely be directly comparable.
Challenge 2: We lack a shared understanding of how to measure digital inclusion outcomes
Among the evaluations of New Zealand digital inclusion initiatives, we found almost no consistency in the digital inclusion outcomes that were measured, even where initiatives were similar enough that there could have been consistency. Several key interviewees commented that New Zealand lacks an agreed set of digital inclusion outcomes, and that they would like advice on what outcomes to measure and how to measure them.
This differs from some other areas. For example, standard measures of various education and health outcomes exist and are commonly used in evaluation.
Challenge 3: For some digital inclusion initiatives, it’s very difficult to measure outcomes and to determine the initiative’s contribution
The following difficulties with evaluating outcomes were described by key interviewees.
- It’s difficult to track longer term outcomes among participants, especially when they're transient and reluctant to trust outsiders. For example, this has made it hard to measure educational and employment outcomes among participants in ‘connectivity and skills for low income families with children’ initiatives. The low-income groups that these initiatives target can be highly transient and reluctant to participate in surveys and evaluation.
- Some initiatives cannot identify participants, making it hard to measure anything about them. This applies to the ‘building online trust’ initiatives and to most of the ‘connectivity for everyone’ initiatives, which often have no built-in way to find out who they’re reaching or what behavioural changes are happening among the people they reach.
- Wellbeing outcomes for individuals and communities have multiple contributing causes. Isolating the effect of an initiative from other factors is difficult. This applies to all types of digital inclusion initiatives and is a very common challenge for evaluation more generally.
Challenge 4: Providers of digital inclusion initiatives lack the resources to carry out evaluation
Key interviewees described major challenges with funding resources for evaluation among community and government providers of digital inclusion initiatives. Many providers:
- lack evaluation capability
- are so busy delivering core services that they cannot find time to do evaluation
- do not receive funding for evaluation (and are under-resourced for administration generally)
- are funded by multiple small grants, creating administrative inefficiencies and resulting in a situation where no 1 grant is large enough to explicitly support evaluation.
Challenge 5: There may be insufficient support for scaling up successful digital inclusion initiatives, which can discourage evaluation
Scale is important for evaluation because larger-scale initiatives can more easily find resources for evaluation and embed evaluation into standard processes. Likewise, evaluation is important for scaling up because evaluation findings can support the case to do so, providing evidence of success and an understanding of the critical factors that should be retained as the initiative grows.
Several key interviewees said that New Zealand lacks long-term funding for digital inclusion initiatives and does not support successful initiatives to scale up.
Through the stocktake, we found at least 10 initiatives that have been successfully rolled out across multiple locations, suggesting that we have supported some scaling up. However, the stocktake also indicated a high turnover of initiatives, with around 10% having ceased operation since the stocktake was first drafted in early 2018. While it’s possible that some initiatives ceased because they were unsuccessful, one key interviewee pointed to some successful initiatives that had stopped because the people running them had ‘burnt out’ from the stress of insecure and short-term funding.
Challenge 6: Conventional perceptions of how social outcomes are achieved do not consider the contribution of digital inclusio
Key interviewees suggested that some digital inclusion initiatives have struggled to retain funding because they are not thought to directly affect the outcomes that government agencies traditionally focus on (such as health, education or employment). Digital inclusion initiatives are at risk of falling between agency siloes, even though there is evidence suggesting that they can facilitate achievement of outcomes in many established areas.
A 2015 evaluation of Computers in Homes found that:
“… the benefits of digital inclusion impact on the outcomes of several agencies including the Ministry of Business, Innovation and Employment, Department of Internal Affairs, Ministry of Social Development and Ministry of Education. It is an archetypal case of an intervention at risk of being orphaned because it is not the priority of any one particular agency, but that has the potential to strongly contribute to whole-of-government outcomes.”[Footnote 5]
This emphasises the need to develop a shared understanding of how digital inclusion affects social outcomes, to help initiatives to demonstrate their value.
Challenge 7: The influence of evaluation on funding decisions has lacked transparency, reducing providers’ motivation to evaluate
There is scepticism about the value of formal evaluation among some providers of digital inclusion initiatives. This scepticism is in part based on their experiences with particular funding decisions that either did not take account of evaluation findings or lacked transparency in how they did so.
Challenge 8: There may be a lack of knowledge about evaluation
Some key interviewees suggested that providers may not have a good understanding of how evaluation can contribute to initiative improvement.
Is the current state good enough?
We need improved evaluation to support increased funding, intelligent scaling up and better outcomes.
As described in the section Most New Zealand digital inclusion initiatives haven’t been evaluated, there has been little formal evaluation of New Zealand digital inclusion initiatives. We could encourage more and better evaluation by addressing the challenges with evaluation capability, knowledge, consistency and motivation. This will need support, as the challenges are long-standing and are unlikely to be resolved on their own.
Or we could continue with the status quo, but this would have the following drawbacks.
- We would struggle to make a good case for increasing funding for digital inclusion. More government funding for digital inclusion would almost certainly require a budget bid, and evaluation would be needed to support that. Budget initiative submissions must present a well-evidenced analysis of how the initiative will benefit wellbeing, a strong intervention logic, and a plan for monitoring and evaluation.[Footnote 6] We cannot yet meet these requirements.
- We will continue to have very little evidence to support decisions about which initiatives should be scaled up. If government intends to fund digital inclusion more extensively, we will need evidence on which initiatives are ready to grow, which will create the most beneficial outcomes, and which are suitable for different groups.
- We are missing opportunities to improve initiatives. While most providers have feedback mechanisms in place to assist service improvement, better evaluation capability would supplement this.
Actions to improve the evaluation of digital inclusion initiatives
There are 4 areas in which actions could be taken to address the evaluation challenges. In this section, we describe each action in detail and note the challenges that may be addressed by these actions.
Appendix 1. Interviews with key people
Fifteen interviews were conducted with key people in organisations that fund, carry out or develop policy related to digital inclusion initiatives. The interviewers asked:
- about the digital inclusion-related work that they did, and any existing monitoring and evaluation
- how they felt evaluation could add value
- what challenges make it difficult to evaluate digital inclusion initiatives
- what kinds of evaluation advice, resources or guidance they would find helpful
- whether they could identify good examples of evaluation of digital inclusion.
The key interviewees were from:
- Policy Regulation and Communities, DIA
- Aotearoa People’s Network Kaharoa (APNK), DIA
- Office of the National Librarian, DIA
- Community Operations (Hāpai Hapori), DIA
- Digital Strategy - Equitable Access, Ministry of Education
- Channels team, Ministry of Education
- Digital Economy team, Ministry of Business, Innovation and Employment (MBIE)
- Infrastructure team, MBIE
- CERT NZ, MBIE
- Ka Hao, Te Puni Kōkiri
- Digital Inclusion Alliance Aotearoa
- 20/20 Trust
- Netsafe
- InternetNZ.
In addition, feedback on early findings was sought from members of the Digital Inclusion sub-group of the Digital Economy and Digital Inclusion Ministerial Advisory Group (DEDIMAG) and the digital inclusion team at DIA.
Appendix 2. Key features of the Superu Evidence Rating Scale
Level 1: Pilot and early stage initiatives
Evidence standards required for funding
There is:
- strong theory of change (or logic model) based on evidence
- an evaluation plan.
Level 2: Small to medium sized initiatives that have been operating for around 1 to 3 years
Evidence standards required for funding
- There is reported information about efficiency (delivery of outputs relative to inputs).
- It has been evaluated at least once, showing some beneficial effects. The evaluation:
- used a convincing method to measure change, such as pre- and post-analysis, or a recognised qualitative method
- used valid, reliable and appropriate methods
- analysed data appropriately and presents conclusions supported by evidence.
- Documentation and procedures provide clarity on how the initiative is implemented and the resources required to deliver it.
Level 3: Medium to large initiatives that have been operating for around 3 to 10 years
Evidence standards required for funding
- There is reported information about efficiency (delivery of outputs relative to inputs).
- It has been evaluated at least once, showing convincing evidence of beneficial effects. The evaluation:
- measured change using pre- and post-analysis of outcomes
- investigated attribution of outcomes to the initiative using a comparison group or other appropriate data, ideally with long-term follow-up
- used other valid methods to examine attribution if it is impossible or extremely difficult to obtain comparison data
- presents good evidence that intermediate outcomes predict long term outcomes, if it is not possible or extremely difficult to do long-term follow-up
- used valid, reliable and appropriate methods
- analysed data appropriately and presents conclusions supported by evidence.
- There is an assessment of the cost of the initiative relative to its impacts.
- There is evidence that shows how and why the initiative leads to outcomes.
- Documentation and procedures provide clarity on how the initiative is implemented and the resources required to deliver it.
- There is regular review of procedures, manuals and staff training processes.
Level 4: Very large initiatives that have been operating for around 8 years or longer
Evidence standards required for funding
- There is reported information about efficiency (delivery of outputs relative to inputs).
- At least 2 evaluations show convincing evidence of beneficial effects. They:
- measured change using pre- and post-analysis of outcomes
- investigated attribution of outcomes to the initiative using a comparison group or other appropriate data, ideally with long-term follow-up
- used other valid methods to examine attribution if it is impossible or extremely difficult to obtain comparison data
- presented good evidence that intermediate outcomes predict long-term outcomes, if it is not possible or very difficult to do long-term follow-up
- used valid, reliable and appropriate methods
- analysed data appropriately and presented conclusions supported by evidence.
- At least 1 cost-benefit analysis completed, using methods that meet established standards.
- There is evidence that shows how and why the initiative leads to outcomes.
- There is evidence about which elements of the initiative are necessary to implement with fidelity, and which can be adapted (e.g. to local conditions).
- There is evidence of the impact of the initiative on different sub-groups in the target population, for example, outcomes for different ages, ethnicities, genders.
- There is evidence that the initiative is consistently delivered as planned and reaches its target groups.
- Documentation and procedures provide clarity on how the initiative is implemented and the resources required to deliver it.
- There is regular review of procedures, manuals and staff training processes.
- Technical support is available to help implement the initiative in new settings.
More detail on each standard is given in Superu (2017).[Footnote 35]
Last updated