Understanding the Return on Digital Skills Investment Starts with Assessment

The results of digital skills training are of growing interest to policymakers and other stakeholders. The National Telecommunications and Information Administration’s (NTIA) Notice of Funding Opportunity (NOFO) for Digital Equity Act (DEA) programs explicitly states that grantees must assess the outcomes of initiatives DEA may fund. That is not the only motivation for understanding the impacts of digital skills programs. States, counties, or local philanthropies that fund such initiatives will want to know their impacts.

The goal of this article is to explore what we know about the return on investment of digital opportunity programs—programs that provide computing devices and digital skills training to those with limited experience in using digital tools and the internet. It does so by highlighting one effort to assess this work in New York—specifically the work of Connect Westchester. Using data from Connect Westchester’s assessment of its services, the article then uses secondary research that characterizes the monetary impacts on people of changes to their lives. The discussion shows that there is a positive return on digital opportunity investments that are evident in potential income boosts for clients of up to five percent of their income over time. This finding is necessarily tentative, but by walking through it, there are lessons for the design of digital skills assessment in the future.

Background

Connect Westchester is funded by Westchester County, New York, and its mission is to provide digital training and tools to those who need it in the county. The STEM Alliance is the primary service provider, and the initiative operates in partnership with the Westchester County Association, a professional membership organization promoting regional economic growth. The Connect Westchester partnership began in 2023, although The STEM Alliance’s work in this area predates that. Thousands of Westchester County citizens have used its services, which include digital skills training, provision of computer devices. The STEM Alliance also provided assistance with free or low-cost internet through enrolling eligible households in the Affordable Connectivity Program when it was operational, providing hotspots, and referring clients to other low-cost internet plans.

In general, Connect Westchester clients had high digital needs: 92% do not own a personal device; 58% need basic digital skills training and 48% need assistance with affordable internet. In the first fifteen months of services, Connect Westchester assisted over 750 clients in the following ways:

• 593 received new Chromebooks and 15 hours of tech education.
• 177 enrolled in the Affordable Connectivity Program.
• 135 received hotspots.
• 82 received devices only because they already had basic skills.

Assessment approach

In the course of client intake, clients receive a “digital needs screener”—a series of questions that allows Connect Westchester to understand client needs and direct resources to them appropriately. This places them into the Connect Westchester client relationship portal, which allows the trained Digital Navigators to follow their needs, track the services they receive, and set the stage for follow-up surveys. This allows for longitudinal trend analysis, meaning that clients receive the same questions at two points in time. Time 1 is prior to the training intervention, and Time 2 is 3-6 months post-intervention, allowing for a comparison of responses. From this, it is possible to explore change over time.

What we found

I was privileged to work with the Connect Westchester team in the design of the survey clients received, as well as analyzing the data. The decision to conduct this assessment demonstrates a commendable commitment by Connect Westchester to an evidence-based understanding of its work.

The two animating questions in exploring impacts were:

• What do the clients get out of training?
• What does the county get out of a more digitally prepared population?

Client impacts

In general, the survey found that impacts on clients fell into two categories:

• Overall confidence in using computers and the internet.
• A strong sense that newly acquired digital skills broadly help improve their lives.

As to confidence in using digital tools, the findings are clearly positive when comparing data from before and after the 15-hour basic training. Approximately 65% of Connect Westchester participants who received training reported being very comfortable powering a computer on and off, a 22-point increase from their pre-training level. Following training, 54% of participants reported being very comfortable sending or receiving emails, compared to 33% who reported this prior to training. And 37% overall were very comfortable using a computer, up from 26% prior to training.

Clients’ sense of the positive results from digital skills training is another striking finding from the survey. For instance:

• 81% say training helped them manage their family schedules.
• 77% say digital skills will help “a lot” in learning new things.
• 52% say they will definitely use training to improve their tech skills to get a job.
• 52% say computer/internet skills training will help “a lot” in connecting with health care providers.
• 46% say they will “definitely” use training to have video doctors’ appointments.

Regional impacts

These clear findings about improved confidence and positive perceptions of the internet’s utility are the preconditions for digital skills and tools to have downstream impacts. The data provide every reason to believe that Connect Westchester’s service will have a material impact on clients’ lives, which in turn will strengthen the region’s economic and social fabric. By how much?

Answering that question begins by turning to other research on the impacts of connectivity, which offers insight into the monetary effects. Work I did several years ago on Comcast’s Internet Essentials (IE) Program aimed to assess the impacts of new internet users gaining service due to the $15 discounted service offered. That work rested on the following premises:

• The ideal way to assess outcomes from digital inclusion initiatives is anywhere from extremely difficult to impractical. One would need to have a set of research subjects who had received digital skills training, a similar control group of those who did not, and the ability to be very intrusive in tracking income or test scores over time. At a minimum, this research design would be cost-prohibitive for most digital skills programs.
• A reasonable “second best” approach relies on a growing body of “well-being” research, which employs surveys to ask respondents to rate their life on a ladder from 1 to 10. Some of this research demonstrates that people’s attitudes about their lives have relationships to material outcomes, such as income levels.

Well-being research builds upon questions asked by the Gallup Organization in worldwide surveys that ask people to rate their lives on a scale of 1 to 10 (that is, place themselves on a rung of a “life ladder”). This is a fairly straightforward question that can be added to assessment surveys, which may also cover programmatic details, client satisfaction levels, and self-reported perceptions of training impacts.

In the context of digital skills, a 2021 exploration of Comcast’s Internet Essentials (IE) program used the “well-being” measure in a survey that examined the program’s impact. Respondents were asked to rate their lives on a 1-10 scale, and statistical analysis revealed the relationship between “life ladder” responses and measures of digital skills. Those who have had formal digital skills training reported 0.5 points higher life satisfaction than those who have not (e.g., their rating on the ladder is 7.5 if they have had formal training and 7.0 if they have not). In the same study, clients who reported that the internet helps “a lot” in carrying out tasks—as the majority of Connect Westchester clients expressed—had a 0.9 point higher life satisfaction.

Important to the context of economic development is the connection between changes in people’s life satisfaction and economic gains. A body of research links income changes to levels of life satisfaction, with one study showing that improving one’s “life satisfaction” rating by 20% translates into a 5% income increase after seven years.

Bringing this study into the Connect Westchester context requires an assumption that the nature of the impacts of the digital inclusion work in Westchester is the same as those in the IE research. Using the 0.9 increase in life satisfaction from the IE research – an 11% boost up the ladder – the potential income increase is between $900 for a family of 3 at 125% of the federal poverty level (FPL) and $1,400 for a family of 3 at 200% of the FPL. This income increase plays out over several years.

One important limitation of this analysis is that it relies on several assumptions (i.e., that the effect of training for Connect Westchester mirrors that of Internet Essentials and that the research finding a 5% income boost is valid) to develop an estimate of monetary impact. Ideally, the Connect Westchester research instruments would have used the same life satisfaction rating scale before and after services. This lesson is part of the ongoing Connect Westchester initiative and is intended for the digital inclusion research community at large. Furthermore, it would be beneficial to have a more extensive body of research characterizing the income effect from changes in how people rate their life satisfaction.

However, the ideal need not be the enemy of the practical when looking to better understand the financial returns from digital inclusion programs. The analysis above indicates that individuals receiving digital skills training can expect a future income increase of $1,400 (at the upper end of the estimate). That increase is not going to be a “Year One” effect, as the 5% increase noted above unfolded over a seven-year period. But this increase is built on a durable foundation of newly acquired digital competencies and confidence, which can place beneficiaries on a higher lifetime income path. As long as the cost for this training per client is below that figure, the return on the investment from digital skills training programs is positive. And although the $1,400 figure may seem small, it adds up. In Westchester County, for instance, if Connect Westchester’s services touch 1,250 clients, that is up to $1.75 million in additional buying power from the income effect.

This estimate of the monetary return of digital skills programs understates their overall impact. As has been documented elsewhere, access to telehealth can result in significant cost savings for healthcare providers. Additionally, having access to online shopping benefits users by allowing them to find goods that better suit their needs and budgets. A recent study highlighted the range of benefits to fostering connectivity for low-income households, which include improved labor productivity and better academic outcomes – which in turn translate into higher earnings. The benefits of having the internet at home and the skills to use it, in other words, help make people more productive in the economy and in day-to-day life.

Conclusion

While tracking the financial return on investment of digital inclusion programs is challenging, it is possible. To be sure, establishing a causal link between the training sessions conducted today and changes in earnings six months or even a year later is a challenging methodological task. Inspecting people’s paychecks is too intrusive and asking for self-reported information on changes in income is not likely to elicit reliable answers.

That is why relying on the “life ladder” survey research approach can serve as a practical and executable way to explore return on investment. Establishing how a participants’ “life ladder” changed over time, in conjunction with the growing body of well-being research connecting life satisfaction to earning power, can provide valuable insight into the economic benefits of device ownership and digital skills training programs.

John B. Horrigan, PhD, is a Benton Senior Fellow. He is a national expert on technology adoption, digital inclusion, and evaluating the outcomes and impacts of programs designed to promote communications technology adoption and use. Horrigan served at the Federal Communications Commission as a member of the leadership team for the development of the National Broadband Plan. Additionally, as an Associate Director for Research at the Pew Research Center, he focused on libraries and their impact on communities, as well as technology adoption patterns and open government data.