12 Ways to Measure EdTech Efficacy

Prove your impact quickly and easily to win more sales

Efficacy. It’s the growing talking point in the world of EdTech. That’s not to say that it’s only just come into vogue. No. Efficacy has been championed for some time by people across the sector, but what’s bringing it into focus more recently is the growing influence of MATs.

While everyone in education has known that technology should deliver impact, the problem in UK education is that EdTech has rarely had a dedicated school ‘owner’ - an individual responsible for implementation, training and ultimately, the transformational impact that EdTech can have.

🚀 The Growing Capabilities of MATs

The UK EdTech landscape is changing fast, predominately driven by the growing capabilities of medium and large Multi-Academy Trusts. As Trusts reach scale, they’re not only creating brand new roles, they’re also bringing in subject matter experts.

In Trusts with 20+ schools, it’s now commonly to see:

  • Chief Technology Officer

  • Chief Digital Officer

  • Chief Transformation Officer

  • Chief Information Officer

  • Technology Support Officers

The result is that larger Trusts are becoming more selective in what is used in their classrooms, and the key consideration in that decision making process is the efficacy of the EdTech.

👊 Say Hello to the Contentious World of Research

5 minutes on Google and you’ll quickly find that there is no ‘right’ way to measure efficacy. It’s highly contentious and there numerous schools of thought.

Everyone’s favourite cartoon scientist

Rather than review the arguments, here’s the simplified version: Research calls for highly stable conditions; a repeatable environment that minimises the interference of external factors in any experiment.

For anyone who’s been a teacher you’ll know that even if you’ve got top set year 12, you’re in for a different experience purely on whether you’ve teach them before or after lunch.

What works in one classroom won’t produce identical results in another, even within the same school. When you’re dealing with as diverse a society and country as England, making peace that there is no ‘right’ way to measure efficacy will make your life easier.

🎯 Giving Schools the Efficacy they Want

If there’s no ‘right’ way to measure efficacy, how do we even get started?

First, let’s rule out what schools don’t want; research papers.

Many people who write about EdTech efficacy champion Randomised Control Trials (RCTs), but most of these people are full time researchers or involved at a government level.

When you’re selecting a product for a country, you need strong evidence to make a compelling recommendation to ministers. These can be multi million pound contracts which leads to a natural focus towards RCTs. However, a 2019 analysis of 141 RCT’s found that 40% produced no impact at all.

When 40% of RCT’s go wrong…

In the highly fragmented UK school system, I’m yet to meet a school or Trust who wants to read RCT write ups. It’s overkill.

What I do hear from UK schools is that they’re fed up of overzealous marketing claims and want company’s to back up their statements.

You better!

Here are 12 areas you can begin to gather evidence in to demonstrate your efficacy:

  1. Learning Outcomes: The most direct measure is to assess changes in student learning outcomes. This could be through test scores, progress gains (faster progress than expected), or other assessments that directly relate to the curriculum.

  2. Engagement Metrics: Track how students and teachers are engaging with the product. This includes usage statistics like log-in frequency, time spent on the platform, and completion rates of tasks or lessons.

  3. Feedback from Users: Collect feedback from both students and teachers using surveys, interviews, or focus groups. This feedback can provide insights into the product's usability, effectiveness, and areas needing improvement.

  4. Comparison Studies: Compare the performance of students who use the EdTech product with those who don’t, controlling for other variables. This can help in understanding the product's impact in a controlled environment.

  5. Teacher Effectiveness: Evaluate how the product affects teacher performance and efficiency. Does it save time, help in better lesson planning, or provide useful analytics for student performance?

  6. Retention and Progression: Analyse student retention rates and progression through courses or grades. A successful EdTech product should ideally contribute to higher retention and successful progression.

  7. Integration with Curriculum: Assess how well the product integrates with the existing curriculum and teaching practices. It should complement and enhance the current educational framework rather than disrupt it.

  8. Cost-Benefit Analysis: Consider the financial investment in the product versus the educational returns. This includes looking at both short-term and long-term benefits.

  9. Scalability and Flexibility: Evaluate whether the product can be scaled and adapted to different classes, grades, or schools, and how easily it can be updated or modified as needs change.

  10. Data Analytics and Reporting: Utilise the product's built-in analytics to track progress and identify trends in student performance, engagement, and other key metrics.

  11. Professional Development and Support: Consider the level of professional development and support provided to teachers and admins to effectively implement and use the product.

  12. Compliance and Privacy: Ensure the product complies with educational standards and regulations, especially regarding student data privacy and security.

👷‍♀️ How to Start Building Your Evidence Base

There’s a good research paper (Understanding Evidence - Driven Digital Learning: A Guide for EdTech Producers) that does a good job of demystifying EdTech evidence.

I recommend focusing on what Kucirkova calls Foundational and Formative evidence. Summative, in my opinion, is overkill for the UK EdTech market.

The key aspect of the paper is that you want to build a portfolio of evidence with various sources that are both qualitative and quantitative.

  • Foundation Evidence = Surveys and interviews

  • Formative Evidence = Observational studies and case studies

🏅 3 Steps to Building a Great Evidence Base

Before we start let me talk about case studies. Case studies have become so ubiquitous (80%+ of EdTechs report that they have more than 3 case studies) that I don’t class them as evidence any longer.

They’re simply a requirement to operate in the school market and every company should have them. For that reason I haven’t include them in the 3 steps.

Step 1: NPS or CSAT

Many companies can’t easily capture usage data, or don’t have compelling usage data to share. If you do, you’re ahead of the game but usage aside, there’s no barrier to any EdTech tracking NPS. You can read up more on the benefits here, but in my travels far and wide across the sector, I’d say less than 5% of companies track this.

Of all the things you can try, it’s the biggest bang for buck thing you can do.

An easy way to set yourself apart is to simply have something your competitors don’t. While most companies race to costly build features, the smart play is to capture NPS that helps prove your efficacy. Generally it’s free (or very low cost) and is an absolutely gold mine of intel.

Step 2: Online Reviews

Google, Trustpilot and Reviews.io are your best options to start capturing public reviews.

Be mindful of the difference between review sites and marketplaces. Many marketplaces enable you to capture reviews, however marketplace business models are built around sales / generating leads. This is not a bad thing, it’s simply a different initiative and don’t confuse the 2. Marketplaces are marketing / lead generation initiatives.

👉 Joining a Marketplace? With new marketplaces popping up every year, here’s a few tips to help you decide whether it will be worth it:

  • Ask to see evidence that it works. Anyone can create a ‘product directory.’ It’s very difficult to get the reach with schools and to stimulate buying demand.

  • Ask to see case studies. Remember, these should be standard in the sector. If the marketplace does not have any, move on.

  • Ask to speak to a few companies who’ve experienced continued benefits for a long time. If the marketplace works, this shouldn’t be an issue.

  • If they have a school or Trust endorsing them, ask about the number of sales made into it that institution. Is the person saying ‘efficacy is a good idea’ or are they actually using the platform to buy for their school?

Google Reviews is completely free, and easy to get started with, however, because it’s connected to a email address, you may find schools aren’t that keen to use it.

Trustpilot and Reviews.io both have great freemium options and are ever popular.

If your solution is a step change from how schools do things today, then you’re going to need a lot of social proof to convince them to change. Schools and Trusts on the whole don’t like ‘transformative’ solutions because it means large scale change. That means risk and complexity, the 2 things schools are in the business of avoiding.

Last year I worked with Mable Therapy, an EdTech that provides online counselling and SALT sessions into schools. There were numerous benefits for schools including better data and big cost savings (the familiar duo!), but schools had traditionally paid a therapist to visit half a day each week, so moving to online only, despite the many benefits, was a big leap for them.

To social norm this change, we gathered as much evidence as we could showcasing that this wasn’t some cutting edge, super difficult thing to try. It was actually being used by hundreds of schools across the country.

Not scary. Just normal.

School revenue growth went from 21% YoY to 73%.

Building a strong evidence base for children’s SALT and counselling services

Step 3: Working with a Researcher

If your goal is to deliver educational impact or accelerated progression, working with a researcher to develop a professional study can add an immense amount of value.

The right partner will scope, design, and implement the study, so you don’t need to become an academic expert to get started. The National Foundation for Educational Research (NFER) is a good place to start as well as UCL’s Education Research Programme. Others include:

Cost vary widely as it depends on the complexity of the study and duration it will last. Generally, the longer the study, the more accurate and powerful the findings will be. In general, £10,000 is a good ballpark figure to think about when thinking about commissioning a study.

For some this might seem like a huge sum, but the power of having a professional study is a game changer for your marketing and sales. You simply blow your competition out the water because very few companies have this level of evidence so you’re in the upper echeleons of efficacy. It’s no coincidence that many of the companies who have a study are market leading solutions in their category.

🧑‍🏫 Wrapping Up

That’s how you can build your evidence base quickly, and prove the efficacy of your product at little cost. Ultimately this is what schools and Trusts want, and once you’ve got your evidence, you can really turbo charge your marketing, and drive up your win rates.

If you’d like to improve how many leads you’re getting, and how fast you’re growing, we’re here to help.

I recently brought on board one of the most experienced EdTech Marketers I’ve met in the last 13 years to The EdTech Forum.

How we can help:

  • Improve your email marketing to generate more leads

  • Improve or even get started with digital marketing to generate more leads

  • Enable you to stand out and differentiate yourself from your competition

  • Support you in putting into action many of the things I’ve talked about in this blog

  • And lots more!

As always, thanks for reading!

Best

Jay

  • 🎓 New to EdTech Forum? Check out more articles here

  • 💌 Share with a friend. If you know someone who would like this article, forward it on.

  • 👋 Connect on LinkedIn. Join the daily conversations here or just say hi. It’s always nice to connect