top of page

Evaluation

Health charity evaluation • Digital health • Theory of change • Learning framework

Understanding the change created by a digital health service, and helping it evolve

This mixed-methods, developmental evaluation helped the Royal Osteoporosis Society understand BoneMed Online’s emerging impact and improve the digital service in its first year.

The challenge

The Royal Osteoporosis Society developed BoneMed Online to support people with osteoporosis who were at high risk of fracture to understand and manage their medication more confidently.

People register with the digital service, answer questions about their experiences and concerns, and receive a personalised summary based on their responses. This is followed by tailored communications over the course of a year.

The service aims to address some of the behavioural and practical factors that can make it difficult for people to take osteoporosis medication consistently, including concerns about side effects, uncertainty about the benefits of treatment and challenges incorporating medication into everyday life.

Its longer-term ambition is to support better medication adherence, reduce avoidable fractures, improve health outcomes and relieve pressure on the wider health system.

The Royal Osteoporosis Society needed to understand whether and how the service was beginning to create this change during its first year.

This required an evaluation that went beyond registrations, website use or satisfaction scores. It needed to explore:
 

  • whether users found the information relevant, accessible and understandable;

  • how the service affected knowledge, confidence and feelings of empowerment;

  • whether it supported medication-taking behaviour and better conversations with healthcare professionals;

  • which elements of the service appeared to enable change;

  • how effectively professionals and Royal Osteoporosis Society services were signposting people to it;

  • who the digital service was and was not reaching;

  • and what could be learned during the pilot to improve the service and support its future development.


The challenge was therefore twofold: to understand the emerging change and impact created by BoneMed Online, while generating timely learning that could help the digital service evolve as it was being delivered.

What we did

The Future Works - working with Linda Jackson of The Loom - designed and delivered a mixed-methods, developmental evaluation of BoneMed Online’s first year.

We treated BoneMed Online both as an intervention intended to create behavioural change and as an evolving digital product whose design, content and user journey could be improved through evidence.

Mapping how change was expected to happen:
We worked with the BoneMed team and its stakeholders to develop a Theory of Change and evaluation framework.

This clarified the pathway between the service’s activities and its intended outcomes - from receiving personalised, accessible information to increased understanding, confidence and empowerment; more informed conversations about medication; improved adherence; and, potentially, fewer fractures and reduced pressure on health services.

Importantly, the Theory of Change also surfaced the assumptions within this pathway. Receiving information does not automatically lead to behaviour change. Medication-taking is influenced by people’s beliefs about risk and treatment, concerns about side effects, previous experiences, confidence, practical circumstances and relationships with healthcare professionals.

The evaluation therefore needed to examine not only whether outcomes occurred, but the mechanisms and conditions through which BoneMed Online might contribute to them.

Combining quantitative and qualitative evidence:
We developed a mixed-methods approach bringing together:
 

  • service reach, registration and usage data;

  • information about how users found and entered the service;

  • a survey completed after four months of using BoneMed Online;

  • qualitative interviews with service users;

  • interviews with Royal Osteoporosis Society helpline nurses;

  • engagement with pharmacists, doctors and Fracture Liaison Service staff;

  • and insight from volunteers, healthcare professionals and the internal BoneMed team.


The quantitative evidence enabled us to examine reach, take-up, user experience and reported outcomes at scale. The qualitative research helped explain the numbers: how people experienced the service, why particular elements were useful, where they encountered friction and what else influenced their medication decisions.

Where data protection constraints limited direct sampling by protected characteristics, we worked with volunteers and intermediaries to support recruitment of a more diverse range of service users. This helped the evaluation consider digital inclusion and accessibility for older people, people with limited digital confidence and communities that could otherwise be underrepresented.

Building learning into delivery:
Rather than waiting until the end of the pilot to produce a retrospective judgement, we organised the evaluation around quarterly learning cycles.

Evidence from monitoring, surveys and interviews was regularly brought together and shared with the Royal Osteoporosis Society team. Facilitated sense-making enabled the team to explore what the evidence meant, identify emerging issues and decide where changes to the digital service or its delivery should be considered.

This made evaluation part of the service-development process, not simply an assessment conducted after the work had finished.

What changed

The evaluation gave the Royal Osteoporosis Society a much clearer picture of BoneMed Online’s reach, user experience and emerging contribution to change. It also generated practical learning about how the service could develop. Priorities included:
 

  • strengthening personalisation so that information felt more directly relevant;

  • aligning messages more closely with different points in the medication and treatment journey;

  • improving signposting into and through the service;

  • continuing to strengthen accessibility and digital inclusion;

  • and understanding more about how professional recommendation affected trust and take-up.


These insights enabled the Royal Osteoporosis Society to consider improvements during delivery, rather than waiting for a final report before acting.

The findings offered promising evidence of reach, user satisfaction, self-reported medication behaviour and emerging short-term outcomes.

Our 'special sauce' on this project

Digital-service evaluation can easily become an analysis of registrations, clicks and satisfaction. Those measures tell you whether people used something. They do not necessarily tell you whether or why it changed anything.

Our contribution came from combining four areas of expertise.

1. Behaviour change knowledge. We understood that medication adherence is not simply an information problem. People’s behaviour is influenced by confidence, beliefs, perceived risk, side-effect concerns, habits, practical circumstances, trust and interactions with healthcare professionals. This enabled us to examine the mechanisms behind change rather than assuming that providing information would automatically improve adherence.

2. Evaluation and research expertise. We combined quantitative data with qualitative research to understand both scale and experience. The quantitative evidence showed patterns in reach, engagement and reported outcomes; interviews helped explain why those patterns might exist and where the service could improve. A theory-based approach also allowed us to distinguish emerging contribution from unsupported claims of causation.

3. Stakeholder engagement and facilitation. BoneMed Online sits within a wider ecosystem involving users, volunteers, helpline nurses, pharmacists, doctors, Fracture Liaison Services and the internal product team. We brought these perspectives together and facilitated collective sense-making, helping the organisation translate varied evidence into practical decisions.

4. Developmental and product-learning expertise. We designed the evaluation to support improvement, not merely judgement. Quarterly learning cycles meant that evidence could inform personalisation, accessibility, signposting and the user journey while the service was still evolving.

An internal product team inevitably holds assumptions about how its service works and is invested in its success. Independent evaluation brought constructive challenge, methodological rigour and a more neutral interpretation of the evidence.

bottom of page