Isn’t prevention a bit of luxury during a crisis? Even in the ‘good’ times prevention seems to be the poor relative, underinvested due to more immediate emergencies taking all the attention. During a pandemic, we have all become even more reactive and necessarily so. Every one of us changed our behaviour and so did our employers. This wasn’t solely in terms of going to the office, but operating our entire lives in a way that took many of us online more than we’d planned.
Whether you find yourself consuming your fitness, sourdough recipes or Saturday night drinks via the Internet, we have something in common, we are all doing things differently. Usually, we do all this face to face, currently, it’s virtual.
This has to mean the results change. In third sector speak, if you’re changing the inputs and outputs then you must be changing the outcomes and impact too – or the difference we set out to make. Just as during lockdown, some of us put on weight, lost weight, gained new Instagram followers or launched new services. Personally, I developed what I’m pretty sure was a caffeine rash!
All of this change to our inputs and outputs will continue to have a broader legacy that we’re yet to see. From more children going into care, to an impact on climate change, our use of office space, homelessness, unemployment, poverty, the high street and closer to home, the difference we make to people’s lives.
It is this issue of shifting outcomes that causes me almost greatest concern, second only to the subject of last week’s blog about keeping people safe. We can’t possibly still be achieving our previous outcomes if we’re doing it all so differently? Moreover, we are using different tools to measure the outcomes. How viable and attractive is this to our commissioners (and worth doing), if we’re not making the same difference?
The perfect is almost certainly the enemy of the good here and in a crisis it arguably matters less if we don’t achieve our outcomes. What commissioner in their right mind is going to haul us over the coals for falling short in 2020? This isn’t a long-term strategy for sustainability as a homelessness prevention organisation, however. As we look towards 2021, let alone reporting on impact for 2020, what will we have actually achieved that has anything to do with our mission?
If you measure it you treasure it!
As the founder of a social enterprise that is still evolving, I am utterly committed to measuring outcomes, (without beating myself up about the imperfection of the art). We have measured relentlessly since we incorporated in 2013 and will continue to do so. I remain stunned by the propensity in the sector to still ignore the importance of measuring the difference they make and yet make huge claims of the lives ‘transformed’. For me it’s the reason I get out of bed in the morning and not to do so pours fuel on the fire of those that wish to devalue the sector and seek to underfund us.
How we achieve our outcomes has evolved over the years and is in truth part art and part science. The art element has revealed itself to Your Own Place in greater colour in recent years, as we acknowledge the importance of our values (and therefore approaches) in achieving our outcomes. The true impact of our values are all but impossible to measure, but I never cease telling the story of those values and how they impact on our outcomes.
When inducting new staff, a key mantra is that you have no outcomes at all if someone doesn’t come back for the second day of training or a second mentor meeting. As such, the only real outcome required on day one is that of a good relationship – so that they come back on day two. Brene Brown has some really interesting things to say about day 2s and how hard they can be to achieve.
So, how has going online changed the inputs, outputs and outcomes? Do we know what the specific changes are that might impact on our ability to achieve our outcomes and how can we replicate them online?
What do we normally do?
In normal times we do manage and measure our impact quite well. Just as lockdown hit, we were completing an external review of our Managing Impact strategy, resulting in new Theories of Change, tools, reporting and review cycles.
Like many organisations, we use a range of imperfect tools and then endeavour not to over-report. Indeed, we probably underreport. I always fear the standard falling back on feel-good quotes and case-studies, when hard data is what’s needed to make strategic decisions. Sadly all too often the press, trustees behind trust funds and even some commissioners and funders still just want stories and exclude the stats (or the other way around). The reality is of course you need both. Having studied ‘What Works’ as part of my Professional Certificate in Effective Practice (PCEP – Youth Justice Board), I am passionate about doing better and using the evidence to move forward.
Measuring a counterfactual is all but impossible, because it’s hard to prove you prevented something from happening that hasn’t happened. That doesn’t stop us trying. Measuring the prevention of homelessness in the purest sense would involve tracking people over a lifetime (something I’d love to be able to do). By definition, people in transition can be hard to track down and as a team of seven we simply don’t have the resource. Like most in the sector we use a range of (semi-evidence-based) proxies that are determined in our Theories of Change.
Tools to collect qualitative as well as quantitative data and information include start and end questionnaires as well as longer-term follow-up impact questionnaires, reviews (for mentors and mentees), photos, case-studies, films, quotes, testimonials as well as a project log that we all update monthly to capture the learning and ascertain the ‘what worked’. We’re conscious of the ‘happy sheets’ that get circulated at the end of so much training, so prefer questionnaires that follow later and don’t ask if people enjoyed the lunch! All of this is collated for reporting, reviewing, marketing and referencing.
An important, often undervalued part of managing impact is of course the review cycle. It’s no good collecting all this great data if it just sits there. Thanks to some pro bono work via a funder this year, we have now contracted with Renaisi to analyse the data from our two main projects once a year. Already the learning has been huge and shows us the variance in outcomes between different demographic groups. This allows us to test changes by altering one variable at a time. The first we’re going to test is a change in our approach to start and end questionnaires.
Managing impact is a bit like unknown unknowns. I quite enjoyed knowing nothing, now I know a bit, I realise not just how much more there is we can do, but now have to confront doing it all virtually.
A lot happens on our courses and interventions, so it’s only natural as well as helpful that we collect as much data as we do and in different ways for different people as well as audiences. This is part of managing impact after all – understanding its different functions – from marketing to reporting to improving. As we collect quantitative and qualitative data, keep logs and take footage as we go, this poses significant challenges in how we do this online and stay true to our values and approaches.
With changes so fundamental, how can we possibly replicate the outcomes? And do we have enough handle on the changing needs and if not, should our outcomes be changing too? Moreover, how much priority should digital outcomes now have? More fundamentally, is our service still the right one to prevent homelessness?
A couple of years ago, I sat down with Jarrod, our first TILS+ trainer, and we analysed all the things we defined as ‘core’ and ‘not core’ to our delivery. Like many founders who design in their own image, Jarrod’s fresh perspective enabled us to codify what we thought was core and not core. When we talk about core, it relates to building that all-important relationship with the people who we support. This means maximising engagement, retention and attendance whilst adhering to our values. This can literally be anything from shaking hands when people arrive, smiling, having their name on a entry board so they feel expected, providing cups of tea and decent refreshments that take their food requirements into account. At the start of a course we turn music on when people arrive, provide a unique and visually appealing learning space, make eye contact, show respect by finishing on time and putting our phones down, never put someone on the spot and always find the positive in the less than perfect answers. Never underestimate the impact of treating people decently and to do it consistently, it has to be codified. How achievable is all this now via Zoom?
When we return to face to face, and we will, all of this can be resumed. For now though we have a significant challenge.
With everything we do, the perfect is the enemy of the good enough or even the done. I think it’s fair to say that I sometimes take this too far. I’ve a team of perfectionists (they do say surround yourself with people better than you), but sometimes, and I would argue that now is that time, you just have to wing it a bit.
My colleague Simone, who runs our Mentoring Training and did an astonishing job of converting it into eMentoring Training, had just this anxiety as we went virtual. She finally articulated it in a way I understood, and unsurprisingly it came from her extraordinarily high levels of empathy. She said that she didn’t want mentors training online to become ‘second class citizens’. When Simone emerged from the first online course with tears of relief and joy (at least I think they were), we knew we were onto something in replicating not just what we do online, but how we make people feel.
We never abandoned our values of fun, interaction, innovation, high quality, equality, professionalism or being restorative. Finding ways of doing this over Zoom brings out the creative in everyone.
I’ve said before it’s not been all plain sailing – this has been hard on staff and they have felt the pressure. But it’s written all over everyone’s faces when it works.
Our solution was to maintain and adapt as many of the core features of our face to face delivery as possible and take them online as was feasible. Although you can’t make someone a cup of tea on Zoom you can welcome them with a smile, invite them to make their own cuppa and ensure they feel expected and welcome. Arguably, with life so tough right now, this matters more than ever. Just last week at the end of a tenancy training course via Zoom, a trainee told Jess ‘you’ve managed to make it feel like a big family and I felt brilliantly comfortable’.
In terms of measuring outcomes, again we stuck to the core principles when searching for the right online tools to make them work. There are unforeseen benefits to some of this and needless to say, some will stay with us way beyond lockdown. For example sending out questionnaires electronically in advance of a course is efficient and even acts as an engagement tool for conversation.
How do we measure?
We now use Google forms for evaluation, collect data from mentors on their meetings using Wufoo forms and send out start and end questionnaires electronically too. I’ve enjoyed using Menti for polls and our use of Canva as our presentation tool allows us to emulate pretty much any visual trick we like, for example using it for scaling statements with the annotate function.
Whilst we may all be a bit bored of the Zoom montage pictures on Instagram, as records of engagement, enjoyment and participation, they remain a valuable record. All of this has an impact on the time to design courses as well as support people to use the functionality, including purchasing and then installing hardware for people living in a large rural county. On the flip slide there are some savings on sending out virtual forms in place of paper ones (including environmentally).
We also chose delivery platforms that would most emulate our offer, style and values whilst creating essential new resources (Zoom and email guides)) where needed. We now offer ‘run through’ sessions for Zoom usage and functionality before workshops and training. We’ve even created a separate Google Drive so that when our mentors are undertaking their eMentoring Training in self-led sessions, they have a learning log that they can bring to the tutor-led sessions, which also then happens on Zoom.
In order to measure like for like, we have kept the same outcome measures across face to face and digital delivery. The inputs (staff time, digital platforms) and outputs (configuration of the delivery) have inevitably changed. For now the outcomes are largely sacrosanct and give us a fixed point to plan from (and there aren’t many of those right now!). Commissioners are still commissioning us to prevent homelessness and not only to increase digital resilience after all.
There are so many variables in the picture described above, from the delivery itself, the evaluation tools and the digital resilience of the trainees and mentees themselves, that it’s hard to imagine our outcomes not being affected. In not letting the perfect be the enemy of the good enough, our priority right now is that people are still receiving a service and we’re confident we’re still having a positive impact. With bigger datasets we can start to compare outcomes. There’s certainly a lot less paper and much more order in how we store things.
I haven’t yet determined whether the fact that we were in the midst of overhauling our Impact Strategy when lockdown hit was a help or a hindrance. That it got behind schedule is in no doubt, but it also meant that our heads were in the game.
Values and principles
Having a set of values and approaches that a team is on board with (if you measure it you treasure it) undoubtedly makes all the difference. Trusting them to get on with the ideas and then reigning in for detailed scrutiny and sense-checking is working increasingly well.
Our values are our principles that we go back to again and again. When I am muddled and struggling to find a solution, my go-to question is always ‘what is the desired outcome?’. It’s the same with every member of the team. It doesn’t matter if I’m preparing for a meeting, designing a Theory of Change or struggling with a social media post – the question is always the same. It’s important too to be honest with commissioners. We’ve asked them to trust us and let us try new things.
The big questions
Some big questions lie ahead. At Your Own Place we take an organic and embedded approach to digital skills – delivering them as part of the bigger programme, not a programme in of itself. They are delivered by a kind of osmosis. What will it take for commissioners who wish to prevent homelessness to pay for this and how can we do it if they don’t?
This raises some pretty fundamental questions for the sector in terms of regarding ‘digital’ as another overhead if it is to become business as usual and then having to deliver it and measure it. The likelihood is that it will change the impact we have and our organisations forever.
I hope you’ll join me for blog five when I’ll be exploring when, how or even if we return to full face to face delivery.
