Transforming Civic Engagement with Caddy

Here at i.AI, we see our job as more than just building AI in government: we’re committed to the power of AI to do good, and so are always looking for new ways AI can improve the lives of citizens everywhere. That’s why last year, we started thinking about contact centres… and after a few months of work with our friends at Citizens Advice, built Caddy, our copilot for front-line contact centres that is already helping support advisors all around the country.

Picture a contact centre, and you might associate that more with upgrading your phone contract than reaching out to the government for help. But contact centres are a core part of government for a whole range of departments: whether you're looking to file your taxes, find subsidies for your farm, or renew your visa, you might be one of the millions of people who find themselves waiting to speak to a public servant for help or advice every month. If AI can help ensure those reaching out to the government for help have an excellent experience, those benefits can be felt across those swathes of services.

The good news is that we know AI can probably help improve customer service work: robust research has shown improvements of up to 34% for the number of issues resolved per hour, although the effects were most important for novice call handlers. In government, these interactions can be especially challenging: support staff might have to provide guidance on new or emerging legislation, with specialties ranging from legal advice to import tariffs. And of course, people rarely reach out to ask government for help when things are going well: support staff often have to answer challenging questions, in what might be the most difficult moment in people's lives.

Building Caddy with CASORT

Government isn't alone in providing help to people when they need it most. Since its creation in 1939, Citizens Advice has been providing invaluable support and impartial advice to people all over Britain: four in ten of the British population contact Citizens Advice at some point during their lives. When we heard Citizens Advice Stockport, Oldham, Rochdale & Trafford (CASORT) had already built and started testing a prototype tool to support their  advisors using AI, we reached out to learn more. In the months that followed, we worked closely with CASORT, along with volunteer developers from across-government, to expand and build on their original prototype, which we've now affectionately come to call Caddy.

We've collaborated closely with CASORT every step of the way, and it's been so exciting to work directly with advisers and digital teams to build a tool that really works to improve service. Together, we think we've built something really special.

Caddy will always be bespoke, custom, and focused to specific public services. Caddy isn't just a small add-on for the language models you might already be used to: we've worked with Citizens Advice to ensure it has access to a whole swathe of public and internal training and guidance, that is constantly kept up to date to make sure its advice is as current, and accurate, as it can be. The customisation also goes beyond just documentation: we use a technique called semantic routing to make sure that when you ask Caddy about a specific topic, it specifically considers factors that experienced advisers trained it to raise.

We also built Caddy to always provide safe, validated answers. The original CASORT prototype built on the concept of always passing generated content for a supervisor to verify, and we’ve kept that innovation core to Caddy.Generated content is never presented directly to the public: instead, it will be reviewed, assessed, and validated by at least one adviser and a supervisor to ensure the answer is correct. By building in these "humans in the loop", we're trying to mitigate AI hallucination and risk.

Finally, we also built Caddy to be as easy as possible to use. We know our front-line colleagues are busy enough, and wanted to make sure Caddy didn’t need them to get used to a fancy user interface or need extensive training: you message Caddy for help on your existing chat platform, just like you would a supervisor or a colleague, and Caddy will take it from there. If you’re a new user in a contact centre, Caddy should be there to take work off your plate, not add to it, so you can focus on what really matters: supporting citizens and providing the best, most high quality government services in the world.

Evaluation and impact

Beyond just building AI, in i.AI we are committed to measuring the impact of our work. That's why, from the moment we started designing Caddy, we built evaluation into it, in the form of a randomised control trial (or A/B test). Advisers were randomly assigned invisibly every time they asked Caddy for help, with the request for help either going straight to Caddy or notifying a supervisor (as they would have done before Caddy).

We also built data collection into the same process: at the end of every single call (whether randomly assigned to use Caddy or not), advisers were surveyed directly in chat about their experiences, and whether or not they'd resolved issues raised by the customer. Over the following three months, over a thousand calls for support were randomly assigned to support through Caddy.

While our trial wasn't without issues (we'll discuss those in a follow-up blog), our initial results are really exciting. We found Caddy generated high quality, detailed messages: 80% of responses were of a high enough quality to be passed directly to advisors without revisions. More than that, Caddy was fast: messages were generated, validated and returned to advisors in around 4 minutes - around half the time it would take most humans just to type out the 400 or so word answers, let alone correctly research them.

Finally, our surveys suggest Caddy was having a real impact on the confidence and quality of service advisors were delivering: advisors that had access to Caddy were more than twice as likely to say they were confident giving advice than those in the control group,  and more than one-and-half times as likely to say they were able to resolve the issues being raised by the client.

Next steps

We're really excited by these initial results. While not perfect, they suggest that Caddy works, producing quick, high-quality answers that seem to improve advisers' confidence and resolve public queries. That's why in the coming months, we're going to continue rolling out Caddy and looking for opportunities to evaluate and build our evidence base for how and why AI works in improving public services.

Our partners in Citizens Advice have already rolled out Caddy to 6 local offices, with plenty more to come - that means if you ask Citizens Advice for help, your adviser might just be getting help from Caddy! We’re incredibly grateful for the hard work and dedication of the Citizens Advice national and SORT teams, who we’re continuing to work closely with to make Caddy the very best experience it can be.

In the meantime, we're looking to deploy Caddy in the government itself. In the coming weeks, we'll be piloting Caddy in a range of government departments, talking to partners to find the very best use case to help Caddy transform public services. And of course, Caddy is just one of the many projects we're continuing to build and evaluate. We're looking forward to sharing how we do that more in the future!