Data-driven decision making: Birmingham City Observatory's journey to a single source of truth

In 2022, Birmingham City Council faced challenges familiar to many local authorities. Despite having access to vast amounts of data, the council lacked a centralised capability to understand the environment in which it operated and to make evidence-based decisions. After years of austerity, the council had reduced its central corporate capability, with the organisation focusing on delivery and directorates without adequate resources to understand the place they served.

This fragmentation of data and analytical capability presented significant challenges for service planning. Without robust analytical capacity, it became difficult to fully respond to Birmingham's changing demographics and diverse needs. The city had undergone substantial demographic changes over recent decades, as reflected in census data, creating a complex landscape that required nuanced understanding.

Meanwhile, partnership working suffered as different agencies - from the NHS to Police and Voluntary sector - each maintained their own separate understanding of communities needs without a common evidence base. For Birmingham, a city of 1.2 million people with complex demographic patterns and high levels of deprivation, this lack of shared insight was increasingly problematic.

The council launched the Birmingham City Observatory with a clear mission to create a single, publicly accessible source of information. Rather than producing insights solely for internal use, the Observatory was designed from the start as an open and transparent resource, publishing information that would simultaneously serve council officers, elected members, strategic partners, voluntary organisations, and the public.

The Observatory team deliberately avoided the traditional approach of commissioning insights for specific internal teams, which would then be buried in PowerPoint presentations and Excel spreadsheets. Instead, they focused on creating shared products that would benefit multiple stakeholders, maximising the impact of their work through radical transparency. This required strong political backing, as it represented a strategic choice about the use of limited resources.

Their first output wasn't a digital platform but printed ward profiles – physically handed to newly elected councillors as they received their election results. This tangible approach helped build awareness and demonstrated the Observatory's immediate value, particularly by visualising where each ward sat in terms of deprivation against the national average. People get used to their local context, and without this comparative data, many weren't aware of how their wards compared to national benchmarks.

The development of Birmingham City Observatory required a thoughtful and practical approach to team building, product development, stakeholder engagement, and partnership creation.

Building the right team proved essential. When recruiting for the Observatory, the council looked beyond its existing workforce, reaching out to local universities to find individuals with both the technical skills and the right mindset. They sought people who were naturally curious and had a genuine desire to understand and communicate insights, not just manipulate data. This selective approach to recruitment meant positions sometimes remained vacant until the right candidates emerged.

The team established a distinctive product development culture focused on continuous improvement. Regular review sessions examined works-in-progress, asking fundamental questions: What purpose does this serve? Who is the audience? Does it meet their needs? How can we make it better? This approach fostered psychological safety, encouraging team members to experiment, learn from mistakes, and focus on step-by-step improvement rather than perfection.

Transparency became a defining principle, though not without challenges. Making data openly available disrupted traditional patterns of information ownership and expertise. Some stakeholders initially resisted this openness, preferring to maintain their position as gatekeepers with privileged access to information. Overcoming this resistance required identifying champions and understanding the underlying motivations of different stakeholders.

From the beginning, the Observatory team recognised that different users needed information presented in different formats. Some wanted raw data they could analyse themselves, others preferred ready made insights with written interpretation, while a third group simply needed actionable bullet points. The platform evolved to accommodate all these needs, offering everything from downloadable datasets to visualisations, publications, and fact sheets.

The Observatory has cultivated a growing network of partnerships, beginning with Birmingham City University for data science expertise, and expanding to include Universities, the NHS, Police, Department for Work and Pensions, and Voluntary and Community organisations. These partnerships have been critical for expanding the platform's capabilities and ensuring its relevance to diverse users.

The City Observatory has transformed how Birmingham City Council uses data to inform decisions and improve services. By mapping need more effectively, the council has targeted Cost of Living support with greater precision, directing resources to areas and communities where they're most needed. One example involved mapping food support initiatives across the city, which revealed geographic gaps in provision. This insight enabled the council to work with faith and community organisations in underserved areas to develop more equitable access to essential support services.

The platform has facilitated better collaboration between previously siloed council departments. The housing team, for example, now shares data automatically, with weekly updates on council-owned housing stock. This openness has reduced duplicated work and created unexpected efficiencies, as teams discover they can use the Observatory instead of purchasing separate data platforms or commissioning external analysis.

The technical architecture of the platform itself delivers significant operational efficiencies. Approximately 90 per cent of data sources are harvested automatically, with the system regularly downloading and updating information from various data sources without manual intervention. This automation allows a small team of just three people to maintain an impressive range of data resources that serve the entire council and beyond. 

For smaller voluntary and community organisations, the Observatory has democratised access to high-quality data. Groups that would otherwise struggle to afford data analysis now have free insights to support their fundraising, service planning, and advocacy work, extending the platform's impact far beyond the council itself.

Perhaps most importantly, the Observatory has strengthened the council's ability to fulfil its statutory duties, particularly around equality impact assessments. By providing clear data on how different communities might be affected by policy changes, the platform helps ensure decisions are made with full awareness of their potential impacts across Birmingham's diverse population.

The Birmingham City Observatory demonstrates how local authorities can transform their approach to data and evidence-based decision-making. Its journey offers several valuable lessons for councils considering similar initiatives:

• Start with purpose, not technology: The Observatory began with a clear strategic intention – creating a shared understanding of the city – rather than a technical solution. This vision guided all subsequent decisions about tools, team, and processes.
• Build the right team with the right mindset: The Observatory's success was built on looking beyond existing workforce constraints, including partnering with local universities to find individuals with both technical skills and natural curiosity. Recruiting for mindset as well as technical ability helped create a team focused on generating insights, not just managing data.
• Build stakeholder engagement into your core processes: The most sophisticated data platform is worthless if people don't use it. The Observatory team continuously worked to understand different users' needs, create appropriate products, and actively engage with stakeholders at all levels.
• Create an environment where innovation can flourish: By establishing a culture that encouraged experimentation and learning rather than perfection, the team was able to develop and improve their offerings rapidly.
• Recognise that transparency can challenge existing power structures: Making data openly available changes who has access to information and can challenge established expertise. This requires sensitivity and persistence to overcome resistance.
• Invest in automation for sustainability: The Observatory's ability to automatically harvest and update data sources has created a sustainable model that doesn't rely on constant manual intervention.

While the Observatory represents significant progress for Birmingham, increasing data maturity and new ways of working is an ongoing journey to consider across a whole council and its partners. The Observatory has demonstrated the potential of how data can be used more effectively making geographic and demographic data accessible in a central location.  More sensitive operational data, such as that used in children's services, presents different challenges requiring tailored approaches to data governance, privacy, and integration, with the council working to consider how to extend good practice and learning into other areas.