A Call to Arms: Open Source Intelligence and Evidence Based Policymaking

Originally posted on King’s College London’s Policy Institute blog Policy Wonkers.

By Mick Endsor, Research Assistant and Dr Bill Peace, Visiting Senior Research Fellow at the International Centre for Security Analysis.

Policymakers have access to a wealth of open source information that has yet to be incorporated into the policymaking process. As Eliot Higgins argued in a previous post on this blog, social media provides a crucial and yet untapped source of evidence which can underpin effective policymaking in relation to conflict zones. We wholeheartedly agree with this – indeed, it can be taken much further. Policymaking based on information that has been acquired through open source intelligence techniques can provide a template for decision-making based on rigorous evidence and a verifiable methodology. In the business world, for example, OSINT is now common analytical practice. But as yet, it has not gained anything like the same kind of traction in the policy sphere, and there is a genuine lack of evidence-based policymaking based on open source intelligence.

From our perspective, this is a worrying state of affairs. There is a powerful case for incorporating OSINT approaches to evidence-based policymaking. In the first place, evidence produced by OSINT methods can be both robust and rigorous, not least because it can be underpinned by extensive datasets. And in the second, it has the potential to be both transparent and verifiable; all open source evidence is, by definition, based on data that is publicly (and often freely) available.

A powerful case indeed, then – so why is open source evidence rarely used to inform policymaking? At the heart of the problem is the fact that OSINT approaches are still relatively ‘young’ and, all too often in our experience, lack the rigour and reliability needed to underpin effective policymaking.

For us then, we need a call to arms – or, more accurately, three calls to arms. First, those who specialise in OSINT need to develop a much stronger, more reliable set of approaches for collecting open source data and, perhaps more importantly, for analysing that data. All too often, analyses rest on a set of assumptions that remain untested. One of the most obvious and frequent of these assumptions is that Twitter and other online populations are representative of wider public opinion and that software can accurately assess the sentiment of these populations. We should critically challenge such assumptions as Nick Halstead, CTO of DataSift, a provider of sentiment analysis software, did when he bluntly stated that any company that claims they can achieve better than 70% accuracy in tracking sentiment is ‘lying’. It is also imperative that we address the real issue of the availability and reliability of information, particularly as organisations are often reluctant to share that information with researchers and academics. This is a policy issue in its own right that could be addressed by developing public-private-academic partnerships, and by broadening the evidence base for policymaking through open data initiatives.

In the second place, those who champion open source intelligence approaches must communicate the advantages of those approaches. Compelling arguments can be, but aren’t, made and as it stands, policymakers lack a real understanding of the potential for OSINT to inform policymaking. To overcome this challenge we need to be far more proactive not only in establishing more effective partnerships and information-sharing practices, but also engaging with policymakers to highlight the policy-relevant benefits of open source research and analysis. OSINT has real potential to add significant value to policy debates: it has the power to identify knowledge gaps, analyse the efficacy of existing policies and, to stress our key point again, to provide new, rigorous evidence bases to support the development of effective policy.

The third point is that policymakers need to be receptive to the enormous advantages that OSINT as an evidence base for policymaking can provide. The big data phenomenon has filtered through to OSINT and opened up new, and as yet underused, avenues of research. As open source approaches become more rigorous, and researchers become better at communicating their applicability to the major policy issues of the day, policymakers should capitalise on this opportunity to improve – significantly – the quality of policymaking.