A Brief History of Open Source Intelligence

“Even a regimented press will again and again betray their nation’s interests to a painstaking observer”.

William Donovan (OSS Director)

“Today you are the media, it is your duty to report and keep the hope alive! You are the media, we are one!”

Mir Hossein Mousavi (former Iranian Prime Minister)

Upstate New York, 1883 is an odd place to start a blog on Open Source Intelligence. It was here that one of intelligence’s most influential figures was born. The son of devout Irish immigrants, William Donovan grew up in a working class family, excelling himself in school and academia.

Donovan’s ambition was to become the first Roman Catholic President of the United States. He did come close to the Presidency – in 1905, Donovan went to Columbia Law School, where a young Franklin D. Roosevelt was among his classmates.

After fighting in World War I, Donovan had a successful career as an International Lawyer, narrowly missing out on becoming the Attorney General in 1925. Throughout the interwar period, Donovan travelled the world as a Lawyer, meeting influential foreign figures and subsequently writing up reports for the US Government in a semi official capacity.

It was Donovan’s connection to F.D.R. that lead to the creation of an intelligence agency in the United States. Until that time, the world of intelligence and spying was seen as ungentlemanly by America. Donovan lobbied F.D.R to formalise his unofficial work for the US Government and on the 11th July, 1941, F.D.R. created for Donovan the post ‘Coordinator of Information’. After Pearl Harbour, the need for intelligence was clear and Donovan’s department was renamed the Office of Strategic Services – the precursor to the CIA. Like the Special Operations Executive in the UK, the OSS was involved in everything from assassination attempts to agent running and information warfare.

It may surprise readers of 2016, that the OSS had an entire branch dedicated to open source intelligence. The OSS’ Research and Analysis Branch meticulously collected dozens of newspapers, journals, press clippings, radio broadcasts reports from around the world, hunting for photos and articles that may give away crucial intelligence about the enemy. In Donovan’s words:

“Even a regimented press will again and again betray their nation’s interests to a painstaking observer”

The OSS poured over obituaries in German regional newspapers, seeking news of important Nazis. Images of new battleships, bomb craters and aircraft were painstakingly collated, and when assessed together, allowed the OSS to assess the state of its enemies. It’s striking how similar the OSS’ activities are to modern day OSINT investigations, albeit without computers. From the OSS and SOE, it’s possible to argue the roots of open source intelligence stretch back almost a century. Indeed one could argue Donvoan’s quote is more true today than ever; amongst the billions of posts, uploads, shares and likes, individuals again and again betray their interests to painstaking observers.

After World War II, the discipline of OSINT became a backwater of most Government and military agencies, staffed with career librarians and researchers. When the noun Library became an unfashionable term in the mid-2000’s, few, if any intelligence professionals sought to work in the field, when the sexy, secret world of agent-running and signals intelligence was available.

For decades, the world of open source (in a Government analysis context) went into a deep sleep, undisturbed by the Cold War or even 9/11.

But then in 2009, something changed. Iran was on the brink of a bottom-up ‘Green Revolution’; many of its citizens were protesting against the regime. Millions of young Iranians took to the internet to coordinate their activities, share viral content and encourage others to join in the campaign. For the first time, the internet was flooded with citizen information about a major political event, largely thanks to the combination of smartphones, internet connections and social media. During the first week of the protests, 60% of all blog links posted on Twitter were about Iranian politics.

Internet use in Iran jumped from 34% in 2008 to 48% in 2009, a huge increase, and mobile phone subscriptions went from 59% to 72% of the population. At the time, the BBC ran an article entitled “Internet Brings Iran to Life”, where it claimed a new form of ‘citizen journalism’ was thriving. During the 2009 protests in Iran, the Washington Post ran a Q&A with the influential digital commentator Evgeny Morozov:

Fairfax, Va.: There’s been a lot written about the coverage in Iran this past weekend and that the U.S. news organizations didn’t really carry their weight but that Twitter and other similar Web sites did spread news and let people know around the world what was going on in Iran. Comments?

Evgeny Morozov: We saw quite a few citizen journalists doing an excellent job of taking photos and videos of protests in Tehran almost in real-time. They have, indeed, filled an important niche. Networks like Twitter, similarly, have played a great role in attracting people’s attention to this user-generated content. So, Flickr provided great photos — and Twitter provided great attention to these photos. There has, indeed, been a lot of criticism of the lack of Iran-related coverage on CNN; Twitter users have even organized an entire campaign to deal with this called #cnnfail. I think they have been successful: CNN executives/reporters eventually had to answer questions about it.

For the first time – any individual around the world could mine these social networks for intelligence-grade content, and in the process, write articles, forecasts and deliver insightful intelligence analysis. Whilst the protests were ultimately unsuccessful, and the Iranian regime quickly reasserted control of the Internet, we can now look back and regard the (ultimately unsuccessful) Green Revolution as a seminal event in the history of open source intelligence.

In hindsight, it is not surprising that Iran was where this new world of OSINT was forged, Iran has more online users as Bahrain, UAE, Israel, Saudi Arabia, Syria, Yemen and Jordan combined. During the Revolution, Matthew Weaver at the Guardian expressed surprise at the realities of this new world; “What people are saying at one point in the day is then confirmed by more conventional sources four or five hours later”.

It is not surprising that citizen journalists and lone investigators (rather than Governments) led the development of OSINT as a discipline – it is a field that is moving at light speed, with new tools and techniques being honed and created every day. The monolithic worlds of academia and government are only just beginning to realise the potential riches of this data. Individuals unconstrained by bureaucracy or bad IT can quickly become adept OSINT operators, and create insights and connections that were previously unknown.

Barely a year after the Green Revolution, revolutions stoked by social media spread across the Arab world. The combination of public anger, smartphones and social media rocked dictatorships across North Africa and the Middle East. According to one of its Directors, the CIA OSINT Centre was ‘unable to forsee the precise development of Internet based social activism in the Arab world’. One explanation for this failure was that Government intelligence was obsessed with collecting intelligence from the powerful elite, rather than “taking advantage of the enormous amount of open-source information out there”.

However, in recent years the US, UK and others clearly have taken notice. Last year, the US military destroyed an Islamic State bomb factory a mere 23 hours after a jihadi posted a selfie revealing the roof structure of the building, which is perhaps the most powerful example of the military using OSINT for targeted operations. In the private sector, dozens of open source intelligence firms have sprung up both in the UK and the US, servicing a range of public and private sector clients. More recently, as WIRED Magazine reported in April 2016, a whole range of Government agencies are looking at ways to better understand the wealth of open data available online, realising that OSINT can make or break operations.

Where will OSINT go next?

Reflecting on this short history of open source intelligence, it could be argued that modern-day OSINT is a result of convergence of technologies. This began around 2009, when three things happened; firstly – a critical mass of smartphones with 3G connections were in the hands of disaffected citizens. Secondly, those citizens used a small number of apps to share a huge amount of content about political events inside their country. Thirdly, this data was free and open for the rest of the world to access and analyse.

However, the world of OSINT will not stand still, and other technologies will continue to augment and change the OSINT practice. The growing appetite to live-stream content presents very real challenges for police, security and OSINT practitioners, from Minnesota to Damascus. Machine Learning, Virtual & Augmented Reality and Artificial Intelligence will again transform OSINT in the coming years. It could be argued OSINT may well become the new hot field of tech investment, taking the mantle from cyber security.

OSINT is increasingly a field that attracts huge attention from investors, the media and the public. After all, with billions of posts, images, streams, records and data uploaded to the internet every day – surprisingly few of us actually realise they are rather good odds for finding useful intelligence.