Using the New Russian Facial Recognition Site SearchFace

Update: SearchFace is dead, long live FindClone! This guide is now mostly for historical interest, as SearchFace has been taken offline and replaced with FindClone, a far easier tool to use than SearchFace was. You need to use a mobile telephone number to create an account for FindClone, which then gives you 20 or so searches over a set number of days to run your searches (unless you register for a relatively low fee). I personally used a “burner phone” (pre-paid phone not tied to my real number) for this, and I am not aware if there are any dangers to your privacy in using your “real” phone number. Additionally, the new FindClone.ru site will directly link you to the profile that matches the face you search, making the latter half of this guide especially useless. The facial recognition algorithms are still outstanding, and we at Bellingcat have used this site quite a bit and broken a long-dormant investigation wide open by identifying a few individuals whom we never previously thought we could confidently identify.

New Update, 7 January 2020: FindClone is still up and alive, and one of the most horrifyingly powerful tools I’ve ever used online. I spoke with Vox recently on FindClone and facial recognition within the larger context of privacy and investigations. Watch their video report, “What facial recognition steals from us”, below:

In September 2018, the Russian website FindFace was taken offline in order to provide “solutions tailored for government and business”. This site received international press for its accuracy in its facial recognition capabilities, with a focus on finding someone’s profile on the Russian social network Vkontakte (VK). Recently, a new site has emerged from the ashes of FindFace: SearchFace.ru.

The owners of SearchFace.ru are not known, with the site providing no information on the tool other than a feedback email address and the title of the page, “FindTwin face search demo.” Meduza and TJournal have spoken with the site’s creator, who says that he is hoping to “test” the tool with the larger reference base of VK photographs. Additionally, he noted that he gathered “half a billion” VK photographs in the summer of 2018.

Like FindFace, SearchFace allows a user to upload a photograph, recognize a face in the image, and then match this face to its “twins” on VK. The results are as accurate as the reference material on VK: if someone has a profile on VK, there’s a good chance that they will be matched. However, if the face in question belongs to a person who does not have a VK profile, you will likely just find his or her Russian (or Ukrainian, or Belarusian, or Kazakh…) doppelganger on the social network.

Just finding similar faces to one in an image you upload on this site does not require you to read this guide, but if you want to take an extra step or two in finding out a potential facial match, you may need to use a few tricks in VK that will be detailed here.

What works, and what doesn’t

The first thing to understand about using this site is its range of reference material: VK, and only VK. While VK is a huge social network with hundreds of millions of registered users, it does not have nearly the same global user base as Instagram or Facebook. SearchFace does not look at photographs uploaded anywhere except VK, thus if you are trying to match the face of someone from outside of a post-Soviet state, you should probably use a different tool.

Images uploaded to VK — found via the userapi.com domain — remain even after a profile is deleted. SearchFace is relying on more on pulling these image files from VK’s server (userapi.com) rather than through the VK interface, meaning that many results found on SearchFace are from deleted, suspended, and even locked/private profiles.

Let’s give the tool a test run to see its capabilities by using a photograph of Bato Dambaev, a Russian soldier from the Republic of Buryatia who was featured in a 2015 investigation by Simon Ostrovsky with the help of Bellingcat (watch it here).

Image of Bato Dambaev used in a SearchFace test.

Running this on SearchFace, we get 16 results. Some of these results are just screenshots from Ostrovsky’s documentary, others are other Buryat men who look a bit like Bato. However, the first three results are indeed Bato from different profiles on VK.

The first result is from a profile of another soldier in Bato’s military unit. None of these photographs feature only Bato, as they are all group shots with the friends of the profile owner. The second profile is Bato as well, likely from the VK of his wife or a member of his family, as Bato has deleted his VK profile following the 2015 documentary. The third person is again Bato from when he was younger–this was likely from an old, abandoned VK profile he had (as uploaded files remain on VK’s servers after a profile is gone), or a family member’s profile.

Other results are clearly not Bato, but instead of other men who either are Buryat or have a similar complexion as Bato. A common feature of these men is that they all have bushy eyebrows, thus revealing one of the details that SearchFace locked in on when it analyzed Bato’s face.

SearchFace was very successful in searching for Bato because he and many of his family members were active on VK, providing a large amount of reference material for SearchFace to survey.

Let’s try a couple more searches to see where this tool’s limits lie with people who are not on VK — two members of this year’s Duke University men’s basketball team. We can’t know exactly what the algorithm is that powers SearchFace, but we make some educated guesses based on what results we get with these Duke searches.

Searching for the face of sophomore guard Alex O’Connell provides no results better than a .600 score (for reference, Bato’s self-matches were around 0.850-0.800), and none of the people are actually Alex. However, the third result is a Ukrainian boy who could probably pass as him if he had the same haircut.

One observation we can make for these results shows that SearchFace does not put much emphasis on a person’s haircut or clothing, instead focusing more on the actual shape and details of the face, such as eyebrows, the shape of the nose, and so on. If we were to run this same search on Yandex Images, which is often very good at matching faces, there would be a number of results of people wearing white tank-tops and white men with curly hair, rather than other young men with longer noses and similar bone structure, as we see with SearchFace.

Searching for the face of superstar freshman Zion Williamson brings back chaotic results, for a few different reasons.

The two results, with scores of 0.71 and 0.7, are a pretty good match, with the nose and eyebrow shapes of the men quite similar. However, the results devolve into nonsense after this, with NBA superstars Giannis Antetokounmpo (score 0.675) and Russell Westbrook (0.673) listed as matches with a fairly high score. These three men do not look much alike at all, and the fact that SearchFace brought back a high score for them reflects a few possible deficiencies in the algorithm, including a lack of capabilities in searching for people of an ethnicity not common in Russia, a lack of reference materials to “learn from” in the VK image database, and/or an algorithmic tick that recognized Zion as a basketball player and thus brought back other basketball players as a result. This same algorithmic tick, if it does exist, could work in other contexts to bring back people of the same profession as the original image — for example, a search for Bato Dambaev may give a higher preference to men who SearchFace recognizes as soldiers (via their uniforms).

Reverse searching results

SearchFace links to the JPG file on VK’s servers without any reference to the user who uploaded the image. In most cases, the URL and filename have no relationship to the VK user ID. The exception to this is with some profile photos, which have the user ID within the URL (following the letter “u”), but this is relatively rare with SearchFace results. In order to trace back where a SearchFace match comes from, we have a couple of options, neither of which are full-proof.

One method is simple — run a reverse image search on the photograph, and hope that you find a result on VK. For this, you should use Yandex, as it is a far superior reverse image search tool than Google, Bing, or TinEye.

The second solution is a bit more complex, and only somewhat reliable. To do this, you need to have a VK account and upload the image there, and then use the VK search engine to find duplicate images. (Thanks to sl0zhny on Twitter for discovering this method)

Let’s run a search for Steven Seagal and try and track back the source of one of his photographs.

Let’s use one of these results — a photograph found at https://pp.userapi.com/c837122/v837122383/1b272/Fif4ECEZtHY.jpg.

Once on VK, upload the photograph as if it were your own.

After posting the image, click the photograph and copy the section of the URL that starts with “photo” and ends before a symbol. There should be eight digits after “photo” and nine digits after the underscore.

Next, go to https://vk.com/feed?section=photos and type in (with the correct digits for your photograph included):

copy:photo12345678_123456789

This will bring back results of other times this photograph has been posted on VK, as seen below:

However, this process is far from perfect — often, the only results will be from large pages, rather than individual personal profiles. It’s worth a try, but don’t expect comprehensive and immediate results.

Conclusion

SearchFace has very limited, yet powerful, uses in looking for a person’s presence on VK. The applications of using the site are far-reaching, from legitimate digital research purposes to more nefarious possibilities, as we have seen with Russian sex workers being targeted through FindFace. Public facial recognition tools will only become stronger in the coming years, with SearchFace perhaps being a harbinger of future tools that use Facebook, Instagram, and other platforms with a far larger set of faces to match.

Update: Meduza and TJournal spoke with the creator of SearchFace and discovered a few details about the site’s purpose and methodologies. This article has been updated to reflect their findings.