At least 45 companies now advertise real-time facial recognition
More than a decade before Spotify, and years before iTunes, there was RealPlayer, the first mainstream solution to playing and streaming media to a PC. Launched in 1995, within five years RealPlayer claimed a staggering 95 million users.
But it was a brief moment of glory for RealPlayer. Amid the dot-com bust and mounting pressure from Microsoft’s Windows Media Player, by March 2001 RealPlayer’s stock had dropped to $21 from $355 just a year prior. Over the course of the 2000s, the entire media software industry reinvented itself. Eventually, RealPlayer receded out of the public eye — most people who grew up using Facebook probably wouldn’t recognize the company at all.
But RealPlayer is still very much alive. Now called RealNetworks, a vast majority of its revenue still comes from licensing media software. But the company has also begun dabbling in an industry that’s suddenly attracting hundreds of firms, most of which operate outside public scrutiny: facial recognition.
Through a startup subsidiary called SAFR, RealNetworks now offers facial recognition for everything from K-12 schools to military drones. The company even claims to have launched a surveillance project in São Paulo, Brazil that analyzes video from 2,500 cameras.
SAFR has also licensed its technology to Wolfcom, a body camera company that is currently building real-time facial recognition into its products. As first reported by OneZero, Wolfcom’s push to bring live facial recognition to hundreds of police departments represents the first such effort within the United States.
Though RealNetworks’ earnings reports say SAFR doesn’t generate significant revenue yet, RealPlayer’s evolution is part of a trend of both large global tech companies and small upstart firms becoming key players in the sprawling facial recognition industrial complex. Over the last decade, Japanese tech firm NEC grew a burgeoning division focused on biometrics, alongside its 100-year-old hardware business. Toshiba, best known for making PCs, claims to be running more than 1,000 facial recognition projects around the world, including identity verification systems at security checkpoints in Russia and for law enforcement in Southeast Asia. Even software contractor Microfocus, one of a handful of companies keeping the aging COBOL language alive, is working on making facial recognition that can scale to thousands of CCTV cameras.
Toshiba claims to be running more than 1,000 facial recognition projects around the world, including identity verification systems at security checkpoints in Russia.
While many of these companies sell facial recognition technology to verify people’s identities in an app, an increasing number are investing in a burgeoning subset of the industry: real-time surveillance, or the ability to recognize individuals in live video footage. Such systems are being sold for law enforcement, military, and security purposes. Many of these companies operate in obscurity, and have never been profiled or scrutinized before.
To understand which companies are operating in this industry and the scope of their contracts, OneZero analyzed the largest repository of facial recognition companies, the facial recognition vendor test run by the U.S. National Institute of Standards and Technology.
While not a complete survey of the entire surveillance field, more than 45 companies now actively advertise facial recognition for real-time surveillance, according to an analysis of NIST-submitted vendors done by OneZero.
These companies and their products are a testament to the A.I. boom of the 2010s, which made the necessary algorithms to conduct live facial recognition trivial to build, and transformed surveillance into a commodity. Large-scale, real-time facial recognition is no longer a potential outcome of the technology, but one that is now being actively sold around the world.
“The public is largely in the dark about the state of the surveillance vendor market, whether its facial recognition or other forms of surveillance,” says Matt Cagle, technology and civil liberties attorney at the ACLU of Northern California. “When you have secretive and unknown vendors pushing their wares on police departments, effectively you have corporate entities making policy decisions without democratic accountability and democratic transparency.”
Some of the companies behind these technologies are household names: Microsoft, Amazon, NEC, and Toshiba; others are small startups of 10 to 200 people that exist outside public scrutiny — like Clearview AI, the facial recognition company that was secretly in the hands of thousands of government and corporate users. But the scope of facial recognition activity of almost all these companies is often overlooked. Increasingly, individuals around the world are being recorded, identified, and tracked — by companies that remain in the shadows.
Despite hundreds of vendors currently selling facial recognition technology across the United States, there is no single government body registering the technology’s rollout, nor is there a public-facing list of such companies working with law enforcement. To document which companies are selling such technology today, the best resource the public has is a governmental agency called the National Institute of Standards and Technology.
NIST is a government organization responsible for setting scientific measurement standards and testing novel technology. As a public service, NIST also provides a rolling analysis of facial recognition algorithms, which evaluates the accuracy and speed of a vendor’s algorithms. Recently, that analysis has also included aspects of facial recognition field like algorithmic bias based on race, age, and sex. NIST has previously found evidence of bias in a majority of algorithms studied.
“The public is largely in the dark about the state of the surveillance vendor market, whether it’s facial recognition or other forms of surveillance.”
By framing its tests as a competition, NIST has incentivized companies developing facial recognition to voluntarily step into the spotlight, and in the process created the most complete available list of companies in the space.
While NIST maintains that its facial recognition vendor test is a purely scientific ranking of accuracy, it’s often supported by law enforcement organizations; the FBI and DHS sponsor NIST’s work, says Dave Maass, senior researcher at the Electronic Frontier Foundation.
“The companies that engage in this challenge and cooperate with NIST are often government and military contractors, and while NIST is measuring accuracy, the utility for a lot of these companies is to use those results in marketing,” Maass said.
A top NIST result may not be an endorsement from the U.S. government, but it’s certainly a gold star marked next to a company’s technology, meaning public funds are being used to power the marketing arms of biometrics companies.
The most recent NIST analysis of facial recognition algorithms surveyed nearly 200 companies and universities, though NIST’s list of facial recognition vendors is incomplete and a NIST verification is not mandatory to sell facial recognition in the United States. The fact that NIST is perhaps the most complete list of facial recognition vendors speaks to a lack of verifiable information on surveillance technology.
OneZero’s review of NIST’s live facial recognition vendors reveals that the companies that sell facial recognition to governments across the world, as well as private spaces like retailers and malls, are often names most Americans have never heard of — small enterprise software companies that don’t sell products to consumers and don’t market themselves outside of trade shows or direct pitches to potential clients.
These are companies like Palo Alto-based Camvi, India’s Awiros, and Los Angeles’ Trueface, which often illustrate their technology with images of crowds of people being identified while walking down the street or attending a sporting event. Each of those companies has fewer than 50 employees, according to LinkedIn data. Another company based in Tokyo, Ayonix, claims that each of its cameras can analyze between 30 and 1,000 faces in a video stream every second. The Slovakian firm Innovatrics tells potential customers that its software requires no expertise in biometrics to set up. AllGoVision says it has projects running in 30 countries. Tech5 says its facial recognition is used city-wide for live facial recognition in Indonesia, as well as monitoring social media.
Some of the companies have been mired in controversy: Ever AI created a photo-sharing app that actually scraped its users’ data to train its facial recognition algorithm. After an NBC News exposé, the company changed its name to Paravision.
“This looks like an egregious violation of people’s privacy,” ACLU attorney Jacob Snow told NBC News. “They are taking images of people’s families, photos from a private photo app, and using it to build surveillance technology. That’s hugely concerning.”
Paravision’s website now reads “A Higher Standard for Facial Recognition.”
NTech Labs, a Russian facial recognition company, is also tested by NIST. The company is most famous for its FindFace app, which The Atlantic wrote in 2016 could “end anonymity.”
FindFace was a consumer app that allowed users to take a picture of anyone, and the app would try to find matches from VKontakte, a Russian social networking site. Company co-founder Alexander Kabakov told The Guardian in 2016 that FindFace could revolutionize dating since you could figure out who a person was without talking to them, and then send a friend request or a message.
“It also looks for similar people. So you could just upload a photo of a movie star you like, or your ex, and then find 10 girls who look similar to her and send them messages,” Kabakov said.
NTech Labs shut down the public-facing app at some point after 2016, and pivoted to building government surveillance for Russia. The company now describes its work as the biggest live facial recognition project in the world, according to an interview with Forbes.
Live facial recognition is offered as a solution for smart cities or policing to governments, but also as a technology to be adopted by businesses to prevent theft, track customer emotions, or for facial recognition payments.
Of the companies publicly advertising real-time facial recognition, none are testing it on the U.S. public. However, many do have contracts in Asia and the Middle East.
One company, called NotionTag, runs an app called FaceTagr, which has been used by police in Chennai to take pictures of people and run it against a facial recognition database, according to the New Indian Express. A short 2018 profile of the company by Wired also claims the system works in real time.
A company called Kedacom installed live facial recognition cameras on police cars in Dubai, while another company called Gorilla Technologies has installed a facial recognition surveillance system into prisons across Taiwan.
Among the crop of companies now selling facial recognition are government contractors that previously specialized in analyzing fingerprints. Two of the largest companies selling facial recognition are Idemia and Gemalto, which each employ upward of 10,000 employees. Gemalto was bought by defense contractor Thales Group in 2019.
Idemia is the biometrics company perhaps most entrenched in American travel — it runs the identity check and fingerprint recognition for TSA Precheck. The company has earned nearly $600 million from the federal government since 2008, for contracts including the TSA program to the FBI’s Next Generation Identification program, which includes facial and fingerprint recognition for FBI investigations. Idemia, and its parent private equity company Advent International, is entitled to $2.2 billion in potential earnings from the TSA alone if all of its current contracts are completed, according to data compiled by government technology watchdog Tech Inquiry.
A few of the companies verified by NIST are also not what their marketing would suggest. Two companies, 3DiVi and VisionLabs, pitch themselves as startups based in California and Amsterdam. However, a majority of their programmers live in Russia, and 3DiVi’s facial recognition is integrated with Papillon, a security company with close ties to the Russian government.
Papillon says its biometric identification system is used by six Russian government agencies, and is exported to Albania, Bangladesh, Kazakhstan, Mongolia, Nigeria, Serbia, Turkey, and Uzbekistan.
Privacy advocates have widely disavowed the use of facial recognition for surveillance purposes, saying that the technology is overly invasive and prone to errors. But as there are no laws banning or strictly regulating facial regulation on a federal level in the U.S. or around the world, the only barrier standing between the use of real-time facial recognition is technical.
That barrier no longer exists. Real-time facial recognition — the power for governments or private organizations to identify and track nearly anyone who frequents public or private spaces — is here.
“Five years ago, the camera systems weren’t there, the algorithms weren’t there,” says Maass. “It was really science fiction-y to a lot of people. Now we’re starting to see the dawn of the horizon of this.”
The true number of companies selling facial recognition is still unknown, despite those publicly marketing their services and submitting algorithms to NIST for testing.
He compares this to the installation of automatic license plate readers over the last decade, a technology mounted on police cars and on public streets that logs information on everyone who drives by. This form of surveillance can be used to track people driving around a city, or even learn their routines. Now that the technology for facial recognition has advanced, Maass has seen the infrastructure for facial recognition cameras start to be similarly built out.
The ACLU says that the biggest danger of this technology is its potential to be used for “suspicionless surveillance,” or as a dragnet identifying people whether they are suspected of a crime or not. Many of the vendors whose algorithms NIST has validated sell this exact technology.
And while the most common images for facial recognition are mugshots of people who have previously committed crimes, governments around the world already have access to images of billions of people through driver’s licenses or other forms of identification.
“State motor vehicles agencies possess high-quality photographs of most citizens that are a natural source for face recognition programs and could easily be combined with public surveillance or other cameras in the construction of a comprehensive system of identification and tracking,” the ACLU writes.
A 2016 project by Georgetown Law’s Center on Privacy and Technology found that more than 117 million adults in the United States can be found in a facial recognition database on a local, state, or federal level. And that was four years ago, before many contracts and databases such as Clearview AI were discovered.
This is also a count of public repositories of data. Convenience stores, supermarket chains, fast food restaurants, sports arenas, museums, or any business open to the public can collect enormous amounts of information on customers through transaction data, and capture faces from video using CCTV cameras. Real-time facial recognition makes all that data far more valuable, as a company can know the second you walk into a store.
The true number of companies selling facial recognition is still unknown, despite those publicly marketing their services and submitting algorithms to NIST for testing. Even with a list of companies like OneZero compiled, there’s no way to know where the technology is being deployed.
“The growth of ‘surveillance as a service’ is where companies create cheap, secretive surveillance products and market them directly to governments, leaving communities out of the loop, and the democratic process out of the loop,” says the ACLU’s Cagle. “People are seeing how easily these technologies can be deployed and how dangerous they can be to people’s privacy, free speech, and rights more generally.”
This leaves the public in a state of surveillance without their knowledge, and without insight into who is making the algorithms that watch them, how those algorithms function, or if they’re even accurate.
Even though U.S. federal, state, and local governments aren’t yet using the technology, it’s not for lack of options. The dozens of companies selling real-time facial recognition all pitch it as the future of public safety and security, vying for public dollars. And in the meantime, those companies are testing their algorithm’s efficacy in other countries.