Automated recognition of faces in images has become a much-debated topic in recent times. To enable an informed discussion about this topic, it is important to know some background about this technology and its current use.

Facial recognition is a very active branch of computer vision that pursues the goal of identifying an individual based on an image of their face. The rationale behind this is that people’s faces are unique to them, like other biometrics, such as fingerprints, DNA, or, curiously, ear shape [1]. An obvious advantage of identifying one’s face, rather than, for example, DNA, is that it is very accessible – all it takes is a picture.

The first automated facial recognition algorithm was developed in 1991 and was called Eigenfaces [2]. This algorithm was a breakthrough since it no longer required the manual annotation of facial features (such as eyes, mouth, etc.). Instead, it used mathematical transformations of facial images to match individuals to images in a database. The general idea of this approach is still used today and can be divided into three steps: face detection, feature extraction, and identification [3]. In the first step, an algorithm scans an image for faces. A detected face is cropped out of the original image and pre-processed to match the images it will be compared to by adjusting, for instance, the framing or the brightness/contrast. In the second step, machine learning is used to extract facial features and to create a description of the face that can be compared to others. This description is then used in the third step to match the face with other faces from a database to determine a person’s identity or medical state. Most of the current research is focused on optimizing feature extraction since the key for efficient and reliable identification of faces lies in their precise, complete, and comparable description [4].

One prominent area of facial recognition is user authentication. Here, the task of the algorithm is to decide whether the face in a picture of a user matches a previously analyzed face and, if so, to grant the user access. For this purpose, all involved data can be stored locally on the user’s device, making it ethically less problematic. Examples for this tool’s application are Apple’s FaceID or Microsoft’s Windows 10 Hello, which are both facial recognition technologies used to unlock electronic devices. Current efforts aim towards expanding the use of such technologies in everyday life [5]. Apart from its uses in electronic devices, authentication via facial recognition has been used since the mid-2000s by governments around the world to identify travelers via the biometric passport [6].

Another – more controversial – area of facial recognition is the identification of people. The main difference with regard to authentication is that to identify an unknown face, it must be compared with many previously recorded faces. This requires a database of face descriptions linked with the names of the corresponding individuals. Such databases can be queried for identification purposes [4]. One example of facial recognition used in this context is the identification of victims or perpetrators of crimes by law enforcement [7]. Despite the obvious benefits that such technology would have in the hands of ethical governments and police, its use remains very controversial. One reason for that is that reliable facial recognition paired with extended public surveillance can enable the automatic tracking and profiling of an individual’s whereabouts and actions [8]. Moreover, facial recognition technology may appear unbiased in its evaluations, when in reality it isn’t. It is highly susceptible to gender or ethnic biases that enter through the data used for training the machine learning algorithms [9]. Both reasons are regarded as highly problematic by ethics researchers and other scientists, who call for legislative restrictions on the use of large scale facial recognition [8, 10]. As a reaction to that – and following public pressure – most big technology companies have now stopped supplying law enforcement with facial recognition technology [11].

One final example of the application of facial recognition software is its use as a decision support system for physicians. In this case, automated recognition of a patient’s face is used to estimate the probability of the patient having a certain condition. This has been demonstrated to work in several cases, including the detection of rare genetic diseases [12-14] and the prediction of physiological parameters such as body fat, blood pressure, or BMI [15]. Obvious points of concern for this type of application are again biases in the selection of the patients used to train the used algorithms and a potential lack of consent given by them [16].

To sum up, facial recognition technology is a rapidly evolving and highly profitable discipline [17], and it seems that its progress has outpaced lawmakers and community leaders. It is crucial that the use of facial recognition technology for authentication, identification, and medical diagnosis becomes well regulated so that its power can be harnessed in an ethically sound way – without discriminating against people or compromising their privacy or personal freedom.