Facial expressions are\u00a0forms of non-verbal communication<\/strong>, providing hints for human emotions. For decades, decoding such emotion expressions has been a research interest in the field of psychology\u00a0(Ekman and Friesen 2003; Lang et al. 1993)\u00a0but also to the Human Computer Interaction field\u00a0(Cowie et al. 2001; Abdat, Maaoui, and Pruski 2011). Recently, the high diffusion of cameras and the technological advances in biometrics analysis, machine learning and pattern recognition have played a prominent role in the development of the FER technology.<\/p>\n Many companies, ranging from tech giants such as NEC or Google to smaller ones, such as Affectiva or Eyeris invest in the technology, which shows its growing importance. There are also several\u00a0EU research and innovation program\u00a0<\/strong>Horizon2020<\/strong><\/a>\u00a0initiatives<\/strong>1<\/sup><\/a>\u00a0exploring the use of the technology.<\/p>\n FER analysis comprises three steps:\u00a0a) face detection, b) facial expression detection, c) expression classification to an emotional state<\/strong>\u00a0(Figure\u00a01). Emotion detection is based on the analysis of facial landmark positions (e.g.\u00a0end of nose, eyebrows). Furthermore, in videos, changes in those positions are also analysed, in order to identify contractions in a group of facial muscles\u00a0(Ko 2018). Depending on the algorithm, facial expressions can be classified to basic emotions (e.g. anger, disgust, fear, joy, sadness, and surprise) or compound emotions (e.g.\u00a0happily sad, happily surprised, happily disgusted, sadly fearful, sadly angry, sadly surprised)\u00a0(Du, Tao, and Martinez 2014). In other cases, facial expressions could be linked to physiological or mental state of mind (e.g.\u00a0tiredness or boredom).<\/p>\n <\/p>\n Figure 1: Steps of Facial Emotion Recognition<\/em><\/p>\n <\/p>\n The source of the images or videos serving as input to FER algorithms vary from surveillance cameras to cameras placed close to advertising screens in stores as well as on social media and streaming services or own personal devices.<\/p>\n FER can also be combined with\u00a0biometric identification<\/strong>. Its accuracy can be improved with technology analysing different types of sources such as voice, text, health data from sensors or blood flow patterns inferred from the image.<\/p>\n Potential uses of FER cover a wide range of applications<\/strong>, examples of which are listed here below in groups by their application field.<\/p>\n Due to its use of biometric data and Artificial Intelligence technologies, FER shares some of the risks of using facial recognition and artificial intelligence. Nevertheless, this technology carries also its own specific risks. Being a biometrics technology, where aiming at identification does not appear as a primary goal, risks related to emotion interpretation accuracy and its application are eminent.<\/p>\n Turning human expressions into a data source to infer emotions touches clearly a part of peoples\u2019 most private data. Being a disruptive technology, FER raises important issues regarding necessity and proportionality.<\/p>\n It has to be carefully assessed, whether deploying FER is indeed\u00a0necessary for achieving the pursued objectives or whether there is a less intrusive alternative<\/strong>. There is risk of applying FER without performing necessity and proportionality evaluation for each single each case, misled by the decision to use the technology in a different context. However proportionality depends on many factors, such as the type of collected data, the type of inferences, data retention period, or potential further processing.<\/p>\n Analysis of emotions based on facial expressions\u00a0may not be accurate<\/strong>, as facial expressions can slightly vary among individuals, may mix different emotional states experienced at the same time (e.g. fear and anger, happy and sad) or may not express an emotion at all. On the other hand, there are emotions that may not be expressed on someone\u2019s\u00a0face<\/em>, thus inference based solely on facial expression may lead to wrong impressions. Additional factors can add to the ambiguity of the facial expressions, such as contextual clauses (sarcasm), and socio-cultural context. In addition, technical aspects (different angles of the camera, lighting conditions and masking several parts of the face) can affect the quality of a captured facial expression.<\/p>\n Furthermore, even in the case of accurate recognition of emotions, the use of the results may lead to\u00a0wrong inferences<\/strong>\u00a0about a person, as FER does not explain the trigger of emotions, which may be a thought of a recent or past event. However, the results of FER, regardless of accuracy limitations, are\u00a0usually treated as facts<\/strong>\u00a0and are input to processes affecting a data subject\u2019s life, instead of triggering an evaluation to discover more about their situation in the specific context.<\/p>\n The accuracy of the facial emotion algorithm results can play an important role in\u00a0discriminating on grounds of skin colour or ethnic origin<\/strong>. Societal norms and cultural differences have been found to influence the level of expression of some emotions while some algorithms have been found to be biased against several groups, based on skin colour. For instance, a study testing algorithms of facial emotion recognition revealed they assigned more negative emotions (anger) to faces of persons of African descent than to other faces. Furthermore, whenever there was ambiguity, the former were scored as angrier (Rhue, 2018).<\/p>\n Choosing the right dataset that is representative is crucial for avoiding discrimination. If the\u00a0training data<\/strong>\u00a0is not diverse enough, the technology might be\u00a0biased<\/strong>\u00a0against underrepresented population. Discrimination triggered by faulty database or by errors in detecting the correct emotional state may have serious effects, e.g.\u00a0inability to use certain services.<\/p>\n In another aspect of the same problem, in case of\u00a0medical conditions or physical impairments<\/strong>\u00a0in which temporary or permanent paralysis of facial muscles occurs, data subjects\u2019 emotions may be misunderstood by algorithms. This may result in a wide range of situations of\u00a0misclassification<\/strong>, with impact ranging from receiving unwished services up to misdiagnosis of having a psychological disorder.<\/p>\n Facial images and video can be captured anywhere, thanks to the ubiquity and small size of cameras. Surveillance cameras in public spaces or stores are not the only cameras remotely capturing facial images as one\u2019s own mobile devices can capture expressions during their use. In these situations, transparency issues arise concerning both the collection and the further processing of personal data.<\/p>\n Where the data subjects\u2019 facial expressions are\u00a0captured in a remote manner<\/strong>, it may not be clear to them which system or application will process their data, for which purposes, and who the controllers are. As a result, they would not be in the position to freely give consent or exercise\u00a0control over the processing<\/strong>\u00a0of their personal data, including sharing with third-parties. Where data subjects are not provided with accurate information, access and control over the use of FER, they are deprived of their freedom to select which aspects of their life can be used to affect other contexts (e.g.\u00a0emotions in social interactions could be used in the context of recruitment). Moreover, data subjects need to control which periods of time their captured data will be processed and aggregated to history records of their emotional situation, as emotion inferences may not be valid for them after a period of time.<\/p>\n Another consequence of the remote capture of facial expressions and the obscurity of their processing is that data subjects might not be provided with information on\u00a0which other sources of data these will be aggregated to<\/strong>. Also, advanced AI algorithms add to the complexity of transparency needs, as they may detect slight movements of facial muscle that are unconscious even for the individuals. This would contribute to the unpleasant feeling of vulnerability due to unwanted exposure.<\/p>\n FER technology can detect the existence, changes or total lack of facial expressions, and link this to an emotional state. As a result, in some contexts, algorithms may\u00a0infer special categories of personal data<\/strong>, such as political opinions or health data. For instance, applying FER technology at political events, political attitudes can be inferred by looking at facial expressions and reactions of the audience. Also, by the lack of facial expressions, algorithms are able to detect signs of alexithymia, a state in which one cannot understand the feelings they experience or lack the words to describe these feelings. This finding can be linked to severe psychiatric and neurological disorders, such as psychosis. Furthermore, analysis of historical data on one\u2019s emotional state may reveal other health conditions such as depression. Such data, if used in the context of healthcare, could assist in prediction and timely treatment of a patient. However, where data subjects are not able to control the flow of derived information and its use in other contexts, they may face a situation of inference and\u00a0use of such sensitive personal data by non-authorised entities<\/strong>, such as employers or insurance companies.<\/p>\n FER technology can be further used to create profiles of people in a number of situations. It could be used to derive one\u2019s acceptance of a product, an advertisement or a proposed idea. It can also be used for classifying productivity and fatigue-resistance in workplaces. The risk lies in the fact that the data subject may not be aware of this type of\u00a0targeting<\/strong>\u00a0and might feel uncomfortable if they found out about it. Further implications can occur by\u00a0erroneous profiling<\/strong>\u00a0or inferences solely based on the association with a certain group of people experiencing the same emotions.<\/p>\n In addition, the knowledge of the individuals\u2019 emotions can make it easier to\u00a0manipulate<\/strong>\u00a0them. For instance, the knowledge of emotions revealing a vulnerable emotional state, can be used to mentally force people to perform actions they would not do otherwise \u2013 e.g.\u00a0to buy goods they do not need.<\/p>\n FER technology could be used for purposes of safeguarding public security, for instance at concerts, sport events or airports, to quickly identify signs of aggression and stress and identify potential terrorists. However, if such an identification was based solely on FER and was not combined with other actions or triggers that this person is dangerous, this could introduce further risks for the data subjects. For instance, a person could be subject to\u00a0unjustified delays to perform further security checks or investigations<\/strong>, causing them to miss participation in an event, boarding on a flight or even lead to unjustified arrest.<\/p>\n Last but not least, FER can\u00a0influence behavioural changes<\/strong>\u00a0in case a person is aware of the exposure to this technology (known as\u00a0Reactivity<\/a>\u00a0in psychology). Individuals may alter their habits or avoid specific areas where the technology is applied in an attempt to self-sensor and protect themselves. One can imagine the chilling effect this could have to a society and the feeling of insecurity among citizens, if such a technology were to be used by non-democratic governments, to infer political attitude of citizens.<\/p>\n![\"techdispatch\"](\"https:\/\/pirc-musar.si\/wp-content\/uploads\/2023\/08\/facial-emotion-recognition-steps.png\")
<\/div>\n<\/div>\nProvision of personalised services<\/h4>\n
\n
Customer behaviour analysis and advertising<\/h4>\n
\n
Healthcare<\/h4>\n
\n
Employment<\/h4>\n
\n
Education<\/h4>\n
\n
Public safety<\/h4>\n
\n
Crime detection<\/h4>\n
\n
Other<\/h4>\n
\n
2. What are the data protection issues?<\/h3>\n
2.1\u00a0Necessity and proportionality<\/h4>\n
2.2\u00a0Data accuracy<\/h4>\n
2.3\u00a0Fairness<\/h4>\n
2.4\u00a0Transparency and control<\/h4>\n
2.5\u00a0Processing of special categories of personal data<\/h4>\n
2.6\u00a0Profiling and automated decision-making<\/h4>\n