# Tagging Multimedia Stimuli With Ontologies

Marko Horvat1, Siniša Popović 1, Nikola Bogunović 2 and Krešimir Ćosić 1 1Department of Electric Machines, Drives and Automation 2Department of Electronics, Microelectronics, Computer and Intelligent Systems Faculty of Electrical Engineering and Computing, University of Zagreb Unska 3, HR-10000 Zagreb, Croatia Phone: (385) 1- 6129 521 Fax: (385) 1-6129 705 E-mail: marko.horvat2@fer.hr Abstract - Successful management of emotional stimuli is a pivotal issue concerning Affective Computing (AC) and the related research. As a subfield of Artificial Intelligence, AC is concerned not only with the design of computer systems and the accompanying hardware that can recognize, interpret, and process human emotions, but also with the development of systems that can trigger human emotional response in an ordered and controlled manner. This requires the maximum attainable precision and efficiency in the extraction of data from emotionally annotated databases While these databases do use keywords or tags for description of the semantic content, they do not provide either the necessary flexibility or leverage needed to efficiently extract the pertinent emotional content. Therefore, to this extent we propose an introduction of ontologies as a new paradigm for description of emotionally annotated data. The ability to select and sequence data based on their semantic attributes is vital for any study involving metadata, semantics and ontological sorting like the Semantic Web or the Social Semantic Desktop, and the approach described in the paper facilitates reuse in these areas as well. 

## I. Introduction

Tagging is a process of annotation or assignment of metadata to information. While annotation can be simply described as adding a note by way of comment or explanation [1] it actually entails significantly more complexity and the accompanying theory or research. Tag can be strictly defined as a 'free-text keyword. Thus tagging becomes an 'indexing process for assigning tags to resources' [2]. A shared collection of tags used within a particular system is called folksonomy. Folksonomies are also referred to as collaborative tagging, social classification, social indexing, and social tagging. 

Development and management of folksonomies, particularly maintenance of their semantics and descriptive values, is an important and demanding task which compromises two distinctive sets of problems: i) problems with local variations and ii) problems with distributed variations. By their very nature tags carry small amounts of information, i.e. semantics, but have many variations which are the result of a large spectrum of issues like spelling, professional and cultural background, but also individual psychological processes [3]. Subsequently, a set of tags does not always have to correctly and consistently represent mental model of its every user. While these problems are related to the local frame of reference, the problems with distributed variations are resulting from the fact that the majority of tagging systems manage and interpret tag semantics in different ways. It is often difficult to find correlations between semantically similar but formally, i.e. lexically, different tags, and thus it becomes a challenge to connect and aggregate metadata from different tagging applications. 

All these issues are ubiquitous to the tagging process itself. Therefore, they also apply to the usage of annotated, or tagged, emotional stimuli in the fields of Affective Computing (AC) and Artificial Intelligence (AI). Broadly speaking, our work in AC is concerned with psychotherapy and psychological training. We aim to develop an integrated system for automated adaptation of virtual reality (VR) based scenarios driven by the subject's physiology, with a rationale regarding application of the system in the Posttraumatic stress disorder (PTSD) 
treatment [4][5]. A part of our efforts is directed at the design and development of the emotional stimuli generator which is able to actuate contextually and temporarily anticipated emotional responses in a human subject [6]. In order to do this the generator is able to display still images and video clips, generate specific sounds, and display synthetic virtual environments based on the control commands from the system's supervisor. In this process it is imperative that the commands result in emotionally and contextually aligned stimuli which are individually conformed to a specific subject and his present mental state. We are using International Affective Picture System (IAPS) and International Affective Digitized Sounds (IADS) databases which store annotated emotional stimuli. 

The databases use free-text keywords, or tags, to describe the meaning of individual stimuli. However, the keywords are semantically scattered, taxonomically disordered, and subsequently cumbersome to use or extract information from. 

In this paper we will explain how to introduce ontologybased tagging in the existing emotionally annotated databases to achieve twofold improvement: a) more efficient extraction of knowledge, and b) higher informative, i.e. semantically descriptive, stimuli data value. We will also demonstrate this concept with StimuliGenerator (StimGen) computer application that we have developed as a part of work on the scientific project 
"Adaptive Control of Scenarios in VR Therapy of PTSD"†. 

The next chapter introduces emotionally annotated databases, their benefits and drawbacks. The third chapter comments why it is necessary to move away from keywords, or tags, towards ontologies in description of emotionally annotated data. Construction of ontologies for description of emotional stimuli semantics is explained in the fourth chapter. Finally, merging of additional data like psychological model values and author's information with the developed ontology is explained in the fifth chapter. 