Affective Rendering: Visual Effect Animations for Affecting User Arousal
Masaki Omata, Kazuya Moriwaki, Xiaoyang Mao, Daisuke Kanuka and Atsumi Imamiya Interdisciplinary Graduate School of Medicine and Engineering
University of Yamanashi Kofu, Japan
Abstract—We propose affective rendering as a new affective computing technology which deals with user emotion in the field of image synthesis. Affective rendering is realized in 3 steps - affect sensing, affect analysis and affect adapted rendering. This paper reports several basic experiments carried out for studying how the typical visual parameters, such as color and shape, affect view’s arousal, and also describes a prototyping of visual effect animations that can affect viewer’s arousal by adapting the visual parameters to user’s current arousal estimated from respiration and blood volume.
Keywords-affective rendering; visual effect; animation;
physiological data; arousal
I. INTRODUCTION
With the advance of image/video processing technologies as well as the popularization of multimedia devices, visual media has now become an important part of human life. We can enjoy visual contents at home, in a train or even when walking on the street. Sometimes, however, visual contents may be presented in a way beyond our necessity. Human- media interaction is a very important but difficult issue since there are so many factors may affect how we feel about a particular visual content. For example, a piece of beautiful video clip may help you feeling relaxing when you just accomplish a project after several days hard working, but the same clip may make you feel annoying when you are frustrated with a deadline. Adapting the visual contents to viewers’
physical and mental status is a very important issue directly contributing to the quality of life. How does a particular visual effect influence one’s emotion? What’s the proper visual effect for a particular person at a particular moment? This paper describes an elementary study aiming to answer those questions and proposes a new affective computing technology calledaffective rendering. Figure 1 depicts the frame work of affective rendering. It is a loop consists of 3 steps: affect sensing, affect analysis and affect adapted rendering. The affect sensing step collects viewer’s physiological data and feed them to affect analysis step for estimating the affect state of the viewer. Then at affect adapted rendering step, visual contents are generated by adjusting rendering parameters, such as color, shape and motion, according to the current as well as the expected affect state. We will call such kind of visual contentsaffective visual contentshereafter.
The concept of affective computing was first proposed by Rosalind Picard’s in 1997 [1] and is now gaining more and more attentions as an interdisciplinary field spanning computer sciences, psychology, and cognitive science [2]. In the field of computer science, most previous studies on affective computing are concerned with either the use of affect for evaluating the usability of products or visualizing the affect with computer graphics [3-9]. This paper address the issue of how to influence user’s affect by dynamically changing the parameters of visual contents according to viewer’s current affect state. The major contribution of this paper can be summarized as the follows:
The new concept of affective rendering which is an extension of the existing affective computing concept to the field of image synthesis.
A fuzzy system for estimating affect from physiological data.
The experiment for investigating the relationship between visual parameters and affect.
A prototype system for validating the framework of affective rendering.
A fundamental model in emotion research describes the affective space by two dimensions: Emotional valence, indicating the hedonic value (positive vs. negative), and arousal, indicating the emotional intensity (see Figure 2) [10].
For example,happycan be represented as the positive emotion while sad as the negative emotion, surprise as the strong emotion, andsleepyas the weak emotion. Mandryk et al. and
Figure 1. A conceptual diagram of affective rendering.
978-1-4673-1520-3/12/$31.00 ©2012 IEEE
Omata et al. have found that it is very difficult to steadily recognize valence [3][4]. Although the estimation of valence is a part of our final target, currently we reduce the affect space to one dimension and only consider the arousal. A fuzzy system has been developed for estimating the arousal value from physiological data (respiration rate and pulse wave).
Followed by a preliminary experiment investigating the relationship between visual parameters and arousal, we implemented a prototype system where affective visual content are presented to a user through an ambient interface and can subconsciously influence his/her arousal. Our experimental results show that by looping through the 3 steps, the proposed affective rendering technique succeeded in inducing the user toward the expected affect.
The remainder of the paper is organized as follows: Section 2 briefly reviews the related works. Section 3 discusses the design of the affective visual contents. Section 4 describes the preliminary experiment investigating the relationship between visual parameters and human affect. Section 5 describes the implementation of the prototype system together with the experiment for verifying the effectiveness of affective rendering. Section 6 concludes the paper.
II. RERATED WORK
As a pioneer work on the dynamic estimation of affect from physiological data, Mandryk et al. proposed a method for estimating the emotion state of a user involved in playing game based on galvanic skin response, electrocardiography, and electromyography of the face [3]. Based on Mandryk et al.’s method, Jones et al. proposed a fully automatic biometric emotion detection system using neural network and 11 dimensional feature vectors computed from blood volume pulse, skin conductance and respiration [11]. The system can detect arousal and valence at a resolution of 5*5. Their experiment showed that the degradability of emotion is not uniform along the valence axis. Similarly, Kim et al. divided emotional states into 9 different states in the Emotional Intelligent Contents framework which they developed [7].
While it is relatively easy to discriminate negative state from neutral state, the discrimination of positive state from neutral state is usually difficult. More recently, Wu et al. proposed to use support vector machine to classify the psychophysiological responses for estimating the arousal level [12]. We also use physiological data to estimate emotion. Specifically, we use respiration rate, pulse wave and skin conductance, which are showed to be effective in estimating emotion in those previous
studies. We have developed a fuzzy system for estimating the arousal from those physiological data
For the visualization of affect, Gebhard developed the pull and push mood change function that attracts a current mood towards a target mood or pushes it away by controlling gestures, facial expressions and so on of the virtual humans [9].
Omata et al. proposed an online text chat system where the shape and color of the text balloons changes according to the physiological signals of users [4]. Shugrina developed an emotion aware painterly rendering system where the color and brush styles of painterly images changes according to user’s affect detected from facial expression [13]. Ståhl et al. and Fagerberg et al. provided a mobile message service enabling a user to inform his/her emotion state to the other side by adjusting the color, texture and animation of background [5][6].
They established a mapping between the 2-dimensional affect space and the Munsell Color Space. We adopt the same mapping for designing the affective visual contents. Instead of visualizing affect, however, our method uses the color, shape and animation for influencing user’s affect, and inducing the user toward an expected arousal level.
III. DESIGN OFAFFECTIVEVISUALCONTENTS This section describes the design of visual contents which can be used for inducing the user toward an expected arousal level. We have designed two animation clips with different visual parameter settings. One is called arousing animation which is used for raising the arousal level, that is, making the viewer more excited, and the other is calledcalming animation, which is expected to calm the user down.
A. Color
We use Munsell color system for specifying colors. Figure 3 shows the mapping between color and affect, where positive emotion is mapped to Yellow (Y), negative emotion to Red (R), high arousal to Purple-Blue (PB), and low arousal to Green (G).
As we mentioned in Section 1, in this study, we consider the 1 dimension of arousal only. Therefore, we map warm colors such as red and orange to high arousal level and cool colors such as blue, aqua and green to low arousal level (See Figure 4).
B. Shape
Similar to Ståhl et al.’s approach [5], we use rounded shapes for arousing animation, and wavy shape for calming animation.
Figure 2. Lang’s arousal-valence 2D emotion model [10].
Figure 3. Relationship between emotion and colors.
C. Animations
To design the animation, we refer to music structure, which is known to be closely related to emotion. The factors of music structure include tempo, rhythm, pitch, loudness, melody, harmony, tonality etc. For example, regular and mellifluous rhythm expresses happiness, dignity and peace. Contrarily, irregular and disturbed rhythm expresses unrest, anger and lightness. We adopt the factors of tempo, rhythm, loudness and staccato in designing affective motion. Specifically, we map tempo to the speed of the motions, rhythm to the regularity of the motions, loudness to the size of the objects and staccato to blink (see TABLE 1).
D. Implementation
Based on the above described design principle, we created an “arousing animation” and a “calming animation” using Adobe Flash9 and actionscript3.0. The “arousing animation”
was created by referring to the Flash visualier released by gskiner.com:gBlog [14] and Clockmaker blog [15]. Figure 5 shows a snapshot of the “arousing animation,” in which red or orange circles placed on the surface of spheres keep rotating while expanding and shrinking strenuously. Figure 6 shows the snapshot of “calming animation”, which is expected to lower the level of arousal by representing calm and relaxations with slow periodical shift of waves in aqua or yellow-green colors.
The wave located at center moves from left to right, while the top and bottom waves move from right to left. Furthermore, together with the motion of the wave, a small number of particles are released.
IV. EVALUATION OF AFFECTIVE CONTENTS
This section describes the experiment to investigate how the two affective animations influence user affect by measuring users’ physiological data. We chose respiration, blood value pulse (BVP) and skin conductance, which have been used by other researcher for recognizing levels of arousal [4][7][8].
A. Physiological data
TABLE 2 summarizes the relationship between arousal levels and the variations of physiological data. The selected physiological data are known to be the indicators of human autonomic nervous system. The autonomic nervous system is involved in organ function such as respiration, circulation, digestion, metabolism and discharge and it consists of sympathetic nerve and parasympathetic nerve. Sympathetic nerve works for activization such as heart beat acceleration, blood pressure elevation, splanchnic vasocon-striction and bronchodilation more than parasympathetic nerve. On the other hand, parasympathetic nerve works for calming down such as secretagogue action of gastric secretion and saliva secretion, vasodilatation, and contraction of the pupil. Basically there are strong relationships between physiological data and arousal such as impatience and excitement.
Respiration rate increases when excited and decreases when relaxing. Of course, respiration rate increases by motion too. But we can exclude the factor of motion in our experiment as the task for subjects is to watch the affective animation in resting state.
BVP is the wave representing temporal change of volume of blood flow and can be used as an alternative to heart rate.
TABLE II. RELATIONSHIP BETWEEN AROUSAL AND PHYSIOLOGICAL DATA
High arousal Low arousal Respiration rate Increase Decrease
Pulse Increase Decrease
HF component Decrease Increase
LF/HF component Increase Decrease Skin conductance Increase Decrease
Figure 6. Calming animation.
Figure 5. Arousing animation.
TABLE I. RELATIONSHIP BETWEEN AROUSAL AND MOTION
Tempo (Speed)
Intensity (Size, Number)
staccato
(Blink) Rhythm High
Arousal Fast Big, Many Applicable Irregular Low
Arousal Slow Small, Few Not
applicable Regular Figure 4. Relationship between emotion and shapes.
When parasympathetic nerve is dominant over sympathetic nerve, one is in relaxed state and cardiac function is suppressed, and vice versa, if sympathetic nerve is dominant over parasympathetic nerve, one is in strained state and the cardiac function is increased. For further analysis, the spectrum of Heart Rate Variability (HRV) is used. Specifically, it is known that LF (Low Frequency) component of HRV, under 0.15 Hz, is related to both sympathetic and parasympathetic nerve, and the HF (High Frequency) component above 0.15 Hz corresponds to parasympathetic nerve only. In our implementation, we first transform BVP to frequency domain, decompose it into LF and HF components and then compute the ratio of power of LF and HF.
Skin conductance increases at excited or stressed state, and decreases in a relaxed state. This phenomenon is caused by the reaction of eccrine sweat gland on palms or feet bottoms.
Eccrine sweat gland responds to not only body temperature change but also physiological stimulation.
B. Experiment Setting
In the experiment, a 50 inches monitor (SANYO, LP- RX50) is used to display the animation. Physiological data is collected with a multi-modality encode (ThoughtTechnolgy ltd., ProCmp-Infiniti). Figure 7 shows the subjects wearing the physiological sensors. Figure 7(a) shows subject’s hand attached with the electrode of a skin conductance sensor and the IR sensor of a BVP sensor. Figure 7(b) is a subject wearing the respiration sensor on waist.
C. Task and procedure
As the task, subjects are asked to watching the two affective animations described in Section 3 for 5 minutes.
Before starting a task, physiological data in normal state are measured and recorded for three minutes and the differences between the normal state and the state when involving in the trial are calculated to evaluate the effect of the animations. In the normal state, subjects were asked to watch the default desktop wallpaper of Windows Vista in order to prevent the extremely-relaxed state occurring when a subject closes his/her eyes. In order to eliminate the factor of the order, half of the subjects watched arousing animation first while the other half watched the calming animation first.
D. Results
Sixteen subjects participated in the experiment. They are undergraduate students and all are accustomed to computer
environment. Figure 8(a) shows the per minute ratio of respiration rate between the normal state and the state when watching the affective animation. Significant difference are found between the normal state and when watching the arousing animation (p < 0.05). The number of respiration increased when watching the arousal animation. On the other hand, a significant difference between the normal state and the calming animation is also found (p < 0.05). The number of respiration decreased when watching the calming animation.
For the number of pulses, there is no significant difference among normal state and the states when watching arousal animation and calming animation. Also, there is no significant difference among normal state and the states of watching arousal animation and watching calming animation in case of the LF/HF ratio of heart rate variability (p < 0.05) (see Figure 8(b)). On the other hand, a significant difference between the normal state and the arousing animation is found for HF (p <
0.05) (see Figure 8(c)). HF increases when subjects were watching the calming animation.
E. Discussions
The experiment shows that there are significant differences about respiration and HF with the simple task. Especially, there is remarkable difference in respiration. We found that the speed of subjects’ respiration changes with the speed of objects in the animations. Therefore we can conclude that it is possible to control the respiration, which is an important indicator of human affect, by controlling the parameters of visual contents.
V. AFFECTIVE RENDERING SYSTEM A. System organization
A wall size screen is placed on the back of the working display for showing animation. It allows the user to see the animation with peripheral vision without being distracted from
(a) Respiration (Mean+SD) (b) LF/HF (Mean+SD) (c) HF (Mean+SD)
Figure 8. Physiological data of subjects watching affective animation (relative to the value in normal state).
Figure 7. Sensors for recording skin conductance, blood volume pulse and respiration movement.
working. The animation on the wall size screen is automatically changed according to user’s arousal levels which are estimated from the physiological data of the user. The software configuration consists of the affect sensor, affect analyzer and affect adapted renderer. The affect sensor measures respiration and BVP, and calculates HF and LF/HF ratio. The affect analyzer recognizes user’s arousal in 10 levels which corresponds to the vertical axis of Lang’s two- dimensional emotion model. Since we only designed two types of affective animations, the current system supports 3 arousal levels, low, neutral and high. However, for the future extension, we have implemented a fuzzy system which can recognize 10 arousal levels. Thus, the user’s arousal is considered to be high when the level is above 5, to be low when the level is lower than five. The level 5 refers to the neutral arousal. The affect adapted renderer shows either the arousing animation or the calming animation on the wall size screen according to the arousal level from the affect analyzer as well as affective filters.
Affective filters are used for specifying the relationship between current arousal state and the expected state. Two affective filters, elevation and inhibition, are supported. The elevation filter induces the user toward an arousal level in the same direction of current level while the inhibition filter changes the arousal level toward an opposite direction. For example, if a user choose elevation filter when the arousal level is high, renderer will show the arousing animation so as to further raise the arousal level. On the other hand, when the current arousal level is low and the inhibition filter is specified, then the render will shows the arousal animation in order to raise arousal level. The next section describes the detail of the affect analyzer.
B. Affect analyzer
In our system, the analyzer estimates the levels of arousal by analyzing respiration, HF and LF/HF of BVP. The system uses fuzzy inference like Mandryk’s method to estimate the affect [3]. The more the number of respiration increases, the high the arousal level is. In addition, the more HF of BVP increases, the high the arousal is. We developed the fuzzy rules as follows;
IF(RespisHigh)OR(HFisLow)OR(LF=HFisHigh)
THEN(ArousalisHigh)
IF(RespisMid)OR(HFisMid)OR(LF=HFisMid)
THEN(ArousalisMid)
IF(RespisLow)OR(HFisHigh)OR(LF=HFisLow)
THEN(ArousalisLow)
To implement the above membership rule, we need to know the maximum varying rate of physiological data in the most relaxed and excited state against the normal state. Since it is relatively easy for one to become excited when playing video game, we obtain the maximum varying rate at excited state by computing the difference between the physiological data measured when playing racing game and the physiological data measured when in normal state. For the maximum varying rate in the most relaxed state, we measured the physiological data when subjects sit on a sofa and close eyes.
Figure 9 shows a diagram illustrating the fuzzy system in relation to the membership functions. In this way, each membership function receives raw physiological data and calculates a grade. Then, the fuzzy system calculates an arousal level based on the fuzzy rules by using the grades from the membership functions. The maximum variation rates are calculated by assuming the value in normal state to be 1.0.
C. Verification experiment
We conducted an experiment to verify the fuzzy estimation and the effectiveness of the affective rendering system. In the experiment, our system shows either of the two affective animations described in Section 3 every three minutes, while a subject is playing a video game (Break- Out360 °http://www.fujiyose.com/play-53.htm). Each trial last twelve minutes and the arousal level of the subject is measured by using their physiological data in the fuzzy system. Before starting a trial, physiological data in normal state are recorded for three minutes to calculate a neutral arousal level. Figure 10 shows a picture of the experiment environment setting. A total of 20 subjects are grouped by ten subjects to verify the elevation filter and the inhibition filter, respectively.
Figure 11 and 12 show the temporal transitions of the fuzzy estimation values of 10 subjects with the elevation filter and the inhibition filter, respectively. The solid lines represent the period when arousing animation is displayed on the background, and the dashed lines denote the period of calming animation.
It is clear from the Figure 11 and 12 that the arousal levels of the subjects with the elevation filter were maintained at constant levels from 5 to 7, while on the other hand, the arousal levels of the subjects with the inhibition filter went up and down. The result demonstrates that the elevation filter works to maintain subjects’ arousals at high level throughout the trial. Additionally, the results show that the inhibition filter works for maintaining subjects’ arousals at their neutral levels.
Figure 10. Experiment environment.
Figure 9. Modeling arousal from physiological data.
For analyzing the effects of the filters, we calculated the averages within groups by quantifying the effect of each filter with a signed number. If the arousal changed in the expected direction, then we assign a positive number and vice verse we assign a negative number. The absolute value of the number is the amount of arousal variation from the previous state. As shown in Figure 13, we obtained a positive average value for both the elevation filter and inhibition filtered. For the group of elevation filter, 5 out of 10 subjects showed a positive number and all the 10 subjects showed a positive number for the inhibition filter group.
From these results, we observed that the two proposed animations are effective for influencing user’s arousal within a certain range from the neutral level when inhibition filter is used. However, we found that the arousing animation cannot further raise a subject’s arousal when the subject is already in a relative high arousal level. Arousal of some subjects in the elevation filter group continued to get down from high arousal levels.
To summarize, the experimental results showed that it is possible to achieve a control over the arousal with the affective animations, but for more effective control, affective contents with large variations of visual parameters are required.
VI. CONCLUSIONS
We have explored the effectiveness of applying the concept of affective computing to the field of image synthesis. Through
simple examples and evaluation experiments, we have showed how to create affective visual contents by considering the mapping between visual parameters and arousal level, how to estimate arousals from different physiological date using fuzzy system, how to induce users toward expected arousal, and demonstrated the potential of using the affective rendering techniques for achieving a better human-media interaction environment.
For future work, we plan to explore the relationship between large variety of visual parameters and human affect and create general guidelines for creating affective visual contents. Currently our fuzzy system can only recognize 10 levels of arousal. Developing a technique for higher resolution recognition in the full 2D affect space is one of our major future works.
ACKNOWLEDGMENT
This work was supported by Grant-in-Aid for Scientific Research (B) (21300033).
REFERENCES
[1] R. W. Picard. “Affective Computing,” The MIT PRess, 1997.
[2] R. W. Picard, “Affective computing: challenges,” International Journal of Human-Computer Studies 59, pp. 1-2, 2003.
[3] R. L. Mandryk, M. S. Atkins, and K. M. Inkpen, “A continuous and objective evaluation of emotional experience with interactive play environments,” In Proceedings of the CHI 2006, pp. 22–27, 2006.
[4] M. Omata, Y. Naito, and A. Imamiya, “Design and evaluation of an online chat system using speech balloons changed by user’s biomedical signals,” Human Interface, 10(2), pp. 179–189, May 2008 (in Japanese).
[5] A. Ståhl, P. Sundstrom, and K. Hook, “A foundation for emotional expressivity,” In Proceedings of the 2005 conference on Designing for User eXperience, pp. 33–54, 2005.
[6] P. Fagerberg, A. Ståhl, and K. Höök, “eMoto: emotionally engaging interaction,” In Journal of the Personal and Ubiquitous Computing, pp.
377-381, 2004.
[7] M. Kim, K. S. Park, D. Kim, and Y. Cho, “Emotional Intelligent Contents: Expressing User’s Own Emotion within Contents,” In Proceedings of the ICEC 2011, pp. 391-394, 2011.
[8] S. Sundar, and S. Kalyanaraman, “Arousal, Memory, and Impression- formation Effects of Animation Speed in Web. Advertising,” Journal of Advertising, 33(1), pp. 7-17, 2004.
[9] P. Gebhard, “ALMA—a layered model of affect,” in Proceedings of the AAMAS ‘05, pp. 177–184, 2005.
[10] P. J. Lang, “The emotion probe. studies of motivation and attention,”
American Psychologist, 50(5), pp. 372–385, May 1995.
[11] M. Jones and T. Troen, “Biometric valence and arousal recognition,” In Proceedings of the 2007 conference of the computer-human interaction special interest group (CHISIG) of Australia on Computer- humaninteraction, pp. 191–194, 2007.
[12] D. Wu, C. G. Courtney, B. J. Lance, S. S. Narayanan, M. E. Dawson, K. S. Oie, and T. D. Parsons, “Optimal Arousal Identification and Classification for Affective Computing Using Physiological Signals:
Virtual Reality Stroop Task,” Virtual Reality 1, pp. 109–118, 2010.
[13] M. Shugrina, M. Betke, and J. P. Collomosse, “Empathic painting:
Interactive stylization using observed emotional state,” In Proceedings of the 4th Intl. Symposium on NPAR 2006, pp. 87–96, 2007.
[14] gskinner.com:gblog.
http://www.gskinner.com/blog/archives/2008/10/simple_ ash_pl.html.
[15] Clockmaker blog. http://clockmaker.jp/blog/2008/10/pv3d_visualizer/.
Figure 13. Quantitative measure of filter effects.
Figure 12. Changing of arousal with the inhibiting filter.
Figure 11. Changing of arousal with elevation filter.
