umu.sePublications
ReferencesLink to record
Permanent link

Direct link
Expressing emotions through vibration for perception and control
2010 (English)Doktorsavhandling, sammanläggning (Other academic)
Abstract [en]

This thesis addresses a challenging problem: “how to let the visually impaired ‘see’ others emotions”. We, human beings, are heavily dependent on facial expressions to express ourselves. A smile shows that the person you are talking to is pleased, amused, relieved etc. People use emotional information from facial expressions to switch between conversation topics and to determine attitudes of individuals. Missing emotional information from facial expressions and head gestures makes the visually impaired extremely difficult to interact with others in social events. To enhance the visually impaired’s social interactive ability, in this thesis we have been working on the scientific topic of ‘expressing human emotions through vibrotactile patterns’.

It is quite challenging to deliver human emotions through touch since our touch channel is very limited. We first investigated how to render emotions through a vibrator. We developed a real time “lipless” tracking system to extract dynamic emotions from the mouth and employed mobile phones as a platform for the visually impaired to perceive primary emotion types. Later on, we extended the system to render more general dynamic media signals: for example, render live football games through vibration in the mobile for improving mobile user communication and entertainment experience. To display more natural emotions (i.e. emotion type plus emotion intensity), we developed the technology to enable the visually impaired to directly interpret human emotions. This was achieved by use of machine vision techniques and vibrotactile display. The display is comprised of a ‘vibration actuators matrix’ mounted on the back of a chair and the actuators are sequentially activated to provide dynamic emotional information. The research focus has been on finding a global, analytical, and semantic representation for facial expressions to replace state of the art facial action coding systems (FACS) approach. We proposed to use the manifold of facial expressions to characterize dynamic emotions. The basic emotional expressions with increasing intensity become curves on the manifold extended from the center. The blends of emotions lie between those curves, which could be defined analytically by the positions of the main curves. The manifold is the “Braille Code” of emotions.

The developed methodology and technology has been extended for building assistive wheelchair systems to aid a specific group of disabled people, cerebral palsy or stroke patients (i.e. lacking fine motor control skills), who don’t have ability to access and control the wheelchair with conventional means, such as joystick or chin stick. The solution is to extract the manifold of the head or the tongue gestures for controlling the wheelchair. The manifold is rendered by a 2D vibration array to provide user of the wheelchair with action information from gestures and system status information, which is very important in enhancing usability of such an assistive system. Current research work not only provides a foundation stone for vibrotactile rendering system based on object localization but also a concrete step to a new dimension of human-machine interaction.

Place, publisher, year, pages
Umeå: Umeå universitet, Institutionen för tillämpad fysik och elektronik, 2010. 159 p.
Series
Digital Media Lab, ISSN 1652-6295 ; 12
Keyword [en]
Multimodal Signal Processing, Mobile Communication, Vibrotactile Rendering, Locally Linear Embedding, Object Detection, Human Facial Expression Analysis, Lip Tracking, Object Tracking, HCI, Expectation-Maximization Algorithm, Lipless Tracking, Image Analysis, Visually Impaired.
National Category
Signal Processing Computer Vision and Robotics (Autonomous Systems) Computer Science Telecommunications Information Science
Research subject
Computerized Image Analysis, Computing Science, Electronics, Systems Analysis
Identifiers
urn:nbn:se:umu:diva-32990 (URN)978-91-7264-978-1 (ISBN)oai:DiVA.org:umu-32990 (OAI)
Public defence
2010-04-28, Naturvetarhuset, N300, Umeå universitet, Umeå, Sweden, 09:00 (English)
Opponent
Supervisors
Projects
Taktil Video
Available from2010-04-07 Created:2010-04-06 Last updated:2010-04-20Bibliographically approved
List of papers
1. Turn your mobile into the football
Open this publication in new window or tab >>Turn your mobile into the football : rendering live football game by vibration
2008 (English)In: IEEE transactions on multimedia, ISSN 1520-9210, Vol. 10, no 6, 1022-1033Artikel i tidskrift (Refereed) Published
Abstract [en]

Vibration offers many potential benefits for the use of mobile phones. In this paper, we propose a new method of rendering live football game on mobile phones using vibration. A mobile phone is “synchronized” with the ball in the real field. By holding the phone, users are able to experience dynamic movements of the ball, to know attacking directions and which team is leading the attack. The usability test of our system shows that vibrotactile display is suitable for rendering live football information on mobile phones by adopting designed coding schemes with a right training process.

National Category
Physical Sciences
Research subject
Physics
Identifiers
urn:nbn:se:umu:diva-19256 (URN)10.1109/TMM.2008.2001352 (DOI)
Available from2009-03-05 Created:2009-03-05 Last updated:2010-04-20Bibliographically approved
2. Lipless tracking and emotion estimation
Open this publication in new window or tab >>Lipless tracking and emotion estimation
2007 (English)In: Proceedings of IEEE 3rd International Conference on Signal ImageTechnology & Internet based Systems , Shanghai, China: IEEE, 2007, 768-774Konferensbidrag (Refereed)
Abstract [en]

Automatic human lip tracking is one of the key components to many facial image analysis tasks, such as, lip-reading and emotion from lips. It has been a classical hard image analysis problem over decades. In this paper, we propose an indirect lip tracking strategy: ‘lipless tracking’. It is based on the observation that many of us don’t have clear lips and some even don’t have visible lips. The strategy is to select and localize stable lip features around the mouth for tracking. For this purpose deformable contour-segments are modelled based on lip features and tracking is done using dynamic programming and viterbi algorithm. The strength of proposed algorithm is demonstrated in emotion estimation domain. Finally, real-time video experiments performed on private and publicly available data sets (MMI face database) have shown the robustness of our proposed lipless tracking technique.

Shanghai, China: IEEE, 2007
National Category
Physical Sciences
Research subject
Physics
Identifiers
urn:nbn:se:umu:diva-19167 (URN)10.1109/SITIS.2007.102 (DOI)978-0-7695-3122-9 (ISBN)
Conference
3rd International Conference on Signal ImageTechnology & Internet based Systems, Shanghai, China, 2007
Available from2009-03-04 Created:2009-03-04 Last updated:2010-04-20Bibliographically approved
3. How to use manual labelers in evaluation of lip analysis systems?
Open this publication in new window or tab >>How to use manual labelers in evaluation of lip analysis systems?
2009 (English)In: Visual speech recognition Lip segmentation and mapping / [ed] Shilin W & Alan Liew, USA: IGI Global, 2009, 239-259Kapitel i bok, del av antologi (Other academic)
Abstract [en]

The purpose of this chapter is not to describe any lip analysis algorithms but rather to discuss some of the issues involved in evaluating and calibrating labeled lip features from human operators. In the chapter we question the common practice in the field: using manual lip labels directly as the ground truth for the evaluation of lip analysis algorithms. Our empirical results using an Expectation-Maximization procedure show that subjective noise in manual labelers can be quite significant in terms of quantifying both human and  algorithm extraction performance. To train and evaluate a lip analysis system one can measure the performance of human operators and infer the “ground truth” from the manual labelers, simultaneously.

USA: IGI Global, 2009
Keyword
Lip Analysis, Expectation Maximization Algorithm, Performance Evaluation
National Category
Physical Sciences Engineering and Technology
Research subject
Signal Processing, Computerized Image Analysis, Computing Science
Identifiers
urn:nbn:se:umu:diva-20300 (URN)978-160566186-5 (ISBN)
Available from2009-03-18 Created:2009-03-18 Last updated:2012-03-20
4. iFeeling
Open this publication in new window or tab >>iFeeling : Vibrotactile rendering of human emotions on mobile phones
2010 (English)In: Mobile multimedia processing fundamentals, methods, and applications / [ed] Xiaoyi Jiang, MatthewY. Ma, Chang Wen Chen, Heidelberg, Germany: Springer Berlin, 2010, 1-20Kapitel i bok, del av antologi (Other academic)
Abstract [en]

Today, the mobile phone technology is mature enough to enable us to effectively interact with mobile phones using our three major senses namely, vision, hearing and touch. Similar to the camera, which adds interest and utility to mobile experience, the vibration motor in a mobile phone could give us a new possibility to improve interactivity and usability of mobile phones. In this chapter, we show that by carefully controlling vibration patterns, more than 1-bit information can be rendered with a vibration motor. We demonstrate how to turn a mobile phone into a social interface for the blind so that they can sense emotional information of others. The technical details are given on how to extract emotional information, design vibrotactile coding schemes, render vibrotactile patterns, as well as how to carry out user tests to evaluate its usability. Experimental studies and users tests have shown that we do get and interpret more than one bit emotional information. This shows a potential to enrich mobile phones communication among the users through the touch channel.

Heidelberg, Germany: Springer Berlin, 2010 Edition: 1st Edition
Series
Lecture notes in computer science, ISSN 0302-9743 (print), 1611-3349 (online) ; 5960
Keyword
emotion estimation, vibrotactile rendering, lip tracking, mobile communication, tactile coding, mobile phone.
National Category
Signal Processing
Research subject
Computerized Image Analysis
Identifiers
urn:nbn:se:umu:diva-32998 (URN)10.1007/978-3-642-12349-8_1 (DOI)978-3-642-12348-1 (ISBN)
Available from2010-04-06 Created:2010-04-06 Last updated:2010-04-20Bibliographically approved
5. Vibrotactile rendering of human emotions on the manifold of facial expressions
Open this publication in new window or tab >>Vibrotactile rendering of human emotions on the manifold of facial expressions
2008 (English)In: Journal of Multimedia, ISSN 1796-2048, Vol. 3, no 3, 18-25Artikel i tidskrift (Refereed) Published
Abstract [en]

Facial expressions play an important role in every day social interaction. To enhance the daily life experience for the visually impaired, we present the Facial Expression Appearance vibroTactile System (FEATS), which uses a vibrotactile chair as the social interface for the visually impaired. An array of vibrating motors are mounted spatially on the back of an office chair. The Locally Linear Embedding (LLE) algorithm is extended to compute the manifold of facial expressions, which is used to control vibration of motors to render emotions. Thus, the chair could provide the visually impaired with on-line dynamic emotion information about the person he/she is communicating with. Usability evaluation of the system is carried out. The results are encouraging and demonstrate usability for the visually impaired. The user studies show that perfect recognition accuracy of emotion type is achieved by the FEATS.

Oulu, Finland: ACADEMY PUBLISHER, 2008
Keyword
locally linear embedding, vibrotactile interface, visually impaired, manifold of facial expressions
National Category
Signal Processing
Research subject
Signal Processing
Identifiers
urn:nbn:se:umu:diva-20236 (URN)
Available from2009-03-17 Created:2009-03-17 Last updated:2010-04-20Bibliographically approved
6. Vibrotactile rendering of head gestures for controlling electric wheelchair
Open this publication in new window or tab >>Vibrotactile rendering of head gestures for controlling electric wheelchair
2009 (English)In: Proceedings of IEEE international conference on systems, man and cybernetics, San Antonio, Texas, USA: IEEE, 2009, 413-417Konferensbidrag (Refereed)
Abstract [en]

We have developed a head gesture controlled electric wheelchair system to aid persons with severe disabilities. Real-time range information obtained from a stereo camera is used to locate and segment the face images of the user from the sensed video. We use an Isomap based nonlinear manifold learning map of facial textures for head pose estimation. Our system is a non-contact vision system, making it much more convenient to use. The user is only required to gesture his/her head to command the wheelchair. To overcome problems with a non responding system, it is necessary to notify the user of the exact system state while the system is in use. In this paper, we explore the use of vibrotactile rendering of head gestures as feedback. Three different feedback systems are developed and tested, audio stimuli, vibrotactile stimuli and audio plus vibrotactile stimuli. We have performed user tests to study the usability of these three display methods. The usability studies show that the method using both audio plus ibrotactile response outperforms the other methods (i.e. audio stimuli, vibrotactile stimuli response).

San Antonio, Texas, USA: IEEE, 2009
Keyword
extended Isomap, Multidimensional Scaling (MDS), head gesture recognition, vibrotactile rendering, wheelchair system, usability.
National Category
Signal Processing
Research subject
Computerized Image Analysis
Identifiers
urn:nbn:se:umu:diva-32993 (URN)10.1109/ICSMC.2009.5346213 (DOI)978-1-4244-2793-2 (ISBN)
Conference
IEEE international conference on systems, man and cybernetics, San Antonio, Texas, USA, 2009
Available from2010-04-06 Created:2010-04-06 Last updated:2010-04-20Bibliographically approved
7. Tongue operated electric wheelchair
Open this publication in new window or tab >>Tongue operated electric wheelchair
2010 (English)In: , 133-136Konferensbidrag (Other academic)
Abstract [en]

In this paper we propose a tongue operated electric wheelchair system to aid persons with severe disabilities. Real-time tongue gestures are detected and estimated from a video camera pointing to the face of the user. The tongue gestures are used to control the wheelchair. Our system is a non-contact vision system, making it much more convenient to use. The user is only required to move his/her tongue to command the wheelchair. To make the system easy to drive, the system is also equipped with a laser scanner for obstacle avoidance. We also mount a 2D array of vibrators on the chair to provide the user with the information of the response from his/her tongue movement and the status of the system. We are carrying out user tests to measure the usability of the system.

National Category
Signal Processing
Research subject
Computerized Image Analysis
Identifiers
urn:nbn:se:umu:diva-32999 (URN)
Conference
Swedish Symposium on Image Analysis
Available from2012-03-29 Created:2010-04-06 Last updated:2012-05-11Bibliographically approved

Open Access in DiVA

fulltext(2243 kB)1799 downloads
File information
File name FULLTEXT01.pdfFile size 2243 kBChecksum SHA-512
055c90aca71bb10376bf96663cc77efef29fb9d3075cdb58d6e5494f8635bbff4585dbc2d12ae5547b9216210a707d997cf757f5c01e7dcf73c3dddfb3ef6e78
Typ fulltextMimetype application/pdf

Search in DiVA

By author/editor
ur Réhman, Shafiq
By organisation
Department of Applied Physics and Electronics
Signal ProcessingComputer Vision and Robotics (Autonomous Systems)Computer ScienceTelecommunicationsInformation Science

Search outside of DiVA

GoogleGoogle Scholar
Totalt: 1799 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available
Totalt: 1520 hits
ReferencesLink to record
Permanent link

Direct link