Funny & Jokes

A Robust Online Touch Pattern Recognition for Dynamic Human-robot Interaction

Description
A Robust Online Touch Pattern Recognition for Dynamic Human-robot Interaction
Categories
Published
of 9
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Related Documents
Share
Transcript
  Contributed Paper Manuscript received 07/15/10 Current version published 09/23/10 Electronic version published 09/30/10. 0098 3063/10/$20.00 © 2010 IEEE A Robust Online Touch Pattern Recognition for Dynamic Human-robot Interaction Young-Min Kim, Seong-Yong Koo, Jong Gwan Lim and Dong-Soo Kwon,  Member  , IEEE Abstract    — This paper presents a novel touch pattern recognition algorithm for dynamic proximate interaction between a robot and a human. At first, in order to guarantee reactive responses to various touch patterns, an online touch  pattern algorithm is proposed based on a Temporal Decision Tree(TDT). Second, dynamic movements of a robot in a real interaction situation usually deteriorate the confidence level of the pattern classifier. A robust method to compensate for inconsistent recognition results in the dynamic interaction is  proposed by a Consistency Index(CI), which estimates consistency degrees of human touch patterns over time. The algorithms are applied to a hard-cover touch recognition module, which is being developed for recognizing the four kinds of emotional touch patterns mainly used in human-robot affective interaction. The recognition performance is evaluated in a simple game scenario environment with KaMERo (KAIST  Motion Expressive Robot), which is an emotionally interactive robot platform. The results show that the proposed algorithm  guarantees commercially applicable recognition performance by compensating for the misclassification inherent in the dynamic movements of a robot  1 .   Index Terms — Emotional touch pattern, Temporal Decision Tree, Consistency Index, Dynamic human-robot interaction, Hard-cover robot I.   INTRODUCTION Touch is an important non-verbal channel for effectively exchanging information in proximate interaction between a human and a robot [1]. There have been extensive studies to utilize touch recognition systems to various robot applications such as teaching robot motions intuitively [17], controlling a robot using tactile commands [18], understanding social  behaviors of humans [19], conveying human’s emotional states [20] and establishing affective relationship between a robot and humans [21]. In particular, many researchers have emphasized that it is important to understand human’s affective gestures from 1  This research was performed for the Intelligent Robotics Development Program, one of the 21st Century Frontier R&D Programs funded by the Ministry of Commerce, Industry and Energy of Korea. Young-Min Kim is with the Mechanical Department of KAIST, Korea (e-mail: kimym@robot.kaist.ac.kr). Seong-Yong Koo is with the Mechanical Department of KAIST, Korea (e-mail: koosy@robot.kaist.ac.kr). Jon-Gwan Lim is with the Mechanical Department of KAIST, Korea (e-mail : limjg@robot.kaist.ac.kr) Dong-Soo Kwon is with the Mechanical Department and the Human-Robot Interaction Research Center of KAIST, Korea (e-mail: kwonds@kaist.ac.kr). touch [2]-[6]. Interaction by touch can be one of the effective communication methods for therapy robot. In the research [4] the experiments using therapy robot with touch interaction have shown the psychological and physiological improvements of elderly people in the care house. Touch interaction is also useful in the entertainment/educational robot areas [22] by providing tangible interface to increase immersion effect. In addition, there has been trying to detect human’s social gestures by touch interaction with a robot [16]. The research also shows that a touch interaction is involved with human’s social and affective interaction. There are two approaches to develop a touch recognition system: touch-receptor-based design (bottom-up approach) and touch-pattern-based design (top-down approach). The cases of in [3] and [5] are reciprocal counterparts to each other in terms of design processing. The touch system of therapeutic robot [3] is modeled on the human physiological sensor which features somatic receptors in human skin, and on the somatosensory system of humans. The somatic receptors such as touch, temperature, kinesthetic and pain in human skin are implemented by electric field/force sensors, thermistors,  potentiometers, and intense sensor signals, respectively, for these stimuli. On the other hand, the robot of [5] , which has  been developed to understand the unexpected physical Interference and intended contact between humans and the robots, identifies various touch gestures in advance and classifies those patterns by extracting features of tactile information like force, pressure, contact area, high-pressure area, number of contact points, etc. Previous studies have shown high recognition performance for various touch patterns. However, the hardware configuration is based on the soft-cover and, hence, not easy to apply to various commercial robotic platforms. In the service robot area, many robots are being developed with hard-covers for their functionality and ease of manufacture. Touch recognition systems for hard-cover robots have been less developed than those for soft-cover robots because it is hard to use various sensors due to the limitations of hardware. Our research focuses on developing a simply structured hard-cover touch recognition system to be applicable to typical service robot platforms and a  pattern recognizer to guarantee reliable performance in a real human-robot interaction (HRI) environment. We take into account two pragmatic issues so that the touch  pattern recognition can be applied to a real HRI. At first, In order to respond reactively with respect to human’s touch stimuli, online touch pattern recognition with a minimum time delay is required. However, it is difficult to detect starting and Y.-M. Kim et al.: A Robust Online Touch Pattern Recognition for Dynamic Human-robot Interaction1979   ending points of human’s continuous touch gestures and the temporal characteristics such as duration or repetition of each touch pattern are not identical. In order to achieve online recognition, Temporal Decision Tree (TDT) [7][8] is proposed  because it contains temporal conditions in the tree nodes to consider temporal properties of features and it is allowed to be executed at any time regardless of starting and ending points of the touch patterns. Second, in order for the touch pattern recognizer to be applicable in a dynamic HRI situation, the robot’s reaction to stimuli should be coherent with previous responses. However, a robot’s dynamic movements or gestures cause inconsistent recognition results. We assume that a normal person’s intention keeps constant during a certain time interval of communication with a robot. The Consistency Index (CI) is proposed to estimate consistency of human touch patterns by temporal coherence of each pattern and correlation between pattern classes. The CI is integrated with our online touch pattern algorithm to filter the inconsistent recognition results caused by the dynamic movements of a robot. The algorithms are applied to a hard-cover touch pattern module, which is being developed to recognize the four kinds of affective touch patterns mainly used in human-robot emotional interaction. The online and consistent recognition  performance of our system can be easily applied to various service robotic systems that are necessary for physical or  proximate interaction. The remainder of this work is organized as follows: In Section II, emotional touch patterns for proximate HRI are categorized. A hard-cover touch pattern recognition system is described in Section III. Section IV presents the online touch  pattern algorithm with TDT and Section V explains algorithms to improve the recognition performance using CI. Experiment results of our proposed algorithms are explained in Section VI and advanced issues to provide further applications are discussed in Section VII. II.   T OUCH P ATTERNS T AXONOMY FOR P ROXIMATE HRI In this section, we categorize feasible touch patterns for hard-cover robots and investigate which features can characterize each pattern.  A.   Touch Patterns Category Situations of physical interference and intended contact with robots are categorized in [5]. TABLE I shows 31 verbs to describe possible contacts with robots. TABLE I V ERBS RELATED TO CONTACT TO THE ROBOT  Touch Beat Pick Scrub Hit Push Pull Grasp Collide Thrust Tug Grip Smite Poke Draw Seize Pat Jab Drag Pinch Tap Jog Tweak Stroke Slap Nudge Scrape Scratch Punch Prod Rub In order to find the touch patterns that can be classified on the hard-cover robot, we investigated the definition of each verb in the Cambridge English Dictionary and found out features such as contact time, repetition, force, purpose, object to touch with, direction, and surface form. According to these feature similarities we categorize the verbs as shown in Fig. 1. TouchShort time No Repetition( Hit )With flat objects(Slap)With fist(Punch)High force(Collide)Destruction(Smite)With thin, sharp objects(Jab)Repetition( Beat ) Low force Noise generation(Tap)Pleasure(Stroke)With flat objects(Pat)Long timeSurface Normal to surface( Push )High force(Thrust)Low forceWith elbow(nudge)With arm, repeat(Jog)With pointed object(Poke, Prod, Pick)Tangential to surface( Rub )High forceDamage(Scrape)With nail(Scratch)Clean(Scrub)Volume No direction(Hold, Seize)High force(Grasp, Grip)With fingers(Pinch)Direction(Draw) Normal to surface(Pull)High force(Tug)Low force(Tweak)Tangential to surface(Drag)   Fig. 1. Touch classification according to feature similarity  B.   Selection of Touch Patterns for Proximate HRI Among the above 31 touch actions we choose four touch  patterns to be recognized: ‘hit’, ‘pat’, ‘push’ and ‘rub’. Touches to the object’s volume such as ‘pinch’ and ‘pull’ are not applicable in hard-cover robots so that those sub trees from volume in Fig. 1 are excluded from our concern. Four high classes of major touch patterns are ‘hit’, ‘beat’, ‘push’ and ‘rub’. In proximate HRI, however, touches are usually utilized for a human to provide a robot with his affective intention. Several touch words, including their affective meanings, can be selected. For instance, ‘hitting’ and ‘pushing’ are dominantly used for negative feedback, and ‘patting’ and ‘rubbing’ are for positive feedback. From this  perspective, ‘pat’ is more appropriate for describing affective intention than ‘beat’. Eventually, four kinds of touch patterns for proximate HRI are chosen as shown in TABLE II. Features for classifying the four touch patterns are defined from the definitions of patterns. As seen in Fig. 1, ‘hit’ and ‘pat’ are discriminated from ‘push’ and ‘rub’ by how long time a human touches the robot. The distinguishable feature for ‘hit’ and ‘pat’ is whether it is conducted repeatedly, and ‘push’ and ‘rub’ are different in the change of contact area,  because when a human pushes an object in the direction normal to the surface, the contact area does not change, while rub causes a change of contact area. 1980IEEE Transactions on Consumer Electronics, Vol. 56, No. 3, August 2010    TABLE II F EATURE S TATES OF E ACH T OUCH P ATTERN   Force Contact time Repeat Contact area change Hit M / H Short No No Pat L / M Short Yes No Rub L Long No Yes Push L / M Long No No L : Low force, M : Moderate force, H : High force III.   S YSTEM D ESCRIPTION   In this section, we describe the hardware and system architecture of our touch interface system in order for it to be used as the head of hard-cover service robots. (a). Round shape hard-cover (b). Sensors below the cover (c). Attachment to the KaMERo Fig. 2. Hardware of touch recognition system   The head of the service robot is covered with round-shaped  plastic as Fig. 2(a). From the survey results of the previous section, we need to detect four features; force, contact time, repetition, and contact area change. Because it is difficult to deform the surface or attach sensors to the cover, we decided to attach two non-contact sensors inside: 3x3 charge-transfer touch sensors and an accelerometer. A charge-transfer touch sensor can detect whether a human hand is within a short distance from the electrode. An accelerometer is used to calculate touch force by sensing the vibration of a hard-cover. Accelerometers can be attached inside of the cover, as shown in Fig. 2(b). Our system is easily and independently embedded in various service robot platforms. Fig. 2(c) shows the system embedded in the emotionally interactive robot KaMERo, which is being developed for more sophisticated emotional interaction with people in entertainment applications such as a game-playing. In this application, touch patterns are utilized for the robot to understand human’s rewards or punishments in the game results. Fig. 3 shows the system architecture. The microprocessor has three processes: a preprocessor to filter raw data and convert it into meaningful data; a feature extractor to make features from data; and a classifier. All sensors data are acquired per every 10ms, and touch patterns are classified every 40ms. 3X3 touch sensorsAccelerometer ADCPreprocessor Multi-windowingFeatureextractor TDTClassifier Sensors Microprocessor    Fig. 3. Real time and online system architecture   IV.   O NLINE T OUCH P ATTERN R  ECOGNITION BY T EMPORAL D ECISION T REE   Online and offline recognitions are distinguished by the recognition time as Fig. 4. Offline recognition determines the results after end point detection of the continuous touch stimulus while online recognition can output the results any time during continuous touch gestures. In this section, multi-windowing will be used for temporal feature extraction of four touch patterns and Temporal Decision Tree will be introduced to allow the online recognition using the temporal features. Fig. 4. Online and offline recognition    A.    Preprocessing Preprocessing is operated every 10ms to filter touch sensor data and store data to be manipulated for features in the next step. Input data are defined as follows, -    Nine touch sensor data : -   Acceleration data : As shown in Fig. 5, charges between human skin and electrodes of capacitor sensor depend on not only distance but also contact area. However, variation of skin contact area makes some high frequency noises. A moving average filter with 30 Hz of cutoff frequency is preprocessed for removing the noises. Now, the last m  filtered touch data and n  acceleration data are stored for the multi-windowing feature extraction. Output data of preprocessing is defined as follows: -   Last four filtered touch data : -   Last 20 acceleration data : 08 [](...) i Tki  = []  Ak  []~[]  fifi TkTkm − []~[] kAkn − Y.-M. Kim et al.: A Robust Online Touch Pattern Recognition for Dynamic Human-robot Interaction1981    DistanceDistance × (1/Contact Area)Time c th c th : Charging threshold10Actual signalFiltered signalThreshold   Fig. 5. Preprocessing of touch sensor signals  B.    Multi-windowing Feature Extraction There are four features to characterize each touch pattern: force, contact time, repetition, and contact area change. Each feature can be extracted by processing sensor data for different periods. Force is calculated by acceleration of cover’s vibration for 200ms, while contact area change is calculated for 40ms, and contact time / repetition are calculated for 10ms by touch sensor data. These three windows are moving forward every 10ms as shown in Fig. 6, so that we can get features in real time. Fig. 6. Multi-windowing feature extraction Force (  F  ) can be calculated as the variance of the last 20 acceleration data ])19[],...,1[],[var(][  −−= k  Ak  Ak  Ak  F   Touch area ( TA ) is a sum of nine touch data and whether there is contact or not ( C  ) is judged by values of TA.   ∑ = = 80 ][][ i fi k T k TA   ⎩⎨⎧ ≠= otherwisek TAif k C  ,00][,1 ][  Start point ( S  ) and end point (  E  ) can be detected by the difference between C  [ k  ] and C  [ k- 1]. ]1[][][  −−= k C k C k S    ][]1[][ k C k C k  E   −−=  Contact time ( CT  ) is the time difference between the current time and the start point time ( ST  ). ⎩⎨⎧ == otherwiseST k S if k  ST  ,1][,   ⎩⎨⎧ =−= otherwisek C if ST k  k CT  ,01][, ][   No contact time (  NCT  ) is the time difference between the current time and the end point time (  ET  ). ⎩⎨⎧ == otherwise ET k  E if k   ET  ,1][,   ⎩⎨⎧ =−= otherwisek C if  ET k  k  NCT  ,00][, ][  Repetition (  R ) is determined by short no contact time at start point. ⎩⎨⎧ =<= otherwisek S and T k  NCT if  k  R  REP  ,01][][,1 ][  Contact area change ( CAC  ) is calculated by the sum of last four contact area differences ( CAD ), which is the sum of touch differences ( TD ) as follows. ]1[][][  −−= k T k T k TD  fi fii   ∑ = = 80 ][][ ii k TDk CAD   ∑ = −= 30 ][][ i ik CADk CAC    Now,  F[k], CT[k], R[k],  and CAC[k]  are utilized to estimate Force, Contact time, Repetition, and Contact area change at time k  , respectively. TABLE II provides standard values of the four features to classify the four kinds of touch  patterns. Based on the standards in TABLE II, we define five thresholds to branch a current feature instance at each node of a decision tree classifier. Threshold values are obtained by averaging feature values of fifteen people’s touch pattern data (13 men, 2 women, 24~39 years of age) as shown in TABLE III. TABLE III T HRESHOLDS OF F EATURE V ALUES    LM   F     MH   F     L T     REP  T    cac  P   5 20 240 480 0.1  LM   F  : Threshold to discriminate between Low and Moderate force  MH   F  : Threshold to discriminate between Moderate and High force  L T  : Threshold to discriminate between Short and Long time  REP  T  : Threshold to discriminate whether stimuli are repeated or not cac  P  : Threshold to discriminate whether contact area changes or not C.   Temporal Decision Tree Classifier TABLE II shows that touch patterns can be classified by four feature values. However the classification based on TABLE II cannot be implemented at every time step  because it takes a different time to make each feature available for use and each touch pattern can be classified in a different amount of time. For example, hitting can be detected right after the touch occurs, while rubbing and  pushing take some time to be detected for extracting ‘long time’ feature. One way to solve this problem is to wait for the completion of all touch inputs and feature extractions and then judge, that is offline recognition. One objective of this research is to classify touch patterns online in order for a robot to recognize human’s touch pattern immediately without waiting until touch gesture ends. A temporal decision tree is proposed to classify touch patterns online. The temporal decision tree, which is an extension of the 1982IEEE Transactions on Consumer Electronics, Vol. 56, No. 3, August 2010   decision tree as time flows, includes temporal information at each node, so that it waits some time to be split into the next branch according to the feature value [7]. The temporal tree can be obtained from the temporal example table that describes features’ time flow according to each situation (touch pattern in this research) in which a specific touch occurs [8]. TABLE IV shows a temporal example table of four situations in which human inputs four touch patterns. The temporal decision tree is composed in Fig. 7, based on TABLE IV. Each node contains feature condition and time condition and the node determines the match of the condition after waiting as much as time condition. This waiting property allows accessible time difference among features. For example, ‘Contact area change’ feature can be calculated after rub time ( T   R ) so that ‘Push’ and ‘Rub’ can  be separated after waiting T   R  in the tree node. However the reason of CAC feature values before T   R  in the example table is another online property that the tree can be executed at any time. When the tree executes again during continuous rubbing, CAC value can be gathered before T   R   but it does not applied in the node decision before T   R . This waiting property makes the tree possible to determine the  patterns online with starting at any time. Fig. 7. Temporal Decision Tree The TDT classifier is executed as Fig. 8. If there is any touch stimulus and the features get out of the neutral states, TDT classifier begins. TDT stores sequence of feature vectors and tries to generate the recognition results every time step with waiting nodes. If TDT get any result, the result becomes the final output of recognition and TDT executes again if touch stimulus is still going on. If (Force, Contact, Repetition and CAC are not neutral) then Begin Result   Nothing While(Result == Nothing) do Begin Result   TDT(Force, Contact, Repetition, CAC) End Output   Result End Fig. 8. Pseudo code for execution of TDT TABLE IV T EMPORAL E XAMPLE T ABLE   Timesit1 sit2 time sit3 sit4 0 M / HL / M 0 L L / M1 M / HL / M 1 L L / M... M / HL / M ... L L / M Force T  end   M / HL / M T   L   L L / M0 1 1 0 1 1 1 1 1 1 1 1 ... 1 1 ... 1 1 T  end   0 0 T   L  1 1 Contact T  end  +50 T   R   1 1 0 0 1 0 0 0 1 0 1 1 0 0 ... 0 1 ... 0 0 Repetition T  end   0 0 T   L  0 0 0 NA NA 0 1 0 1 NA NA 1 1 0 ... NA NA ... 1 0 Contact area change T  end   NA NA T   R  1 0 CL Hit Pat Rub Push DL  L T     L T    T  end  : End time at which touch input terminates  T   L : Long time, boundary between short and long touch  T   R : Rub time from which rub can be recognized  NA: Not available   V.   C ONSISTENCY I NDEX - BASED D ECISION M AKER    A robot should be able to move dynamically to reactively express its emotional intentions during the proximate interaction. This situation usually deteriorates the confidence level of the pattern classifier in real interaction situations. However, it is not easy to model human pattern physical changes with a robot’s dynamic gestures. Bratman[9][10] defined that human intention as a main attitude that directs future planning, and Dennett[11] emphasized human intention as certain beliefs and desires to  behave in such a way as to further the human’s goals in light of its beliefs. Therefore, human behavior evoked from identical intentions tends to have consistent characteristics. Modeling human’s intention can compensate for the limited communication information between a human and a robot[12]. Instead of reflecting the dynamic properties of robot gestures to the pattern classifier, we compensate for this deficiency by utilizing consistency characteristics of human intention during real interaction. We can describe temporal  properties of human intention with a turn-taking scenario  based on a 20 questions game. In this scenario, a human usually gives the robot his true intention (red line in Fig. 9) such as rewards or punishments, and doesn’t change this intention while touching the robot. For instance, Fig. 9 shows that the human’s intention is maintained with positive rewards  by warming touch patterns such ‘pat’ and ‘rub’ for a short time interval, about three seconds. In a turn-taking scenario, the human’s actions and the robot’s actions are piled up in a certain time region, for instance, from 2 sec. to 3.5 sec. as shown in Fig. 9. This region frequently causes misclassification results. In service robot applications, such misclassifications become more critical in cases in which the evaluation of a human’s stimulus is the opposite of the true Y.-M. Kim et al.: A Robust Online Touch Pattern Recognition for Dynamic Human-robot Interaction1983
Search
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks