Science

Declarative Gesture Spotting Using Inferred and Refined Control Points

Description
Presentation given at ICPRAM 2013, International Conference on Pattern Recognition, Barcelona, Spain, February 2013 ABSTRACT: We propose a novel gesture spotting approach that offers a comprehensible representation of automatically inferred spatiotemporal constraints. These constraints can be defined between a number of characteristic control points which are automatically inferred from a single gesture sample. In contrast to existing solutions which are limited in time, our gesture spotting approach offers automated reasoning over a complete motion trajectory. Last but not least, we offer gesture developers full control over the gesture spotting task and enable them to refine the spotting process without major programming efforts. Paper:http://www.academia.edu/2157798/Declarative_Gesture_Spotting_Using_Inferred_and_Refined_Control_Points
Categories
Published
of 83
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Related Documents
Share
Transcript
  • 1. Declarative Gesture Spotting UsingInferred and Refined Control PointsDepartment of Computer Science, Vrije Universiteit Brussel (VUB) Lode Hoste - lhoste@vub.ac.be Brecht De Rooms - bderooms@vub.ac.be Beat Signer - bsigner@vub.ac.be 1
  • 2. Gesture Classification template sample segment e.g. Dynamic Time Warping 2
  • 3. Gesture Classification template sample segment e.g. Dynamic Time Warping 3
  • 4. Gesture Classification 4
  • 5. Gesture Classification 5
  • 6. Gesture Classification 6
  • 7. Gesture Classification 7
  • 8. Gesture Classification 8
  • 9. Gesture Classification Too Complex 9
  • 10. Gesture ClassificationStart? We need gesture spotting End? 10
  • 11. Gesture ClassificationStart? We need gesture spotting reduce amount of work for gesture classification End? 11
  • 12. Gesture SpottingGoal: - High Recall - High PrecisionPopular techniques: - Hidden Markov Models - Continuous Dynamic Programming 12
  • 13. Our Declarative Gesture Spotting Approach 1. Automatically infer control points - from one representative sample - one shot learning 2. Generate declarative codeSpotted? - understandable and extensible spotting definitions 3. Call a gesture classification algorithm 13
  • 14. Automatically Infer Control PointsOne representative sample Z gesture 14
  • 15. Automatically Infer Control PointsOne representative sample - start with the first point Z gesture 15
  • 16. Automatically Infer Control PointsOne representative sample - start with the first point - search for the smallest angle - within a given time-frame Z gesture 16
  • 17. Automatically Infer Control PointsOne representative sample - start with the first point - search for the smallest angle - within a given time-frame Z gesture 17
  • 18. Automatically Infer Control PointsOne representative sample - start with the first point - search for the smallest angle - within a given time-frame - refine control points (optionally) Z gesture 18
  • 19. Comprehensible Code GenerationBased on the inferred control points: 19
  • 20. Comprehensible Code Generation Based on the inferred control points:1 (defrule symbol_z2 ?p1 (Point2D) there is a start point ?p1 20
  • 21. Comprehensible Code Generation Based on the inferred control points:1 (defrule symbol_z2 ?p1 (Point2D)3 ?p2 (Point2D) there is a second point ?p1 ?p2 21
  • 22. Comprehensible Code Generation Based on the inferred control points:1 (defrule symbol_z2 ?p1 (Point2D)3 ?p2 (Point2D) ?p1 ?p24 (test (< ?p1.time ?p2.time)) temporal constraint: that comes after the start point 22
  • 23. Comprehensible Code Generation Based on the inferred control points:1 (defrule symbol_z2 ?p1 (Point2D)3 ?p2 (Point2D) ?p1 ?p24 (test (< ?p1.time ?p2.time))5 (test (inside_control_point ?p1 ?p2 185 4 76)) spatial constraint: 23
  • 24. Comprehensible Code Generation Based on the inferred control points:1 (defrule symbol_z2 ?p1 (Point2D)3 ?p2 (Point2D) ?p1 ?p24 (test (< ?p1.time ?p2.time))5 (test (inside_control_point ?p1 ?p2 185 4 76)) 4 185 spatial constraint: .. a circular area with centerpoint x: 185 y:4 (pixels) 24
  • 25. Comprehensible Code Generation Based on the inferred control points:1 (defrule symbol_z2 ?p1 (Point2D)3 ?p2 (Point2D) ?p1 ?p24 (test (< ?p1.time ?p2.time))5 (test (inside_control_point ?p1 ?p2 185 4 76)) 4 185 spatial constraint: .. a circular area with centerpoint x: 185 y:4 (pixels) and radious 76 25
  • 26. Comprehensible Code Generation Based on the inferred control points:1 (defrule symbol_z2 ?p1 (Point2D)3 ?p2 (Point2D) ?p1 ?p24 (test (< ?p1.time ?p2.time))5 (test (inside_control_point ?p1 ?p2 185 4 76)) 4 185 26
  • 27. Comprehensible Code Generation Based on the inferred control points:1 (defrule symbol_z2 ?p1 (Point2D)3 ?p2 (Point2D) ?p1 ?p24 (test (< ?p1.time ?p2.time))5 (test (inside_control_point ?p1 ?p2 185 4 76)) 46 ?p3 (Point2D) 1857 (test (< ?p2.time ?p3.time))8 (test (inside_control_point ?p1 ?p3 15 178 76))9 ?p4 (Point2D)10 (test (< ?p3.time ?p4.time))11 (test (inside_control_point ?p1 ?p4 197 175 76)) ?p3 ?p4 27
  • 28. Comprehensible Code Generation Based on the inferred control points:1 (defrule symbol_z2 ?p1 (Point2D)3 ?p2 (Point2D) ?p1 ?p24 (test (< ?p1.time ?p2.time))5 (test (inside_control_point ?p1 ?p2 185 4 76))6 ?p3 (Point2D)7 (test (< ?p2.time ?p3.time))8 (test (inside_control_point ?p1 ?p3 15 178 76))9 ?p4 (Point2D)10 (test (< ?p3.time ?p4.time))11 (test (inside_control_point ?p1 ?p4 197 175 76)) each point expressed relative to ?p1 ?p3 ?p4 28
  • 29. Comprehensible Code Generation Based on the inferred control points:1 (defrule symbol_z2 ?p1 (Point2D)3 ?p2 (Point2D) ?p1 ?p24 (test (< ?p1.time ?p2.time))5 (test (inside_control_point ?p1 ?p2 185 4 76))6 ?p3 (Point2D)7 (test (< ?p2.time ?p3.time))8 (test (inside_control_point ?p1 ?p3 15 178 76))9 ?p4 (Point2D)10 (test (< ?p3.time ?p4.time))11 (test (inside_control_point ?p1 ?p4 197 175 76)) temporal constraints: the points should be ordered ?p3 ?p4 29
  • 30. Comprehensible Code Generation Based on the inferred control points:1 (defrule symbol_z2 ?p1 (Point2D)3 ?p2 (Point2D) ?p1 ?p24 (test (< ?p1.time ?p2.time))5 (test (inside_control_point ?p1 ?p2 185 4 76))6 ?p3 (Point2D)7 (test (< ?p2.time ?p3.time))8 (test (inside_control_point ?p1 ?p3 15 178 76))9 ?p4 (Point2D)10 (test (< ?p3.time ?p4.time))11 (test (inside_control_point ?p1 ?p4 197 175 76))12 =>13 (call DynamicTimeWarping14 (select-between ?p1.time ?p4.time)15 (gesture-set “CharacterZ”)) ?p3 ?p4 call an external algorithm for classification 30
  • 31. A State Machine is Not Enough template continuous stream State 1State 1 State 2State 3 State 4 31
  • 32. A State Machine is Not Enough template continuous stream State 2State 1 State 2State 3 State 4 32
  • 33. A State Machine is Not Enough template continuous stream State 2State 1 State 2 reset state machine?State 3 State 4 33
  • 34. A State Machine is Not Enough template continuous stream State 1State 1 State 2 reset state machine?State 3 State 4 34
  • 35. A State Machine is Not Enough template continuous stream State 1State 1 State 2State 3 State 4 35
  • 36. A State Machine is Not Enough template continuous stream State 1State 1 State 2State 3 State 4 36
  • 37. A State Machine is Not Enough template continuous streamState 1 State 2 gesture overshot  missed gestureState 3 State 4 37
  • 38. A State Machine is Not Enough template continuous streamState 1 State 2State 3 State 4 38
  • 39. A State Machine is Not Enough template continuous streamState 1 State 2 refused by gesture classificationState 3 State 4 39
  • 40. A State Machine is Not Enough template continuous streamState 1 State 2 refused by gesture classificationState 3 State 4 overlapping gesture  missed gesture 40
  • 41. Incremental Evaluation of Every Possible Combination 1 (defrule symbol_z 2 ?p1 (Point2D) 41
  • 42. Incremental Evaluation of Every Possible Combination 1 (defrule symbol_z 2 ?p1 (Point2D) 3 ?p2 (Point2D) 4 (test (< ?p1.time ?p2.time)) 5 (test (inside_control_point ?p1 ?p2 185 4 76)) 42
  • 43. Incremental Evaluation of Every Possible Combination 1 (defrule symbol_z 2 ?p1 (Point2D) 1 (defrule symbol_z 2 ?p1 (Point2D) 3 ?p2 (Point2D) 4 (test (< ?p1.time ?p2.time)) 5 (test (inside_control_point ?p1 ?p2 185 4 76)) 43
  • 44. Incremental Evaluation of Every Possible Combination 1 (defrule symbol_z 2 ?p1 (Point2D) 1 (defrule symbol_z 2 ?p1 (Point2D) 3 ?p2 (Point2D) 4 (test (< ?p1.time ?p2.time)) 5 (test (inside_control_point ?p1 ?p2 185 4 76)) 44
  • 45. Incremental Evaluation of Every Possible Combination 1 (defrule symbol_z 2 ?p1 (Point2D) 1 (defrule symbol_z 2 ?p1 (Point2D) 3 ?p2 (Point2D) 1 (defrule symbol_z 2 ?p1 (Point2D) 3 ?p2 (Point2D) 4 (test (< ?p1.time ?p2.time)) 5 (test (inside_control_point ?p1 ?p2 185 4 76)) 45
  • 46. Incremental Evaluation of Every Possible Combination 1 (defrule symbol_z 2 ?p1 (Point2D) 1 (defrule symbol_z 2 ?p1 (Point2D) 3 ?p2 (Point2D) 1 (defrule symbol_z 2 ?p1 (Point2D) 1 (defrule symbol_z 3 ?p2 (Point2D) 2 ?p1 (Point2D) 3 ?p2 (Point2D) 4 (test (< ?p1.time ?p2.time)) 5 (test (inside_control_point ?p1 ?p2 185 4 76)) 46
  • 47. Incremental Evaluation of Every Possible Combination 1 (defrule symbol_z 2 ?p1 (Point2D) 1 (defrule symbol_z 2 ?p1 (Point2D) 3 ?p2 (Point2D) 1 (defrule symbol_z 2 ?p1 (Point2D) 1 (defrule symbol_z 3 ?p2 (Point2D) 2 ?p1 (Point2D) 3 ?p2 (Point2D) 4 (test (< ?p1.time ?p2.time)) 5 (test (inside_control_point ?p1 ?p2 185 4 76)) 47
  • 48. Incremental Evaluation of Every Possible Combination 1 (defrule symbol_z 2 ?p1 (Point2D) 1 (defrule symbol_z 2 ?p1 (Point2D) 3 ?p2 (Point2D) 1 (defrule symbol_z 2 ?p1 (Point2D) 1 (defrule symbol_z 3 ?p2 (Point2D) 2 ?p1 (Point2D) 3 ?p2 (Point2D) 4 (test (< ?p1.time ?p2.time)) 5 (test (inside_control_point ?p1 ?p2 185 4 76)) RETE engine takes care of caching intermediate calculations 48
  • 49. Incremental Evaluation of Every Possible Combination ?p1 49
  • 50. Incremental Evaluation of Every Possible Combination ?p1 ?p1 50
  • 51. Incremental Evaluation of Every Possible Combination ?p1 ?p1 ?p1 51
  • 52. Incremental Evaluation of Every Possible Combination ?p1 ?p1 ?p1 52
  • 53. Incremental Evaluation of Every Possible Combination ?p1 ?p1 ?p1 53
  • 54. Incremental Evaluation of Every Possible Combination ?p1 ?p1 ?p1 ?p1 54
  • 55. Incremental Evaluation of Every Possible Combination ?p2 ?p1 ?p1 ?p1 ?p1 55
  • 56. Incremental Evaluation of Every Possible Combination ?p2 ?p1 ?p1 ?p1 ?p1 ?p1 56
  • 57. Incremental Evaluation of Every Possible Combination ?p2 ?p1 ?p1 ?p1 ?p1 ?p3 ?p1 57
  • 58. Incremental Evaluation of Every Possible Combination ?p2 ?p1 ?p1 ?p1 ?p1 ?p3 ?p1 58
  • 59. Incremental Evaluation of Every Possible Combination ?p2 ?p1 ?p1 ?p1 ?p1 ?p3 ?p1 59
  • 60. Incremental Evaluation of Every Possible Combination ?p2 ?p1 ?p1 ?p1 ?p1 ?p1 ?p3 ?p1 60
  • 61. Incremental Evaluation of Every Possible Combination ?p2 ?p1 ?p1 ?p1 ?p2 ?p1 ?p1 ?p3 ?p1 61
  • 62. Incremental Evaluation of Every Possible Combination ?p2 ?p1 ?p1 ?p1 ?p2 ?p1 ?p1 ?p3 ?p3 ?p1 62
  • 63. Incremental Evaluation of Every Possible Combination ?p2 ?p1 ?p1 ?p1 ?p2 Limit calculations to a sliding window ?p1 ?p1 ?p3 ?p3 ?p1 63
  • 64. Incremental Evaluation of Every Possible Combination ?p2 ?p1 ?p1 ?p1 ?p2 Limit calculations to a sliding window ?p1 ?p1 ?p3 ?p3 64
  • 65. Incremental Evaluation of Every Possible Combination ?p2 ?p1 ?p1 ?p1 ?p2 Limit calculations to a sliding window ?p1 ?p3 ?p3 65
  • 66. Incremental Evaluation of Every Possible Combination ?p2 ?p1 ?p1 ?p1 ?p2 Limit calculations to a sliding window ?p1 ?p3 ?p1 ?p3 66
  • 67. Incremental Evaluation of Every Possible Combination ?p2 ?p1 ?p1 ?p1 ?p2 Limit calculations to a sliding window ?p1 ?p3 ?p1 ?p4 ?p3 67
  • 68. Incremental Evaluation of Every Possible Combination ?p2 ?p1 ?p1 ?p1 ?p2 Limit calculations to a sliding window ?p3 ?p1 ?p4 ?p3 68
  • 69. Incremental Evaluation of Every Possible Combination ?p2 ?p1 ?p1 ?p1 ?p2 Limit calculations to a sliding window ?p3 ?p1 ?p4 ?p3 69
  • 70. Incremental Evaluation of Every Possible Combination ?p2 ?p1 ?p1 ?p1 ?p2 Limit calculations to a sliding window ?p3 ?p1 ?p4 ?p3 70
  • 71. Incremental Evaluation of Every Possible Combination ?p2 ?p1 ?p1 ?p1 ?p2 Limit calculations to a sliding window ?p2 ?p3 ?p1 ?p4 ?p3 71
  • 72. Incremental Evaluation of Every Possible Combination ?p2 ?p1 ?p1 ?p1 ?p2 Limit calculations to a sliding window ?p2 ?p3 ?p1 ?p4 ?p3 72
  • 73. Incremental Evaluation of Every Possible Combination ?p2 ?p1 ?p1 ?p1 ?p2 Limit calculations to a sliding window ?p2 ?p3 ?p1 ?p4 ?p3 73
  • 74. Extending the ‘Flick Right’ Definition Reducing false positives using negation ?p1 ?p2 ?p3 74
  • 75. Extending the ‘Flick Right’ Definition Reducing false positives using negation ?p1 ?p2 ?p3 75
  • 76. Extending the ‘Flick Right’ Definition ?p1 ?p2 ?p3 76
  • 77. Extending the ‘Flick Right’ Definition ?p1 ?p2 ?p3(not (and (Point2D (y ?b y) (time ?b time)) … there should not be a point where the … (test (< ?b time ?p3.time)) (test (> ?b time ?p1.time)) (test (> (abs (- ?p1.y ?b y)) 245)))) 77
  • 78. Extending the ‘Flick Right’ Definition ?p1 ?p2 ?p3(not (and (Point2D (y ?b y) (time ?b time)) … there should not be a point where the … (test (< ?b time ?p3.time)) … time lies in between ?p1 and ?p3 … (test (> ?b time ?p1.time)) (test (> (abs (- ?p1.y ?b y)) 245)))) 78
  • 79. Extending the ‘Flick Right’ Definition y ?p1 ?p2 ?p3 y(not (and (Point2D (y ?b y) (time ?b time)) … there should not be a point where the … (test (< ?b time ?p3.time)) … time lies in between ?p1 and ?p3 … (test (> ?b time ?p1.time)) (test (> (abs (- ?p1.y ?b y)) 245)))) … for which y is greater than 245. 79
  • 80. Evaluation22 77.50 52.10 78.75 56.5024 83.13 47.16 84.38 52.5326 90.63 42.40 91.25 46.7928 93.75 39.47 94.38 43.2630 97.50 35.37 97.50 39.2932 98.75 32.78 98.75 36.41 • 16 gestures • 1760 gesture samples • 10 subjects 80
  • 81. Future WorkScale / rotational Visual 3D gestureinvariant spotting definitions 81
  • 82. Summary lhoste@vub.ac.be bderooms@vub.ac.be bsigner@vub.ac.beAuto infer control points Generate Spotting Code Call classification • Extensive spotting: support for real-time continuous streams. • No gesture overshooting: support for overlapping submatches. • Comprehensible: easy to understand and to correct. • Extensible: allows for expert adjustments. 82
  • 83. Lode Hoste, Brecht De Roomsand Beat Signer, DeclarativeGesture Spotting UsingInferred and Refined ControlPoints, Proceedings ofICPRAM 2013, InternationalConference on PatternRecognition, Barcelona, Spain,February 2013 83
  • Search
    Related Search
    We Need Your Support
    Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

    Thanks to everyone for your continued support.

    No, Thanks