Visual-haptic temporal perception
How do we perceive the crossmodal visual-haptic events during our own action? This seems a trivial question, yet it is an important issue for the human-machine interaction (HMI) involving delayed feedback, such as tele-operation. In a typical telepresence scenario, users send action commands through internet to the remote side, remote robots then send back multimodal information to the operator. The communication between action and feedback requires certain amount of time, ranging from dozen millisend to minutes (e.g. Operation in space), which inevitably impedes perception and action. Even in our daily actions, sound, light, and force also take different amount of time to reach our central nervous systems as different forms of senses take different sensory pathways. This project thus devotes to assess the influence of action and feedback delay of visual-haptic temporal perception.
Effects of packet loss and latency on visual-haptic temporal discrimination
Abstract of Shi et al. 2010: Temporal discontinuities and delay caused by packet loss or communication latency often occur in multimodal telepresence systems. It is known that such artifacts can influence the feeling of presence . However, it is largely unknown how the packet loss and communication latency affect the temporal perception of multisensory events. In this article, we simulated random packet dropouts and communication latency in the visual modality and investigated the effects on the temporal discrimination of visual-haptic collisions. Our results demonstrated that the synchronous perception of crossmodal events was very sensitive to the packet loss rate. The packet loss caused the impression of time delay and influenced the perception of the subsequent events. The perceived time of the visual event increased linearly, and the temporal discrimination deteriorated, with increasing packet loss rate. The perceived time was also influenced by the communication delay, which caused time to be slightly overestimated.
Perception of delay in haptic telepresence systems
Abstract of Rank et al. 2010. Time delay is recognized as an important issue in haptic telepresence systems as it is inherent to long-distance data transmission. What factors influence haptic delay perception in a time-delayed environment are, however, largely unknown. In this article, we examine the impact of manual movement frequency and amplitude in a sinusoidal exploratory movement as well as the stiffness of the haptic environment on the detection threshold for delay in haptic feedback. The results suggest that the detection of delay in force feedback depends on the movement frequency and amplitude, while variation of the absolute feedback force level does not influence the detection threshold. A model based on the exploration movement is proposed and guidelines for system design with respect to the time delay in haptic feedback are provided.
Influence of own action on visual-haptic simultaneous perception
Abstract of Shi et al. 2008: Perceiving temporal simultaneity of cross-modal (e.g., visuo-haptic) events is very important for coherent perception and high performance in multi-modal interaction (e.g. telepresence) systems. The perception of simultaneity, however, depends on many factors, including stimulus intensity, redundant information and selective attention. In this study, we examined the effects of two factors on perceived simultaneity in a visuo-haptic task scenario: active (self- generated) motor control and additional visual motion feedback. We used a psychophysical method to measure the differences in points of subjective simultaneity (PSSs) and just noticeable differences (JNDs) in visuo-haptic events. The results showed that the PSS decreased significantly under conditions of active motor control with concurrent visual feedback, and the JNDs were narrowed with either active motor control or additional visual motion feedback.
Shi, Z., Zou, H., Rank, M., Chen, L., Hirche, S., Müller, H. J. (2010) Effects of packet loss and latency on the temporal discrimination of visual-haptic events. IEEE Transactions on Haptics, 3(1), 28-36,
Shi, Z., Zou, H., & Müller, H. J. (2010) Temporal perception of visual-haptic events in multimodal telepresence system. In M. H. Zadeh (Eds.), Advances in Haptics, 437-449.
Rank, M., Shi, Z., Müller, H. & Hirche, S. (2010) Perception of Delay in Haptic Telepresence Systems, Presence,19(5), 389-399.
Rank, M., Shi, Z., Müller, H. J. (2010)The influence of difference Haptic Environments on Time Delay Discrimination in Force Feedback, Lecture Notes in Computer Science, 6191, Eurohaptics 2010, 205-212.
Shi, Z., Hirche, S., Schneider W., Müller, H. (2008), Influence of visuomotor action on visual-haptic simultaneous perception: A psychophysical study, Proceedings of IEEE Symposium on Haptic Interfaces for Virtual Environment and Teleoperator systems 2008, 65-70