Deep Learning for Gaze Trackingimages.nvidia.com/cn/gtc/downloads/pdf/ecs/6 Deep Learning for...
24
Deep Learning for Gaze Trackingimages.nvidia.com/cn/gtc/downloads/pdf/ecs/6 Deep Learning for Eye... · What is eye tracking? Eye tracking is the process of estimating the gaze direction
Deep Learning for Eye Tracking�Presented by: �Thomas Huang@7invensun�2016·9·13 视线追踪中的深度学习
Presenter
Presentation Notes
What is eye tracking?�Eye tracking is the process of estimating the gaze direction & point.�Eye tracking device can capture users' pupil location and visual information, estimate gaze vectors and gaze point coordinates, and further to achieve the data of fixation track, fixation duration, fixation frequency and pupil diameter etc. 视线追踪是测量注视点的过程 视线追踪器是分析眼睛位置以及眼动的设备。 视线追踪：又称注视点追踪、眼球追踪。是利用眼动测量设备测量眼睛的运动情况估计视线方向或者视线落点位置的技术。
Presenter
Presentation Notes
7invensun�The Leader of Eye-Tracking Company in China. �7invensun is the first technology company in China which is committed to Eye-Tracking and Eye-Controlling technology with independent intellectual property rights. Since foundation, we still focus on researching and innovation of vision capture, artificial intelligence as well as committing to upgrading and optimizing the interaction between devices and end-users. Honors: 1. Innovative star in the final “Demo China” 2014 � 2. Runner-up of Zero2IPO “Growing Star” 2014� 3. Runner-up of 2014 Qualcomm Redwood Mobile Internet entrepreneurship competition� 4. Top 100 Innovation Growth Enterprises in 2014 China 七鑫易维是国内首家拥有自主知识产权的视线追踪和眼动控制技术的科技公司。帮助渐冻人与外界沟通，为普通人提供新的交互方式…… 企业荣誉： 2014年获得“创新中国”总决赛创新之星； 2014清科“成长力之星” 第二名； 2014年中国创新成长企业100强； 2015年代表中国先进沟通辅具受到李克强、默克尔总理接见； 2014、15、16年连续3年获得美国高通公司风险投资；
Presenter
Presentation Notes
Eye-Tracking Technology�Eye image processing is important for eye tracking�Deep Learning can improve the eye image processing 视线追踪主要处理流程是：1）输入图像；2）图像处理；3）视线估计 图像处理和视线估计都是视线追踪的关键部分。 图像处理包括人眼区域检测（蓝框）和瞳孔中心定位（红点） 传统方法在图像处理中有时会有一些问题，特别是戴眼镜反光的情况下，如图所示，瞳孔定位不准。 借助深度卷积网络，我们图像处理的结果更加准确、稳定。如图所示，瞳孔基本都定位准确了。
Deep learning for eye detection�Fully convolutional networks�Input Image, CNN model, Eye region�Input images, Results of traditional�method, Results of CNN 使用深度学习进行人眼区域检测（p11图中蓝色方框） 主要流程是：1)输入图像；2）CNN（卷积神经网络）模型；3）输出人眼区域 现有的有关检测的神经网络有：R-CNN，FAST R-CNN, FASTER R-CNN, YOLO, SSD和残差网络（Deep Residual Networks）等 我们使用的人眼区域检测方法跟YOLO较为接近，只不过我们没有使用全连接网络，而是使用的全卷积网络。这样将大大减少网络的权重（weight）,提高网络的收敛速度，减少训练时间。在使用时，也可以大大降低模型的大小。 全卷积模型
Presenter
Presentation Notes
Deep learning for eye detection�Yolo model, Conv.Layer, Maxpool Layer�Our model, Conv.Layer, Max 使用深度学习进行人眼区域检测（p11图中蓝色方框）; 主要流程是：1)输入图像；2）CNN（卷积神经网络）模型；3）输出人眼区域; 现有的有关检测的神经网络有：R-CNN，FAST R-CNN, FASTER R-CNN, YOLO, SSD和残差网络（Deep Residual Networks）等; 我们使用的人眼区域检测方法跟YOLO较为接近，只不过我们没有使用全连接网络，而是使用的全卷积网络。这样将大大减少网络的权重（weight）,提高网络的收敛速度，减少训练时间。在使用时，也可以大大降低模型的大小。 全卷积网络需要更少的空间。
Presenter
Presentation Notes
Deep Learning for eye tracking Frame t-1 Local eye area, Tracking model, Eye region Frame t Local eye area, Detection model, Eye region Results of CNN 使用深度学习进行人眼区域跟踪（p11图中蓝色方框） 由于检测花时间稍长，我们会使用跟踪方法。在上一帧人眼区域的附件去顶这一帧的人眼区域。 我们在跟踪中使用了一个跟踪网络和一个检测网络。 跟踪网络基于上一帧人眼区域确定这一帧的人眼区域。 检测网络基于小图像区域确定这一帧的人眼区域。
Presenter
Presentation Notes
Deep learning for eye tracking�Tracking, TRAIN, Input Image 1/2, A(CNN)/B(CNN), Loss Layer, Label�TEST, Input Image 1/2, Trained A/B, Matching 使用深度学习进行人眼区域跟踪（p11图中蓝色方框） 由于检测花时间稍长，我们会使用跟踪方法。在上一帧人眼区域的附件去顶这一帧的人眼区域。 我们在跟踪中使用了一个跟踪网络(所谓SIAMESE网络)和一个检测网络。 跟踪网络基于上一帧人眼区域确定这一帧的人眼区域。 检测网络基于小图像区域确定这一帧的人眼区域。
Presenter
Presentation Notes
Deep learning for pupil location Fully Convolutional Networks Eye region, CNN model, Pupil Center Results of traditional method, Results of CNN 使用深度学习进行瞳孔中心定位（p11图中红色点） 使用全卷积网络进行瞳孔中心定位。
Presenter
Presentation Notes
Deep Learning for pupil location Our model, Conv.Layer, Maxpool Layer Fully Convolutional networks 使用深度学习进行瞳孔中心定位（p11图中红色点） 使用全卷积网络进行瞳孔中心定位。
Presenter
Presentation Notes
Deep leaning boost Eye-Tracking research Thanks NVIDIA’s support for deep learning, which makes deep learning algorithm get well application in actual problem. Time-consuming Comparison of GPU & CPU NVIDIA在GPU通用运算中的贡献，对深度学习的应用与普及帮助非常巨大，在此向英伟达和“黄老板”表示感谢与敬仰！ 我们拥有一个自己的计算中心，其核心是由若干tesla k80显卡组成的服务器集群。 我们一直用这个“强大的”计算中心进行眼球追踪相关的数据处理与模型训练。 如图所示，（最右边的柱），k80在机器学习中的运算效率是一般CPU的20多倍。也就是说，如果用CPU训练一个月的模型，在NVIDIA k80显卡上也就需要1天多。这对我们模型优化提供了巨大的帮助。 在这里再次感谢NVIDIA对深度学习的给力支持!!!
Presenter
Presentation Notes
Products aMouse(Head Tracking without mark), AlmightyEye(Monocular Eye Tracking), aSee(Binocular Eye Tracking), aSee AIO(Binocular Eye Tracking All in One Tablet), VR Eye-Tracking Module 7invensun has helped numerous people with ALS, high paraplegia and cerebral palsy to communicate with the outside world. 我们的主要产品
Eye movement data & analysis Heat map of an advertisement with tiny difference of model girl’s eye’s fixation point. 眼动分析表明：仅仅放上一张美女脸蛋图是远远不够的！“她在看什么”才是关键！
Presenter
Presentation Notes
Business model Products Assistive devices / Eye tracker, VR & AR Eye-Tracking solution, Foveated Rendering, Interaction Remote Eye-Tracking Device(TO C), Eye-Tracking module ofr HTC Vive(TO B) 七鑫易维产品分为两种：第一种是独立的产品：眼控交互产品、动分析产品；第二种解决方案：采用眼控、注视点渲染技术的VR可穿戴式设备、眼控模组等。 眼动分析可以进行视点路径分析Scan path，线图分析Line graph，兴趣区顺序分析Fixation sequence，凝视图分析Focus map，热区图分析Heat Map，柱形图分析Binning chart 很可能，眼球追踪会成为“眼球经济”“Eyeball Economy”的最佳实现手段 。
Eye-Tracking technology is the crucial part of VR and AR 眼球追踪已经成为VR&AR的关键技术。
Presenter
Presentation Notes
Interaction In the real world, we’re used to rotate our eyes to locate targets, However, currently in VR, we have to rotate our head to locate targets based on simple gyroscopic theory, which will increase the motion sickness and prolong the time of targeting. While, integrating eye tracking technology into VR devices can reduce the motion sickness cause by forcible head movement, more convenient is, your can interact with any MENU, BUTTON, PICTURE, OBJECT, NPC etc. by look at them naturally. Event game characters can feel your eyes.
Presenter
Presentation Notes
Foveated rendering Full high-definition screen plays a key role in VR device. The current rendering method has high demand of hardware and high power consumption. With Eye tracking technology, rendering can be carried out in fixation points. In this way, images definition can be obtained, GPU efficiency can be enhanced, hardware demanding can reduced as well. Local rendering can also alleviate eye fatigue on certain level.
7invensun Technology Superiority Full FOV tracking, High precision, Low latency Tracking range can reach the limit of human eye rotation (vertical 30°, horizontal 50 °), and cover the mainstream VR devices which FOV more than 110 °. It not only realizes a extensive tracking range, but also covers the entire VR display screen. It has reached the international leading level (0.5 °), and improved the accuracy of gaze tracking technology by optimizing the Eye-Tracking algorithm and cumulative eye movement data; Whether Asians or Europeans and Americans, myopia or eye diseases can have a good usability. Since VR demands strong real-time; tracking speed of Eye-Tracking VR algorithm maximum up to 380 hz, latency time less than 5ms, so that users almost do not feel the latency in the using process . 技术优越性：全视场角追踪、高精度、低延迟、滑动补偿；
Presenter
Presentation Notes
The world’s first PNP VR Eye-Tracking module For HTC Vive, Plug & Play, Coming soon! 这应该是全球首款即插即用的VR眼控模组，这款是HTC Vive专用，支持自带远/近视镜片，支持散光，可以实现今天给大家讲述的注视点渲染、眼控交互等功能，下个月（10月份）即将发布预售，敬请大家扫码关注。
