The HandNet dataset contains depth images of 10 participants hands non-rigidly deforming infront of a RealSense RGB-D camera. This dataset includes 214971 annotated depth images of hands captured by a RealSense RGBD sensor of hand poses. Annotations: per pixel classes, 6D fingertip pose, heatmap. Images: Train - 202198 Test - 10000 Validation - 2773 Recorded at GIP Lab, Technion The annotations are generated by a magnetic annotation technique as described in our paper Rule of Thumb: Deep derotation for improved fingertip recognition, A Wetzler, R Slossberg, R Kimmel, BMVC 2015 6D pose is available for the center of the hand as well as the five fingertips (i.e. position and orientation of each).