Development of an autonomous mango picking system using a 7-dof robot manipulator

Abstract

As the mango industry requires intensive skilled labor for harvesting cultivation, robotics technology can potentially turn this task into an autonomous process to reduce human labor. This report outlines an approach for autonomous mango picking using a 7-DOF robot manipulator with a novel scissor gripper tool and RGB-D camera. The study mainly focused on the cutting-point generation at the mango stem and provided an autonomous mango harvesting process on the platform. Mask R-CNN was the major algorithm for searching the potential wrapped mango in the scene acquired from RGB-D data, then providing the mask data of the detected bags. The wrapped mango coordinate was defined as the points at the top of the masked mango bag and calculated from the 3-d points data inside the partial area at the top of the bag mask. The 3-d points were merged statistically: the median of the x and y values of the point set being the target x and y value in cartesian space respectively while the z value was chosen as the minimum value out of the entire point set data. The cutting point was obtained as the offset point of the mango bag’s top region vertically by constant stroke. The transformation approach of the coordinate of reference from camera into robot world frame on the targeted point also provided in this manuscript. The cutting point which references on the same frame coordinate of the robot is the basis for robot arm motion planning. The additional via points such as the search and approach postures were defined based on the cutting points to complete the path definition for the entire manipulation stems under self-collision. The harvesting experiment by using this platform could achieve the entire process with 75.75% success rate (the crops harvesting into the basket) on 66 samples while the performance of the cutting point generation algorithm is 87.8% in accuracy. Under some circumstances failure could occurred on the tasks caused by either software or hardware. The software possibly provides the wrong mango bag coordinate because of the occlusion of the stems and leaves around the mango bag. The physical issue which makes the mission fail possibly the handling of the mango crop which stem diameter smaller than 3 mm, the wind can make the target move and then drop, and the density of the occlusion element which prohibits the gripper to reach the targeted stem. The issues found during the test on both hardware and software are also discussed as the idea of future development.