CN115250745B - A fully automatic fruit picking robot and picking method based on vision technology - Google Patents
A fully automatic fruit picking robot and picking method based on vision technology Download PDFInfo
- Publication number
- CN115250745B CN115250745B CN202210936842.2A CN202210936842A CN115250745B CN 115250745 B CN115250745 B CN 115250745B CN 202210936842 A CN202210936842 A CN 202210936842A CN 115250745 B CN115250745 B CN 115250745B
- Authority
- CN
- China
- Prior art keywords
- fruit
- fruits
- mechanical arm
- picking
- bionic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- A—HUMAN NECESSITIES
- A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
- A01D—HARVESTING; MOWING
- A01D46/00—Picking of fruits, vegetables, hops, or the like; Devices for shaking trees or shrubs
- A01D46/30—Robotic devices for individually picking crops
-
- A—HUMAN NECESSITIES
- A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
- A01D—HARVESTING; MOWING
- A01D91/00—Methods for harvesting agricultural products
- A01D91/04—Products growing above the soil
Landscapes
- Life Sciences & Earth Sciences (AREA)
- Environmental Sciences (AREA)
- Engineering & Computer Science (AREA)
- Robotics (AREA)
- Manipulator (AREA)
Abstract
本发明公开了一种基于视觉技术的全自动水果采摘机器人及采摘方法，包括移动平台、升降平台、工控机、机械臂、视觉系统、仿生夹爪、收集装置；机械臂、工控机和升降平台安装在移动平台的支撑底板上，收集装置安装在升降平台上；机械臂的末端安装有仿生夹爪和视觉系统。本发明的仿章鱼夹爪设计能够增大夹爪与水果的接触面积，使夹爪更加稳固的抓取目标水果，能实现对不同目标体积的自适应性，增加了通用性。
The invention discloses a fully automatic fruit picking robot and a picking method based on vision technology, which includes a mobile platform, a lifting platform, an industrial computer, a robotic arm, a visual system, a bionic clamping claw, and a collection device; a robotic arm, an industrial computer, and a lifting platform. It is installed on the support base of the mobile platform, and the collection device is installed on the lifting platform; the end of the robotic arm is equipped with a bionic gripper and a vision system. The octopus-like gripper design of the present invention can increase the contact area between the gripper and the fruit, allowing the gripper to more firmly grasp the target fruit, achieve adaptability to different target volumes, and increase versatility.
Description
Technical Field
The invention belongs to the field of agricultural machinery, and particularly relates to a full-automatic fruit picking robot and a picking method based on a vision technology.
Background
The orchard planting area and the fruit yield in China are steadily located in the world throughout the year, and fruit picking is an important link of orchard planting. An existing fruit picking machine, such as patent CN110506501a, discloses an automatic fruit picking machine, which is characterized in that a fruit tree is fixed and shaken by a meshing vibration device on a vehicle body to drop fruits, and then a receiving device is used for receiving the dropped fruits, so that the fruits are harvested; however, in the picking and shaking process, the target fruits and branches and leaves can be mixed and fall down, the subsequent step of separating the fruits and the leaves is added, and the device can cause certain damage to the surfaces of the fruit trees and the fruits. Patent CN109302885a discloses a fruit picking robot, picking fruits by a manipulator, and placing the picked fruits into a collecting device; however, the manipulator of the picking robot cannot adapt to fruits with different sizes, causes a certain degree of damage to fruits with larger volumes when in actual picking, and meanwhile, needs manual operation, so that full automation cannot be realized completely. Therefore, how to reduce the damage to fruits and fruit trees in the fruit picking process and improve the picking efficiency is a technical problem which needs to be solved in the field urgently.
Disclosure of Invention
The invention aims to overcome the defects in the prior art, and provides a full-automatic fruit picking robot and a picking method based on a vision technology, wherein a bionic clamping jaw of the full-automatic fruit picking robot can adapt to the size of fruits, damage to the fruits and fruit trees is small, the picking robot can determine the picking position and angle through vision, then the height of a lifting platform is automatically controlled to expand the picking range, and the picking efficiency is improved.
The aim of the invention is achieved by the following technical scheme:
a full-automatic fruit picking robot based on a vision technology comprises a moving platform, a lifting platform 2, an industrial personal computer 3, a mechanical arm 4, a vision system 5, a bionic clamping jaw 6 and a collecting device; the mechanical arm 4, the industrial personal computer 3 and the lifting platform 2 are arranged on a supporting bottom plate 8 of the mobile platform, and the collecting device is arranged on the lifting platform 2; the tail end of the mechanical arm 4 is provided with a bionic clamping jaw 6 and a vision system 5.
The mobile platform comprises a mobile trolley 1, a supporting bottom plate 8 and an ultrasonic ranging sensor 9; a supporting bottom plate 8 is fixed at the top of the mobile trolley 1; an ultrasonic ranging sensor 9 is mounted at the front end of the mobile trolley 1.
The vision system 5 comprises a first binocular camera 13, a second binocular camera 14, a camera support 15 and an inertial navigation module 16, wherein the two binocular cameras are installed on the camera support 15, the camera support 15 is fixedly installed on a mechanical arm flange 17, and the inertial navigation module 16 is fixed behind the camera support 15. The first binocular camera 13 and the inertial navigation module 16 form a visual slam system to complete construction work of an orchard map; the second binocular camera 14 captures the camera current frame image in real time while in motion and recognizes and locates fruit present in the current frame image through deep learning.
The 4 bionic clamping jaws 6 are uniformly distributed on the connecting piece 18, the connecting piece 18 is fixed on the flange 17 of the mechanical arm, and a motor main shaft of the stepping motor 19 is sleeved with the connecting piece 18. The bionic clamping jaw 6 is designed by adopting a bionic octopus and comprises an outer jaw 25, an inner jaw 22 and a sucker 24; the front end of the inner claw 22 is hinged with the front end of the outer claw 25, and two springs 21 are fixedly arranged between the inner claw 22 and the outer claw 25; the front end of the second connecting rod 27 is hinged with the tail end of the outer claw 25, the convex part of the second connecting rod 27 is provided with an arc-shaped notch, and the tail end of the inner claw 22 can slide in the arc-shaped notch; the hinge joint of the outer claw 25 and the second connecting rod 27 is sleeved with a torsion spring 28, and a torsion spring torsion adjusting device 26 is arranged at the joint of the outer claw 25 and the second connecting rod 27 and controls the length of the torsion spring 28 by adjusting the depth of a nut of the torsion spring torsion adjusting device, so that the torsion force of the torsion spring is controlled.
The inner claw 22 is of an arc-shaped structure, more than one sucking disc 24 is arranged on the surface of the inner claw, the sucking disc 24 is made of flexible materials and is arranged on a ball stud, and the other end of the ball stud is fixed on the inner claw 22.
One end of the first connecting rod 20 is connected with a motor seat 23 of the stepping motor 19 to form a revolute pair; the other end of the first link 20 is connected with the protruding portion of the second link 27 in a living hinge manner, and the end of the second link 27 is connected with the connecting member 18 in a living hinge manner, and the rotation angle of the outer jaw is restricted to 0 to 30 °.
The working principle of the bionic clamping jaw 6 is as follows: (1) The bionic clamping jaw 6 is designed by adopting a bionic octopus, and the surfaces of most fruits are arc-shaped, so that the inner claw 22 is designed into an arc shape to fit the surfaces of the fruits for picking work; a row of suckers 24 are designed on the surfaces of the inner claws, the suckers can freely rotate on the ball stud according to stress conditions, when the fruits are grabbed, the surfaces of the suckers are tightly attached to the surfaces of the fruits due to stress of the suckers, so that the contact area between the bionic clamping jaw and the fruits is increased, the surfaces of the fruits are protected from being damaged, and the grabbing stability of the bionic clamping jaw is also increased; (2) When the inner jaw 22 is pressed, the inner jaw 22 starts to rotate towards the outer jaw 25, and the spring 21 generates a reaction force to increase the grabbing force of the inner jaw 22 on the target; when the spring compression between the inner claw and the outer claw is maximum, the bionic clamping jaw can grasp the maximum volume of fruits, if the volume of the fruits to be grasped exceeds the maximum volume, the torsion of the torsion spring can be reduced by changing the scale of the nut on the torsion spring torsion adjusting device, so that the outer claw can rotate by a certain angle, and the effective grasping volume of the clamping jaw is increased. The double functions of the inner claw and the outer claw can realize the self-adaption to different target volumes without algorithm control, and the universality of picking fruits with different sizes is enhanced.
A full-automatic fruit picking method based on vision technology adopts the full-automatic fruit picking robot, and comprises the following steps:
(1) Early preparation: calibrating the two binocular cameras to obtain an inner parameter matrix and an outer parameter matrix and a re-projection matrix of each camera; performing hand-eye calibration on the second binocular camera 14 to obtain a rotation translation matrix of a camera coordinate system and a mechanical arm base coordinate system, wherein the rotation translation matrix is used for converting points in the camera coordinate system into points in the mechanical arm base coordinate system;
(2) Building a map: constructing a map in an orchard by utilizing a visual slam system formed by the first binocular camera 13 and the inertial navigation module 16, and obtaining a three-dimensional point cloud map of the whole orchard, wherein the three-dimensional point cloud map comprises position information of a starting point and a terminal point;
(3) Visual inspection: loading a three-dimensional point cloud map of an orchard, and automatically walking the mobile trolley along a map track; the second binocular camera 14 acquires and detects each frame of image in real time by utilizing the YOLOv5 network, when the target fruit appears, the mobile trolley stops, the second binocular camera stores the current frame of image, and otherwise, the robot continues to walk;
(4) Autonomous obstacle avoidance: when the mobile trolley walks, the ultrasonic ranging sensor 9 detects an obstacle in real time and feeds back the distance between the ultrasonic ranging sensor and the obstacle to the industrial personal computer; when the distance is less than 1m, stopping the mobile trolley, and if the obstacle disappears within 10 seconds, continuing to walk by the mobile trolley; otherwise, the mobile trolley turns left or right in situ, and continues to walk after bypassing the obstacle;
(5) Automatic lifting: when the mobile trolley stops due to the detection of fruits, firstly extracting the outline of each fruit in the stored image in the step (3), solving the mass center of all the fruit outlines, obtaining a mass center intermediate value, taking the three-dimensional space position of the mass center intermediate value as the initial picking gesture of the mechanical arm, namely, transmitting the position information of the mass center intermediate value to an industrial personal computer, and controlling the height of a lifting platform by the industrial personal computer so that the tail end of the mechanical arm is opposite to the position of the mass center intermediate value.
(6) Positioning a target: calculating the center point of the fruit and the fruit stalks in the image, judging the inclination angle of the fruit and the horizontal plane, and controlling the tail end gesture of the mechanical arm; then matching fruit centroids in left and right images of the binocular camera one by using an SAD matching algorithm, calculating the spatial position of each fruit centroid, and converting the spatial position of the fruit centroid into a mechanical arm base coordinate;
(7) Picking fruits: the mechanical arm sequentially goes to the mass center point of each fruit; when the target fruit completely enters the bionic clamping jaw, the bionic clamping jaw starts to be closed, the sucker on the inner jaw rotates through the reaction force of the fruit surface until the sucker contacting the fruit is tightly attached to the fruit surface, and meanwhile, the spring between the inner jaw and the outer jaw gives an acting force, so that the sucker can more firmly grasp the fruit;
(8) Placing fruits: after the bionic clamping jaw grabs the fruits, the mechanical arm moves to the upper part of the storage box, and the fruits are placed in the storage box until the fruits in the current field of view are picked; an infrared sensor on the storage box can detect whether fruits in the storage box are full; the mobile trolley can continue to carry out picking along the three-dimensional point cloud map of the orchard until the whole orchard is picked.
In the step (2), the first binocular camera and the inertial navigation module are adopted to construct the map, and the inertial navigation module can accurately convey rotation and displacement generated in the motion process in real time, so that the accuracy of constructing the map is improved.
In the step (3), a large number of fruit samples and fruit stem pictures thereof are prepared in advance, a fruit image library is constructed, and then the fruit image library is trained by utilizing a YOLOv5 deep learning network, so that the YOLOv5 deep learning network can identify fruits and fruit stems in the pictures.
In the step (5), after the stored image is extracted, gray processing is carried out on a detection frame containing fruits, then binarization operation is carried out on the detection frame by using an OTUS method, and then noise is eliminated by carrying out operations such as filtering processing on the image; finally, extracting contour information by using a canny operator and an edge detection algorithm function, and calculating the centroid (x) of the contour i ,y i ) The method comprises the steps of carrying out a first treatment on the surface of the Repeating the operation until the barycenters of all fruit outlines in the image are obtained, and adding the transverse coordinates and the longitudinal coordinates of each barycenter to obtain the sum w i ＝x i +y i The method comprises the steps of carrying out a first treatment on the surface of the Sum it to w i The method comprises the steps of arranging in sequence from large to small, and taking intermediate values; when the total fruit number (n) in the field of view is even, the center of mass median is (x) n/2 ,y n/2 ) The method comprises the steps of carrying out a first treatment on the surface of the When the total fruits in the visual field are odd, the center of mass median is (x) (n+1)/2 ,y (n+1)/2 ) Or (x) (n-1)/2 ,y (n-1)/2 ) The method comprises the steps of carrying out a first treatment on the surface of the Then calculating the actual space position of the center of mass intermediate value by using a triangulation principle, and controlling the height of the lifting platform by the industrial personal computer so that the tail end of the mechanical arm is opposite to the fruit where the center of mass intermediate value is located; at this time, the posture of the mechanical arm is set to be the initial posture, so that all fruits are in a state of picking under the view angle of the mechanical arm.
In the step (6), the detection frames of the fruits and the fruit stalks are identified by deep learning, and the center points (Fx, fy) of the fruit detection frames and the center points (Fx, fy) of the corresponding fruit stalk detection frames are obtained; the slope k= (Fy-Fy)/(Fx-Fx) of the two points is obtained, and then the alpha is obtained by using an inverse trigonometric function arctan alpha = k, namely the inclination angle of the fruit relative to the horizontal plane; if alpha is more than 15 degrees, rotating the bionic clamping jaw by a corresponding angle; if α < = 15 °, the bionic jaw remains unchanged in the initial attitude.
The working principle of the invention is as follows: building a map of an orchard through a visual slam system, and reading the map by a mobile trolley and walking along the map; when the trolley walks, the ultrasonic ranging sensor carried by the trolley can feed back the distance between the trolley and the obstacle in the walking direction, so that the trolley is driven to bypass the obstacle to realize autonomous obstacle avoidance; meanwhile, in the walking process, the other binocular camera performs real-time detection. The working space of the mechanical arm is relatively small, and for the initial pose of the mechanical arm, a phenomenon that picking fails due to the fact that a plurality of fruits are far away exists, so that after the target fruits are detected, the industrial personal computer sends a signal to enable the trolley to stop moving, the height of the lifting platform is controlled, and all the fruits are in a state of picking under the visual angle of the mechanical arm, so that the picking rate is improved; before picking, changing the tail end posture by judging the inclination angle of the fruits and the horizontal plane; when picking, the reaction force of the fruit enables the sucker arranged on the ball stud to rotate, so that the sucker clings to the surface of the fruit, the contact area between the clamping jaw and the fruit is increased, and the double action of the inner jaw and the outer jaw enables the bionic clamping jaw to self-adaptively grab the fruit. After picking, fruits are placed in the storage box, and whether the storage box is full of fruits is judged through the infrared sensor.
Compared with the prior art, the invention has the following advantages and effects:
(1) According to the invention, an orchard map is constructed by adopting a mode of combining the binocular camera and the inertial navigation module, so that an accurate pose can be provided when the mobile trolley moves violently, the condition of losing characteristic points is avoided, and the accuracy of the obtained map point cloud is greatly improved.
(2) According to the invention, the height of the lifting platform and the initial pose of the mechanical arm during each picking are determined through the algorithm, so that the picking success rate and the picking range can be increased, and the situation of picking failure caused by exceeding the working range of the mechanical arm is avoided.
(3) The invention adopts the ultrasonic ranging sensor, and can realize the autonomous obstacle avoidance function of the mobile trolley when walking.
(4) The invention has good bionic effect, and in the picking process, the design of the octopus-like clamping jaw can enlarge the contact area of the clamping jaw and the fruit, so that the clamping jaw can grasp the target fruit more firmly. For fruits with moderate volumes, the spring between the inner claw and the outer claw can increase the grabbing force of the tail end, and the surfaces of the fruits are not damaged; for fruits with larger volume, if the spring between the inner claw and the outer claw is compressed to the maximum value, the outer claw can continuously rotate outwards under the action of the fruits to increase the effective capacity of the clamping jaw, and the double action of the inner claw and the outer claw enables the tail end to realize the self-adaptability to different target volumes without an algorithm, so that the universality is improved.
(5) The fruit tilting degree is calculated through an algorithm, and then the rotation angle of the clamping jaw is controlled. When facing the fruit with relatively inclined growth posture, the clamping jaw can well grab the fruit, and the condition that the clamping jaw part pushes the fruit open can not occur.
(6) According to the bionic octopus clamping jaw, the inner jaw and the sucker arranged on the inner jaw are made of soft silica gel materials, so that the weight of the tail end can be reduced, and the surface of fruits can be protected from being damaged.
(7) The invention is suitable for full-automatic picking of various fruits and has strong universality.
Drawings
Fig. 1 is a schematic diagram of the overall structure of a fully automatic fruit picking robot.
Fig. 2 is a schematic structural diagram of a bionic jaw and a vision system.
Fig. 3 is a schematic structural diagram of a bionic clamping jaw.
Fig. 4 is a picking workflow diagram of a fully automated fruit picking robot.
Wherein: 1. a moving trolley; 2. a lifting platform; 3. an industrial personal computer; 4. a mechanical arm; 5. a vision system; 6. bionic clamping jaws; 7. a storage tank; 8. a support base plate; 9. an ultrasonic ranging sensor; 10. a power device; 11 control cabinet and power cabinet of mechanical arm; 12. an infrared sensor; 13. a first binocular camera; 14. a second binocular camera; 15. a camera mount; 16. an inertial navigation module; 17 mechanical arm flange plates; 18. a connecting piece; 19. a stepping motor; 20. a first link; 21. a spring; 22. an inner claw; 23. a motor base; 24. a suction cup; 25. an outer claw; 26. torsion adjusting parts of torsion springs; 27. a second link 2; 28. and (3) a torsion spring.
Detailed Description
In order that the invention may be readily understood, a detailed description of the invention will be provided below with reference to specific examples. The following examples will assist those skilled in the art in further understanding the present invention, but are not intended to limit the invention in any way. It should be noted that the present invention can be modified and improved by those skilled in the art without departing from the spirit of the present invention, which falls within the scope of the present invention.
Example 1
A full-automatic fruit picking robot based on a vision technology is shown in fig. 1, and comprises a moving platform, a lifting platform 2, an industrial personal computer 3, a mechanical arm 4, a vision system 5, a bionic clamping jaw 6 and a collecting device; the mechanical arm 4, the industrial personal computer 3 and the lifting platform 2 are arranged on a supporting bottom plate 8 of the mobile platform, and the collecting device is arranged on the lifting platform 2; the tail end of the mechanical arm 4 is provided with a bionic clamping jaw 6 and a vision system 5. The mobile platform comprises a mobile trolley 1, a supporting bottom plate 8 and an ultrasonic ranging sensor 9; a supporting bottom plate 8 is fixed at the top of the mobile trolley 1; an ultrasonic ranging sensor 9 is mounted at the front end of the mobile trolley 1. The movable trolley 1 can be a crawler-type movable trolley, can adapt to various terrains, can also drive over smaller obstacles, and meanwhile, the movable trolley 1 also provides 220V power for other components. The lifting platform 2 is a hydraulic lifting platform, the hydraulic lifting platform and a power device 10 thereof are fixedly arranged on the supporting bottom plate 8, and the lifting function is realized through the pressure transmission of hydraulic oil. The industrial personal computer 3 is equipment for carrying out algorithm processing on the whole system, realizes whole communication by the equipment, and is fixed on the supporting bottom plate 8. The mechanical arms 4 are mechanical arms with six degrees of freedom, each mechanical arm is driven by a special motor, and the mechanical arms have great flexibility in practical application, so that picking actions can be completed under different growth postures of fruits. The control cabinet and the power cabinet 11 are fixed on the supporting bottom plate 8.
As shown in fig. 2, the vision system 5 includes a first binocular camera 13, a second binocular camera 14, a camera support 15, and an inertial navigation module 16, where the two binocular cameras are mounted on the camera support 15, the camera support 15 is fixedly mounted on a mechanical arm flange 17, and the inertial navigation module 16 is fixed behind the camera support 15. The first binocular camera 13 and the inertial navigation module 16 form a visual slam system to complete construction work of an orchard map; the second binocular camera 14 captures the camera current frame image in real time while in motion and recognizes and locates fruit present in the current frame image through deep learning.
As shown in fig. 2, 4 bionic clamping jaws 6 are uniformly distributed on a connecting piece 18, the connecting piece 18 is fixed on a flange 17 of the mechanical arm, and a motor main shaft of a stepping motor 19 is sleeved with the connecting piece 18. As shown in fig. 3, the bionic clamping jaw 6 adopts a bionic octopus design, and comprises an outer jaw 25, an inner jaw 22 and a sucking disc 24; the front end of the inner claw 22 is hinged with the front end of the outer claw 25, and two springs 21 are fixedly arranged between the inner claw 22 and the outer claw 25; the front end of the second connecting rod 27 is hinged with the tail end of the outer claw 25, the convex part of the second connecting rod 27 is provided with an arc-shaped notch, and the tail end of the inner claw 22 can slide in the arc-shaped notch; the hinge joint of the outer claw 25 and the second connecting rod 27 is sleeved with a torsion spring 28, and a torsion spring torsion adjusting device 26 is arranged at the joint of the outer claw 25 and the second connecting rod 27 and controls the length of the torsion spring 28 by adjusting the depth of a nut of the torsion spring torsion adjusting device, so that the torsion force of the torsion spring is controlled. The inner claw 22 is of an arc-shaped structure, more than one sucking disc 24 is arranged on the surface of the inner claw, the sucking disc 24 is made of flexible materials and is arranged on a ball stud, and the other end of the ball stud is fixed on the inner claw 22. One end of the first connecting rod 20 is connected with a motor seat 23 of the stepping motor 19 to form a revolute pair; the other end of the first link 20 is connected with the protruding portion of the second link 27 in a living hinge manner, and the end of the second link 27 is connected with the connecting member 18 in a living hinge manner, and the rotation angle of the outer jaw is restricted to 0 to 30 °. The collecting device comprises a storage tank 7 and an infrared sensor 12. The storage box 7 is placed on the lifting platform 2, and the freedom degrees of the storage box in five directions are limited through the clamping grooves and the lifting platform, so that the stable state during movement is ensured. An infrared sensor 12 is mounted at the top edge of the bin for determining whether the interior of the bin has been filled with fruit.
As shown in fig. 4, the full-automatic fruit picking method based on the vision technology comprises the following steps:
(1) Building a map: constructing a map in an orchard by utilizing a visual slam system formed by the first binocular camera 13 and the inertial navigation module 16, and obtaining a three-dimensional point cloud map of the whole orchard, wherein the three-dimensional point cloud map comprises position information of a starting point and a terminal point;
(2) Visual inspection: loading a three-dimensional point cloud map of an orchard, and automatically walking the mobile trolley along a map track; the second binocular camera 14 acquires and detects each frame of image in real time by utilizing the YOLOv5 network, when the target fruit appears, the mobile trolley stops, the second binocular camera stores the current frame of image, and otherwise, the robot continues to walk;
(3) Autonomous obstacle avoidance: when the mobile trolley walks, the ultrasonic ranging sensor 9 detects an obstacle in real time and feeds back the distance between the ultrasonic ranging sensor and the obstacle to the industrial personal computer; when the distance is less than 1m, stopping the mobile trolley, and if the obstacle disappears within 10 seconds, continuing to walk by the mobile trolley; otherwise, the mobile trolley turns left or right in situ (turns right if the left side also has an obstacle), and continues to walk after bypassing the obstacle;
(4) Automatic lifting: when the mobile trolley stops due to the detection of fruits, firstly extracting the outline of each fruit in the stored image in the step (3), solving the mass center of all the fruit outlines, obtaining a mass center intermediate value, taking the three-dimensional space position of the mass center intermediate value as the initial picking gesture of the mechanical arm, namely, transmitting the position information of the mass center intermediate value to an industrial personal computer, and controlling the height of a lifting platform by the industrial personal computer so that the tail end of the mechanical arm is opposite to the position of the mass center intermediate value.
(5) Positioning a target: calculating the center point of the fruit and the fruit stalks in the image, judging the inclination angle of the fruit and the horizontal plane, and controlling the tail end gesture of the mechanical arm; then matching fruit centroids in left and right images of the binocular camera one by using an SAD matching algorithm, calculating the spatial position of each fruit centroid, and converting the spatial position of the fruit centroid into a mechanical arm base coordinate;
(6) Picking fruits: the mechanical arm sequentially goes to the mass center point of each fruit; when the target fruit completely enters the bionic clamping jaw, the bionic clamping jaw starts to be closed, the sucker on the inner jaw rotates through the reaction force of the fruit surface until the sucker contacting the fruit is tightly attached to the fruit surface, and meanwhile, the spring between the inner jaw and the outer jaw gives an acting force, so that the sucker can more firmly grasp the fruit;
(8) Placing fruits: after the bionic clamping jaw grabs the fruits, the mechanical arm moves to the upper part of the storage box, and the fruits are placed in the storage box until the fruits in the current field of view are picked; an infrared sensor on the storage box can detect whether fruits in the storage box are full; the mobile trolley can continue to carry out picking along the three-dimensional point cloud map of the orchard until the whole orchard is picked.
Example 2
Under the condition of not changing the torsion adjusting device of the torsion spring, when the compression of the spring between the inner claw and the outer claw is maximum, the bionic clamping jaw can grasp the maximum volume of fruits. When the grabbed target volume is larger than the value, torsion of the torsion spring can be reduced by changing the scale of the nut on the torsion spring torsion adjusting device, so that the outer claw of the clamping jaw rotates outwards under the action of force, and the effective grabbing volume of the clamping jaw can be increased. The double action of the inner claw and the outer claw enables the tail end to achieve self-adaption to different target volumes without an algorithm, and universality of picking fruits with different sizes is enhanced.
Example 3
When the growth posture of the grabbed fruits is not vertical, that is to say, when the fruits and the horizontal plane have a certain inclination angle, the positions of the fruits and the fruit stalks are predicted through deep learning, and then the inclination angle of the fruits relative to the horizontal plane is calculated through the positions of the central points of the fruit and the fruit stalk detection frames; when the angle is smaller than 15 degrees, the clamping jaw is used for grabbing forward in an initial posture, and when the angle is larger than 15 degrees, the clamping jaw can rotate by a corresponding angle according to the calculated value, so that the clamping jaw is prevented from pushing fruit stalks or fruits when the clamping jaw goes to a picking point, and picking failure is avoided.
The foregoing is illustrative of the present invention, and the present invention is not limited to the above embodiments, but is capable of other modifications, adaptations, alternatives, combinations, and simplifications without departing from the spirit and principles of the invention.
Claims (6)
1. Full-automatic fruit picking robot based on vision technique, its characterized in that: the device comprises a moving platform, a lifting platform, an industrial personal computer, a mechanical arm, a vision system, a bionic clamping jaw and a collecting device; the mechanical arm, the industrial personal computer and the lifting platform are arranged on a supporting bottom plate of the mobile platform, and the collecting device is arranged on the lifting platform; the tail end of the mechanical arm is provided with a bionic clamping jaw and a vision system;
the 4 bionic clamping jaws are uniformly distributed on the connecting piece, the connecting piece is fixed on a flange plate of the mechanical arm, and a motor main shaft of the stepping motor is sleeved with the connecting piece; the bionic clamping jaw is designed by adopting a bionic octopus and comprises an outer jaw, an inner jaw and a sucker; the front end of the inner claw is hinged with the front end of the outer claw, and two springs are fixedly arranged between the inner claw and the outer claw; the front end of the second connecting rod is hinged with the tail end of the outer claw, the convex part of the second connecting rod is provided with an arc-shaped notch, and the tail end of the inner claw can slide in the arc-shaped notch; the hinge joint of the outer claw and the second connecting rod is sleeved with a torsion spring, and a torsion adjusting device of the torsion spring is arranged at the joint of the outer claw and the second connecting rod and controls the length of the torsion spring by adjusting the depth of a nut of the torsion adjusting device, so that the torsion force of the torsion spring is controlled; the inner claw is of an arc-shaped structure, more than one sucking disc is arranged on the surface of the inner claw, the sucking disc is made of flexible materials and is arranged on the ball stud, and the other end of the ball stud is fixed on the inner claw;
one end of the first connecting rod is connected with a motor seat of the stepping motor to form a revolute pair; the other end of the first connecting rod is connected with the protruding part of the second connecting rod in a movable hinge mode, the tail end of the second connecting rod is connected with the connecting piece in a movable hinge mode, and the rotation angle of the outer claw is limited to be 0-30 degrees.
2. The fully automatic fruit picking robot of claim 1, wherein: the mobile platform comprises a mobile trolley, a supporting bottom plate and an ultrasonic ranging sensor; a supporting bottom plate is fixed at the top of the mobile trolley; an ultrasonic ranging sensor is installed at the front end of the mobile trolley.
3. The fully automatic fruit picking robot of claim 2, wherein: the vision system comprises a first binocular camera, a second binocular camera, a camera support and an inertial navigation module, wherein the two binocular cameras are arranged on the camera support, the camera support is fixedly arranged on a flange plate of the mechanical arm, and the inertial navigation module is fixed at the back of the camera support.
4. A full-automatic fruit picking method based on visual technology is characterized in that: a fully automatic fruit picking robot as claimed in claim 3, comprising the steps of:
(1) Early preparation: calibrating the two binocular cameras to obtain an inner parameter matrix and an outer parameter matrix and a re-projection matrix of each camera; performing hand-eye calibration on the second binocular camera to obtain a rotation translation matrix of a camera coordinate system and a mechanical arm base coordinate system, wherein the rotation translation matrix is used for converting points in the camera coordinate system into points in the mechanical arm base coordinate system;
(2) Building a map: constructing a map in an orchard by utilizing a visual slam system formed by a first binocular camera and an inertial navigation module, and obtaining a three-dimensional point cloud map of the whole orchard, wherein the three-dimensional point cloud map comprises position information of a starting point and a terminal point;
(3) Visual inspection: loading a three-dimensional point cloud map of an orchard, and automatically walking the mobile trolley along a map track; the second binocular camera acquires and utilizes a YOLOv5 network to detect each frame of image in real time, when a target fruit appears, the mobile trolley stops, the second binocular camera stores the current frame of image, and if not, the mobile trolley continues to walk;
(4) Autonomous obstacle avoidance: when the mobile trolley walks, the ultrasonic ranging sensor detects an obstacle in real time and feeds back the distance between the ultrasonic ranging sensor and the obstacle to the industrial personal computer; when the distance is less than 1m, stopping the mobile trolley, and if the obstacle disappears within 10 seconds, continuing to walk by the mobile trolley; otherwise, the mobile trolley turns left or right in situ, and continues to walk after bypassing the obstacle;
(5) Automatic lifting: when the mobile trolley stops due to the detection of fruits, firstly extracting the outline of each fruit in the stored image in the step (3), solving the mass center of all the fruit outlines, obtaining a mass center intermediate value, taking the three-dimensional space position of the mass center intermediate value as the initial picking gesture of the mechanical arm, namely, transmitting the position information of the mass center intermediate value to an industrial personal computer, and controlling the height of a lifting platform by the industrial personal computer so that the tail end of the mechanical arm is opposite to the position of the mass center intermediate value;
(6) Positioning a target: calculating the center point of the fruit and the fruit stalks in the image, judging the inclination angle of the fruit and the horizontal plane, and controlling the tail end gesture of the mechanical arm; then matching fruit centroids in left and right images of the binocular camera one by using an SAD matching algorithm, calculating the spatial position of each fruit centroid, and converting the spatial position of the fruit centroid into a mechanical arm base coordinate;
(7) Picking fruits: the mechanical arm sequentially goes to the mass center point of each fruit; when the target fruit completely enters the bionic clamping jaw, the bionic clamping jaw starts to be closed, the sucker on the inner jaw rotates through the reaction force of the fruit surface until the sucker contacting the fruit is tightly attached to the fruit surface, and meanwhile, the spring between the inner jaw and the outer jaw gives an acting force, so that the sucker can more firmly grasp the fruit;
(8) Placing fruits: after the bionic clamping jaw grabs the fruits, the mechanical arm moves to the upper part of the storage box, and the fruits are placed in the storage box until the fruits in the current field of view are picked; an infrared sensor on the storage box can detect whether fruits in the storage box are full; the mobile trolley can continue to carry out picking along the three-dimensional point cloud map of the orchard until the whole orchard is picked.
5. The vision-based technique of claim 4, fully automaticThe fruit picking method is characterized in that: in the step (5), after the stored image is extracted, gray processing is carried out on a detection frame containing fruits, then binarization operation is carried out on the detection frame by using an OTUS method, and then filtering processing operation is carried out on the image to eliminate noise; finally, extracting contour information by using a canny operator and an edge detection algorithm function, and calculating the centroid (x) of the contour i ,y i ) The method comprises the steps of carrying out a first treatment on the surface of the Repeating the operation until the barycenters of all fruit outlines in the image are obtained, and adding the transverse coordinates and the longitudinal coordinates of each barycenter to obtain the sum w i ＝x i +y i The method comprises the steps of carrying out a first treatment on the surface of the Sum it to w i The method comprises the steps of arranging in sequence from large to small, and taking intermediate values; when the total fruit number (n) in the field of view is even, the center of mass median is (x) n/2 ,y n/2 ) The method comprises the steps of carrying out a first treatment on the surface of the When the total fruits in the visual field are odd, the center of mass median is (x) (n+1)/2 ,y (n+1)/2 ) Or (x) (n-1)/2 ,y (n-1)/2 ) The method comprises the steps of carrying out a first treatment on the surface of the And then calculating the actual space position of the center of mass intermediate value by using a triangulation principle, and controlling the height of the lifting platform by the industrial personal computer so that the tail end of the mechanical arm is opposite to the fruit where the center of mass intermediate value is located.
6. The vision-based fully automatic fruit picking method of claim 4, wherein: in the step (6), the detection frames of the fruits and the fruit stalks are identified by deep learning, and the center points (Fx, fy) of the fruit detection frames and the center points (Fx, fy) of the corresponding fruit stalk detection frames are obtained; the slope k= (Fy-Fy)/(Fx-Fx) of the two points is obtained, and then the alpha is obtained by using an inverse trigonometric function arctan alpha = k, namely the inclination angle of the fruit relative to the horizontal plane; if alpha is more than 15 degrees, rotating the bionic clamping jaw by a corresponding angle; if α < = 15 °, the bionic jaw remains unchanged in the initial attitude.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210936842.2A CN115250745B (en) | 2022-08-05 | 2022-08-05 | A fully automatic fruit picking robot and picking method based on vision technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210936842.2A CN115250745B (en) | 2022-08-05 | 2022-08-05 | A fully automatic fruit picking robot and picking method based on vision technology |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115250745A CN115250745A (en) | 2022-11-01 |
| CN115250745B true CN115250745B (en) | 2024-03-12 |
Family
ID=83749611
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210936842.2A Active CN115250745B (en) | 2022-08-05 | 2022-08-05 | A fully automatic fruit picking robot and picking method based on vision technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115250745B (en) |
Families Citing this family (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| MX2021003767A (en) * | 2018-10-08 | 2021-05-27 | Advanced Farm Tech Inc | Autonomous crop harvester. |
| CN115643902B (en) * | 2022-11-03 | 2024-05-10 | 华南农业大学 | Litchi targeting accurate vibration harvesting equipment and harvesting method thereof |
| CN116079712B (en) * | 2022-12-01 | 2024-08-06 | 杭州电子科技大学 | Three-dimensional space clamping height control method of mechanical arm |
| CN115937314B (en) * | 2022-12-23 | 2023-09-08 | 南京林业大学 | Method for detecting growth posture of oil tea fruits |
| CN115781766B (en) * | 2023-02-13 | 2023-04-18 | 安徽文达信息工程学院 | Mechanical arm with visual positioning system |
| CN116965236A (en) * | 2023-04-28 | 2023-10-31 | 中国农业大学 | Small watermelon picking robot and picking method based on greenhouse three-dimensional cultivation mode |
| CN116439018B (en) * | 2023-05-05 | 2024-01-02 | 仲恺农业工程学院 | A seven-degree-of-freedom fruit picking robot and its picking method |
| CN116602123A (en) * | 2023-07-03 | 2023-08-18 | 华南农业大学 | Shaddock fruit harvesting robot and harvesting method |
| CN116985270B (en) * | 2023-07-19 | 2024-07-02 | 玉溪师范学院 | A bionic fossil repairing device |
| CN116872207B (en) * | 2023-07-27 | 2025-11-14 | 苏州大学 | Intelligent Fruit Harvesting Device with Robotic Arm Based on Radar Array and RGB-D Camera Fusion Positioning and Its Control Method |
| CN116868772B (en) * | 2023-08-14 | 2025-08-26 | 深圳市普蓝机器人有限公司 | Robot for fruit recognition and picking based on vision and method for using the robot |
| CN117530053A (en) * | 2023-11-09 | 2024-02-09 | 中国农业科学院农业信息研究所 | Picking robot and tail end grabbing device and method thereof |
| CN117413686B (en) * | 2023-11-13 | 2025-10-03 | 吉林大学 | A picking robot with learning ability and a picking method |
| CN118370088A (en) * | 2024-04-18 | 2024-07-23 | 山东农业大学 | A wide-area picking device for modern orchards and an all-round posture adjustment method |
| CN118180008A (en) * | 2024-04-26 | 2024-06-14 | 双鸭山双煤机电装备有限公司 | A track cleaning device for coal mine inspection robots |
| CN118511737A (en) * | 2024-06-04 | 2024-08-20 | 海南大学 | A potato picking robot and a potato picking method |
| CN119014209B (en) * | 2024-10-16 | 2025-04-22 | 宜元素生态农场(扬州)有限公司 | A cherry tomato picking robot |
| CN120036186A (en) * | 2025-02-28 | 2025-05-27 | 江苏华骏生物科技有限公司 | Grifola frondosa picking robot |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101273688A (en) * | 2008-05-05 | 2008-10-01 | 江苏大学 | Flexible picking device and method for citrus picking robot |
| CN203752160U (en) * | 2014-03-25 | 2014-08-06 | 甘肃农业大学 | Quick under-actuated absorbing picking manipulator |
| CN106508282A (en) * | 2016-09-22 | 2017-03-22 | 华南农业大学 | Clamping and shearing type manipulator end effector |
| CN107471212A (en) * | 2017-08-24 | 2017-12-15 | 佛山伊贝尔科技有限公司 | A kind of flexible pneumatic manipulator |
| CN107711078A (en) * | 2017-10-12 | 2018-02-23 | 谷新运 | Tomato absorption clamps synchronous picking end effector and corresponding picking mechanism and method |
| CN207836206U (en) * | 2017-11-21 | 2018-09-11 | 祝锐 | A kind of portable white heart shaddock stripper unit |
| CN109479522A (en) * | 2018-12-27 | 2019-03-19 | 甘肃农业大学 | A kind of fruit picking robot and its picking method |
| CN111656959A (en) * | 2020-07-27 | 2020-09-15 | 荆门掇刀双井刘伟西瓜专业合作社 | A pick device for watermelon |
| CN112743566A (en) * | 2021-03-05 | 2021-05-04 | 广东海洋大学 | Pulling-in type mechanical claw capable of changing form |
| CN112976029A (en) * | 2021-03-11 | 2021-06-18 | 南京农业大学 | Soft electrostatic adhesion coating type bionic octopus manipulator |
| CN113330915A (en) * | 2021-05-26 | 2021-09-03 | 华南农业大学 | Self-adaptive cotton harvesting method based on binocular vision recognition and intelligent mechanical harvesting device |
| CN114402806A (en) * | 2022-02-18 | 2022-04-29 | 湖北汽车工业学院 | A kind of spherical fruit picking robot and picking method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10327399B2 (en) * | 2016-11-29 | 2019-06-25 | Invia Robotics, Inc. | Harvesting robots for hydroponics |
-
2022
- 2022-08-05 CN CN202210936842.2A patent/CN115250745B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101273688A (en) * | 2008-05-05 | 2008-10-01 | 江苏大学 | Flexible picking device and method for citrus picking robot |
| CN203752160U (en) * | 2014-03-25 | 2014-08-06 | 甘肃农业大学 | Quick under-actuated absorbing picking manipulator |
| CN106508282A (en) * | 2016-09-22 | 2017-03-22 | 华南农业大学 | Clamping and shearing type manipulator end effector |
| CN107471212A (en) * | 2017-08-24 | 2017-12-15 | 佛山伊贝尔科技有限公司 | A kind of flexible pneumatic manipulator |
| CN107711078A (en) * | 2017-10-12 | 2018-02-23 | 谷新运 | Tomato absorption clamps synchronous picking end effector and corresponding picking mechanism and method |
| CN207836206U (en) * | 2017-11-21 | 2018-09-11 | 祝锐 | A kind of portable white heart shaddock stripper unit |
| CN109479522A (en) * | 2018-12-27 | 2019-03-19 | 甘肃农业大学 | A kind of fruit picking robot and its picking method |
| CN111656959A (en) * | 2020-07-27 | 2020-09-15 | 荆门掇刀双井刘伟西瓜专业合作社 | A pick device for watermelon |
| CN112743566A (en) * | 2021-03-05 | 2021-05-04 | 广东海洋大学 | Pulling-in type mechanical claw capable of changing form |
| CN112976029A (en) * | 2021-03-11 | 2021-06-18 | 南京农业大学 | Soft electrostatic adhesion coating type bionic octopus manipulator |
| CN113330915A (en) * | 2021-05-26 | 2021-09-03 | 华南农业大学 | Self-adaptive cotton harvesting method based on binocular vision recognition and intelligent mechanical harvesting device |
| CN114402806A (en) * | 2022-02-18 | 2022-04-29 | 湖北汽车工业学院 | A kind of spherical fruit picking robot and picking method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115250745A (en) | 2022-11-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115250745B (en) | A fully automatic fruit picking robot and picking method based on vision technology | |
| CN113330915B (en) | Self-adaptive cotton harvesting method based on binocular vision recognition and intelligent mechanical harvesting device | |
| CN103950033B (en) | The mechanical arm of fruit picking robot and end effector and fruit picking method | |
| CN111587665B (en) | Four-degree-of-freedom multi-eye visual rotary-flying picking robot and picking method thereof | |
| CN103503639B (en) | A kind of both arms fruits and vegetables are gathered robot system and fruits and vegetables collecting method thereof | |
| CN109699301B (en) | Intelligent citrus picking machine and citrus picking method | |
| US20200376656A1 (en) | Mobile Robot Morphology | |
| CN108401685A (en) | A kind of spheral fruit picking robot | |
| CN115553132A (en) | Litchi recognition method based on visual algorithm and bionic litchi picking robot | |
| CN114402806A (en) | A kind of spherical fruit picking robot and picking method | |
| Yang et al. | Development of a pumpkin fruits pick-and-place robot using an RGB-D camera and a YOLO based object detection AI model | |
| CN216058333U (en) | Intelligent movement fruit picking robot | |
| CN108811766A (en) | A kind of man-machine interactive fruits and vegetables of greenhouse harvesting robot system and its collecting method | |
| CN203775716U (en) | Fruit picking manipulator | |
| CN112249324A (en) | Garbage picking robot device facing high-risk scenic spot and working method | |
| CN116965236A (en) | Small watermelon picking robot and picking method based on greenhouse three-dimensional cultivation mode | |
| Oliveira et al. | End-effectors for harvesting manipulators-state of the art review | |
| Zhao et al. | Research on design and experiment of rear-drive apple harvesting robotic arm based on obstacle avoidance posture conditions | |
| CN218042576U (en) | Fruit picking device | |
| CN119836935A (en) | Multi-arm apple picking robot | |
| CN212520048U (en) | Four-degree-of-freedom multi-vision rotary flying type picking robot | |
| CN120476846A (en) | Multi-arm picking robot based on YOLOv identification algorithm and fruit identification method | |
| US20250135636A1 (en) | Systems and methods for grasping objects with unknown or uncertain extents using a robotic manipulator | |
| CN114451146B (en) | Method and system for picking fruits and vegetables | |
| CN208175422U (en) | A kind of spheral fruit picking robot |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |