Precise Bin Picking – Open Problems and an Alternative Approach
Introduction
Precise picking from a box of bulk material for machine tending or assembly remains one of the most coveted applications in industrial automation. And the economic potential of a fully autonomous bin picking system is enormous: There are no specific requirements on how items need to be packaged in the previous machining step. The pieces can be just dropped into a box and transported with high density. There are no additional costs for the construction or loading of object-specific magazines.
Even though a wide range of bin picking systems are available on the market, the number of installations found in production plants is still surprisingly low. What could be the reason? The so-called Pareto principle offers a possible explanation: it states that 80% of the effort is spent solving the top 20% of challenges in the development of a bin picking system.
Yet, only a 100% autonomous system is commercially viable, i.e., the remaining 20% of issues that a system may have can hardly be ignored since they will lead to errors requiring manual interventions from a human operator.
In this blog post we want to shed some light on these “last problems” in precise bin picking.
From a technical point of view, the goals and requirements are as follows: The vision system detects the precise location and orientation (a pose) of one or multiple work pieces randomly distributed inside a box or bin. Next it will apply to each detection a pose that the gripper (i.e., the TCP of the robot) should assume relative to the detected object for picking. We call this desired relative pose between object and gripper a grip. Typically, there are multiple alternatives to pick an object, and operators are advised to define more than a single grip to increase the chance of finding one that is free of collisions (more on this further below). Ideally, the robot moves directly from the pick to the desired place pose without having to adjust the grip with the help of mechanical fixtures, requiring a costly redesign for every new object geometry to be processed by the robot cell.
For the sake of illustration, we have put together a typical bin picking demo, in which the robot picks nozzles and inserts them precisely into a mold on a vacuum cleaner case. It did not take long for the edge cases to be caught on video.
Deadlocks
A deadlock occurs when the vision system cannot find a single collision-free grip, not due to shortcomings of the vision system but just because there are none.
One can distinguish between two types of potential collisions:
- those with other pieces or the box itself while picking,
- those between the part attached to the gripper and, e.g., the machine while placing.
The former is relatively easy to resolve by slightly rearranging items in the box. This perturbation can be random and executed by the robot itself or an additional actor (e.g., a vibrating table) but will negatively impact the achievable mean cycle time. The latter on the other hand will require significant efforts to prevent. Take for example a situation where the piece is laying “upside-down”, i.e., where the grip that is required to assemble it or get it into the machine without collision is occluded. The only remedy here is an extra mechanism outside the box to flip the piece before re-gripping it from the appropriate side.
Unreachable poses
The vision system has selected a picking pose which is feasible from its own point of view but turns out to be unreachable under the kinematic constraints of the robot. Obviously, this aggravates the situation above by limiting the possibly small number of gripping options even further. The risk of encountering a deadlock increases.
Some robot controllers can check isolated poses for feasibility, reject them if necessary, and skip ahead to the next best collision-free pose from the vision system. Often, however, the path to the pick pose closely passes a so-called kinematic singularity which makes an interruption of the motion program inevitable. The only fail-safe solution seems to be selecting a robot with possibly oversized reach, raising the obvious question after an adequate balance between cost and benefit.
Path planning failures
Here, the actual pick was successful but the piece sticking at the end-effector collides with the box or other pieces on the way out. Finding a trajectory connecting pick and place pose that is collision-free and adheres to the kinematic constraints of the robot (see last issue) is a very hard optimization problem, requiring efficient fusion of data from multiple sources, in particular, a 3d image capturing the unknown portion of the scene (i.e., the pieces inside the box), a geometric/CAD model of the known portion of the scene (i.e., the box itself and mechanical extensions), and a kinematic model of the robot.
Very few deployments in real production environments tackle this optimization but, for the sake of simplicity and speed, resort to simple heuristics such as moving towards safety in a direction where collisions are least likely, e.g., straight up relative to the bottom of the box. A feasible path to the part destination is well-known and can be hardcoded before runtime.
Insufficient contact
This problem mainly affects vacuum grippers such as the one seen in the video. In order to establish the vacuum, the gripper needs to make even contact with the surface of the piece which is not always possible due to measurement uncertainty or the (uneven) shape of the suction cup itself. What starts as a slight touch quickly becomes a collision. If the mass of pieces is sufficiently inelastic (i.e., provides enough counterforce) that “collision” could even be helpful but otherwise the target piece will slip away.
An insufficiently firm grip can have two consequences: Either 1. the piece is never picked or lost on the way to the machine or, even worse, 2. the position of the part relative to the gripper through slippage and thus the place pose is not as intended, leading to damaging of a machine or assembly rig in the worst case.
Moving pieces
For safety reasons, most pick & place systems will capture an image after each pick because as we just saw. This works well as long as the scene remains static during the time interval between capture and pick. Admittedly, this is a bit of an exotic incident, but as the video proves, this assumption may well be violated at times. The consequences are similar to those of the last issue: an “air pick” with repercussions to cycle time in the best case, damage to a machine in the worst.
Conclusion
We observe that one of the major challenges in bin picking is a direct transfer from the bin to the desired destination. Most vision systems on the market invest a significant amount of processing power to find poses inside the box to enable this direct transfer, yet, you will hardly find a system for precise bin picking in practice without a mechanical centering device.
Wouldn’t it be wiser to rely on the vision system only for separation, much like in (imprecise) pick & pack systems for the intralogistics vertical, i.e., to first isolate an object from the box using a coarse grip and refine the grip after imaging it outside the box? Relaxing the requirements on the grip will certainly reduce the risk of deadlocks. But also the object recognition results will improve, given that confusion with neighboring parts inside the box have a negative impact on the confidence of each result. If cycle time permits, in a second step, the vision system can devote all of its intelligence to grip refinement as to empower the robot to achieve turning and/or centering without the help of an external device or actor.
But of course, choosing the optimal strategy (one-step vs. two-step) under the boundary conditions of the process and weighing reliability, speed, and cost against each other needs to ultimately rely on the judgement of the professional machine builder/system integrator.
