Smart cobots thanks to 3D computer vision | Cobots manual - part 4

This article was created by ACRO (KU Leuven) within the framework of the COOCK+ ROBUST project, in collaboration with Sirris.

What makes vision technology so suitable for cobot applications

Automation is not self-evident for SMEs active in the sheet metal processing sector. Production environments are dynamic, batch sizes are small and the supply of parts is constantly changing. As a result, investments in process automation are hard to recoup, even with the shortage of technicians in the sector.

Cobots offer a potential alternative, but their deployment is limited by a crucial factor: visibility. Without some form of visual perception, it is difficult for a cobot to deal independently with variations in parts, position and orientation. This is precisely where mobile cobots come in: they can be flexibly deployed at different workstations, thereby increasing the cobot utilisation rate. Combined with advanced vision systems, they offer solutions to the challenges facing the sheet metal processing industry.

Visual perception as leverage

As part of the COOCK+ ROBUST project, Sirris and KU Leuven's ACRO research group are investigating how cobots can become more visually "intelligent". Using 3D computer vision, cobots learn to recognise and correctly position objects.

A system like this consists of three core components:

• Object recognition: the cobot recognises the workpiece based on shape, contour or depth
• Location: the position and orientation (pose) of the object is calculated
• Reference determination: the location of the object is converted to the cobot's coordinate system

This allows a cobot to orient itself autonomously and perform gripping tasks, including with products of varying shape and position.
 

Localisation of lasercut workpieces in depth images using MVTec HALCON's 3D shape-based matching

Tested in practice in an industrial case study

A feasibility study conducted with a Flemish sheet metal processor looked into whether a cobot could handle sheet metal pieces from an unstructured or semi-structured supply. These were thin, laser-cut parts of varying shape, orientation and size.

The biggest challenges:

• Variable backgrounds and lighting
• No prior knowledge of sequence or parts type
• Barely any depth information with thin plates
• Cycle time and fault tolerance requirements

Experiments were carried out with multiple algorithms using a combination of 3D scanning and HALCON vision software (MVTec). The method that performed best? 3D shape-based matching.

Why 3D shape-based matching works

This approach works on the basis of CAD models of the parts. Instead of a full 3D point cloud, a depth image is created in which the relevant contours are highlighted. This is done by means of distance projection along the scene's normal vector.

Advantages:

• Fast: 1.22 seconds per depth image on average
• Accurate: matching scores up to 96%
• Robust: resistant to shadows and lighting variation
• Scalable: can also be used for large parts by means of CAD segmentation

Combined with calibration objects and hand-eye calibration, the output of the vision system is correctly translated into movements of the cobot, enabling precise gripping tasks.
 

Localisation of lasercut workpieces in greyscale images using MVTec HALCON's 3D shape-based matching

What does this mean for your workplace?

Thanks to this vision technology, a cobot can:

• Pick up workpieces without a fixed setup or custom moulds
• Make corrections for deviating positions or orientations
• Reorient itself on the basis of visual reference points rather than physical anchors
• Work efficiently with variable product ranges

The approach is particularly useful for sheet metal processing companies that frequently switch between small batches and different products, but still want to commit to automation.