Vision sensor object masks

[mohammadhosseink]

Hello,

Does Coppeliasim provide any method to get the mask of a specific shape in a vision sensor? Basically, the objective would be to retrieve the pixels in a vision sensor that are associated with a particular object, i.e. where the object is present in the pixel coordinate of the vision sensor. Any ideas?

Cheers,

[fferri]

You can achieve something like selective rendering by setting the vision sensor to explicit handling, then adjusting visibility layer of all objects before/after rendering to only "see" the interesting objects. Depth map gives you something almost resembling a binary map (simply apply some threshold). E.g.:

Code: Select all

function sysCall_init()
    sim = require('sim')
    simVision = require('simVision')
    sensor = sim.getObject('.')
    targetObject = sim.getObject('/Cuboid')
end

function sysCall_sensing()
    local origVis = changeVisibility({[targetObject] = 1})
    sim.handleVisionSensor(sensor)
    simVision.sensorDepthMapToWorkImg(sensor)
    simVision.workImgToSensorImg(sensor)
    changeVisibility(origVis)
end

function changeVisibility(objMap)
    local old = {}
    for i, h in ipairs(sim.getObjectsInTree(sim.handle_scene)) do
        old[h] = sim.getObjectInt32Param(h, sim.objintparam_visibility_layer)
        local new = objMap[h] or 0
        sim.setObjectInt32Param(h, sim.objintparam_visibility_layer, new)
    end
    return old
end

[fferri]

Actually, even better (and faster) than that, you can set the vision sensor render mode to "color coded handles", then each object is rendered with a unique color corresponding to its handle.

Then you can go through all pixels and decode which object it relates to (handle = rgbData[3*i+0]+(rgbData[3*i+1]<<8)+(rgbData[3*i+2]<<16)).

Also, the main difference in the two approaches is occlusions, as the first approach (manually toggling object visibility before/after render) would always render the object without any occluding object.

[mohammadhosseink]

The second solution is great. thanks. Also, how can I know which color code (RGB) is assigned to which object handle? I need this to find the mask of the object.

[fferri]

handle = rgbData[3*i+0]+(rgbData[3*i+1]<<8)+(rgbData[3*i+2]<<16)