Page 1 of 1
Vision sensor object masks
Posted: 21 Apr 2024, 11:02
by mohammadhosseink
Hello,
Does Coppeliasim provide any method to get the mask of a specific shape in a vision sensor? Basically, the objective would be to retrieve the pixels in a vision sensor that are associated with a particular object, i.e. where the object is present in the pixel coordinate of the vision sensor. Any ideas?
Cheers,
Re: Vision sensor object masks
Posted: 22 Apr 2024, 09:04
by fferri
You can achieve something like selective rendering by setting the vision sensor to explicit handling, then adjusting visibility layer of all objects before/after rendering to only "see" the interesting objects. Depth map gives you something almost resembling a binary map (simply apply some threshold). E.g.:
Code: Select all
function sysCall_init()
sim = require('sim')
simVision = require('simVision')
sensor = sim.getObject('.')
targetObject = sim.getObject('/Cuboid')
end
function sysCall_sensing()
local origVis = changeVisibility({[targetObject] = 1})
sim.handleVisionSensor(sensor)
simVision.sensorDepthMapToWorkImg(sensor)
simVision.workImgToSensorImg(sensor)
changeVisibility(origVis)
end
function changeVisibility(objMap)
local old = {}
for i, h in ipairs(sim.getObjectsInTree(sim.handle_scene)) do
old[h] = sim.getObjectInt32Param(h, sim.objintparam_visibility_layer)
local new = objMap[h] or 0
sim.setObjectInt32Param(h, sim.objintparam_visibility_layer, new)
end
return old
end
Re: Vision sensor object masks
Posted: 22 Apr 2024, 09:25
by fferri
Actually, even better (and faster) than that, you can set the vision sensor render mode to "color coded handles", then each object is rendered with a unique color corresponding to its handle.
Then you can go through all pixels and decode which object it relates to (handle = rgbData[3*i+0]+(rgbData[3*i+1]<<8)+(rgbData[3*i+2]<<16)).
Also, the main difference in the two approaches is occlusions, as the first approach (manually toggling object visibility before/after render) would always render the object without any occluding object.
Re: Vision sensor object masks
Posted: 25 Apr 2024, 09:09
by mohammadhosseink
The second solution is great. thanks. Also, how can I know which color code (RGB) is assigned to which object handle? I need this to find the mask of the object.
Re: Vision sensor object masks
Posted: 25 Apr 2024, 12:50
by fferri
fferri wrote: ↑22 Apr 2024, 09:25
handle = rgbData[3*i+0]+(rgbData[3*i+1]<<8)+(rgbData[3*i+2]<<16)