## How to simulate a depth camera?

Typically: "How do I... ", "How can I... " questions
hxg
Posts: 2
Joined: 09 Jul 2020, 04:37

### How to simulate a depth camera?

Hi,

I am trying to simulate a depth camera with vrep and each pixels need R,G,B values and relative x,y,z values. It is simple to have RGB but I've come across some problems with x,y,z values. The customization script of the vision sensor was written as

Code: Select all

function sysCall_vision(inData)
simVision.sensorImgToWorkImg(inData.handle)
trig,packedPacket=simVision.coordinatesFromWorkImg(inData.handle,{512,512},false)

sim.setStringSignal("camera_d_msg",packedPacket)
simVision.workImgToSensorImg(inData.handle,false)

end

Then I visualized the point cloud with MATLAB. The rerurned z values seems incorrect as the QR codes on the wall were detected "behind" the wall, which means these relative z values were larger than those of pixels of the wall. And all of the pixels of floor had similar z values and were also larger than the wall. Did I use this function in a wrong way? Are there any other methods that can be used to simulate a depth camera?

Thanks a lot,
hxg

coppelia
Posts: 7966
Joined: 14 Dec 2012, 00:25

### Re: How to simulate a depth camera?

Hello,

are your extracted points distorted? I guess that you simply forgot to multiply the points with the vision sensor frame transformation, i.e. each point should be multiplied with the matrix of the vision sensor in order to obtain absolute coordinates. Maybe try something like (not tested):

Code: Select all

function sysCall_vision(inData)
simVision.sensorImgToWorkImg(inData.handle)
trig,packedPacket=simVision.coordinatesFromWorkImg(inData.handle,{512,512},false)
local ptsX=sim.unpackFloatTable(packedPacket,0,1)
local ptsY=sim.unpackFloatTable(packedPacket,1,1)
local pts=sim.unpackFloatTable(packedPacket,2)
local correctedPacket={}
local m=sim.getObjectMatrix(inData.handle,-1)
for i=0,(ptsX*ptsY)-1,1 do
local v={pts[4*i+1],pts[4*i+2],pts[4*i+3]}
sim.multiplyVector(m,v)
pts[4*i+1]=v[1]
pts[4*i+2]=v[2]
pts[4*i+3]=v[3]
end
sim.setStringSignal("camera_d_msg",correctedPacket)
simVision.workImgToSensorImg(inData.handle,false)
end
Above might eb quite slow. In next release (V4.1.0, out within 1-2 weeks), you'll be able to do:

Code: Select all

    trig,packedPacket=simVision.coordinatesFromWorkImg(inData.handle+sim.handleflag_abscoords,{512,512},false) 
to retrieve points relative to the absolute reference frame, which will be much faster than above.

Cheers

hxg
Posts: 2
Joined: 09 Jul 2020, 04:37

### Re: How to simulate a depth camera?

Hello,

Thanks for your reply. The solution you've provided is to transform the relative values to absolute coordinate values, which makes the point cloud more comprehensive but does not solve the problem. For example, in this situation I put two QR code on the wall and this is the image observed by vision sensor

And then the observed point cloud with absolute coordinate values were shown in 3D:

The interesting problem is that the QR codes were splitted from the wall and appeared behind it, and the floor was "standing upright" on the image. Do you have any idea?

Thanks,
hxg

coppelia