How to simulate a depth camera?

Typically: "How do I... ", "How can I... " questions
Post Reply
hxg
Posts: 2
Joined: 09 Jul 2020, 04:37

How to simulate a depth camera?

Post by hxg »

Hi,

I am trying to simulate a depth camera with vrep and each pixels need R,G,B values and relative x,y,z values. It is simple to have RGB but I've come across some problems with x,y,z values. The customization script of the vision sensor was written as

Code: Select all

function sysCall_vision(inData)
    simVision.sensorImgToWorkImg(inData.handle)
    trig,packedPacket=simVision.coordinatesFromWorkImg(inData.handle,{512,512},false)

    sim.setStringSignal("camera_d_msg",packedPacket)
    simVision.workImgToSensorImg(inData.handle,false)

end
Then I visualized the point cloud with MATLAB. The rerurned z values seems incorrect as the QR codes on the wall were detected "behind" the wall, which means these relative z values were larger than those of pixels of the wall. And all of the pixels of floor had similar z values and were also larger than the wall. Did I use this function in a wrong way? Are there any other methods that can be used to simulate a depth camera?

Thanks a lot,
hxg

coppelia
Site Admin
Posts: 10339
Joined: 14 Dec 2012, 00:25

Re: How to simulate a depth camera?

Post by coppelia »

Hello,

are your extracted points distorted? I guess that you simply forgot to multiply the points with the vision sensor frame transformation, i.e. each point should be multiplied with the matrix of the vision sensor in order to obtain absolute coordinates. Maybe try something like (not tested):

Code: Select all

function sysCall_vision(inData)
    simVision.sensorImgToWorkImg(inData.handle)
    trig,packedPacket=simVision.coordinatesFromWorkImg(inData.handle,{512,512},false)
    local ptsX=sim.unpackFloatTable(packedPacket,0,1)
    local ptsY=sim.unpackFloatTable(packedPacket,1,1)
    local pts=sim.unpackFloatTable(packedPacket,2)
    local correctedPacket={}
    local m=sim.getObjectMatrix(inData.handle,-1)
    for i=0,(ptsX*ptsY)-1,1 do
        local v={pts[4*i+1],pts[4*i+2],pts[4*i+3]}
        sim.multiplyVector(m,v)
        pts[4*i+1]=v[1]
        pts[4*i+2]=v[2]
        pts[4*i+3]=v[3]
    end
    sim.setStringSignal("camera_d_msg",correctedPacket)
    simVision.workImgToSensorImg(inData.handle,false)
end
Above might eb quite slow. In next release (V4.1.0, out within 1-2 weeks), you'll be able to do:

Code: Select all

    trig,packedPacket=simVision.coordinatesFromWorkImg(inData.handle+sim.handleflag_abscoords,{512,512},false) 
to retrieve points relative to the absolute reference frame, which will be much faster than above.

Cheers

hxg
Posts: 2
Joined: 09 Jul 2020, 04:37

Re: How to simulate a depth camera?

Post by hxg »

Hello,

Thanks for your reply. The solution you've provided is to transform the relative values to absolute coordinate values, which makes the point cloud more comprehensive but does not solve the problem. For example, in this situation I put two QR code on the wall and this is the image observed by vision sensor

Image

And then the observed point cloud with absolute coordinate values were shown in 3D:

Image
Image

The interesting problem is that the QR codes were splitted from the wall and appeared behind it, and the floor was "standing upright" on the image. Do you have any idea?

Thanks,
hxg

coppelia
Site Admin
Posts: 10339
Joined: 14 Dec 2012, 00:25

Re: How to simulate a depth camera?

Post by coppelia »

Hum... how does the point cloud or visualization look inside of CoppeliaSim? Is it the same or different? Can you share a minimalistic, self-contained scene that illustrates your problem?

Cheers

Post Reply