Mapping from 3D world coordinates to 2D image coordinates

[lisiyi1992]

Hi all,

I am trying to do something related to object tracking by quadcopter in the v-rep simulator, here is the environment setup:

1).I have a vision sensor mounted on the quadcopter model which can captures the image which the quadcopter sees.
2).I also have a moving object (like a Bill person) in the scene, which is designed to move randomly in the scene.
3).I know how to get the absoluete world coordinates of the quadcopter, the vision sensor and the Bill target by the APIs of v-rep.

Then my question is as follow:
How to get the bounding box location of the target Bill in the image plane captured by the vision sensor?
I think it should be possible since all the relevant transforming matrices can be obtained by some specific API. But I am not that familiar to how these transformations actually work. Could you please provide me some detailed guidance on how to achieve this.

Thanks very much.

[coppelia]

Hello,

You will have to use transformation matrices. Make sure you read about that, since this is one basic element when computing coordinates in another reference frame.
Here a basic example. Imagine you have 2 objects:

world --> objectA --> objectB

You can get the transformation matrix of the objects relative to the world (i.e. absolute transformation matrix) with:

Code: Select all

ma=simGetObjectMatrix(objectAHandle,-1)
mb=simGetObjectMatrix(objectBHandle,-1)

You can also get for instance the transformation matrix of objectB relative to objectA with:

Code: Select all

mb_relToA=simGetObjectMatrix(objectBHandle,objectAHandle)

mb_relToA can also be computed with matrices like:

Code: Select all

ma=simGetObjectMatrix(objectAHandle,-1)
mb=simGetObjectMatrix(objectBHandle,-1)
ma_inverse=simGetInvertedMatrix(ma)
mv_relToA=simMultiplyMatrices(ma_inverse,mb)

Now you can also do other matrix operations, for instance find the reference frame matrix of objectB rotated around objectB frame Z axis by pi:

Code: Select all

rot=simBuildMatrix({0,0,0},{0,0,math.pi})
mb_rotated=simMultiplyMatrices(mb,rot)

If your case, you probably want to compute all the matrices/positions relative to the vision sensor's reference frame. Have also a look at simMultiplyVector.

Cheers

[lisiyi1992]

Thanks for your reply. I tried to use the transformation matrix do the coordinates mapping but still encounters some problem.

First I extract the transformation matrix of the target object relative to the vision sensor frame:

Code: Select all

transformMatrix = simGetObjectMatrix(shapeObj, frontSen)

Then I get the coordinates of the target object in the world frame like this:

Code: Select all

shapePos = simGetObjectPosition(shapeObj, -1)

Assuming the transformation matrix is the camera mapping matrix, I can get the 2D coordinates of the target on the vision sensor plane like this:

Code: Select all

coord = simMultiplyVector(transformMatrix, shapePos)

The inhomogeneous pixel location on the image should be:

Code: Select all

{coord[1]/coord[3], coord[2]/coord[3]}

However, I found the result seems to be incorrect. For now I set the resolution of the vision sensor to 128x128, but the above computed coordinates are not in that range. How can I get the pixel location in the range of 128x128 image?

To be more clear, I also upload the sample scene(*.ttt) file I used for testing. https://www.dropbox.com/s/7e4lk3o1yikjs ... t.ttt?dl=0

[coppelia]

If you need to x/y position of the target object in the vision sensor's reference frame, then simply do:

Code: Select all

transformMatrix = simGetObjectMatrix(shapeObj, frontSen)
xRel=transformMatrix[4]
yRel=transformMatrix[8]
zRel=transformMatrix[12]

The reference frame of a vision sensor has Y pointing up, x pointing left, and z pointing in the viewing direction.

Cheers

[lisiyi1992]

Thanks very much. I've got it.