Hi all,
I am trying to do something related to object tracking by quadcopter in the vrep simulator, here is the environment setup:
1).I have a vision sensor mounted on the quadcopter model which can captures the image which the quadcopter sees.
2).I also have a moving object (like a Bill person) in the scene, which is designed to move randomly in the scene.
3).I know how to get the absoluete world coordinates of the quadcopter, the vision sensor and the Bill target by the APIs of vrep.
Then my question is as follow:
How to get the bounding box location of the target Bill in the image plane captured by the vision sensor?
I think it should be possible since all the relevant transforming matrices can be obtained by some specific API. But I am not that familiar to how these transformations actually work. Could you please provide me some detailed guidance on how to achieve this.
Thanks very much.
Mapping from 3D world coordinates to 2D image coordinates

 Posts: 9
 Joined: 31 Jan 2016, 16:16
Re: Mapping from 3D world coordinates to 2D image coordinate
Hello,
You will have to use transformation matrices. Make sure you read about that, since this is one basic element when computing coordinates in another reference frame.
Here a basic example. Imagine you have 2 objects:
world > objectA > objectB
You can get the transformation matrix of the objects relative to the world (i.e. absolute transformation matrix) with:
You can also get for instance the transformation matrix of objectB relative to objectA with:
mb_relToA can also be computed with matrices like:
Now you can also do other matrix operations, for instance find the reference frame matrix of objectB rotated around objectB frame Z axis by pi:
If your case, you probably want to compute all the matrices/positions relative to the vision sensor's reference frame. Have also a look at simMultiplyVector.
Cheers
You will have to use transformation matrices. Make sure you read about that, since this is one basic element when computing coordinates in another reference frame.
Here a basic example. Imagine you have 2 objects:
world > objectA > objectB
You can get the transformation matrix of the objects relative to the world (i.e. absolute transformation matrix) with:
Code: Select all
ma=simGetObjectMatrix(objectAHandle,1)
mb=simGetObjectMatrix(objectBHandle,1)
Code: Select all
mb_relToA=simGetObjectMatrix(objectBHandle,objectAHandle)
Code: Select all
ma=simGetObjectMatrix(objectAHandle,1)
mb=simGetObjectMatrix(objectBHandle,1)
ma_inverse=simGetInvertedMatrix(ma)
mv_relToA=simMultiplyMatrices(ma_inverse,mb)
Code: Select all
rot=simBuildMatrix({0,0,0},{0,0,math.pi})
mb_rotated=simMultiplyMatrices(mb,rot)
Cheers

 Posts: 9
 Joined: 31 Jan 2016, 16:16
Re: Mapping from 3D world coordinates to 2D image coordinate
Thanks for your reply. I tried to use the transformation matrix do the coordinates mapping but still encounters some problem.
First I extract the transformation matrix of the target object relative to the vision sensor frame:
Then I get the coordinates of the target object in the world frame like this:
Assuming the transformation matrix is the camera mapping matrix, I can get the 2D coordinates of the target on the vision sensor plane like this:
The inhomogeneous pixel location on the image should be:
However, I found the result seems to be incorrect. For now I set the resolution of the vision sensor to 128x128, but the above computed coordinates are not in that range. How can I get the pixel location in the range of 128x128 image?
To be more clear, I also upload the sample scene(*.ttt) file I used for testing. https://www.dropbox.com/s/7e4lk3o1yikjs ... t.ttt?dl=0
First I extract the transformation matrix of the target object relative to the vision sensor frame:
Code: Select all
transformMatrix = simGetObjectMatrix(shapeObj, frontSen)
Code: Select all
shapePos = simGetObjectPosition(shapeObj, 1)
Code: Select all
coord = simMultiplyVector(transformMatrix, shapePos)
Code: Select all
{coord[1]/coord[3], coord[2]/coord[3]}
To be more clear, I also upload the sample scene(*.ttt) file I used for testing. https://www.dropbox.com/s/7e4lk3o1yikjs ... t.ttt?dl=0
Re: Mapping from 3D world coordinates to 2D image coordinate
If you need to x/y position of the target object in the vision sensor's reference frame, then simply do:
The reference frame of a vision sensor has Y pointing up, x pointing left, and z pointing in the viewing direction.
Cheers
Code: Select all
transformMatrix = simGetObjectMatrix(shapeObj, frontSen)
xRel=transformMatrix[4]
yRel=transformMatrix[8]
zRel=transformMatrix[12]
Cheers

 Posts: 9
 Joined: 31 Jan 2016, 16:16
Re: Mapping from 3D world coordinates to 2D image coordinate
Thanks very much. I've got it.