Object Orientation from Depth Buffer

[jack_mcevoy]

Hi All,

I am using a RGB + Depth vision sensor to extract the x,y,z coordinates of a cuboid within the camera's field of view. Then I am using a second RGB+D camera to extract the angle of rotation at which the cuboid makes with respect to the reference frame of the world.

Here the angle of rotation equals the amount by which the gripper must rotate for the fingers to be parallel to the cuboid edge.

The camera views the object side on, To do this I am indexing the depth buffer whereby the object lies, obtaining the maximum and minimum of those depth values, performing basic trigonometry, and printing out the computed angle. However, after playing around with the camera's resolution, examining the depth buffer values for outliers and dealing with those, I can only ever compute the correct angle of rotation 60% of the time, between 0-90 degrees.

I feel this task is common enough and I'm making it a lot harder than it needs to be, Is there an easier way to do what I am trying to achieve? Or maybe a helpful tip which could improve the current accuracy of my camera methods.

Thank you in advance! Kind regards,
Jack



Here is my working progress simulation model, hopefully one might understand what I'm talking about a bit more after looking. Simply download the model and press play to see what I'm talking about.
https://1drv.ms/u/s!Aruq-Cl-uV5cjgq2tyF ... C?e=4Jo4Ln

Note in the model:
The first loop never returns an angle
An acceptable 'Computed angle' lies within 2.20 degrees of the 'Angle of Rotation'

[coppelia]

Hello,

so from what I understand from your message, in an ideal world, given your calculations, you should be able to always extract the correct orientation of the cube. Since this is not the case, there must be a problem with the vision sensor's data?
The only thing I can think of is (of course next to a faulty calculation on your side), a lack of resolution (in x/y and/or depth). Since you already tried to adjust the resolution in x/y, did you try to narrow down the visible area in depth, via the near and far clipping plane settings?
Additionally, I guess you are aware that sim.getVisionSensorDepthBuffer normally returns values between 0 and 1, for each pixel. If you require values in meters, use the sim.handleflag_depthbuffermeters flag.

Cheers

[jack_mcevoy]

Hi Coppelia,

The fact the model outputs the correct orientation most of the time leads me to believe the calculations don't contain inaccuracies, but I could be wrong. I changed the resolution (X/Y = 1024/256) in an attempt to increase the accuracy of the depth buffer. I also narrowed down the visible area by reducing the clipping planes and perspective angle to as small as possible, again in the hope of better accuracy.

After doing all of this there is no observed change in accuracy, it is still roughly as good as the initial model. I am guessing there is no easier/existing method of doing this. I will look into the vision sensors data, what do you think could be causing errors with the vision sensor data?


P.S. My methodology is as follows:
1) selective colour on work image
2) store all indices where object lies within image in 'objectIndicies'
3) get vision sensor depth buffer and index 'objectIndices' from this to give the depth buffer for the object only 'objectDepthBuffer'
4) remove outlier data from 'objectDepthBuffer' and get the max and min values of the object depth buffer
5) convert these max min values to meters
6)perform trigonometry to object angle of rotation




**EDIT ON 8TH MARCH**
I ran with the above camera model and its inaccuracies. I reverted to a camera resolution of 256x256 after that I changed nothing else.
After mounting the camera onto a robotic arm its accuracy jumped to 84% (and I can't explain why but I can say shadows/lighting did not effect this cameras estimation of orientation)