Vision sensor sync with kuka youbot

Typically: "How do I... ", "How can I... " questions
Trilo
Posts: 5
Joined: 22 Dec 2016, 22:20

Vision sensor sync with kuka youbot

Postby Trilo » 16 May 2017, 17:51

hello there,

Basically my project is to use the vision to detect the object then pick and place the object at the desired position.

The problem that i am facing right now is: The position of the object in the vision sensor is different from the position of the object in the real world..

So is there any solution for this...?


https://www.dropbox.com/sh/a6ogrg9p09qz ... R55va?dl=0

coppelia
Site Admin
Posts: 5728
Joined: 14 Dec 2012, 00:25

Re: Vision sensor sync with kuka youbot

Postby coppelia » 17 May 2017, 16:20

Hello,

yes. You will have to transform the X/Y pixel position, and pixel depth value, to a 3D position relative to the world coordinate frame.

For that you need to take several things into account, among others:

  • The view angle of the vision sensor
  • The resolution of the vision sensor
  • The transformation matrix of the vision sensor
  • And of course the X/Y pixel position, and depth value at that pixel position.

If your vision sensor is in orthogonal view mode, then things get simpler. But the approach is still the same.

Cheers

Trilo
Posts: 5
Joined: 22 Dec 2016, 22:20

Re: Vision sensor sync with kuka youbot

Postby Trilo » 17 May 2017, 19:22

Sorry i really don't get it...
I have change my vision to orto...

My cam code

Code: Select all

simSetThreadSwitchTiming(2) -- Default timing for automatic thread switching
simDelegateChildScriptExecution()
--out=simAuxiliaryConsoleOpen("vision",8,1)
out2=simAuxiliaryConsoleOpen("realworld",8,1)
cam=simGetObjectHandle("cam")
target=simGetObjectHandle("target")
target2=simGetObjectHandle("target2") 

while (simGetSimulationState()~=sim_simulation_advancing_abouttostop)do
    result,t0,t1=simReadVisionSensor(cam)
    if (t1) then
       posx=t1[5]
        posy=t1[6]
        --simAuxiliaryConsolePrint(out,string.format("\nx: %0.2f, y %0.2f",posx,posy))
 
   local r,nearClip=simGetObjectFloatParameter(cam,(1000))
    local r,farClip=simGetObjectFloatParameter(cam,(1001))
   local depthMapPortion=simGetVisionSensorDepthBuffer(cam,posx,posy,1,1)
    local blobDepth=nearClip+(farClip-nearClip)*depthMapPortion[1]
   

transformMatrix = simGetObjectMatrix(target, -1)
xRel=transformMatrix[4]
yRel=transformMatrix[8]
zRel=transformMatrix[12]
simAuxiliaryConsolePrint(out2,string.format("\nx:%0.4f, y %0.4f, z %0.4f",xRel,yRel,zRel)) --display the value of y z of target when moving

--simInvertMatrix(transformMatrix)
--local v={1,2,3}
--local vp=simMultiplyVector(transformMatrix,v)

simSwitchThread()
end
end


i have no idea how to code the vision sensor... Can you please help me through this?

coppelia
Site Admin
Posts: 5728
Joined: 14 Dec 2012, 00:25

Re: Vision sensor sync with kuka youbot

Postby coppelia » 20 May 2017, 05:20

You need to focus on an image element, e.g. a specific pixel. If your sensor is orthogonal, then you have something like:

Code: Select all

-- Input is sensor (i.e. sensor handle), and pixelX/pixelY
local result,nclip=simGetObjectFloatParameter(sensor,sim_visionfloatparam_near_clipping)
local result,fclip=simGetObjectFloatParameter(sensor,sim_visionfloatparam_far_clipping)
local result,orthoS=simGetObjectFloatParameter(sensor,sim_visionfloatparam_ortho_size)
local result,resX=simGetObjectInt32Parameter(sensor,sim_visionintparam_resolution_x)
local result,resY=simGetObjectInt32Parameter(sensor,sim_visionintparam_resolution_y)
local sensorM=simGetObjectMatrix(sensor,-1)
local sensorRelX,sensorRelY
if resX>=resY then
    sensorRelX=(resX*0.5-pixelX)*orthoS/resX
    sensorRelY=((pixelY-resY*0.5)*orthoS/resY)*resY/resX
else
    sensorRelX=((resX*0.5-pixelX)*orthoS/resX)*resX/resY
    sensorRelY=(pixelY-resY*0.5)*orthoS/resY
end
local db=simGetVisionSensorDepthBuffer(sensor,pixelX,pixelY,1,1)
local sensorRelZ=nclip+(fclip-nclip)*db[1]
local pixelCoordRelativeToSensor={sensorRelX,sensorRelY,sensorRelZ}
local pixelCoordRelativeToWorld=simMultiplyVector(sensorM,pixelCoordRelativeToSensor)


Cheers

Trilo
Posts: 5
Joined: 22 Dec 2016, 22:20

Re: Vision sensor sync with kuka youbot

Postby Trilo » 20 May 2017, 18:47

Hello,
Good day...

i have done what you have told me to do...

This is my dropbox link:
https://www.dropbox.com/s/ucuvasx2hcp4n ... g.ttt?dl=0

my cam code is :

Code: Select all

--threadFunction=function()
simSetThreadSwitchTiming(2) -- Default timing for automatic thread switching
simDelegateChildScriptExecution()
out=simAuxiliaryConsoleOpen("vision posx,y & focal",8,1)
--out2=simAuxiliaryConsoleOpen("real",8,1)
--out3=simAuxiliaryConsoleOpen("realworld",8,1)
cam=simGetObjectHandle("cam")
target=simGetObjectHandle("target")

 

while (simGetSimulationState()~=sim_simulation_advancing_abouttostop)do
    result,t0,t1=simReadVisionSensor(cam)
    if (t1) then
       pixelX=t1[5]
       pixelY=t1[6]
       

-- Input is sensor (i.e. sensor handle), and pixelX/pixelY
local result,nclip=simGetObjectFloatParameter(cam,sim_visionfloatparam_near_clipping)
local result,fclip=simGetObjectFloatParameter(cam,sim_visionfloatparam_far_clipping)
local result,orthoS=simGetObjectFloatParameter(cam,sim_visionfloatparam_ortho_size)
local result,resX=simGetObjectInt32Parameter(cam,sim_visionintparam_resolution_x)
local result,resY=simGetObjectInt32Parameter(cam,sim_visionintparam_resolution_y)
local sensorM=simGetObjectMatrix(cam,-1)
local sensorRelX,sensorRelY
if resX>=resY then
    sensorRelX=(resX*0.5-pixelX)*orthoS/resX
    sensorRelY=((pixelY-resY*0.5)*orthoS/resY)*resY/resX
else
    sensorRelX=((resX*0.5-pixelX)*orthoS/resX)*resX/resY
    sensorRelY=(pixelY-resY*0.5)*orthoS/resY
end


local db=simGetVisionSensorDepthBuffer(cam,pixelX,pixelY,1,1)
local sensorRelZ=nclip+(fclip-nclip)*db[1]
local pixelCoordRelativeToSensor={sensorRelX,sensorRelY,sensorRelZ}
local pixelCoordRelativeToWorld=simMultiplyVector(m,pixelCoordRelativeToSensor)
simAuxiliaryConsolePrint(out,string.format("\nx: %0.3f, y: %0.3f, RelX: %0.3f, RelY: %0.3f, RelZ: %0.3f",pixelX,pixelY,sensorRelX,sensorRelY,sensorRelZ))

end

end


There is a error :Lua runtime error: [string "SCRIPT cam"]:39: One of the function's argument type is not correct. (simMultiplyVector)
stack traceback:
[C]: in function 'simMultiplyVector'
[string "SCRIPT cam"]:39: in main chunk


Now i got all the values needed... I am kind of confuse for my next step... Should i get the matrices for the values i get and relate it to the real world?

Thanks you so much...

coppelia
Site Admin
Posts: 5728
Joined: 14 Dec 2012, 00:25

Re: Vision sensor sync with kuka youbot

Postby coppelia » 22 May 2017, 13:39

I had an error in my code: instead of sensorM, I had m in the last line.
But it seems you are missing several basics on matrices, transformation frames, etc.

Cheers


Return to “General questions”

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 22 guests