Python API Vision Sensor

Typically: "How do I... ", "How can I... " questions
Post Reply
series9
Posts: 8
Joined: 04 Mar 2015, 07:01

Python API Vision Sensor

Post by series9 » 04 Mar 2015, 21:08

Hi there, basically i am working on a attached vision sensor to detect blobs on the floor. I need to know the z coordinate as well as the x, y coordinate of the detected blob.

From this thread: viewtopic.php?f=9&t=985

I had managed to follow the procedure until:

original image to work image (if you want blob detection on a RGB image)
some filter to make the blob detection detect only what is needed
blob detection on work image --> returns the second packet of data (blob information)
original depth image to work image
extract coordinates from work image --> returns the third packet of data (positions (x,y,z) of an array of points)

But i am stuck at obtaining

Code: Select all

 ErrorCode,result,packet1 packet2, packet3 = vrep.simxReadVisionSensor(clientID,CamHandle_bp_vs,vrep.simx_opmode_streaming) 
Python gives:

File "C:/Users/joy/Dropbox/PythonWorkspace/Project_v1.py", line 76, in <module>
ErrorCode, result, packet1, packet2, packet3 = vrep.simxReadVisionSensor(clientID,CamHandle_bp_vs,vrep.simx_opmode_streaming)

ValueError: need more than 3 values to unpack

When i change to this line:

Code: Select all

 ErrorCode, packet2, packet3 = vrep.simxReadVisionSensor(clientID,CamHandle_bp_vs,vrep.simx_opmode_streaming) 
it seems to work fine with the variable packet2 and packet3 showing a boolean value and a list of size 3. What does this mean?
any help would be appreciative. Thanks alot

series9
Posts: 8
Joined: 04 Mar 2015, 07:01

Re: Python API Vision Sensor

Post by series9 » 04 Mar 2015, 21:53

ok i sort of get it, how do i retrieve the coordinate of the detected blob using python? i was thinking if it is possible to first retrieve the x y coordinate of the blob and follow by using its coordinate to find the depth field. which is getDepthBuffer function. Please advise

coppelia
Site Admin
Posts: 6740
Joined: 14 Dec 2012, 00:25

Re: Python API Vision Sensor

Post by coppelia » 04 Mar 2015, 22:03

Hello,

the function returns 3 values, not 4.
In your case, the second value is of no interest. But it could be used to check a detection triggered by a specific filter.
The third values represents following:

{packet1, packet2, packet3, etc.}

The number of packets depends on the number of filters and if the filters return values. If no filter returns values, there is just one packet:

Code: Select all

packet1={minI,minR,minG,minB,minDepth,maxI,maxR,maxG,maxB,maxDepth,avgI,avgR,avgG,avgB,avrDepth}
Basically: the minimum/maximum/average of the Intensity/Red/Green/Blue/Depth values over the whole picture.

In a similar way, the blob extraction filter will also return a packet of values:

Code: Select all

blobFilterPacket={blobCount,valueCountPerBlob,blob1Size,blob1Orientation,blob1PositionX,blob1PositionY,blob1Width,blob1Height,...,blob2Size,blob2Orientation, etc.}
The first value of blob N is located at index 1+(N-1)*valueCountPerBlob.

Blob1PositionX and blob1PositionY represents the X/Y positions in pixels (if the X resolution is 256, then blob1PositionX might vary between 0 and 255).

You can check that if you double-click on the filter.

Cheers

series9
Posts: 8
Joined: 04 Mar 2015, 07:01

Re: Python API Vision Sensor

Post by series9 » 04 Mar 2015, 23:12

hi many thanks. Does the orthographic vision sensor able to retrieve depth parameter as well? Also, i attempted to use 2D laster scanner but it doesnt seem to output a graph. How do i interface with those range sensor using python? for e.g Hokuyo and 2D laser scanner. They both have child script and they are a built on of primitive sensors, what api function in python should i call to extract data out of them?

coppelia
Site Admin
Posts: 6740
Joined: 14 Dec 2012, 00:25

Re: Python API Vision Sensor

Post by coppelia » 05 Mar 2015, 08:39

Laser scanners can be modeled in V-REP using proximity sensors or vision sensors. But in either case, they are not a basic proximity/vision sensor: they are a higher-level sensor, that will produce a different type of output/data.

Check the child script attached to those sensors, and you will see that at some point a table of points is generated. If you want to access those points from a remote API client, you need to pack that data into a string signal (inside of the child script), for instance:

Code: Select all

local data=simPackFloats(myCalculatedPoints)
simSetStringSignal('myPoints',data)
Then on your client side, use simxGetStringSignal.

Cheers

zhenghao102
Posts: 5
Joined: 20 Jan 2015, 07:48

Re: Python API Vision Sensor

Post by zhenghao102 » 12 Mar 2015, 15:58

Hi!
I'm in trouble as following:

Code: Select all

errorCode1,cam1_handle=vrep.simxGetObjectHandle(clientID,'cam1',vrep.simx_opmode_oneshot_wait)
errorCode2, resolution, image1= vrep.simxGetVisionSensorImage(clientID,cam1_handle,0,vrep.simx_opmode_streaming)
I got the result as: errorCode1 = 0, errorCode2 = 1. I can't get the vision Sensor data. Who can help me?
Thanks a lot!!

coppelia
Site Admin
Posts: 6740
Joined: 14 Dec 2012, 00:25

Re: Python API Vision Sensor

Post by coppelia » 13 Mar 2015, 00:04

Hello,

You are not doing the streaming correctly. Make sure you have understood how streaming is done, this page might help.

Have also a look at following example, that shows how to do a blocking call, stream data, and a non-blocking call:

Code: Select all

try:
    import vrep
except:
    print '--------------------------------------------------------------'
    print '"vrep.py" could not be imported. This means very probably that'
    print 'either "vrep.py" or the remoteApi library could not be found.'
    print 'Make sure both are in the same folder as this file,'
    print 'or appropriately adjust the file "vrep.py"'
    print '--------------------------------------------------------------'
    print ''

import time	
	
print ('Program started')
vrep.simxFinish(-1) # just in case, close all opened connections
clientID=vrep.simxStart('127.0.0.1',19999,True,True,5000,5) # Connect to V-REP
if clientID!=-1:
    print ('Connected to remote API server')

    # Now try to retrieve data in a blocking fashion (i.e. a service call):
    res,objs=vrep.simxGetObjects(clientID,vrep.sim_handle_all,vrep.simx_opmode_oneshot_wait)
    if res==vrep.simx_return_ok:
        print ('Number of objects in the scene: ',len(objs))
    else:
        print ('Remote API function call returned with error code: ',res)

    time.sleep(2)
	
    # Now retrieve streaming data (i.e. in a non-blocking fashion):
    startTime=time.time()	
    vrep.simxGetIntegerParameter(clientID,vrep.sim_intparam_mouse_x,vrep.simx_opmode_streaming) # Initialize streaming
    while time.time()-startTime < 5:	
        returnCode,data=vrep.simxGetIntegerParameter(clientID,vrep.sim_intparam_mouse_x,vrep.simx_opmode_streaming) # Try to retrieve the streamed data
        if returnCode==vrep.simx_return_ok: # After initialization of streaming, it will take a few ms before the first value arrives, so check the return code
            print ('Mouse position x: ',data) # Mouse position x is actualized when the cursor is over V-REP's window

    # Now send some data to V-REP in a non-blocking fashion:
    vrep.simxAddStatusbarMessage(clientID,'Hello V-REP!',vrep.simx_opmode_oneshot)

    # Before closing the connection to V-REP, make sure that the last command sent out had time to arrive. You can guarantee this with (for example):
    vrep.simxGetPingTime(clientID)

    # Now close the connection to V-REP:	
    vrep.simxFinish(clientID)
else:
    print ('Failed connecting to remote API server')
print ('Program ended')
Cheers

farah amirah
Posts: 11
Joined: 05 Oct 2016, 07:31

Re: Python API Vision Sensor

Post by farah amirah » 07 Dec 2017, 06:16

hi! I want to ask about getting a moving target coordinate values captured by vision sensor in vrep to python. The target is red sphere that move in circular path by dummy.

Code: Select all

import vrep
import sys
import time
import numpy as np
import matplotlib.pyplot as mlp

t = 0


vrep.simxFinish(-1)
clientID=vrep.simxStart('127.0.0.1',19999,True,True,5000,5)

if clientID!=-1:
    print 'connected to remote API server'
else:
    print 'connection not succesful'
    sys.exit('error')
    


errorcode,sensor1=vrep.simxGetObjectHandle(clientID,'cam#0',vrep.simx_opmode_oneshot_wait)
returnCode,resolution,image=vrep.simxGetVisionSensorImage(clientID,sensor1,0,vrep.simx_opmode_streaming)
time.sleep(0.1)
#while t < 20:
#    t = t +2
returnCode,resolution,image=vrep.simxGetVisionSensorImage(clientID,sensor1,0,vrep.simx_opmode_buffer)
i = np.array(image, dtype=np.uint8)
i.resize([resolution[0],resolution[0],3])
mlp.imshow(i)


by using the code above, I can only obtain the target image display in python but i don't get the value of coordinate (x,y,z) of the target. My question is how to make the vision sensor send the coordinate of target to python?Is there any idea or code u can give me? Thanks.

coppelia
Site Admin
Posts: 6740
Joined: 14 Dec 2012, 00:25

Re: Python API Vision Sensor

Post by coppelia » 09 Dec 2017, 18:16

Hello,

have a look here.
Also, hopefully in less than one week we will make a beta release of next V-REP version, which will include a model doing what you describe (Models/components/sensors/Blob to 3D position.ttm). In that model, a vision sensor with a blob detection filter is used, and following code in a non-threaded child script (the code below uses the new API notation, but you can still get the idea):

Code: Select all

function sysCall_init()
    sensor=sim.getObjectAssociatedWithScript(sim.handle_self)
    sphereContainer=sim.addDrawingObject(sim.drawing_spherepoints,0.03,0,-1,9999,{1,0,1})
    local res
    res,nearClip=sim.getObjectFloatParameter(sensor,sim.visionfloatparam_near_clipping)
    res,farClip=sim.getObjectFloatParameter(sensor,sim.visionfloatparam_far_clipping)
    res,xAngle=sim.getObjectFloatParameter(sensor,sim.visionfloatparam_perspective_angle)
    res,resX=sim.getObjectInt32Parameter(sensor,sim.visionintparam_resolution_x)
    res,resY=sim.getObjectInt32Parameter(sensor,sim.visionintparam_resolution_y)
    yAngle=xAngle
    local ratio=resX/resY
    if resX>resY then
        yAngle=2*math.atan(math.tan(xAngle/2)/ratio)
    else
        xAngle=2*math.atan(math.tan(yAngle/2)/ratio)
    end

end

function sysCall_sensing()
    local m=sim.getObjectMatrix(sensor,-1)
    sim.addDrawingObjectItem(sphereContainer,nil)
    local res,packet1,packet2=sim.handleVisionSensor(sensor)
    if res>=0 then
        local blobCnt=packet2[1]
        local valCnt=packet2[2]
        for i=0,blobCnt-1,1 do
            local blobSize=packet2[2+valCnt*i+1]
            local blobOrientation=packet2[2+valCnt*i+2]
            local blobPositionX=packet2[2+valCnt*i+3]
            local blobPositionY=packet2[2+valCnt*i+4]
            local blobWidth=packet2[2+valCnt*i+5]
            local blobHeight=packet2[2+valCnt*i+6]
            local depthV=sim.getVisionSensorDepthBuffer(sensor,1+math.floor(blobPositionX*(resX-0.99)),1+math.floor(blobPositionY*(resY-0.99)),1,1)
            local depth=nearClip+(farClip-nearClip)*depthV[1]
            local coord={0,0,depth}
            local x=0.5-blobPositionX
            local y=blobPositionY-0.5
            coord[1]=depth*math.tan(xAngle*0.5)*(0.5-blobPositionX)/0.5
            coord[2]=depth*math.tan(yAngle*0.5)*(blobPositionY-0.5)/0.5
            coord=sim.multiplyVector(m,coord)
            -- coord now contains the position of the blob, in world coordinates
            sim.addDrawingObjectItem(sphereContainer,coord)
        end
    end
end

function sysCall_cleanup()
    sim.removeDrawingObject(sphereContainer)
end
Cheers

Post Reply