Visual Servoing Coppelia SIm

[linoleico]

Hi there,

I'm new to this simulator and I've already seen a lot of tutorials about it, but the problem is that there are so many informations that I'm kinda lost here. I need to build a scene with the kuka iiwa 14 robot arm in which the robot will follow a green target (could be any color) using a visual servoing application. I already attached the camera at the tip of the robot, made a path for the target to follow and coded a simple blob detection that gives the x and y position everytime the green target crosses the sensor. But now, I'm stuck at how to make the robot move. I researched about how the visual servoing uses Inverse jacobians, but I really don't know what to do. Also, I'm not very familiar with lua scripts. Is there any way to do this scene in another language, let's say, python, for example?

Cheers.

[fferri]

You probably need another target (which you will control in x,y,z position) that the robot will follow, after you have configured robot's IK functionality. See models/7 DoF manipulator or models/ABB IRB 140 or the Inverse kinematics tutorial.

Top 2 results for visual servoing:

• https://en.wikipedia.org/wiki/Visual_servoing
• http://www.diag.uniroma1.it/deluca/rob2 ... rvoing.pdf

Yes, it is possible to use Python (or C++, or Java, or MATLAB) via the Remote API.

[linoleico]

I'm already digging into the python API to boost the image processing power. I'm 100% not a expert in this topic, but I would need to set a target to do inverse kinematics to control de robot? I mean, with just the camera information I would be able to do that, setting the joint velocities and etc, right? Not quite sure how to approach this problem. By the way, I'm very glad you repplied my topic, thanks, man.

[fferri]

Sure, you can also work directly with joint velocities. In that case you don't need a target or the IK calculation module at all.

[linoleico]

Hey, man, let me start thanking you again.

So, I read about joint motion in the manual and the ''motion mode'' was depreciated. I tried setting the joints to the ''passive mode'' to see how it would work. After detecting the green object and reading it's x and y position, I'm trying to set the joints velocities so that the object would stay in the middle of the vision sensor or as close to it as possible. I tried setting the velocity only of the 6th joint, to see how it would work, but the joint does not move at all.

https://imgur.com/a/n6WzcA5

The image above represents where I'm currently at and the below the code I wrote on the vision sensor at the tip of the robot.

Sorry if it's a lot to ask.

Code: Select all

function sysCall_threadmain()
    out = sim.auxiliaryConsoleOpen("debug",8,1)
    cam = sim.getObjectHandle("camrobo")
    joint6 = sim.getObjectHandle("joint6")

    while sim.getSimulationState()~=sim.simulation_advancing_abouttostop do
        simVision.sensorImgToWorkImg(cam)
        simVision.selectiveColorOnWorkImg(cam,{0,1,0},{0.2,0.2,0.2},true,false,true)
        simVision.swapWorkImgWithBuffer1(cam)
        trigger, pack = simVision.blobDetectionOnWorkImg(cam,0.1,0,true,{1,0,1})
        resBlob = sim.unpackFloatTable(pack)
        simVision.workImgToSensorImg(cam)
        
        if resBlob[1]>0 then
            xtarget = resBlob[5]
            ytarget = resBlob[6]
            sim.auxiliaryConsolePrint(out,string.format("\n %0.2f %0.2f",xtarget, ytarget))
            sim.setJointTargetVelocity(joint6, xtarget)

        end


        sim.switchThread()
    end
end
function sysCall_cleanup()
    -- Put some clean-up code here
end

-- See the user manual or the available code snippets for additional callback functions and details

[coppelia]

Your robot can operate in different modes, depending on how you configured it.
If your joints are operating in inverse kinematics mode, then you should move and rotate the robot target tip (as in a tip-target relationship for an IK element), in order to have the end-effector follow.
If your joints are in Force/torque mode, make sure the motors are enabled, and the control loop disabled. Then you can set the target velocities for your joints with sim.setJointTargetVelocity. Make sure that the maximum force/torque for the individual joints is large enough.

Cheers