Making the quadricopter follow a path

Typically: "How do I... ", "How can I... " questions
anish
Posts: 22
Joined: 27 Jan 2020, 14:15

Re: Making the quadricopter follow a path

Post by anish »

Hai,

Thanks for your response. I was able to complete that task successfully. I was having exams so couldn't respond before!

Now I am planning to control the quad with a neural network and the vision sensor images as input. Based on the image the quad sees I would like it to follow a track which it is currently following using path function in Coppelia-sim.

The present problem is I would like to know what inputs can I give to the quad. Simply I want to command the quad to move forward on a straight path, to move forward & change its yaw when there is a curve.

1. to move forward
Is there a way to give pitch input to the quad so that it moves forward rather than the position input?

2. change its yaw
It seems like in this model I cannot read the orientation of the quad w.r.t its own axis although I can set it (don't understand why I can only read its orientation w.r.t the world & parent frame but not w.r.t to its own axis which is the yaw)?

anish
Posts: 22
Joined: 27 Jan 2020, 14:15

Re: Making the quadricopter follow a path

Post by anish »

Hai,

Now I have to implement the path follow using a neural network.

When I add codes for data collection (saving image seen through the vision sensor added to the quad) to the loop of path follow the whole execution gets delayed.

Without the image processing code, the quad runs smoothly and there is no delay between each loop i.e, quad moves from one point to the other smoothly. But when saving image codes come in the quad moves to one point then there is a small delay then moves to the next. Is there any way to make this motion smooth?

anish
Posts: 22
Joined: 27 Jan 2020, 14:15

Re: Making the quadricopter follow a path

Post by anish »

Hai,

I figured out that the below line of code is the reason for the non-smooth motion of the quad

Code: Select all

err_code, vision_handle = sim.simxGetObjectHandle(clientID, 'Vision_sensor', sim.simx_opmode_blocking)
err_code, resolution, image = sim.simxGetVisionSensorImage(clientID, vision_handle, 0, sim.simx_opmode_streaming)  
Can you please advise how to resolve this discontinuous motion of the quad and what could be the reason for this?

Note: getting vision_handle doesn't affect the motion of quad, but it is the line with 'sim.simx_opmode_streaming' causing the problem.
Complete code below;

Code: Select all

#getting handle for the floorcamera of the copter
err_code, vision_handle = sim.simxGetObjectHandle(clientID, 'Vision_sensor', sim.simx_opmode_blocking)
err_code, resolution, image = sim.simxGetVisionSensorImage(clientID, vision_handle, 0, sim.simx_opmode_streaming)  
 
while True:
    err_code1, resolution, image = sim.simxGetVisionSensorImage(clientID, vision_handle, 0, sim.simx_opmode_buffer)    #getting images
    if err_code1 == sim.simx_return_ok:
        break
    
 while True:
    err_code,target_handle = sim.simxGetObjectHandle(clientID,'Quadricopter_target',sim.simx_opmode_blocking)
    err_code,dummy_handle = sim.simxGetObjectHandle(clientID,'Dummy',sim.simx_opmode_blocking)
  
    err_code, dummy_orien = sim.simxGetObjectOrientation(clientID, dummy_handle, -1, sim.simx_opmode_blocking)    
    err_code, dummy_pos = sim.simxGetObjectPosition(clientID, dummy_handle, -1, sim.simx_opmode_blocking)
    dummy_pos[2] =  +5.0000e-01 
     
    err_code = sim.simxSetObjectPosition(clientID, target_handle, -1, dummy_pos, sim.simx_opmode_oneshot)
    err_code = sim.simxSetObjectOrientation(clientID, target_handle, -1, dummy_orien, sim.simx_opmode_oneshot)

    err_code1, resolution, image = sim.simxGetVisionSensorImage(clientID, vision_handle, 0, sim.simx_opmode_oneshot)    
    image = np.array(image, dtype = np.uint8)                                 #signedint -> unsigned int now each value range 0-255
    image.resize([resolution[0],resolution[1],3])                               #resize to 512*512*3
    image = np.flip(image,0)
    data_collector.save_image('R',image)
    

coppelia
Site Admin
Posts: 8125
Joined: 14 Dec 2012, 00:25

Re: Making the quadricopter follow a path

Post by coppelia »

It seems you are not correctly using the legacy remote API. You should retrieve handles once, just after connecting to CoppeliaSim, and you should also enable data streaming once, at the same time. When disconnecting, you should disable data streaming with the simx_opmode_discontinue operation mode.

While simulation is running, for all data where you enabled streaming, use the simx_opmode_buffer operation mode. Have a look how streaming works here.

Cheers

anish
Posts: 22
Joined: 27 Jan 2020, 14:15

Re: Making the quadricopter follow a path

Post by anish »

Hai,

Thanks again for your response!

I have made changes like you mentioned, retrieving handles and streaming data only once, using buffer mode for streamed data during data collection and disabling the streaming at the end. Here is the updated code(The entire code is given below including initializing connection with the server)

Code: Select all

import sim
import sys
import numpy as np

sim.simxFinish(-1)

clientID=sim.simxStart('127.0.0.1',19999,True,True,5000,5)

if clientID!=-1:
    print ('Connected to remote API server')
else:
    print("Not connected to remote API server")
    sys.exit("Could not connect")

#getting handle of dummy, quad and vision sensor
err_code,target_handle = sim.simxGetObjectHandle(clientID,'Quadricopter_target',sim.simx_opmode_blocking)
err_code,dummy_handle = sim.simxGetObjectHandle(clientID,'Dummy',sim.simx_opmode_blocking)
err_code, vision_handle = sim.simxGetObjectHandle(clientID, 'Vision_sensor', sim.simx_opmode_blocking)


#initialise streaming of image data
err_code, resolution, image = sim.simxGetVisionSensorImage(clientID, vision_handle, 0, sim.simx_opmode_streaming)  
#verify images are being recieved
while True:
    err_code1, resolution, image = sim.simxGetVisionSensorImage(clientID, vision_handle, 0, sim.simx_opmode_buffer)    
    if err_code1 == sim.simx_return_ok:
        break
    
im =  np.empty(shape=[0, 786432])
       
while True:
    #get target pos & orien
    err_code, dummy_orien = sim.simxGetObjectOrientation(clientID, dummy_handle, -1, sim.simx_opmode_blocking)    
    err_code, dummy_pos = sim.simxGetObjectPosition(clientID, dummy_handle, -1, sim.simx_opmode_blocking)
    dummy_pos[2] =  +5.0000e-01 
    
    #set pos & orien
    err_code = sim.simxSetObjectPosition(clientID, target_handle, -1, dummy_pos, sim.simx_opmode_oneshot)
    err_code = sim.simxSetObjectOrientation(clientID, target_handle, -1, dummy_orien, sim.simx_opmode_oneshot)
   
   #getting images    
    err_code1, resolution, image = sim.simxGetVisionSensorImage(clientID, vision_handle, 0, sim.simx_opmode_buffer)    
    image = np.asarray(image)
    im = np.vstack((im,image))
    
    #end the process when final poit is reached
    if round(dummy_pos[0], 3) == 0.126 and  round(dummy_pos[1], 3) == -0.855:
        #stop streaming images
        err_code, resolution, image = sim.simxGetVisionSensorImage(clientID, vision_handle, 0, sim.simx_opmode_discontinue)
        break
unfortunately, the motion of the quad is still discontinuous. I will also show you a picture of the set up in CoppeliaSim.
https://drive.google.com/drive/folders ... p=sharing

I tried a lot of things but the motion which is continuous and smooth without the data collection code becomes discontinuous just after adding vision sensor image data collection codes. Can't figure out the reason?

coppelia
Site Admin
Posts: 8125
Joined: 14 Dec 2012, 00:25

Re: Making the quadricopter follow a path

Post by coppelia »

Ok, when streaming vision sensor data, it is normal that things slow a bit down, since there is more data to transmit. What happens if you reduce the resolution of the vision sensor? What happens if you just stream the data of the vision sensor without collecting it (i.e. enabling vision sensor data streaming only, without checking if data has arrived)?

Cheers

anish
Posts: 22
Joined: 27 Jan 2020, 14:15

Re: Making the quadricopter follow a path

Post by anish »

Hai,

I got some positive results!

Streaming data without collecting it did not work. Reducing the resolution worked, but not using python remote API functions.

I used Legacy Remote API to extend the LUA function written in a child script at the server-side. There I collected the images using the below regular API functions and returned the images to the client where I have python.

Code: Select all

    image = sim.getVisionSensorImage(Vision_handle, 0, 0, 0, 0, 1)
    resolution = sim.getVisionSensorResolution(Vision_handle)
after receiving the images I converted it into an array and carried out the image processing. And a relatively smooth motion was achieved. A little low on the resolution side though! it was possible only with 128x128 resolution.

If you have any other ideas to work with, around 256x256 resolution please let me know. Otherwise, if this is the maximum achievable result guess I can proceed with my work using this solution.

Thank you so much for your help!

coppelia
Site Admin
Posts: 8125
Joined: 14 Dec 2012, 00:25

Re: Making the quadricopter follow a path

Post by coppelia »

Streaming data without collecting it did not work
What does that mean?

You should try to enable data streaming for the vision sensor (one line of code), that's it. Then check if there is a severe slow-down or similar. If you do not collect the data, then of course you won't get the image, but the image will still be streamed and arrive on your client side (simply that you won't collect it).
Why can't you go at a higher resolution? Does it get too slow? That should not be. Are you in synchronous mode?


I highly recommend you to try also the B0-based remote API. Have a look at the demo scene scenes/synchronousImageTransmissionViaRemoteApi.ttt and the demo client application b0RemoteApiBindings/python/synchronousImageTransmission.py. If you do not need to use the remote API in synchronous mode, then things will run even faster.

Cheers

anish
Posts: 22
Joined: 27 Jan 2020, 14:15

Re: Making the quadricopter follow a path

Post by anish »

Hello,

By "Streaming data without collecting it did not work" - I meant that it did not give me the smooth operation of the quad, the slow-down was notable. But I was getting images without the line of codes for making sure the image was being received.

Using Legacy Remote API, resolutions above 128x128 was making the process slow.

Also, I am not running in synchronous mode.

I haven't tried B0-based remote API yet. I will try it today and let you know the results soon.

Thank you!

wode
Posts: 18
Joined: 12 Oct 2019, 03:47

Re: Making the quadricopter follow a path

Post by wode »

Hi,
I tried to make the dummy move along the path and followed your steps.
But my dummy is not moving and there is no child script in my dummy or path.
What is the problem?

Cheers
anish wrote:
03 Feb 2020, 20:18
Hai,

I need to do a similar task. But my dummy is not moving at all when I run the simulation. Here is what all I did;-
added a circular path.
added a dummy and made it child of the path.
in dummy properties, the trajectory is set to 'Follow Parent path'
Placed the dummy at the start(seen as red on the path) of the path.
Then I ran the simulation.
Is there any thing more to be done?

Also I am not able to find path planning in the calculation modules?

Post Reply