## Streaming video camera

Typically: "How do I... ", "How can I... " questions
Vegus
Posts: 12
Joined: 22 Mar 2014, 21:41

### Streaming video camera

Hello,

I have a question regarding to stream of video camera. What are the possbile ways to streaming. I know that it is possible by ROS but i wanna avoid this way. My purpose is streaming video camera from V-REP to my application written in the language c#.

Cheers.

coppelia
Posts: 7606
Joined: 14 Dec 2012, 00:25

### Re: Streaming video camera

Hello,

then use the remote API. Make sure you understand how the remote API works.

In following C/C++ example, an image is read from a V-REP vision sensor (streamed from V-REP), then applied to a different (passive) vision sensor in V-REP:

Code: Select all

// Make sure to have the server side running in V-REP:
// in a child script of a V-REP scene, add following command
// to be executed just once, at simulation start:
//
// simExtRemoteApiStart(19999)
//
// then start simulation, and run this program.
//
// IMPORTANT: for each successful call to simxStart, there
// should be a corresponding call to simxFinish at the end!

#include <stdio.h>
#include <stdlib.h>

extern "C" {
#include "extApi.h"
}

int main(int argc,char* argv[])
{
int clientID=simxStart((simxChar*)"127.0.0.1",19999,true,true,2000,5);
if (clientID!=-1)
{
printf("Connected to remote API server\n");

// Now retrieve some handles:
int passive;
int active;
simxGetObjectHandle(clientID,"PassiveVision_sensor",&passive,simx_opmode_oneshot_wait);
simxGetObjectHandle(clientID,"Vision_sensor",&active,simx_opmode_oneshot_wait);
int resolution[2];
simxUChar* image;
simxGetVisionSensorImage(clientID,active,resolution,&image,0,simx_opmode_streaming);

// Now retrieve streaming data (i.e. in a non-blocking fashion):
while (simxGetConnectionId(clientID)!=-1)
{
if (simxGetVisionSensorImage(clientID,active,resolution,&image,0,simx_opmode_buffer)==0)
{
simxSetVisionSensorImage(clientID,passive,image,resolution[0]*resolution[1]*3,0,simx_opmode_oneshot);
}
extApi_sleepMs(5);
}

// Now close the connection to V-REP:
simxFinish(clientID);
}
return(0);
}

Use the demo scene Scenes/rosTopicPublisherAndSubscriber.ttt, and replace the child script attached to object Vision_sensor with following code:

Code: Select all

if (sim_call_type==sim_childscriptcall_initialization) then
simExtRemoteApiStart(19999)
end

Cheers

lpitonakova
Posts: 17
Joined: 22 Nov 2017, 12:32

### Re: Streaming video camera

Hi,

I am using the method of streaming described in this thread (slightly altered) but I see some strange behaviour.

The first 15-20 seconds stream just fine, but then the streaming image freezes or is very slow, while the simulation instance is running fine. I am using localhost, so network speed is not an issue. If the main simulation instance is running in the GUI mode (i.e., with V-REP editor on) and I change focus from the streaming window to any other window on my computer, the freezing stops and streaming is smooth again. However, when the main simulation is running in the headless mode, i.e., has no GUI itself, switching focus between program windows doesn't help and the freezing is permanent.

In any case, it looks like that it is the streaming code that slows down rather than the simulation being streamed.
Has anyone experienced this or could give any hints as to what could be happening?

Thanks.

coppelia
Posts: 7606
Joined: 14 Dec 2012, 00:25

### Re: Streaming video camera

Hello,

so if I understand you correctly, the simulation runs fine, but the client eventually hangs or freezes?
Can you provide a very simple sample code that illustrates this?

Cheers

lpitonakova
Posts: 17
Joined: 22 Nov 2017, 12:32

### Re: Streaming video camera

Hi,

the code is essentially the same as is provided in the first reply in this thread. It's spread across classes so it's difficult to show but this is roughly how it goes:

[MainRendererClass]

Code: Select all

logger->Log("Getting image data...");
simUChar* imageData = CameraControl::Instance()->GetRemoteVisionSensorImage();
if (imageData != NULL) {
simSetVisionSensorCharImage(renderingVisionSensor,imageData);
}
extApi_sleepMs(5);

[CameraControl::GetRemoteVisionSensorImage]:

Code: Select all

   if (simxGetConnectionId(simulatorInterface->GetServerId()) != -1) {
if (simxGetVisionSensorImage(simulatorInterface->GetServerId(),remoteVisionSensor,remoteVisionSensorResolution,&remoteVisionSensorImage,0,simx_opmode_buffer) == 0) {
return remoteVisionSensorImage;
} else {
logger->Log("Could not get remote vision sensor image",RNELogger::MESSAGE_TYPE::ERROR);
}
} else {
logger->Log("Could not connect to server while retrieving remote vision sensor image",RNELogger::MESSAGE_TYPE::ERROR);
}
return NULL;

You are correct - the simulation itself runs but it is the client (the renderer) that freezes. I can switch to the simulation window and I see it working and then switch back to the renderer and it becomes unstuck. But, the renderer gets stuck each 15-20 seconds. The time in which is becomes stuck is much smaller when I run the simulation in headless mode.

Also, if you notice, I output every program loop in MainRendererClass. This output continues while the image is stuck.

I should add that my renderer program (MainRendererClass) is a custom plug-in. The rendering loop is inside of the

Code: Select all

VREP_DLLEXPORT void* v_repMessage(int message, int* auxiliaryData, void* customData, int* replyData) {
function, when message "sim_message_eventcallback_instancepass" gets handled. I check that the rendering only occurs when the remote simulation is running. - Could this be the problem though, trying to process the remote image in this function?

coppelia
Posts: 7606
Joined: 14 Dec 2012, 00:25

### Re: Streaming video camera

Before going further, because I am not sure I understood you.

I understood following: you have created a V-REP plugin that communicate via the remote API with V-REP? If this is the case, why don't you directly transmit the image data to the plugin from a script, or by reading the vision sensor?

If this is not the case... what exactly is the purpose of the plugin?

Cheers

lpitonakova
Posts: 17
Joined: 22 Nov 2017, 12:32

### Re: Streaming video camera

Yes, sorry to be confusing. You are right - there is the Server, which is one V-REP simulation (also a plugin runs there) and there is Renderer, which is another V-REP instance (with the plugin I have problems with). The point of having it split up is that the Renderer plugin demonstrates the RemoteAPI functionality (I have written a bunch of functions that use the RemoteAPI but could theoretically be used in any C++ code). The only reason that the Renderer is also a V-REP plugin is the ease of rendering on screen, which seemed much faster than using OpenGL to process the streamed image data. Also, my Renderer can now be used on a different machine than the Server runs on which is also useful.

lpitonakova
Posts: 17
Joined: 22 Nov 2017, 12:32

### Re: Streaming video camera

Just wanted to add here a function that I initially use to start streaming from a given vision sensor. The CameraControl class keeps a list of visual sensors (obtained at the beginning of the program from the RemoteAPI) - this list basically has the sensor names and resolution. Then there is a drop-down box and the following function is called when user selects a sensor to stream from using the drop down:

Code: Select all

void CameraControl::AttachViewToSensor(int sensorIndex) {

std::vector<RNESimulatorInterface::VisualSensorInfo>* visualSensors = simulatorInterface->GetVisualSensors();
if (int(visualSensors->size()) <= sensorIndex) {
logger->Log("Cannot attach to sensor because sensor index is out of bounds. Sensor index:", sensorIndex, " Number of visual sensors in the remote scene:", int(visualSensors->size()));
return;
}

//-- index is ok, proceed to retrieving the selected vision sensor info
RNESimulatorInterface::VisualSensorInfo selectedSensorInfo = visualSensors->at(sensorIndex);
remoteVisionSensor = selectedSensorInfo.handle;

if (remoteVisionSensor >=0) {
//-- initial mandatory call to get the image from the remote sensor
simxGetVisionSensorImage(simulatorInterface->GetServerId(),remoteVisionSensor,remoteVisionSensorResolution,&remoteVisionSensorImage,0,simx_opmode_streaming);

//-- setup the local rendering vision sensor attributes to match those of the remote sensor
simSetObjectInt32Parameter(renderingVisionSensor,sim_visionintparam_rendering_attributes,sim_displayattribute_renderpass+sim_displayattribute_forvisionsensor+sim_displayattribute_ignorerenderableflag);
simSetObjectInt32Parameter(renderingVisionSensor,sim_visionintparam_resolution_x,selectedSensorInfo.resolutionX);
simSetObjectInt32Parameter(renderingVisionSensor,sim_visionintparam_resolution_y,selectedSensorInfo.resolutionY);

isViewAttachedToASensor = true;
} else {
logger->Log("Failed to attach to sensor at index ", sensorIndex, RNELogger::MESSAGE_TYPE::ERROR);
isViewAttachedToASensor = false;
}
}
So in the Renderer V-REP plugin, that holds an instance of Camera Control:

Code: Select all

VREP_DLLEXPORT void* v_repMessage(int message, int* auxiliaryData, void* customData, int* replyData) {
...

} else if (message == sim_message_eventcallback_instancepass) {

//--
//-- Inside the V-REP program loop
//--

if (isRemoteSimulationRunning) {
if (CameraControl::Instance()->GetIsViewAttachedToASensor()) {
logger->Log("Getting image data...");
simUChar* imageData = CameraControl::Instance()->GetRemoteVisionSensorImage();
if (imageData != NULL) {
simSetVisionSensorCharImage(renderingVisionSensor,imageData);
} else {
logger->Log("Image data empty", RNELogger::MESSAGE_TYPE::ERROR);
}
extApi_sleepMs(5);
} else {
//-- link the local rendering and remote sensors, picking the 0th sensor available
CameraControl::Instance()->AttachViewToSensor(0);
}
}
} 
I gave the code of CameraControl::Instance()->GetRemoteVisionSensorImage(); in my previous post.

coppelia
Posts: 7606
Joined: 14 Dec 2012, 00:25

### Re: Streaming video camera

It is difficult to say what is going wrong. Does the problem persist when images are smaller? When simulation runs in headless mode, it is usually faster since it doesn't wait for rendering. And if it gets stuck faster in that case, it could mean that the problem is linked to the amount of data streamed. Not sure.
For streaming images between two V-REP instances, you will be able to use the BlueZero framework, which is a much better fit than the remote API. The BlueZero framework will be available in next V-REP release (V3.5.0). You can test the beta version here. Have a look at the demo scene blueZeroDemo1.ttt, which is using BlueZero.

Cheers

lpitonakova
Posts: 17
Joined: 22 Nov 2017, 12:32

### Re: Streaming video camera

Thanks, but I really need to use the RemoteAPI, because eventually, I will want to write a renderer in Unity or similar. Or would BlueZero handle that too? I.e. using the vision sensor data in an arbitrary program?

My problem does persist when images are smaller and yes, it gets stuck quicker when the simulation runs in headless mode. I also thought that the problem could be with the amount of data streamed, as the image always gets stuck at the exact same moment (for the same scene). Is there any documentation that I could read regarding streaming visual sensor data? I couldn't find anything that I could use, apart from this post. A few ideas:
• Is there a way of limiting how often the sensors stream, or similar?
• Should I be releasing the contents of imageData at some point? If yes, what is the safe way of doing this?