Text-based save file format

Requests or suggestions for new features
arennuit
Posts: 58
Joined: 03 Oct 2014, 23:04

Text-based save file format

Post by arennuit »

Hi guys,

I believe you could introduce a text-based file format. This would allow for two things:

1. In terms of workflow, this would ease the integration of v-rep into large applications
___* Because v-rep files could be generated out of another description of the scene (kind of what can be done in programming with complex build chains)
___* For example this would allow to only maintain a single file when you have another description of your robot (like a URDF or anything else...)
2. This would ease version control as you could easily check what has changed between 2 revisions of the same file

Cheers,

Antoine.

arennuit
Posts: 58
Joined: 03 Oct 2014, 23:04

Re: Text-based save file format

Post by arennuit »

Here are two use cases related to the workflow:

1. I need to design a new mechanism which will fit onto my mobile base. The mobile base is already designed and working well. The new mechanism will be designed by trial and error. So it is expected that I create it in solidworks, import and integrate it in vrep, realize that my design is wrong, go back to solidworks, delete it and re-import/integrate it in vrep... and loop like this a few times. Each time I modify the design of my new mechanism I need to destroy all the integration work and start the integration again from scratch, this is a loss of time.

With a text-based description file I could have a file describing my mobile platform, a file describing my new mechanism and a file integrating both. When modifying the design of the new mechanism, only the file describing the new mechanism would change. The integration work would be kept unchanged. This would tremendously clean vrep's workflow.

2. I have just integrated ros_control within vrep (obviously I will provide an example package on this forum once it is reviewed in a few hours/days). ros_control requires a urdf description of the system in order to work properly (it contains information on the hardware interfaces found nowhere else + a description of the joints). This means that there is a redundant description of the system (both in the urdf and in the ttt files). When you modify on file you must also modify the other one. This is painful and error prone.

With a text-based description file (and possibly a custom pre-load step but this is not mandatory), we could keep a single description of the scene in a format of our own and generate a urdf and a ttt-text at will! So when a scene is modified we would only have to modify it in one place! Cool, no?

Regards,

Antoine.

coppelia
Site Admin
Posts: 10339
Joined: 14 Dec 2012, 00:25

Re: Text-based save file format

Post by coppelia »

Hello Antoine,

I perfectly agree with you.
THere are however some difficulties: a ttt file contains a lot of information of all sorts (child script code, colors, transformation matrices, all the parameters of the different physics engines, etc.). That file is also forward/backward compatible through a special mechanism, and many other things (compression, redudancy handling, consistency check, etc.).

The idea is rather to write a small add-on/customization script that allows to merge things from an external file into an existing scene or model. There are many API functions that allow to do this. In order to replace a single mesh in an existing model/scene, you would follow this procedure:
Cheers

arennuit
Posts: 58
Joined: 03 Oct 2014, 23:04

Re: Text-based save file format

Post by arennuit »

Hello Coppelia,

That would be a bit of dev, but it could be a workaround. I guess this will take a bit of annoyment with the dirty way of doing things before I decide it is worth going for the workaround though;)

Thanks for you help,

Antoine.

ahundt
Posts: 112
Joined: 29 Jan 2015, 04:21

Re: Text-based save file format

Post by ahundt »

I second the request for a text based save file format. It could be a folder where the main file refers to separate child script files. Versioning should be just as achievable with a text based file format.

Compression is actually making things worse for me, because my version control system has to track a whole new file every time I make a change.

Boost.serialization implements an XML format with versioning built in, including ways of only writing an object pointed to from multiple pointers once. However their format isn't really for moving between languages or anything like that, I would just consider it as a reference for certain useful techniques.

http://www.boost.org/doc/libs/release/l ... index.html

ahundt
Posts: 112
Joined: 29 Jan 2015, 04:21

Re: Text-based save file format

Post by ahundt »

I tried to sell a group in my lab on using V-REP, but one of the major problems in the way of them adopting it was the save file format. They have a number of users that ask them to help support the machines they use. This frequently involves the users sending our group code and config files, which our group then diffs against the basic upstream configurations to diagnose the changes that caused the problem.

While they were very impressed with V-REP overall, as am I, the current V-REP format makes any such comparison or merging of changes impossible, so they don't think they can use it.

Currently I believe they are leaning towards choosing gazebo, because all the configurations are done in plaintext XML files.

coppelia
Site Admin
Posts: 10339
Joined: 14 Dec 2012, 00:25

Re: Text-based save file format

Post by coppelia »

Having a text fileformat would not be too difficult to implement: one way would be to replace the current binary serialization routines with a text-based serialization routine.

It becomes much more difficult if the user manipulates those text files, since consistency needs to be checked carefully. There are several thousand different parameters and attributes that can be set, and each object can reference many other objects. So there would be major parts that would need to be rewritten.

But in the end, it is indeed an important feature, and we pushed it to a higher priority in the todo-list.

Cheers

ahundt
Posts: 112
Joined: 29 Jan 2015, 04:21

Re: Text-based save file format

Post by ahundt »

Thanks! That would be a fantastic improvement I'd make a lot of use of from day to day, which could also potentially benefit adoption as well.

coppelia
Site Admin
Posts: 10339
Joined: 14 Dec 2012, 00:25

Re: Text-based save file format

Post by coppelia »

We have started with the implementation, but it will not make it for next release (scheduled for in about 4 weeks of time). There are many additional difficulties compared to a binary file format, since data consistency need to be always checked, which is not the case when you know the data wasn't manually modified. Probably we will do this in a stepped fashion, leaving not that important data (e.g. textures or other details out) out in a first time.

Cheers

ahundt
Posts: 112
Joined: 29 Jan 2015, 04:21

Re: Text-based save file format

Post by ahundt »

Wonderful, I am very excited for this! Iterative or stepped improvements is always a reasonable way to go.

Simply having all the non-binary data (scene configuration, child scripts, simple variables, positions/rotations etc) that isn't purely binary already (textures, models) would be a wonderful step!

I'm curious, what serialization library is your choice for each of binary and text data?

Post Reply