Page 1 of 1

Externalizing scripts for better version control

Posted: 07 Sep 2017, 13:32
by Klaus
Greetings,

I have an issue in a workaround we have been using for version control of scripts.

We have been working on a way to mitigate the difficulty of using vrep simulations with version control tools.
The approach was to leave all lua scripts outside the simulation in .lua files (as suggested in another thread http://www.forum.coppeliarobotics.com/viewtopic.php?f=7&t=199).
This way, all modifications to scripts could be tracked by version control, and we would reduce the number of commits with modifications to the binary .ttt.

Just to give you a minimum working example, it works like this. Let's say your simulation is called "testScene.ttt":

1) Create a Cuboid, and rename it to "Cuboid_Threaded", just to make it easier to know what it is about latter.

2) Add a threaded script, with the following sample code:

Code: Select all

myVariable=0
while simGetSimulationState()~=sim_simulation_advancing_abouttostop do
   myVariable=myVariable+1
   print("myVariable: "..myVariable)
   simSwitchThread() -- resume in next simulation step
end

Nothing fancy, this code just initializes a variable called myVariable, an adds up 1 in a loop while printing it. If you play the scene you should see something like this:
    Initializing the Bullet physics engine in plugin 'DynamicsBullet_2_78'...
    Engine version: 2.78
    Plugin version: 9
    Initialization successful.
    myVariable: 1
    myVariable: 2
    myVariable: 3
    myVariable: 4
    myVariable: 5
    myVariable: 6
Now... if, for instance, we altered this code in anyway and had to commit this to a git repository, we would end up having to push the whole binary to it. Which is quite bad. We would like this script to actually be outside of the .ttt, in its own .lua file. So...

3) Create a lua file at the same folder as your simulation called "testScene_Cuboid_Threaded.lua"

4) Add the same aforementioned code to it and save it.

5) Now, go back to that threaded script in Cuboid_Threaded, erase everything, and replace it by the following code:

Code: Select all

    local sceneFullName=simGetStringParameter(sim_stringparam_scene_name)
    local sceneName = string.gsub(sceneFullName, ".ttt", "_")
    local objectHandle=simGetObjectAssociatedWithScript(sim_handle_self)
    local selfName=simGetObjectName(objectHandle)
    local nameSuffix=simGetNameSuffix(selfName)
    selfName = string.gsub(selfName, "#"..nameSuffix, "")
    local appPath=simGetStringParameter(sim_stringparam_scene_path)
    filenamepath=appPath.."/"..sceneName..selfName..".lua"
    local file = assert(loadfile(filenamepath))
    file()


This code builds the name of the lua file we want to access (i.e. "filenamepath") based on the scene's name and the name of the object the script is attached to.
Then it uses the loadfile and file commands to load the contents inside "testScene_Cuboid_Threaded.lua" to the present script.
If you play the simulation the result should be the same we saw in step 2.

However, if we try to follow the same steps for a Non-threaded script, things get trickier. Because a non-threaded script gets called again and again, it will load the file from disk again and again too. And if you have many objects doing this you would end up with a lot of unnecessary access to disk being made many times per second.

So my question is: is there a way to perform the same (or similar) trick explained above but with non-threaded scripts?

Thanks in advance for any feedback.

Re: Externalizing scripts for better version control

Posted: 10 Sep 2017, 13:31
by coppelia
Hello Klaus,

have a look at the demo scene scenes/blueRealityTeaser.brs: in there, most scripts are external, and included with following two instructions:

Code: Select all

require('utils')
include('/BlueWorkforce/modelScripts/genericPartTagger_child.lua')


Above works fine, but in next release, things will be simplified even further, and you will be able to simple write:

Code: Select all

sim.include('/BlueWorkforce/modelScripts/v1/genericPartTagger_child.lua')


Cheers

Re: Externalizing scripts for better version control

Posted: 11 Sep 2017, 14:56
by Klaus
Hello,

Thanks for the reply.
I tried this solution but it seems to be constrained to using Vrep's location as root folder, and I get the followinf error:

    Simulation started.
    Lua runtime error: C:\Program Files\V-REP3\V-REP_PRO_EDU\lua\utils.lua:95: cannot open C:/Program Files/V-REP3/V-REP_PRO_EDUC:/Users/klaus/Desktop/testScene_Cuboid_Non_Threaded.lua: Invalid argument
    stack traceback:
    [C]: in function 'assert'
    C:\Program Files\V-REP3\V-REP_PRO_EDU\lua\utils.lua:95: in function 'include'
    [string "SCRIPT Cuboid_Non_Threaded"]:17: in main chunk

Is there a way for it NOT to include the "C:/Program Files/V-REP3/V-REP_PRO_EDU"?

K

Re: Externalizing scripts for better version control

Posted: 12 Sep 2017, 14:56
by coppelia
That path is included by default. Otherwise, you should specify an other path to search via additionalLuaPath value in system/usrset.txt.

Otherwise you can always manipulate the Lua search path with following code:

Code: Select all

    // append some paths to the Lua path variable:
    luaWrap_lua_getglobal(L,"package");
    luaWrap_lua_getfield(L,-1,"path");
    std::string cur_path=luaWrap_lua_tostring(L,-1);
    cur_path+=";";
    cur_path+=desiredFolder;
    cur_path+="/lua/?.lua";


(c++ code, but in Lua it is very similar)

Cheers

Re: Externalizing scripts for better version control

Posted: 13 Sep 2017, 13:14
by Klaus
Hello, thanks for the help.

I'm not sure I understood the suggestions (i) add additional lua path and (ii) manipulate lua search path.

Here is what I did and the results:

(i) I've added the following to system/usrset.txt

    additionalLuaPath = c:/Users/klaus/Desktop

But the error remains the same:

    Simulation started.
    Lua runtime error: C:\Program Files\V-REP3\V-REP_PRO_EDU\lua\utils.lua:95: cannot open C:/Program Files/V-REP3/V-REP_PRO_EDUtestScene_Cuboid_Non_Threaded.lua: No such file or directory
    stack traceback:
    [C]: in function 'assert'
    C:\Program Files\V-REP3\V-REP_PRO_EDU\lua\utils.lua:95: in function 'include'
    [string "SCRIPT Cuboid_Non_Threaded"]:21: in main chunk
    Simulation paused.

It does not seem that include is taking the new information into consideration.

(ii) Is this what you meant?

Code: Select all

require('utils')
filepathname='C:/Users/klaus/Desktop/testScene_Cuboid_Non_Threaded.lua'
include(';'..filepathname)


Produces the following error:

    Simulation started.
    Lua runtime error: C:\Program Files\V-REP3\V-REP_PRO_EDU\lua\utils.lua:95: cannot open C:/Program Files/V-REP3/V-REP_PRO_EDU;C:/Users/klaus/Desktop/testScene_Cuboid_Non_Threaded.lua: Invalid argument
    stack traceback:
    [C]: in function 'assert'
    C:\Program Files\V-REP3\V-REP_PRO_EDU\lua\utils.lua:95: in function 'include'
    [string "SCRIPT Cuboid_Non_Threaded"]:21: in main chunk
    Simulation paused.

Regards

Re: Externalizing scripts for better version control

Posted: 13 Sep 2017, 13:53
by coppelia
Sorry, I realized I wasn't very helpful with the c code, which actually translates into something like this in Lua:

Code: Select all

package.path='d:/v_rep/?.lua;' .. package.path


Cheers

Re: Externalizing scripts for better version control

Posted: 13 Sep 2017, 20:47
by Klaus
It seems to me that the current folder is already on package.path (the script is in the same folder as the .ttt).
I have prepared a minimum working example of it. See if you can open it:

https://www.dropbox.com/s/dg10er115ctyb8i/testScene.zip?dl=0

Btw, I made it so it modifies the path only once:

Code: Select all

firstRun=tonumber(sim.getScriptSimulationParameter(sim.handle_self,'firstRun'))
--print("firstRun: "..firstRun)
if(firstRun==1)then
    setCount=sim.setScriptSimulationParameter(sim.handle_self,'firstRun',0)
    local sceneFullName=sim.getStringParameter(sim.stringparam_scene_name)
    sceneName = string.gsub(sceneFullName, ".ttt", "_")
    local objectHandle=sim.getObjectAssociatedWithScript(sim.handle_self)
    selfName=sim.getObjectName(objectHandle)
    local nameSuffix=sim.getNameSuffix(selfName)
    selfName = string.gsub(selfName, "#"..nameSuffix, "")
    local appPath=sim.getStringParameter(sim.stringparam_scene_path)
    filename=sceneName..selfName..".lua"
    filenamepath=appPath.."/"..filename
    package.path=filenamepath .. package.path
    print('=======')
    print(package.path)
end
require('utils')
include(filename)

Re: Externalizing scripts for better version control

Posted: 18 Sep 2017, 08:26
by coppelia
Sorry, maybe there was a confusion: the search path that you can modify will only affect require. So following should work:

Code: Select all

require('testScene_Cuboid_Non_Threaded')


as long as the two files are in the same folder (or as long as file testScene_Cuboid_Non_Threaded.lua is in one of the folders listed in package.path.

Do not use script simulation parameters to remember if it was the first run: a script's state is always newly initialized the first time the script is run, so something like:

Code: Select all

if not notFirst then
    ...
    notFirst=true
end


Should work. You basically need a very similar code as what is in function include, except that the path should be same as the current scene. So something like following:

Code: Select all

if not __notFirst__ then
    local file='/testScene_Cuboid_Non_Threaded.lua'
    local scenePath=simGetStringParameter(sim_stringparam_scene_path)
    __notFirst__=true
    __scriptCodeToRun__=assert(loadfile(scenePath..file))
    if cmd then
        local tmp=assert(loadstring(cmd))
        if tmp then
            tmp()
        end
    end
end
if __scriptCodeToRun__ then
    __scriptCodeToRun__()
end


Cheers