## extraction of points positions from a moving surface

Typically: "How do I... ", "How can I... " questions
medp
Posts: 12
Joined: 27 Jun 2013, 14:53

### extraction of points positions from a moving surface

Hello everyone.
I'd like to ask some help about a project that me and my collegues are implementing for a university course.
We are asked to extract some features position (in our case two coloured dots) placed on the surface of a pulsing human heart.
We saw some of the tutorials on how to use vision sensors like the blob detection but we are not able to retrieve the wanted information, in particular we need to extract the "features" information in order to reconstruct the periodical motion of the heart via FFT and Klaman filtering and then use this information for a visual servoing task during a minimal invasive surgery.

We finded a built-in filter in the vision sensor to retrieve particular colors but we dont know how to extract information from the filter.
Can someone help us to obtain the position of the points during the heart motion?

PS
If someone need a little overview on what we are asking here is the introduction to what we need to do
Abstract— In this paper , we present a novel approach for
mapping periodically moving visual features with a monocular
camera. Our target application is the estimation of moving
surfaces during minimally invasive surgery for the purpose of
aiding in the guidance of surgical tools. Our approach uses
a bank of Kalman filters to estimate FFT parameters that
encode the periodic motion of visually detected features. T o
ensure convergent estimation for this highly nonlinear problem,
we have developed an iterative update procedure that treats
the Kalman filter measurement update step as an optimization
problem. Unlike existing solutions that rely on stereo vision,
our approach estimates periodic motion with a single moving
camera. With an experiment involving a beating heart phantom,
we have shown that our approach is able to successfully estimate
the periodic motion of visual features.

coppelia
Posts: 7597
Joined: 14 Dec 2012, 00:25

### Re: extraction of points positions from a moving surface

Hello,

using vision sensors, you can compose following filter sandwich:
• Original image to work image
• Selective color on work image
• Blob detection on work image
• Work image to output image
First, disable the 3rd filter component and adjust the 2nd filter in order to filter out all colors that don't interest you (i.e. just keeping your dots visible). Then enable the 3rd filter component. Make sure the dots are detected (i.e. the blob detection algorithm will add small boxes around the detected blobs)

From a child script, you can then access the blob positions with a code similar to this one:

Code: Select all

result,t0,t1=simReadVisionSensor(camera) -- Here we read the image processing camera!
if (t1) then -- in t1 we should have the blob information if the camera was set-up correctly
blobCount=t1[1]
dataSizePerBlob=t1[2]
-- Now we go through all blobs:
for i=1,blobCount,1 do
blobSize=t1[2+(i-1)*dataSizePerBlob+1]
blobOrientation=t1[2+(i-1)*dataSizePerBlob+2]
blobPos={t1[2+(i-1)*dataSizePerBlob+3],t1[2+(i-1)*dataSizePerBlob+4]}
blobBoxDimensions={t1[2+(i-1)*dataSizePerBlob+5],t1[2+(i-1)*dataSizePerBlob+6]}
end
end
The blob positions (blobPos) indicate the x/y position of the blob relative to the image (x=0 --> left border, x=1 --> right border, y=0 --> lower border, y=1 --> upper border)

Cheers

medp
Posts: 12
Joined: 27 Jun 2013, 14:53

### Re: extraction of points positions from a moving surface

I really appreciate your help, I'll try and let you know.