Rotem Kupfer & Alon Gat Under supervision of

Dr. Ohad Ben Shahar & Mr. Yair Adato

Nonlinear optical flow estimation

Department of Computer ScienceBen-Gurion University

Undergraduate Project Presentation DayMay 2011

Given: Two or more frames of an image sequence.Wanted: Displacement field between two consecutive frames optical flow

Problem definition:

We want to find and that minimize the energy functional:

Horn & Schunck Algorithm

dxdyvvuu

tyxItvyuxIyxvyxuE

yxyx ))()((

),,()1,,()),(),,((

2222

2

Smoothness

term

Data term brightness constancy

• Data term - Penalizes deviations from constancy assumptions.• Smoothness term - Penalizes dev. from smoothness of the solution.• Regularization parameter α - Determines the degree of smoothness.

Output – the optical flow!

Output

Minimizing using Euler-Lagrange doesn’t guarantee global minimum, so a Pyramid Method is enforced, which causes loss of vital information in some type of problems.

Main Goal: trying to find a method that wont need Pyramids.

Problems with this method.

Instead of solving E-L equations (which need to be linearized) we wrote the Functional as a set of n*m equations and minimized it with regular minimization methods (Gradient decent, Quasi Newton, and others).

So what we tried to do?

dxdyvvuu

tyxItvyuxIyxvyxuE

yxyx ))()((

),,()1,,()),(),,((

2222

2

Mathemactica for symbolic calculations.

Where P are the Image derivatives, and u, v is the optical flow components in the vertical and horizontal directions.

Mathematica Symbolic Toolbox for MATLAB--Version 2.0(http://library.wolfram.com/infocenter/MathSource/5344/)

The functional in polar coordinates:

, ,

Trying Polar Representation

Quasi Newton IterationProblem: Calculation time of the inverse of non sparse matrix is long. O(n^3) -> resulting ((n*m)^3)

Gradient Decent.Advantage: Doesn’t require calculation of the Hessian matrix.

MATLABfor numeric calculations

Translating the symbolic expression to numeric scheme:for i=2:imageSizeN-1 for j=2:imageSizeM-1 gradC(i,j)=gradC(i,j)+alpha*(2*(-c(i-1,j)+c(i,j))+2*(-c(i,j-1)+c(i,j))-2*alpha*(-c(i,j)+c(i,j+1)) - 2*alpha*(-c(i,j)

+c(i+1,j))+4*exp((-1+c(i,j)^2+s(i,j)^2)^2)*c(i,j)*(-1+c(i,j)^2+s(i,j)^2) +2*(Ix(i,j)*m(i,j)+Ixz(i,j)*m(i,j)+2*Ixx(i,j)*c(i,j)*m(i,j)^2+Ixy(i,j)*m(i,j)^2*s(i,j))*(Iz(i,j)+Izz(i,j)+Ix(i,j)*c(i,j)*m(i,j)+Ixz(i,j)*c(i,j)*m(i,j)+Ixx(i,j)*c(i,j)^2*m(i,j)^2+Iy(i,j)*m(i,j)*s(i,j) +Iyz(i,j)*m(i,j)*s(i,j)+Ixy(i,j)*c(i,j)*m(i,j)^2*s(i,j)+Iyy(i,j)*m(i,j)^2*s(i,j)^2));

gradS(i,j)=gradS(i,j)+4*exp((-1+c(i,j)^2+s(i,j)^2)^2)*s(i,j)*(-1+c(i,j)^2+s(i,j)^2)+2*(Iy(i,j)*m(i,j)+Iyz(i,j)*m(i,j)+Ixy(i,j)*c(i,j)*m(i,j)^2+2*Iyy(i,j)*m(i,j)^2*s(i,j))*(Iz(i,j)+Izz(i,j) +Ix(i,j)*c(i,j)*m(i,j)+Ixz(i,j)*c(i,j)*m(i,j)+Ixx(i,j)*c(i,j)^2*m(i,j)^2+Iy(i,j)*m(i,j)*s(i,j)+Iyz(i,j)*m(i,j)*s(i,j)+Ixy(i,j)*c(i,j)*m(i,j)^2*s(i,j)+Iyy(i,j)*m(i,j)^2*s(i,j)^2) +alpha*(2*(-s(i-1,j)+s(i,j))+2*(-s(i,j-1)+s(i,j))-2*alpha*(-s(i,j)+s(i,j+1))-2*alpha*(-s(i,j)+s(i+1,j)));

gradM(i,j)=gradM(i,j)+beta*(2*(-m(i-1,j)+m(i,j))+2*(-m(i,j-1)+m(i,j)))-2*beta*(-m(i,j)+m(i,j+1))-2*beta*(-m(i,j)+m(i+1,j))+2*(Ix(i,j)*c(i,j)+Ixz(i,j)*c(i,j) +2*(Ixx(i,j)*c(i,j)^2*m(i,j)+Iy(i,j)*s(i,j)+Iyz(i,j)*s(i,j)+2*Ixy(i,j)*c(i,j)*m(i,j)*s(i,j)+2*Iyy(i,j)*m(i,j)*s(i,j)^2)*(Iz(i,j)+Izz(i,j)+Ix(i,j)*c(i,j)*m(i,j)+Ixz(i,j)*c(i,j)*m(i,j) +Ixx(i,j)*c(i,j)^2*m(i,j)^2+Iy(i,j)*m(i,j)*s(i,j)+Iyz(i,j)*m(i,j)*s(i,j)+Ixy(i,j)*c(i,j)*m(i,j)^2*s(i,j)+Iyy(i,j)*m(i,j)^2*s(i,j)^2));

end end end

Leaving Mathematica

Calculation time of 60-360 minutes! While high end algorithms run at around 20 secs.(CPU time complexity)

There are couple of parameters (alpha, number of iteration, with or without filters (texture decomposition and median filter)) that are needed to be set each run, and they are problem specific.

(OUR time complexity)

Resulting Big HEADACHE !

Optimization

So why use only one core when we have two or more? The OpenMP Application Program Interface supports multi-

platform shared-memory parallel programming in C/C++ on all architectures, OpenMP is a portable, scalable model that gives shared-memory parallel programmers a simple and flexible interface for developing parallel applications for platforms ranging from the desktop to the supercomputer.

Optimization OPENMP – interface matlab OPENMP

:// . /http openmp org

MATLAB MEX FILES (In C++) OPENMP

Condor is a specialized workload management system for compute-intensive jobs.

So instead of specifically trying to find the ‘best’ parameters, we just ‘tried them all’.

We used condor in order run multiple (100 and more) experiments in parallel.

Optimization Condor to The Rescue

:// . . . / /http www cs wisc edu condor

It’s installed only on EE Labs… It’s easy to use, very practical, it’s FREE and

CONDOR VIEW

Some preliminary results – Cartesian functional:

Starting from boundaries

Some preliminary results – Polar functional

6 pyramids, 1000 iterations 840secs

NO pyramids, 1000 iterations 283 sec ~ 3mins!

Best Results

We have successfully ran experiments without the need for Pyramids (in “fair” play).

Gradient Decent uses normalized steps - very small step size, resulting |u| |v| that are small.

Using problem specific tools (like Condor and OPENMP) is HIGHLY recommended. (saves time, and time is money)

Conclusion and future direction

The Ground Truth.

Questions?