Embed Size (px)
Citation preview
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Nikos Paragioshttp://cermics.enpc.fr/~paragios
CERTISCERTISEcole Nationale des Ponts et ChausseesEcole Nationale des Ponts et Chaussees
Paris, FranceParis, France
Level Set Methods in Medical Image Analysis: Segmentation
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Nikos Paragioshttp://cermics.enpc.fr/~paragios
Atlantis Research GroupAtlantis Research GroupEcole Nationale des Ponts et ChausseesEcole Nationale des Ponts et Chaussees
Paris, FranceParis, France
Stanley Osherhttp://math.ucla.edu/~sjo
Department of MathematicsDepartment of MathematicsUniversity of California, Los Angeles University of California, Los Angeles
USAUSA
Http://cermics.enpc.fr/~paragios/book/book.html
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Outline
Introduction/Motivation
On the Propagation of Curves The snake model
The level set method Basic Derivation, algorithms Boundary-driven and Region-driven model free segmentation
The Level Set Method as a Direct Optimization Space Multiphase Motion Region-driven model free image segmentation Knowledge-based Object Extraction Shape Registration
Discussion
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Motivation
Image Segmentation and image registration are core components of medical imaging
2002 The word “Segmentation” appears 34 times at MICCAI’02 program
The word “Registration” appears 22 times at MICCAI’02 program
2003: The word “Segmentation” appears 47 times at MICCAI’03 program ~
25%
The word “Registration” appears 53 times at MICCAI’03 program ~
25%
2004: The word “Segmentation” appears 51 times at MICCAI’04 program ~
25%
The word “Registration” appears 67 times at MICCAI’04 program ~
35%
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Overview of Segmentation Techniques
Boundary-driven Edge Detectors (model free) Active Contours/snakes (model free + knowledge-based) Active Shape Models (knowledge-based)
Region-driven Deformable templates (knowledge-based) Statistical/clustering techniques (model free + knowledge-based) MRF-based techniques (model free) Active Appearance Models (knowledge-based)
Boundary + Region-driven Active Contours (model free + knowledge-based) Graph-based Techniques (model free) Level Set Methods (model free + knowledge-based)
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
On the propagation of Curves
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
On the Propagation of Curves
Snake Model (1987) [Kass-Witkin-Terzopoulos]
Planar parameterized curve C:R-->RxR A cost function defined along that curve
The internal term stands for regularity/smoothness along the curve and has two components (resisting to stretching and bending)
The image term guides the active contour towards the desired image properties (strong gradients)
The external term can be used to account for user-defined constraints, or prior knowledge on the structure to be recovered
The lowest potential of such a cost function refers to an equilibrium of these terms
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Active Contour Components
The internal term…
The first order derivative makes the snake behave as a membrane
The second order derivative makes the snake act like a thin plate
The image term…
Can guide the snake to Iso-photes , edges
and terminations
Numerous Provisions…: balloon models, region-snakes, etc…
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Optimizing Active Contours
Taking the Euler-Lagrange equations:
That are used to update the position of an initial curve towards the desired image properties
Initial the curve, using a certain number of control points as well as a set of basic functions,
Update the positions of the control points by solving the above equation
Re-parameterize the evolving contour, and continue the process until convergence of the process…
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Pros/Cons of such an approach
Pros Low complexity Easy to introduce prior knowledge Can account for open as well as closed structures A well established technique, numerous publications it works User Interactivity Demetri Terzopoulos is a very good friend
Cons Selection on the parameter space and the sampling rule affects the
final segmentation result Estimation of the internal geometric properties of the curve in
particular higher order derivatives Quite sensitive to the initial conditions, Changes of topology (some efforts were done to address the
problem)
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Level Set: The basic Derivation
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
The Level Set Method
Osher-Sethian (1987) Earlier: Dervieux, Thomassett, (1979, 1980)
Introduced in the area of fluid dynamics
Vision and image segmentation Caselles-Catte-coll-Dibos (1992) Malladi-Sethian-Vermuri (1994)
Level Set Milestones Faugeras-keriven (1998) stereo reconstruction Paragios-Deriche (1998), active regions and grouping Chan-Vese (1999) mumford-shah variant Leventon-Grimson-Faugeras-etal (2000) shape priors Zhao-Fedkiew-Osher (2001) computer graphics
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
The Level Set Method
Let us consider in the most general case the following form of curve propagation:
Addressing the problem in a higher dimension…
The level set method represents the curve in the form of an implicit surface:
That is derived from the initial contour according to the following condition:
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
The Level Set Method
Construction of the implicit function
And taking the derivative with respect to time (using the chain rule)
And we are DONE…
(1)
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
The Level Set Method
Let us consider the arc-length (c) parameterization of the
curve, then taking the directional derivative of in that dire- ction we will observe no change:
leading to the conclusion that the is ortho-normal to C where
the following expression for the normal vector
Embedding the expression of the normal vector to:
the following flow for the implicit function is recovered:(2)
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Level Set Method (the basic derivation)
Where a connection between the curve propagation flow and the flow deforming the implicit function was established
Given an initial contour, an implicit function is defined and deformed at each pixel according to the equation (2) where the zero-level set corresponds to the actual position of the curve at a given frame
Euclidean distance transforms are used in most of the cases as embedding function
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Overview of the Method
The level set flow can be re-written in the following form
where H is known to be the Hamiltonian. Numerical approximations is then done according to the form of the Hamiltonian
Determine the initial implicit function (distance transform) Evolve it locally according to the level set flow Recover the zero-level set iso-surface (curve position) Re-initialize the implicit function and Go to step (1) of the loop
Computationally expensive
Open Questions: re-initialization…and numerical approximations
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Implementation Details…
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
The normal to the curve/surface can be determined directly from the level set function:
The curvature can also be recovered from the implicit function, by taking the second order derivative at the arc length
Where we observe no variation since the implicit function has constant “zero” values, and given that as well as one can easily prove that:
That can also be extended to higher dimensions
Level Set Method and Internal Curve Properties
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Minimize the Euclidean length of a
curve/surface:
The corresponding level set variant with a
distance transform as an implicit function:
Things become little bit more complicated
at 3D (Gaussian Curvature)
Results are courtesy Prof. J. Sethian
(Berkeley) & G. Hermosillo (INRIA)
Examples: Mean/Gaussian Curvature Flow
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
From theory to Practice (Narrow Band) [Chop:93, Adalsteinsson-Sethian:95]
Central idea: we are interested on the motion of the zero-level set and not for the motion of each iso-phote of the surface
Extract the latest position Define a band within a certain distance Update the level set function Check new position with respect
the limits of the band Update the position of the band
regularly, and re-initialize the implicit function
Significant decrease on the computational complexity, in particular when implemented efficiently and can account for any type of motion flows
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Narrow Band (the basic derivation)
Results are courtesy: R.
Deriche
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Handling the Distance Function
The distance function has to be frequently re-initialized…
Extraction of the curve position & re-initialization: Using the marching cubes one can recover the current position of the
curve, set it to zero and then re-initialize the implicit function: the
Borgefors approach, the Fast Marching method, explicit estimation of
the distance for all image pixels…
Preserving the curve position and refinement of the existing function (Susman-smereka-osher:94)
Modification on the level set flow such that the distance transform property is preserved (gomes-faugeras:00)
Extend the speed of the zero level set to all iso-photes, rather
complicated approach with limited added value?
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
From theory to Practice (Fast Marching) [Tsitsiklis:93,Sethian:95]
Central idea: “move” the curve one pixel in a progressive manner according to the speed function while preserving the nature of the implicit function
Consider the stationary equation
Such an equation can be recovered for all flows where the speed function has one sign (either positive or negative), propagation takes place at one direction
If T(x,y) is the time when the implicit function reaches (x,y):
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Fast Marching (continued)
Consider the stationary equation in its discrete form:
And using the assumption
that the surface propaga-
tes in one direction, the so-
lution can be obtained by
outwards propagation from
the smallest T value…
active pixels, the curve has already reached them alive pixels, the curve could reach them at the next stage far away pixels, the curve cannot reach them at this stage
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Fast Marching (continued)
INITIAL STEP
Initialize for the all pixels of the front (active), their first order neighbors alive and the rest far away
For the first order neighbors,
estimate the arrival time according to: While for the rest the crossing time is set to infinity
PROPAGATION STEP Select the pixel with the lowest arrival time from the alive ones Change his label from alive to active and for his first order
neighbors: If they are alive, update their T value according to
If they are far away, estimate the arrival time according to:
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Fast Marching Pros/Cons, Some Results
Fast approach for a level set
implementation
Very efficient technique for re-setting the
embedding function to be distance
transform
Single directional flows, great importance
on initial placement of the contours
Absence of curvature related terms or
terms that depend on the geometric
properties of the curve…
Results are courtesy: J. Sethian, R.
Malladi, T. Deschamps, L. Cohen
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Level Sets in imaging and vision…the edge-driven case
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Emigration from Fluid Dynamics to Vision
(Caselles-Cate-Coll-Dibos:93,Malladi-Sethian-Vemuri:94)
have proposed geometric flows to boundary extraction
Where g(;) is a function that accounts for strong image
gradients
And the other terms are application specific…that either
expand or shrink constantly the initial curve
Distance transforms have been used as embedding
functions
Malladi-Sethian-Vemuri:94
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Geodesic Active Contours [Caselles-Kimmel-Sapiro:95, Kichenassamy-Kumar-etal95]
Connection between level set methods and snake driven optimization
The geodesic active contour consists of a simplified snake model without second order smoothness
That can be written in a more general form as
Where the image metric has been replaced with a monotonically decreasing function:
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Geodesic Active Contours [Caselles-Kimmel-Sapiro:95, Kichenassamy-Kumar-etal95]
Leading to the following more general framework…
,
One can assume that smoothness as well as image terms are equally important and with some “basic math”
That seeks a minimal length geodesic curve attracted by the desired image properties…
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Geodesic Active Contours
That when minimized leads to the following geometric flow:
Data-driven constrained by the curvature force Gradient driven term that adjusts the position of the contour
when close to the real 0bject boundaries…
By embedding this flow to a level set framework and using a distance transform as implicit function,
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Geodesic Active Contours…
That has an extra term when
compared with the flow
proposed by Malladi-Sethian-
Vemuri.
Single directional flow…requires
the initial contour to either
enclose the object or to be
completely inside...
Results are courtesy: R.
Deriche
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Gradient Vector Flow Geometric Contours [paragios-mellina-ramesh:01]
Initial conditions are an issue at the active contours since they are propagated mainly at one direction
Region terms (later introduced) is a mean to overcome this limitation…
an alternative is somehow to extend the boundary-driven speed function to account for
directionality, thus recovering a field (u,v)
One can estimate this field close to the object boundaries…where
The image gradient at the boundaries is tangent to the curve While the inward normal normal points towards the object
boundaries
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Gradient Vector Flow Geometric Contours [paragios-mellina-ramesh:01]
Let (f) be a continuous edge detector with values close to 1 at the presence of noise and 0 elsewhere…
The flow can be determined in areas with important boundary information (Important f)
And areas where there changes on f, |Gradient(f)|
While elsewhere recovering such a field is not possible and the only way to be done is through diffusion
This can be done through an approximation of image gradient at the edges and diffusion of this information for the rest of the image plane
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Gradient Vector Flow Geometric Contours
This flow can be used within a geometric flow towards image segmentation…
The direction of the propagation should be the same with the one proposed by the recovered flow, therefore one can penalize the orientation between these two vectors.
That is integrated within the classical
Geodesic active contour equation and is
implemented using the level set function
using the Additive Operator Splitting
The inner product between the curve
normal and the vector field guides the curve propagation
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Additive Operator Splitting [Weickert:98, Goldenberg-Kimmel:01]
Introduced for fast non-linear diffusion
Applied to the flow of the geodesic active contour
Where one can consider a signed Euclidean distance function to be the implicit function, leading to:
That can be written as:
That can be solved in an explicit form:
Or a semi-implicit one:
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Additive Operator Splitting (Weickert:02)
Or in a semi-implicit one
That refers to a triagonal system of equations and can be done using the Thomas algorithm…at O(N) and has to be done once…
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Some Comparison (Weickert:02)
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Level Sets in imaging and vision…the region-driven case
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
The Mumford-Shah framework [chan-vese:99, yezzi-tsai-willsky-99]
The original Mumford-Shah framework aims at partitioning the image into (multiple) classes according to a minimal length
curve and reconstructing the noisy signal in each class
Let us consider - a simplified version - the binary case and the fact that the reconstructed signal is piece-wise constant
Where the objective is to reconstruct the image, using the mean values for the inner and the outer region
Tractable problem, numerous solutions…
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
The Mumford-Shah framework [chan-vese:99, yezzi-tsai-willsky-99]
Taking the derivatives with respect to piece-wise constants, it straightforward to show that their optimal value corresponds to the means within each region:
While taking the derivatives with respect and using the stokes theorem, the following flow is recovered for the evolution of the curve:
An adaptive (directional/magnitude)-wise balloon force A smoothness force aims at minimizing the length of the partition
That can be implemented in a straightforward manner within the level set approach
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
The Mumford-Shah framework – Criticism & Results
Account for multiple classes? Quite simplistic model, quite often the means are not a good
indicator for the region statistics Absence of use on the edges, boundary information
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Geodesic Active Regions [paragios-deriche:98]
Introduce a frame partition paradigm within the level set space that can account for boundary and global region-driven information
KEY ASSUMPTIONS Optimize the position and the geometric form of the curve by
measuring information along that curve, and within the regions that compose the image partition defined by the curve:
(input image) (boundary) (region)
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Geodesic Active Regions
We assume that prior knowledge on the positions of the objects to be recovered is available - - as well as on the expected intensity properties of the object and the background
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Geodesic Active Regions
Such a cost function consists of: The geodesic active contour A region-driven partition module that aims at separating the
intensities properties of the two classes (see later analogy with the Mumford-Shah)
And can be minimized using a gradient descent method leading to:
Which can be implemented using the level set method as follows…
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Geodesic Active Regions
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Some Results…
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
…REMINDER…
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Level Set & Geometric Flows
While evolving moving interfaces with the level set method is quite attracting, still it has the limitation of being a static approach
The motion equations are derived somehow, The level set is used only as an implementation tool…
That is equivalent with saying that the problem has been somehow already solved…since there is not direct connection between the approach and the level set methodology
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Level Set: Optimization space
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Level Set Dictionary
Let us consider distance transforms as embedding function
Then following ideas introduced in [evans-gariepy:96], one can introduce the Dirac distribution
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Level Set Dictionary
Using the Dirac function and integrating within the image domain, one can estimate the length of the curve:
While integrating the Heaviside Distribution within the image domain
Such observations can be used to define regional partition modules as follows according to some descriptors
That can be optimized with respect to the level set function (implicitly with respect to a curve position)
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Level Set Optimization
And given that :
An adaptive (directional & magnitude wise) flow is recovered for the propagation of an initial surface towards a partition that is optimal according to the regional descriptors…
The same idea can be used to introduce contour-driven terms…
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Level Set Optimization
and optimize them directly on the level set space
Curve-driven terms:
Global region-driven terms:
According to some image metrics…defined along the curve and within the regions obtained through the image partition according to the position of the curve, that can be multi-component but is representing only one class
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Multiphase Motion [zhao-chan-merinman-osher:96]
Up to now statistics and image information have been used to partition image into two classes,
Often, we need more than object/background separation, and therefore the case of multi-phase motion is to be considered…
N objects/curves, represented by N level set functions
How to deal with occlusions,
one image pixel cannot be
assigned to more than one curve…
How to constrain the solution
such that the obtained partition
consists of all image data
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Multi-Phase Motion (continued)
For each class, boundary, smoothness as well as region components can be considered
Subject to the constraint at each pixel:
a hard and local constraint difficult to be imposed that could be replaced with a more convenient
That can be optimized through Lagrange multipliers method…
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Multiphase Motion & Mumford-Shah [samson-aubert-blanc-feraud:99]
Image Segmentation and Signal Reconstruction (direct
application of the (zhao-chan-merinman-osher:96) within the
Mumford Shah formulation…)
Separate the image into regions with consistent intensity properties
Recover a Gaussian distribution that expresses the intensity properties of each class, or force the intensity properties of each class to follow some predefined image characteristics
That when optimized leads to a set of equations that deforming simultaneously the initial curves according to:
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Multiphase Motion & Mumford-Shah [samson-aubert-blanc-feraud:99]
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Multi-Phase Motion
PROS Taking the level set method to another level Dealing with multiple (multi-component) objects, and multiple
tasks Introducing interactions between shape structures that evolve in
parallel
CONS
Computationally expensive Difficult to guarantee convergence Numerically unstable & hard to implement Prior knowledge required on the number of classes and in some
cases on their properties…
PARTIAL SOLUTION: The multi-phase Chan-Vese model
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Introduce classification according to a combination of all level sets at a given pixel
LEVEL SET DICTIONARY
Class 1: Class 2: Class 3: Class 4:
And therefore by taking these products one can define a modified
version of the mumford-shah approach to account with four classes
while using two level set functions…
Multi-Phase Motion [vese-chan:02]
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Multi-Phase Motion
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Multi-Phase Motion with more advanced data-driven terms
The assumption of piece-wise constant is rather weak in particular in medical imaging…
Several authors have proposed more advanced statistical formulations that are recovered “on the fly” to determine the statistics of each class
The case of non-parametric approximations of the histogram within each region is a promising direction
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Knowledge-based Object Extraction
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Knowledge-based Object Extraction
Objective: recover from the image a structure
of a particular – known to some extend
– geometric form
Methodology Consider a set of training examples Register these examples to a common pose Construct a compact model that expresses the
variability of the training set Given a new image, recover the area where the
underlying object looks like that one learnt
Advantages of doing that on the LS space: Preserve the implicit geometry Account with multi-component objects… … all wonderful staff you can do with the LS
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Knowledge-based Segmentation [leventon-faugeras-grimson-etal:00]
Concept: Alternate between segmentation & imposing prior knowledge
Learn a Gaussian distribution of the
shape to be recovered from a training
set directly at the space of implicit functions The elements of the training set are registered
A principal component analysis is use to recover
the covariance matrix of probability density function of this set
ALTERNATE Evolve a let set function according to the geodesic active contour
Given its current form, deform it locally using a MAP criterion so it fits
better with the prior distribution
Until convergence…
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Knowledge-based Segmentation [leventon-faugeras-grimson-etal:00]
Limitations: Data driven & prior term are decoupled Building density functions on high dimensional spaces is an ill
posed problem, Dealing with scale and pose variations (they are not explicitely
addressed)
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Knowledge-based Segmentation [chen-etal:01]
Concept level:
Use an average model as prior in its implicit function For a given curve find the transformation that projects it closer to
the zero-level set of the implicit representation of the prior For a given transformation evolve the curve locally towards
better fitting with the prior… Couple prior with the image driven term in a direct form…
Issues to be addressed: Model is very simplistic (average shape) – opposite to the leventon’s
case where it was too much complicated…
Estimation of the projection between the curve and the model space is
tricky…not enough support…data term can be improved…
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Knowledge-based Segmentation [chen-etal:01]
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Knowledge-based Segmentation [tsai-yezzi-etal:01]
At a concept level, prior knowledge is modeled through a Gaussian distribution on the space of distance functions by performing a singular value decomposition on the set of registered training set,
The mumford-shah framework determined at space of the model is used to segment objects according to various data-driven terms
The parameters of the projection are recovered at the same time with the segentation result…
A more convenient approach than the one of Leventon-etal Which suffers from not comparing directly the structure that is
recovered with the model…
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Knowledge-based Segmentation [paragios-rousson:02]
Prior is imposed by direct comparison between the model and evolving contour modulo a similarity transformation…
The model consists of a stochastic level set with two components,
A distance map that refers to the average model And a confidence map that dictates the accuracy of the model
Objective: Recover a level set that pixel-wise looks like the prior modulo some transformation
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Model Construction
From a training set recover the most representative model;
If we assume N samples on the training set, then the distribution that expresses at a given point most of these samples is the one recovered through MAP
Where at a given pixel, we recover the mean and the variance that best describes the training set composed of implicit functions at this point, where the mean corresponds to the average value
Constraints on the variance to be locally smooth is a natural assumption
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Model Construction (continued)
The calculus of variations can lead to the estimation of the mean and variance (confidence measure) of the model at et each pixel,
However, the resulting model will not be an implicit function in the sense of distance transform (averaging distance transforms doesn’t necessary produce one)
One can seek for a solution of the previously defined objective function subject to the constraint the “means” field forms a distance transform using Lagrange multipliers…
An alternative is to consider the process in repeated steps where first a solution that fits the data is recovered and then is projected to the space of distance functions…
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Imposing the (Static) Prior
Define/recover a morphing function “A” that creates correspondence between the model and the prior
In the absence of scale variations, and in the case of global morphing functions one can compare the evolving contour with the model according to
That modulo the morphing function will evolve the contours towards a better fit with the model
One can prove that scale variations introduce a multiplicative factor and they have to be explicitly taken into account
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Static Prior (continued)
Where the unknowns are the morphing function and the position of the level set
Calculus of variations with respect to the position of the interface are straightforward:
The second term is a constant inflation term aims at minimizing the area of the contour and eventually the cost function and can be ignored…since it has no physical meaning.
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Static Prior, Concept Demonstration
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Static Prior (continued)
One can also optimize the cost function with respect to the unknown parameters of the morphing function
Leading to a nice “self-sufficient” system of motion equations that update the global registration parameters between the evolving curve and the model
However, the variability of the model was not considered up to this point and areas with high uncertainties will have the same impact on the process
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Some Results (non-medical)
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Taking Into Account the Model Uncertainties
Maximizing the joint posterior (segmentation/morphing) is a quite attractive criterion in “inferencing”
Where the Bayes rule was considered and given that the probability for a given prior model is fixed and we can assume that all (segmentation/morphing) solutions are equally probable, we get
Under the assumption of independence...within pixels…and then finding the optimal implicit function and its morphing transformations is equivalent with
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Taking Into Account the Model Uncertainties
That can be further developed using the Gaussian nature of the model distribution at each image pixel
A term that aims at recovering a transformation and a level set that when projected to the model, it is projected to areas with low variance (high confidence)
A term that aims at minimizing the actual distance between the level set function and the model and is scaled according to the model confidence…
would prefer have a better match between the model and level set in
areas where the variability is low,
while in areas with important deviation of the training set, this term
will be less important
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Taking the derivatives…
Calculus of variations regarding the level set and the morphing function:
The level set deformation flow consists of two terms: that is a constant deflation force (when the level set function
collapses, eventually the cost function reaches the lowest potential)
An adaptive balloon (directional/magnitude-wise) force that inflates/deflates the level set so it fits better with the prior after its projection to the model space…In areas with high variance this term become less significant and data-terms guide the level set to the real object boundaries...
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Comparative Results…
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Some Videos…(again non-medical)
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Some medical results
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Implicit Active Shapes [rousson-paragios:03]
The Active Shape Model of Cootes et al. is quite popular to object extraction. Such modeling consists of the following steps:
Let us consider a training set of registered surfaces (implicit representations can also be used for registration [4]). Distance maps are computed for each surface:
The samples are centered with respect to the average representation :
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Implicit Active Shapes [rousson-paragios:03]
Training set:
The principal modes of variation are recovered through Principal Component Analysis (PCA). A new shape can be generated from the (m) retained modes:
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
The model…
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
The prior
A level set function that has minimal distance from a linear from the model space…
The unknown consist of: The form of the implicit function The global transformation between the average mode and the
image, The set of linear coefficients that when applied to the set of basis
functions provides the optimal match of the current contour with the model space
And are recovered in a straightforward manner using a gradient descent method…
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Some nice results…
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Conclusions
PROS
Elegant tool to track moving interfaces Implicit Curve Parameterization & estimation of the geometric
Properties Able to account with topological changes, able to describe multi-
component objects
CONS
Computational complexity Numerical approximations, redundancy Open Curves, sorry we CANNOT do anything about that…
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Nikos Paragioshttp://cermics.enpc.fr/~paragios
Atlantis Research GroupAtlantis Research GroupEcole Nationale des Ponts et ChausseesEcole Nationale des Ponts et Chaussees
Paris, FranceParis, France
Stanley Osherhttp://math.ucla.edu/~sjo
Department of MathematicsDepartment of MathematicsUniversity of California, Los Angeles University of California, Los Angeles
USAUSA
Http://cermics.enpc.fr/~paragios/book/book.html
C
en
tre
d‘E
eig
nem
en
t e
t d
e R
ech
erc
he
en
Tech
nolo
gie
s
de L‘I
nfo
rmati
on
et
Syste
mes
Resources
Books James Sethian (1996,1999): Level Set & Fast Marching Methods,
Cambridge, Introductory. Stan Osher & Ronald Fedkiw (2002): Level Set Methods and Dynamic
Implicit Surfaces, Springer, Introductory. Stan Osher & Nikos Paragios (2003): Geometric Level Set in Imaging
Vision and Graphics, Springer, …Mostly Vision…bit advanced…
People [non-exclusive list] Laurent Cohen (medical), David Breen (graphics), Rachid Deriche
(segmentation, tracking, DTI), Eric Grimson (medical), Olivier Faugeras (stereo), Renaud Keriven (stereo, segmentation), Ron Kimmel (segmentation, shape from shading, tracking), Jerry Prince (topology preserving), Guillermo Sapiro (segmentation, tracking, implicit surfaces), James Sethian, Baba Vemuri (Diffusion, Segmentation, Registration) Joachim Weickert (diffusion, segmentation), Ross Whitaker (Graphics), Allan Willsky (medical), Anthony Yezzi (medical), …
