1PhD defense M. García – 19/01/2009 – CERFACS
PhD defense: Marta García
Development and validation of the Euler-Lagrange formulation on a parallel and
unstructured solver for large-eddy simulation
Director: T. Poinsot & Co-director: V. Moureau
2PhD defense M. García – 19/01/2009 – CERFACS
THE CONTEXT
Human nature: try to understand phenomena, comprehension effort, power of fire, energy conversion …
• Observation• Experimentation• Numerical simulation
Gas turbine
engines
fire
locomotive
This thesis is focused on the improvement of current tools to the comprehension of multiphase flows by
using numerical simulation.
3PhD defense M. García – 19/01/2009 – CERFACS
THE CONTEXT: EXPERIMENTS vs NUMERICAL SIMULATIONS
EXPERIMENTS
NUMERICAL SIMULATION
• expensive• destructives• difficult to reproduce exactly
• less expensive• not destructives• reproductibility
Ham et al. Annual Research Briefs 2003 CTR Stanford Univ.
Spray evolution from a realistic gas-turbine injector.
4PhD defense M. García – 19/01/2009 – CERFACS
THE CONTEXT: INCREASE OF COMPUTER POWER
(1) Flops: floating point operations per second
1.65x1012 Flops(1)
280x1012 Flops
1012 = 1.000.000.000.000
1105x1012 Flops
12.64x1012 FlopsIn the last 3 years CPU time divided by 8 (approx.)
8 weeks 1 week
5PhD defense M. García – 19/01/2009 – CERFACS
THE CONTEXT: TWO-PHASE FLOW NUMERICAL SIMULATION
Eulerian formulation
PhDcerfacs2
21
PhDimft1
PhDimft1
up~
TPF Team
Current treatment of the dispersed phase in AVBP
(TPF)
PhDPhD
PhD
PhD
PhD
PhD
PhD
PhD PhD
PhD
PhD
PhD
PhD
PhD
PhD
Lagrangian formulation ?
in 2005 …
PhD
… considering the increasing computer power and my
experience accumulated in the past 3 years …
???Kaufmann 2004
PhD’s: CERFACS & IMFT
Mossa 2005Pascaud 2006
Boileau 2007Riber 2007
Lavedrine 2008 Lamarque 2007
…
PhDcerfacs2
6PhD defense M. García – 19/01/2009 – CERFACS
THE CONTEXT: TWO-PHASE FLOW NUMERICAL SIMULATIONEuler-Euler vs Euler-Lagrange
Euler-Lagrange
Individual particle trajectories are computed
+ Easy modeling of particle movements and interactions.+ Robust and accurate if enough particles are used.+ Size distributions easy to describe.+ Easy to implement physical phenomena (e.g. heat and mass transfer, wall-particle interaction).
- Delicate coupling with combustion.- Difficult to run in parallel.- Each ‘particle’ actually represents an ensemble of particles.
Euler-Euler
+ Easy treatment of dense zones.
+ Similarity with gaseous equations.
+ Direct transport of Eulerian quantities.
+ Similarity with gaseous parallelisme.
- Difficult description of polydispersion.- Difficulty of crossing sprays treatment.- Limitation of the method in very dilute zones.
Particle ensemble viewed as a continuous field
7PhD defense M. García – 19/01/2009 – CERFACS
THE OBJECTIVES OF THE WORK
• Develop a Lagrangian formulation for two-phase flow treatment within a parallel, unstructured and hybrid solver AVBP.
• Perform the first simulations on academic and complex geometries.
• Verify the efficient parallel implementation to maintain good performance on massively parallel machines.
8PhD defense M. García – 19/01/2009 – CERFACS
THE PLAN OF THE PRESENTATION
I. Presentation of the simulation tool: AVBP solver
II. Description of the Lagrangian module
III. Application test cases:
IV. Conclusions and perspectives
• Decaying homogenous isotropic turbulence• Polydisperse two-phase flow of a confined bluff body
• Particle equations of motion• Particle tracking algorithm
• Quick introduction• Domain partitioning• Rounding errors and repetitivity of LES
9PhD defense M. García – 19/01/2009 – CERFACS
THE AVBP SOLVER
Parallel solver started in 1993. Unstructured solver capable of handling hybrid grids of different cell types. Computational Fluid Dynamics (CFD) code to solve laminar and turbulent compressible Navier-Stokes equations in 2 and 3 space dimensions. Built upon a modular software library that includes integrated parallel domain partition and data reordering tools, message passing (MPI) and includes supporting routines for dynamic memory allocation, routines for parallel I/O and iterative methods. Written in standard Fortran 77 and C, but it is being upgraded to Fortran 90 in a gradual fashion. Highly portable to different parallel machines.
10PhD defense M. García – 19/01/2009 – CERFACS
THE AVBP SOLVER: PARTITIONING ALGORITHMS
RCB
RIB
RGB
R = recursiveB = bisection
C = coordinate
I = inertial
G = graph
… currenly available in AVBP
DUAL MESH
NODAL MESH
11PhD defense M. García – 19/01/2009 – CERFACS
THE AVBP SOLVER: PARTITIONING ALGORITHMS
MESH
Total No of nodes
495,232
503,230
530,852
367,313(before partitioning)
(after partitioning)
RCB
RIB
RGB
+ 35%
+ 37%
+ 45%
CPU time of 1000 it. (s)
361.5
366.96
405.64
+ 1.5%
+ 12%
The choice of partitioning algorithm has an effect on the CPU time of your simulation.
Need of a new partitioning algorithm:
• Faster partitioning• Lower number of total nodes after partitioning• With parallel version• With multi-constraint partitioning options
Choice done: METIS package implemented during this thesis
12PhD defense M. García – 19/01/2009 – CERFACS
THE AVBP SOLVER: PARTITIONING ALGORITHMSSome results obtained with METIS multilevel partitioning algorithm …
ARRIUS2_10M
ARRIUS2_44M
COMPARISON OF ALGORITHMSNo of nodes after partitioning
No of nodes after partitioning
No of subdomains
No of subdomains
No of subdomains
29 minutes21 minutes
4096 procs
METIS algorithm is faster It produces a lower
number of nodes after partitioning
13PhD defense M. García – 19/01/2009 – CERFACS
THE AVBP SOLVER: ROUNDING ERRORS… and repetitivity of LES
MESH
Finite precision computation: lack of associativity property !!
AVBP: Parallel solver, highly portable to solve laminar and turbulent compressible Navier-Stokes equations.
What that means …
€
R1
€
R2
€
R3
€
R4
zoom
€
14
R4 + R3 + R2 + R1( )[ ]{ }RCM
€
R1
€
R2
€
R3
€
R4
CM
RCM
€
14
R4 + R3 + R2 + R1( )[ ]{ }CM
€
≠ABCD
A B C D
Work published in the AIAA Journal publication:
AIAA Journal Vol. 46, No 7, July 2008“Growth of Rounding Errors and Repetitivity of Large-Eddy Simulations”
J.-M. Senoner, M. García, S. Mendez, G. Staffelbach, O. Vermorel and T. Poinsot
14PhD defense M. García – 19/01/2009 – CERFACS
THE AVBP SOLVER: ROUNDING ERRORSAxial velocity fields of a turbulent channel (TC) at different instants
(t1)
4 procs
8 procs
Axial velocity (m/s)
(t2)Axial velocity (m/s)(t3)
Axial velocity (m/s)
4 procs
8 procs
4 procs
8 procs
1. Instantaneous solutions in unsteady simulations.
2. Same initial conditions.3. Different number of processors.
DIFFERENCES OBSERVED BETWEEN
TWO SNAPSHOTS
TWO NORMS ARE USED TO COMPARE RESULTS BETWEEN TWO SOLUTIONS
15PhD defense M. García – 19/01/2009 – CERFACS
THE AVBP SOLVER: ROUNDING ERRORSDifferent effects observed on repetitivity of LES
Effect of node reordering
Effect of initial conditions
Reprinted by permission of the American Institute of Aeronautics and Astronautics.
Effect of machine precisionquadruple
double
simple
Effect of turbulence
turbulent
laminar
Machine precision differences
Norm saturation
Any sufficiently turbulent flow computed in LES exhibits significant sensitivity to small perturbations, leading to instantaneous solutions which can be totally different.
The divergence of solutions is due to 2 combined facts:
1. The exponential separation of trajectories in turbulent flows.
2. The different propagation of rounding errors induced by domain partitioning and scheduling operations.The validation of an LES code after modifications may
only be based on statistical fields.
16PhD defense M. García – 19/01/2009 – CERFACS
THE PLAN OF THE PRESENTATION
I. Presentation of the simulation tool: AVBP solver
II. Description of the Lagrangian module
III. Application test cases:
IV. Conclusions and perspectives
• Decaying homogenous isotropic turbulence• Polydisperse two-phase flow of a confined bluff body
• Particle equations of motion• Particle tracking algorithm
• Quick introduction• Domain partitioning• Rounding errors and repetitivity of LES
17PhD defense M. García – 19/01/2009 – CERFACS
THE LAGRANGIAN MODULE: PARTICLE EQUATIONS… of motion
Individual particle trajectories are computed with a Lagrangian solver coupled to the LES code for the gas phase.
N droplets to track (order of a few millions)
€
dx p,i
dt= up,i
€
dup,i
dt= − 3
4ρ g
ρ p
CD
dp
vr vr,i + gi = −up ,i − ˜ u g,i
τ p
+ gi
€
τ p = 34
ρ p
ρ g
dp
CD vr
€
with
[ Schiller & Nauman. 1935 ]
€
ρp >> ρ g
Assumptions:
spheres
€
CD
€
vr,i = up .i − ˜ u g,i
€
and
Particles equation of motion
Need to know the gas velocity at each particle location (linear interpolation)
€
˜ u g,i = ˆ u g ,i
The effect of the subgrid fluid velocity is not considered in this thesis.
[ Fede & Simonin 2006 ]
drag + gravity
18PhD defense M. García – 19/01/2009 – CERFACS
THE LAGRANGIAN MODULE: KEY POINTS
X3
X1
X2
n1
n2
n3
Xp
( Xp-Xi ) ni ≥ 0
Locating particles in cellsKnowing particles positions at time n: exchange particles between processors
cell i
particle
Subdomain 1 Subdomain 2
influence node
Two-way coupling
Particles load-balancing
Interpolation algorithm
€
dx p,i
dt= up,i
€
τ p =dp
2ρ p
18μg
€
with
€
dup,i
dt= −
up,i − ˜ u g,i
τ p
+ gi
gas velocity ug,i at each particle location
Injection…
Particle-wall treatment
… for Lagrangian schemes in unstructured meshes
19PhD defense M. García – 19/01/2009 – CERFACS
THE LAGRANGIAN MODULE: LOCATING PARTICLES
Shape functions:
€
ρ x p = N i r x i
i∑
€
N i =1i
∑
€
ρ x p =
r X
r N →
r N =
r X −1r
x p
€
min(N i,1− N i) ≥ 0, ∀ i
€
Vii=1,nv∑ = Vc
Calculation of partial volumes:… in elements of arbitrary shape 2D:
3D:
To decide if the particle is in the cell or not, the scalar product between the vector starting from the vertex of the cell to the particle and the inward normal
vector of the corresponding edge is taken. The particle is inside the cell if all the scalar products of each edge are positive.
X1
X2
n1
n2
n3
Xp
Xp•
X3
n3
-
+n3
( Xp-Xi ) ni ≥ 0
Face-normals:
20PhD defense M. García – 19/01/2009 – CERFACS
THE LAGRANGIAN MODULE: SEARCH ALGORITHMS… for different situations
Search particles for the first time Search injected particles
Search particles during simulation Search particles crossing boundaries between processors
Cells of the interface (type 2)
Initial particle location
New particle location
Interface between processors
Old cell containing the particle
Cells surrounding the old cell
Initial particle location
Nodes of the containing cell
New particle location
Cells of the injection area
Particles injected
Interface between processors
Injection area
Nodes of the injection area
Quad/Octree
F. Collino(CERFACS)Use of different search algorithms depending on
the situation to reduce memory and CPU time requirements.
21PhD defense M. García – 19/01/2009 – CERFACS
THE LAGRANGIAN MODULE: INTERPOLATION
∏≠= −
−=
n
imm mi
mLi xx
xxxP1
)(
Ex. 3D with hexa: n=2 (trilinear interpolation)
)()()(),,(),,(111
zPyPxPzyxfzyxf Lk
Lj
Likji
n
i
n
j
n
k∑∑∑===
=
Lagrange interpolation (only for coordinate grids with quads or hexahedras)
1 2
2
2
Linear Least Squares (LAPACK subroutine DGELS)
1st order Taylor Serie
€
f (x) = f (n )(a)n!n= 0
∞
∑ (x − a)n
€
f (x) = f (a) + f '(a)(x − a)
Ex. 1D and 1st order
… of gaseous-phase properties at particle position
22PhD defense M. García – 19/01/2009 – CERFACS
THE LAGRANGIAN MODULE: TWO-WAY COUPLING
€
fc,i = Coupling force
€
Vfc,iP(x,y,z ) = α proj( fd ,i
n ,P(x,y,z))n in V∑
1 2 3
4 6
7 8 9
€
α = constant of proportionality
Exist an analytical solution
Validation test of two-way coupling
Momentum eq. = cte
[ Boivin et al. 1998, Boivin et al. 2000 ] [ Ph.D. O. Vermorel 2003 ]
x
y
€
up (0) = cte
€
ug (0) = 0
In the framework of PIC methods
… source terms and validation
23PhD defense M. García – 19/01/2009 – CERFACS
THE LAGRANGIAN MODULE: PARTICLE INJECTION
INJECTION GEOMETRY: simple injection options available• Point injection: all droplets are injected at the same
point.• Disk injection: droplets are injected over a disk.Example of input parameters:
Coordinates of the injection point, disk diameter, normal to define disk direction, tolerance …
PARTICLE SIZE DISTRIBUTION:• Monodisperse: all particles have the same diameter.• Polydisperse: different particle diameters.
1. Gaussian distribution2. Log-normal distribution
Example of input parameters: Type of distribution, maximum and minimum diameters, mean and standard deviation
…
24PhD defense M. García – 19/01/2009 – CERFACS
THE LAGRANGIAN MODULE: PARTICLE INJECTION
- Particle mass flow rate
- Particle diameter(s), density, mean/rms velocity ...
ZOOM Injection tube
z=-3mm
Inject # particles by timestep
… example of a disk injection
25PhD defense M. García – 19/01/2009 – CERFACS
THE PLAN OF THE PRESENTATION
I. Presentation of the simulation tool: AVBP solver
II. Description of the Lagrangian module
III. Application test cases:
IV. Conclusions and perspectives
• Decaying homogenous isotropic turbulence• Polydisperse two-phase flow of a confined bluff body
• Particle equations of motion• Particle tracking algorithm
• Quick introduction• Domain partitioning• Rounding errors and repetitivity of LES
26PhD defense M. García – 19/01/2009 – CERFACS
THE APPLICATION TEST CASES
Decaying Homogeneous Isotropic turbulence (HIT)
Polydisperse two-phase flow of a confined bluff body
• Academic test case• Well documented
• Complex recirculating flow
• Large amount of data available
27PhD defense M. García – 19/01/2009 – CERFACS
THE APPLICATION TEST CASES: DECAYING HIT
Simple configuration to:• Validate first developments of the Lagrangian version.• Localisation algorithms, interpolation, processor exchanges,
etc.
28PhD defense M. García – 19/01/2009 – CERFACS
THE APPLICATION TEST CASES: DECAYING HIT
Illustration of preferential concentrationValidation
of particle kinetic energy results
Performance analysis of
particle location
3224168
15
10
5
0
20x103
29PhD defense M. García – 19/01/2009 – CERFACS
THE APPLICATION TEST CASES: CONFINED BLUFF BODY
Borée, J., Ishima, T. and Flour, I. 2001. The effect of mass loading and inter-particle collisions on the development of the polydispersed two-phase flow downstream of a confined bluff body. J. Fluid Mech., 443, 129-165.
•z (m)
•r (m)
€
R2 =150 mm
€
R j =10 mm
€
U j = 3.4 m /s
€
(max(Ue ) = 6 m /s)
€
U j€
Ue
€
L =1500 mm
€
R1 = 75 mm
€
Q j = 3.4 m3 /h
€
Qe = 780 m3 /h
€
U e = 4.1 m /s
•EDF - R&D
€
ρp = 2470 kg /m3
Description of the configuration
Work published in Journal of Computational Physics:
J. Comput. Phys. Vol. 228, No 2, pp. 539-564 2009“Evaluation of numerical strategies for large eddy simulation of
particulate two-phase recirculating flows”E. Riber, V. Moureau, M. García, T. Poinsot and O. Simonin
30PhD defense M. García – 19/01/2009 – CERFACS
THE APPLICATION TEST CASES: CONFINED BLUFF BODYNumerical parameters
Grid Tetrahedra Hexahedra
Nb cells 2 058 883 3 207 960
Nb nodes 367 313 3 437 576
Nb particles ~ 560 000 ~ 370 000
Time step (ms) 3,2 4,22
LES model Smagorinsky WALE
Turbulence injection on gas No Yes
Turbulence injection on particles
Yes Yes
Wall treatment Law of the wall (Schmitt et al. JFM 2006) No slip
Scheme 3rd order TTGC scheme compressible Two-way coupling Yes
In the following: results of velocity profiles of the polydisperse simulation
At the end of the presentation: study of particle load imbalance
31PhD defense M. García – 19/01/2009 – CERFACS
THE APPLICATION TEST CASES: CONFINED BLUFF BODYAnimation with AVBP-EL: gas velocity modulus with particles
32PhD defense M. García – 19/01/2009 – CERFACS
THE APPLICATION TEST CASES: CONFINED BLUFF BODYParticle trajectories for the polydisperse case
20 microns 40 microns
60 microns 80 microns
Particle trajectories of polydisperse case give expected results, behavior is different
depending on the particle size.
Lighter particles respond to the flow faster. Their trajectories are deviated and more influenced by turbulence.
Heavier particles penetrate more into the recirculation bubble.
33PhD defense M. García – 19/01/2009 – CERFACS
THE APPLICATION TEST CASES: CONFINED BLUFF BODY
z (m)
`r (m
)
20 microns 40 microns
60 microns 80 microns
4 i
nt
12
int
20
int
3 m
m
8 i
nt
16
int
10
int
4 i
nt
12
int
20
int
3 m
m
8 i
nt
16
int
10
int
4 i
nt
12
int
20
int
3 m
m
8 i
nt
16
int
10
int
4 i
nt
12
int
20
int
3 m
m
8 i
nt
16
int
10
int
0.130
0.075
0.010
3 days 12 days 45 days
Cross-section velocity profiles (effect of the No of samples)
34PhD defense M. García – 19/01/2009 – CERFACS
THE APPLICATION TEST CASES: CONFINED BLUFF BODY
20 microns 40 microns
60 microns 80 microns
Axial mean particle velocity profiles: [-2, 6] (m/s)EXP:AVBP_EL: 0.25, 1, 4 (s)
Location of recirculation zone is shifted by a few mm.
35PhD defense M. García – 19/01/2009 – CERFACS
THE APPLICATION TEST CASES: CONFINED BLUFF BODY
20 microns 40 microns
60 microns 80 microns
EXP:AVBP_EL: 0.25, 1, 4 (s)Axial RMS particle velocity profiles: [0.0, 1.5] (m/s)
Minor problem to capture the first stagnation point
36PhD defense M. García – 19/01/2009 – CERFACS
THE APPLICATION TEST CASES: CONFINED BLUFF BODY
20 microns 40 microns
60 microns 80 microns
EXP:AVBP_EL: 0.25, 1, 4 (s)Radial mean particle velocity profiles: [-1, 1] (m/s)
37PhD defense M. García – 19/01/2009 – CERFACS
THE APPLICATION TEST CASES: CONFINED BLUFF BODY
20 microns 40 microns
60 microns 80 microns
EXP:AVBP_EL: 0.25, 1, 4 (s)Radial RMS particle velocity profiles: [0.0, 1.5] (m/s)
38PhD defense M. García – 19/01/2009 – CERFACS
THE APPLICATION TEST CASES: CONFINED BLUFF BODY
050000
100000150000200000250000300000 No cells
Nprocs
Important point to retain of a two-phase Lagrangian parallel simulation Gaseous phase
NOT A GOOD PARALLEL SIMULATION !!
020000400006000080000
100000120000
Nprocs
Gaseous phase Dispersed phaseNo cells
050000
100000150000200000250000300000 No particles
Nprocs
Gaseous phase Dispersed phaseA GOOD PARALLEL LAGRANGIAN SIMULATION !!
05000
10000150002000025000
Nprocs
No particles
020000400006000080000
100000120000
Nprocs
No cells
(B)
(A)
NOT A GOOD PARALLEL LAGRANGIAN SIMULATION !!
39PhD defense M. García – 19/01/2009 – CERFACS
THE APPLICATION TEST CASES: CONFINED BLUFF BODY
050000
100000150000200000250000300000
n particles
Nprocs
05000
10000150002000025000
Nprocs
(A)
(B)
(A)
(B)
Imbalanced simulation
Balanced simulation
Single-constraint (RIB) vs two-constraints (METIS) partitioning algorithm
Mesh
edge-cut
(A)
(B)
RIB
METIS
Load-balancing the disperse phase with a two-constraint partitioning algorithm improves the
performance of the two-phase Lagrangian simulation
40PhD defense M. García – 19/01/2009 – CERFACS
CONCLUSIONS AND PERSPECTIVES
• The effects of rounding errors on the repetitivity of LES was demonstrated and analysed.
• An efficient implementation of a Lagrangian formulation is related to the study of partitioning algorithms, data structure, load-balancing capabilities and parallel facilities, between others.
• The increase of computer power opens a new way for two-phase Lagrangian simulations that were considered prohibitive years ago.
• Validation of the Lagrangian module in an Homogeneous Isotropic Turbulence (HIT) which allows a simple analysis of several aspects of performances and particle behavior.
• A more complete study and validation has been done in a particle-laden bluff-body configuration. Results are in good agreement with experiments. Feasibility demonstrated of load-balancing capabilities.
CONCLUSIONS
41PhD defense M. García – 19/01/2009 – CERFACS
CONCLUSIONS AND PERSPECTIVES
Modeling• Evaporation model (Ph.D. F. Jaegle).• Treatment of particle-wall interactions (Ph.D. F. Jaegle).• Improvement of particle injection (Ph.D. J.M. Senoner + C. Habchi IFP). • Introduction of collision and coalescence models.• Introduction of subgrid-scale fluid velocity on particle components.
PERSPECTIVES
Numerics• Improvement of searching algorithms and data structure.• Improve analysis of current performances: communications, algorithms, memory requirements, etc.
42PhD defense M. García – 19/01/2009 – CERFACS
Thank you for your attention !
Any question ?
43PhD defense M. García – 19/01/2009 – CERFACS
ARRIUS2_44M with RCB: 0.5 [hours] * 4096 [processors] * 0.2 [euros/processor/hour]
= 409.6 euros !!
Need of a new partitioning algorithm:
• Faster partitioning• Less number of total nodes after partitioning• With parallel version• With multi-constraint partitioning options
Choice done: METIS package
THE AVBP SOLVER: PARTITIONING ALGORITHMSEffect of different partitioning algorithms on CPU time
SAME MESH, DIFFERENT ALGORITHMS SAME ALGORITHM, DIFFERENT MESHESRIB
4.7 hours; 4096 procs
44PhD defense M. García – 19/01/2009 – CERFACS
THE AVBP SOLVER: ROUNDING ERRORSThe representation of numbers
B CB+B- C+C-
A
…
A+A-
zoom
(71)10 = 7 x 101 + 1 x 100
(1000111)2 = 1 x 26 + 0 x 25 + 0 x 24 + 0 x 23 + 1 x 22 + 1 x 21 + 1 x 20 decimalbinary
(5.5)10 = (101.1)2 = 1 x 22 + 0 x 21 + 1 x 20 + 1 x 2-1binary (real)
-1-2 0 1 2
1/2-1/2 0.1Numbers represented in a line
A+D=CA+D’=B
45PhD defense M. García – 19/01/2009 – CERFACS
THE APPLICATION TEST CASES: CONFINED BLUFF BODY
050000
100000150000200000250000300000
n particles
Nprocs
05000
10000150002000025000
Nprocs
(A)
(B)
(A)
(B)
Bad speedup
Good speedup
Single-constraint (RIB) vs two-constraints (METIS) partitioning algorithm
Mesh
edge-cut
(A)
(B)
RIB
METIS
Load-balancing the disperse phase with a two-constraint partitioning algorithm improves the
performance of the two-phase Lagrangian simulation