PhD defense: Marta Garc í a

1PhD defense M. García – 19/01/2009 – CERFACS

PhD defense: Marta García

Development and validation of the Euler-Lagrange formulation on a parallel and

unstructured solver for large-eddy simulation

Director: T. Poinsot & Co-director: V. Moureau


THE CONTEXT

Human nature: try to understand phenomena, comprehension effort, power of fire, energy conversion …

• Observation• Experimentation• Numerical simulation

Gas turbine

engines

fire

locomotive

This thesis is focused on the improvement of current tools to the comprehension of multiphase flows by

using numerical simulation.


THE CONTEXT: EXPERIMENTS vs NUMERICAL SIMULATIONS

EXPERIMENTS

NUMERICAL SIMULATION

• expensive• destructives• difficult to reproduce exactly

• less expensive• not destructives• reproductibility

Ham et al. Annual Research Briefs 2003 CTR Stanford Univ.

Spray evolution from a realistic gas-turbine injector.


THE CONTEXT: INCREASE OF COMPUTER POWER

(1) Flops: floating point operations per second

1.65x1012 Flops(1)

280x1012 Flops

1012 = 1.000.000.000.000

1105x1012 Flops

12.64x1012 FlopsIn the last 3 years CPU time divided by 8 (approx.)

8 weeks 1 week


THE CONTEXT: TWO-PHASE FLOW NUMERICAL SIMULATION

Eulerian formulation

PhDcerfacs2

21

PhDimft1

PhDimft1

up~

TPF Team

Current treatment of the dispersed phase in AVBP

(TPF)

PhDPhD

PhD

PhD

PhD

PhD

PhD

PhD PhD

PhD

PhD

PhD

PhD

PhD

PhD

Lagrangian formulation ?

in 2005 …

PhD

… considering the increasing computer power and my

experience accumulated in the past 3 years …

???Kaufmann 2004

PhD’s: CERFACS & IMFT

Mossa 2005Pascaud 2006

Boileau 2007Riber 2007

Lavedrine 2008 Lamarque 2007

…

PhDcerfacs2


THE CONTEXT: TWO-PHASE FLOW NUMERICAL SIMULATIONEuler-Euler vs Euler-Lagrange

Euler-Lagrange

Individual particle trajectories are computed

+ Easy modeling of particle movements and interactions.+ Robust and accurate if enough particles are used.+ Size distributions easy to describe.+ Easy to implement physical phenomena (e.g. heat and mass transfer, wall-particle interaction).

- Delicate coupling with combustion.- Difficult to run in parallel.- Each ‘particle’ actually represents an ensemble of particles.

Euler-Euler

+ Easy treatment of dense zones.

+ Similarity with gaseous equations.

+ Direct transport of Eulerian quantities.

+ Similarity with gaseous parallelisme.

- Difficult description of polydispersion.- Difficulty of crossing sprays treatment.- Limitation of the method in very dilute zones.

Particle ensemble viewed as a continuous field


THE OBJECTIVES OF THE WORK

• Develop a Lagrangian formulation for two-phase flow treatment within a parallel, unstructured and hybrid solver AVBP.

• Perform the first simulations on academic and complex geometries.

• Verify the efficient parallel implementation to maintain good performance on massively parallel machines.


THE PLAN OF THE PRESENTATION

I. Presentation of the simulation tool: AVBP solver

II. Description of the Lagrangian module

III. Application test cases:

IV. Conclusions and perspectives

• Decaying homogenous isotropic turbulence• Polydisperse two-phase flow of a confined bluff body

• Particle equations of motion• Particle tracking algorithm

• Quick introduction• Domain partitioning• Rounding errors and repetitivity of LES


THE AVBP SOLVER

Parallel solver started in 1993. Unstructured solver capable of handling hybrid grids of different cell types. Computational Fluid Dynamics (CFD) code to solve laminar and turbulent compressible Navier-Stokes equations in 2 and 3 space dimensions. Built upon a modular software library that includes integrated parallel domain partition and data reordering tools, message passing (MPI) and includes supporting routines for dynamic memory allocation, routines for parallel I/O and iterative methods. Written in standard Fortran 77 and C, but it is being upgraded to Fortran 90 in a gradual fashion. Highly portable to different parallel machines.


THE AVBP SOLVER: PARTITIONING ALGORITHMS

RCB

RIB

RGB

R = recursiveB = bisection

C = coordinate

I = inertial

G = graph

… currenly available in AVBP

DUAL MESH

NODAL MESH


THE AVBP SOLVER: PARTITIONING ALGORITHMS

MESH

Total No of nodes

495,232

503,230

530,852

367,313(before partitioning)

(after partitioning)

RCB

RIB

RGB

+ 35%

+ 37%

+ 45%

CPU time of 1000 it. (s)

361.5

366.96

405.64

+ 1.5%

+ 12%

The choice of partitioning algorithm has an effect on the CPU time of your simulation.

Need of a new partitioning algorithm:

• Faster partitioning• Lower number of total nodes after partitioning• With parallel version• With multi-constraint partitioning options

Choice done: METIS package implemented during this thesis


THE AVBP SOLVER: PARTITIONING ALGORITHMSSome results obtained with METIS multilevel partitioning algorithm …

ARRIUS2_10M

ARRIUS2_44M

COMPARISON OF ALGORITHMSNo of nodes after partitioning

No of nodes after partitioning

No of subdomains

No of subdomains

No of subdomains

29 minutes21 minutes

4096 procs

METIS algorithm is faster It produces a lower

number of nodes after partitioning


THE AVBP SOLVER: ROUNDING ERRORS… and repetitivity of LES

MESH

Finite precision computation: lack of associativity property !!

AVBP: Parallel solver, highly portable to solve laminar and turbulent compressible Navier-Stokes equations.

What that means …

€

R1

€

R2

€

R3

€

R4

zoom

€

14

R4 + R3 + R2 + R1( )[ ]{ }RCM

€

R1

€

R2

€

R3

€

R4

CM

RCM

€

14

R4 + R3 + R2 + R1( )[ ]{ }CM

€

≠ABCD

A B C D

Work published in the AIAA Journal publication:

AIAA Journal Vol. 46, No 7, July 2008“Growth of Rounding Errors and Repetitivity of Large-Eddy Simulations”

J.-M. Senoner, M. García, S. Mendez, G. Staffelbach, O. Vermorel and T. Poinsot


THE AVBP SOLVER: ROUNDING ERRORSAxial velocity fields of a turbulent channel (TC) at different instants

(t1)

4 procs

8 procs

Axial velocity (m/s)

(t2)Axial velocity (m/s)(t3)

Axial velocity (m/s)

4 procs

8 procs

4 procs

8 procs

1. Instantaneous solutions in unsteady simulations.

2. Same initial conditions.3. Different number of processors.

DIFFERENCES OBSERVED BETWEEN

TWO SNAPSHOTS

TWO NORMS ARE USED TO COMPARE RESULTS BETWEEN TWO SOLUTIONS


THE AVBP SOLVER: ROUNDING ERRORSDifferent effects observed on repetitivity of LES

Effect of node reordering

Effect of initial conditions

Reprinted by permission of the American Institute of Aeronautics and Astronautics.

Effect of machine precisionquadruple

double

simple

Effect of turbulence

turbulent

laminar

Machine precision differences

Norm saturation

Any sufficiently turbulent flow computed in LES exhibits significant sensitivity to small perturbations, leading to instantaneous solutions which can be totally different.

The divergence of solutions is due to 2 combined facts:

1. The exponential separation of trajectories in turbulent flows.

2. The different propagation of rounding errors induced by domain partitioning and scheduling operations.The validation of an LES code after modifications may

only be based on statistical fields.











THE LAGRANGIAN MODULE: PARTICLE EQUATIONS… of motion

Individual particle trajectories are computed with a Lagrangian solver coupled to the LES code for the gas phase.

N droplets to track (order of a few millions)

€

dx p,i

dt= up,i

€

dup,i

dt= − 3

4ρ g

ρ p

CD

dp

vr vr,i + gi = −up ,i − ˜ u g,i

τ p

+ gi

€

τ p = 34

ρ p

ρ g

dp

CD vr

€

with

[ Schiller & Nauman. 1935 ]

€

ρp >> ρ g

Assumptions:

spheres

€

CD

€

vr,i = up .i − ˜ u g,i

€

and

Particles equation of motion

Need to know the gas velocity at each particle location (linear interpolation)

€

˜ u g,i = ˆ u g ,i

The effect of the subgrid fluid velocity is not considered in this thesis.

[ Fede & Simonin 2006 ]

drag + gravity


THE LAGRANGIAN MODULE: KEY POINTS

X3

X1

X2

n1

n2

n3

Xp

( Xp-Xi ) ni ≥ 0

Locating particles in cellsKnowing particles positions at time n: exchange particles between processors

cell i

particle

Subdomain 1 Subdomain 2

influence node

Two-way coupling

Particles load-balancing

Interpolation algorithm

€

dx p,i

dt= up,i

€

τ p =dp

2ρ p

18μg

€

with

€

dup,i

dt= −

up,i − ˜ u g,i

τ p

+ gi

gas velocity ug,i at each particle location

Injection…

Particle-wall treatment

… for Lagrangian schemes in unstructured meshes


THE LAGRANGIAN MODULE: LOCATING PARTICLES

Shape functions:

€

ρ x p = N i r x i

i∑

€

N i =1i

∑

€

ρ x p =

r X

r N →

r N =

r X −1r

x p

€

min(N i,1− N i) ≥ 0, ∀ i

€

Vii=1,nv∑ = Vc

Calculation of partial volumes:… in elements of arbitrary shape 2D:

3D:

To decide if the particle is in the cell or not, the scalar product between the vector starting from the vertex of the cell to the particle and the inward normal

vector of the corresponding edge is taken. The particle is inside the cell if all the scalar products of each edge are positive.

X1

X2

n1

n2

n3

Xp

Xp•

X3

n3

-

+n3

( Xp-Xi ) ni ≥ 0

Face-normals:


THE LAGRANGIAN MODULE: SEARCH ALGORITHMS… for different situations

Search particles for the first time Search injected particles

Search particles during simulation Search particles crossing boundaries between processors

Cells of the interface (type 2)

Initial particle location

New particle location

Interface between processors

Old cell containing the particle

Cells surrounding the old cell

Initial particle location

Nodes of the containing cell

New particle location

Cells of the injection area

Particles injected

Interface between processors

Injection area

Nodes of the injection area

Quad/Octree

F. Collino(CERFACS)Use of different search algorithms depending on

the situation to reduce memory and CPU time requirements.


THE LAGRANGIAN MODULE: INTERPOLATION

∏≠= −

−=

n

imm mi

mLi xx

xxxP1

)(

Ex. 3D with hexa: n=2 (trilinear interpolation)

)()()(),,(),,(111

zPyPxPzyxfzyxf Lk

Lj

Likji

n

i

n

j

n

k∑∑∑===

=

Lagrange interpolation (only for coordinate grids with quads or hexahedras)

1 2

2

2

Linear Least Squares (LAPACK subroutine DGELS)

1st order Taylor Serie

€

f (x) = f (n )(a)n!n= 0

∞

∑ (x − a)n

€

f (x) = f (a) + f '(a)(x − a)

Ex. 1D and 1st order

… of gaseous-phase properties at particle position


THE LAGRANGIAN MODULE: TWO-WAY COUPLING

€

fc,i = Coupling force

€

Vfc,iP(x,y,z ) = α proj( fd ,i

n ,P(x,y,z))n in V∑

1 2 3

4 6

7 8 9

€

α = constant of proportionality

Exist an analytical solution

Validation test of two-way coupling

Momentum eq. = cte

[ Boivin et al. 1998, Boivin et al. 2000 ] [ Ph.D. O. Vermorel 2003 ]

x

y

€

up (0) = cte

€

ug (0) = 0

In the framework of PIC methods

… source terms and validation


THE LAGRANGIAN MODULE: PARTICLE INJECTION

INJECTION GEOMETRY: simple injection options available• Point injection: all droplets are injected at the same

point.• Disk injection: droplets are injected over a disk.Example of input parameters:

Coordinates of the injection point, disk diameter, normal to define disk direction, tolerance …

PARTICLE SIZE DISTRIBUTION:• Monodisperse: all particles have the same diameter.• Polydisperse: different particle diameters.

1. Gaussian distribution2. Log-normal distribution

Example of input parameters: Type of distribution, maximum and minimum diameters, mean and standard deviation

…


THE LAGRANGIAN MODULE: PARTICLE INJECTION

- Particle mass flow rate

- Particle diameter(s), density, mean/rms velocity ...

ZOOM Injection tube

z=-3mm

Inject # particles by timestep

… example of a disk injection











THE APPLICATION TEST CASES

Decaying Homogeneous Isotropic turbulence (HIT)

Polydisperse two-phase flow of a confined bluff body

• Academic test case• Well documented

• Complex recirculating flow

• Large amount of data available


THE APPLICATION TEST CASES: DECAYING HIT

Simple configuration to:• Validate first developments of the Lagrangian version.• Localisation algorithms, interpolation, processor exchanges,

etc.


THE APPLICATION TEST CASES: DECAYING HIT

Illustration of preferential concentrationValidation

of particle kinetic energy results

Performance analysis of

particle location

3224168

15

10

5

0

20x103


THE APPLICATION TEST CASES: CONFINED BLUFF BODY

Borée, J., Ishima, T. and Flour, I. 2001. The effect of mass loading and inter-particle collisions on the development of the polydispersed two-phase flow downstream of a confined bluff body. J. Fluid Mech., 443, 129-165.

•z (m)

•r (m)

€

R2 =150 mm

€

R j =10 mm

€

U j = 3.4 m /s

€

(max(Ue ) = 6 m /s)

€

U j€

Ue

€

L =1500 mm

€

R1 = 75 mm

€

Q j = 3.4 m3 /h

€

Qe = 780 m3 /h

€

U e = 4.1 m /s

•EDF - R&D

€

ρp = 2470 kg /m3

Description of the configuration

Work published in Journal of Computational Physics:

J. Comput. Phys. Vol. 228, No 2, pp. 539-564 2009“Evaluation of numerical strategies for large eddy simulation of

particulate two-phase recirculating flows”E. Riber, V. Moureau, M. García, T. Poinsot and O. Simonin


THE APPLICATION TEST CASES: CONFINED BLUFF BODYNumerical parameters

Grid Tetrahedra Hexahedra

Nb cells 2 058 883 3 207 960

Nb nodes 367 313 3 437 576

Nb particles ~ 560 000 ~ 370 000

Time step (ms) 3,2 4,22

LES model Smagorinsky WALE

Turbulence injection on gas No Yes

Turbulence injection on particles

Yes Yes

Wall treatment Law of the wall (Schmitt et al. JFM 2006) No slip

Scheme 3rd order TTGC scheme compressible Two-way coupling Yes

In the following: results of velocity profiles of the polydisperse simulation

At the end of the presentation: study of particle load imbalance


THE APPLICATION TEST CASES: CONFINED BLUFF BODYAnimation with AVBP-EL: gas velocity modulus with particles


THE APPLICATION TEST CASES: CONFINED BLUFF BODYParticle trajectories for the polydisperse case

20 microns 40 microns


Particle trajectories of polydisperse case give expected results, behavior is different

depending on the particle size.

Lighter particles respond to the flow faster. Their trajectories are deviated and more influenced by turbulence.

Heavier particles penetrate more into the recirculation bubble.



z (m)

`r (m

)



4 i

nt

12

int

20

int

3 m

m

8 i

nt

16

int

10

int

4 i

nt

12

int

20

int

3 m

m

8 i

nt

16

int

10

int

4 i

nt

12

int

20

int

3 m

m

8 i

nt

16

int

10

int

4 i

nt

12

int

20

int

3 m

m

8 i

nt

16

int

10

int

0.130

0.075

0.010

3 days 12 days 45 days

Cross-section velocity profiles (effect of the No of samples)





Axial mean particle velocity profiles: [-2, 6] (m/s)EXP:AVBP_EL: 0.25, 1, 4 (s)

Location of recirculation zone is shifted by a few mm.





EXP:AVBP_EL: 0.25, 1, 4 (s)Axial RMS particle velocity profiles: [0.0, 1.5] (m/s)

Minor problem to capture the first stagnation point





EXP:AVBP_EL: 0.25, 1, 4 (s)Radial mean particle velocity profiles: [-1, 1] (m/s)





EXP:AVBP_EL: 0.25, 1, 4 (s)Radial RMS particle velocity profiles: [0.0, 1.5] (m/s)



050000

100000150000200000250000300000 No cells

Nprocs

Important point to retain of a two-phase Lagrangian parallel simulation Gaseous phase

NOT A GOOD PARALLEL SIMULATION !!

020000400006000080000

100000120000

Nprocs

Gaseous phase Dispersed phaseNo cells

050000

100000150000200000250000300000 No particles

Nprocs

Gaseous phase Dispersed phaseA GOOD PARALLEL LAGRANGIAN SIMULATION !!

05000

10000150002000025000

Nprocs

No particles

020000400006000080000

100000120000

Nprocs

No cells

(B)

(A)

NOT A GOOD PARALLEL LAGRANGIAN SIMULATION !!



050000

100000150000200000250000300000

n particles

Nprocs

05000

10000150002000025000

Nprocs

(A)

(B)

(A)

(B)

Imbalanced simulation

Balanced simulation

Single-constraint (RIB) vs two-constraints (METIS) partitioning algorithm

Mesh

edge-cut

(A)

(B)

RIB

METIS

Load-balancing the disperse phase with a two-constraint partitioning algorithm improves the

performance of the two-phase Lagrangian simulation


CONCLUSIONS AND PERSPECTIVES

• The effects of rounding errors on the repetitivity of LES was demonstrated and analysed.

• An efficient implementation of a Lagrangian formulation is related to the study of partitioning algorithms, data structure, load-balancing capabilities and parallel facilities, between others.

• The increase of computer power opens a new way for two-phase Lagrangian simulations that were considered prohibitive years ago.

• Validation of the Lagrangian module in an Homogeneous Isotropic Turbulence (HIT) which allows a simple analysis of several aspects of performances and particle behavior.

• A more complete study and validation has been done in a particle-laden bluff-body configuration. Results are in good agreement with experiments. Feasibility demonstrated of load-balancing capabilities.

CONCLUSIONS


CONCLUSIONS AND PERSPECTIVES

Modeling• Evaporation model (Ph.D. F. Jaegle).• Treatment of particle-wall interactions (Ph.D. F. Jaegle).• Improvement of particle injection (Ph.D. J.M. Senoner + C. Habchi IFP). • Introduction of collision and coalescence models.• Introduction of subgrid-scale fluid velocity on particle components.

PERSPECTIVES

Numerics• Improvement of searching algorithms and data structure.• Improve analysis of current performances: communications, algorithms, memory requirements, etc.


Thank you for your attention !

Any question ?


ARRIUS2_44M with RCB: 0.5 [hours] * 4096 [processors] * 0.2 [euros/processor/hour]

= 409.6 euros !!

Need of a new partitioning algorithm:

• Faster partitioning• Less number of total nodes after partitioning• With parallel version• With multi-constraint partitioning options

Choice done: METIS package

THE AVBP SOLVER: PARTITIONING ALGORITHMSEffect of different partitioning algorithms on CPU time

SAME MESH, DIFFERENT ALGORITHMS SAME ALGORITHM, DIFFERENT MESHESRIB

4.7 hours; 4096 procs


THE AVBP SOLVER: ROUNDING ERRORSThe representation of numbers

B CB+B- C+C-

A

…

A+A-

zoom

(71)10 = 7 x 101 + 1 x 100

(1000111)2 = 1 x 26 + 0 x 25 + 0 x 24 + 0 x 23 + 1 x 22 + 1 x 21 + 1 x 20 decimalbinary

(5.5)10 = (101.1)2 = 1 x 22 + 0 x 21 + 1 x 20 + 1 x 2-1binary (real)

-1-2 0 1 2

1/2-1/2 0.1Numbers represented in a line

A+D=CA+D’=B



050000

100000150000200000250000300000

n particles

Nprocs

05000

10000150002000025000

Nprocs

(A)

(B)

(A)

(B)

Bad speedup

Good speedup

Single-constraint (RIB) vs two-constraints (METIS) partitioning algorithm

Mesh

edge-cut

(A)

(B)

RIB

METIS

Load-balancing the disperse phase with a two-constraint partitioning algorithm improves the

performance of the two-phase Lagrangian simulation

Documents

PhD defense: Marta Garc í a