Vous êtes sur la page 1sur 6

Parallelization of a DNS/LES code

for Eulerian/Lagrangian simulations


of turbulent dispersed channel flow
at high Reynolds numbers
CRISTIAN MARCHIOLI
Dipartimento di Energetica e Macchine, Universita di Udine
Abstract. The dynamics of small bubbles or droplets transported by an
incompressible turbulent flow is in general much more complicated than the
motion of a passive tracer (i.e. a small volume element of the fluid itself).
Within this project, we performed Direct Numerical Simulations (DNS) and
Large-Eddy Simulations (LES) of fully-developed turbulent shear flow in a
periodic channel with large swarms of heavy particles. To perform these simula-
tions the DNS/LES code, solving for three-dimensional time-dependent Navier-
Stokes (NS) equations, and the Lagrangian particle tracking routine, solving for
ordinary differential equations (ODEs) for particle velocity and position, have
been parallelized by means of the Message Passing Interface (MPI) paradigm. In
the most common (and simplest) assumption, heavy particles are assumed to
interact with the fluid only through the viscous drag. Here, we have made particle
interaction with the fluid more realistic by including other effects such as those of
gravity and hydrodynamic lift (which are of importance for the case of heavy
particle dispersion) and those of added mass, pressure drag and Basset forces
(which of importance for the case of microbubble dispersion). The database of
trajectories and velocities produced will be used to provide detailed investigation
and validation of the statistical properties (and structural characteristics) of two-
phase flows in wall-bounded turbulence, starting from plane channel flow. It will be
possible to gather a large and accurate set of statistics that will improve the current
understanding of crucial phenomena such as dispersion, preferential distribution,
deposition and resuspension of particles, droplets or microbubbles in the turbulent
boundary layer.
Introduction
Two-phase turbulent flows (gas-particle flows, gas-droplet flows) are of impor-
tance in a number of engineering, industrial and environmental problems such as clean
air technology, mixing and combustion processes and entrainment, saltation and
dispersal of contaminants. An accurate prediction of turbulent flows and particle/
bubble transport processes occurring in such flows is thus important. Yet, prediction is
difficult due to the wide spectrum of length and time scales characterizing turbulence
and to the strong and complex interaction between the continuous fluid phase and the
dispersed phase (particles/bubbles).
In the field of computational fluid dynamics, the direct numerical simulation of
two-phase turbulent flows rank among the applications with the highest demand on
computational power and system resources. In the present project, we have run such
simulations on large-scale High Performance Computing (HPC) infrastructure using
the parallelised version of a sequential DNS solver. High performance computing is
crucial to provide detailed investigation and validation of the statistical properties
(and structural characteristics) of two-phase flows in wall-bounded turbulence. At
our laboratory different projects make use of DNS and LES to study two-phase
turbulent flows, with specific focus on particle/microbubble behavior in the
turbulent boundary layer. In particular, we concentrate on the dipersion of (1) solid
particles flyashes, aerosols in pressure-driven gas flowand (2) non-deformable air
microbubbles in gravity-driven water flow. The reference geometry for these
problems is a flat channel, where the flow is bounded by two infinite parallel walls:
periodic boundary conditions are imposed on the fluid velocity field both in
streamwise and spanwise directions and no-slip boundary conditions are enforced
at the walls.
Numerical Scheme
The flow field is calculated by integrating incompressible Continuity and NS
equations in dimensionless form:
@u
i
@x
i
= 0; (1)
@u
i
@t
= u
j
@u
i
@x
j

1
Re
t
@
2
u
i
@x
j
@x
j

@p
@x
i
d
1;i
; (2)
where u
i
is the i
th
component of the velocity vector, p is the fluctuating kinematic
pressure, d
1;i
is the mean pressure gradient driving the flow, and Re
t
= hu
t
=n is the
shear Reynolds number. Equations (1) and (2) are solved directly using a pseudo-
spectral method [1] based on transforming the field variables into wavenumber space,
through Fourier representations for the streamwise and spanwise directions (using k
1
and k
2
wavenumbers respectively) and a Chebyshev representation for the wall-normal
(non-homogeneous) direction (using n coefficients). A two-level explicit Adams-
Bashforth scheme for the nonlinear terms S
i
and an implicit Crank-Nicolson method
for the viscous terms, are employed for time advancement.
The equation of motion for heavy particles or microbubbles is described by a set of
ordinary differential equations for particle/bubble velocity and position at each time
PARALLELIZATION OF A DNS/LES CODE FOR EULERIAN/LAGRANGIAN ETC. 421
step. In vector form:
dx
dt
= v
p
; (3)
dv
p
dt
= 1
r
r
p
!
g
u v
p
( )
t
p
1 0:15Re
0:687
p

(4)

r
r
p
Du
Dt
C
L
r
r
p
u v
p
( ) v [ [

9m
d
p
r
p

pn
_
Z
t
0
du
dt

dv
p
dt

dt
t t ( )
0:5

r
2r
p
Du
Dt

dv
p
dt

where x is particle position; v is particle velocity; u and v are fluid velocity and
vorticity at particle/bubble position; d
p
and d
p
are particle/bubble diameter and
density. Right hand side terms in Eq. 4 represent the forces per unit of mass acting on
a particle/bubble and describe the effect of gravity, drag, pressure drag, lift, time-
history Basset and added mass force respectively. C
D
=
24
Re
p
(1 0:15 Re
0:687
p
) is the
Stokes drag coefficient, whereas C
L
is the lift coefficient. Both coefficients depend
on the particle Reynolds number Re
p
= d
p
[v u[=n.
The Lagrangian particle tracking routine uses sixth-order Lagrangian polynomials
to interpolate fluid velocities at particle/bubble position; with this velocity the
equations of particle/bubble motion are advanced in time using a fourth-order
Runge-Kutta scheme.
Parallelization
During a previous HPC visit, a parallel version of the numerical code was
developed to perform DNS/LES of channel flow laden with solid particles subject to
inertia, drag and buoyancy under dilute flow conditions (one-way coupling). The
original sequential version of the code is written in Fortran 77 and has more than 10 000
lines. The parallel programming paradigm used is the MPI standard and the paralleliza-
tion is achieved by one-dimensional domain decomposition (see [2] for further details
and for scalability and speed-up tests).
During the present HPC visit, we completed and optimized the parallelised version
of the code to study more complicated (yet more realistic) simulation settings. These
simulation settings now include a larger set of external hydrodynamic forces which
422 CRISTIAN MARCHIOLI
significantly affect the motionof bothsolidparticles andmicrobubbles (e.g. lift, pressure
drag, Basset and added mass). It was not straightforward to include these forces into the
parallel version of the Lagrangian tracking routine, the reason being that accurate
evaluation of fluid velocity time and space derivatives is required. Furthermore, the lift
force has to be modeled and the choice of the model has an impact on the prediction of
particle/bubble behavior: we have implemented a lift force model which was previously
tested in the serial version of the code. Interaction with the local scientific host has been
crucial to accomplish the abovementioned tasks successfully.
The inclusionof the lift, pressure drag, Basset andaddedmass forces intothe parallel
version of the Lagrangian tracking routine is based on the knowledge of the fluid velocity
time and space derivatives in the physical space, which in turn require the knowledge of
the fluid velocity components in the physical space. Since the flow solver works in the
frequency domain, parallelization has been implemented in the following way:
1) The frequency domain has been split among the processes into sub-domains
along either the (wall-normal) z-direction or the (spanwise) y-direction. Specifically: (i)
The Fourier frequency domain corresponds to the x-y domain planes: to perform the
Fast Fourier Transforms (FFTs), a 1D decomposition of the domain is made in the z-
direction so that each processor holds a block of data associated with a number of x-y
planes. All blocks in z-direction are equal in size to ensure an optimal load balancing
over the processors. (ii) The Chebychev frequency domain corresponds to the z
direction: to perform the Discrete Chebychev Transform (DCT), a 1D decomposition
of the domain is made in the y-direction so that each processor holds a block of data
associated with a number of x-z planes. The blocks in z-direction are also equal in size.
Note that all x-y planes, and all x-z planes as well, are independent in the frequency
domain: thus, FFTs and DCT can be easily parallelized without communication
between neighbouring processes.
2) the Chebychev coefficients are transformed from the frequency domain (k
1
, k
2
,
n) to the (k
1
, k
2
, z) domain using an inverse DCT; ii) the (k
1
, k
2
, z) domain is mapped
from the x-z blocks to the x-y blocks through data redistribution; iii) the Fourier
coefficients are transformed from the (k
1
, k
2
, z) domain to the physical domain (x, y, z)
using inverse FFTs. Data redistribution in-between DCT and FFTs is achieved
transposing each data block and using point-to-point communication.
3) The MPI_ISEND, the MPI_IRECV and the MPI_WAITALL, which perform
separate blocking send and receive respectively, are used for local (point-to-point)
communication between processors. Blocking communications assure a tight syn-
chronization between the processes. In the present case, the MPI_ISEND/MPI_IRECV/
MPI_WAITALL calls are combined to perform the same send/receive operations as the
MPI_ISENDRECV, thus removing the risk of deadlock between parallel processes
(that can occur when all processes are trying to receive but no process is sending
anything). Also, one does not have to worry about the buffering of messages when one
process performs a send and the receiving process is not yet ready to receive. In the
present case, using MPI_ALLTOALL (collective communication) would not provide
proper data transposition.
PARALLELIZATION OF A DNS/LES CODE FOR EULERIAN/LAGRANGIAN ETC. 423
Parallelization of the Lagrangian particle tracking routine also accounts for the
non-homogeneity of particle/microbubble concentration distribution in the flow
field [3, 4], and the complex data dependence between the storage of fluid flow data
(distributed among processors) and the local data requirements for the prediction
of particle/bubble motion. The parallel solution algorithm that was adopted
provides, using the MPI_BCAST, the entire fluid flow data to calculate a given
particle/bubble trajectory segment in the local memory of the processor node.
Particle/bubble tracking is then performed by dividing the particle/bubble sample
N
p
into sub-samples of equal size N
p
=N
t
, where N
t
is the total number of processors;
each processor holds one specific sub-sample (namely, each processor tracks all
particles/bubbles from (1 N
p
=N
t
r) to N
p
=N
t
(r 1), where r = 0; :::; N
t
1 is the
rank of the processor).
Acknowledgements. This work was carried out under the HPC-EUROPA project
(RII3-CT-2003-506079), with the support of the European Community Research
Infrastructure Action under the FP6 (Structuring the European Research Area)
Programme.
FIGURE 1. Zoomin the near wall region of a channel cross-section (i.e. plane orthogonal to the
streamwise direction): the black circles indicate bubble positions (only bubbles located at a
distance lower than 10 wall-units from the considered section are considered); the grey scale
map shows the fluid streamwise velocity field in the cross-section. The sketches are referred to
the last time-step of the simulation currently available: a) upflow, cross-section located at
x

=; 800; b) downflow, cross-section located at x

= 1400.
424 CRISTIAN MARCHIOLI
References
[1] S. PEDINOTTI, G. MARIOTTI and S. BANERJEE, Int. J. Multiphase Flow 18, 927-941 (1992).
[2] C. MARCHIOLI, Science and Supercomputing in Europe, Report 2006.
[3] C. MARCHIOLI and A. SOLDATI, J. Fluid Mech., 468, 283-315 (2002).
[4] M. PICCIOTTO, C. MARCHIOLI and A. SOLDATI, Phys. Fluids, 17, 098101 (2005).
PARALLELIZATION OF A DNS/LES CODE FOR EULERIAN/LAGRANGIAN ETC. 425

Vous aimerez peut-être aussi