PNAM: parallel software for air quality simulations in
the Naples area
Pasqua D'Ambra
Center for Research on Parallel Computing and Supercomputers (CPS-CNR),
Naples, Italy
Guido Barone
University of Naples ``Federico II'' and CPS-CNR, Naples, Italy
Daniela di Serafino
The Second University of Naples, Caserta and CPS-CNR, Naples, Italy
Giulio Giunta
Naval University of Naples and CPS-CNR, Naples, Italy
Almerico Murli
University of Naples ``Federico II' and CPS-CNR, Naples, Italy
Angelo Riccio
University of Naples ``Federico II'' and CPS-CNR, Naples, Italy
1. Introduction
The numerical solution of air quality models (AQMs) plays a key role in studying the atmospheric chemistry, e.g. to understand the relations between anthropogenic emissions and distributions of primary and secondary pollutants, and hence in planning and evaluating air pollution control
strategies. Conversely, the effective treat-ment of AQMs, where physical and chemical processes are adequately described, requires fast and reliable numerical algorithms and software that exploit the great potential power of parallel and distributed computers (Blomet al., 1998; Brownet al., 1995; Bruegge et al., 1995; Dabdub and Seinfeld, 1996; Dabdub and Manohar, 1997; Elbern, 1997, Kessleret al., 1995).
The parallel Naples airshed model (PNAM) is a prototypal software package for the numerical simulation of air pollution episodes in urban scale domains, on MIMD distributed-memory parallel machines. This is a first result of a research activity at the Center for Research on Parallel Computing and Supercomputers (CPS-CNR), aimed at developing operational software for air qual-ity simulations in the Naples area, and more generally in southern Italy (Baroneet al., 1998a, 1998b, 1998c, 1998d; D'Ambraet al., 1997).
In this paper we give a general description of PNAM (Section 2) and discuss some results
of the simulation of a severe photosmog episode which occurred in the Naples area (Section 3). In particular, we perform a comparison between simulated and mea-sured data, as a first step toward the valida-tion of our model (Secvalida-tion 3.1), and show also some results concerning its parallel perfor-mance (Section 3.2). Finally, we report a few conclusions (Section 4).
2. The parallel Naples airshed
model
2.1 Model description
In PNAM the chemical transport is modeled by the following system of atmospheric advection-diffusion-reaction equations:
concentrationsci t;xofnchemical species,
u t;xis the wind velocity field,K t;zis the
eddy diffusivity coefficient driving the ver-tical diffusion,Riis the reaction term, and
x x;y;z. In the above equations the fields
describing the atmospheric weather phe-nomena are considered known, i.e. supplied by an off-line circulation model.
The chemical kinetics is modeled as in the CIT photochemical model (Harleyet al., 1993). This is based on the LCC/SAPRC kinetic mechanism (Lurmannet al., 1987), involving 107 gas-phase reactions among 42 species.
The current issue and full text archive of this journal is available at http://www.emerald-library.com
Environmental Management and Health
10/4 [1999] 209±215
#MCB University Press [ISSN 0956-6163] Keywords Models, Simulation, Parallel computing
Abstract
The parallel Naples airshed model (PNAM) is a parallel software package for the numerical simula-tion of photosmog episodes in urban scale domains. It solves the at-mospheric diffusion equations, which model the air pollution dynamics in a Eulerian approach, using a symmetric time-splitting, where the advection is separated from the (coupled) diffusion and chemistry. Presents some results of a numerical simulation of a severe photochemical smog episode, which occurred in the Naples area. A preliminary comparison with mea-sured data is reported, as a first step toward the validation of PNAM. Some parallel performance results, obtained on an IBM SP machine, are also shown.
The numerical solution of system (1) is based on a symmetric time-splitting techni-que that decouples the horizontal transport operator (LT) from the vertical diffusion
and chemistry operator (LDC), as in many advanced AQMs (see, for example, Bruegge et al., 1995; Harleyet al.,1993; Moussiopoulos et al., 1995):
c tt LT t=2LDC tLA t=2c t: 2
The equations are discretized on a three-dimensional rectangular grid, with uniform spacing in the horizontal directions and with variable spacing in the vertical direction, so that the region close to the terrain can be resolved accurately. A terrain-influenced z-co-ordinate transformation, with a one-dimensional stretching function, is used in the vertical direction. The solution procedure is based on the method of lines and the spatial discretization of the equations is performed using cell-centered finite-difference schemes.
The advective transport is treated using a third-order upwind positive and mass-con-servative scheme with flux limiting in the horizontal directions (Hundsdorferet al., 1995) and a first-order upwind in the vertical direction. Lateral boundary conditions of inflow type are computed by constant extra-polation, in order to avoid the arising of not real extrema in the solution, while outflow boundary conditions are obtained by a second-order extrapolation. At the ground level, the wind velocity is set to zero. This allows us to keep the concentrations on the bottom constant, during the treatment of the vertical advection. A zero-gradient boundary condition is considered at the top of the computational domain, i.e. constant extrapo-lation is used. The ODEs arising from the semi-discretization of the advective transport are solved using a two-stage second-order explicit Runge-Kutta method (Heun).
The vertical diffusion is semi-discretized with a second-order finite-difference scheme, and the diffusion-chemistry ODEs are treated implicitly using the general-purpose VODE package (Brownet al., 1989), modified to exploit the sparsity of the linear systems in the solution procedure. For details see Bar-oneet al.(1998b). We use the following ground level boundary condition:
ÿK t;0@ci
@z t;0 Ei t ÿDi t;0; i1;. . .;n;
whereKis the turbulent diffusivity coeffi-cient,EiandDiare emission and dry deposition terms, respectively. We do not consider wet deposition in our model, since we are interested in the simulation of photo-smog episodes related to dry periods. The
dry deposition term is based on a three-resistance approach, to account for the transport through the boundary layer and the laminar sublayer and for pollutant-surface interactions (McRaeet al., 1982). The emissions[1] are modeled using data from the EC CorinAir archive; some details on the methodology used to realize an emission basecase for the photosmog episode consid-ered in this paper can be found in Barone et al.(1998c). The above bottom boundary condition and a zero-gradient top boundary condition are discretized using first-order approximations.
2.2 Parallel implementation
The parallelism has been introduced decom-posing the computational domain into verti-cal air columns, i.e. partitioning the three-dimensional grid along thexandy direc-tions, and mapping the subdomains onto a two-dimensional logical grid of processors. This choice is a natural approach in atmo-spheric modeling (Dabdub and Seinfeld, 1996; Dabdub and Manohar, 1997; Elbern, 1997). The coupled solution of vertical diffusion and chemistry introduces a global coupling among points in the vertical direction, while requiring no interactions between different vertical columns. Therefore, no data com-munication must be performed during the diffusion-chemistry steps. Moreover, this data decomposition allows the use of avail-able efficient and reliavail-able sequential soft-ware, such as VODE, to solve the stiff diffusion-chemistry ODEs. Nearest-neigh-bour communications occur only to exchange subdomain boundary data during the advec-tion steps. Furthermore, global communica-tions are performed each time the
meteorological data are read, to compute the maximum advection step allowed by the CFL condition.
Numerical experiments have shown that a uniform and static domain decomposition does not lead to load balanced executions, and hence degrades the parallel performance (Baroneet al., 1998d). The main source of load imbalance is the stiffness of the chemical kinetics, which varies in time and space, and the use of a variable-step stiff ODE solver in different vertical columns.
The use of dynamic load balancing in PNAM is currently under experiment. A dynamic load balancing strategy, similar to that used in the MM90 software (Michalakes, 1997), has been implemented in PNAM. A global-decision/local-migration algorithm is applied to periodically redistribute vertical columns among the processors, i.e. the decision of remapping the computational grid onto the processor grid is based on a Pasqua D'Ambra,
Guido Barone, Daniela di Serafino, Giulio Giunta, Almerico Murli and Angelo Riccio
PNAM: parallel software for air quality simulations in the Naples area
Environmental Management and Health
measure of the workload of all the proces-sors, but the columns are moved only among neighbouring processors. A description of the algorithm can be found in Baroneet al. (1998d).
PNAM has been implemented in Fortran 90, using the Runtime System Library (RSL) developed at Argonne National Laboratory, which provides high-level routines to perform domain decomposition, nearest-neighbor and global communication, dynamic load balancing, and also grid re-finement and irregularly-shaped nesting (Michalakes, 1994). In PNAM the size of the
x; y-domain is specified by giving the number of horizontal rows and columns, and the RSL default algorithm is used to divide the domain into subdomains of about the same dimensions. RSL high-level communi-cation routines are used to execute data exchanges required to update ghost-bound-ary values and data redistributions needed by the dynamic load balancing strategy.
3. Application of PNAM to a
photosmog episode
PNAM has been applied to a photosmog episode which occurred on 26 July 1995, in the Campania region. On this day, owing to the stagnant conditions and the intensive solar radiation, a high ozone concentration (180ppb) was reported for the Naples basin.
The modeled domain is centered at the Naples Gulf and extends about 384384km2,
hence including the Campania region and some surrounding areas (a large part of this domain is shown in Figure 1). A maximum height of 2,360m is considered. Details on the topographical features and on the related air quality situation are given in Baroneet al. (1998c).
The domain has been discretized using a grid with 646416 cells, with a spacing of 6km2in thexandydirections, and a spacing
ranging from 10m at the ground level to 640m at the upper level in thezdirection.
Meteorological data have been generated by the topographic vorticity-mode mesoscale (TVM) model, a three-dimensional and non-hydrostatic atmospheric circulation model (Bornsteinet al., 1996; Schayeset al., 1996). TVM has been run, using the above grid, off-line, i.e. hourly two- and three-dimensional fields have been first generated and then supplied to PNAM. To initialize TVM, synoptic data from the European Centre for Medium Range Weather Forecasting (ECMWF) of Reading (UK) have been used. Topographic data have been obtained from the National Oceanographic and
Atmospheric Administration (NOAA), and the land-use field has been extracted from the Italian Military Geographic Institute (IGM) archive. Finally, initial concentrations have been obtained interpolating data available from the monitoring network, Monitoraggio Ambientale Regione Campania (MARC), op-erated by the Campania Regional Board. Some details on the data setup are given in Baroneet al.(1998c).
A start-up time of 24 hours has been used to obtain reasonable input data for the pollution episode under consideration. A splitting interval of 15 minutes has been chosen. The meteorological data have been read every hour and kept constant during the hour.
3.1 Comparison between simulated and measured data
We compare the computed data with the data measured by five monitoring stations of the MARC network, located in the urban areas of Naples, Caserta, Salerno, Benevento and Avellino (these stations are identified by the bullets in Figure 1). Daily emissions of atmospheric pollutants for 26 July 1995 are given in Table I.
Observed and computed concentrations of O3, NO, NO2and CO at the surface level are
reported in Figure 2. The frequency distri-butions of the corresponding residuals, i.e. of the difference between observed and
computed concentrations, are shown in Figure 3, and the mean and the standard deviation are summarized in Table II. Note that the data from all the monitoring stations have been aggregated.
Accumulation processes in the night and emissions, which become relevant even at dawn, are responsible for the local maxima of CO and NO in the early morning. After that time, the concentrations of CO and NO undergo a decrease during the day; this is mainly due to the influence of the transport processes (high mixing height and wind velocity) for the slightly reactive CO, and to the rapid oxidation processes for the highly photochemical reacting NO. The O3
concen-tration grows up steadily to a maximum at midday. The maximum daytime NO2value
depends on photochemical production and occurs almost simultaneously with the CO peak.
The previous results show that the model underestimates the global production of O3
and NO2, mainly in the high polluted
down-town Naples area. For the NO and CO values, the disagreement of the prediction is reason-able and related to the inaccuracy of the input data. However, note that the disagree-ment between predicted and measured values is contained, in about 75 percent of the cases, Pasqua D'Ambra,
Guido Barone, Daniela di Serafino, Giulio Giunta, Almerico Murli and Angelo Riccio
PNAM: parallel software for air quality simulations in the Naples area
Environmental Management and Health
in an interval of width 25ppb, for O3, NO, NO2
and 1ppb for CO. The analysis of the sample cross-correlation coefficients at various lags (Table III), as far as the accuracy of the prediction in both the maximum concentra-tions values and the time of their occur-rence[2] (Tables IV and V) is concerned,
Figure 2
Measured data (dotted line) and computed data (solid line)
Figure 1
Topography of the modeled domain
Table I
Daily emissions (tonnes/day) of atmospheric pollutants from the Campania provinces on 26 July 1995
NOx NMHC CO
155.0 113.3 1,406.3
Pasqua D'Ambra, Guido Barone, Daniela di Serafino, Giulio Giunta, Almerico Murli and Angelo Riccio
PNAM: parallel software for air quality simulations in the Naples area
Environmental Management and Health
suggests that we have a good undelayed prediction of the ozone diurnal variation on the whole domain. The significant inaccura-cies and the temporal bias in the prediction of the other species, mainly NO2, are at
present unexplained and deserve a more careful tuning of the model.
3.2 Some performance results Numerical experiments to evaluate the parallel performance of PNAM have been carried out on an IBM SP machine at CPS-CNR, using the previous air pollution episode. This machine has 12 Power2 Super Chip thin nodes (160MHz), each with 512MBytes of memory; the nodes are con-nected via an SP switch, with a peak bi-directional bandwidth of 110MBytes/sec.
The available Fortran 90 compiler is XL Fortran, version 4.1, and the RSL parallel environment is implemented on the top of the IBM proprietary version 2.3 of MPI.
Table VI shows the execution times, in seconds, the speed-up and the efficiency of PNAM on 1, 2, 4, 8 and 12 processors[3]. We note that, in going from 1 to 12 processors, the execution time reduces from 13.17 hours to 1.80 hours; the corresponding speed-up is more than 7, with a parallel efficiency of about 60 percent. Higher efficiency values are obtained on 2, 4 and 8 processors.
A detailed analysis of the effects of the workload distribution on the parallel perfor-mance and of the results obtained with and without the application of dynamic load balancing is reported in Baroneet al.(1998d).
Figure 3
Frequency distribution of residuals
Pasqua D'Ambra, Guido Barone, Daniela di Serafino, Giulio Giunta, Almerico Murli and Angelo Riccio
PNAM: parallel software for air quality simulations in the Naples area
Environmental Management and Health
4. Concluding remarks
The simulation of an air pollution episode which occurred in the Naples basin has been performed with the PNAM parallel air quality system. The results of a comparison between measured and computed data seem
promising, although some discrepancies have been observed. However, a further analysis is needed, and PNAM will be applied to other air pollution episodes. Aerosol microphysics and chemistry will also have to be considered in the future.
The parallel performance of PNAM appears to be satisfactory. Future work will be devoted to experimenting with load bal-ancing strategies and to testing PNAM on other parallel machines.
Notes
1 They could have been equivalently in-cluded in the differential equation (1) as a source term in the ground level cells. 2 These parameters are defined respectively
as:
maxk cMEAS k
maxk cCOMP k andt
MEAS ÿtCOMP; where
cMEAS kandcCOMP k
are the measured and the computed local values of the concentrationcin thek-th time interval, andtMEASandtCOMPare the time intervals where the maximum values ofcMEAS(k)andcCOMP(k)are reached.
3 The speed-up and the efficiency onp processors are defined respectively as
Sp
whereTiis the execution time oni
processors.
References
Barone, G., D'Ambra, P., di Serafino, D., Giunta, G. and Riccio, A. (1998a), ``Numerical simulation of air pollution phenomena in the Neapolitan urban area (southern Italy): first experiences'', in Griebel, M., Iliev, O.P., Margenov, P.O. and Vassilevski, P.S. (Eds), Large-Scale Computations of Engineering and Environmental Problems, Vol. 62, Kluwer, pp. 128-35.
Barone, G., D'Ambra, P., di Serafino, D., Giunta, G. and Riccio, A. (1998b), ``A comparison of numerical methods for solving diffusion-re-action equations in air quality models'', Center for Research on Parallel Computing and Supercomputers (CPS-CNR), Naples, Italy.
Barone, G., D'Ambra, P., di Serafino, D., Giunta, G., Murli, A. and Riccio, A. (1998c), ``Application of a parallel photochemical air quality model to the Campania Region (southern Italy)'',Proceedings of the Interna-tional Conference on Air Pollution Modelling
Table IV
Accuracy of the prediction of the maximum concentrations
O3 NO NO2 CO
Naples 1.6 1.4 1.1 1.3
Caserta 1.3 2.8 1.4 1.4
Avellino ± 2.5 1.5 1.7
Benevento ± 2.0 1.2 1.8
Salerno ± 1.7 1.3 1.0
Table V
Accuracy of the prediction of the time of occurrence of the maximum concentrations (hours)
Execution times (seconds), speed-up, and efficiency of PNAM
Procs 1 2 4 8 12
Time 47,415 25,987 14,387 8,639 6,479
Speed-up ± 1.82 3.30 5.49 7.32
Efficiency ± 0.91 0.82 0.69 0.61
Table II Daniela di Serafino, Giulio Giunta, Almerico Murli and Angelo Riccio
PNAM: parallel software for air quality simulations in the Naples area
Environmental Management and Health
and Simulation, APMS '98, Champs-sur-Marne, Paris, 26-29 October.
Barone, G., D'Ambra, P., di Serafino, D., Giunta, G., Murli, A. and Riccio, A. (1998d), ``Parallel numerical simulation of air pollu-tion in southern Italy'', in Zlatev, Z. (Ed.), Large-Scale Computations in Air Pollution Modelling, Kluwer.
Blom, J.G., Lioen, W.M. and Verwer, J.G. (1998), ``HPCN and air quality modelling'', CWI Tech. Rep. MAS-R9801, Centrum voor Wiskunde en Informatica, Amsterdam, The Netherlands. Bornstein, R., Thunis, P., Grossi, P. and
Schayes, P. (1996), ``Topographic vorticity-mode mesoscale ±b(TVM) model. Part II: evaluation'',J. Appl. Meteor., Vol. 35, pp. 1824-34.
Brown, J., Wasniewski, J. and Zlatev, Z. (1995), ``Running air pollution models on massively parallel machines'',Parallel Computing, Vol. 21, pp. 971-91.
Brown, P.N., Byrne, G.D. and Hyndmarsh, A.C. (1989), ``VODE: a variable coefficient ODE solver'',SIAM J. Sci. Stat. Comput., Vol. 10, pp. 1038-51.
Bruegge, B., Riedel, E., Russell, A., Segall, E. and Steenkiste, P. (1995), ``Heterogeneous distrib-uted environmental modelling'',SIAM News, No. 28.
Dabdub, D. and Manohar, R. (1997), ``Performance and portability of an air quality model'', Parallel Computing, No. 23, pp. 2187-200. Dabdub, D. and Seinfeld, J.H. (1996), ``Parallel
computation in atmospheric chemical model-ling'',Parallel Computing, No. 22, pp. 111-30. D'Ambra, P., di Serafino, D., Giunta, G. and
Riccio, A. (1997), ``Parallel numerical simula-tions of reacting flows in air quality models'', in Schiano, P., Ecer, A., Periaux, J. and Safotuka, N. (Eds),Parallel Computational Fluid Dynamics, Algorithms and Results Using Advanced Computers, Elsevier Applied Science, Barking, Essex, and Amsterdam, pp. 116-23.
Elbern, H. (1997), ``Parallelization and load balancing of a comprehensive atmospheric chemistry transport model'',Atmos. Environ., No. 31, pp. 3561-74.
Harley, R.A., Russell, A.G., McRae, G.J., Cass, G.R. and Seinfeld, J.H. (1993), ``Photo-chemical modelling of the southern Califor-nia air quality study'',Environ. Sci. Technol., No. 27, pp. 378-88.
Hunsdorfer, W., Koren, B., van Loon, M. and Verwer, J.W. (1995), ``A positive finite-difference advection scheme'',J. of Comput. Phys., No. 117, pp. 35-46.
Kessler, C., Blom, J.G. and Verwer, J.G. (1995), ``Porting a 3D-model for the transport of reactive air pollutants to the parallel machine T3D'',CWI Tech. Rep. NM-R9519, Centrum voor Wiskunde en Informatica, Amsterdam, The Netherlands.
Lurmann, F.W., Carter, W.P.L. and Coyner, L.A. (1987), ``A surrogate species chemical reaction mechanism for urban-scale air quality simu-lation models. I. Adaptation of the mechan-ism. II. Guidelines for using the mechanism'', Tech. Rep. EPA/600/3-87/014, US Environ-mental Protection Agency and Statewide Air Pollution Research Center.
McRae, G.J., Goodwin, W.R. and Seinfeld, J.H. (1982),Mathematical Modeling of Photo-chemical Air Pollution, Environmental Quality Laboratory 18, California Institute of Technology, Pasadena, CA.
Michalakes, J. (1994), ``A parallel runtime system library for regular grid finite difference models using multiple nests'',Tech. Rep. ANL/MCS-TM-197, Mathematics and Com-puter Science Division, Argonne National Laboratory, Argonne, IL.
Michalakes, J. (1997), ``MM90: a scalable parallel implementation of the Penn State/NCAR mesoscale model (MM5)'',Parallel Comput-ing, No. 23, pp. 2173-86.
Moussiopoulos, N., Sahm, P., Kessler, C.H. (1995), ``Numerical simulation of photochemical smog formation in Athens, Greece ± a case study'',Atmos. Environ., No. 29, pp. 3619-32. Schayes, G., Thunis, P. and Bornstein, R. (1996),
``Topographic vorticity-mode mesoscale ±b (TVM) model. Part I: formulation'',J. Appl. Meteor., No. 35, pp. 1815-23.
Pasqua D'Ambra, Guido Barone, Daniela di Serafino, Giulio Giunta, Almerico Murli and Angelo Riccio
PNAM: parallel software for air quality simulations in the Naples area
Environmental Management and Health
