Vous êtes sur la page 1sur 8

2007 IEEE Asia-Pacific Services Computing Conference

Rapid and Automated Deployment of Monitoring Services in Grid Environments*


Mei Yiduo, Dong Xiaoshe, Li Junyang, Xu Jing, Xue Zhenghua
School of Electronic and Information Engineering, Xi′an Jiaotong University, Xi′an, 710049, China
meiyiduo@gmail.com
xsdong@mail.xjtu.edu.cn

Abstract Automation of deployment for monitoring system is


beneficial for improved correctness and speed.
The monitoring service is a crucial component in the Usually, the monitoring systems require the
service oriented grid infrastructure. Objectives and deployment of agents (sensors) on all sites and often on
requirements for the grid monitoring system have been all end nodes, with the potential for thousands of
summarized in this paper. To cope with the dynamic and resources and applications at geographically distant nodes
large-scale nature of the grid, a scalable distributed in grid environments, it is crucial that the deployment and
monitoring system is proposed, which can support easy distribution mechanisms for the monitoring system scale
and rapid deployment of monitoring services in the grid well. Furthermore, an update of the monitoring system
environments. The BitTorrent protocol is adopted to always involves changes to all the deployed nodes, which
facilitate the distribution and automated deployment of requires a high efficient automated deployment and
the monitoring system. The deployment of monitoring update approaches to the monitoring system.
service for the newly joined node and the update of the In this paper, we describe a grid Monitoring System
deployed components can be initiated by the nodes within capable of Rapid and Automated Deployment (MSRAD).
the system to reduce the administrator′s participation. The objective of our proposed system is to support
The maintenance cost and human errors might happen in automated deployment to a number of nodes efficiently.
the deployment process can be reduced. With the help of Peer-to-peer protocols such as BitTorrent [12][13] can be
the peer-to-peer networks, our proposed system can employed to facilitate the distribution and automated
automatically adapt to failures in network connections or deployment of the monitoring system, which is inspired
nodes. Service capacity of our proposed system is given by SystemImager [15]. The deployment of monitoring
out. Simulation results show that our proposed system service for the newly joined node and the update of the
supports efficient and rapid deployment to a large scale deployed components can be initiated by the nodes within
grid and provides a robust platform to efficiently monitor the system to reduce the administrator′s participation.
the grid resources. This can reduce the maintenance cost and human errors
might happen in the deployment. With the help of the
1. Introduction peer-to-peer networks, MSRAD can automatically adapt
to failures in network connections or nodes. Service
Grid computing is quickly becoming the new means capacity of our proposed system is given out. Simulation
for creating distributed infrastructures and virtual results in the framework of the Constellation Model [14]
organizations (VOs) for multi-institutional research and show that MSRAD can support efficient and rapid
enterprise applications [1]. In a grid environment, the deployment to a large scale grid and provide a robust
monitoring system is an essential part to keep the platform to efficiently monitor the grid resources.
complex distributed system efficient [2]. Computational The rest of the paper is organized as follows. Section 2
and data grids require a substantial amount of monitoring summarizes previous works on monitoring systems in
data be collected for various tasks such as fault detection, grid environments. Section 3 presents objectives and
performance analysis, performance tuning, performance requirements of our proposed monitoring system. In
prediction, and scheduling [3]. section 4, we introduce the monitoring architecture of our
Since the software deployment management dominates proposed system. This section will also show how to
system administration costs, and configuration is a major deploy the components to target nodes. Service capacity
source of errors in system deployment [4]. Research on is analysed in section 5. Finally, we end this paper with
the automated deployment mechanism of the monitoring discussion on simulation results in section 6 and
services in grid environments has been initialized [5]. conclusion in section 7.

*
This research is supported by 863 Project of China (Grant No. 2006AA01A109),
National Natural Science Foundation of China (Grant No. 60773118) and
Program for Changjiang Scholars and Innovative Research Team in University.

0-7695-3051-6/07 $25.00 © 2007 IEEE 328


DOI 10.1109/APSCC.2007.65
2. Related Work
As the monitoring system is a key part of the grid
infrastructure, many systems and tools have been
developed to provide the monitoring services. The Grid
Monitoring Architecture (GMA) [3] developed by the
Global Grid Forum Performance Working Group
describes components of a grid monitoring architecture
and their essential interactions. GMA provides a minimal
specification that will support required functionality and
allow interoperability. The Relational Grid Monitoring
Architecture (R-GMA) [6] which has been developed
within the European DataGrid Project as a grid
information and monitoring system is based on the GMA. Figure 1. Objectives and requirements of MSRAD
Ganglia [7] is a scalable distributed monitoring system
for high performance computing systems such as clusters z Proper granularity of data processing. Data
and grids. It is based on a hierarchical design targeted at processing in a GMS mainly involves real-time
federations of clusters. It relies on a multicast-based monitoring and the processing of historical records or
listen/announce protocol to monitor state within clusters logs. Real-time monitoring is required to reflect the
and uses a tree of point-to-point connections amongst status of jobs, resources and services at present, and
representative cluster nodes to federate clusters and can help real-time decision making such as error
aggregate their state. detection and correction, job scheduling. The objective
ChinaGrid Super Vision (CGSV) [2] provides is to collect data in time without causing too much
monitoring functions for ChinaGrid [8]. CGSV is based overhead. Long-term records are necessary for
on the GMA, and it is designed to collect status performance improvements and accounting. The
information of each entity (such as resources, services, objective is to save sufficient data according to the
users, jobs, networks) and provide corresponding system′s need and the node′s storage capability.
information data query and mining services. GlobalWatch z Low overhead. The objective is to minimize the load
[9] is a distributed platform to monitor various resources introduced by the monitoring system. The monitoring
of grid platforms so as to improve the flexibility and process should carry out with low interfering side-
usability of grid systems. Reference [5] proposes a script- effect on the node′s performance. And the monitoring
based approach to automate the deployment of grid system should transmit monitoring data with low
monitoring service component. This monitoring system is latency [3].
compatible with the GMA, and scalable integrates with z High adaptability. Heterogeneity is one of the
existing grid platforms and monitoring systems. characteristics in grid. It is important to overcome this
These research efforts have focused on scalability and challenge to provide a unified logical view of
low overhead aspects of the monitoring system, while our resources and services, including their status
work has not only focused on the above aspects, but also (monitoring data). Moreover, service providers and
concerned about automated and rapid distribution and virtual organizations are autonomous in grid, and have
deployment of the monitoring services. their own security policies and access control
strategies respectively. The monitoring system should
3. Objectives and Requirements of MSRAD not conflict with the node′s security policies.
z Scalable. As the deployed grid size increases from
With the potential for thousands of resources at tens to thousands of nodes, the objects to monitor will
geographically distant sites and tens-of-thousands of increase dramatically. Thus, it is necessary to design a
simultaneous grid users, it is critical that data collection monitoring system with high scalabilities.
and distribution mechanisms scale [3]. Several design z Flexible. Nodes participating in a VO often have
issues should be considered in the construction of a Grid deployed grid infrastructure already. A grid monitoring
Monitoring System (GMS), see Fig. 1. The pyramid is system must interoperate with the existing grid
designed based on the analysis of related works and our middlewares and should be easy to integrate into
goals. existing grid platforms.
As the time consumption and the probability of errors
caused by configuration increase with the size and
complexity of the grid, the grid requires an Automated
Deployment Monitoring System (ADMS) to automate the

329
deployment of grid monitoring service components on maintenance cost and human errors happened in the
newly joined resources. deployment process can be reduced.
z Transparent deployment. The scale and complexity z Fault tolerant. In the automated deployment or update
of today′s grid systems make them increasingly process, with the help of the peer-to-peer networks, our
difficult and time-consuming to deploy. As grid nodes proposed system can automatically adapt to failures in
join and leave quickly, automated resource and service network connections or nodes, as well as to a transient
monitoring must be performed and launched at a population of nodes [11]. And the deployment can
frequency compatible with the dynamic of grid carry on as long as there is one complete copy of the
elements [10]. Further more, automated approaches to monitoring system components, which decreases the
deployment prevent human errors and make the dependence on the central server.
deployment process easier [4].
z Transparent update. The grid is an evolutionary 4. MSRAD
system in which new types of resources and
applications will join at anytime, in this case, an 4.1. Monitoring Architecture
update on the monitoring system is necessary. An
update might require changes to all the nodes which Many sites or institutions participating in the grids
are the potential customers for the new types of comprise multiple clusters or supercomputers. Different
resources and applications. Automated approaches to clusters or supercomputers work independently from each
update should also be employed to facilitate the other. A hierarchical structure is employed to transfer the
massive update of monitoring systems. large amount of monitoring data from the computing
Besides the above requirements, distribution and nodes to the interaction interface hosted at the monitor
deployment of the monitoring system can benefit from the center, see Fig. 2.
following characters of MSRAD. There are three roles in the monitoring architecture: the
z Rapid deployment. Since the deployment and update computing node, the front-end node and the monitor
of the monitoring system involve all the nodes in grid, center. Nodes in different layer are required to deploy
it is important to delivery and deploy all the different modules. To the computing nodes, which locate
components in an efficient way. Our proposed system at the bottom layer, deployment agent, update agent,
can provide rapid deployment of monitoring services controller and sensors have to be installed on them.
to the nodes in a large scale grid. Time consumption Deployment agent has to be installed at the beginning
grows very slowly with the number of target nodes, phase of the deployment. Technically, it can be
which achieves better scalability. implemented using the peer-to-peer client, such as
z Self-organizing. The nodes in our proposed system are BitTorrent. After the transfer of the deployment agent is
able to self-organize into a peer-to-peer overlay to complete, the other modules can be treated as the content
facilitate the distribution and deployment of the delivered by the deployment agent. Sensors are
components. The deployment of monitoring service for responsible for collecting the status of the target
the newly joined node and the update of the deployed resources. Monitoring data are processed into a standard
components can be initiated by the nodes within the format, such as XML file. Update agent executes the
system to reduce the administrator′s participation. The update actions. The controller module is listening to the
network

Monitor Center
Front-End Node
Computing Node
Interaction Interface Database
Communication Module

Deployment Agent Archiver Global Aggregator


Controller

Data Aggregator ...


Update Agent
Deployment Agent
Controller

Update Agent
...
Sensor
Sensor
Sensor
Cluster Supercomputer Cluster
Figure 2. Monitoring architecture of MSRAD

330
to receive instructions from upper layer. Instructions are sends a deployment request to the registration center to
then dispatched to corresponding modules. require the deployment of monitoring service (1).
Besides deployment agent, update agent and controller, b. According to the role of the target node, registration
data aggregator is running on the front-end node of a center replies the request with a source node which
cluster or supercomputer. The data aggregator is in charge contains the required part of the monitoring system.
of aggregating the monitoring data collected from the Meanwhile, registration center will send a notification
individual nodes within clusters or supercomputers. to the source node (2).
Monitor center holds all the monitoring data and users c. Communications between the target node and the
or other grid services can access the data through the source node are established through handshaking.
interaction interface. Communication module is in charge Then, deployment agent module is transferred from the
of the transferring of the instructions and monitoring data. source node to the target node (3). An auxiliary
All the monitoring data is gathered by the global protocol such as the GridFTP is needed to distribute
aggregator and processed by the archiver. Historical the deployment agent.
records can be retrieved within the database. d. The registration center could act as a tracker, which
helps target nodes downloading the same files to find
4.2. Automated Deployment and Update each other in the BitTorrent networks. The target node
contacts the registration center periodically to show it
In MSRAD, the monitoring system images are hosted has which pieces of the deployment files. Then the
in a central repository on a server, called the source node. registration center returns a list of nodes that require
The nodes to deploy the monitoring system are referred to the same deployment file but have different file pieces
as the target nodes. A registration center is needed to (4).
provide the registration services for the source node and e. The target node then establishes a connection to other
the target nodes. Administrator can customize the target nodes and finds out which pieces reside in each
monitoring system′s configuration according to the needs other node [12]. Other modules are distributed via the
for efficiently monitoring, and be assured that the BitTorrent protocol (5).
monitoring system, once deployed, will behave correctly Since the target nodes attempting to deploy the same
in different nodes. The content can be distributed among part of monitoring system could contact to each other
the target nodes using the BitTorrent protocol. simultaneously and download different pieces of the
The MSRAD architecture supports two interactions for deployment file from different nodes, the distribution and
transferring data between source nodes and target nodes: deployment process can be accelerated. The nodes
subscription and push, which correspond to the involved in the deployment form a peer-to-peer overlay,
deployment and update process respectively. Fig. 3 the deployment of the monitoring system is voluntary and
illustrates the subscription and push mode, where solid self-organized, without the administrator′s participation.
lines represent the interactions happened in the With the help of the reliability brought by the BitTorrent
subscription mode and dashed lines represent the ones in protocol, the deployment can continue even under
the push mode. temporary failure of some nodes. In a worst case, the
deployment can carry on as long as there is a complete
Source Node Registration Center
copy in the network, even if the source node is down.
In the push mode, interactions happen in the following

⑥ sequence.
⑦ ⑦ a. When a new update is available, the push mode is
⑦ ③ ⑦ ④ enabled by the source node, which sends a notification
⑦ ① to the registration center, and the registration center
Node1 ⑤ ⑤ ② Node5
⑤ will reply with a list contains the IDs of the target
⑤ ⑤ nodes need to deploy the update component (6).
Node2 Node4
Node3 b. The target nodes can download the new update from
the source nodes, or they can download the file blocks
Figure 3. Subscription mode and push mode
for update from other target nodes with the help of the
registration center and the update agent (7).
In the subscription mode, the interaction is initiated by
the target nodes, as shown in Fig. 3. Suppose that node 3
is a newly joined node (target node). Interactions happen 4.3. Interaction Interface
in the following sequence.
a. At first, the target node contacts the registration center We provide a web based interface to administrator and
to notify its arrival and register itself. After that, it users, as shown in Fig. 4. Through the interface they can
query the load on the target nodes and browse the status

331
Figure 4. Web based interface

of the target resources. The web based interface is easy to node is referred to as a client. In this case, the bandwidth
use and barrier to first use is low. u between a provider and a client is fully utilized. This
ideal transferring process is called as one-to-one service,
5. Service Capacity as shown in Fig. 5.

In this section, we will analyze the service capacity of Source Node ...
our proposed system. To simplify the demonstration, we
consider a deployment process in a cluster, the involved ...
nodes are homogeneous. Suppose that there are y target
nodes, and only one source node. Since the BitTorrent ...
protocol divides the file into equal-sized blocks (pieces) Target Nodes

...
[13], we assume that the whole deployment files are
devided into n pieces. ...
In a cluster, the bandwidth between nodes is
represented by u. The size of the file to deploy is I. Then one-to-one service
t
to each node, I = ∫t c( t )dt , where t0 is the starting time,
1

0
Figure 5. Examples of one-to-one service
and t1 is the finish time, c(t) is the download rate.
The optimal model for the whole deployment is as During the beginning phase of the deployment, the
follows. According to the BitTorrent protocol, the bandwidth of the source node is a bottleneck. To
contents are divided into pieces (file blocks), as shown in accelerate the deployment process, an optimal scheme is
Fig. 5, a block stands for a piece of data, a blank block that the source node sends a block to a different target
means that the target node has not the corresponding node within each round, and each target node sends a
piece yet, a grained block means that the target node has block to another node. Fig. 6 illustrates this process, for
received the corresponding piece already. example, in round one, the source node sends one block to
To explain our analysis, we give the following the first target node, in round two, the source node sends
assumptions. Since the distribution is carried out within one block to the second target node, at the same time, the
the clusters, and the nodes in the clusters are always first node shares its piece with the third node. Note that,
organized by a switch, then assume that, blocks are the number in the figure indicates the round number.
delivered synchronously, which means in every round, x Suppose that, in round r, pr pieces of files are delivered,
then during the beginning phase of the deployment, pr =
(1 ≤ x ≤   ) blocks, each of which is held by one node
y 2
2 1 + ∑ 2 r−i = 2 r−1 , where 1 refers to the piece transferred by
i=r
are transferred to the other x nodes. Every round, a
sending node is referred to as a provider, and a receiving the source node, 2 ≤ i ≤ r.

332
During the steady phase of the deployment, an ideal 2
re = rs + rt = 2 n − 2 + log 2 y + (1)
situation is that   + 1 (1 refers to the source node)
y y
2 I /n I
Every round, the time consumption is tr = = .
u n×u
nodes upload pieces for the other   nodes in each
y
2 Therefore, the total time consumption is
round, which means that in every round, at most 2 I
T = re × tr = (2n − 2 + log 2 y + ) × (2)
y n ×u
 y
 2  pieces are replicated. Note that, during the steady The approximate result is T ≈ 2n ×
I
=
2×I
. The
n×u u
phase of the deployment, the maximal number of pieces
could be transferred in each round remains at a certain formula shows an important feature of our system, i.e. our
proposed system can be deployed in a rapid way and the
value (   ), contrasts to the value of the beginning phase
y
total time remains at a certain level. In practice, we can
2
revise Formula 2 to Tt = T + Ta, where Tt represents the
(exponential growth). Suppose that the deployment total time and Ta represents the additional time caused by
process steps into the steady phase at round rs, let pr = extra communications.
2
 y  y Reference [5] proposed a linear deployment method,
 2  , i.e.  2  = 1 + ∑ 2 rs −i , solve this equation, we can get
which means the target nodes are deployed one by one.
i = rs

rs = log 2 y . I
The time consumption can be calculated as Ts = n × ,
u
i.e. the time grows linearly with the number of target
Source Node ... nodes, contrasts to the value of our proposed system.
① ② ③
Target Node 1 ... 6. Simulation Results
Target Node 2 ② ...
... To verify the efficiency of our proposed method, we
Target Node 3
adopted a simulation based test. Using a simulator can
③ ③ benefit the experiments for the sake of providing the
flexibility of precisely controlling the parameters for the
networks and the target nodes. For our proposed system,
...

③ the typical scenarios include computer labs or institutions,


... and the number of simultaneous target nodes for
Target Node y deployment could be of the order of several tens or even a
few hundreds. The simulator adopted by us is the one
one-to-one service introduced in reference [13]. Our simulation is carried out
Figure 6. Ideal deployment in the framework of the Constellation Model [14].
50
Until the round rs, the number of pieces which have
Analysis Result
been deployed to the target nodes can be calculated as Simulation Result
Time Consumption (seconds)

follows. Recall that, every round, there are totally pr 45

blocks are distributed during the beginning phase, and this


process lasts for rs round, so there are totally 40
rs
d = ∑ 2 r −1 = 2 rs − 1 file blocks distributed in different
r =1 35
target nodes, since rs = log 2 y , we can get d = y − 1.
Because there are totally y × n blocks to be delivered 30
during a deployment process, and during the steady phase,
 y
 2  blocks can be transferred in each round, so the
20 30 40 50 60 70 80 90 100
Numer of Target Nodes

remaining number of round can be calculated as follows. Figure 7. Analysis results and simulation results
y × n − ( y − 1) 2
rt = = 2( n − 1) + , the entire number of round
y y
 2 
is

333
Figure 8. Deployment time with respect to bandwidth and number of target nodes

To simulate a real scenario, we set the join rate The bandwidth of the network varies from 100Mb/s to
(request rate) to 2, 4, 6, 8 and 10 respectively, and the join 1000Mb/s, and the numbers of target nodes are set to 20,
time lasts for 10 seconds. The network bandwidth of each 40, 60, 80 and 100 respectively. The results are shown in
node is set to 100Mb/s, and there is only one source node Fig. 8. The upper surface represents the results got by
in the network, the results are shown in Fig. 7. The simulating the linear manner used by [5] and the lower
analysis results are calculated by Formula 2. The surface represents the results of our system. As illustrated
simulation results are got by running the simulator with by Fig. 8, our proposed method works more efficiently
corresponding input parameters. As observed from the and can support a rapid deployment to a large scale grid.
figure, there are differences between the analysis results
and the simulation results. This is because the Formula 2 7. Conclusion
is got under an ideal situation without considering the
additional communications among nodes, moreover, the In this paper, we have summarized the objectives and
simulation is carried out with the assumption that the requirements for the grid monitoring system. To cope
target nodes send requests to the source node in a certain with the dynamic and large-scale nature of the grid, a
rate, as is usually the case in real-world situations. scalable distributed monitoring system is proposed. The
However, each curve remains at a certain value. The monitoring system can support rapid and automated
trends of the two curves remain the same, from which we deployment. The monitoring architecture and deployment
can see that the number of the target nodes has little process of our proposed system have been introduced.
impact on the deployment time. This feature can The BitTorrent protocol is adopted to facilitate easy and
guarantee that our proposed system can be deployed rapid deployment of monitoring services in the grid
efficiently in a large scale grid. environments. Nodes in our proposed system can self
In the second test, comparison methodology is organized into a peer-to-peer overlay to automate the
adopted. We compare our method with the one presented deployment of the monitoring service. Also, this system
in [5], in which the number of target nodes and the can automatically adapt to failures in network connections
network bandwidth are two factors for determining the or nodes.
time consumption caused by the deployment process.

334
Service capacity has been analyzed. Comparison [7] M.L. Massie, B.N. Chun, D.E. Culler, ″The Ganglia
methodology is adopted. Simulation results in the Distributed Monitoring System: Design, Implementation, and
framework of the Constellation Model can support our Experience″, Parallel Computing, Vol 30, June 2004, pp. 817-
analysis. From the results, we can see that our proposed 840.
system supports efficient and rapid deployment to a large [8] H. Jin, ″ChinaGrid: Making Grid Computing a Reality″,
scale grid and provides a robust platform to efficiently ICADL 2004, Springer-Verlag, Berlin Heidelberg, LNCS 3334,
2004, pp. 13-24.
monitor the grid resources.
[9] S. Di, H. Jin, S. Li, et al, ″GlobalWatch: A Distributed
Service Grid Monitoring Platform with High Flexibility and
References Usability″, Proceedings of the 2006 IEEE Asia-Pacific
Conference on Services Computing (APSCC'06), IEEE Press,
[1] S. Hastings, S. Oster, S. Langella, et al, ″Introduce: An Open 2006, pp. 440-446
Source Toolkit for Rapid Development of Strongly Typed Grid [10] F. Bonnassieux, R. Harakaly, P. Primet, ″Automatic
Services″, Journal of Grid Computing, Springer Netherlands, Services Discovery, Monitoring and Visualization of Grid
available online, Environments: The MapCenter Approach″, Across Grids 2003,
http://www.springerlink.com/content/u301u225wg5356w3 Springer-Verlag Berlin Heidelberg, LNCS 2970, 2004, pp. 222-
[2] W. Zheng, L. Liu, M. Hu, et al, ″CGSV: An Adaptable 229.
Stream-Integrated Grid Monitoring System″, NPC 2005, [11] S. Androutsellis-Theotokis, D. Spinellis, ″A Survey of
Springer-Verlag, Berlin Heidelberg, LNCS 3779, 2005, pp. 22- Peer-to-Peer Content Distribution Technologies″, ACM
31. Computing Surveys, ACM Press, 2004, 36(4), pp. 335-371.
[3] B. Tierney, R. Aydt, D. Gunter, et al, ″A Grid Monitoring [12] D. Qiu, R. Srikant, ″Modeling and Performance Analysis of
Architecture″, Tech. Rep. GWD-Perf-16-3, Global Grid Forum BitTorrent-Like Peer-to-Peer Networks″, SIGCOMM’04, ACM
(GGF), 2002, http://www-didc.lbl.gov/GGF-PERF/GMA- Press, 2004, Portland, Oregon, USA, pp. 367-377.
WG/papers/GWD-GP-16-3.pdf [13] A.R. Bharambe, C. Herley, V.N. Padmanabhan, ″Analyzing
[4] V. Talwar, D. Milojicic, Q. Wu, et al, ″Approaches for and Improving a BitTorrent Networks Performance
Service Deployment″, IEEE Internet Computing, IEEE Press, Mechanisms″, Proceedings of 25th IEEE International
March /April 2005, pp. 70-80. Conference on Computer Communications(INFOCOM 2006),
[5] X. Dong, Y. Wang, Z. Qin, et al, ″Research on an Automatic IEEE Press, 2006, Barcelona, Spain, pp. 1-12.
Deployment Mechanism of Monitor Service in Grid [14] Y. Wang, X. Dong, X. He, et al, ″A Constellation Model
Environment″, Proceedings of the Fifth International for Grid Resource Management″, The Sixth International
Conference on Grid and Cooperative Computing Workshops Workshop on Advanced Parallel Processing Technologies
(GCCW′06), IEEE Press, 2006, pp. 63-70. (APPT 2005), Springer-Verlag, Berlin Heidelberg, LNCS 3756,
[6] A. Cooke, W. Nutt, J. Magowan, et al, ″Relational Grid 2005, pp. 263-272.
Monitoring Architecture (R-GMA)″, http://www.r- [15] http://wiki.systemimager.org/index.php/Main_Page
gma.org/pub/Cracow-2003-rgma.pdf

335

Vous aimerez peut-être aussi