The Journal of Systems and Software 114 (2016) 69–81

Contents lists available at ScienceDirect

The Journal of Systems and Software

journal homepage: www.elsevier.com/locate/jss

Quality assurance in software ecosystems: A systematic literature

mapping and research agenda
Jakob Axelsson∗, Mats Skoglund
Software and Systems Engineering Laboratory, Swedish Institute of Computer Science (SICS), PO Box 1263, SE-164 29 Kista, Sweden

a r t i c l e i n f o a b s t r a c t

Article history: Software ecosystems are becoming a common model for software development in which different ac-
Received 12 March 2015 tors cooperate around a shared platform. However, it is not clear what the implications are on software
Revised 11 December 2015
quality when moving from a traditional approach to an ecosystem, and this is becoming increasingly
Accepted 11 December 2015
important as ecosystems emerge in critical domains such as embedded applications. Therefore, this pa-
Available online 18 December 2015
per investigates the challenges related to quality assurance in software ecosystems, and identifies what
Keywords: approaches have been proposed in the literature. The research method used is a systematic literature
Software ecosystems mapping, which however only resulted in a small set of six papers. The literature findings are comple-
Quality mented with a constructive approach where areas are identified that merit further research, resulting in
Verification a set of research topics that form a research agenda for quality assurance in software ecosystems. The
Testing agenda spans the entire system life-cycle, and focuses on challenges particular to an ecosystem setting,
which are mainly the results of the interactions across organizational borders, and the dynamic system
integration being controlled by the users.
© 2015 Elsevier Inc. All rights reserved.

1. Introduction non-negotiable characteristic. For some embedded systems, certain

Software ecosystems are an increasingly popular model for soft-
ware development and the associated business aspects. Tradition-
ally, companies work in a fairly closed setting in which software
systems are provided as more or less monolithic entities. In a
software ecosystem, this changes to an open environment where
a community of developers from different organizations gather
around an open or semi-open platform, and thrive from each
other in a coopetition. This gives a potential for numerous ben-
efits, including improved customer offers through the use of the
innovation potential in the ecosystem (Iansiti and Levien, 2004)
and reduced complexity of development by allowing composabil-
ity (Bosch and Bosch-Sijtsema, 2010a).
Many of the examples of successful software ecosystems come
from the IT domain, and include general applications such as
PC operating systems, mobile phone apps, etc. (Bosch, 2009; van
der Schuur et al., 2011). There are also initial signs of software
ecosystems emerging in more critical applications, based on em-
bedded systems, which are to a much higher degree tailor-made
for a specific purpose (Eklund and Bosch, 2014). For all software
products, quality is of high importance, and for critical embed-
ded systems, on which the lives of people may depend, it is a
∗
Corresponding author. Tel.: +46 72 734 29 52.
E-mail address: jakob.axelsson@sics.se (J. Axelsson).
aspects of quality are even formal requirements that must be ful-
filled to obtain certification from authorities.
Our current research deals with ecosystems for federated em-
bedded systems, a concept where traditional embedded systems
can be dynamically extended with plug-in software components
(Axelsson and Kobetski, 2013) which can be provided by exter-
nal developers. In a previous case study around the characteristics
of such an ecosystem, quality assurance was identified as one of
the most important factors (Axelsson and Kobetski, 2014; Axelsson
et al., 2014), and some indication was given about important fac-
tors that contribute to quality in the embedded domain. However,
according to Manikas and Hansen (2013) and Barbosa et al. (2013),
there appears to be a general lack of knowledge about the relation
between quality assurance and software ecosystems.
The purpose of this paper is therefore to: (a) provide an ini-
tial characterization of the state-of-the-art in the area, as well as
(b) provide directions for further research needs in the form of a
research agenda for the field.
To investigate the relationship between quality assurance and
software ecosystems scientifically, it is necessary to first provide
some clarity what is meant by these two rather broad terms. Thus,
the following subsections deal with defining the terms software
ecosystems and quality assurance. This also provides a theoreti-
cal framework that defines the scope of the study. After this, a
more exact definition of the research questions in the paper will
be given, together with an overview of the remainder of the paper.

http://dx.doi.org/10.1016/j.jss.2015.12.020
0164-1212/© 2015 Elsevier Inc. All rights reserved.
70 J. Axelsson, M. Skoglund / The Journal of Systems and Software 114 (2016) 69–81
System instance
User
Software extension
Software extension
Software extension
Infrastructure
Niche
player
Keystone
Platform
Fig. 1. Technical and organizational concepts in a software ecosystem.
1.1. Definition of software ecosystems
Although most of the work on software ecosystems use the
term in similar ways, there is no definition which is universally
agreed upon. In a recent systematic literature review (Manikas and
Hansen, 2013), it is found that 44% of the papers do not state any
definition at all. Among those that do provide a definition, the
most common one (27%) used is the one by Jansen et al., who de-
fine a software ecosystem as “a set of businesses functioning as
a unit and interacting with a shared market for software and ser-
vices, together with the relationships among them. These relation-
ships are frequently under-pinned by a common technological plat-
form or market and operate through the exchange of information,
resources and artifacts” (Jansen et al., 2009).
Another commonly used definition (14% of the papers in the
above study) is provided by Bosch and Bosch-Sijtsema: “A software
ecosystem consists of a software platform, a set of internal and ex-
ternal developers and a community of domain experts in service
to a community of users that compose relevant solution elements
to satisfy their needs” (Bosch and Bosch-Sijtsema, 2010a, b).
Based on their review, Manikas and Hansen (2013) identify
common elements in the various definitions, and propose the fol-
lowing definition, which is also the one we will use in this paper:
Definition. A software ecosystem is the interaction of a set of ac-
tors on top of a common technological platform that results in a
number of software solutions or services.
The actors in the ecosystem are commonly divided into a key-
stone and a set of niche players (Iansiti and Levien, 2004). In the
conceptual model used in the analysis in this paper, the keystone
is responsible for developing the platform and also plays an im-
portant role in the success of the ecosystem through orchestration.
The niche players develop software extensions which execute on top
of the platform, and the extensions can be added by the user in
different combinations. A system instance, whose quality we are in-
terested in, thus consists of a platform combined with a set of dif-
ferent extension components (Jansen and van Capelleveen, 2013).
To support the interaction between the actors, and between actors
and systems, an infrastructure is needed. All these are illustrated
in Fig. 1. (Some ecosystems may not map easily to this conceptual
model, and in those cases, other challenges may be present when
it comes to quality assurance.)
1.2. Definition of quality and quality assurance
The focus of this paper is quality assurance, but to able to de-
fine that, a discussion of quality in general is needed. Whereas
software ecosystems are a relatively new phenomenon, product
quality has been discussed for many decades. Already in the 1930s,
in his effort to characterize quality and provide quality control
mechanisms, Shewhart identified that “there are two common as-
pects of quality: one of them has to do with the consideration of
the quality of a thing as an objective reality independent of the
existence of man. The other has to do with what we think, feel or
sense as a result of the objective reality. In other words, there is a
subjective side of quality” (Shewart, 1931). In software engineering,
a similar difference was much later captured in a description of
validation vs. verification, where validation checks if we are build-
ing the right product, and verification if we are building the prod-
uct right (Boehm, 1979). Both these definitions concern primarily
how the product is performing during its operational use, and this
is also the emphasis of this paper, where we focus on how well
the system under operation fulfills the needs or requirements as
defined (implicitly or explicitly) by various stakeholders.
For software systems, the word “quality” is however often used
in a different and broader sense, which also covers many aspects
related to the efficiency of development, maintenance, evolution,
etc. of the system. The well-established quality standard ISO 9126
and its follower ISO 25010 are examples of this, where two of the
quality characteristics are maintainability, defined as “the capabil-
ity of the software product to be modified”, and portability, de-
fined as “the capability of the software product to be transferred
from one environment to another” (ISO/IEC, 2001; ISO/IEC, 2011).
Those are aspects which are beyond the primary scope of this pa-
per, and instead our focus is quality in use, even though all aspects
are sometimes intertwined with each other.
In addition, much research related to software ecosystems has a
relation to software architecture, a field where it is common prac-
tice to discuss about the “quality attributes” of an architecture.
The architecture is in itself an abstraction of the real product, and
not all quality in use aspects are considered since they may relate
more to details at a lower level than to the architecture. Also, the
quality attributes of the architecture usually go beyond the quality
in use, and emphasize how to manage the product during its de-
velopment and maintenance phases, in order to reduce complexity.
The definition of quality we will use in this paper is as follows:
Definition. The quality of a software product is how the system
under operation fulfills the needs and requirements as defined (im-
plicitly or explicitly) by its stakeholders.
Given this definition of quality, quality assurance can be defined
as follows:
Definition. Quality assurance is the set of activities carried out to
ensure that the system has sufficient quality.
This includes activities for ensuring that the software product
was designed to be fit for its intended purpose, i.e. validation, but
also that mistakes in the implementation are eliminated, i.e. veri-
fication. However, quality assurance cannot be seen as an isolated
activity, but rather as a set of activities that take place during dif-
ferent phases of the product’s life-cycle. This life-cycle can be de-
scribed in many ways, and one example is provided in the standard
ISO12207 (ISO/IEC/IEEE, 2008) for software life-cycle processes.
This standard defines four high-level process areas, namely agree-
ment processes; organizational project-enabling processes; project
processes; and technical processes, with sub-processes defined for
each of them. For example, the technical processes include activ-
ities such as requirements definition; system architectural design;
implementation; integration; and operations. Quality assurance re-
lated activities can occur in any of these processes.
Product quality has a relationship with the much broader con-
cept of ecosystem health used by many software ecosystem re-
searchers. This concept is defined as the ability of the ecosystem
to endure and remain variable and productive over time (Manikas
and Hansen, 2013), or alternatively as longevity and a propensity
 J. Axelsson, M. Skoglund / The Journal of Systems and Software 114 (2016) 69–81
for growth (Jansen, 2014). The term, which is also used for busi-
ness ecosystems, encompasses factors such as productivity, robust-
ness, and innovation that take place in the ecosystem, which are
important for its long-term survival. In the framework proposed
by Manikas and Hansen (2013), software ecosystem health is bro-
ken down into actor, software, and orchestration health, where the
software part is further broken down into software component
health, platform health, and software network health. Clearly, the
software component health and platform health are dependent on
good practices for quality assurance, and the orchestration can also
affect quality assurance by stipulating rules and processes to be
followed within the ecosystem. Jansen (2014) proposes an oper-
ationalization of open source ecosystem health measurements on
the network level and project level, with measurements related to
productivity, robustness, and niche creation. The effects of product
quality on the ecosystem have also been studied in the context of
open source software ecosystems (Wahyudin et al., 2007).
1.3. Research questions
Given the above definitions of software ecosystems and quality
assurance, this paper addresses the following research questions:
1. RQ1: What does the scientific literature say about quality assur-
ance in software ecosystems?
2. RQ2: What are the research needs related to quality assurance
in software ecosystems?
Obviously, many challenges and approaches that exist for soft-
ware development in a more traditional setting also apply to soft-
ware ecosystems. Therefore, the focus in this paper is only on as-
pects where a software ecosystem fundamentally differs compared
to traditional approaches, such as when development is carried out
internally within a company using a closed platform.
1.4. Overview of the paper
The remainder of the paper is structured as follows: In the next
section, the research method used, which is a systematic literature
mapping, is motivated and it is explained how it was applied. In
Section 3, the results from the literature are analyzed in more de-
tail, ending with a summary of the current state-of-the-art. Then,
in Section 4, the gaps in knowledge are identified, and a set of re-
search topics are derived, forming a research agenda for the area.
Finally, in Section 5, the conclusions are summarized.
2. Systematic literature mapping method
This study aims at providing a summary of the current state
of research as well as identifying research directions for quality
assurance in the software ecosystems domain. In order to get a
broad perspective on both these aspects, we decided to search
the research literature and compose a literature base to use as a
stepping stone for formulating the research agenda. We performed
a systematic mapping following the process defined in Pedersen
et al. (2008), which consists of five steps: (1) Definition of research
questions; (2) Conduct search; (3) Screening of papers; (4) Key-
wording using abstracts; and (5) Data extraction and mapping. The
research questions have already been stated in the previous sec-
tion, and we will now present how the search and screening was
performed, whereas the analysis steps are detailed in Section 3.
2.1. Search strategy
The search strategy was a combination of manual and auto-
matic search and extraction of conference and workshop proceed-
ings and journal articles from the following digital libraries:
71
• ACM (http://dl.acm.org)
• IEEE (http://ieeexplore.ieee.org)
• ScienceDirect (http://www.sciencedirect.com)
• Springer (http://link.springer.com)
These digital libraries were selected since they have been con-
firmed to cover relevant journals and conference and workshop
proceedings within software engineering (Dybå et al., 2007).
The search strategy used is shown in Fig. 2 and is divided into
the following three steps:
1. Define search terms and perform search in the digital li-
braries.
2. Extract metadata and clean the result from Step 1 to cope
with duplicates and differences occurring as a result of the
varying search possibilities of the individual digital libraries.
3. Make a qualitative analysis of the result based on the inclu-
sion and exclusion criteria (described below).
The rest of this section describes the steps more thoroughly.
2.1.1. Define search terms and perform search
We used the following search terms and logical operators for
the search:
(“testing” OR “verification” OR “quality”) AND (“software
ecosystem” OR “software ecosystems”)
As in most cases, these search terms were arrived at after a few
iterations to achieve relevant and complete results. Our objective
is to find work related to quality assurance in software ecosys-
tems. To reduce the risk of missing relevant papers in that area, we
choose to use the broader search term “quality” instead of “qual-
ity assurance”, and then narrow the scope to assurance through
the manual reviews. Also, we included “testing” and “verification”,
since these are activities often associated with quality assurance,
and we wanted to ensure that papers in those areas were not
missed even if they did not explicitly use the word “quality”. One
of the additional terms considered was “validation”, since that is
clearly relevant to quality as discussed in Section 1.2. However, in-
cluding this term did not yield any additional results, and it was
removed from the final query for simplicity reasons, although it is
in itself an early indicator that the emphasis of research in the field
is on testing and verification. Similarly, “software supply networks”
and “software supply industry” were tried as alternative terms for
software ecosystems, but did also not yield additional results.
We searched in the metadata fields Title, Abstract and Key-
words when available. There were differences in search possibili-
ties among the digital libraries and thus the search queries varied
slightly. The detailed queries used in each database and the result-
ing number of hits are shown in Table 1.
For Springer both articles and chapters were considered. The
search and extraction (described below) was performed during Au-
gust and September 2014. Only papers written in English were
considered.
For this particular study, there is one potentially interest-
ing event, the International Workshop on Software Ecosystems
(IWSECO), which has in certain years published their proceedings
outside the above sources. As a precaution, we therefore manu-
ally searched those proceedings, but this yielded no relevant pa-
pers. In addition, we also did a validation by searching the Scopus
database, in which 54 papers matched the search criteria, but all
the papers that met the inclusion criteria were already present in
the previously identified set.
2.1.2. Metadata extraction and cleaning of the results
Titles and abstracts for the results were extracted and stored
in an internal SQL database. This was done using a script where
any overlap between the results from the libraries was stored only
72 J. Axelsson, M. Skoglund / The Journal of Systems and Software 114 (2016) 69–81

ACM IEEE Explore ScienceDirect Springer

Search terms

208 unique

Query clean

32 papers

Full text exclusion

3 papers

Fig. 2. Overview of the search process and the resulting literature base.

once to avoid duplicate entries. In total 208 unique hits were col-
lected and stored. For Springer only full text search could be per-
formed which gave too many search hits since only title, abstract
and keywords should be considered. Papers only having the de-
fined search terms in full text but not in title or abstract were
excluded by running SQL queries on the internal database. The
resulting literature base contained 32 papers. These papers were
thus found in ACM, IEEE Xplore and ScienceDirect considering ti-
tle, abstract and keywords and papers from Springer considering
title and abstract.
2.1.3. Qualitative analysis
The qualitative analysis was performed by reading the title, ab-
stract and full text of the 32 papers in the final literature base.
First the titles were scanned to exclude papers that were not re-
search papers, e.g. “Keynotes”. Then the abstracts were analyzed to
exclude papers not within the software engineering field, e.g. work
using software to analyze organic material.
We found that the phrase “software ecosystem” is sometimes
used more as a buzzword than to describe something that is in
line with the software ecosystems definition stated above. Papers
of this kind were excluded together with papers where the phrase
“software ecosystem” was mentioned very few times in the pa-
per and where software ecosystem was not an integral part of the
work. The goal was to find work addressing both software ecosys-
tems and quality assurance. The quality assurance aspects in the
papers were also analyzed and papers not addressing quality in
line with our quality definition and in relation to software ecosys-
tems were removed. For example, a paper focusing on business
models and only mentioning briefly that quality is an important
aspect was considered not to contribute to answering the research
questions of this study and thus excluded.
The total literature base of only 201 papers and the final base
of only 3 papers are both very small compared to other system-
atic literature studies in software engineering, such as Engström
et al. (2010) with 2923 papers in the initial list, or the systematic
review on software ecosystem reported in Manikas and Hansen
(2013) having 420 papers in the gross list. One reason for this can
be the selection of search phrases we used in the search. For ex-
ample, we found two papers covering quality assurance of mobile
applications (Corral et al., 2015) and test suite understanding in
the Eclipse environment (Greiler and van Deursen, 2013) very rele-
vant to our study even though the phrase “software ecosystem(s)”
were not included in their metadata fields. We also came across
one paper on quality review methods (Jansen and van Capelleveen,
2013) published as a book chapter, and hence not indexed in the
databases. We added these three papers to our final literature base,
in addition to the three papers found in the literature search. Thus,
our final literature base is comprised of 6 papers covering quality
assurance in software ecosystems from different perspectives, see
Table 2.
3. Characteristics of the literature base
The resulting literature base covers the areas architecture; on-
line testing; assurance practices; software operation knowledge;
test suite understanding; and review methods. The papers have
been published in journals (3 papers), conferences (2 papers), and
as a book chapter (1 paper). Five of the papers were published in
the years 2013–2014 and one paper was published in 2011, which

Table 1
Search strings and number of hits from each database.

Database Search string No. of hits

ACM (Advanced search) ((Title:“quality” or Title:“testing” or Title:“verification” or Abstract:“quality” or Abstract:“testing” or 12

IEEE (Command search, Metadata only)
ScienceDirect (Expert search)
Springer
Abstract:“verification” or Keywords:“quality” or Keywords:“testing” or Keywords:“verification”) and
(Title:“software ecosystem” or Title:“software ecosystems” or Abstract:“software ecosystem” or
Abstract:“software ecosystems” or Keywords:“software ecosystem” or Keywords:“software
ecosystem”))
((“testing” OR “verification” OR “quality”) AND (“software ecosystem” OR “software ecosystems”)) 19
(title-abs-key(“testing” OR “verification” OR “quality”)) AND (title-abs-key(“software ecosystem” OR 3
“software ecosystems”))
(quality OR verification OR testing) AND (“software ecosystem” OR “software ecosystems”) 201
 J. Axelsson, M. Skoglund / The Journal of Systems and Software 114 (2016) 69–81
Table 2
Resulting literature base.
Title
Architecture for embedded open software ecosystems
Governance policies for verification and validation of service choreographies
The power of propagation: on the role of software operation knowledge within
software ecosystems
Software assurance practices for mobile applications
What your plug-in test suites really test: an integration perspective on test suite
understanding
Quality review and approval methods for extensions in software ecosystems
indicates that this is a research field that has very recently started
to emerge and has so far been given fairly little effort or recogni-
tion.
All papers can be categorized according to Shaw’s classifica-
tion of research to be “Qualitative or descriptive models”, which
is defined as “Structure or taxonomy for a problem area; archi-
tectural style, framework, or design pattern; non-formal domain
analysis, well-grounded checklists, well-argued informal general-
izations, guidance for integrating other results, well-organized in-
teresting observations” (Shaw, 2003). One paper (Greiler and van
Deursen, 2013) may also be considered in the category “Specific
solution, prototype, answer, or judgment“, which is defined as “So-
lution to application problem that shows application of software
engineering principles – may be design, prototype, or full imple-
mentation; careful analysis of a system or its development, result
of a specific analysis, evaluation, or comparison”.
The rest of this section describes the final literature base in
more detail.
3.1. System architecture
Eklund and Bosch (2014) discuss the desired characteristics of
the platform architecture in order to support an open software
ecosystem for the automotive domain. The system architecture is
an important enabler for many properties of a product, and the
term used to describe those properties is architectural quality at-
tributes. However, it should be noted that the word “quality” here
refers to the broader definition of quality used in software archi-
tecture, and only a subset of quality attributes normally refers to
quality of the system in operation (see Section 1.2).
One particular aspect of software ecosystems highlighted by the
authors is that system integration is not only decoupled from the
keystone player and moved to another actor (e.g., the end user),
but it is also decoupled in time. This means that software can be
deployed continuously at any time during the system’s life-cycle.
Although the authors do not mention it explicitly, a consequence of
such a model is that quality assurance must also be a continuous
activity throughout the system’s lifetime.
The authors elicit a set of quality attributes that an architecture
suitable for ecosystems of embedded software must satisfy. These
include composability, deployability, stability, and configurability as
general properties, and consistent user interface and dependability
as properties that apply in certain domains. Among these aspects,
it is primarily the last two that relate to the quality of the prod-
uct in operation, whereas the others point at mechanisms that are
needed in the platform to support an ecosystem approach where
applications can be developed, integrated, validated, and deployed
independently.
Based on the identified quality attributes, the authors con-
tinue to describe a set of design decisions that are fundamental
to the entire approach, that facilitate the ecosystem, or that are
on the platform implementation level. Among these decisions, cer-
tain have a direct bearing on product quality, and this includes
73
Authors Year Reference
Eklund and Bosch 2014 (Eklund and Bosch, 2014)
Cordeiro et al. 2013 (Cordeiro et al., 2013)
van der Schuur et al. 2011 (van der Schuur et al., 2011)
Corral et al. 2015 (Corral et al., 2015)
Greiler and van Deursen 2013 (Greiler and van Deursen, 2013)
Jansen and van Capelleveen 2013 (Jansen and van Capelleveen, 2013)
to provide mechanisms that let applications get differentiated ac-
cess to protected hardware resources depending on certification; to
be able to isolate application execution; to deploy critical applica-
tions to dedicated hardware; to support data access implementing
necessary integrity; to let the platform hide from the applications
the implementation of data exchange to fulfill necessary safety in-
tegrity levels; and to let the architecture be layered from a veri-
fication perspective. The primary driver for these decisions is the
dependability requirements.
The paper further defines a set of design patterns, and discusses
how four existing platforms satisfy the architecture as described by
the authors.
The contribution of this paper is primarily the characteriza-
tion of the necessary platform architectural concepts to support an
ecosystem. The aspects related to product quality are mainly those
that deal with the dependability aspects of many embedded do-
mains.
3.2. Governance policies for verification and validation
Cordeiro et al. (2013) identify and propose governance policies
for verification and validation of services and service choreogra-
phies in a service oriented architecture (SOA). SOA services avail-
able on the Internet need to be governed, i.e. administered and
managed with respect to, e.g. functionality, performance and qual-
ity during the services’ life-cycle. The authors present a number
of policies that can be used to create a governance framework.
The policies are categorized in activation policies (rules for how a
service is activated), policies for rating quality aspects, policies for
ranking quality parameters, choreography enactment policies (rules
for setting up a choreography) and test case selection policies that
regulate the testing strategies that can be used. The authors elab-
orate on these policies, how they can be formulated and used. The
policies are meant to be used when running online testing of ser-
vices in a service ecosystem. The online testing should be triggered
according to the policies and perform tests on new or changed ser-
vices in the ecosystem according to the identified policies.
The paper provides insight into how to organize online testing,
which is also highly relevant for software ecosystems.
3.3. Software operation knowledge propagation
van der Schuur et al. (2011) discuss the importance of propagat-
ing software operation knowledge between actors in the ecosys-
tem. Operation knowledge includes things like error reports and
performance measures. Of course, such knowledge has routinely
been collected and analyzed by companies for a long time, and the
information has been used for quality assurance and for planning
new products, including which features should be added or which
problems should be fixed. The difference in a software ecosys-
tem compared to other development models is that the operation
knowledge is distributed among the actors, and no one has a com-
plete picture. In order to efficiently use the knowledge, actors have
to share it effectively and transparently between them.
74 J. Axelsson, M. Skoglund / The Journal of Systems and Software 114 (2016) 69–81
According to the authors, the keystone should establish an in-
frastructure for publication and propagation of acquired operation
knowledge to niche players in their ecosystem. Niche players in
their turn must ensure that they respond adequately to operation
knowledge received from the keystone, as well as frequently in-
forming the keystone on end-user feedback knowledge. To encour-
age this, keystones may reward niche players with different priv-
ileges in their relation with the keystone, such as access to more
information or other resources from the keystone. They emphasize
the importance for quality assurance of having access to the oper-
ation knowledge. The keystone may also monitor the quality of the
niche players’ products.
The authors present information from interviews with four
ecosystem actors. In relation to quality management, these inter-
views give examples of companies that collect diagnostic reports
from the field. One approach used in this is to give each niche
player their own development key, which makes it possible to
trace if malformed API requests or responses originated in their
application. Another company has its own portal for sharing oper-
ation knowledge, such as crash dumps. Different support systems
are thus important in actually implementing the operation knowl-
edge propagation.
Keystones may also provide services to assist niche players in
realizing improvements in software quality, and these actions can
improve the attractiveness of their ecosystem and build trust. On
the other hand, if a particular participant distorts ecosystem sta-
bility, the keystone can diminish that actor’s participation, for ex-
ample by limiting their platform privileges. Keystones should then
be prepared to cope with niche player distrust resulting from us-
ing operation knowledge in this way, such as when that knowledge
indicates frequent instability of certain niche player’s software.
One aspect which is not discussed in the paper is the trade-
off between sharing information with other players for the benefit
of the ecosystem as a whole, versus the need to not share infor-
mation in order to secure competitive advantages such as intellec-
tual property or knowledge about the customers. The actors in the
ecosystem are in one sense partners, but in another sense they are
competitors, and finding the appropriate level of information shar-
ing is a key to a successful coopetition.
3.4. Mobile applications
Corral et al. (2015) performed a systematic literature study to
investigate software assurance practices for mobile applications.
Since mobile applications nowadays are developed within ecosys-
tems, their work is potentially relevant for the research questions
stated above. They identify three levels of practices: development
processes, product assurance, and implementation. The paper is
not very clear what perspective on quality they take, but it gives
the impression that they use the broader quality definition, and do
not restrict themselves to product quality.
The paper lists different requirements and constraints that are
characterizing mobile systems, including limited resources, avail-
ability, efficiency, and responsiveness. They also emphasize the re-
quirements put up by the ecosystem’s shared market, which is
usually dictated by the keystone in the form of publishing poli-
cies. Those policies typically comprise requirements related to the
operation of target devices, application content, application func-
tionality, user experience, and others. For an ecosystem, the qual-
ity of each product offered on that market impacts the quality of
the ecosystem as a whole, and the publishing policies represent a
comprehensive and well-settled set of quality expectations.
Among the literature reviewed in the paper, the most rele-
vant for product quality assurance in ecosystems is the part de-
scribing test-based assessments. In particular, they refer to another
work (Dantas et al., 2009) in which the authors discuss testing
requirements for mobile applications. These include testing of mo-
bile applications in both emulators and in the actual devices,
which is highly relevant for ecosystems, where the niche players
may not have access to the platform during all stages of the de-
velopment. Another requirement is that a mobile application may
not harm anything that is already deployed on the platform, and
this is similar to what Eklund and Bosch (2014) discuss concerning
independent deployment of applications.
3.5. Test suite understanding
Greiler and van Deursen (2013) discuss how to effectively per-
form testing in plug-in based systems like Eclipse. They present
data from interviews with 25 practitioners (from both open source
and closed source ecosystems) on their testing practices, which
can be summarized as a focus on unit testing, but with a large
need being felt for better integration testing. The authors then fo-
cus on the need for improved test suite understanding, both for
individual test cases and for the entire suites, as a consequence of
the extensive use of automatic testing. In particular, they address
the understanding problem beyond the unit level, to capture inter-
actions between plug-ins and provide an understanding of which
plug-ins are actually exercised in test suites. For software ecosys-
tems, this is a particular challenge since the different plug-ins and
platforms will be developed by different organizations, and there
may be limited access to the information needed to achieve un-
derstanding.
To improve test suite understanding, they propose five architec-
tural views which can be presented to developers. They describe
an implementation of those views in an IDE, and perform an eval-
uation of this.
3.6. Quality review methods
Jansen and van Capelleveen (2013) have studied methods for re-
view and approval of software extensions in platform based soft-
ware ecosystems. Their focus is on how the keystone or platform
coordinators can ensure that submitted extensions from niche
players have a sufficient quality, although the paper is not explicit
about what they mean with quality. The method used is a multi-
case study of 27 software ecosystems, primarily focusing on IT and
mobile platform applications. They give particular attention to how
review methods scale in very large ecosystems.
The three main methods considered are reviews, certification,
and community feedback. The reviews can be made either by
peers; by management; by experts; as walk-throughs; through self-
testing; or by the use of tool support. The certification can apply
either to the software, or to the developers to assess their skills.
The community feedback can be based on ratings by end users;
“developer karma” which lets developers rate other developers; or
developer tracking based on commits to a shared software reposi-
tory. The paper discusses the pros and cons of the different review
methods, and investigates their usage in the cases studied.
Based on the type of submission system, different patterns can
be observed in the case studies regarding the quality review meth-
ods used. For open source repository submissions, there are al-
ways peer reviews, and sometimes also management and expert
reviews. If submissions are to a marketplace, end-user ratings are
always used. These types of reviews are the most common ones
in the studied cases, whereas the other types mentioned are rarely
used. The data indicates that the manual review practices can scale
to very large ecosystems, with thousands of submitted extensions.
Automatic techniques, such as code analyses, are very rarely used,
and there appears to be a potential for future improvements in this
area. The authors conclude by stating that current methods are not
sufficiently used to improve quality.
 J. Axelsson, M. Skoglund / The Journal of Systems and Software 114 (2016) 69–81
It is worth noting that many of the methods, in particular re-
views and software certification, assume that source code is avail-
able. This is naturally the case for open source ecosystems, but
may not be true for commercially oriented ones, and as noted in
Axelsson et al. (2014), there can often be suspicion regarding in-
tellectual property protection in the relations between a keystone
and the niche players, leading to a situation where the parties may
not agree to share source code. Therefore, manual reviews need to
be complemented by testing and analysis techniques. User ratings
seem to be more generally applicable, at least for ecosystems with
large enough user groups, but have also many drawbacks that may
reduce trustworthiness, as explained in the paper.
3.7. Synthesis of findings from literature
To provide a synthesis of the literature findings with relation to
quality assurance in software ecosystems, we rely on our defini-
tion of quality assurance (see Section 1.2), stating that all activities
that contribute to quality are important. We therefore find it natu-
ral to map the literature findings according to life-cycle processes.
As a basis, the ISO12207 standard for software life-cycle processes
has been used (ISO/IEC/IEEE, 2008). This standard was already in-
troduced in Section 1.2, and it defines four high-level process ar-
eas, namely agreement processes; organizational project-enabling
processes; project processes; and technical processes, with sub-
processes defined for each of them. In some of these areas, not so
much findings exist, whereas in particular the technical processes
the findings are much more detailed. Therefore, we have chosen
to focus on the technical processes area, where the material has
been broken down to the subset of applicable sub-processes, and
the organizational processes, which are not broken down further.
To avoid misunderstandings, it should be emphasized that the pro-
cesses in this standard primarily describe activities which are typ-
ically needed in software and systems development, but does not
prescribe or assume any particular order in which they should be
carried out. The mapping is shown in Table 3.
Other studies of the software ecosystem literature have used
three perspectives to structure their findings, namely the tech-
nical (or software engineering, or architecture, and including as-
pects related to the development process); business (including
management, organization, monitoring, decision making, intellec-
tual property, and value creation); and social (or relationships, in-
cluding communities, communication and interaction among ac-
tors) (Barbosa et al., 2013; Campbell and Ahmed, 2010; Manikas
and Hansen, 2013). As pointed out by Campbell and Ahmed (2010),
all three dimensions are closely integrated through software engi-
neering processes, and they are also cross-cutting the processes in
our analysis. Even though we have chosen to use the life-cycle pro-
cesses as the main analysis dimension, the three software ecosys-
tem perspectives are also relevant, and are therefore indicated for
each literature finding in Table 3.
It is interesting to compare our mapping with the two broad
software ecosystems literature studies by Manikas and Hansen
(2013) and Barbosa et al. (2013). Both are structured according to
the three perspectives just mentioned, and in the details of their
descriptions of the perspectives they touch upon several of the life-
cycle processes used in our mapping. In the technical perspective,
both these studies put a large emphasis on the system architectural
design process, which is evidently very important, not only for
achieving quality, but for the software ecosystem as a whole. The
requirements definition process is also pointed out as a particular
challenge (Manikas and Hansen, 2013). The business and manage-
ment dimension in the reviews cover aspects such as monitoring
the ecosystem health (as discussed in Section 1.2), and manag-
ing the ecosystem through strategic planning, release planning,
75
etc., in particular by the keystone. The relationships perspective,
finally, deals with the cooperation between actors, and is appar-
ent in many of the life-cycle processes in our study. In particu-
lar, the communication between the actors is highlighted in these
studies, and also shows up in our results in the implementation,
operation, and organizational processes. However, it should be em-
phasized once more that neither of these studies have found any
significant results on quality assurance activities within software
ecosystems in the literature, and both stress the need for research
in that area in their conclusions.
4. Toward a research agenda
The results found in the literature are interesting and address a
number of valid needs for quality assurance in software ecosys-
tems. However, they are quite few and scattered over various
subtopics. In order to provide a more complete picture of the area,
a number of constructive exercises were conducted in the form of
workshops, some of them together with industrial partners, and
discussions between the involved researchers. The result can be
seen as a set of challenges that need to be addressed in future
research, and they thus constitute a research agenda for quality as-
surance in software ecosystems. Some of the items are open ques-
tions, whereas others also have informal hypothesis on what solu-
tions merit further investigations.
Jansen, Finkelstein and Brinkkemper already presented a gen-
eral research agenda for software ecosystems in Jansen et al.
(2009). Their research agenda mentions quality management as
one of several challenges in software ecosystems, but it does not
provide a more detailed list of research topics for this area. Sim-
ilarly, Barbosa et al. (2013) also identify quality as an important
challenge for software ecosystems, although their focus appears to
be more on how to measure quality than to assure it. A contri-
bution of this paper is thus a comprehensive list of research chal-
lenges specific to quality assurance in software ecosystems.
The agenda is structured around the major life-cycle processes.
Most of these processes were already introduced in Table 3, but we
have extended them with an additional process from ISO 12207,
the software installation process. It is worth noting that this pro-
cess is placed prior to the system integration process, instead of
after integration, which would have been normal in traditional de-
velopment models where integration takes place before the prod-
uct is delivered to the user. Table 4 provides an overview of the
research agenda, and each topic is explained in more detail in a
separate section below. The table is based on the findings from lit-
erature, as shown in Table 3, but we have added new topics, and
in some cases merged and extended several related topics from lit-
erature to provide a more comprehensive picture. Again, the table
also indicates the primary software ecosystem dimension of each
research topic. When looking at the topic titles, one can get the
impression that most topics are primarily technical, but as the ta-
ble indicates, there are in many cases business or social considera-
tions that make the topics relevant. This is further detailed in the
description of each topic below.
4.1. Stakeholder requirements definition
The purpose of the Stakeholder requirements definition process
is to define the requirements for a system that can provide the
services needed by users and other stakeholders in a defined en-
vironment. In a software ecosystem, the normal stakeholders are
complemented with the roles of keystone and niche player, and
the challenges in this life-cycle process focuses on the interactions
between these two actor categories.
76 J. Axelsson, M. Skoglund / The Journal of Systems and Software 114 (2016) 69–81

Table 3
Literature findings mapped to life-cycle processes based on ISO 12207.

Primary software
Life-cycle process ecosystem dimension Literature finding Reference

Stakeholder requirements
definition
System architectural design
Implementation
System integration
Software operation
Organizational processes
Business The requirements defined by the keystone for accepting a niche (Corral et al., 2015)
player application in the shared market sets a minimal quality
standard for the ecosystem.
Technical The platform system architecture is an important enabler for (Eklund and Bosch, 2014)
making it possible to assure quality for independently developed
applications from niche players.
Technical There is a need of emulators of the platform product for niche (Corral et al., 2015)
players to test their applications without having access to the
actual product.
Social Sharing information about test suites between actors is necessary (Greiler and van Deursen, 2013)
for developers to understand what combinations of applications
have actually been tested.
Technical Code reviews can be used to ensure that extensions meet the (Jansen and van Capelleveen, 2013)
quality standards of the ecosystem.
Business Since application development is decoupled in time, integration (Eklund and Bosch, 2014)
and quality assurance become continuous activities throughout
the platform’s lifetime.
Technical Integration testing is a particular challenge in ecosystems, since (Greiler and van Deursen, 2013)
the integration is not done by any of the developing actors but
dynamically on request by the users.
Social Operation knowledge is distributed among actors. To use that (van der Schuur et al., 2011)
knowledge for quality assurance, it must be shared between
them.
Social User ratings can be applied to assess the quality of extensions. (Jansen and van Capelleveen, 2013)
Business Keystones may need to monitor the quality of niche player (van der Schuur et al., 2011)
products.
Technical Online testing can be used for continuous quality assurance of (Cordeiro et al., 2013)
services in ecosystems and it should be underpinned by policies
regarding e.g. test case selection to regulate the testing strategies
that need to be used to ensure product quality.
Business Mechanisms are needed to reward or punish niche players who (van der Schuur et al., 2011)
do not act to assure quality for the ecosystem as a whole.
Social Keystones may need to support niche players in different ways to (van der Schuur et al., 2011)
make it possible for them to assure quality.

Table 4
Overview of the proposed research agenda for quality assurance within software ecosystems.

Life-cycle process Primary software ecosystem dimension Research topics

Stakeholder requirements definition Business Niche player requirements on platform

Social Keystone requirements on software extensions
Business Balancing between different stakeholder’s requirements
System architectural design Technical Ecosystem architecture quality attributes
Technical Diagnosis mechanisms in the platform
Technical Protective mechanisms in the platform
Implementation Social Support for platform development
Business Support for extension development
Technical Test infrastructure
Software installation Social Pre-release testing
Technical Keystone verification of extensions
System integration Technical Dynamic configuration management
Social Test suite sharing
Technical Test case selection
Technical Automatic test case generation
Software operation Social Sharing of operational knowledge
Technical On-line testing
Organizational processes Business Niche player certification

4.1.1. Niche player requirements on platform
The platform provided by the keystone must be attractive to
the niche players for them to be interested in being part of the
ecosystem. The level of attractiveness is dependent on the business
model involved when being a part of the ecosystem but also on
the keystone’s responsiveness to niche players’ needs and wishes
and on the quality of the interfaces used to collaborate between
the keystone’s platform and the niche player’s software. The niche
players require the platform interfaces to be well documented,
easy to start with and continue using, backward compatible, and
have the necessary level of openness so that the niche players are
able to use the needed features of the platform for adding value
(see also Barbosa et al., 2013). There is a need for more research,
primarily empirical, to better understand those niche player needs,
and to define the appropriate technical support in the platform and
infrastructure to deal with them effectively.
4.1.2. Keystone requirements on software extensions
From the end user’s perspective the quality experience is de-
termined by the joint efforts of the keystone and the niche player.
Thus from the keystone’s view it is of interest that software exten-
sions are of high quality with respect to functionality, that it looks
 J. Axelsson, M. Skoglund / The Journal of Systems and Software 114 (2016) 69–81
and feels nice, and that it has good behavior. This can be achieved
by for example imposing guidelines and rules to which niche
players must conform in order to deploy their software in the
ecosystem (Jansen et al., 2009). A challenge is how the keystone
can verify that a niche player’s software conforms to the rules in
order to prevent poor quality software to be released to the users,
and niche players are also interested in verifying that they con-
form to the rules to avoid delays, rework and uncertainty. Depend-
ing on the degree of formalism of the quality requirements, differ-
ent levels of arbitrariness apply. For example a rule stating solely
that the software package must include a specific file can be auto-
matically verified by both the keystone and the niche player. How-
ever, a guideline stating for example that the software must not
"contain content or materials of any kind that … may be found
objectionable or inappropriate" (Apple Inc., 2014) gives room for
various interpretations and may require human judgment to ver-
ify, leading to conflicts about subjective interpretations. This ex-
ample illustrates the sometimes intricate power relations that can
occur between keystone and niche players in an ecosystem. A re-
search direction is how to communicate quality requirements be-
tween keystone and niche players, by for example formal lan-
guages and tools for expressing and verifying such requirements
(Dietrich et al., 2007).
4.1.3. Balancing between different stakeholder’s requirements
A strong keystone may dictate the conditions that all niche
players must conform to. However, in an ecosystem with a strong
niche player (a dominator in the terminology of (Iansiti and Levien,
2004)), this player might influence the evolution of the platform
leading to more advantages for itself but disadvantages for the
smaller niche players as well as for the ecosystem as a whole. A
direction for research is how to balance all ecosystem players’ re-
quirements, wishes and priorities with respect to platform evolu-
tion, to ensure a win–win situation where all stakeholder’s values
are protected (Fricker, 2010). A challenge in this is that the set of
stakeholders is dynamic, and adapting too much to existing stake-
holders could make the ecosystem less attractive to prospective
newcomers.
A key element in quality management is the definition of a
quality model, which could be based on one of the standards men-
tioned in Section 1.2, or a specific model suitable in the particular
context of the system. Such a quality model would be an important
basis for balancing the stakeholder requirements. More research is
needed on how to formulate appropriate quality models for use in
an ecosystem setting, and how different stakeholders should relate
to it (Barbosa et al., 2013).
4.2. System architectural design
The system architecture consists of the fundamental elements
of the system and their interrelations, and the principles of its de-
sign and evolution (ISO/IEC/IEEE, 2011). As a life-cycle process for
software ecosystems, it is primarily connected to the design of the
platform. The architectural concerns specific to ecosystems are fo-
cused around the interface provided by the platform to the exten-
sions, but also to the infrastructure of the ecosystem, since it has
close relations to the platform. As pointed out in Bosch (2009), the
architecture is a key to enable different actors to work indepen-
dently in development of the platform and extensions by formaliz-
ing the rules of interoperability.
4.2.1. Ecosystem architecture quality attributes
An ecosystem architecture forms the basis of the ecosystem
platform and infrastructure and the quality attributes of the ar-
chitecture are thus of high importance. The quality attributes of
an architecture are different compared to product quality in the
77
sense that quality attributes can be seen as aspects where more
or less of some attribute is somewhat better than the opposite but
there is no absolute level that must be met and thus can be ver-
ified. For example, in Eklund and Bosch (2014) a quality attribute
is “Configurability” which is described as “the platform must sup-
port variability in the hardware configuration of sensors and actua-
tors since individual products can vary within the product family”
but the level of minimally acceptable configurability and how to
measure the level is not obvious. Additional research is needed
to better understand what quality attributes are important to pro-
vide a platform that supports quality assurance within a software
ecosystem. Also, as discussed in Section 1.2, only some quality at-
tributes relate to the quality in use, which is the focus of this work.
In particular, dealing with dependability, security, and resource us-
age is important, especially for ecosystems built around platforms
with limited computational power which are used in critical appli-
cations, such as embedded systems (Axelsson and Kobetski, 2014;
Eklund and Bosch, 2014).
4.2.2. Diagnosis mechanisms in the platform
For a niche player developing software for a platform in an
ecosystem, it is of interest to be able to get diagnostic data from its
own extension’s behavior, and to isolate this from the platform’s
behavior, to ease fault-tracing and debugging. Thus the platform
must provide support for niche players to diagnose the extensions
and their interactions with the platform. This is also of interest
from the keystone’s perspective in order to evaluate niche player
software with respect to its use of the platform, as discussed in
Jansen et al. (2009), van der Schuur et al. (2011).
4.2.3. Protective mechanisms in the platform
As described in Eklund and Bosch (2014), there is a need for
protective mechanisms in the platform architecture to deal with
differentiated access to resources for extensions depending on
niche player privileges, to isolate extension execution, and to sup-
port data integrity. One way of coping with this is to only allow
software extensions to run in a sandbox environment with dic-
tatorship control over the software’s external behavior (Axelsson
and Kobetski, 2013). However, there is a need for additional re-
search on how to design the platform interface to the extensions
for these mechanisms, and what design trade-offs exist with re-
lation to architectural quality attributes. A particular challenge is
the dynamic nature of the ecosystem, where it is unknown what
extensions may use the platform in the future.
4.3. Implementation
The implementation process contains activities such as detailed
design, construction, and qualification testing, with a close interac-
tion between these activities, being iterated in short cycles. In an
ecosystem context, there is a need to cover both the evolution of
the platform over time as handled by the keystone, and the devel-
opment of software extensions by niche players.
4.3.1. Support for platform development
Evolution of the platform is to a large extent connected to un-
derstanding the niche-players’ requirements (see Section 4.1.1) and
balancing these against ecosystem architecture quality attributes
(Section 4.2.1). Too much restriction on what software extensions
are allowed to do can have negative impact on the possible busi-
ness opportunities and could lead to niche players losing interest
in the platform and fleeing the ecosystem, thus reducing its health
(Manikas and Hansen, 2013). On the other hand, by opening up the
platform to niche players, some degree of additional risk regarding
poor quality is introduced with respect to negative effects on the
keystone’s and the ecosystem’s reputation, which also affects the
78 J. Axelsson, M. Skoglund / The Journal of Systems and Software 114 (2016) 69–81
health negatively. A key research challenge is how to feed the key-
stone with sufficient information to make these decisions, and use
that information in the verification of platform updates. Being able
to run combinations of software extensions in test beds of new
platform releases could provide such real world knowledge. How-
ever, a successful ecosystem may have a huge number of available
software extensions that could be integrated with the platform in
an endless number of combinations making it practically impossi-
ble to test run every possible combination. Having access to con-
figuration information about what real world combinations exist,
how they are used and what effects they have may be used to sim-
ulate real world usage of the platform. This knowledge can then be
used to make design and implementation decision when evolving
the platform, and for verification purposes. For this, support in the
ecosystem infrastructure is needed.
4.3.2. Support for extension development
A key to achieving quality is to have proper review mecha-
nisms for extensions submitted by niche players, possibly comple-
mented by certification of both the extensions and the developers,
as discussed in Jansen and van Capelleveen (2013). However, this is
effective only in cases where source code can be shared, and there-
fore needs to be complemented with other mechanism. In partic-
ular, a test environment of the platform is needed, both includ-
ing real hardware as well as emulators if non-standard hardware
is used in the platform (Corral et al., 2015). If the niche players
need to acquire the necessary hardware themselves the cost of en-
tering the ecosystem might be too high for small niche players
if the hardware is expensive. This can be addressed by providing
necessary test environments for niche players to use. Emulators
can be provided as software packages either from keystone play-
ers or from other third party players. Also, test environments us-
ing real hardware can be provided as an online service by either
the keystone or third parties. Not only test environments but also
other parts of the software development tool chain may be pro-
vided in this manner, e.g. ecosystem specific compilers, static and
dynamic analysis tools, as well as continuous integration and test
tools. These can be tailor-made to support the hardware as well
as supporting the policies imposed by the keystone, e.g. graphi-
cal user interface, test coverage, memory usage and other nonfunc-
tional requirements. Apart from lowering the cost, test support as a
software service can help in mitigating intellectual property issues,
by reducing the amount of information that needs to be shared be-
tween the different actors. This includes all kinds of information,
including source code, test cases, and operational knowledge. Fur-
ther research is needed on what support is beneficial, and how it
can be provided in the best possible way. In particular, providing
tools and test environments as a service is an interesting research
direction. This is relevant in situations where the platform test en-
vironment is prohibitively expensive for a niche player to acquire,
which is often the case for embedded systems that are connected
to physical equipment. In other cases, the keystone may want to
avoid a transfer of a platform to protect intellectual property, and
similarly, the niche player may want to limit the amount of in-
formation transferred to the keystone regarding a new extension
under development.
4.3.3. Test infrastructure
In the overall software ecosystem, there will be many sources
of quality related information, and an open research question is
how to organize the overall test system infrastructure to allow ef-
ficient development. This includes how to exchange requirements,
test cases, and other test related information between actors. In
some cases, it could also include testing as a service solutions, e.g.,
by letting a niche player test their extension on a platform without
having direct access to the platform, as described in the previous
section (this is briefly discussed in Kajan et al. (2011)). Due to the
distributed nature of the information in the first place, the infras-
tructure is also likely to be distributed, and could be designed as
a system-of-systems which connect the different actors’ own test
systems. The principles and interfaces for those connections are ar-
eas where more research is motivated.
4.4. Software installation
The installation of software applications in the target environ-
ment is one aspect where software ecosystems differ from tradi-
tional development approaches. Traditionally, it is done in a well-
defined step, where the integrated software is being transferred to
the user’s system. In an ecosystem, it is decoupled in time (Eklund
and Bosch, 2014), meaning that the user selects extensions, and
installs them on the system in an iterative fashion (Bosch, 2009).
In this context, we find it meaningful to define software installa-
tion as the time when an extension becomes available to users for
installation, and the actual installation is part of the system inte-
gration process, which then occurs after installation.
4.4.1. Pre-release testing
When the extension development is finished from the niche
player’s point of view some additional steps may be needed that
involve the keystone and the ecosystem. For example, the niche
player may want to perform A/B-testing of a business hypothesis
before releasing the software publicly. The keystone may orches-
trate this through functionality in the infrastructure to release soft-
ware in stages to various subsets of users. A research direction is
to explore the use of knowledge of user profiles and configurations
for the purpose of A/B-testing of software extensions. Also, other
collaborations between niche and keystone players are possible in
the deployment stage to enhance market penetration and user ex-
perience, e.g. by connecting niche players with possible beta test
users.
4.4.2. Keystone verification of extensions
A keystone may also be interested in doing some verification of
a software extension before it is released in the ecosystem. A part
of this is the manual reviews discussed in Jansen et al. (2009) and
in Section 4.3.2, but there is also a need for automated approaches.
This could be stability and reliability tests and verification of com-
pliance to rules and guidelines. The research challenge is how to
handle this in the infrastructure, with a focus on automation is-
sues, and how to communicate feedback to the extension devel-
oper in the best way. It is worth noting that the test suite used
for this could also evolve, as new issues are discovered. This may
lead to the need for retesting of already deployed extensions, and
a test failure may then lead to a need to put those extensions in
a quarantine, and even recall them from systems where they are
installed (this is briefly mentioned in German et al. (2013)).
4.5. System integration
The purpose of the system integration process is to integrate
the system elements to produce a complete system that will sat-
isfy the system design and the customers’ expectations expressed
in the system requirements. After an extension has been installed,
and is thus available to users in the ecosystem infrastructure, a
user may choose to deploy it in a specific system instance, at
which point an integration takes place. Only in connection to this
can integration testing be performed, which leads to a number of
specific considerations in software ecosystems.
 J. Axelsson, M. Skoglund / The Journal of Systems and Software 114 (2016) 69–81
4.5.1. Dynamic configuration management
The system integration of the software extension with the
ecosystem platform is done at runtime by the users. As mentioned
above, the combination of software extensions can vary indefinitely
leading to integration challenges. Keystones and niche players can
make use of knowledge regarding possible combinations that the
users may choose to install. For example, having two different ex-
tensions using the same actuators in an embedded system ecosys-
tem may lead to race conditions and deadlocks. Policies for prior-
itizing the use of actuators during installation, or even the pos-
sibility to prohibit certain combinations to be installed, may be
required. Research on how to do this analysis is needed, includ-
ing the application of static analysis of the actual software, us-
ing metadata about the software’s behavior such as annotations
or manifest files, through testing, or a combination. Similar tech-
niques can be used for ensuring that matching versions of software
extensions is used.
Integration testing challenges include several different types of
interactions: between an individual extension and the platform;
directly between extensions without involvement of the platform;
and the combinatorial problem with several extensions interacting
with the platform. A keystone with knowledge about possible com-
binations as describe in Section 4.3.1 can perform integration test-
ing of selected combinations and then use test results as a basis
for estimating valid and invalid combinations.
Closely related to configuration management is also change
management, i.e. the process of selecting which changes to encour-
age, and which to prevent. As discussed in Bosch (2009), the activ-
ities here have to shift focus from overall system functionality to
instead ensure backward compatibility and compositionality of the
different software artifacts, in order to enable decentralized devel-
opment.
4.5.2. Test suite sharing
A keystone that has access to niche players’ test suites can exe-
cute tests of a niche player’s software extension in a context where
other extensions also execute to find integration issues. Niche play-
ers having access to other niche players’ test suites can also per-
form integration tests of relevant combinations of their software
together with other software earlier in the development process. A
direction for future research is the distribution and sharing of test
suites between the ecosystem actors. This includes the challenge of
test suite understanding (Greiler and van Deursen, 2013) in order
for actors to be able to use such test suites in valuable ways.
4.5.3. Test case selection
With the combinatorial issues described above as a backdrop,
test case selection and prioritization are also directions for the fu-
ture. Since the time of executing tests increases with the number
of tests, there will be an increasing need for strategies to select
subsets of the complete set of test cases that should be executed,
to find as many defects as possible. It is also relevant to find opti-
mal ordering for executing test cases to be able to find defects as
early as possible during a test run since finding fatal defects at the
end of a test run is a waste of time.
4.5.4. Automatic test case generation
Without access to other actors’ test suites, integration tests can
be performed by creating own test suites to run on other actor’s
software. Automatic generation of test cases is thus also a direction
for future research. This includes automatic generation of test input
that is submitted to a software’s user and service interfaces, and
mechanisms to record the behavior and detect crashes. Depending
on the available information about the extension under test, more
or less random testing approaches may be used.
79
4.6. Software operation
The purpose of the Software Operation Process is to operate
the software product in its intended environment and to provide
support to the customers of the software product. The variety of
operating conditions that might occur around a system instance
is only known once it is put in operation, and on-line testing be-
comes an important tool for continuous validation under real con-
ditions.
4.6.1. Sharing of operational knowledge
The issue of collecting operational data of a software’s be-
havior and sharing this knowledge within the ecosystem is dis-
cussed in van der Schuur et al. (2011) and should also be a di-
rection for future efforts within software ecosystem research. This
includes different kinds of user and developer ratings (Jansen and
van Capelleveen, 2013), but also other mechanisms are needed, in
particular for ecosystems that have a small user base, or a com-
mercial or technical structure that does not allow users to easily
assess extensions.
4.6.2. On-line testing
Due to late integration and the combinatorial challenges de-
scribed above, testing in operation may be the only possibility in
some cases. Testing in operation can also provide opportunities not
possible in pre-release environments. For example, during opera-
tion a request can be sent and the response time can be measured.
On-line testing to assure availability of dependent software is also
relevant in a software ecosystem context as well as security and
reliability testing. Interesting research questions include what in-
terfaces and services are needed to control the testing in the prod-
ucts from the infrastructure, and manage the acquired data.
4.7. Organizational processes
The organizational processes manage the organization’s capabil-
ity to acquire and supply products or services through the initia-
tion, support and control of projects. They provide resources and
infrastructure necessary to support projects and ensure the satis-
faction of organizational objectives and established agreements.
Many of the challenges in software ecosystems relate to the
fact that system development involves several organizations, where
information needs to flow across the organizational borders. The
mechanisms for control of the ecosystem are important, and in-
volve primarily the relation between keystone and niche players.
4.7.1. Niche player certification
A quality mechanism that can be used from the organizational
perspective, is to give niche players different access to privileged
resources. Only trusted and certified niche players would then be
allowed to access resources that might affect the overall experi-
ence of the ecosystem, as described in Eklund and Bosch (2014).
Certification (Jansen and van Capelleveen, 2013) can be done by
manual evaluation and through contractual agreements between
keystone and niche players, a process that might be both cum-
bersome and dependent on manual judgments. Strategies for au-
tomatically certifying niche players based on past performance can
be a direction for future research. For example, niche players show-
ing a history of delivering high quality software over time and are
responsive when incidents and problems occur can be automati-
cally certified.
5. Conclusions
Software ecosystems are becoming an increasingly common
model for software development, in which different actors coop-
erate around a shared platform. In this paper, we have studied
80 J. Axelsson, M. Skoglund / The Journal of Systems and Software 114 (2016) 69–81
the implications that this development model has on quality assur-
ance. Of course, there are many quality assurance challenges that
exist in traditional development practices, which also carry over
to ecosystems, but the focus in this study has been to understand
what additional issues arise, or require increased attention in the
ecosystem setting.
The first research question was “What does the scientific liter-
ature say about quality assurance in software ecosystems?” It was
answered through a systematic literature mapping, which how-
ever resulted in only six papers. They presented findings on sys-
tem architecture, knowledge propagation, test suite understanding,
and concerns from mobile systems and service-oriented architec-
tures. Although these papers contain valuable information, a map-
ping of their findings to the life-cycle processes made it apparent
that there is a need for more research.
The second research question of the paper was therefore “What
are the research needs related to quality assurance in software ecosys-
tems?” This was answered by complementing the results from the
literature search with further challenges that we could identify, re-
sulting in a list of important topics to address in order to advance
the state-of-the-art of quality assurance in software ecosystems.
Most of our findings have their root in some of the key charac-
teristics of software ecosystems, and expand upon an early charac-
terization of quality assurance briefly provided in Bosch (2009).
Since the complete system instance becomes a combination of
the keystone’s platform, the niche players’ available extensions,
and the selection by the user of which extensions should be in-
stalled in the configuration, nobody has the complete picture.
Therefore, the actors must share information on every aspect of the
system, which poses challenges related to communication across
organizational borders, incentives for sharing information, and rea-
sons for not doing so. A second aspect is that the final system
integration is carried out by the user by selecting a particular
combination of extensions, which means that the keystone and
niche players cannot carry out traditional quality assurance activi-
ties such as integration testing. Finally, the execution context is to
some extent unknown, and both the platform and extensions must
be prepared to co-exist with an unknown combination of other ex-
tensions. Many of these challenges must, or could, be addressed in
the technical infrastructure on which the ecosystem relies. This in-
cludes means for communication between stakeholders, but also
testing as a service approaches and tracking of configuration com-
binations in actual use.
As a final remark, it is easy to focus on the problems and chal-
lenges that are caused by using a new development model. To
balance this, we would like to highlight that a software ecosys-
tem also provides new possibilities to improve quality assurance.
In particular, the infrastructure needed in the ecosystem gives op-
portunities for better feedback of data from operations, and this is
particularly valuable in embedded systems that traditionally suf-
fer from a difficulty in accessing operational data due to the fact
that they are enclosed within another product. To maximize these
benefits, it is necessary to provide mechanisms for implicit data
collection in the platform, and for sharing data between actors.
5.1. Future work
The results presented above show that the existing research on
quality assurance in software ecosystems is very limited, and the
proposed research agenda gives a number of directions that would
be valuable to study in the future. In addition to these more de-
tailed topics, it would also be interesting to expand the scope of
this study in different directions.
First, we have focused primarily on quality assurance, and even
though we have touched upon the more general topic of software
quality, a more thorough study would be needed to give a full ac-
count of the challenges. This also comes even closer to the topic of
ecosystem health, and it would be interesting to further analyze
how product quality and quality assurance influence the health,
and how important quality is compared to other factors. Like-
wise, it would be worthwhile to investigate how quality assurance
in software ecosystems compares to how it is done in other ap-
proaches to software reuse, such as software product lines. Finally,
even though the definition of ecosystem that we have chosen is
common in literature, it is somewhat limiting in its assumptions of
a common technical platform, and it would be valuable to under-
stand better how quality assurance can be handled in other types
of ecosystems, including those that resemble systems-of-systems.
Acknowledgments
This research has been supported by Vinnova, the Swedish
Agency for Innovation Systems (grant no. 2013-03492), and a con-
sortium of Swedish companies.
References
Apple Inc., 2014. iOS Developer Program License Agreement.
Axelsson, J., Kobetski, A., 2013. On the conceptual design of a dynamic component
model for reconfigurable Autosar systems. SIGBED Rev. 10 (4), 45–48. doi:10.
1145/2583687.2583698.
Axelsson, J., Kobetski, A., 2014. Architectural concepts for federated embedded sys-
tems. In: Proceedings of 2nd International Workshop on Software Engineering
for Systems of Systems. Vienna, Austria doi:10.1145/2642803.2647716.
Axelsson, J., Papatheocharous, E., Andersson, J., 2014. Characteristics of software
ecosystems for federated embedded systems: a case study. Inf Software Tech-
nol 56 (11), 1457–1475 http://dx.doi.org/10.1016/j.infsof.2014.03.011.
Barbosa, O., dos Santos, R.P., Alves, C., Werner, C., Jansen, S., 2013. A systematic map-
ping study on software ecosystems from a three-dimensional perspective. In:
Jansen, S., Brinkkemper, S., Cusumano, M. (Eds.), Software Ecosystems – Ana-
lyzing and Managing Business Networks in the Software Industry. Edward Elgar
Publishing, pp. 59–81. doi:10.4337/9781781955635.
Boehm, B.W., 1979. Software engineering; R&D trends and defense needs. Research
Directions in Software Technology. MIT Press.
Bosch, J., 2009. From software product lines to software ecosystems. In: Proceedings
of the 13th International Software Product Line Conference, pp. 111–119.
Bosch, J., Bosch-Sijtsema, P., 2010a. From integration to composition: on the impact
of software product lines, global development and ecosystems. J. Syst. Software
83 (1), 67–76 http://dx.doi.org/10.1016/j.jss.2009.06.051.
Bosch, J., Bosch-Sijtsema, P., 2010b. Softwares product lines, global development and
ecosystems: collaboration in software engineering. In: Mistrik, I., Grundy, J.,
Hoek, A., Whitehead, J. (Eds.), Collaborative Software Engineering. Springer,
Berlin, Heidelberg, pp. 77–92. doi:10.1007/978-3-642-10294-3_4.
Campbell, P.R.J., Ahmed, F., 2010. A three-dimensional view of software ecosystems.
In: Proceedings of the 2nd International Workshop on Software Ecosystems
doi:10.1145/1842752.1842774.
Cordeiro, J., Krempels, K.-H., Bertolino, A., De Angelis, G., Polini, A., 2013. Gover-
nance Policies for Verification and Validation of Service Choreographies, 140.
Springer, Berlin, Heidelberg, pp. 86–102. doi:10.1007/978-3-642-36608-6_6.
Corral, L., Sillitti, A., Succi, G., 2015. Software assurance practices for mobile appli-
cations. Computing 97 (10), 1001–1022. doi:10.1007/s00607-014-0395-8.
Dantas, V., Marinho, F., da Costa, A., Andrade, R., 2009. Testing requirements for mo-
bile applications. In: 24th International Symposium on Computer and Informa-
tion Sciences, 2009, ISCIS 2009, pp. 555–560. doi:10.1109/ISCIS.2009.5291880.
Dietrich, J., Hosking, J., Giles, J., 2007. A formal contract language for plugin-based
software engineering. In: Proceedings of the 12th IEEE International Conference
on Engineering Complex Computer Systems, pp. 175–184. doi:10.1109/ICECCS.
2007.7.
Dybå, T., Dingsøyr, T., Hanssen, G., 2007. Applying systematic reviews to diverse
study types: an experience report. In: First International Symposium on Em-
pirical Software Engineering and Measurement, 2007, ESEM 2007, pp. 225–234.
doi:10.1109/ESEM.2007.59.
Eklund, U., Bosch, J., 2014. Architecture for embedded open software ecosystems. J.
Syst. Software 92 (0), 128–142 http://dx.doi.org/10.1016/j.jss.2014.01.009.
Engström, E., Runeson, P., Skoglund, M., 2010. A systematic review on regression
test selection techniques. Inf. Software Technol. 52 (1), 14–30 http://dx.doi.org/
10.1016/j.infsof2009.07.001.
Fricker, S., 2010. Requirements value chains: stakeholder management and require-
ments engineering in software ecosystems. In: Paech, B., Rolland, C. (Eds.), Re-
quirements Engineering: Foundation for Software Quality, pp. 60–66. doi:10.
1007/978-3-642-14192-8_7.
German, D.M., Adams, B., Hassan, A.E., 2013. The evolution of the R software ecosys-
tem. In: 17th European Conference on Software Maintenance and Reengineering
(CSMR), pp. 243–252. doi:10.1109/CSMR.2013.33.
Greiler, M., van Deursen, A., 2013. What your plug-in test suites really test: an inte-
gration perspective on test suite understanding. Empirical Software Eng. 18 (5),
859–900. doi:10.1007/s10664-012-9235-7.
 J. Axelsson, M. Skoglund / The Journal of Systems and Software 114 (2016) 69–81
Iansiti, M., Levien, R., 2004. Strategy as ecology. Harvard Bus. Rev. 82 (3), 68–78
http://www.cs.cmu.edu/∼jhm/Readings/HBR-Strategy_as_Ecology.pdf.
ISO/IEC, 2001: ISO/IEC 9126. Software engineering – product quality.
ISO/IEC/IEEE, 2008: ISO/IEC/IEEE 12207. Systems and software engineering –
software life cycle processes.
ISO/IEC, 2011: ISO/IEC 25010 System and software quality models.
ISO/IEC/IEEE, 2011: ISO/IEC/IEEE 42010. Systems and software engineering – archi-
tecture description.
Jansen, S., Finkelstein, A., Brinkkemper, S., 2009. A sense of community: a research
agenda for software ecosystems. In: 31st International Conference on Software
Engineering – Companion Volume, 2009. ICSE-Companion 2009, pp. 187–190.
doi:10.1109/ICSE-COMPANION.2009.5070978.
Jansen, S., van Capelleveen, G., 2013. Quality review and approval methods for ex-
tensions in software ecosystems. In: Jansen, S., Brinkkemper, S., Cusumano, M.
(Eds.), Software Ecosystems – Analyzing and Managing Business Networks
in the Software Industry. Edward Elgar Publishing, pp. 187–217. doi:10.4337/
9781781955635.
Jansen, S., 2014. Measuring the health of open source software ecosystems: beyond
the scope of project health. Inf. Software Technol. 56, 1508–1519. doi:10.1016/j.
infsof.2014.04.006.
Kajan, E., Lazic, L., Maamar, Z., 2011. Software engineering framework for digi-
tal service-oriented EcoSystem. In: 19th Telecommunications Forum (TELFOR),
pp. 1320–1323. doi:10.1109/TELFOR.2011.6143796.
Manikas, K., Hansen, K.M., 2013. Software ecosystems – a systematic literature re-
view. J. Syst. Software 86 (5), 1294–1306. doi:10.1016/j.jss.2012.12.026.
Manikas, K., Hansen, K.M., 2013. Reviewing the health of software ecosystems –
a conceptual framework proposal. In: Proceedings of 5th International Work-
shop on Software Ecosystems. Potsdam, Germany http://ceur-ws.org/Vol-987/3.
pdf.
Pedersen, K., Feldt, R., Mujtaba, S., Mattsson, M., 2008. Systematic mapping studies
in software engineering. In: Proceedings of the 12th International Conference
on Evaluation and Assessment in Software Engineering. Swinton, UK, pp. 68–77.
http://dl.acm.org/citation.cfm?id=2227123.
81
Shaw, M., 2003. Writing good software engineering research papers: minitutorial.
In: Proceedings of the 25th International Conference on Software Engineering,
ICSE ’03. IEEE Computer Society, Washington, DC, USA, pp. 726–736. http://dl.
acm.org/citation.cfm?id=776816.776925.
Shewart, W.A., 1931. Economic control of Quality of Manufactured Product. Van Nos-
trand Reinhold Co, New York.
van der Schuur, H, Jansen, S., Brinkkemper, S., 2011. The power of propagation:
on the role of software operation knowledge within software ecosystems. In:
Proceedings of the International Conference on Management of Emergent Dig-
ital EcoSystems, MEDES ’11. ACM, New York, NY, USA, pp. 76–84. doi:10.1145/
2077489.2077503.
Wahyudin, W., Mustofa, K., Schatten, A., Tjoa, A., Biffl, S., 2007. Monitoring “health”
status of open source web engineering projects. Int. J. Web Inf. Syst. 1/2 (3),
116–139. doi:10.1108/17440080710829252.
Jakob Axelsson is director of the Software and Systems Engineering Laboratory at
the Swedish Institute of Computer Science (SICS) in Stockholm, and is also profes-
sor in Computer Science at Mälardalen University in Västerås. He received his MSc
in Computer Science in 1993, and a PhD in Computer Systems in 1997, both from
Linköping University. He has worked almost 15 years in industry, primarily in the
automotive sector at Volvo Cars and AB Volvo, but also at ABB. His research inter-
ests are primarily in cyber-physical systems, systems-of-systems, systems architec-
ture, and software ecosystems.
Mats Skoglund received an MSc in Computer and Systems Science in 1999, and a
PhD in Software Engineering in 2006, both from Stockholm University. His main re-
search interests are in the field of software quality assurance and testing. He has
held positions at Stockholm University, Lund University, Swedish Institute of Com-
puter Science (SICS), and as a consultant at various companies in the IT sector. Cur-
rently, he is with Scania, working as a software development methodology coach.