Evaluation of Awareness and Acceptability of Using eGovernment Services in Developing Countries: the Case of Jordan

Saheer Al-Jaghoub, Hussein Al-Yaseen and Mouath Al-Hourani Al-Ahliyya Amman University, Amman, Jordan
saljaghoub@ammanu.edu.jo mouath.hourani@ammanu.edu.jo hyaseen@ammanu.edu.jo
Abstract: Similar to other developing countries, Jordan started a national e-Government initiative aiming to streamline government procedures and make information and government services available to business and citizens online. This paper presents the results of a pilot study that aims to assess factors which could influence the awareness and use of e-Government services in Jordan. It investigates issues such as: accessibility of egovernment, citizen's attitude toward various privacy and security, the required services and costs. The data was collected using quantitative and qualitative methods including a survey and interviews with e-Government officials. The results of this preliminary study suggest that awareness of e-government did not reach the required level. These findings are hoped to be useful for researchers, practitioners and policy makers. Keywords: ICT, e-government, developing countries, Jordan, evaluation, mixed research

1. Introduction
Many governments in developed and developing countries are now developing, implementing and improving their strategies to transform government services using information and communication technologies (ICTs) (Borras, 2004). This transformation of services is referred to as eGovernment, eGov., digital government, online government, or transformational government (Gupta et al., 2008). E-Government, which is the term used in this paper, can be described as the use of any type of information and communication technology to improve services and operations provided to different parties such as: citizens, businesses, and other government agencies (Grant and Chau, 2005; Gronlund and Horan, 2005; Adeshara et al., 2004; Arif, 2008). E-Government has been classified in terms of activities and delivering models into four categories: the Government to Citizens or Government to Customer (G2C); Government-to-Business (G2B); Government-to-Employees (G2E); Government-to-Government (G2G); and Citizen-to-Citizen (C2C) (Lee et al., 2005; Carter and Belanger, 2003). A number of studies have focused on many issues related to e-Government such as: e-Government strategies (Beynon-Davies, 2004; Williams and Beynon-Davies, 2004); e-Government program challenges (Barc and Cordella, 2004); e-Government technical issues (Cottam et al., 2004); e-Government usability websites (Mosse and Whitley, 2004); eGovernment adoption (Ciborra and Navarra, 2005; Elsheikh et al., 2008; Dwivedi and Williams., 2008; Mofleh et al., 2008b). The government of Jordan has started implementing an e-Government program in 2002 which seeks to improve service delivery and increase the involvement of citizens through the use of ICT. This paper presents the results of a research done as a pilot study in one of the Jordanian universities as a preliminary evaluation of Jordanian citizens awareness and acceptability of the e-Government program. The data was collected using both a survey and quantitative methods (focus groups and interviews). The "Levels of e-Government application model" (NAO, 2002) is used as a theoretical framework to aid in the analysis. The paper is structured as follows: the introduction is followed by presenting the research approach, then the theoretical framework is discussed, after which the e-Government initiative in Jordan is briefly presented. Section 5 presents the results, followed by the discussion and conclusion.

ISSN 1566-6379 1 Academic Conferences Ltd Reference this paper as: Al-Jaghoub, S, Al-Yaseen, H and Al-Hourani, M. (2010) Evaluation of Awareness and Acceptability of Using eGovernment Services in Developing Countries: the Case of Jordan The Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010, (pp1 - 8), available online at www.ejise.com

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (1 8)

2. Research approach
Mixed research approach (quantitative and qualitative methods) was used to collect the data for this paper, including a survey, focus group discussions and interviews. Using a survey research method is considered to be an appropriate research approach to address the citizens' awareness and usage of e-Government services (Choudrie and Dwivedi, 2005). However, combining quantitative and qualitative (Mixed Research method) sources of information and analytical methods can build on the strength of each type of data collection and minimize the weaknesses of any single approach (Tashakkori and Teddlie, 2003). A multi-method approach can increase both the reliability and validity of evaluated data. A mixed research approach is argued to be the most appropriate technique when investigating a complex and emerging phenomena such as e-Government (Creswell, 2003; Green and Preston, 2005; Sammons et al., 2005; Mofleh et al., 2008a; Shareef et al., 2009). In order to achieve the objectives of this paper in exploring the issues related to the awareness and acceptability of e-Government in Jordan, the following questions needed to be answered by participants and were included in the questionnaire: What percentage of the Jordanian population uses computers and the Internet? How do people use these tools (applications and Internet)? Where do people feel comfortable using computers and the Internet? For what purposes do they use the Internet? Why do they NOT use the Internet? What are peoples' attitudes toward the Internet and the costs of its use? Would people use government services if they were available on the Internet? How much would they pay for these services? What are the privacy and security concerns with respect to e-Government applications? What are the main barriers of using government services over the Internet?

As mentioned earlier, this is a pilot study for a more comprehensive research that aims to evaluate the Jordanian citizens' awareness and acceptability of e-Government services. The data for the survey and the focus groups was collected at Al-Ahliyya Amman University (AAU). Four sources of data and information were identified: General information on e-Government services (Government online, 2009) Data from a survey: we administered a survey instrument to a sample of undergraduate students at Al-Ahliyya Amman University. Many studies in ICT adoption have used students as a sample (Moon and Kim, 2001) as they have experience using the Internet; differ from the demographics of the population of citizens; and the majority of undergraduate students use and have easy access to Internet services. The questionnaire contains four stages: designing and testing the questionnaire based on two focus groups and some interviews; followed by data collection and then data analysis; and finally the preliminary findings of the questionnaire. Focus group discussions: focus groups were organized both before and after the survey. The first focus group meetings were designed to take place prior to the survey, and their main goal was to discuss with participants their attitudes, usage and awareness of the internet and e-Government services in Jordan to find out the reasons for no or low levels of usage. The questionnaire was also distributed to the first focus group participants. The second round of focus group meetings was conceptualized as a forum for discussing the survey results and, on this basis, developing recommendations. In addition to the above, interviews were conducted with the Director of e-Government Projects and the e-Government initiative manager at the Ministry of Information and Communication Technology (MoICT) in Jordan.

The results and the conclusions from the interviews were integrated and synthesized with the result of the survey.

3. Theoretical framework of e-government project

Similar to any project, an e-Government project is a unique, complex, and one time effort, with specified limitations (time, budget, resources and performance) designed to meet governmental and agencies goals or stakeholders (Citizens, Businesses, Employees, and Agencies) needs (Al-Yaseen et al., 2008).
www.ejise.com

Academic Conferences Ltd

Saheer Al-Jaghoub et al.

The model used in this paper is based on the "Levels of e-Government application model" (NAO, 2002). This model proposes that each e-Government project goes through different phases from the easy phase (Basic site) which mainly involves developing websites, piloting a few applications, and putting these services online to the most advanced phase in which e-government reaches a high level of maturity and becomes fully integrated into the governance framework and activities of each sector (Joined-up e-Government) (Yang and Paul, 2005; Santos and Heeks, 2003). Figure 1 shows the main levels of the e-Government project, which are discussed briefly below (NAO, 2002).

Electronic Publishing

Interaction Publishing Basic Site Transaction

Joined-Up eGovernment

Account Management

Figure 1: Levels of e-government application model (NAO, 2002) Stage 1: Basic site provision of basic information, usually electronic versions of print documents; no email contact is given. Stage 2: Electronic Publishing provision of extensive downloadable, static information via ICTs; basic email contact is given but there is no online interaction. Stage 3: Interactive Publishing provision of dynamic, customizable information via ICTs; with a capacity to interact (e.g. via email and online submissions). Stage 4: Transaction ability to authenticate users and register their identities reliably in order to undertake complete transactions online (e.g. submitting a tax return); users may be able to manage their own 'accounts' (i.e. a set of personal interactions/transactions). Stage 5: Joined-up e-Government interlinking of different government structures/processes to produce electronic one-stop shops; users can access their own files/accounts and manage their relationships with government wholly via the Internet.

4. e-Government initiative in Jordan

Jordan is a developing country in the Middle East with a population of about 5,000,000 and limited natural resources. The e-Government initiative in Jordan has been one of a number of ICT related initiatives launched with the succession of King Abdullah II to the throne in 1999, the aim of which is to transform the country into a knowledge-based economy (Al-Jaghoub and Westrup, 2003). As the projects manager in MoICT stated: "In 2000, the Government of Jordan launched a national eGovernment initiative; in 2003 the initiative has been chartered by His Majesty King Abdullah II as an e-Government program, aiming to make information and services available to citizens on the Internet". Jordan's national e-Government program aims to provide government and agencies

www.ejise.com

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (1 8) services throughout various electronic channels such as internet, SMS gate, mail and others where work is in progress to develop a number of electronic services. The e-Government program aims to deliver high-quality services to consumers, businesses and organizations; improve government performance and efficiency; enhance Jordans competitiveness; ensure public sector transparency and accountability; reduce costs and increase ease of interaction with government; promote development of Jordans ICT sector; develop skills within the public sector; boost e-commerce activities; and improve information security. The e-Government initiative has been launched in terms of activities and delivering models into four categories: the Government to Citizens or Government to Customer (G2C) section aims to provide comprehensive information about all the services provided by the Jordanian Ministries and other Jordanian government agencies. Citizens can benefit from services related to life including birth, school, work, retirement, health services and so on. Government to Business (G2B) aims to provide investors in Jordan with information about all the services provided by the Ministries and other government agencies, including how to start a new business, running a business, investment incentives, privatization program, and investment benefits at free zones and so on. Government to Employees (G2E) aims to provide governmental employees in different government agencies in Jordan with information about all the services provided by the Ministries and government agencies (Director of e-Government program in Jordan).

5. Preliminary results of awareness and usage of e-government services

The survey was distributed to a sample of 1200 students from different departments at AAU. Of the 1200 questionnaires, 1200 completed responses were collected; giving a response rate of 100%. The response level is due to the fact that the questionnaires were answered and collected with the presence of the researcher(s) during lectures. This rate is considered to be above expectation given that the generally accepted average responses to non-incentive based questionnaires are around 20%. We analyzed the data from the responses of the questionnaire using a combination of the parametric statistical methods, Descriptive Analysis and Factor Analysis (Pett, et al., 2003). Students were asked to select from the list the closest choice of many variables. Each of these variables were measured using a five point Likert scales (1 = not important and 5 = very important). For technically interested readers we report that a factor analysis technique was employed in order to identify possible categories. Factor analysis was performed in three steps (following Berthold and Hand, 2003): A matrix of correlation coefficients for all possible pairings of the variables was generated. Factors were then extracted from the correlation matrix using principal factors analysis. The factors were rotated to maximize the relationships between the variables and some of the factors and minimize association with others using Varimax Kaiser Normalization, which maintained independence among the mathematical factors. The Eigenvalues determined which factors remained in the analysis. Following Kaisers criterion, factors with an Eigenvalue of less than 1 were excluded. A Screen plot provides a graphic image of the Eigenvalue for each component extracted

5.1 Computer and internet use

A large majority -100%- of the sample currently uses computers and most of the computer users also use the Internet. This rate is reasonable given the characteristics of the sample which consists of university students who have access to computers and the internet, and tend to rely on it for educational and other purposes. This level of access may not be applicable to the the majority of Jordanians.

5.2 Importance of place and purpose of using internet

Most of the people in this sample report using computers at home, at work or at the university. Of the people who use computers: 100% use it at home; 60.2% use it at University; 14.7% use it at work as most of the students are unemployed-; while 31.8% use it at a public place. As noted, most computer users are also Internet users. Home is main place 84.1% for connecting to the Internet, 13.5% of Internet users at work; 40.8% of Internet users at University; and 30.2% connect to the Internet at a public place.
www.ejise.com

Academic Conferences Ltd

Saheer Al-Jaghoub et al.

5.3 Attitudes toward the Internet, cost and other factors

The reasons for not using the Internet are varied. The main reason is associated with 'Not enough time'; and the least important reason is 'Don't use computers', as shown in Table 1 below: Table 1: Reasons for not using the Internet
Reasons for not using the Internet Not enough time Phone bill too high Internet charge too high For security reasons Concerned about kids Don't use computers % 17.9 15.1 12.3 7.4 2.6 0.0

The most frequent Internet use was browsing the Internet (100%), entertainment (86.2%); sending and receiving emails (56.2%); getting information (36.4%); shopping over the Internet (16.7%) and for paying bills online (1.9%).

5.4 Using government services over the Internet

When we asked the participants if they know what e-Government is in Jordan, (74.1%) answered 'Yes'; while when we asked the same question in a different way within the survey, we found that more than (75%) of the participants do not actually know about e-Government services or its Website. Moreover, the study found that more than (85%) of the participants never logged in to e-Government Website or never got any information. In order to investigate how people might feel about using government services on the Internet, we asked a series of questions about peoples most current use of different services by going to government departments, and then we asked if they would use these services if they become available on the Internet. The respondents were also asked how much they are willing to pay for the convenience of using such services online. The most frequently used service included checking traffic tickets, while the least service was renewing the family document. As Table 2 indicates, many of the actual users of the services are very interested in having an Internet-based delivery system: Table 2: e-Government services
e-Government used services Information about checking traffic tickets Information about the weather Renew passport Renew ID card renew a drivers license Paying bills Information about car tax Apply for job Renew health card Pay taxes Tax refund Income tax settlement Tax situation Renew family document % 81.3 51.1 39.4 28.4 18.9 17.2 16.8 10.1 8.1 7.9 7.3 7.1 6.5 1.6

The results are presented in Table 3. Using a factor analysis cut-off level of 0.5, four factors were considered the main reasons of using e-Government services, which we described as: 'personal information', 'security and health information', 'tax information', and 'other information'. As mentioned earlier in Table 1, -for the privacy and security issues- nearly 7.4% of the study sample agreed that they were worried about privacy on the Internet.

www.ejise.com

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (1 8) Table 3: Factor analysis of e-government services
Factors Security and health Tax information information

Personal e-Government services information Information about checking traffic 0.973 tickets Information about the weather 0.869 Renew ID card 0.865 renew a drivers license 0.776 Paying bills 0.973 Information about car tax 0.973 Apply for job 0.784 Renew health card 0.974 Pay taxes Tax refund Income tax settlement Tax situation Renew family document Note: Only loadings greater than 0.50 are shown

Other information

0.928 0.874 0.842 0.841 0.933

6. Discussion and conclusion

A striking result of the above survey is the low level of awareness of the e-Government program within the sample. The fact that our sample consisted of students who are young, most of them have high income, have access to the Internet and use it for various reasons, implies that they should at least know what e-Government is. Given that many other Jordanians will not have the same level of income, Internet access, and easiness of usage means that they will most likely have even lower awareness and usage levels of e-Government services. According to the e-Government officials, this is expected and there are awareness plans in place to address this issue which includes conducting workshops, making visits to different agencies such as schools, universities, companies and media promotions. However, such plans face the challenge of the limited resources available which include budget, qualified personnel, and the culture (Director of projects). The demanded services as shown in table 3 based on factor analysis indicates that such services are related to the needs of the people. For example, in our sample the least demanded service was the family document renewal which is reasonable taken into consideration that students in most cases do not have such a document which is required once a person starts his own family. For other Jordanians, the importance of services may be ranked differently. Results indicated that people are willing to pay a fee to use some of the Internet-based services. With respect to how much people might pay for the convenience of such services (if it is available online), responses varied depending on the nature of the services provided by government. On average, people are willing to pay more to the first factor which we described as 'personal information', than they are for the other factors, which we described as 'security and health information', ''tax information', and 'other information', respectively. Payment for services may be linked to the person's income especially that even within the focus group discussion a number of participants expressed concerns about cost of using services and using the internet in general. Therefore, for other Jordanians with lower income levels this concern about cost is most likely applicable. A very important issue that was pointed out during focus group discussions is the concern about security of giving information over the internet. Generally speaking, there is a lack of trust of using online transactions especially when it comes to important documents or payments. This may also be related to the culture in general within Jordan where the internet still seems to be used mainly for entertainment. For example, only 16% of the sample did shopping online. Using e-Government services within such a culture is still problematic and needs serious attention. NAO (2002) proposed a model to present different levels of e-Government services development. This model has five components: 1) Basic site; 2) Electronic Publishing; 3) Interactive Publishing; 4) Transaction; and 5) Joined-up e-Government. This research applied the model of the Jordanian eGovernment program in order to examine the current situation of the e-Government project. EGovernment project in Jordan launched nine years ago, and based on the different sources of data
www.ejise.com

Academic Conferences Ltd

Saheer Al-Jaghoub et al.

and information; there is a demand for e-Government services amongst Jordanian citizens. However, the percentage of citizens using these services is still modest. Based on the discussion during the interviews pointed out that e-Government program still in the beginning of the second phase (Electronic Publishing) based on NAO model (see Figure 1). ICTs are useful instruments, capable of increasing government agencies' effectiveness, efficiency and transforming their services. In order to improve the effectiveness of the Jordanian e-Government practice, government agencies need to move towards a higher level (Figure 1) of e-Government development, which will require more technical, personal and financial commitments. Jordanian government also needs to establish systematic and comprehensive e-Government plans of citizens' encouragement and awareness of the e-Government services web sites. Issues like privacy and security, costs, acceptance appear to be the major obstacles of adopting e-Government services and need attention in the deployment of e-Government. However, the challenge of using the country's limited resources needs to be kept in mind. This research has been a preliminary attempt that aims to investigate the current level of awareness of e-Government and to understand some of the reasons behind the results that were apparent from the sample used for this paper. Future research should be targeted towards a more comprehensive sample which includes people that represent different demographics to further analyze the current situation and enable the decision makers in the Jordanian government to plan and implement the e-Government more successfully in the future. Finally, developing countries are not a homogenous group and therefore the results of this paper may not be generalizable. However, our findings may be useful as they provide rich insights (Walsham, 1995) to other developing countries in planning and implementing their e-Government initiatives.

References
Adeshara, P., Juric, R., Kuljis, J. and Paul, R. (2004) A survey of acceptance of e-Government services in the UK, Journal of Computing and Information Technology CIT 12, Vol. 2, Pp. 143-150. Al-Jaghoub, S. and Westrup, C (2003) Jordan and ICT Led Development: Towards a competition State, Information Technology and People, Vol. 16, No. 1, Pp. 93-110. Al-Yaseen, H., Eldabi, T., Paul, R. and El-Haddadeh, R. (2008) Post-implementation evaluation of IT systems: A close review of practice. In Irani, Z. and Love, P. (2008) Evaluating Information Systems: Public and Private Sector, Butterworth-Heinemann, Oxford, UK. Arif, M. (2008) Customer orientation in e-Government project management: a case study, The Electronic Journal of e-Government, Vol. 6, No. 1, Pp. 1-10. Barc, C. and Cordella, A. (2004) Seconds Out, Round Two: Conceptualizing e-Government projects within their Institutional Milieu-A London Local Authority Case Study. In Proceedings of the 12th European Conference on Information Systems, Turku, Finland. Berthold, M. and Hand, D. J. (2003) Intelligent Data Analysis, 2nd Ed., Springer-Verlag, Berlin. Beynon-Davis, P. (2004) Constructing electronic government: the case of the UK Inland Revenue. In th Proceedings of the 12 European Conference on Information Systems, Turku, Finland. Borras, J. (2004) International Technical Standards for e-Government, Electronic Journal of e-Government, Vol. 2, Pp. 75-80. Creswell, J. W. (2003) Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. SAGE Publications, Thousand Oaks, CA. Carter, L. and Belanger, F. (2003) "Diffusion of Innovation and Citizen Adoption of E-Government Services", the Proceedings of the First International E-Services Workshop, Vol. 1, No. 1, Pp. 57-63. th Carter, L. and Belanger, F. (2004) Citizen adoption of electronic government initiatives. In Proceeding of the 37 Hawaii International Conference on System Sciences. Choudrie, J. and Dwivedi, Y. (2005) Citizens' awareness and adoption of e-Government initiatives in the UK, eGovernment Workshop '05 (eGOV05), Brunel University, West London, UK. Ciborra, C. and Navarra, D. (2005) Good governance, development theory, and aid policy: Risks and challenges of e-Government in Jordan, Information Technology for Development, Vol. 11, No. 2, Pp. 141-159. Cottam, I., Kawalek, P. and Shaw, D. (2004) A local government CRM maturity model: a component in the transformational change in the UK councils. In Proceedings of the Americas Conference on Information Systems, New York. Dwivedi, Y. and Williams, M. D. (2008) Demographic influence on UK citizens' e-Government adoption, Electronic Government, an International Journal (EG), Vol. 5, No. 3, Pp. 261-274. Elsheikh, Y., Cullen, A. and Hobbs, D. (2008) e-Government in Jordan: challenges and opportunities, Transforming Government: People, Process and Policy, Vol. 2, No. 2, Pp. 83-103. Government online (2009) the official website of Jordanian e-Government, http://www.jordan.gov.jo/wps/portal, (accessed March 2009) Gupta, B., Dasgupta, S., and Gupta, A. (2008) Adoption of ICT in government organization in a developing country: An empirical study, Journal of Strategic Information Systems, Vol. 17, Pp. 140-154. Grant, G.and Chau, D. (2005) Developing a generic framework for e-Government, Journal of Global Information Management, Vol. 13, No. 1, Pp. 1-29.

www.ejise.com

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (1 8)

Green, A. and Preston, J. (2005) Editorial: Speaking in Tongues Diversity in Mixed Methods Research, International Journal of Social Research Methodology, Vol. 8, No. 3, Pp. 167-171. Gronlund, A. and Horan, T. A. (2005) Developing a generic framework for e-Government, Communications of the Association for Information Systems, Vol. 15, Pp. 713-729. Lee, S. M., Tan, X., Timi, S. (2005) Current practices of leading e-Government countries, Communications of the Association for Information Systems, Vol. 48, No. 10, Pp. 99-104. Mofleh, S., Wanous, M. and Strachan, P. (2008a) Developing countries and ICT initiatives: Lessons learnt from Jordan's Experience, The Electronic Journal on Information Systems in Developing Countries, Vol. 34, No. 5, Pp. 1-17. Mofleh, S., Wanous, M. and Strachan, P. (2008b) The gap between citizens and e-Government projects: the case for Jordan, Electronic Government, an International Journal (EG), Vol. 5, No. 3, Pp. 275-287. Moon, J. M. and Kim, Y. G. (2001) Extending the TAM for a World-Wide-Web Context, Information and Management, Vol. 28, Pp. 217-230. Mosse, B. and Whitley, E.A. (2004) Assessing UK e-Government websites: classification and benchmarking. In th Proceedings of the 12 European Conference on Information Systems, Turku, Finland. NAO (2002) Government on the Web II, National Audit Office, London. Pett, M. A., Lackey, N. R. and Sullivan, J. J. (2003) Making sense of factor analysis: the use of factor analysis for instrument development in health care research, Sage Publications, London. Sammons, P., Siraj-Blatchford, I., Sylva, K., Melhuish, E., Taggart, B. and Elliot, K. (2005) Investigating the Effects of Pre-School Provision: Using Mixed Methods in EPPE Research, International Journal of Social Research Methodology, Vol. 8, No. 3, Pp. 207-224. Santos, R. and Heeks, R. (2003) ICTs and Intra-Governmental Structure at Local, Regional, and Central Levels: Updating Concentional Ideas, available at: http://www.sed.manchester.ac.uk/idpm/research/publications/wp/igovernment/short/igov_sp07.pdf Shareef, M., Kumar, U., Kumar, V., Dwivedi, Y. (2009) Identifying critical factors for adoption of e-Government, Electronic Government, an International Journal (EG), Vol. 6, No. 1, Pp. 70-96. Tashakkori, A., and Teddlie, C (2003) The past and the future of mixed methods research: from Methodological Triangulation to Mixed Methods Designs. In Handbook of Mixed Methods in Social and Behavioral Research, Tashakkori and Teddlie (Eds.). Thousand Oaks, CA: Sage. Walsham, G., Interpretive case studies in IS research: nature and method, European Journal of Information Systems, (4), 1995, pp. 74-81. Wiiliams, M.D. and Beynon-Davies, P. (2004) Implementing e-Government in the UK: An analysis of local-level strategies. In Proceedings of the Americas Conference on Information Systems, New York. Yang, J. and Paul, S. (2005) E-Government application at local level: issues and challenges: an empirical study, International Journal of an Electronic government, Vol. 2, No. 1, Pp. 56-76.

www.ejise.com

Academic Conferences Ltd

Post-Implementation Evaluation of HealthCare Information Systems in Developing Countries

Hussein Al-Yaseen, Saheer Al-Jaghoub, Maher Al-Shorbaji and Maher Salim Al-Ahliyya Amman University, Amman, Jordan
hyaseen@ammanu.edu.jo mahersho@ammanu.edu.jo saljaghoub@ammanu.edu.jo president@ammanu.edu.jo
Abstract: Information and Communication Technology (ICT) project managers require accurate and reliable evaluation to allocate and control project resources. In addition, many private hospitals indicate that a number of their projects have failed; and between one and two thirds of ICT projects exceed their budget and time. Further, about half of the expensive ICT projects at the end will be considered out of control and cancelled. Justifying ICT investments is a long standing problem, and managers for the past decades have expressed concerns about the value they are getting from their investments, and they have been searching for ways to evaluate and justify these projects. Hence, evaluation of ICT is therefore becoming an important issue for both managers and practitioners. This paper aims to investigate the current practice of both types of evaluation: Prior Operational Use evaluation -POUe- and Operational Use evaluation -OUe- in Jordanian private hospitals to better understand what is required for the evaluation process and its associated benefits; secondly, to collect information about how hospitals carry out the evaluation process. In doing so, we attempts to answer specific questions, such as: How prevalent is POUe and OUe? What criteria are being used in both types of evaluation? What are their main benefits and uses of each type of evaluation? Results suggest that most decision makers do not place much importance on OUe of their IT/IS. Most managers tend to think of it only as a formality rather than a proper evaluation process. Without adopting a formal OUe the cost of future health informatics would seem likely to be less accurately estimated. Keywords: healthcare information systems, health informatics, evaluation, developing countries, Jordan

1. Introduction
Many organizations in developed and developing countries -in both private and public sectors- turned to Information Technology/Information Systems (IT/IS) to meet the increasing demands on organizations to increase their efficiency and effectiveness (Jones and Hughes, 2000). This implies that investment in IT/IS is high, which has been a problematic issue for more than one decade. In addition, there is a contradictory evidence as to whether or not IT/IS expenditure has resulted in creating economic value for the organizations (Willcocks and Lester, 1999; Eldabi et al., 2003; Irani et al., 2002; Al-Yaseen, et al., 2006; 2008). Investments in IT/IS are growing extensively in most organizations; managers worry about the fact that benefits from IT/IS investments may not be as high as expected as large amounts of money are invested in IT/IS and there is not enough return from this expenditure (Irani et al., 2002; Kumar, 1990; Remenyi et al., 2000; Al-Yaseen et al., 2007). Organizations specialized in healthcare are no exception as they have also joined in building information systems that also require investments in IT/IS (Wetter, 2007). These systems are known as healthcare information systems which focus on optimizing and using information to increase efficiency and effectiveness in healthcare organizations (Reichertz, 2006; Hayrinen et al., 2007). This paper investigates the evaluation process of health information systems implemented in Jordanian hospitals and explores the main issues related to the evaluation process such as: how prevalent prior operational use and operational use evaluation; main uses and benefits of adopting both types of evaluation. The paper is organized as follows. Section 1 is an introduction about the importance of ICTs in healthcare information systems. Section 2 discusses healthcare systems evaluation and the need for justification. The details of the research methodology used are presented in section 3. Section 4 presents data analysis and preliminary findings of the adoption of IT/IS evaluation types in healthcare sector. Finally, the discussion and conclusions are presented in section 5.

ISSN 1566-6379 9 Academic Conferences Ltd Reference this paper as: Al-Yaseen,H, Al-Jaghoub, S, Al-Shorbaji, M and Salim, M. (2010) Post-Implementation Evaluation of HealthCare Information Systems in Developing Countries The Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010, (pp9 - 16), available online at www.ejise.com

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (9 16)

2. IT/IS evaluation and justification

Justifying expenditure on IT/IS is a long standing problem, and managers for the past few decades have expressed concerns about the value they are getting from IT/IS investments; moreover they have been searching for ways to evaluate and justify the use of IT/IS (Gunasekaran et al., 2006; AlYaseen, et al., 2007; 2008). Such a continuous increase in investment coupled with continuous need for justification presents a challenge to the IT/IS community. Many organizations reported that they are uncertain about how to measure the impact and the outcomes of their IT/IS investments (Bradford and Florin, 2003; Eldabi et al., 2003; Farbey et al., 1993; Gunasekaran et al., 2001; Lin and Pervan, 2003; Liu et al., 2003; Skok et al., 2001; Al-Yaseen et al., 2007).

3. IT/IS evaluation types

Evaluation can be defined as: 'to judge or determine the significance, worth, or quality' (Webster's Dictionary). Willcocks (1992) defined IT/IS evaluation as the 'process of establishing by quantitative and/or qualitative techniques the worth of IT/IS projects to the organizations'. Or the process of assessing the worth of something (Beynon-Davies et al., 2000). Evaluation can also be considered in terms of the effectiveness of the IT system in situ what a system actually accomplishes in relation to its stated goals (Al-Yaseen et al., 2004, Eldabi et al., 2003). We take the stance that evaluation is a process that takes place at different points in time, or continuously, explicitly searching for (quantitatively or qualitatively) the impact of IT projects (Eldabi et al., 2003). The value of this latter definition is that it explicitly recognises the different stages in the full lifecycle of an IT/IS project in which evaluation is performed, and provides the opportunity to discriminate between two decidedly different views of the evaluation process, each serving different aims. The evaluation process is a fundamental and critical activity and needs to be thoroughly conducted in any phase of the system's life cycle (Galal et al., 2000). Furthermore, it is increasingly acknowledge that evaluation of information systems is recognized as a complex and challenging activity, and there is no agreement on an ideal way to evaluate or how to make the evaluation process better (Dabrowska and Cornford, 2001). Evaluation types can be classified into two types with regards to the development stage of the system or the timing of evaluation (Eldabi et al., 2003). Type A is a Prior Operational Use Evaluation; sometimes referred to as ex-ante; formative, or Prior-Implementation Evaluation; or as we shall refer to it, Prior Operational Use evaluation (POUe). POUe is a 'predictive evaluation performed to forecast the impact of the project. This type of evaluation is carried out prior the system becomes into operational use through the development stages of IT/IS- to justify the investment. Type B of evaluation is carried out when the system becomes into operational use; this form of evaluation draws on real rather than projected data, and can be used to justify adoption (Love and Irani, 2001; Irani, 2002); estimate the direct cost of the system, estimate the tangible benefits of the system (Liu et al., 2003); ensure that the system meets requirements (Irani, 2002); measure the system effectiveness and efficiency (Poon and Wagner, 2001); measure the quality of programs and to estimate indirect costs and other costs (Love and Irani, 2001); or to measure the quality of programmes (Eldabi et al., 2003). We shall refer to this type as Operational Use evaluation (OUe). Figure 1 shows these forms of evaluation with respect to the system's life cycle from a systems inception to the end of its useful life.
Systems life cycle

Prior Operational Use Evaluation Time Development Stage

e Us al on tion ti era ua Op Eval

System into operational use

Figure 1: IT/IS evaluation types in the systems life cycle

www.ejise.com

10

Academic Conferences Ltd

Hussein Al-Yaseen et al. There are also a number of empirical studies such as those reviewed by (Ballantine et al., 1996) which examined ex-ante evaluation (POUe), yet only a few {for example Kumar (1990) and to some extent Beynon-Davies et al., 2004)} that have explored the ex-post evaluation (OUe).

4. Research methodology
In order to gather as much data as possible to understand the whole picture about prior operational use and operational use evaluation as practices within Jordanian private hospitals and to understand the benefits and obstacles of these two types of evaluation; the following data sources were used: General information on healthcare systems in Jordan (Government websites such as: Ministry of Health MOH-, Private Hospitals Association PHA-, 2009) Data from a survey: we administered a survey instrument to all private hospitals. Questionnaire contains four stages: designing and testing the questionnaire; followed by data collection and then data analysis; and finally the preliminary findings of the questionnaire. The survey was sent to 60 private hospitals in Jordan.

In doing so, the following questions need to be answered by practitioners who are involved with IT/IS evaluation in healthcare sector: How prevalent is prior operational use and operational use evaluation of healthcare systems in private hospitals in Jordan? What criteria are being used in order to evaluation healthcare systems? What are the main uses and benefits of adopting the two types of evaluation of healthcare systems?

We analyzed the data from the responses of the questionnaire using a combination of the parametric statistical methods, Descriptive Analysis and Factor Analysis (Pett, et al., 2003). Respondents were asked to select from the list the closest choice of many variables. Each of these variables were measured using a five point Likert scales (1 = not important and 5 = very important). For technically interested readers we report that a factor analysis technique was employed in order to identify possible categories. Factor analysis was performed in three steps (following Berthold and Hand, 2003): A matrix of correlation coefficients for all possible pairings of the variables was generated. Factors were then extracted from the correlation matrix using principal factors analysis. The factors were rotated to maximize the relationships between the variables and some of the factors and minimize association with others using Varimax Kaiser Normalization, which maintained independence among the mathematical factors. The Eigenvalues determined which factors remained in the analysis. Following Kaisers criterion, factors with an Eigenvalue of less than 1 were excluded. A Screen plot provides a graphic image of the Eigenvalue for each component extracted.

5. Data analysis and preliminary findings

This section presents aggregated results from direct answers to the research questions mentioned above. The basic issues considered here are: reasons for adopting either type of evaluations, criteria used for evaluations, and uses and benefits of adopting the two types of evaluation of healthcare systems. Of the 60 questionnaires addressed to all private hospitals, 19 completed questionnaires were returned for a total response rate of 31.6%. The average IT/IS costs for the private hospitals in the research sample is ($328,000); while within the sample, 15% had implemented systems that cost more than ($1,140,000). 26.3% of the respondent hospitals have adopted IT/IS as a response to problem(s), while 73.7% of the respondent hospitals have adopted IT/IS systems searching for ways of improving their efficiency and effectiveness.

6. Reasons for adopting POUe in healthcare systems

The results are presented in Table 1. Using a factor analysis cut-off level of 0.5, four factors were considered the main reasons of adopting Prior Operational Use evaluation, which we described as system completion and justification, system costs, system benefits, and other reasons.

www.ejise.com

11

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (9 16) Table 1: Reasons for adopting Prior Operational Use evaluation factor analysis
Factors System completion and justification 0.767 0.782 0.791 0.786 0.750 0.742 0.772

System System costs benefits Reasons System meets requirements System effectiveness System usage System efficiency Justify adoption System security System performance Quality and completeness of system documentation 0.670 Hardware performance 0.655 Quality of programs 0.666 Operational costs 0.598 Training costs 0.619 Maintenance costs 0.584 Upgrade costs 0.542 Reduction in clerical salaries 0.649 Reduction in other staff costs 0.630 Other expenses saved 0.652 Direct costs 0.682 Indirect costs 0.676 Other costs 0.686 Tangible benefits 0.525 Intangible benefits 0.611 Other benefits Barriers of adopting the system Note: Only loadings greater than 0.50 are shown

Other reason

0.578 0.542

The first factor System completion and justification is highly correlated with seven variables, the second factor system costs is highly correlated with seven variables, and the third factor system benefits are highly correlated with eight factors, whilst the fourth factor other reasons is highly correlated with two variables 'Other benefits' and 'barriers for adopting the system' which was also found to be the least evaluated reason in practice, as shown in Table 1.

7. Reasons for adopting OUe in healthcare systems

The results are presented in Table 2. Employing a factor analysis cut-off level of 0.5, three factors were considered. The most important reasons for adopting Operational Use evaluation were identified from a five-point Likert scale ranging from 1 (not important) to 5 (very important). The results are presented in Table 2. Three factors were considered as the main reasons of adopting Operational Use evaluation, which we described as: system costs, system benefits, and other reasons. Table 2: Reasons for adopting Operational Use evaluation factor analysis
Other reasons 0.751 0.751 0.741 0.612 Factors System benefits System costs

Variables Estimating of system life Justify system adoption Risks Barriers Tangible benefits 0.677 Intangible benefits 0.678 Other benefits 0.682 Direct costs Indirect costs Other costs Note: Only loadings greater than 0.50 are shown

0.646 0.519 0.522

www.ejise.com

12

Academic Conferences Ltd

Hussein Al-Yaseen et al. The first factor other reasons is highly correlated with four variables, the second factor systems benefits' is highly correlated with three variables, and the third factor systems costs' is highly correlated with three variables 'direct costs', 'indirect costs', and 'other costs' which was also found to be the least evaluated reason in practice, as shown in Table 2.

8. OUe criteria used in healthcare systems

The results are presented in Table 3. Employing a factor analysis cut-off level of 0.5, four factors were considered. OUe criteria were identified from a five-point Likert scale ranging from 1 (not important) to 5 (very important). The results are presented in Table 3. four factors were considered as the main criteria of OUe, which we described as: system completion, system information, system impact, and other criteria. Table 3: Operational Use evaluation criteria factor analysis
System completion 0.873 0.769 0.794 0.765 0.676 0.873 0.773 Factors System information System impact Other criteria

Criteria Internal controls Project schedule compliance System security and disaster protection Hardware performance System performance versus specifications System usage Quality and completeness of system documentation Accuracy of information 0.984 Timeliness and currency of information 0.974 Adequacy of information 0.979 Appropriateness of information 0.874 Quality of programs 0.842 User satisfaction and attitude towards systems User friendliness of system-user interface System's impacts on users and their jobs System's fit with the impact upon organization Net operating costs (savings of system) Note: Only loadings greater than 0.50 are shown

0.959 0.874 0.928 0.849 0.633

The first factor system completion is highly correlated to seven criteria, the second factor system information are highly correlated to five criteria, the third factor system impact is highly correlated to four criteria, whilst other criteria is correlated to one criterion net operating costs, which was also found to be the least evaluated criteria in practice. For more information, see Table 4 which shows the construct loadings for the reasons of adopting Operational Use evaluation.

9. Discussion and conclusions

All of the responding private hospitals have adopted a formal POUe, but only about a third (31.5%) currently perform a formal OUe IT/IS. This means that about two thirds (68.5%) of the private hospitals do not gather any evidence to establish how successful their IT/IS were, therefore cannot use such information from OUe to improve their evaluation techniques and outcomes and decrease deviation. Results suggest that most decision makers do not place much importance on OUe of their IT/IS. Most managers tend to think of it only as a formality rather than a proper evaluation process. For example, amongst the 6 hospitals who consider adopting OUe those hospitals that seriously perform it tend to gain considerable benefits, including the validation of their original POUe estimates. In some cases, OUe is adopted in order to move responsibility from developers to users and for formal closure. When performed, the most popular reasons for adopting OUe were related to formal aspects of signing off the project (based around traditional measures such as meeting requirements, and achieving agreed metrics for effectiveness, usage, efficiency, security, performance, etc.), and system costs. The two factors -systems benefits and adoption barriers were found to be less important. On the other hand, amongst the 19 private hospitals, the most frequent reason for adopting OUe was related to systems benefits (both tangible and intangible). Most of the sampled private hospitals attach greater importance to the measurement of benefits rather than the measurement of costs. The most important claimed use and benefit of adopting OUe was system cost (operational cost, training cost, maintenance cost, upgrade cost, reduction in other staff cost, reduction in salaries, and other expenses saved).As for the criteria used, the most frequently cited criterion for OUe was

www.ejise.com

13

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (9 16) system information followed by system impact, especially user satisfaction and friendliness. This implies that hospitals focus more on what the system provides rather than system completion and some of the related factors. Having such information about the system, its impact and the information it provides is useful for the future as it provides managers with the negative and positive aspects of the current system and therefore, learn lessons for the implementation of other systems in the future. It is clear that the practitioners are not appreciating the full benefits of OUe and need to be aware of such benefits. Such lack of appreciation is evidently behind the apparent shortage of implementations of OUe, which negatively feeds back into perceptions and so forth. This research focuses on the private sector of hospitals but it does not include any information about the public sector. Private hospitals in Jordan are more developed in terms of using IT/IS than public healthcare organizations. The results of OUe in private hospitals, if and when it is carried out could be useful for public healthcare organizations. In a developing country like Jordan which has limited resources, the lessons learnt from the implementation of information systems and the deviations that occur in such systems could be useful in estimating the budgets required for the implementation of such systems in the public sector, given the limited resources available for such organizations. It is understandable that the scope of systems for public healthcare is different; nonetheless, the results of OUe of systems in the private sector could still be useful for decision makers. Without adopting a formal OUe the cost of future health informatics would seem likely to be less accurately estimated. Our research results are entirely consistent with this observation. Without OUe how can we know whether this is true or not, or much else about what is going on? Our study confirms that diffusion of the importance of OUe amongst both the academic and practitioners communities could play an important role in more IT/IS effectiveness and less disappointment and less healthcare systems failure.

References
Al-Yaseen, H., Al-Gweri, A., and Al-Jaghoub, S. (2007) IS/IT Investment Appraisal: Prior and Post Implementation Evaluation in Jordanian Firms, International Conference Applied Computing (IADIS), Spain. Al-Yaseen, H., Eldabi, T., Lees, D. and Paul, R. (2006) Operational Use Evaluation of IT Investments: An Investigation into Potential Benefits, European Journal of Operational Research (EJOR), Vol. 173, AlYaseen, H., Eldabi, T., Paul, R. J. (2004) A quantitative assessment of operational use evaluation of information technology: benefits and barriers, in Proceedings of the Tenth Americas Conference on Information Systems, New York, August 2004, pp 688 692. Al-Yaseen, H., Eldabi, T., Paul, R. and El-Haddadeh, R. (2008) Post-implementation evaluation of IT systems: A close review of practice. In Irani, Z. and Love, P. (2008) Evaluating Information Systems: Public and Private Sector, Butterworth-Heinemann, Oxford, UK. Ballantine, J.A., Galliers, R. D., and Stray, S.J. (1996) Information systems/technology evaluation practices: evidence from UK organizations, Journal of Information Technology, Vol. 11, pp 129 141. Beynon-Davies, P., Owens, I. and Lloyd-Williams, M. (2000) IS Failure, evaluation and organisational learning, UKAIS, Cardiff, pp 444 452. Beynon-Davies, P., Owens, I. and Williams, M. D. (2004) Information systems evaluation and the information systems development process, Enterprise Information Management, Vol. 17, pp 276 282. Bradford, M. and Florin, J. (2003) Examining the role of innovation diffusion factors on the implementation success of enterprise resources planning systems, International Journal of Accounting Information systems, Issue 4, pp 205 225. Berthold, M. and Hand, D. J. (2003) Intelligent Data Analysis, 2nd Ed., Springer-Verlag, Berlin. Dabrowska, E. K. and Cornford, T. (2001) Evaluation and Telehealth an Interpretative Study. Proceedings of the Thirty-Fourth Annual Hawaii International Conference on System Sciences (HICSS)-34, Maui, Hawaii, Computer Society Press of the IEEE. Piscataway, N.J., June 2001. Eldabi, T., Paul, R.J. and Sbeih, H. (2003) Operational use evaluation / post implementation evaluation of IT, UKAIS, 2003, Warwick. Farbey, B., Land, F. and Targett, D. (1993) How to assess your IT investment: a study of methods and practice, Butterworth-Heinemann Ltd., London. Galal, G. H., McDonnell, J. T. and Paul, R. J. (2000) The Role of Interpretative Evaluation in Engineering Information Systems Requirements. Proceedings of the Americas Conference on Information Systems (AMCIS) (10-13 August, Long Beach, Ca.), Editor: Chung, H. M., Association for Information Systems, Atlanta, Pp. 1102-1108. Gunasekaran, A., Love, P.E.D., Rahimi, F. and Miele, R. (2001) A Model for Investment Justification in Information Technology Projects, International Journal of Information Management, Vol. 21, pp 349 Gunasakeran, A., Ngai, E.W.T., McGaughey, R.E. (2006) Information technology and systems justification: A review for research and application, European Journal of Operational Research, Vol. 173, Pp. 957-983.

www.ejise.com

14

Academic Conferences Ltd

Hussein Al-Yaseen et al.

Hayrinen, K., Saranto, K., and Nykanen, P. (2007) Definition, structure, content, use and impacts of electronic health records: A review of the research literature, International Journal of Medical Informatics, Vol. 77, No. 5, Pp. 291-304. Irani, Z. (2002) Information systems evaluation: navigating through the problem domain, International Journal of Information and Management, Vol. 40, Pp. 11 24. Irani, Z., Sharif, A., Love, P.E.D. and Kahraman, C. (2002) Applying concepts of fuzzy cognitive mapping to model: The IT/IS investment evaluation process, International Journal of Production Economics, Issue 75, pp 199 211. Jones, S. and Hughes, J. (2000) Understanding IS Evaluation as a Complex Social Process. Proceedings of the 2000 Americas Conference on Information Systems (AMCIS) (10-13 August, Long Beach, Ca.), Editor: Chung, H. M., Association for Information Systems, Atlanta, pp 1123 1127. Kumar, K. (1990) Post Implementation Evaluation of Computer Information Systems: Current Practices. Communications of the Association for Computer Machinery (ACM), Vol. 33, No. 2, N.Y., (January), pp 203 212. Lin, C. and Pervan, G. (2003) The practice of IS/IT benefits management in large Australian organisations, International Journal of Information and Management, Issue 41, pp 13 24. Liu, Y., Yu, F., Su, S.Y.W. and Lam, H. (2003) A Cost-Benefit Evaluation Server for decision support in ebusiness, Journal of Decision Support Systems, Issue 36, pp 81 97. Love, P. E. D. and Irani, Z. (2001) Evaluation of IT costs in construction, Journal of Automation in Construction, Vol. 10, Pp. 649 658. Ministry of Health (MoH) Jordan, www.moh.gov.jo/MOH/En/home.php, (Accessed March, 2009). Pett, M. A., Lackey, N. R. and Sullivan, J. J. (2003) Making sense of factor analysis: the use of factor analysis for instrument development in health care research, Sage Publications, London. Poon, P. and Wagner, C. (2001) Critical success factors revisited: success and failure cases of information systems for senior executives, Journal of Decision Support Systems, Vol. 30, Pp. 393 418. Private Hospitals Association (PHA) Jordan, www.pha-jo.com, (Accessed March, 2009). Reichertz, P. L. (2006) Hospital information systems Past, present, future, International Journal of Medical Informatics, Vol. 75, No. 3-4, Pp. 282-299. Remenyi, D., Money, A., Sherwood-Smith, M. and Irani, Z. (2000), The effective management and management of IT costs and benefits, Butterworth-Heinemann Ltd., London. Skok, W., Kophamel, A. and Richardson, I. (2001) Diagnosing information systems success: importanceperformance maps in the health club industry, International Journal of Information and Management, Issue 38, pp 409 419. Wetter, T. (2007) To decay is system: The challenges of keeping a health information system alive, International Journal of Medical Informatics, Vol. 76, Pp. 252-260. Willcocks, L. (1992) Evaluating information technology investments, research findings and reappraisal, Journal of Information Systems, Vol.2, pp 243 268. Willcocks, L. and Lester, S. (1999) Beyond the IT Productivity Paradox, Wiley, Chichester.

www.ejise.com

15

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (9 16)

www.ejise.com

16

Academic Conferences Ltd

Towards a Model for Determining the Scope of ICT Integration in the Enterprise: the Case of Enterprise Resource Planning (ERP) Systems
Fergal Carton and Frederic Adam University College Cork, Ireland
f.carton@ucc.ie fadam@afis.ucc.ie
Abstract: The question of integration of information systems (IS) into the planning and execution of operational activities has been the focus for researchers from different constituencies. Organisational theorists recognise the need for integrating mechanisms for co-ordinating the actions of sub-units within an organisation. Centralisation has been seen as a defensive reaction by organisations when placed under increasing external control , and also as a way to improve the efficiency of information processing, at least for routine tasks. In the meantime, researchers have been sceptical about the ability for structured information systems to deal with the complexity of the information flows within the organisation. Frameworks have also been identifying characteristics of the tasks themselves that have a bearing on the amount of information processing required. The real world is complex and moving, thus managers require flexibility in their interpretation of the mixed signals arising from this complexity. However, managers are working in environments where highly integrated information systems blur the distinction between what is real and what is virtual. There is a need for an integration approach allowing organisations to question which areas of activity are worth integrating, and conversely which areas are better left under local control. Where integrated, managers require processes for the maintenance of data integrity (people, tools, procedures). Based on field work involving two multi-national manufacturing companies, this paper proposes a framework for ERP integration, which describes the evolution of functionality gaps as an ongoing and inevitable process that requires management. Keywords: ERP, enterprise, integration, framework, complexity

1. Introduction
In examining the extent to which information systems (IS) can be integrated into the planning and execution of an organisations activities, the researcher may draw from rich seams of research from different constituencies. The question of how best to control and co-ordinate the organisation to meet the needs of its customers, while optimising the use of available resources, has tantalised organisational theorists from the earliest days of industrial and economic growth. Key to the concept of control is information, such that goals may be communicated downwards and performance may be measured and communicated upwards. It has been acknowledged by the earliest students of administrative systems that the major cost of implementing control systems of any sort is the investment in human resources to design and maintain the system. Today, advances in data capture, communication and dissemination technologies have allowed designers to integrate the bureaucratic overhead of administration of task monitoring into the activities themselves. However, this very synergy between task and control can have negative side effects in terms of flexibility that are undermining the gains in efficiency. Indeed, the ubiquity of technology in all areas of the enterprise has displaced the question of efficiency towards the administrative aspect of the task and away from the task itself. In this section the two strands of research (organisational and informational) are drawn together to tease out the dimensions of a framework that might permit researchers to situate the adoption of integrated control systems within the broader context of the organisation and its aims.

1.1 Integration is a design attribute for organisational rationality

To Mintzberg (1989), organisation means collective action in the pursuit of a common mission. Organisation theory is the body of research that addresses itself to the problem of how to organise (Pugh 1997). How organisations should be controlled, by whom, and to what ends, are the fundamental issues for the private sector (Mintzberg 1989). An organisation is instrumentally rational if the job gets done, economically rational if it gets done at the least cost (Thompson 1967). Thompson (1967) uses the analogy of the ad-hoc organisation of the community in the face of a natural disaster to emphasise that instrumental rationality can be achieved in the face of necessity, but that economic rationality requires more co-ordination and advance knowledge of what resources are required and when. Interestingly, research into how managers actually work suggests a constant mode of crisis ISSN 1566-6379 17 Academic Conferences Ltd
Reference this paper as: Carton, F and Adam, F. (2010) Towards a Model for Determining the Scope of ICT Integration in the Enterprise: the Case of Enterprise Resource Planning (ERP) Systems The Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010, (pp17 - 26), available online at www.ejise.com

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (17 26) management (Mintzberg 1989), denoting the inability to work to a co-ordinated plan because of the contingencies of the moment. It would appear that despite best efforts to control and co-ordinate, managers have to deal with many sorts of unplanned events thrown at them in quick succession and requiring immediate attention. Organisational attributes are choices about the optimal design of the organisation in view of its aims. Some structural variables considered in organisational theory are specialisation (functional and role), standardisation, formalisation, and centralisation (Inkson et al. 1968). An increasingly important organisational attribute is the incorporation of information systems into ways of working such that task execution and visibility of that performance are instantaneous. The challenge of studying such organisational attributes is that there are underlying interdependencies between these attributes and contextual factors which are difficult to conceptualise. Formalisation of procedures is a step towards standardisation, which itself is a step towards centralisation. Furthermore, business context, culture and managerial capability are additional softer attributes of the organisation which will confound generalisations based on any one set of structural variables. In his work on organisational configurations, Henry Mintzberg (1989) suggests that it is the combination of basic attributes of an organisation that define its culture, rather than a focus on any one single attribute. One of the essential dichotomies in the planning and management of routine activities is the trade-off between control and flexibility. With Enterprise Resource Planning (ERP) systems, the interdependence between control and other structuring attributes (such as formalisation, centralisation, specialisation) is configured in the software. As researchers had forewarned, the complexity of the control system made it necessary to invest in specialised skills to design and maintain the system (Child 1973). Without the ability to conceptualise integration as an organisational attribute among others, organisations risk becoming slaves to their self-imposed technostructure.

1.2 The use of information technologies for control

Since the earliest days of the application of information technologies to automate manual tasks, the question of integration has intrigued practitioners and academics alike. The potential for the technology to make information flow was understood, but its use in planning and monitoring of performance entailed a trust in the parameters and algorithms in the machine that had hitherto not existed. As far back as 1958, Leavitt & Whisler suggested that information technologies would undermine the decision making role of the middle manager, consigning them instead to jobs that were highly structured. The authors argued that the spread of information technology would be rapid because it would make centralisation much easier, making top executives less dependent on subordinates. In addition, the faster processing of information would shorten the feedback loop that tests the accuracy of original observations and decisions. There is a relationship between the need to process information and uncertainty deriving from the external business environment. Organisations require integrating mechanisms (Galbraith in Pennings, 1983) for co-ordinating the actions of sub-units towards a common objective. Centralisation and standardisation have been seen as a defensive reaction by organisations when placed under increasing external control (Mintzberg 1989), and also as a way to improve the efficiency of information processing, at least for routine tasks (Galbraith 1974). An information system by definition supports the centralisation of control (Markus & Robey 1988). Under conditions of uncertainty, there is a tendency to increase the amount of information sought (Thompson 1967). Earl & Hopwood (in Lucas et al. 1980) refer to the tendency in the MIS area to perceive uncertainty as threatening rather than inevitable, and, rather than exploiting information for its educative (Gorry 1971) potential, information systems professionals tend to design models that mask reality with assumed certainties. In parallel to the evolution of the capacity of systems to handle information at speed and in large volumes, the determinist consequences of information systems for centralisation has been the subject of much debate (Orlikowski in Knights, D. and Willmott, H. 1988). ERP systems simultaneously centralise ownership of information resources and democratise access to that information, mixing therefore, to an unprecedented extent, centralising and decentralising effects. On the one hand they promote local, cross-functional cooperation and control, breeding skills that are eminently transferable across sub-units. On the other, a best practice model of transaction processing imposes a common standard across the organisation. As Davenport (1998) notes, the real challenges hinges on where to draw the boundary between centralization of control and autonomy of decision making.
www.ejise.com

18

Academic Conferences Ltd

Fergal Carton and Frederic Adam

Researching the organisational impact of information systems is challenging because the interaction of people with technology in the execution of business processes is so intertwined that it becomes difficult to differentiate organisational from technical factors (Markus et al. 2000). Researchers must differentiate between the symptoms and the cure. Inefficiency and frustration may be the result of poor process design, or incomplete implementation, or a combination of both factors. When the template falls short of actual physical procedures, data quality will inevitably be impacted. Researchers have noted that although ERP systems may be introduced, physical procedures are not always changed, such that a mismatch develops between virtual processes and physical processes (Staehr, Shanks and Seddon in Adam and Sammon, 2004; Lee and Lee 2000). The literature that provides the theoretical framework for this study of integration unites the themes of organisations and control mechanisms. Organisations require co-ordination mechanisms to transform strategy into operational targets. ERP systems are considered to be an advanced form of control mechanism. Managers are the enactors of this co-ordination, and their decisions are informed by data concerning both performance targets and actual consumption of resources. ERP systems superimpose new patterns on this co-ordination, with an, as yet, poorly researched impact on the quality of decision making.

2. The research objective

The objective of this research is to propose a framework which would allow the question of integration to be modelled alongside other related organisational attributes. It was anticipated that some of these attributes, identified already in the literature, such as centralisation, standardisation and specialisation, will figure in the framework. Equally, it was anticipated that other attributes will emerge from the empirical work, which is based around the implementation of ERP in two multinational manufacturing organisations. The research objective was operationalised into three separate research questions which together yield a picture of the scope of integration in the organisation, and a suggestion for the interdependence between integration and other organisational attributes. Research Question 1 was concerned with discovering the key organisational attributes relating to the implementation of ERP. Research Question 2 was concerned with developing a causality map between these attributes. Research Question 3 drew on the output from Question 1 and 2 to suggest a conceptual framework for the consideration of integration as a set of interdependent organisational attributes. It is envisaged that this framework could be used by managers to evaluate the potential impact of integration on the organisation, as well as a providing a diagnostic tool for the isolation of dysfunctional behaviour and its potential causes.

2.1 Profile of the case studies

In this research, two case studies of successful multinational companies (KPC and SIT) are used to explore the impact of integration on the organisation. Both cases studied are multi-national manufacturing organisations with mature ERP implementations. Table 2 compares the profiles of the two cases studied. Table 1: Comparison of case study profiles
Industry Turnover 05 ($bn) Employees WW operations Manufacturing sites Head Office ERP System Architecture Server location Go-live Firm A Pharmaceutical 38.72 100,000 119 80 London, UK SAP R/3 Single instance Pennsylvania, USA Phased 2004 Firm B Data management 9.66 26,500 52 3 Boston, USA Oracle 11.03 Single instance Boston, USA Big-bang 2001

www.ejise.com

19

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (17 26) The Key Pharma Company, KPC (real name withheld to allow more detailed reporting), is a leading manufacturer of pharmaceutical products, with a highly successful product portfolio in consumer healthcare, prescriptions drugs and vaccines. With annual sales of nearly 30 billion, and a R&D budget of 5 billion, KPC is in a dominant position in its marketplace. The manufacturing organisation involves more than 20 autonomous plants worldwide. Managing the supply chain to efficiently satisfy demand is extremely complex. With over 30,000 Stock Keeping Units (SKUs) or lines of product, any one of the 600 sources of demand could be ordering 300-600 SKUs each. The case study focuses on the KPC plant in Cork, Ireland, a bulk site which is part of the global Manufacturing and Supply organisation. KPC Cork ships 4,000 batches of goods per year and local managers are proud of their customer service record for deliveries, in the context of the complex and sometimes unpredictable scenario described above. KPC Cork was part of the roll-out programme for a new ERP system (based on SAP version 4.0), which involved all the sites in the Manufacturing and Supply organisation. The ERP project had the overall goal of implementing FDA compliant business processes throughout KPC, using the best practice templates that had been designed by KPC around SAP standard functionality. The scope of the ERP project at KPC Cork was the integration of processes in all the main business areas (Production, Finance, Sales, Quality), excluding process control at the manufacturing execution level. The second case is SIT, a market leader in data management solutions, which sees itself as specialising in helping customers to derive more value from their corporate data. The company is following an aggressive growth path, with 17% growth in consolidated revenue in 2005, which was the year in which the case study was carried out. Revenues have since then continued their upward growth trend, topping $11.2 billion in 2006. SIT Ltd employs over 26,500 people in 52 operations worldwide. Manufacturing is concentrated in three sites, one of which is Cork, Ireland. SIT has evolved into a solutions company, delivering not just hardware, but also information lifecycle tools and consulting services. A key complexity of such full service offerings is the management of the information flows related to executing a single customer order, which increasingly is constituted of hardware, software and services. Many of these revenue lines are executed by multiple locations, over different time horizons, yet the customer will require a single sales order, single invoice and single goods shipment. SIT implemented a single instance global ERP system in 2001. This big bang implementation addressed user requirements for transaction processing in all back office activities relating to sales order processing, manufacturing, materials planning, distribution and finance. The Oracle based system supports 4,500 users in 52 countries worldwide, 3 of which involve manufacturing operations.

3. The research methodology

The premise of this study is that data and process integration are inherent parts of an ERP package. Organisations make considerable assumptions about the level of integration between functions, plants, and headquarters when implementing an ERP system. Although differences exist between the different ERP systems, and these differences will have an impact on the fit with individual business process requirements, it is assumed, as with Gattiker & Goodhue (2005), that the differences are less important than the similarities. The similarities are that all ERP systems are highly integrated at a data level, and all ERP systems use workflow logic to automate the flows of information through the different stages of its transaction process. It is via this data and process integration that different functions can access and transact information relating to the use of common enterprise resources. Following the Barua et al. (1995) recommendations for evaluating IT investments, Gattiker & Goodhue (2005) selected a fine grained unit of analysis (the plant) and within that the functions examined are close to the operating core of the business (manufacturing planning and control, as well as execution processes). The logic here is that longer lasting transformative benefits on the organisation are more likely to be derived from core value-adding activities of the company, rather than from administrative functions Barua et al. (1995). This research study includes both core operational and support functions, as ERP is inherently an administrative (support function) tool, yet its impact is most felt at the transactional level (operations). The data from interviews regarding ERP impact was classified with respect to organisational parameters, and these parameters had been identified as seed categories from the literature, but other themes also emerged from the analysis of managerial perceptions. The seed categories included themes such as centralisation of responsibility, standardisation of processes and gaps between template process and reality. The emergent attributes include granularity of information, process flexibility and manual manipulation of data.

www.ejise.com

20

Academic Conferences Ltd

Fergal Carton and Frederic Adam

Business processes were used to analyse findings as an embedded unit of analysis, in order to better identify areas of the organisation where the integration impact was most strongly felt. ERP systems superimpose a process based view of business activity, with integration at data and workflow level. In many cases, these new processes cross traditional functional boundaries, thus it was considered important to be able to view the impact equally from the process level. The definition of processes is closely allied the physical management of resources in the organisation, such as inventory, customer orders, cash, labour or plant capacity. Information flows in a logical and chronological order between different stages of the process as the physical resources are transferred, consumed and transformed through different stages of the supply and demand cycles. Interviews with managers from both cases were carried out in the period from April 2005 to August 2005 and involved meetings with 76 managers from different functions affected by the implementation of the ERP system. SIT had gone live on their ERP system in October 2001, so these interviews reflect the views of managers using a relatively mature system. Table 1 shows a summary of the numbers of managers interviewed, broken down by case and by function. Table 2: Breakdown of interviews by case and by functional area
Function Finance Manufacturing / Distribution Sales IS Engineering HR Total Cork 5 13 4 4 2 2 30 SIT US 9 6 3 2 Total 14 19 7 6 2 2 50 Cork 1 9 1 3 1 15 KPC UK Total 1 2 7 16 1 3 6 1 11 26

20

Interviews were carried out using a semi-structured format, and each interview lasted one hour. The interviews were recorded and transcribed, yielding over 400,000 words of raw research material. A robust coding methodology was applied to reduce the data and avoid paralysis by data analysis (Yin 2003). Observations from the transcripts were extracted to a matrix structured by research question, yielding a total of 3,202 observations. Cell entries were either abridged versions of the original quote, summarised to capture the issue raised. Using hyperlink functionality between Microsoft Excel and Word, each extracted observation was linked back to the original transcript, thereby retaining richness and avoiding too thin cell entries (Miles & Huberman 1994). The observations were coded at three levels, identifying the interviewee, the business process commented, and the theme (organisational attribute). The business process was recorded independently of the functional affiliation of the interviewee. For example, observations regarding shipment decisions were classified as pertaining to the Deliver process, and were commented upon by managers from Finance, Materials, Manufacturing, Distribution, and Sales. The themes began as a set of 13 seed categories, developed from the literature review, and complemented by themes which emerged from the data.

4. Findings of the case study

The empirical data was analysed to answer the three research questions mentioned in the research objective section of this paper. These questions form a vehicle to investigate the effect of ERP driven integration in todays multinational company. Research question 1: Key organisational attributes Research question 1 identifies the organisational attributes associated with ERP driven integration. These observations were coded based on whether the impact of the attribute was perceived to be related to organisational, decisional or integration themes, and categorised by process area. The field data is aggregated across both cases (n=2,818), and plotted according to three dimensions. Figure 1 summarises the results of this analysis of organisational attributes. Organisational themes (y-axis values) included Centralisation, Goals, Compliance and Skills. The 695 observations classified under the organisational themes related to the positive aspects of centralisation, goal clarity, compliance and the development of new skills. Integration themes (x-axis) included Correlation, Granularity,
www.ejise.com

21

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (17 26) Accuracy / Consistency, Automation and Aggregation. Again, the 883 observations in this category emphasised the positive aspects of integration, particularly the analytical potential. The Decisional themes (denoted by size of bubble) were Manual, Flexibility, Gap Virtual vs. Physical, Uncertainty and Latency. The 1,060 observations under the decisional theme were largely negative feedback regarding the difficulties of exploiting ERP data.

400 350 Make 300

Organisational attributes

250 200 150 Deliver 100 50 0 -50 -50 Integration attributes 0 Forecast 50 100 150 200 250 300 350 Approve & Pay Buy Plan Quote to Cash

Report

Figure 1: The organisational impact of ERP integration by business process area These results reveal a strong linear relationship between centralisation and integration. That is, the stronger the consensus around the need for a centralising force in the organisation, the stronger the perceived need (among managers) for integration. This relationship holds across the different process areas. It could be deduced from this data that organisational goals are well understood and assimilated in the different functions, and that centralisation of the organisation is wholly accepted. There is a good understanding among managers of the impact of centralisation for data integrity, and a good understanding of the potential to use the associated wealth of highly granular information for different decision scenarios. This linear relationship would seem to suggest a broad acceptance among managers that centralisation is necessary from the point of view of goal focus, and that the more centralised the organisation, the more integration is required and accepted. The corollary of this finding, not tested in this sample, would be that more decentralised organisations would have difficulty accepting integration. On the other hand, what is equally striking from these results is the level of unanimity in the frustration felt throughout the business in exploiting corporate information for decision support. Because of performance and security issues with the technology, and its fit with actual business activities, managers are obliged to resort to much manual manipulation of the information derived from the ERP system in order to get to the meaning they require. It was of interest to note that the acceptance of centralisation varied between the two cases observed, with SIT showing more consensus on centralisation for Buying and Shipping, than KPC who were more forceful in their support for centralisation for Planning and Making. It is thought that the regulatory environment explains the KPC willingness to accept compliant manufacturing processes, whereas SIT had moved Purchasing to a global process, and were debating how to change the Deliver process from a site specific activity to a global basis. These nuances demonstrate that the acceptance of integration should be considered from a site and functional level. Research question 2: Causality map between attributes

www.ejise.com

22

Academic Conferences Ltd

Fergal Carton and Frederic Adam

The analysis of Research question 1 served to establish the existence of a strong relationship between the centralisation attributes and the integration attributes. Integration is accepted by managers as it is understood that it is a pre-requisite for better control and visibility of organisational activity. The granularity of information available possibly encourages requirements for managers to correlate data in ways that were not envisaged by the integrated applications used, and with information from other sources. The increased accuracy of information benefits Manufacturing and Finance particularly. The downside of this integration is that ERP systems are inflexible and unforthcoming with meaningful information. Organisations seek to exercise control and reduce uncertainty by multiplying the number of data points, but the implementation of tools and procedures to collect the data in a changing reality becomes a cause of bureaucratic inflexibility. This inflexibility is a cause for users to resort to workarounds, which engender gaps between the physical reality and the virtual picture used to monitor its progress. These gaps or virtual blind-spots mean managers return to manual methods in manipulating information for decision making purposes. This additional manipulation introduces latency in reporting, which is compounded by response time issues deriving from the technical architecture. The net result is a requirement for skilled resources in the manipulation of corporate data to derive meaning. Thus, as integration increases with centralisation, there is a corresponding increase in the necessity for skills to manipulate and digest the ensuing glut of data. Finally, and perhaps most worryingly, users in the most sensitive execution areas (Manufacturing and Deliver processes) are handicapped by inflexibility in the transaction processing system, whereas their focus should be on material availability and customer satisfaction respectively. These elements are depicted graphically in Figure 2.

Control

Centralisation Co-ordination

More analysis potential

Integration

Granularity Accuracy / consistency

Inflexibility

Deteriorating data integrity Technical skills required for reporting

Latency Manual

Figure 2: Understanding the causality between control, integration and its impact Research question 3: Conceptual framework for ERP integration and functionality gaps The first two research questions highlighted the themes related to the impact of ERP driven integration on the organisation, and the causality associated with these themes. The final research question draws on this output to suggest a framework for the conceptualisation of integration in the 0 organisation, named here as the Zipper effect and shown in Figure 3. At go-live (t ), a certain number of gaps will exist between the configured ERP processes and the way the company actually works. These will typically be dealt with by workarounds, but managers do not have visibility of the impact for data integrity of these workarounds, or their cost. The suggestion behind the Zipper effect is that these gaps between the physical and the virtual will inevitably emerge over time, but that acknowledging that they exist is the first step in addressing them. The second step is to establish

www.ejise.com

23

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (17 26) processes for monitoring these gaps, analysing and explaining them. Once recognised, the third step for organisations is to have the resources and procedures in place to be able to resolve them.

Data integrity gaps evolve over time t

n

t2

Increasing use of tools to manipulate data in order to derive meaning from data (corporate DW, BI tools and spreadsheets)

Go-live + n years

Business model changes, new channel partners, outlsourcing supply chain, volumes increase, margins shrink,

Gaps in data integrity

Gaps filled at transactional level by more workarounds or system customisation or at reporting level with BI

Workarounds found or changes are made to ways of working to maintain ERP data integrity

ERP templates customised to allow for company specific requirements

t1 Reality t0
5. Conclusions

Go-live

ERP
(virtual picture of use of resources)

(physical consumption of resources)

Figure 3: The Zipper effect: modelling the divergence of ERP from reality

This research highlights a fundamental difficulty for the use of ERP systems in large corporations. Integration involves virtualising the communication process through the use of information systems such as ERP. The accuracy of the information is a function of the closeness of the template parameters to the actual way of doing business. The two cases studied demonstrate that the virtualisation of business processes engenders constraints that negate the original purpose of improved control. These constraints arise from either the virtual or the physical context. The combination of a centralised technical architecture and large volumes of highly granular information creates latency issues (virtual context) which impacts at both an execution and a reporting level. The inflexibility of the standardised ways of working to the evolving business imperatives (physical context) obliges managers to resort to manual methods when analysing performance variances, defeating the original aim of an integrated approach to performance control, and incurring an additional cost. Another key lesson from this study has been that an integrated model of how businesses function is theoretical and ill adapted to the reality of how businesses operate. Managers in both studies confessed to their inability to build conceptual models of the decision processes required in the demand and supply processes of the firm because of their sheer complexity. In such complex scenarios, the imposition of standardised procedures appears to inhibit rather than encourage an understanding of the key variables. Accordingly, the notion of integration should be treated as a normative force in organisational administration, along with other organisational attributes, but not as a once-off solution embodied in the ERP software. Raising the awareness of integration as a

www.ejise.com

24

Academic Conferences Ltd

Fergal Carton and Frederic Adam

parameter of organisational design should be an objective for researchers and practitioners alike, rather than being left in the hands of vendors of solutions.

References
Adam F. and Sammon D. (2004) The Enterprise Resource Planning Decade: Lessons Learned And Issues For The Future, IPG, Hershey, PA Barua, A., Kriebel, C. H., and Mukhopadhyay, T. (1995) Information Technologies and Business Value: An Analytic and Empirical Investigation, Information Systems Research, Vol. 6, No.1, pp 3-23 Child, J. (1973) Strategies of Control and Organisational Behaviour, Administrative Science Quarterly Davenport , T.H. (1998) Putting the Enterprise into the Enterprise System, Harvard Business Review, Jul-Aug Dearden, J. (1972) MIS is a mirage, Harvard Business Review, Jan/Feb, Vol. 50, No. 1, pp 90-99 Gattiker, T.F. and Goodhue. D.L. (2005) What Happens After ERP Implementation: Understanding the Impact of Interdependence and Differentiation on Plant-Level Outcomes, MIS Quarterly, Vol. 29, No. 3, September, pp 559-585, Galbraith J.R. (1974) Organisation design: an information processing view, Interfaces, Vol. 4, No. 3, pp 8-37. Gorry, G. (1971) The Development of Managerial Models, Sloan Management Review, Winter, pp 1-16 Gorry G. & Scott Morton, M. (1971) A Framework for Management Information Systems, Sloan Management Review, Fall, Vol. 13 No. 1, p 49-61 Inkson, J., Hickson, D. & Pugh, D. (1968), Administrative reduction of variance in organization and behaviour, unpublished paper given to the British Psychological Society, Annual Conference, April Knights, D. and Willmott, H. (1988), New technology and the labour process, Macmillian, London, pp 20-49 Leavitt, H. and Whisler, T. (1958) Management in the 1980s : New information flows cut new organisation channels, Harvard Business Review, Nov-Dec, pp 41-48 Lee, Z. and Lee J. (2000), An ERP implementation case study from a knowledge transfer perspective, Journal of Information Technology, Vol. 15, pp 281288 Lucas et al. (1980) The Information Systems Environment, North-Holland Publishing Company, IFIP Markus, M.L. and Robey, D. (1988) Information technology and organisational change causal structure in theory and research. Management Science, Vol. 34, No. 5, May, pp 583-598 Markus et al. (2000), Learning from adopters experiences with ERP: Problems encountered and success achieved, Journal of Information Technology, Vol.15, 245265 nd Miles, M., and Huberman, A. (1994) Qualitative Data Analysis: An Expanded Sourcebook, 2 edition, Sage Publications, CA Mintzberg, H. (1989) Mintzberg on Management, The Free Press, New York Pennings (1983) Decision Making : an Organisational Behaviour Approach, Markus Wiener Publishing Inc., New York, pp 131-139 th Pugh, D.S. (1997), Organization Theory: Selected Readings, 4 edition, Penguin Books, England rd Simon, H. (1977) The new Science of Management Decision, 3 edition, Prentice Hall, Englewood Cliffs, NJ Thompson J. (1967) Organisations in Action, McGraw-Hill, New York rd Yin, R.K. (2003) Case Study Research, Design and Methods, 3 edition, Sage Publications, CA

www.ejise.com

25

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (17 26)

www.ejise.com

26

Academic Conferences Ltd

Understanding IT Management in SMEs

Paul Cragg, Annette Mills and Theek Suraweera University of Canterbury, Christchurch, New Zealand
paul.cragg@canterbury.ac.nz
Abstract: There is evidence in the IT literature indicating that IT management is one factor that influences IT success. In addition, there is much literature indicating that IT management is important in the SME context. However, much of this literature has focused on the important role of the owner and/or other senior managers. For example, Thong et al (1996) focused on top management support and its influence on IT success. This paper argues that top management support is only one aspect of IT management and other aspects of IT management have received little attention in studies of SMEs. The study commenced with a review of the literature which identified many different definitions of IT management. However, the broader management literature indicated that the classical functions of planning, organising, controlling and leading provide an excellent way of conceptualising the many activities involved in managing (Carroll and Gillen, 1987). The management literature also provided working definitions for the four management functions. A multiple-case study approach was then used to collect evidence from four SMEs (with between four and 50 employees). The data identified IT management practices associated with each of the four IT functions, ie, IT planning, IT organising, IT controlling and IT leading. For example, one firm had an IT committee made up of staff from different levels of the firm. Some on the committee would be assigned specific IT responsibilities, eg, training of other staff. This practice provided an example of IT organising, ie, of defining tasks and assigning personnel. Similarly, many other IT management practices were both identified and classified. The results provide a significant foundation for researchers of IT management in SMEs. For example, the conceptualisation indicates four IT management functions. Also, the definitions clarify important aspects of IT management. The study also indicated that some aspects of IT management have received little attention in prior studies. For example, while IT planning has received considerable attention, IT leading in SMEs has only been examined in the narrower context of top management support. Furthermore, few studies have examined the role and importance of IT organising and IT controlling in the SME context. Thus the study also indicates directions for future research, including the identification of IT management best practices. Keywords: small business, IT management, IT management practices, accounting firms

1. Introduction
Although the significance of IT management in the context of small firms has been emphasised in the IT literature, so far a consensus has not been reached with respect to the definition of this construct. At present, we have a poor understanding of the concept of IT management, particularly in the small firm environment. This study aimed to characterise IT management in the context of small firms with a view to exploring the concept of IT management sophistication, that is, of some firms being more sophisticated than others in their approach to IT management. For example, a firm that had no agreed IT plan would be considered less sophisticated in terms of IT management than a firm that had an agreed IT plan. Similarly, a firm that did not review its IT plan would be considered less sophisticated than a firm that reviewed its IT plan annually. The research objective was addressed using case study research of four small chartered accountancy firms in New Zealand. The case studies explored the IT management practices of small firms. The evidence was used to derive a pool of indicators that can be used to measure IT management sophistication in small firms. This paper is organised under five sections. The literature review is presented next and then the research approach. This is followed by a discussion of the case study data and the resultant pool of indicators that characterise IT management sophistication in small firms. Research conclusions and limitations are also presented.

2. Literature review
This study focuses on the concept of IT management. As different authors define the term IT management in different ways, we start by clarifying the term IT management, before proceeding to discuss the concept of IT management sophistication. In this study IT management is viewed as a broad concept incorporating topics like managing the IT resource as a competitive weapon as well as the development and operation of information systems (Luftman, 2004). Thus it includes the ISSN 1566-6379 27 Academic Conferences Ltd
Reference this paper as: Cragg, P, Mills, A and Suraweera, T. (2010) Understanding IT Management in SMEs The Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010, (pp27 - 34), available online at www.ejise.com

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (27 34) management of IT resources including personnel and technical infrastructure, and management of information systems. The definition recognises that some of these management processes are not carried out by IT personnel. This was particularly important for the study's focus on small firms as many small firms have no IT department (Cragg, 2002). There have been a number of attempts by IS researchers to characterise IT management. For example, as early as 1973, Nolan referred to the important IT management activities of planning, organising and control. Planning, organising and controlling are common to many frameworks characterising IT management (Earl, 1989; Raymond and Pare, 1992; Gupta et al, 1997). However, some studies include aspects not shared by other frameworks. For example, IS/IT leadership, relationship building and business systems thinking are identified as core IS capabilities by Feeny and Willcocks (1998), but these aspects were not incorporated by others. Thus researchers have adopted a range of different dimensions to characterise IT management, indicating that there is no commonly accepted characterisation of the term IT management. With a lack of a consensus within the IT literature, we turned to the management literature for clarification of the concept of 'management'. The management literature also shows a range of definitions of what is meant by management. However, many authors refer to the 'functions of management'. For example, Daft (2000) refers to the The four management functions of: planning, organizing, leading and controlling" (p. 5). Schermerhorn (1989) uses exactly the same framework. Overall, the IT literature shows no consensus for the term IT management. Some researchers have referred to IT management in as few as three dimensions. Others view IT management having many more dimensions. However, many frameworks incorporate three dimensions that reflect the three management functions of planning, organising and controlling. The management literature agrees with this typology but adds 'leading' as a fourth function. The study thus initially viewed IT management as the four functions of IT planning, IT organising, IT leading and IT controlling. The following brief definitions from Schermerhorn (1989) proved useful in clarifying what each function referred to. Planning determining what is to be achieved, setting goals, and identifying appropriate action steps. Planning centres on determining goals and the means to achieve them. Organizing allocating and arranging human and material resources in appropriate combinations to implement plans. Organizing turns plans into action potential by defining tasks, assigning personnel, and supporting them with resources. Leading guiding the work efforts of other people in directions appropriate to action plans. Leading involves building commitments and encouraging work efforts that support goal attainment. Controlling monitoring performance, comparing results to goals, and taking corrective action. Controlling is a process of gathering and interpreting performance feedback as a basis for constructive action and change.

3. Research methodology
This study relied on the case study method. This method was found to be useful in small firm studies and also in IS research leading to theory development (Caldeira and Ward, 2002; Yin, 1994). A multiple-case design was chosen. An exploratory case' was followed by multiple case studies. The objective in seeking subsequent cases was to identify additional dimensions of IT management and confirm those dimensions already identified. Also, the firms were selected with the aim of covering a wide range of practices related to sophistication of IT management in the small chartered accountancy business sector. The unit of analysis identified for the study was the individual firm. Small and medium sized, independently owned and operated chartered accounting (CA) firms were targeted. Independence was an important criteria as the IT management practices of subsidiaries (or branch units) of larger organisations are often influenced by the parent organisations. CA firms were selected because CA firms allowed the study to focus on IT management sophistication rather than technical sophistication.

www.ejise.com

28

Academic Conferences Ltd

Paul Cragg et al.

Most CA firms employ similar IT and use IT in similar ways. Thus choosing CA firms helped minimise distractions due to differences in technical sophistication. The case studies were conducted in four small CA firms in Christchurch, New Zealand. These are subsequently identified in this paper as CAF1, CAF2, CAF3 and CAF4 respectively. Interviews were the primary data collection technique. In all cases, interviews were held with a partner who had a senior role within the firm, was a shareholder, and was responsible for IT. Interviews were also held with other staff with IT responsibilities. These discussions lasted about one hour each and were guided by open-ended questions. A detailed questionnaire was not used. Instead, questions were asked about the following four broad areas of IT management: planning, organising, controlling and leading, based on the literature review. The sessions were audio recorded and the content transcribed. Follow-up visits and the telephone were used to clarify matters arising from the interviews. In addition to the four case firms, data was also collected from three IT consultants, who were closely associated with CA firms. These investigations were identified as Consultant 1, Consultant 2 and Consultant 3 (abbreviated to CON1, CON2 and CON3 respectively). These professionals were familiar with the broad range of IT management practices and other aspects such as use of technology and the issues and problems related to IT in CA firms. Such supplementary information strengthened the validity of the study through triangulation. The background information of the CA firms investigated is given in Table 1. The total staff ranged from 3 to 50 and the number of partners ranged from 1 to 9. Two firms were using PC networks while the smallest firm (CAF2) used standalone PCs. A variety of accounting software was used by these firms. Obtaining the services of a designated IT consultant was a feature in the three larger firms. Table 1: Profile of the case study firms
CAF1 (CA Firm 1) Number of partners Number of accounting staff Total number of staff Computer system Accounting software* 3 15 CAF2 (CA Firm 2) 1 (part-time) CAF3 (CA Firm 3) 6 20 CAF4 (CA Firm 4) 9 45

~20

~25 IBM mini system with 14 computer terminals plus two PCs Customised accounting package with IBM system APS for tax processing Business Service Manager Yes

~50 PC Network (~50 Pcs)

PC Network (14 pcs ) CA Systems (MYOB practice systems), moving to MS Windows base Partner (in-charge IT) Yes

2 stand alone Attache Moving on to CA systems Principal No

APS Partner/ Director (IT) Yes

Principal Contact External IT Consultant

(* CA systems, MYOB, Attache, APS are PC based software packages)

4. Case results
This part of the paper reports evidence of IT management practices in SMEs, under the four dimensions: IT planning, IT organising, IT controlling and IT leading.

4.1 IT planning
Planning determining what is to be achieved, setting goals, and identifying appropriate action steps. Planning centres on determining goals and the means to achieve them. (Schermerhorn, 1989, p. 20)

www.ejise.com

29

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (27 34) The cases provided considerable evidence of IT planning activities. For example, some firms had embraced a rigorous planning process. Some managers viewed this as more sophisticated than an informal, ad-hoc approach to planning. Furthermore, some firms review their IT plans to accommodate the changing needs of the firm. In one firm this review was undertaken formally every year by the senior management team. Ad-hoc reviews were conducted by a partner during the year. Oh, yes definitely [we place a high priority on IT planning]; because [in the] long term whatever solution you decide on is going to be cheaper, the better plan it is. (CAF3) The case studies revealed that the IT plan in a small firm may range from a simple reflection of the owner/manager, to a detailed written plan identifying requirements, time frame, cost structure, personnel requirements and possible constraints. The firms highlighted two important characteristics of a plan: they should be written and they should be comprehensive. We start off saying what we intend to do for next year or two And then we specify exactly what we need technically, what gear we need what training we need, and we set up a time table and try and work for that. (CAF4) We have a blueprint, say for the coming year which we try and work towards. So we dont get diverted You certainly have to know where you are trying to go to; but you have to be very flexible as well because, the ground rules change. (CAF4) Many CA firms have a practice of gathering and sharing IT related information, particularly with peers. In part this reflects the requirement for all practising accountants to engage in professional development activities, which include seminars where they meet and engage with their peers. In addition, the software suppliers run user-groups where accountants share their experiences with peers from other CA firms. These formal and informal information gathering activities can be viewed as part of the planning process where information can spark a new IT project or feed into analysis, e.g., when evaluating alternatives. ...in the last user group they were talking about converting (from DOS environment) across onto windows. We spoke about the need for having to get the hardware right. That has been a major problem in the firms that have converted across. (CAF 1).

4.2 IT organising
Organising turns plans into action potential by defining tasks, assigning personnel, and supporting them with resources. (Schermerhorn, 1989, p 20). One common practice in large firms is to have an IT department with specialist IT staff, who take much of the responsibility for IT. While none of the CA firms had an IT department, some employed staff who are IT specialists, i.e., they have formal IT training, e.g., a degree or diploma in computing/IT, and their job focuses primarily on IT activities. However, most CA firms do not have an IT specialist on their payroll. Instead they tend to delegate IT duties to at least one of their staff. This person is often a qualified accountant with some IT in their education, plus on-the-job IT experience within the firm. In addition, CA firms rely on external IT support. For example, many CA firms have contracts with local computer firms who they call in on an as-needed basis to fix problems, typically operating problems. We have had (trained IT staff) once. Like in any accounting firms there is always staff turnover. In the past we had two very good people that were very knowledgeable about computers. I got it (i.e. management function of IT in the firm) by default when (they) left. (CAF1) [It is] left to me on a day-to-day basis. I [i.e., partner in-charge of IT] am the only one who does backups. (CAF1) Its really me [in-charge of IT]. I own it, so, I basically look after that. I keep up-to-date; see whats available. (CAF2)

www.ejise.com

30

Academic Conferences Ltd

Paul Cragg et al.

Many CA firms also have an IT committee that includes staff from different levels of the firm, and includes those with specific IT responsibilities as well as those without IT responsibilities. The IT committees have been set up for many reasons, but particularly for IT planning, adoption and implementation. [We have identified a] small little IT group in our firm which; we'd say, OK, you are in charge of training and installation. (CAF4) We have two ways of managing our firm I suppose. Weve a lot of staff training external and internal; we have a small IT committee, and we work with outside consultants. (CAF4) Many CA practices employ the services of an IT vendor. Some firms prefer this practice over employing internal IT specialists and believe that it places them in a stronger position. [We employ] one. ... He has his own firm and he would deal with network problems, installation of new hardware [and] software. If there is anything wrong with the network [or] something like that then he would come in and attend to it. (CAF3) In addition, CA firms use IT consultants. We use the title consultant to differentiate these activities from the typically technical support activities carried out by IT vendors. IT consultants are called on to offer advice. For example, to provide independent advice about a new technology, e.g., should they upgrade to Vista? In addition, consultants are often used in the hardware and software selection process, sometimes in a significant way, e.g., as project manager, and to make a recommendation after identifying possible alternative systems. While many CA firms have an IT committee, and when needed create an IT project team, they also need to maintain relationships with external IT providers like consultants and vendors. This is another important aspect of organising, as it reflects a mechanism for co-ordinating tasks.

4.3 IT controlling
Controlling monitoring performance, comparing results to goals, and taking corrective action. Controlling is a process of gathering and interpreting performance feedback as a basis for constructive action and change. (Schermerhorn, 1989, p. 20) The CA firms provided considerable evidence of monitoring practices, including procedures for staff. The monitoring practices in some firms reflected strong involvement of staff, and some of the monitoring was conducted by external providers. The firms monitored IT projects using a mix of both formal and informal techniques. For example, it was common for any IT committee to seek progress reports on IT projects. Weve got meetings with all the staff every fortnight and computers is one of the issues that is discussed every time. Probably our monitoring is based on identifying how many problems we have if it is running smoothly. We are OK. (CAF1) The controlling activities included procedures that had been devised and implemented. In particular, data security was an issue, partly because CA firms hold and analyse financial data for clients. Some firms took few risks with this data. "At the moment internally we dont really have much [internal] security. Our systems are open; Just about everyone in the office can actually look at anything in the system." (CAF4) Other examples of procedures include version control, when to replace hardware, and who has internet access. There is no control over what software has been loaded on the PCs there is often different versions of all software on all the PCs depending on when they brought them. (CON2) Procedures also relate to the IT adoption process. For example, to increase project success, CA firms monitored their IT implementation plans.

www.ejise.com

31

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (27 34) Many CA firms also provide training to their staff. Providing training can be viewed as part of the firms policies and procedures that ensure high quality work, ie, to minimise risks and increase the chances that things will go well in the future. Yes, we do [value staff training on IT very much and spend a lot of money on training.] I think more time than money. (CAF4)

4.4 IT leading
Leading guiding the work efforts of other people in directions appropriate to action plans. Leading involves building commitments and encouraging work efforts that support goal attainment. (Schermerhorn, 1989, p. 20) The case studies indicated that the leadership qualities of managers have a strong impact on the effectiveness and efficiency of the IT systems of the firm. Reference was made to a number of broad issues of leadership and the leadership qualities of managers relating to IT. The cases showed that top management involved themselves in many different ways. Strategy and planning was one key area with a partner often in charge of IT. High priority [is given] to the [IT review] team. (CAF3) One firm displayed a strong IT management focus that had eliminated people whose attitude towards IT was not supportive. Encouraging and promoting positive attitudes and ensuring staff satisfaction are characteristics of strong leadership. They (partners) have just said go with it (new IT proposals). So theyve been very supportive. (CAF4) There was also evidence of senior managers taking responsibility for IT by, e.g., pushing IT staff: If some one does not push people in the [IT] team, they would just reach the comfort zone and stay at that level I think. (CAF4) Some IT partners were also very close to operational issues. For example, Its really me [in-charge of IT]. I own it, so, I basically look after that. I keep up-to-date; see whats available. (CAF2) Small CA firms do believe that staff training on IT is important for success. For example, some firms had regular training programmes and placed a high priority on staff training. Yes, we do [value staff training on IT very much and spend a lot of money on training.] I think more time than money. (CAF4)

5. Discussion
The most important contribution of this study is that it identified four dimensions that characterise IT management sophistication in small firms: IT planning, IT organising, IT controlling, and IT leading. To a large extent, this major finding conforms with past research findings. For example, IT planning, IT organising and IT controlling have been identified as three major dimensions that determine IT management in the context of large firms (Earl, 1989; Gupta et al., 1997; Nolan, 1973) and in small firms (Raymond and Pare, 1992). Also, leading has been well recognised as a function of management within the management literature (Schermerhorn, 1989) and within a growing body of IS literature (Feeny and Willcocks, 1998; Karahanna and Watson, 2006). However, of the eight studies that attempted to define IT management that were identified in the literature review, only Feeny and Willcocks (1998) refer directly to IT leadership as an important dimension of IT Management. Thus IT leading has not been regarded as a major sub-dimension of IT management by past researchers, although others have referred directly, or indirectly, to this aspect of managing information technology. For example, Caldeira and Ward (2002) recognised the importance of managerial skills in gaining IT success in SMEs. Gupta et al. (1997) had a measure related to the leading role of top management under IT integration (i.e. In my firm, top management perceives that future exploitation of IT is of strategic importance). Although the small firm instrument

www.ejise.com

32

Academic Conferences Ltd

Paul Cragg et al.

by Raymond and Pare (1992) has a measure of top management involvement, it did not treat IT Leadership as a major factor. This study indicates that IT Leadership is a major aspect of IT management sophistication in small firms. This important finding supports many prior studies of IT in small firms, e.g., Caldeira and Ward (2002), Cragg (2002). Compared to IT Planning, relatively much less IS research has examined the concept of IT Organising, although many authors discuss aspects of topics like IT outsourcing, IT governance, and the structure and location of the IT department (Earl, 1989; Gupta et al., 1997; Nolan, 1973). Of three existing instruments, only Raymond and Pare (1992) paid much attention to IT Organising. This study focused on small firms, unlike most prior studies of IT management. Importantly, the study provides evidence that dimensions of IT management discussed in the large firm IS literature are also applicable in the context of small firms, i.e., IT planning, IT organising, IT controlling and IT leading. Another significant contribution of this study is that it identified indicators of the four dimensions that make up IT management sophistication in small firms. Importantly, these indicators could be used by other researchers to, for example, examine all or a selection of the dimensions. However, other researchers must recognise that further testing should be conducted on the indicators, for example, of validity and reliability using survey type data.

6. Conclusions
This study of IT management sophistication within small firms identified many indicators of IT management sophistication in SMEs, covering the four dimensions of IT planning, IT organising, IT controlling and IT leading. These results fit well with the findings from past research. Many of the indicators relating to planning, controlling and organising have been used in prior research to characterise IT management sophistication. IT leading emerged as a relevant factor of IT management sophistication in small firms according to the current study. Past research and the case study evidence also supported the inclusion of indicators pertaining to IT external expertise. The studys major contribution is the characterisation of IT management as four dimensions. This understanding can help IS researchers characterise IT management sophistication in small firms' and further explore the relationships between this and other related constructs, such as IS success, IT enabled organisational performance, and competitive advantage. The characterisation may also help practitioners determine strengths and weaknesses of IT management in small firms. These results could then be used to help formulate appropriate strategies aimed at improving IT management sophistication in a small firm. A major limitation of this research was the restriction of the study to four small and medium sized chartered accountancy firms in New Zealand. Given the diversity within the small business sector, this element restricts the generalisability of the findings. Furthermore, most of the firm level data was collected from one senior person within each firm, who could present a biased view of IT management within their firm.

References
Bergeron, F., Raymond, L. and Rivard, S. (2004). Ideal patterns of strategic alignment and business performance, Information & Management, 41, pp 1003-1020. Caldeira, M.M. and Ward, J.M. (2002). Understanding the successful adoption and use of IS/IT in SMEs: an explanation from Portuguese manufacturing industries, Information Systems Journal, 12(2), 121-152. Cragg, P.B. (2002). Benchmarking Information Technology Practices in Small Firms. European Journal of Information Systems. 11(4), pp. 267-282. Daft, R.L. (2000). Management. The Dryden Press, New York. Earl, M.J. (1989). Management Strategies for Information Technology. Prentice Hall, United Kingdom. Feeny, D.F., and Willcocks, L.P. (1998). Core IS Capabilities for Exploring Information Technology, Sloan Management Review, Vol. 39, No. 3, pp 9-21, 1998. Gupta, Y.P., Karimi, Somers, M. (1997). Alignment of Firms Competitive Strategy and Information Technology Management Sophistication: the Missing Link, IEEE Transactions of Engineering Management, Vol. 44, No. 4, pp 399-412. Karahanna, E. and Watson, R.T. (2006). Information Systems Leadership, IEEE Transactions on Engineering Management, 53:2, May 2006, 171-176. Luftman, J. (2004). Managing the Information Technology Resource, Pearson. Nolan, R.L. (1973). Managing the Computer Resources: A Stage Hypothesis, Communications of the ACM, Vol. 16, No. 7, pp 399-405.

www.ejise.com

33

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (27 34)
Raymond, L., Pare, G. (1992). Measurement of Information Technology Sophistication in Small Manufacturing Business, Information Resources Management Journal, Vol. 5, No. 2, pp 4-15 . Sabherwal, R. and Kirs, P. (1994). The Alignment between Organisational Critical Success Factors and Information Technology Capability in Academic Institutions, Decision Sciences, Vol. 25, No. 2, pp 301- 330, 1994. Schermerhorn, J.R. (1989). Management for Productivity, John Wiley & Sons, New York. Yin, R.K. (2003). Case Study Research Design and Methods, 3rd Ed, Sage, California.

www.ejise.com

34

Academic Conferences Ltd

Testing of a Model Evaluating e-Government Portal Acceptance and Satisfaction

Cora Sio Kuan Lai1 and Guilherme Pires2 1 Macao Polytechnic Institute, Macao SAR, PRC 2 University of Newcastle, Australia
H

coralai@ipm.edu.mo

Guilherme.Pires@newcastle.edu.au
Abstract: e-Government has the potential to improve public administration efficiency by increasing convenience, performance and accessibility of government information and service to users. But knowledge about eGovernment remains limited. To realize its potential, e-Government needs to be grounded on in-depth understanding of target users needs, perceptions and other factors influencing its uptake. This cross-sectional study identifies and examines factors influencing e-Government portal satisfaction and adoption by individual citizens in Macao, three years after its inauguration. It is an adaptation to the e-Government context of a model developed for assessing e-commerce websites. To understand the determinants of e-government portal adoption, an integrated model of user satisfaction and technology acceptance is empirically tested. The integrated model involves four success factors -- information quality, system quality, perceived effectiveness and social influence -- which impact user satisfaction with the e-Government website, influencing intention to reuse. Overall, the study proposes that user perceptions about the e-Government portal influence user attitude towards the portal. An Internet survey collected data from 464 online users of Macaos e-government portal. The model was found to explain a large proportion of the variance in citizens intention to reuse the portal. The portal partially mediates the relationship between success factors and intention-to-reuse. The results provide evidence that Information Quality, System Quality and Social Influence (but not Perceived Effectiveness) are success factors influencing user satisfaction and adoption. It is recommended that portal management needs to ensure ease-ofuse, currency and accuracy of the supplied information. Timely information updating is a major concern for the eGovernment portal in Macao. The content an e-government portal that is perceived by users to be easier to navigate is likely to facilitate satisfaction and reuse. Finally, the importance of social influence justifies, managerial actions aimed at improving e-Government portal acceptance by individual users and government employees.

Keywords: e-government portal, adoption, satisfaction, TAM, EUS

1. Introduction
0B

E-Government has been comprehensively defined as a broad-based transformation initiative, enabled by leveraging the capabilities of information and communication technology (ICT); (1) to develop and deliver high quality, seamless and integrated public services; (2) to enable effective constituent relationship management; and (3) to support the economic and social development goals of citizens, business, and civil society at local, state, national and international levels (Grant & Chau, 2004,p. 8), although there are other definitions with varying emphasis and perspectives. Such a definition recognizes that the integration of a citizen / customer orientation with the adoption of the Internet and other ICT as new delivery channels may result in satisfaction for individual citizens and businesses alike (Ho, 2002). Moreover, e-Government has the potential to build better relationships between the government and its constituents, through citizen empowerment and increased transparency (World Bank, 2002). Nevertheless, e-Government inherits the less positive characteristics of information systems (IS), including a high failure rate of information technology (IT) based projects (Schepers & Wetzels, 2007) and user adoption rates below expectations (Davis, 1993). Low adoption may be explained as a temporary characteristic of the transition process of users from the traditional public service delivery mode to the new e-version, and by reasons ranging from supply side imperfect dissemination of information about the new services and system complexity, to demand related user lack of proficiency in the use of e-services and resistance to innovation. E-Government deployment success depends on achieving a state of adoption that combines user satisfaction gains with the discontinuation of traditional processes. As long as users can choose between using e-Government and traditional
ISSN 1566-6379 35 Academic Conferences Ltd Reference this paper as: Lai, C, S, A, and Pires, G. (2010) Testing of a Model Evaluating e-Government Portal Acceptance and Satisfaction The Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010, (pp35 - 46), available online at www.ejise.com

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (35 46)

methods to access public services, adoption can be expected to depend on external user attitudes and perceptions.

2. e-Government in Macao
1B

Similar to other countries, Macaos SAR government, in 1999, had already deployed e-government initiatives decentralized at agencies level by providing information and service on Internet and Intranet (Janowski et al., 2004). A survey revealed that the norm is for government agencies to establish a web presence with little cross-departmental cooperation (Janowski et al., 2005). The Macau SAR government portal was officially launched in December 2004, offering one single point of entry for government information targeted at residents, visitors, business and public servants (SAFP, 2004). Using United Nations classification (UNPA & ASPA,2001), more than half of the eservices (58%) accessible in the portal are at the enhanced stage, with information being regularly updated and offering search capability and downloadable forms, while 38% are in the interactive stage, characterized by online forms submission, user log-in, and specialized databases. Only 4% of the e-services are in the transactional stage, where users can complete and pay for services online. This distribution supports the findings of Janowski et al., (2005) survey of Macaos e-government.

3. Purpose and theoretical framework

2B

Notwithstanding significant investment by many governments, knowledge about e-Government remains limited. To realize its potential, e-Government initiatives need to be grounded on an in-depth understanding of potential users needs, perceptions and other factors influencing its uptake. Still in its infancy, e-government lacks a theoretically grounded and empirically proven model to evaluate its effectiveness based on external user satisfaction. This study seeks to identify and examine factors influencing the satisfaction and ultimately adoption of the e-Government portal by individual citizens in Macao. The theoretical framework used in this study is an adaptation to the e-Government context of a model developed by Schaupp et al., (2006) for assessing e-commerce websites. Hence the paper examines an existing research model applied elsewhere and tests its applicability in the Asian eGovernment context of Macao. The model integrates measures from the technology acceptance model or TAM (Davis, 1989; Davis et al., 1989) and end user satisfaction (EUS) theories (DeLone & McLean, 1992; Doll et al., 1995), examining user satisfaction when using a computer application. Both the TAM and the EUS approaches are theoretically grounded and empirically supported (Wixom & Todd, 2005). Although providing limited guidance about how to influence usage through design and implementation (Taylor & Todd, 1995) and assuming that all behaviour is under an individuals complete volitional control (Furneaux, 2006), TAM identifies perceived usefulness and ease of use as factors influencing IS adoption. The strength of EUS is in its ability to link design characteristics with user satisfaction (Wixom & Todd, 2005), despite of its inability to link attitude to behavior and to fully capture the significant determinants of satisfaction (Au et al., 2002). It is proposed that taking both approaches into account may yield a better understanding of user acceptance and adoption (Melone, 1990 and Seddon, 1997). The integrated model used in this research identifies four success factors -information quality (IQ), System Quality (SQ), perceived effectiveness (PE) and social influence (SI) -which impact user satisfaction with a e-Government website (SAT), influencing intention to reuse (IU). IQ is defined as the degree to which information on the e-Government portal is perceived to be accurate, relevant, complete and in the format required by the user (Schaupp et al., 2006). SQ is defined as the degree to which the e-Government portal is easy to use to accomplish a desired goal. It involves three dimensions, namely access (the speed and availability of the website when users attempt to use it), usability (the extent a website is visually appealing, consistent, fun and easy to use) and navigation (the extent to which the website allows a user to easily find needed information (Huizingh, 2000; McKinney et al., 2002). PE is the degree to which a user believes the website to be useful, or the website enables users to gather information about government services quickly and enhances users effectiveness in searching for and using government services on the Internet. It is similar to perceived usefulness (Davis et al., 1989), usefulness and relativeness (Moore & Benbasat, 1991) and performance expectancy (Venkatesh et al., 2003). SI refers to user perception of peers expectations about their use of the website. Deemed to be a significant predictor of IT adoption (Moore & Benbasat, 1991), SI is equivalent to social norm (Seddon, 1997) subjective norm (Thompson & Higgins, 1991) and image.

www.ejise.com

36

Academic Conferences Ltd

Cora Sio Kuan Lai and Guilherme Pires

The outcome measures are user satisfaction with the e-Government portal (SAT) and user intention to reuse (IU). Overall, the study proposes that user perceptions about the e-Government portal, influence user attitude towards the portal. The following research questions are examined: RQ1: What factors affect SAT? RQ2: What factors influence the e-Government portal adoption individual citizens? RQ3: Are there relationships between the identified factors in the explanation of portal acceptance by Macao citizens? H1: The effect of IQ on IU is mediated by SAT H1a: IQ is positively related to IU. H1b: IQ is positively related to SAT. H1c: SAT significantly predicts IU after controlling for IQ. H2: The effect of SQ on IU is mediated by SAT. H2a: SQ is positively related to IU. H2b: SQ is positively related to SAT. H2c: SAT significantly predicts IU after controlling for SQ. H3: The effect of SI on IU is mediated by SAT. H3a: SI is positively related to IU. H3b: SI is positively related to SAT. H3c: SAT significantly predicts IU after controlling for SI. H4: The effect of PE on IU is mediated by SAT. H4a: PE is positively related to IU. H4b: PE is positively related to SAT. H4c: SAT significantly predicts IU after controlling for PE. H5: Together with SAT, the success factors (IQ, SQ, SI) are a good predictors of IU.
3B

The following four hypotheses were developed from the literature to address the questions:

4. Methodology
Using a web questionnaire, this cross-sectional study is an ex post evaluation of Macaos eGovernment portal, three years after its inauguration (Doherty & King, 2004). Estimated to exceed 260,000 online population (RTHK, 2007), the target is the set of all non-business users of Macao's eGovernment portal, 18 years or older self-reported to have used the portal more than once. Table 1: The instrument used in this research was adapted from instruments previously tested for validity and reliability (see Table 1). All scales were phrased as questions on a five-point Likert scale ranging from strongly disagree to strongly agree. The satisfaction construct used a five-point semantic scale from very dissatisfied to very satisfied, very displeased to very pleased, frustrated to contented and disappointed to delighted to gauge the degree of satisfaction. Table 1: Source of scales and their reliabilities

www.ejise.com

37

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (35 46)

Construct
Success Information quality System quality

Doll & Torkzadehs,1988) Cheung & Lee, 2005; McKinney et al., 2002 Access Usability Navigation

Source of Scale

Dimensions

Cronbach's Alpha
0.92 0.8 0.97 0.86 0.8827 0.92 0.9195 0.98

Perceived Effectiveness Social influence Outcomes Intention-toreuse Portal satisfaction

Carter & Belanger, 2005; Gefen et al .,2002 Venkatesh et al., 2003 Carter & Belanger, 2005; Gefen et al ., 2002 Cheung & Lee, 2005; McKinney et al ., 2002

A web-link to the survey was made available on the e-Government portal and the questionnaire was posted on the web-hosting QuestionPro.com website for publicizing the research and collecting data. An information letter about the purpose of the survey was sent to four local organizations and community groups requesting their participation in the survey. Potential participants learnt about the survey either through their self-initiated contact with the website hosting the questionnaire, or by email via the referred organizations. Using self-selection (Ticehurst & Veal, 200) non-probability sampling, data collection took place August 15 to September 16, 2007. Data processing and analysis was performed using SPSS 13.0. To ensure data accuracy and completeness, invalid responses were excluded. Perception scales were replaced with the median for responses with missing values. The negatively worded item (PU3) was recoded. Table 2: Frequency distributions of demographic variables of repeated users

www.ejise.com

38

Academic Conferences Ltd

Cora Sio Kuan Lai and Guilherme Pires

Variables Gender Valid male female Total 18-27 28-37 38-47 48-57 >=58 Total Missing Total

Frequency 243 217 460 223 118 78 25 2 446 14 460

Variables Macau Resident or Not

Frequency 440 20 460 368 82 1 4 3 2 460

% 95.65 4.35 100.00 80.00 17.83 0.22 0.87 0.65 0.43 100.00

AgeGroup Valid

52.83 Valid yes 47.17 no 100.00 Total Internet Usage Frequency 48.48 Valid 25.65 16.96 5.43 0.43 96.96 3.04 100.00 at least once a day several times a week once each week several times a months once a month less than once a month Total

Occupation Valid public servant other prof./ exec./ managerial clerical/ service worker production worker self-employed student homemaker unemployed Total Education Level Valid primary school secondary college >=University Total Missing Total 138 4 91 85 1 14 115 4 8 460 1 49 126 278 454 6 460 30.00 0.87 19.78 18.48 0.22 3.04 25.00 0.87 1.74 100.00 0.22 10.65 27.39 60.43 98.70 1.30 100.00

Internet Experience Valid < 1 yr 1 - 3 yr 4-6 yr 7-9 yr >=10 yr Total Missing Total 1 84 99 147 128 459 1 460 0.22 18.26 21.52 31.96 27.83 99.78 0.22 100.00

Macau Government Portal Usage Frequency Valid 1 or 2 times a year several times a year once a month once a week more than once a week Total 32 166 70 76 116 460 6.96 36.09 15.22 16.52 25.22 100.00

5. Data analysis and findings

4B

The web-questionnaire was visited 1932 times, with 849 attempts to answer the questionnaire. In addition to 25 responses from once-users and 103 from non-users, 460 usable web-survey questionnaires were received from repeat users (the sample), for a total 588 questionnaires. Table 2 depicts the characteristics of the sample. Though being incongruent with the profile of Macaos population distribution, the respondents in the sample seems to be compatible to the demographic characteristics of the online population in Macau. Respondents were found to be mostly young, Macao residents, highly educated, experienced and frequent users of the Internet. The gender and age distributions were similar to the Internet user profile gathered for the Macao Internet Usage Survey (Cheong, 2007) and there is support for another

www.ejise.com

39

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (35 46)

Internet Usage study indicating that the higher the education, the more frequent the Internet usage (China.com.cn, 2006). Table 3: VARIMAX rotated loading,reliability and overall reliability of 24 items instrument

FACTOR
ITEMS
SAT1 SAT4 SAT2 SAT3 USA1 USA2 USA4 USA5 NAV1 NAV2 NAV3 SI4 SI2 SI3 SI1 UI2 UI1 ECS2 ECS3 ECS4 ECS5 ECS6 ECSB ECSC IQ UI SI SQ

Construct
SAT

Manifest Variable
satisfied delighted pleased contented simple layout ease of use clear design user friendly ease to go back and forth locate information with a few click easy to navigate influential people important people senior management of SAR Macau SAR government Intend to use predict to use relevance applicable sufficient information information accuracy accuracy timely up-to-dated

SQ

IQ

SAT
0.68 0.72 0.79 0.77

SI

UI

0.66 0.76 0.71 0.74 0.64 0.55 0.78 0.61 0.63 0.72 0.62 0.71 0.79 0.63 0.70 0.53 0.76 0.75 0.66 0.67 4.29 17.86 4.20 17.49 2.95 12.30 2.27 9.46 1.64 6.82

Sum of Square (eigenvalue) Percentage of trace

Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization. Rotation converged in 6 iterations.

5.1 Principal component analysis

7B

Exploratory factor analysis using principal component factoring with Varimax-rotation was conducted to examine the factor structure of the 38 items in the questionnaire. The ratio of sample size to number of scale items was 16:4, exceeding the minimum 10:1 ratio recommended by Kerlinger (Doll & Torkzadeh, 1988). After iterative factor analysis and item deletion, the number of items was reduced to 24 in the final instrument (as shown in Table 3), with an overall reliability of 0.94, representing 63.93% of the variance of the dataset with five factors, one less than the Schaupp et al., (2006) model used. Table 4: Mean, standard deviation, internal reliability and correlation coefficients for the latent constructs

www.ejise.com

40

Academic Conferences Ltd

Cora Sio Kuan Lai and Guilherme Pires

No. Items Success Factors Component 1 2 3 4 5 IQ SQ SI IU SAT

7 7 4

Mean

S.D.

IQ

SQ

SI

IU

SAT

3.18 3.21 2.89

0.65 0.67 0.63

0.88 0.66 0.55 0.89 0.52 0.7

Success Measures
2 4 3.17 3.77 0.63 0.68 0.47 0.67 0.41 0.64 0.36 0.54 0.61 0.42 0.9

Correlation is significant at the 0.01 level (2-tailed). Cronbach Alpha is shown on the diagonal in boldface

Table 3 shows that accuracy, relevance, completeness and timeliness are valid sub-dimensions of IQ while usability and navigation are important proportions of SQ. Calculated by using the means of the items that belonged to the scale, the descriptive statistics for the composite scale are reported in Table 4. The analysis reveals that Macaos repeat users were satisfied with the e-government portal. Comparing with SQ, users were less satisfied with the information and service provided within the portal. Table 5: Correlation matrix of measurement model (n = 460)
ECS2 ECS2 ECS3 ECS4 ECS5 ECS6 ECSB ECSC USA1 USA2 USA4 USA5 NAV1 NAV2 NAV3 SI1 SI2 SI3 SI4 SAT1 SAT4 SAT2 SAT3 IU1 IU2 ** 1 0.58 0.52 0.53 0.49 0.56 0.55 0.36 0.4 0.42 0.5 0.39 0.5 0.48 0.34 0.32 0.26 0.31 0.52 0.51 0.5 0.49 0.27 0.43 1 0.49 0.6 0.55 0.65 0.49 0.32 0.49 0.41 0.5 0.42 0.53 0.42 0.37 0.33 0.24 0.33 0.47 0.48 0.45 0.5 0.23 0.39 1 0.43 0.39 0.48 0.4 0.37 0.32 0.33 0.39 0.29 0.42 0.33 0.3 0.3 0.29 0.26 0.42 0.42 0.38 0.41 0.26 0.3 1 0.71 0.6 0.5 0.29 0.43 0.45 0.44 0.39 0.38 0.38 0.33 0.25 0.25 0.37 0.45 0.48 0.47 0.47 0.28 0.34 1 0.53 0.53 0.26 0.38 0.43 0.46 0.37 0.38 0.4 0.28 0.28 0.26 0.29 0.45 0.43 0.44 0.43 0.26 0.37 1 0.5 0.32 0.46 0.41 0.53 0.46 0.5 0.4 0.41 0.34 0.31 0.41 0.52 0.51 0.49 0.48 0.26 0.38 1 0.27 0.32 0.37 0.4 0.32 0.34 0.34 0.28 0.34 0.32 0.32 0.39 0.45 0.39 0.38 0.25 0.32 1 0.46 0.53 0.49 0.38 0.47 0.5 0.24 0.28 0.26 0.29 0.34 0.37 0.33 0.34 0.2 0.28 1 0.6 0.68 0.6 0.49 0.65 0.31 0.27 0.24 0.31 0.46 0.48 0.44 0.48 0.18 0.29 1 0.65 0.49 0.43 0.59 0.33 0.21 0.28 0.37 0.51 0.49 0.42 0.41 0.2 0.3 1 0.58 0.49 0.67 0.32 0.26 0.26 0.36 0.51 0.5 0.44 0.48 0.23 0.32 1 0.44 0.56 0.32 0.21 0.3 0.27 0.46 0.46 0.45 0.44 0.22 0.35 1 0.58 0.35 0.31 0.33 0.3 0.44 0.43 0.42 0.42 0.25 0.37 1 0.28 0.26 0.27 0.3 0.44 0.48 0.43 0.44 0.22 0.36 1 0.43 0.35 0.38 0.37 0.34 0.36 0.33 0.28 0.31 1 0.29 1 1 0.38 0.4 0.35 0.43 0.21 0.27 1 0.7 0.64 0.65 0.29 0.31 1 0.7 0.7 0.28 0.36 1 0.77 0.33 0.34 1 0.27 0.34 1 0.44 1 ECS3 ECS4 ECS5 ECS6 ECSB ECSC USA1 USA2 USA4 USA5 NAV1 NAV2 NAV3 SI1 SI2 SI3 SI4 SAT1 SAT4 SAT2 SAT3 IU1 IU2

0.34 0.41 0.33 0.32 0.37 0.34 0.27 0.3 0.33 0.3 0.27 0.08 0.28 0.09

Correlation is significant at the 0.01 level (2-tailed).

5.2 Reliability and validity testing

8B

Scale reliability was assessed using Cronbachs coefficient alpha analysis. The 24-item instrument had an overall coefficient alpha of 0.94 (Table 3), showing that all the items measured the same construct as intended. In addition, Table 4 shows the reliabilities of five summated scales were between 0.9 (SAT) and 0.61(IU). All scales exceeded the 0.70 threshold except for IU(0.61), which is still an acceptable level for newly developed constructs (Hair et al., 2006). Convergent, discriminant and criterion-related validity were assessed using factor and correlation analyses, to ensure the instrument only correlated with the expected variables. In factor analysis, significant loading of all the items on the single factor indicated unidimensionality while no cross-loading items supported instrument discriminate validity (Wang et al., 2001). These results were confirmed using the correlation approach reported in Table 5. Finally, criterion-related validity was assessed to examine the correlation between total scores (sum of 20 items) and the measure of global rating of satisfaction

www.ejise.com

41

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (35 46) (sum of SAT items), using Doll & Tokzadehs (1988) approach. Since r= 0.72 and p <.001, the instrument was deemed capable of measuring user satisfaction. In summary, the observed convergent and discriminate reliability and criterion validity jointly indicated that the psychometric properties for the measurement model warranted further analysis. Hypotheses testing was then conducted with user perceptions as independent variables, portal satisfaction (SAT) as mediator and usage intention (IU) as the dependent variable.

5.3 Hypothesis testing

9B

Hypothesis were tested to examine the factors affecting e-Government portal satisfaction and adoption, as expressed in the research questions and hypotheses stated earlier. Following Baron & Kenny (1986)s procedures, mediation analyses were conducted to understand how user perceptions affect adoption. Sobel Statistics (1982, cited in Preacher and Hayes, 2004) and bootstrapping (Precher & Hayes, 2004) were also adopted to compare and estimate the indirect effect defined as the reduction of the effect of the independent variable on the outcome (Kenny, 2006). Finally, an exploratory multiple regression analysis was carried out to better understand the mechanism underlining the adoption of an e-government portal, a relationship not previously examined (H5). 5.3.1 Mediation analyses
10B

Success factors (IQ, IQ and SI) were the independent variables; SAT was the mediator and IU the dependent variable. Several regression analyses were performed to test [condition 1] significance of the total effect of the independent variable on the dependent variable; [condition 2] significance of the effect of the independent variable on the mediator; and [condition 3] that the mediator significantly predicts the dependent variable, controlling for independent variable. H1: The Effect of IQ on IU is mediated by SAT Using Baron and Kenny (1986)s causal steps, all conditions were met thus all three sub-hypotheses were supported. The results were: H1a) IQ significantly predicted IU (b=0.49, p < 0.01). H1b) IQ significantly predicted SAT (b=0.65, p < 0.01). H1c) IQ and SAT significantly predicted IU (b=0.21, p < 0.01 and b=0.35, p < 0.01)

This means when IU is regressed on both SAT and IQ, the mediator significantly predicts IU, while the predictive utility of the independent variable is reduced from b=0.49 to b=0.35; SAT and IQ together explained 24.1 percent of the variance in IU (significant based on the ANOVA table F(1, 458)=72.67, P<0.01). That is, holding IQ constant, the effect of SAT on IU and the direct effect of IQ on IU were significant. Furthermore, the Sobel and bootstrap tests supported the mediation. The former estimated the existence of the indirect effect (Z=3.43, P<.001) and the estimated indirect effect of IQ on IU through SAT was 0.13. The latter confirmed an indirect effect between .04 and 0.22 at a 95% confidence level. The indirect effect is significantly different from zero at p < 0.01 (two tailed). The overall conclusion is that SAT mediates the relationship between IQ and IU. Hypothesis 1 is supported. H2: The effect of SQ on IU is mediated by SAT. Using Baron and Kenny (1986)s causal steps, all conditions were met thus all three sub-hypotheses were supported. The results were: H2a) SQ significantly predicted IU (b=0.41, p < 0.01:); H2b) SQ significantly predicted SAT (b=0.60, p < 0.01); H2c) Both SQ and SAT significantly predicted IU (b=0.30, p < 0.01 and b=0.23, p < 0.05); the positive relationship between SQ and IU (b=0.41) is smaller after controlling for SAT (b=0.23) with R-square =0.21).

Results suggest that SAT mediates the effect of SQ on IU. The indirect effect is.18 using the Sobel test (Z =4.88, p < 0.01); the bootstrap estimate is that the indirect effect lies between .10 and 0.26, at

www.ejise.com

42

Academic Conferences Ltd

Cora Sio Kuan Lai and Guilherme Pires

95% confidence level. The overall conclusion is that SAT mediates the relationship between SQ and IU and hypothesis 2 is supported. H3a: The effect of SI on IU is mediated by SAT. Using Baron and Kenny (1986)s causal steps, all conditions were met thus all three sub-hypotheses were supported. The results were: H3a) SI significantly predicted IU (b=0.38, p < 0.01); H3b) SI significantly predicted SAT (b=0.54, p < 0.01); H3c) SAT and SI significantly predicted IU (b=0.35, p < 0.01 and b=0.20, p < 0.01). SI predictive utility fell from b=0.38 to b=0.20. R-square =0.20).

Both the partial effect of SAT on IU after controlling for SI and the direct effect of SI on IU are statistically significant. The indirect effect is.19 using the Sobel Test (Z=5.85, p < 0.01); the bootstrap estimate is that the indirect effect lies between .11 and 0.26, at 95% confidence level. The overall conclusion is that SAT mediates the relationship between SI and IU and hypothesis 3 is supported. H4: The effect of PE on IU is mediated by SAT. This hypothesis was not supported, as Perceived Effectiveness was not a construct of the measurement model of this research. 5.3.2 Exploratory multiple regression analysis
11B

Since support for H1 to H3 took into account the effect of one independent variable, multiple regression analysis was performed to explore whether SAT is the significant predictor of IU in the presence multiple independent variables IQ, SQ, PE, SI and to determine the strengths of effect of the independent variables on the dependent variable (identified as H5). The objective was to better understand the mechanism underlining adoption of the e-Government portal. H5: Together with SAT, the success factor (IQ, SQ, SI) is a good predictor of IU. The analysis (table 6), shows that SAT (b = .14, p <.05) and IQ (b=0.28, p < 0.01) significantly predict IU while the effects of SQ (b=0.10, p > 0.05) and SI (b=0.10, p > 0.05) became insignificant in the presence of three success factors and SAT. This suggests that the effects of SQ and SI on IU are a complete mediation (becoming zero after controlling for IU), whereas the effect of IQ is a partial mediation (its effect is reduced but different from zero) when the mediator is controlled (Kenny, 2006). IQ is deemed the best predictor due to its higher standardized regression coefficient. That is, when all three success factors were taken into consideration concurrently to predict e-Government portal adoption, only IQ and SAT were significant. The models explained 25.3% of the variance and the overall model was significant (F (4,455) =38.47, p<.001). Finally, the multiple regression assumptions of normality, linearity and homoscedasticity and independence of residuals were tested and the integrity of the assumptions was not questioned. Hypothesis 5 is partially supported as the exploratory regression analysis found that when all the variables were included in the regression SQ and SI became insignificant in predicting IU, whereas the effect of IQ was still positive, significant and larger than SAT. This is in line with Shrout & Bolger (2002)s claim that one possible explanation for partial mediation is that the independent variable has a specific and direct effect on the dependent variable, in addition to an indirect effect through the mediator. Users perception of the reliability of the information available in the portal plays a vital role in influencing IU. Overall, IQ, SQ and SI are deemed as significant predictors of SAT, itself a significant predictor of IU. Table 6: Multiple regressions using success factors to predict intention-to-use

www.ejise.com

43

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (35 46)

Unstandardized Coefficients B Std. Err. Constant SAT SQ 1.85 0.14 0.1 0.16 0.06 0.06

Std. Coefficients Beta

Sig.

Collinearity Statistics Tolerance VIF

Statistics

11.5 0.13 0.1 2.17 1.68

0 0.03 0.09 0.46 0.48 2.19 2.08 R Square =0.2527 Adjusted R Square =0.2462 F = 38.47 P < 0.01

SI IQ

0.1 0.28

0.05 0.06

0.09 0.27

1.75 4.36

0.08 0

0.63 0.43

1.6 2.3

Dependent Variable: IU

6. Conclusion and implications

5B

This research tested an integrated model of e-Government satisfaction that incorporated constructs from the technology acceptance model (TAM) and end-user satisfaction (EUS). The model was found to explain a large proportion (63.93%) of the variance in citizens IU. Empirical results provided support for the hypothesis that SAT partially mediates the relationship between success factors and IU. As in prior literature, IQ, SQ and SI (but not PE) were confirmed as determinants of the adoption of, and satisfaction with the e-Government portal. IQ is the most important element of SAT in Macao while SAT is a significant predictor of IU. This suggests that a good portal from a web designers eye might not be able to attract users adoption. Portal satisfaction is a requirement for intention-to-reuse the portal. In terms of theoretical implications, empirical testing of existing measures of ICT adoption and user satisfaction in an Asian e-Government context indicate that the determinants of website satisfaction vary and depend on the objectives set for the website. This supports the importance of portal design in terms of IQ and SQ, without overlooking the impact of SI as a direct and positive influencer of SAT. Therefore, portal management needs to ensure ease-of-use, currency and accuracy of the supplied information. The findings also indicate that the content an e-government portal that is perceived to be easier to manage is likely to indirectly facilitate citizens adoption intention and satisfaction. Timely information updating is a major concern for the e-Government portal in Macao. Given the importance of social influence, managerial actions should aim at improving e-Government portal acceptance by individual users and government employees. This might be achieved by combining the development of a support program directed at managing internal resistance to change with a program of integrated marketing communications focused on promoting awareness of and benefits from using the portal. In addition, it appears sensible to actively seek citizen input and feedback on how to improve the design of the e-Government portal. Estranged from the development process, potential users may lack indepth understanding of portal capabilities, with negative implications for portal adoption. In terms of limitations, while self-selection sampling captured online users, of Macaos e-Government portal, who are interested in responding to a web-survey, generalization to Macaos population is not warranted. Since the research promoted access to target respondents by engaging with four local organizations or community groups (including one higher education institution), the sample may reflect participation by institutional members. Another limitation is that IU is subjective to respondents statement of their intentions. No objective measures were used. Since data was collected for one only portal over a short period of time, different portals, and portals at different stages of development and different time periods may have yielded different responses.
6B

References

Au, N., Ngai, E. and Cheng, T. (2002) A critical review of end-user information system satisfaction research and a new research framework, Omega, 30, 6, 451-78. Baron, R. and Kenny, D. (1986) The moderator-mediator variable distinction is social psychological research: conceptual, strategic and statistical considerations, Journal of personality and social psychology, 51, 117382. Boyer, K., Olson, J., Calantone, R. and Jackson, E. (2002) Print versus electronic surveys: a comparison of two data collection methodologies, Journal of Operations Management, 20, 357-73. Carter, L. and Belanger, F. (2005) The utilization of e-government services: citizen trust, innovation and acceptance factors, Information Systems Journal, 15, 1, 5-25.

www.ejise.com

44

Academic Conferences Ltd

Cora Sio Kuan Lai and Guilherme Pires

Cheong, W. (2007) Macau Internet Usage Report, accessed Aug.20, 2007, http://www.rthk/org.hk/mediadigest/20070215_76_121332.html. China.com.cn (2006) High Internet Usage Rate Among the Young People in Macau, accessed Aug. 24, 2007, http://big5.china.com.cn/chinese/TCC/1101125.htm. Chu, P. Hsiao, N., Lee, F. and Chen, C. (2004) Exploring success factors for Taiwan's government electronic tendering system: behavioral perspectives from end users, Government Information Quarterly, 21, 2, 21934. Davis, F. (1989) Perceived usefulness, perceived ease of use, and user acceptance of information technology, MIS Quarterly, 13, 340-51. Davis, F., Bagozzi, R. and Warshaw, P. (1989) User Acceptance of Computer Technology: A Comparison of Two Theoretical Models, Management Science, 35, 8, 982. Davis, F. D. (1993) User Acceptance of Information Technology: System Characteristics, User Perceptions and Behavioral Impacts, Int. J. Man-Machine Studies, 38, 475-487 DeLone, W. and McLean, E. (1992) Information Systems Success: The quest for the dependent variable, Information Systems Research, 3, 1, 60-95. Doherty, N. and King, M. (2004) The Treatment of Organisational Issues in Systems Development Projects: The Implications for the Evaluation of Information Technology Investments, Electronic Journal of Information Systems Evaluation, 4, 1, 1. Doll, W., Raghunathan, T., Jeen-Su, L. and Gupta, Y. (1995) A Confirmatory Factor Analysis of the User Information Satisfaction Instrument, Information Systems Research, 6, 2, 177-88. Doll, W. and Torkzadeh, G. (1988) The measurement of end-user computing satisfaction, MIS Quarterly, 12, 2, 259-74. Furneaux, B. (2006) Theories used in IS research : Technology Acceptance Model, accessed Aug. 6, 2007, http://www.istheory.yorku.ca/. Gant, J., Gant, D. and Sprague, R. (2002) Web portal functionality and state government e-service, In Proceedings of the 35th Annual Hawaii International Conference on System Sciences, Los Alamitos, CA, 1627-36. Gefen, D., Warkentin, M., Pavlou, P. and Rose, G. (2002) EGovernment Adoption, In Proceedings of the 8th Americas Conference on Information Systems, 569-76. Grant, G. and Chau, D. (2004) Developing a Generic Framework for E-Government, Journal of Global Information Management, 13, 1, 1-30. Hair, J., Black, W., Babin, B., Anderson, R. and Tatham, R. (2006) Multivariate data analysis, Pearson, Upper Saddle River, NJ. Ho, A. (2002) Reinventing local governments and the e-government initiative, Public Administration Review, 62, 4, 434-44. Hu, P., Chau, P., Liu Sheng, O. and Tam, K. (1999) Examining the Technology Acceptance Model Using Physician Acceptance of Telemedicine Technology, Journal of Management Information Systems, 16, 2, 91-112. Huizingh, E. (2000) The content and design of web site: an empirical study, Information and Management, 27, 3, 123-34. Janowski, T., Estevez E., and Ojo A. (2004) eMacao Domain Analysis Concluding Workshop, Macao eGovernment Survey Summary and Findings, Public Administration Building Auditorium, Dec. 17. Janowski, T., Ojo A., & Estevez E. (2005) The State of Electronic Government in Macao, Volume 1: Survey, eMacao Task1 Report, United Nations University International Institute of Software Technology Kenny, D. (2006) Mediation, accessed Sep. 14, 2007, http://davidkenny.net/cm/mediate.htm. Lee, C. and Lei, U. (2007) Adoption of eGovernment Services in Macao, In Proceedings of the 1st International Conference on Theory and Practice of Electronic Governance, Macao. Malhotra, Y. and Galletta, D. (1999) Extending the technology acceptance model to account for social influence: theoretical bases and empirical validation, In 32nd HICSS, Hawaii, 1-14. McKinney, V., Yoon, K. and Zahedi, F. (2002) The measurement of Web-customer satisfaction: An expectation and disconfirmation approach, Information Systems Research, 13, 3, 296-315. Melone, N. (1990) A theoretical assessment of the user-satisfaction construct in information systems research, Management Science, 36, 1, 76-91. Moore, G. and Benbasat, I. (1991) Development of an Instrument to Measure the Perceptions of Adopting an Information Technology Innovation, Information Systems Research, 2, 3, 192-222. Nunnally, J. (1978) Psychometric Theory, McGraw-Hill, New York. Pikkarainen, K., Pikkarainen, T., Karjaluoto, H. and Pahnila, S. (2006) The measurement of end-user computing satisfaction of online banking services: empirical evidence from Finland, International Journal of Bank Marketing, 24, 3, Preacher, K. and Hayes, A. (2004) SPSS and SAS procedures for estimating indirect effects in simple mediation models Behavior Research Methods, Instruments, & Computers, 36, 4, 717-31. Rao, H., Chai, S., Herath, T. and Park, I. (2006) Repeated Use of E-Gov Web Sites: A Satisfaction and Confidentiality Perspective, International Journal of Electronic Government Research, 2, 3, 1-22. RTHK (2007) The current status of Internet in Macau (in Chinese), accessed Aug. 24, 2007, http://www.rthk.org.hk/mediadigest/20070215_76_121332.html. SAFP (2004) The Integration of Macau SAR government portal and InforMac completed, Department of Public Administration and Civil Service Bureau, assessed Aug. 4, 2006, available from the Government Intranet.

www.ejise.com

45

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (35 46)

Schepers, J. and Wetzels, M. (2007) A meta-analysis of the technology acceptance model: Investigating subjective norm and moderation effects, Information & Management, 44, 1, 90-103. Schaupp, L., Fan, W. and Belanger, F. (2006) Determining success for different website goals, In Proceedings of the 39th Hawaii International Conference on System Sciences (HICSS'06), Hawaii, 1-10. Seddon , P. (1997) A respecification and extension of the DeLone and McLean's model of IS success, Information Systems Research, 8, 3, 240-53. Shrout, P. and Bolger, N. (2002) Mediation in experimental and nonexperimental studies: new procedures and recommendations, Psychological methods, 7, 4, 422-45. Straub, D. W. (1989) Validating Instruments in MIS Research, MIS Quarterly, 13, 2, 147-69. Taylor, S. and Todd, P. A. (1995) Understanding Information Technology Usage: A Test of Competing Models, Information Systems Research, 6, 2, 144-76. Thomas, S. J. (2004) Using web and paper questionnaires for data-based decision making - from design to interpretation of the results, Corwin Press, Thousand Oaks, CA. Thompson, R. and Higgins, C. (1991) Personal computing: Toward a conceptual model of utilization, MIS Quarterly, 15, 1, 125. Ticehurst, G. and Veal, A. (2000) Business Research Methods - a managerial approach, Pearson Education, Frenchs Forest, New South Wales. UNPA and ASPA (2001) Benchmarking e-Government: A Global Perspective, accessed Jun. 24, 2007, http://unpan1.un.org/intradoc/groups/public/documents/un/unpan003984.pdf Venkatesh, V., Morris, G. and Davis, F. D. (2003) User acceptance of information technology: towards a unified view, MIS Quarterly, 27, 3, 425-78. Wang, Y., Tang, T. and Tang, J. (2001) An instrument for measuring customer satisfaction toward we sites that market digital products and service, Journal of Electronic Commerce Research, 2, 3, 89-102. Wangpipatwong, S., Chutimaskul, W. and Papasratorn, B. (2005) Factors Influencing the Adoption of Thai eGovernment Websites: Information Quality and System Quality Approach, In Proceedings of the Fourth International Conference on eBusiness, Bangkok, Wixom, B. and Todd, P. (2005) A theoretical integration of user satisfaction and technology acceptance, Information Systems Research, 16, 1, 85-102. World Bank (2002) The e-Government Handbook for developing countries, Centre for Democracy & Technology, World Bank, assessed Aug. 2,2006, www.infodev.org .
H H

www.ejise.com

46

Academic Conferences Ltd

Firms Patterns of e-Business Adoption: Evidence for the European Union-27

Tiago Oliveira and Maria Fraga Martins ISEGI, Universidade Nova de Lisboa, Lisbon, Portugal
toliveira@isegi.unl.pt mrfom@isegi.unl.pt
Abstract: Research has shown that firms using e-business achieve considerable returns through efficiency improvements, inventory reduction, sales increase, customer relationship enhancement, new market penetration, and ultimately financial returns. However, there is little systematic research in terms of e-business adoption patterns in firms across countries and industries. This study addresses the research gap by analysing the pattern of e-business adoption by firms across European Union (EU) members. For that, we used the survey data from 6,964 businesses in EU27 members (excluding Malta and Bulgaria). The choice of variables that we will use in our study is based on the technology-organization-environment (TOE) theory. In the TOE framework, three aspects may possibly influence e-business adoption: technological context (technology readiness and technology integration), organizational context (firm size, expected benefits and barriers of e-business and improved products or services or internal processes) and environmental context (internet penetration and competitive pressure). We performed a factor analysis (FA) of multi-item indicators to evaluate the validity and to reduce the number of variables. We used the principal component technique with varimax rotation to extract four eigenvalue, which were all greater than one. The first four factors explain 72.4% of variance contained in the data. The four factors found are: expected benefits and obstacles of e-business, internet penetration, technology readiness and technology integration. These factors are in accordance with the literature review. Afterwards, we performed a cluster analysis (CA) using variables obtained from the FA and the other variables were gathered directly (firm size, employees education, improved products or services or internal processes and competitive pressure) from the e-Business W@tch survey. In the CA we used hierarchical and non hierarchical methods. We obtained four distinct groups of e-business adoption. The pattern of these groups suggested that in the European context the most important factor to characterize e-business adoption is the specific characteristics of the industry and is not the country to which the firms belong. Keywords: e-business adoption, information and communication technology (ICT), technology-organizationalenvironment (TOE) framework, cluster analysis (CA), European Union (EU) members

1. Introduction
The development of e-business capability is crucial since it is swiftly chaining the way that companies buy, sell, and deal with customers, becoming a more integral part of its business strategies (AbuMusa 2004). E-business adoption becomes a significant research topic because it enables the firm to execute electronic transactions along value chain activities (Straub and Watson 2001, Zhu and Kraemer 2002). It represents a new way to integrate Internet-based technologies with core business potentially affecting the whole business (Zhu 2004a). The European Commission (2005) claims that more efforts are needed to improve e-business in European firms if the Lisbon targets of competitiveness are to be accomplished. European firms, under the pressure of their main international competitors, need to find new opportunities to reduce costs and improve performance. For this reason, it is fundamental to identify the patterns of ebusiness adoption among firms in European Union (EU) members. To the best of our knowledge, very limited empirical research has been performed to identify the patterns of e-business adoption among firms in EU27 using the technological, organizational and environmental (TOE) contexts. This study fills this gap. The main objectives of this study are the following: To identify distinct clusters of e-business adoption; To characterize the pattern of e-business adoption by firms across these clusters; To understand the extent to which industry e-business adoption characteristics are more or less relevant than country specific characteristics.

ISSN 1566-6379 47 Academic Conferences Ltd Reference this paper as: Oliveira, T, and Martins, M, F. (2010) Firms Patterns of e-Business Adoption: Evidence for the European Union27 The Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010, (pp47 - 56), available online at www.ejise.com

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (47 56)

The paper is organized as follows. The next section presents the literature review of the factors affecting e-business adoption. After we describe the data used. Then, we define the methodology used and present the results obtained. Finally, we present conclusions and future research.

2. e-Business adoption by firms: literature review

Several authors (Hong and Zhu 2006, Iacovou et al. 1995, Kuan and Chau 2001, Lin and Lin 2008, Oliveira and Martins 2008, Pan and Jang 2008, Premkumar and Ramamurthy 1995, Thong 1999, Zhu et al. 2003, Zhu et al. 2006) used the TOE framework, developed by Tornatsky and Fleisher (1990), to analyse IT adoption by firms. Based on the TOE framework, we stipulate that three aspects may possibly influence e-business adoption: technological context (technology readiness and technology integration); organizational context (firm size, expected benefits and obstacles of e-business and improved products or services or internal processes); and environmental context (internet penetration and competitive pressure). Technological context Technology readiness can be defined as technology infrastructure and IT human resources. Technology readiness is reflected not only by physical assets, but also by human resources that are complementary to physical assets (Mata et al. 1995). Technology infrastructure establishes a platform on which internet technologies can be built; IT human resources provide the knowledge and skills to develop web applications (Zhu and Kraemer 2005). Theoretical assertions are supported by several empirical studies (Armstrong and Sambamurthy 1999, Hong and Zhu 2006, Iacovou et al. 1995, Kwon and Zmud 1987, Zhu 2004b, Zhu et al. 2003, Zhu and Kraemer 2005, Zhu et al. 2006, Pan and Jang 2008). Evidence from the literature suggests that technology integration helps improve firm performance by reduced cycle time, improved customer service, and lowered procurement costs (Barua et al. 2004). E-business demands close coordination of various components along the value chain. Correspondingly, a greater integration of existing applications and the internet platform represent a greater capacity of conducting business over the internet (Al-Qirim 2007, Mirchandani and Motwani 2001, Premkumar 2003, Zhu et al. 2006). Organizational context Firm size is one of the most commonly studied determinants of IT adoption (Lee and Xia 2006). Large firms are more likely to undertake innovation. Three major arguments support the positive role of firm size in determining IT adoption: appropriability (the benefits of the new IT), the greater availability of funds and the quicker capture of economies of scale. However, larger firms have multiple levels of bureaucracy and this can impede decision-making processes about new ideas and projects. Moreover, IT adoption often requires close collaboration and coordination that can be easily achieved in small firms. Empirical studies consistently found that perceived benefits have a significant impact in IT adoption (Beatty et al. 2001, Gibbs and Kraemer 2004, Iacovou et al. 1995, Kuan and Chau 2001, Lin and Lin 2008). Perceived obstacles are particularly relevant because the adoption process may be complicated and costly (Pan and Jang 2008, Zhu et al. 2006). Improved products or services or internal process that are enabled by or related to a subset of IT, namely e-business technologies (Koellinger 2008). Environmental context Internet penetration measures the adoption and diffusion of computer and internet of individual and household in the population of each country. It is a important factor for decision makers of e-business adoption because it reflects the potential market (Zhu et al. 2003). Competitive pressure refers to the degree of pressure felt by the firm from competitors within the industry. Porter and Millar (1985) analyzed the strategic rationale underlying competitive pressure as an innovation-diffusion driver. They suggested that, by using a new innovation, firms might be able to alter the rules of competition, affect the industry structure, and leverage new ways to outperform
www.ejise.com

48

Academic Conferences Ltd

Tiago Oliveira and Maria Fraga Martins

rivals, thus changing the competitive landscape. This analysis can be extended to IT adoption. Empirical evidence suggests that competitive pressure is a powerful driver of IT adoption and diffusion (Al-Qirim 2007, Battisti et al. 2007, Dholakia and Kshetri 2004, Gibbs and Kraemer 2004, Grandon and Pearson 2004, Hollenstein 2004, Iacovou et al. 1995, Mehrtens et al. 2001, Zhu et al. 2003).

3. Data
Our data source is the e-Business W@tch (w@tch 2006a, w@tch 2006b), which collects data concerning the use of ICT and e-business in European enterprises. The data, collected by means of representative surveys of firms that used computers, are related to EU27 members and had a scope of 12,439 telephone interviews with decision-makers in enterprises. Interviews were carried out in March and April 2006, using computer-aided telephone interview (CATI) technology. The sample drawn was a random sample of companies from the respective sector population in each of the countries. According to the methodological recommendations of Eurostat, the situation of an operator that did not answer or does not know the answer to a specific question should not imply its imputation, in any case, based on the answer of the other operators. Consequently, we obtained a smaller sample that we compared, by a proportion test, with the original one. The proportion test for the variable ebusiness adoption reveals that the only country where statistically significant differences exist is Bulgaria. For this reason we excluded it from our analysis. We also used, as additional information, the Eurostat data (Survey on ICT Usage in Households and by Individuals 2006) to compute the internet penetration index by country. We excluded Malta because no data were available. The final sample includes 6,694 firms belonging to the EU27 members excluding Malta and Bulgaria. About 80 percent (79.0%) of the data was collected from owners, managing directors, heads of IT and other senior members of IT, suggesting the high quality of the data source.

4. Methodology and results

As a first step, we group the items to reduce the number of variables of the survey; for that we applied a factor analysis (FA). Then, to determine homogenous groups of firms in terms of e-business adoption, we applied cluster analysis (CA).

4.1 Factor analysis results

We performed a FA of multi-item indicators to reduce the number of variables of the survey and to evaluate the validity. We used the principal component technique with varimax rotation (see Table 1) to extract four eigen-value, which were all greater than one. The first 4 factors explain 72.4% of variance contained in the data. The Kaiser-Meyer-Olkin (KMO) measures the adequacy of sample; KMO general is 0.91, which reveals that the matrix of correlation is appropriate for the FA. The KMO for individual variables is also adequate. All the factors have a loading greater than 0.50 (except TI4). This indicates that our analysis is based on a well-explained factor structure. The four factors found are: expected benefits and obstacles of e-business, internet penetration, technology readiness and technology integration. These factors are in accordance with the literature review. Table 1: Factor and validity analysis
Items measured Expected benefits and obstacles of e-business (EBOEB) Why did your company decided to engage in e-business activities? (0-not at all; 1not important; 2-important) EB1 - Because your customers expected it from you EB2 - Because your company believes that e-business will help to get an edge over your competitors EB3 - Because your competitors also engage in e-business EB4 - Because your suppliers expected it from you Important obstacles for not practising e-business in your company? (0-not at all; 1-not important; 2-important) EO1 - My company is too small to benefit from any e-business activities 1 2 Factor 3 4

0.91 0.90 0.88 0.87

0.01 0.04 0.02 0.03

0.11 0.13 0.08 0.10

0.06 0.05 0.04 0.05

-0.11

www.ejise.com

49

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (47 56)

0.93 Items measured EO2 - E-business technologies are too expensive to implement EO3 - Our systems are not compatible with those of suppliers or customers EO4 - We are concerned about potential security risks and privacy issues EO5 - We think that there are important unsolved legal issues involved EO6 - It is difficult to find reliable IT suppliers Internet penetration (IP) IP1 - Individuals using computer in the European Union IP2 - Individuals using Internet in the European Union IP3 - Households with Internet access at home IP4 - Households with computer (International Benchmarking) IP5 - Households using a broadband connection to the Internet (International Benchmarking) IP6 - Individuals using Internet commerce in the European Union Technology readiness (TR) TR1 - Sum of the following network applications: a Local Area Network (LAN); Wireless LAN; Voice-over-IP; Fixed line connections; Wireless-Local-AreaNetworks or W-LANs, Mobile communication networks; Virtual Private Network (VPN) TR2 - Sum of the following technologies: Internet; intranet; web site; TR3 - Sum of the following questions ICT skills: your company currently employ ICT practitioners; your company regularly send employees to ICT training programmes TR4 - Sum of the following security applications: secure server technology, for example SSL, TLS or a comparable technical standard; digital signature or public key infrastructure; a firewall TR5 - Sum of the following online applications other than e-mail: to share documents between colleagues or to perform collaborative work in an online environment; to track working hours or production time; to collaborate with business partners to forecast product or service demand; to collaborate with business partners in the design of new products or services; to manage capacity or inventories; to send e-invoices to customers in the public sector; to send einvoices to customers in the private sector; to receive e-invoices from suppliers. TR6 - Percentage of employees that have access to the internet Technology Integration (TI) Does your company use any of the following systems or applications for managing information in the company (0- do not know what this is; 1-no; 2-yes)? TI1 - a SCM system, that is a Supply Chain Management System TI2 - an EDM system, that is an Enterprise Document Management System TI3 - an ERP system, that is Enterprise Resource Planning System TI4 - Knowledge Management software Eigen value Percentage of variance explained Note: variables are marked according to factor loading 0.08 0.08 0.06 0.06 8.72 33.5 5 0.02 0.01 0.03 0.02 0.05 0.04 1 0.94 0.94 0.94 0.95 0.95

0.02 Factor 2 3 -0.08 0.03 -0.06 0.03 -0.05 0.03 -0.05 0.04 -0.06 0.03 0.98 0.98 0.97 0.97 0.92 0.92 0.05 0.05 0.03 0.01 0.01 0.06

0.04 4 0.02 0.01 0.01 0.01 0.01 0.05 0.05 0.03 0.01 0.00 0.00

0.18 0.24 0.14

0.08 0.08 0.01

0.75 0.68 0.66

0.16 0.08 0.18

0.22

0.11

0.66

0.08

0.23

0.05

0.55

0.32

0.12

0.17

0.55

0.25

0.07 0.15 0.06 0.03 5.57 21.4 3

0.06 0.11 0.27 0.21 2.73 10.5 0

0.68 0.65 0.65 0.49 1.80 6.94

www.ejise.com

50

Academic Conferences Ltd

Tiago Oliveira and Maria Fraga Martins

4.2 Cluster analysis results

To perform the CA we used the variables presented in Table 2 obtained from the FA (TR, TI, EBOEB and IP) and also some variables computed directly (SIZE, IPSIP and CP) from the e-Business W@tch survey. Table 2: Description of variables used in CA
Technological context Technology readiness (TR) Technology integration (TI) Variables Organizational context Firm size (SIZE) Expected benefits and obstacles of ebusiness (EBOEB) Improved products or services or internal processes (IPSIP) Environmental context Internet penetration (IP) Competitive pressure (CP) AF index of technology readiness AF index of technology integration Description The logarithm number of employees AF index of expected benefits and barriers of e-business Binary =1 if firms improved products or services or internal processes AF index of internet penetration Binary =1 if firms think that ICT has an influence on competition in their industry

The objective of the CA is to classify firms in homogenous groups, so that firms from the same group are as similar as possible in what concerns the pattern of e-business adoption, and as different as possible from firms belonging to other groups. The variables used to perform the CA have measurement scales that are both quantitative and qualitative, and so, it was necessary to use the dissimilarity matrix calculated through the method proposed by Gower (Gower 1971). For that, we used the SAS software and the distance macros (SAS 2003). Once the dissimilarity matrix was computed, we performed, as usual, a hierarchical CA, through the most known methods: Ward, median, centroid, complete linkage and single linkage. From the results obtained from these five methods, it was possible to determine the optimal number of groups (four), as well as the method that best fits these data (Ward), (see annex). The centroids of the clusters obtained through Wards method were used as initial seeds for the non hierarchical model (k-means), which allowed us to refine the previous solution. According to Sharma (1996), this is the best solution to obtain clusters. To characterise the groups we used as auxiliary variables the firm size (by classes), industries and countries (Table 3). Table 3: Description of adoptions and auxiliary variables
Variables Adoption E-business (EB) e-purchasing adoption (e1) e-selling adoption (f4) Auxiliary Size by classes (micro, small, medium and large) Industry (manufacture, construction, tourism and telecommunications) EU27 members (excluding Malta and Bulgaria) Description Binary =1 if firm adopts e-purchasing or e-selling Binary =1 if firm adopts e-purchasing Binary =1 if firm adopts e-selling Four binary variable for each size Four binary variables for each industry Twenty-five binary variables for each country

Summary statistics for variables in each cluster are provided in Table 4. Clusters patterns were compared using chi-squared tests for binary variables and Kruskal-Wallis test for quantitative variables. All variables, except auxiliary variables related to countries, present statistically significant differences across clusters, suggesting that our cluster analysis generated groups of firms that are statistically distinct according to the variables characterising e-business adoption. Table 4: Summary statistics for CA
All Number of firms Percentage of firms TOE Variables Technology readiness (TR) www.ejise.com 6,964 100.0% 1 1,699 24.4% Cluster 2 3 1,150 1,509 16.5% 21.7% 4 2,606 37.4% Statistic Test (p-value)

0.000

-0.482

-0.041

-0.215

0.457

991.69 (<0.001) ISSN 1566-6379

51

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (47 56)

Technology integration (TI)

0.000 All

-0.140 1 1,699 24.4%

Number of firms Percentage of firms Adoption variables Firm size (SIZE) Expected benefits and barriers of ebusiness (EBOEB) Improved products or services or internal processes (IPSIP) Internet penetration (IP) Competitive pressure (CP) e-purchasing adoption e-selling adoption E-business Auxiliary variables Micro Small Medium Large Manufacture

6,964 100.0%

0.090 -0.155 Cluster 2 3 1,150 1,509 16.5% 21.7%

0.141 4 2,606 37.4%

83.76 (<0.001) Statistic Test (p-value)

2.733 0.000 0.539 0.000 0.591 0.619 0.310 0.693 0.423 0.306 0.211 0.060 0.512

2.406 -0.480 0.000 0.071 0.000 0.448 0.169 0.516 0.489 0.312 0.167 0.033 0.504

2.948 -0.205 1.000 0.017 0.000 0.602 0.250 0.666 0.357 0.339 0.241 0.063 0.647

2.473 0.141 0.000 0.002 1.000 0.606 0.307 0.693 0.493 0.279 0.180 0.048 0.427

3.003 0.322 1.000 -0.055 1.000 0.746 0.430 0.820 0.368 0.302 0.245 0.084 0.507

183.69 (<0.001) 710.83 (<0.001) 3207.97 (<0.001) 52.43 (<0.001) 2848.98 (<0.001) 149.59 (<0.001) 242.62 (<0.001) 139.10 (<0.001)

64.84 (<0.001) 8.04 (0.045) 42.24 (<0.001) 49.06 (<0.001) 62.33 (<0.001) 159.74 0.169 0.265 Construction 0.127 0.192 0.112 (<0.001) 0.181 0.233 31.48 (<0.001) Tourism 0.166 0.149 0.175 185.11 0.138 0.206 (<0.001) Telecommunications 0.065 0.077 0.148 Note: We also analyze summary statistics by cluster for each country but we do not identify any pattern. In addition, several countries (Netherlands, Sweden, Germany, Ireland, Luxembourg, Italy, Slovakia, Cyprus and Latvia) did not have statistically differences by clusters.

A description of each of the clusters, drawn from Table 4 is given below: Cluster 1 (lowest e-business adoption).These firms had the lowest level of Technology readiness compared to the other three clusters; their expected benefits of e-business are the lowest, but they had the highest level of internet penetration index. They are therefore referred to here as lowest ebusiness adoption group, that is, firms who are at the very start of their e-business adoption but are currently operating in countries with a high internet penetration. The competitive pressure they faced is low. Most of the firms within this group are micro and small firms without improved products or services and the most common activity sector is the construction. This cluster contains 24.4% of the whole sample firms. Cluster 2 (medium e-business adoption with technology integration). The firms in this cluster were making some use of integrated technologies and most of them are improving their products or internal processes. They were presenting a medium level of Technology readiness but a low competitive pressure. Firms in cluster 2 are small and medium size firms coming from manufacturing industry and having a medium level of e-business adoption. This cluster includes 16.5% of the firms. Cluster 3 (medium e-business adoption with high competitive pressure). Firms in this cluster had the lowest index of technology integration and a low level of technology readiness. Contrarily to firms in cluster 2, most of them dont improve their services or internal processes, but all of them are facing competitive pressure. In this group the most common firms are micro firms coming from the tourism industry. This cluster contains 21.7% of firms. Cluster 4 (highest e-business adoption). Firms in cluster 4 were found to have high levels of all variables, except for the internet penetration. Cluster 4 incorporates firms that operate in the telecommunications industry and had medium or large size. This is the biggest cluster with 37.4% of firms.
www.ejise.com

52

Academic Conferences Ltd

Tiago Oliveira and Maria Fraga Martins

The findings suggest that the four clusters identified represent a set of e-business patterns (Figure 1) that are much more related to industry sector and firm size than to countries. Micro and small firms from the Construction Sector can be viewed as laggards and big firms from Telecommunications as the Pioneers. Moreover, our results also suggested a positive relationship between e-purchase and eselling adoption.

- Medium e-business adoption - Low TR - Lowest TI - Medium IP - High EBOEB - All firms have CP - Without IPSIP - Micro size firms - Tourism - Low e-business adoption - Medium TR and TI, - Low EBOEB - All firms have IPSIP - Lowest e-business adoption - Without CP - Lowest TR and EBOEB - High IP - Low TI - Small and medium size firms - Without IPSIP and CP - Manufacture - Highest IP - Micro and small size firms - Construction

- Highest e-business adoption - Highest TR, TI and EBOEB - All firms have IPSIP and CP - Lowest IP - Medium and large size firms - Telecommunications

Figure 1: Characteristics of four e-business groups

5. Conclusions and future research

This study sought to explore the patterns of e-business adoption by European firms. Four distinct clusters of e-business adoption were found. The major conclusions are the following. Firstly, in general, firms with high levels of TOE factors have also enhanced levels of e-business (Figure 2, 3, 4 and 5). Secondly, the two clusters (cluster 3 and 4) that have the highest level of e-business adoption incorporate firms all of them with the higher level of competitive pressure (CP). This reveals the importance of environmental factors to improve e-business adoption. Thirdly, the comparison of ebusiness patterns, between cluster 3 and cluster 2 (figures 1, 2 and 3), suggest that the technology context is more important for the manufacture industry than for the tourism industry. Finally, the internet penetration index, which is the specific variable for each country, has a different behaviour from e-business adoption. This index does not follow the trends of e-business adoption as can be seen in Figure 6. Additionally, the variables related to countries do not have statistically significant differences across clusters. These reveal that in the European context the most important to characterise e-business adoption is the industry and their specific characteristics and not the country to which the firms belong.

Figure 2: Technology readiness

Figure 3: Technology

Figure 4: Size versus e-

www.ejise.com

53

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (47 56)

index versus e-business adoption

integration index versus ebusiness adoption

business adoption

Figure 5: Expected benefits and obstacles of e-business versus ebusiness adoption

Figure 6: Internet penetration index versus e-business adoption

In terms of future research, it would be interesting to study one model that determines e-business adoption for each industry in the European context. It would be also important to compare the impacts of TOE variables in different industries (manufacture, construction, tourism and telecommunications).

6. Annex
Ward's methods

Firms

Figure 7: R of different methods

Figure 8: Dendrogram of Wards methods

Acknowledgements
We would like to acknowledge the e-Business W@tch survey 2006 for providing us with the data.

References
Abu-musa, A.A. (2004) "Auditing e-business: new challenges for external auditors", Journal of American Academy of Business, Vol 4, No. 1, pp 2841. Al-qirim, N. (2007) "The adoption of eCommerce communications and applications technologies in small businesses in New Zealand", Electron. Commer. Rec. Appl., Vol 6, No. 4, pp 462-473. Armstrong, C.P. and Sambamurthy, V. (1999) "Information technology assimilation in firms: The influence of senior leadership and IT infrastructures", Information Systems Research, Vol 10, No. 4, pp 304-327. Barua, A., Konana, P., Whinston, A.B. and Yin, F. (2004) "Assessing internet enabled business value: An exploratory investigation", MIS Quart, Vol 28, No. 4, pp 585-620. Battisti, G., Hollenstein, H., Stoneman, P. and Woerter, M. (2007) "Inter And Intra Firm Diffusion Of Ict In The United Kingdom (Uk) And Switzerland (Ch) An Internationally Comparative Study Based On Firm-Level Data", Economics of Innovation and New Technology, Vol 16, No. 8, pp 669 - 687. Beatty, R.C., Shim, J.P. and JONES, M.C. (2001) "Factors influencing corporate web site adoption: a time-based assessment", Information & Management, Vol 38, No. 6, pp 337-354. Dholakia, R.R. and Kshetri, N. (2004) "Factors Impacting the Adoption of the Internet among SMEs", Small Business Economics, Vol. 23, pp 311-322. Europeans Commission (2005) "Information Society Benchmarking Report", [online], http://ec.europa.eu/information_society/eeurope/i2010/docs/benchmarking/051222%20Final%20Benchmark ing%20Report.pdf. Gibbs, L.J. and Kraemer, K.L. (2004) "A Cross-Country Investigation of the Determinants of Scope of Ecommerce Use: An Institutional Approach", Electronic Markets, Vol. 14, No. 2, pp 124-137.

www.ejise.com

54

Academic Conferences Ltd

Tiago Oliveira and Maria Fraga Martins

Gower, J.C. (1971) "A general coefficient of similarity and some of its properties", Biometrics, Vol. 27, pp 857872. Grandon, E.E. and Pearson, J.M. (2004) "Electronic commerce adoption: an empirical study of small and medium US businesses", Information & Management, Vol. 42, No. 1, pp 197-216. Hollenstein, H. (2004) "Determinants of the adoption of Information and Communication Technologies (ICT). An empirical analysis based on firm-level data for the Swiss business sector", Structural Change and Economic Dynamics, Vol. 15, pp 315-342. Hong, W.Y. and Zhu, K. (2006) "Migrating to internet-based e-commerce: Factors affecting e-commerce adoption and migration at the firm level", Information & Management, Vol. 43, No. 2, pp 204-221. Iacovou, C.L., Benbasat, I. and Dexter, A.S. (1995) "Electronic data interchange and small organizations: Adoption and impact of technology", Mis Quarterly, Vol. 19, No. 4, pp 465-485. Koellinger, P. (2008) "The relationship between technology, innovation, and firm performance - Empirical evidence from e-business in Europe", Research Policy, Vol. 37, No. 8, pp 1317-1328. Kuan, K.K.Y. and Chau, P.Y.K. (2001) "A perception-based model for EDI adoption in small businesses using a technology-organization-environment framework". Information & Management, Vol. 38, No. 8, pp 507-521. Kwon, T.H. and Zmud, R.W. (1987) Unifying the fragmented models of information systems implementation. In critical issues in Information Systems Research (Boland RJ and Hirschheim RA, Eds). IN WILEY, J. (Ed.) New York. Lee, G. and Xia, W.D. (2006) "Organizational size and IT innovation adoption: A meta-analysis", Information & Management, Vol. 43, No. 8, pp 975-985. Lin, H.F. and Lin, S.M. (2008) "Determinants of e-business diffusion: A test of the technology diffusion perspective", Technovation, Vol. 28, No. 3, pp 135-145. Mata, F., Fuerst, W. and Barney, J. (1995) "Information technology and sustained competitive advantage: A resource-based analysis", MIS Quart, Vol. 19, No. 4, pp 487505. Mehrtens, J., Cragg, P.B. and Mills, A.M. (2001) "A model of Internet adoption by SMEs", INFORMATION & MANAGEMENT, Vol. 39, pp 165-176. Mirchandani, D.A. and Motwani, J. (2001) "Understanding small business electronic commerce adoption: An empirical analysis", Journal of Computer Information Systems, Vol. 41, No. 3, pp 70-73. Oliveira, T. and Martins, M.F.O. (2008) "A comparison of web site adoption in small and large Portuguese firms", Ice-B 2008: Proceedings of the International Conference on E-Business, pp. 370-377, Oporto, Portugal, July. Pan, M.J. and Jang, W.Y. (2008) "Determinants of the adoption of enterprise resource planning within the technology-organization-environment framework: Taiwan's communications", Journal of Computer Information Systems, Vol. 48, No. 3, pp 94-102. Porter, M.E. and Millar, V.E. (1985) "How Information Gives You Competitive Advantage", Harvard Business Review, Vol. 63, No. 4, pp 149-160. Premkumar, G. (2003) "A meta-analysis of research on information technology implementation in small business", Journal of Organizational Computing and Electronic Commerce, Vol. 13, No. 2, pp 91-121. Premkumar, G. and Ramamurthy, K. (1995) "The role of interorganizational and organizational factors on the decision mode for adoption of interorganizational systems", Decision Sciences, Vol. 26, No. 3, pp 303-336. SAS (2003) The Distance Macro. 8.2 ed., SAS Institute Inc. Sharma, S. (1996) Applied Multivariate Techniques, John Wiley & Sons, New York. Straub, D.W. and Watson, R.T. (2001) "Research commentary: Transformational issues in researching IS and net-enabled organizations", Information Systems Research, Vol. 12, No. 4, pp 337-345. Thong, J.Y.L. (1999) "An Integrated Model of Information Systems Adoption in Small Businesses", Journal of Management Information Systems, Vol. 15, No. 4, pp 187-214. Tornatsky, L. and Fleischer, M. (1990) The Process of Technology Innovation, Lexington, MA, Lexington Books. e-Business W@tch (2006a) "e-Business Decision Maker Survey in European enterprises", [online], http://www.ebusiness-watch.org/about/documents/eBiz_Questionnaire_2006.xls. e-Business W@tch (2006b) "The e-Business Survey 2006 Methodology Report", [online], http://www.ebusiness-watch.org/about/documents/DMS2006_Methodology.pdf. Zhu, K. (2004a) "The complementarity of information technology infrastructure and e-commerce capability: A resource-based assessment of their business value", Journal of Management Information Systems, Vol. 21, No. 1, pp 167-202. Zhu, K. (2004b) "Information transparency of business-to-business electronic markets: A game-theoretic analysis", Management Science, Vol. 50, No. 5, pp 670-685. Zhu, K., Kraemer, K. and Xu, S. (2003) "Electronic business adoption by European firms: a cross-country assessment of the facilitators and inhibitors", European Journal of Information Systems, Vol. 12, No. 4, pp 251-268. Zhu, K. and Kraemer, K.L. (2002) "e-Commerce metrics for net-enhanced organizations: Assessing the value of e-commerce to firm performance in the manufacturing sector", Information Systems Research, Vol. 13, No. 3, pp 275-295. Zhu, K. and Kraemer, K.L. (2005) "Post-adoption variations in usage and value of e-business by organizations: Cross-country evidence from the retail industry", Information Systems Research, Vol. 16, No. 1, pp 61-84. Zhu, K., Kraemer, K.L. and Xu, S. (2006) "The process of innovation assimilation by firms in different countries: A technology diffusion perspective on e-business", Management Science, Vol. 52, No. 10, pp 1557-1576.

www.ejise.com

55

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (47 56)

www.ejise.com

56

Academic Conferences Ltd

Creating Strategic Value through Executive Information Systems: an Exploratory Study

Elmarie Papageorgiou1 and Herman de Bruyn2 1 University of the Witwatersrand, Johannesburg, South Africa 2 University of Johannesburg, South Africa
Elmarie.Papageorgiou@wits.ac.za hdebruyn@uj.ac.za
Abstract: Over the past few years, information technology has grown so rapidly that businesses had to adjust very quickly to keep abreast of fast growing technologies and international trends. An increasing number of South African companies listed on the Johannesburg Stock Exchange (JSE) have implemented Executive Information Systems (EISs) that have resulted in the widespread use of computers in companies. Users of EISs need systems that provide them with access to diverse types of information in order to take decisions, to solve problems and to compete with competitors. This article discusses whether South African companies create strategic value through the use of EISs. The strategic value of the business is explained as an advantage to improve businesses performances firstly, by gains in profitability and financial strength and secondly, gains in the businesses competitive strength and market standing. Executives and top management need to be aware of the opportunities available to them by using information technology as a business tool to analyse their businesses performance and competitiveness. This study is an exploratory study and the research method is quantitative of nature. A structured questionnaire was designed and was sent to 334 listed JSE companies in order to investigate the existence of an EIS, the gathering, selecting and use of information in companies in order to make decisions and to solve problems. Many businesses have chosen EIS technology to provide relevant and accurate information to top management and executives. Currently the EIS provides information that is only available to executives and top management, but the need exists to expand EISs to other users in the business. The research questions investigated in this study are to establish what EISs offer to fulfil the needs of users and to determine the impact on creating strategic value within the business in order to keep pace with on-going changes in technology. In addressing these problems the existence of EISs was investigated to debate, express, and understand the role and use of an EIS and resulted in creating strategic value for businesses. The value of the study explains the vital importance of executives influence towards the adoption, commitment and use of EISs at strategic management levels, creating and adding strategic value in companies. The findings of the study add to the current understanding and awareness of EISs in listed JSE companies and therefore create an environment in which the business can enhance sustainability and strategic competitiveness. Responses were favourable since respondents requested a report on the outcome of the results as they expressed an interest in the underlying motivation of the study and how their company compares with their competitors in the industry. Conclusions drawn from the results are that EISs need to incorporate all the unmet needs of users in order for EISs to add strategic value and to be used as effective business tools in companies. Keywords: competitive advantage, executive information system, information technology, performance, strategic value, sustainability

1. Introduction and background

The business environment exposes top management and executives to so much data that data needs to be converted into useful information that is organised and summarised. Information definitely establishes a competitive advantage and enables business change by creating a strategic value through an Executive Information System (EIS). Businesses are using EISs as a business tool on a daily basis that enable top management and executives to extract useful information for decisionmaking and problem solving. In todays business environment, information is also required and utilised on all levels of the business. The driving force behind EISs is the strategic value, as information software technology improves in a digital environment, top management and executives must utilise the new technology available for strategic decision making and managing daily business activities in order to remain competitive. Sawy (1989:57) states that practitioners and academics spent the last ten years bringing out the characteristics that distinguish the varieties of information systems, which often resulted in a hairsplitting exercise. Therefore some references used in this exploratory study are older than ten years as these consulted references include all available references to investigate the study to its full extent to incorporate the history and literature of EISs and also to compare the thoughts of older studies to the digital era that are still valid and trustworthy.
ISSN 1566-6379 57 Academic Conferences Ltd Reference this paper as: Papageorgiou, E, and de Bruyn, H. (2010) Creating Strategic Value through Executive Information Systems: an Exploratory Study The Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010, (pp57 - 76), available online at www.ejise.com

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (57 76)

2. Research questions and research objectives

The research questions are to establish what EISs offer to fulfil the needs of users and to determine the impact on creating strategic value within the business in order to keep pace with on-going changes in technology. The objectives of this study are twofold. The first objective is to determine the use, purpose and strategic value of EISs in listed JSE companies to investigate the purpose and criteria for success of EIS and believing that investing in EISs will benefit the company. The second objective is to explore to what extent strategic value can be created through an EIS system to enhance sustainability and strategic competitiveness.

3. Literature study
A literature study investigates the definition, capabilities, problems, purpose of an EIS and discusses the strategic value as a driving force behind EISs. Furthermore the intention of the study is to unlock the strategic value in EISs in order to enhance business performance and decision-making. Derived from the literature study pertains as to whether top management and executives are prepared to be involved in information technology or whether they are unintentionally forced into the digital environment.

3.1 Defining EISs

Twenty different studies have been investigated regarding the definition of EISs. The definitions were documented and compared in the original research and for the purpose of this study the following definition of EIS is used: An EIS is a computerised system that provides executives, top management and other senior managers access to internal and external information. Information that is relevant, accurate, timely and up-to-date in order to make decisions, solve problems, determine critical success factors and satisfy information needs. These interest groups are primarily interested in summarised data that has been transformed into meaningful information, using graphs, reports and on-line screens. The definition of the EIS was stated in the cover letter of the questionnaire sent to listed JSE companies. In the questionnaire listed JSE companies indicated whether they have an EIS. Although an EIS is an extremely important information system within businesses, it is not the one and only formal framework that provides information within businesses. Eleven other information systems were identified and compared with EISs. The following were documented regarding the eleven information systems: several different information systems may exist in one business; information systems are completely independent; some information systems are interconnected; a collection of several information systems is referred to as an information system; and information systems are connected by means of electronic networks.

3.2 Capabilities of EISs

Twenty-three major capabilities of EISs of thirteen studies are tabled in Table 1. These capabilities were identified to determine the criteria for the success of EISs in listed JSE companies as per Table 6. Table 1: Capabilities of an EIS
Studies
Rainer Watson Young Byrd Khan and Watson and and Watson Singh Watson Marshall and Holmes (1990) (1992) (1995) (1995) (1995) (1995) (1995) (1996) (1996) (4) (2) (12) (7) (13) (11) (8) (9) (3) Drill down x x x x x x x x Easy access x x x x x x Status x x x x access mode Exception x x x reporting SterOlson Chi Nord and and and Court- Turban Nord ney Turban Mclean and Wetherbe (1996) (1) x x Walstrom and Wilson (1997) (10) x Galliers Laudand don Leid- and ner Lauddon (2004) (2005) (6) (5) x x x x

Capabilitie rens of EIS berg

T o t a l
12 7 6

www.ejise.com

58

Academic Conferences Ltd

Elmarie Papageorgiou and Herman de Bruyn

External databases access SterOlson Chi Nord and and and Court- Turban Nord ney

Studies Capabilitie rens of EIS berg

Rainer Watson Young Byrd Khan and Watson and and Watson Singh Watson Marsand hall Holmes (1990) (1992) (1995) (1995) (1995) (1995) (1995) (1996) (1996) (4) (2) (12) (7) (13) (11) (8) (9) (3) x x x x x x x x Turban Mclean and Wetherbe (1996) (1) x x Walstrom and Wilson (1997) (10) Galliers Laudand don Leid- and ner Lauddon (2004) (2005) (6) (5) x

T o t a l
4 4 4

Trend analysis Ad hoc facilities Text, graphs, tables on one screen Internal database access Extensive graphics Electronic mail Internal & external information on one screen Security & control Improve managemen ts performance Critical success factors Multidimensional analysis Quick & immediate access Nonkeyboard interface Directly used by executives Colour screens Forecasting capabilities Open-ended problems explanations Standard definitions across businesses

x x x x

x x x

3 3 x 3

x x x

2 2

x x x

1 1 1

3.3 Problems of EISs

The planning, developing, implementing and maintaining of EISs are not a trouble-free endeavour. The problems of EISs are investigated and discussed in twelve different studies, with regard to the limitations, shortcomings, obstacles, implications and disadvantages of EISs. The main problems are summarised as follows: defining EIS objectives; 59
ISSN 1566-6379

www.ejise.com

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (57 76)

defining the information requirements for executives; time versus cost; an EIS is not timely; an EIS is a personalised system which is influenced by the characteristics of the CEO; executives and developers must work closely together in order to establish an effective EIS; and EISs are personalised systems and are customised for only executives.

Problems of EISs were identified to determine the reasons why listed JSE companies do not invest in EISs.

3.4 Purposes of EISs

The primary purpose of an EIS is to provide an essential link between management, which includes top management, executives, senior management, other managers, and the employees of the business (Khan, 1996:17). From readings the following purposes of an EIS are summarised: To reduce the amount of useless information: By using summarised reports of financial and nonfinancial information. To promote cost saving: A variety of businesses suggested cost savings through paper flows and reductions in staff and middle management (Khan, 1996:17). To encourage real-time and on-time reporting: Turban, Mclean and Wetherbe (1996:557) refer to it as status access mode; the latest and most recent data can be accessed at any time. To provide drill-down facilities: According to Bocij, Chaffey, Greasley and Hickie (2003:257) an EIS provides a drill-down feature which gives the executive the opportunity to access detailed information other than of the summarised information. Brody (1988:45) explains that by using drilling down facilities, executives can identify a problem area. To function as a management tool: An EIS can be used as a management tool in order to manage a business effectively. To plan and control information: Rockart and Treacy (1982:83) suggest a central purpose of EISs is to provide the Chief Executive Officer (CEO) with information that is planned and controlled. To provide a comparative evaluation: Over time, data has been captured in each business unit that enables the business to make use of comparative evaluation in order for executives to compare major competitors, key customers (Olson & Courtney, 1992:214), and financial reviews of past years with the current financial year.

A study by Volonino and Drinkard (1989:37) successfully integrates different management information systems into the strategic business plan. The objectives and purposes of an EIS by Volonini and Drinkard (1989:41) are summarised as follows: to provide all levels of management with current information, sophisticated and highly advanced control reporting on customers and accounting reports which reflect accurately summarised transactions; to reduce the dependence on customer service personnel; to provide the facility to all management to generate their own ad hoc reports; and to reduce the cost of paper, printing, mail and messengers.

From these studies the purposes for using EISs were identified and in the questionnaire, listed JSE companies identified the main purposes for using EISs as a business tool in order to determine the use and availability of EISs for executives to make strategic decisions.

3.5 Strategic value as the driving force behind EISs

In its original sense strategy is referred to as the development plan for deceiving or outwitting an enemy (Bocij et al., 2003:515). The term strategic management covers the full range of activities in the strategy process, including data collection and interpretation (Manheim, 1989:16). The strategic plan of the business needs to be evaluated to ensure that objectives and strategies are clear and practical while the strategic success factors need to be reviewed and monitored (Kogan, 1986:10). Business management processes are outlined and discussed to compete and to contribute to add a
www.ejise.com

60

Academic Conferences Ltd

Elmarie Papageorgiou and Herman de Bruyn

strategic value through EISs to accomplish how best to shape the business management processes of businesses. The main issues are: challenges to position and shape information technology within the business the strategy, relationships and work methods are required to capture value (Marchand, Davenport & Dickson, 2000:283); effective management leads to the intensifying of competition and the emerging of new markets (Lohman, Sol & De Vreede, 2002: Introduction.par.1); adequate management planning ensures effective digital strategy (Hackbarth & Kettinger, 2000:78); creating an effective environment for users to enjoy competitive advantage over competitors (Chen & Nath, 2005:64); effective planning versus user perception of effective information technology activity (Cash, Mcfarlan & Mckenney, 1993:267); a information system allows users to examine the strategic relationship between businesses that share data electronically (Young, Carr & Rainer, 1999:32); a new and fresh information technology indentifies new opportunities, and capabilities(Typanski, 1999:32); CEOs need to participate in strategic information system planning (Kearns & Lederer, 1999:59); and management to provide an environment that is robust, extensible and strategy implementable (Rabin, 2003:62).

Lohman et al. (2002: Abstract, para.1) suggest that the availability of effective management information is essential when dealing and coping with the complexity and dynamism, both internal and external in businesses. Yoon (1999:64) examines the steps to discover knowledge when users extract information that is useful for decision support and how management can apply these steps more effectively for meeting the overall corporate strategies of the business. Yoon (1999:70) concludes that there is a growing gap between powerful massive storage retrieval systems and the ability to analyse the collected data effectively to extract useful knowledge for decision-making processes. Stokes (2004:86) discusses the competencies needed by top management, executives and senior managers that contribute to the sustainable effectiveness of information technologies. Clarke and Cameron (1992:105) further argue that with the introduction of new information technologies are often associated with significant changes to the business. Kearns and Lederer (1999:59) suggest that CEOs need to participate in strategic IS planning that may be important under conditions of increased environmental uncertainty and/or information intensity, especially for businesses highly dependent upon information technology. From these different studies the strategic value of EISs were identified and in the questionnaire, listed JSE companies indicated if investing in EISs will benefit their company and reasons why companies invested in EISs.

4. The link between EISs and strategies

Executives are familiar with information systems, some have used them, extract information, maintain and develop them, but what executives want are to manage better as a team that lead their business to execute business strategy better and faster in the face of a changing business environment (Gunner, 1986:55). According to Kogan (1986:12) every business has some form of executive reporting system. Yet often these systems do not focus adequately on the most important task of senior management the implementation of strategy. Bocij et al. (2003:563) state that if a business does not have a clear picture of what its strategy is, it is difficult to see how the right information systems can be put in place. Also if the information needs are unclear it is difficult to determine what the right information technology must be to satisfy those needs. A list of main limitations to overcome in providing an EIS: Accessing accurate data Executives ever-changing information requirements

www.ejise.com

61

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (57 76)

Defining objective of EIS Sufficient staff and computer resources Variety, complexity and defining information requirements Timely data Information not in a standard format Development of EISs according to the executives individual needs Distribution of information management responsibility New developments require frequently information technology changes to survive in an extremely competitive and global environment Decisions regarding buy or build Personalised systems are customised for executives only Focus on technical support rather than information support Executives and developers must work closely together in order to establish an effective EIS

5. Research design, research methodology and data collection

A literature review was conducted with the specific aims of identifying, firstly, EISs and, secondly, the strategic value of the business. Observations were made and specific areas were identified from the review. A questionnaire (See Appendix A) was designed to identify if companies are using EISs to create strategic value. This study is an exploratory and discovery-orientated study and the research methodology is quantitative in nature. A questionnaire with a cover letter was e-mailed to 334 listed JSE companies on the 19 of October 2007. Most of the questions were only a "click" of the mouse that required a response and other questions were open-ended questions. Some questions were compulsory and respondents could not proceed if required fields were not completed. Each e-mail sent was numbered and linked to a listed JSE company in order to facilitate the follow-up process for non-responses. A spreadsheet to monitor the results was developed.
th

6. Findings
The link to the survey was deactivated on the 23 of November 2007. Of the 334 listed JSE companies, eighty seven CEOs, Financial Directors (FDs), Managing Directors (MDs), directors, executives and senior managers responded, a response rate of 26%. The percentage reported is based on the number of responses received from each completed questionnaire. However, in some cases more than one person per company responded to the questionnaire and since the unit of analysis is a listed JSE company, sixty five listed JSE companies responded to the questionnaire, a response rate of 19.5%. Twenty five of the respondents indicated that they had an EIS and the remainder of forty respondents indicated that they did not have an EIS at that time. Thirteen of the forty respondents that did not have an EIS were planning to implement an EIS in the near future. The support of EISs will therefore increase from 19.5% to 59% after future planned EISs were implemented. Some of the addressees responded to the senders e-mail via e-mail that they were unable to respond to the survey due to various reasons. The sixty five companies that responded to the questionnaire represents 16% (R0.995-trillion of the total market capitalisation (R6.219-trillion as October 2007) of the all the JSE listed companies. The findings of the questionnaire are discussed in the following seven tables. Number of employees: This question was asked to determine the number of employees in the company to compare the association between the size of the company and the existence of an EIS. The JSE listed companies in the survey represent a variety of companies, very small, small, medium, large and extra large as per the number of employees. The extra large companies represent 40.0% of the 65 respondents. In the cross tabulation between associations Table 3 number of employees were associated with the number of companies that have an EIS. Table 2: Number of employees
rd

www.ejise.com

62

Academic Conferences Ltd

Elmarie Papageorgiou and Herman de Bruyn

Number of Employees Less than 50 50-100 100-1000 1000-2500 More than 2500 Total

Frequency Valid Percent 10 5 19 5 26 65 15.4 7.7 29.2 7.7 40.0 100.0

In Table 3, forty of the sixty five companies responded that they do not have an EIS. The size of the company and the existence of an EIS were compared to determine if an association exist. A Pearson Chi-square test for independence was conducted to determine whether the size of a company is related to having an EIS. The proportion of large companies that have implemented an EIS (65.4%) is significantly different from the proportion of both; small (less than 100; 6.7%) companies and medium (100 to 2500; 29.2%) companies ( (2) = 15.245, < 001). The magnitude of the association was moderate (Cramers V: 0.484). The practical value of the test determines that larger companies have the tendency to have an EIS in relation to smaller companies. Table 3: Association between the size of the company in three categories and the existence of an EIS
Number of Employees/ Size of the Company Have EIS Yes No Total

Count Less than 100 100-2500 More than 2500 Total 1 7 17 25

% within Number of Employees 6.7% 29.2% 65.4% 38.5%

Count 14 17 9 40

% within Number of Employees 93.3% 70.8% 34.6% 61.5%

Count 15 24 26 65

% within Number of Employees 100.0% 100.0% 100.0% 100.0%

Respondents by position in the company: This question indicates if the person forms part of the executive team of the company because an EIS is aimed at the executives of the company. The executive(s) of a business can be one or more persons in the business whose job position(s) and/or title(s) varies from director, chairman, president, chairperson, chairwoman, chief executive officer, managing director, financial director, senior manager, manager, controller to name a few (Collins, 2002:163). There is a very fine line between executives and non-executives, because executives are treated differently in relation to non-executives in respect of staff benefits and other special privileges. Table 4 shows the rank order of the persons position in the company. Clearly from all the respondents the majority of the respondents which have completed the questionnaire were: CEOs (29.3%), followed by the FDs (24.7%). Although questionnaires were addressed to top management it is interesting to note that other levels participated substantially, surprisingly the response rate of managers was 17.0%. Nine respondents (13.5%) indicated other positions that were not listed in the questionnaire. Table 4: Respondents by position in the company
Respondents by position in the company CEO FD Manager Director MD OTHER Total Frequency Valid Percent 19 16 11 8 2 9 65 29.3 24.7 17.0 12.4 3.1 13.5 100.0

www.ejise.com

63

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (57 76)

Purpose/s of using EISs: Potentially an EIS provides a wide range of purposes and capabilities but only the main purposes were listed as per Table 5. This question explains the main reasons why EISs are beneficial to companies. This question was not compulsory and respondents could select more than one purpose for using EISs. Nearly thirty seven percent (36.9%) use EIS as a decision-making tool and twenty three (23.1%) use EIS as a problem solving tool. Table 5: Purpose/s for using EISs
Purpose/s for using EISs Marked Count Percentage Decision-making Problem solving Quick reference Scheduling Electronic mail Other 24 15 8 6 6 3 36.9% 23.1% 12.3% 9.2% 9.2% 4.6%

Criteria for the success of EISs: This question explains if EISs are successful in listed JSE companies. This question was not compulsory and the respondents could select more than one criterion for the success of EISs. Decision-making and problem solving scored the highest as the most important purposes of EISs and therefore executives rated the quality (35.4%) and integrity (32.3%) of information as the most important criteria for the success of EISs. The criterion cost was not a major factor for the success of EISs and the criterion interest of staff was not even considered important by respondents. Surprisingly the criterion better communication among management (15.4%) was added as a last criterion to test the diversity of staff in current South African companies as well as the relationship of information technology professional staff versus professional financial staff. The latter is an on-going debate in companies as to who is responsible for what as there are definite grey areas where both parties are responsible for certain managerial duties. Table 6: Criteria for the success of EISs
Criteria for the success of EISs Count Quality of information Integrity of information User-friendly Quick response time Decision-making Regularly updated Management involvement Better communication among management Cost Interest of staff Other: Integration - visibility across the value chain 23 21 16 15 13 13 12 10 9 0 1 Marked Percentage 35.4% 32.3% 24.6% 23.1% 20.0% 20.0% 18.5% 15.4% 13.8% 0.0% 1.5%

Believing that investing in EISs will benefit your company: This question was a Yes/No question to determine if investing in EISs will benefit the company. In Table 7, 96% of the companies that have an EIS agreed that investing in EISs will benefit their companies and the reasons why are listed as per Table 8.

www.ejise.com

64

Academic Conferences Ltd

Elmarie Papageorgiou and Herman de Bruyn

This question was compulsory. An overwhelming response were received that companies believe in EISs that guarantee the continuous use of EISs in listed JSE companies. Only one company of the Information Technology industry did not believe that investing in EISs will benefit the company.

Table 7: Believing that investing in EISs will benefit your company

Believing that investing in EISs will benefit your company Frequency Valid Percent Yes No Sub Total No EIS Total 24 1 25 40 65 96.1% 4.9% 100.0

This question was not compulsory and seventeen addressees responded to the question as per Table 8. Table 8: Why investing in EISs will benefit your company
Why investing in EISs will benefit your company All executive easily able to compare and access across the group. Availability of critical data. Consolidate and make available to key decision makers group information which was not available before due to our decentralised management philosophy. EIS allows for control; Quick decision making; Poor performance detection and quick action; and Good communication. Fast, efficient and cost effective way of implementation. For increasing productivity and using people to full potential; Managing the business instead of spending time number crunching; As a competitive advantage; To stay on the leading edge; and To have visibility across the value chain - to create synergy. Group wide information capturing and sharing were required. The manipulation of management information is the essential key to success in business. It's not the technology that matters, but how the information is used. It has enhanced the quality of decision-making in the organisation. Management of data and information flow. Mechanism to deliver reporting. Our company operates in a very competitive environment and decision-makers are highly dependant on accurate, near-real-time information. On the spot decisions can be made with the availability of information. Quick decision making. Sometimes difficult to explain to non financial and non technical people that information systems have a place in an organisation. To realise the benefit

7. Conclusion
Today there is an urgent need for businesses to adapt or to adjust constantly to changing business conditions in an ever-changing dynamic environment. Executives and top management need to empower and encourage lower level members to use information systems that enable them to work more efficiently. In the last decade a major shift has been experienced into a new era leading to a new digital era and digital environment. The driving force is new technology, notably information
www.ejise.com

65

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (57 76)

technology, and it is emphasized as follows: when the landscape changes the map changes. Information systems are designed and implemented every day in a variety of different ways to support the business functions and processes. Management needs to decide and choose how the business needs to be structured to best fit the information system, as each business is different. By introducing an EIS, all users of businesses will have a competitive advantage over their competitors and are positioned strategically in an ever-changing business environment. The results of the questionnaire indicated that thirty eight companies (nearly 60%) of the sixty five listed JSE companies have an EIS or are planning to implement an EIS. As this study was an exploratory study there was no other evidence and formal measures of previous EISs studies to compare the results to, within a South African context. Reasons for implementing an EIS varied as an EIS is: an excellent management tool which makes use of standard tools and built-in features, critical for real-time decision-making, familiar to users in all companies, a competitive edge for competitors and it is not the technology that matters but how the information is used. However it is clear from the results of the questionnaire that an EIS creates and adds strategic value in companies. In conclusion it can be stated that the main objective of the study was met and was supported by the evidence gathered. The respondents to the questionnaire indicated that they understand and show an interest in EISs as a useful part of their company, irrespective whether the company has an EIS or not.

8. Appendix A
Questionnaire

1. 2.

Name of Listed Company:

Number of employees: Name of person completing this questionnaire: Highest level of Education
If other, please specify:

Choose one

3.

4. 5.

Choose one

6.

Position of person completing this questionnaire:

If other, please specify:

Choose one

7. 8.

E-mail Address: Web Address: Type of Industry:

Choose one

9.

If other, please specify:

10.

Which Annual Financial Elements do you download on your Web Site?

Yes Full Annual Report Chairmans Report Corporate Information No

www.ejise.com

66

Academic Conferences Ltd

Elmarie Papageorgiou and Herman de Bruyn

Year-in-Review Management Report Financial Summary Balance Sheet Income Statement Cash Flow Statement Notes to the accounts Changes in Equity Statement Audit Report Links to non-financial information If any other, please specify:

11.

Does your business have an EIS (Executive Information System)?

Yes

No 12.

Did your business have an EIS previously?

Yes

No 13. If Yes, Why did you stop using the EIS Cost Not User-friendly Invasiveness No involvement from management Not regularly updated If other, please specify: 14.

www.ejise.com

67

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (57 76)

Is your business planning to implement an EIS?

Yes

No If Yes, when do you foresee implementing an EIS?

Choose one

15.

Will you accept another EIS, like a standard EIS for all businesses in a standard format?
Yes No

16.

If Yes, under which conditions?

Cost User-friendly Non-invasiveness Management Involvement Regularly updated If other, please specify:

(Tick all that apply)

17.

Who will be your main decisionmaker concerning the implementing of an EIS if required in the future?
CEO Executive Accountant MIS (Management information system) Manager If other, please specify:

(Tick all that apply)

If Question 11 was answered "No", please ignore the rest of the questionnaire and click on submit now
Submit Reset

18.

Who was responsible to implement the original EIS in your company?

CEO Executive Accountant MIS (Management information system)

(Tick all that apply)

www.ejise.com

68

Academic Conferences Ltd

Elmarie Papageorgiou and Herman de Bruyn

Manager If other, please specify: 19.

For what purpose does your company use an EIS?

Decsion-making Problem solving Scheduling Quick reference Electronic mail . 20. If other, please specify:

(Tick all that apply)

Who is defining the requirements of the EIS for executives?

CEO Executive Accountant MIS (Management information system) Manager If other, please specify:

(Tick all that apply)

21.

What software is used for your existing EIS? If other, please specify:
22.

Choose one

How many people in your business can access the complete EIS?
23.

Choose one

Who are the main users of EIS?

CEO Executive Accountant MIS (Management information system) Manager Other users outside the company If other, please specify:

(Tick all that apply)

www.ejise.com

69

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (57 76)

24.

What are the criteria for the success of an Executive Information System in (Tick all that apply) your company?
Cost

User-friendly
Quality of information Integrity of information Quick response time Management involvement Regularly updated Interest of staff Decision-making If other, please specify: 25. Who decides what content must be included in the EIS? CEO Management Committee Executive Accountant MIS Manager Other (please specify)

Involvement
Main Some Minor None

26.

Who adds new features to the EIS? (Tick all that apply)
CEO Executive Middle Management Auditors Investors Accountants Other users outside the company If other, please specify:

27.

Who ensures the integrity of the data in the EIS?

(Tick all that apply)

www.ejise.com

70

Academic Conferences Ltd

Elmarie Papageorgiou and Herman de Bruyn

CEO Executives Middle Management Auditors Investors Accountants Other users outside the company If other, please specify: 28.

Who assures ownership of the EIS?

CEO Executives Middle Management Auditors Investors Accountants IT Person Non-Management Staff Other users outside the company If other, please specify:

(Tick all that apply)

Yes

No

29.

What Financial information is available on EIS for Executives?

Balance Sheet Income Statement Cash flow Statement 5-Year Review Comparative Figures Actual versus Budgets figures for the company Taxation Enquiries VAT Reports Graphs on Ratios/Budgets vs. actual of the company Drill down to Chart of Account level On-time, real-time

(Select all that apply)

Yes, Regularly Upgraded

Yes, but not always No Upgraded

www.ejise.com

71

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (57 76)

Exchange Rates Interest Rates Links to the JSE Trend Analysis Capability to do what-if-analysis Audit Report Exception Reports Investors Guide Ratios Fixed Asset Register Database Access Report Writer for Formal Reports Management Report Financial Summary If other, please specify:

30.

What type of Non-Financial information is available on EIS?

Links to News Agencies Mission Statement Logo of Firm Contact details of Firm Directors of the Firm Shareholders of the Firm Unions Complement of Staff (Headcount) Internal Telephone list/E-mail of staff Other Important Contacts, like Auditors, Consultants, Investors, Contractors, Other etc Order form for Stationery and Printing Scheduling Meetings

(Select all that apply)

Yes, Regularly Upgraded

Yes, but not always No Upgraded

www.ejise.com

72

Academic Conferences Ltd

Elmarie Papageorgiou and Herman de Bruyn

Leave Forms/Leave Available Salary and Wage Negotiations/Agreements Canteen facility Parking bay allocation Application form for personal loans Medical Aid for Staff (Claims, Policies etc) Pension Fund for Staff (Retirement, Contributions etc) IRP5 Street Map for local deliveries Fixed Assets per office, room, section etc Booking of Internal Venues Travel Agencies/Accommodation Wine and Dine Car Hiring Computer and workstation assistant First Aid Fire Drill Multimedia and Hypermedia Products/Services Organisational Structure of Firm UIF Internet Banking Calendar of events/meetings Labour Laws If other, please specify:

31.

Which of the following factors affect the efficiency of your EIS?

Response Time Involvement of Executives

(Select all that apply)

Crucial Important Minor

None

www.ejise.com

73

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (57 76)

Lack of resources Lack of interest Saving Cost Quality of Information Identify New Opportunities Increase Productivity Improves control and communication On-time, real-time Respond time to competitors must quicker Availability of Market Indicators Knowledge of new technology User-friendliness Software upgrades Security Viruses Availability of information regularly required Links to JSE, Databases etc If other, please specify:

32.

Will you believe that investing in EISs will benefit your company Yes No If Yes, Why?

33.

Will you propose a standard EIS for all companies?

Yes No If Yes, Why?

www.ejise.com

74

Academic Conferences Ltd

Elmarie Papageorgiou and Herman de Bruyn

34.

What other valuable experiences of EIS would you like to share?

End. Thank You for your Participation

References
Bocij. P., Chaffey, D., Greasley, A. and Hickie, S. (2003) Business information systems: Technology, development and management for the e-business, 2nd ed, Prentice-Hall, Harlow. Brody, H. (1988) Computers invade the executive suite, High Technology Business, Vol 8, No. 2, pp 41-45. Byrd, T.A. and Marshall, T. (1996) Corporate culture, related chief executive officer traits, and the development of executive information systems, Computers in Human Behaviour, Vol.12, No. 3, pp 449-464. Clarke, R. and Cameron, J. (1992) Managing information technology's organisational impact: II, Elsevier, NorthHolland. Cash, J.I., Mcfarlan, F.W. and Mckenney, J.I. (1992) Corporate information systems management, 3rd ed, Irwin, Illinois USA. Chen, L. and Nath, R. (2005) Nomadic culture: Cultural support for working anytime, anywhere. Information Systems Management, Vol 22, No. 4, pp 56-64. Chi, R.T. and Turban, E. (1995) Distributed intelligent executive information systems, Decision Support Systems, Vol 14, No. 2, pp 117-130. Collins Thesaurus. (2002) Collins Thesaurus, Harper Collins, Glasgow. Galliers, R.D. and Leidner, D.E. (2004) Strategic information management: Challenges and strategies in managing information systems, 3rd ed, Elsevier, Oxford. Gunner, H. (1986) Executive Information Systems: The real payoff in improving the link between information and management processes: DSS-86 Transactions, Sixth international conference on Decision Support Systems, Edited by J. Fedorowicz, Washington: College on Information Systems. Hackbarth, G. and Kettinger, W.J. (2000) Building an e-business strategy, Information Systems Management, Vol 17, No 3, pp 78-94. Kearns, G.S. and Lederer, A.l. (1999) The influence of environmental uncertainty on the strategic use of information systems, ACM SIGCPR, Vol 20, No. 3, pp 40-68. Khan, S.J. (1996) The benefits and capabilities of executive information systems, Johannesburg:University of the Witwatersrand (MBA). Kogan, J.N. (1986) Executive Information Systems: Information for motivation: A key to Executive information systems that translate strategy into results for management: DSS-86 Transactions, Sixth international conference on Decision Support Systems, Edited by J. Fedorowicz, Washington: College on Information Systems. Laudon, K.C. and Laudon, J.P. (2005) Essentials of management information systems, 6th ed, Prentice-Hall, New Jersey. Lohman, F.A.B., Sol, H.G. and De Vreede, G-J. (2002) The illusion of effective management information: A critical perspective, Computer Society, Proceedings of the 36th Hawaii International Conference on System Sciences. Manheim, M.L. (1989) Toward true executive support: Managerial and theoretical perspectives: DSS-89 Transactions, Ninth International conference on Decision Support Systems, Edited by G.R. Widmeyer, Providence: College on Information Systems. Marchand, D.A., Davenport. T.H. and Dickson, T. (2000) Mastering Information Management, Prentice-Hall, London. Nord, J.H. and Nord, G.D. (1995) Executive information systems: A Study and comparative analysis. Information and Management, Vol 29, No. 2, pp 95-106. Olson, D.l. and Courtney, J.F. (1992) Decision support models and expert systems, Macmillan, New York. Rabin, S. (2003) The real-time enterprise, the real-time supply chain, Information Systems Management, Vol 20, No. 2, pp 58-62. Rainer, R.K. and Watson, H.J. (1995) What does it take for successful executive information system? Decision Support Systems, Vol 14, No. 2, pp147-156. Rockart, J.F. and Treacy, M.E. (1982) The CEO goes on-line, Harvard Business Review, Jan-Feb, pp 82-88. Sawy, A.E. (1989) How to make DSS EIS ready: DSS-89 Transactions, Ninth International conference on Decision Support Systems, Edited by G.R. Widmeyer. Providence: College on Information Systems. Sterrenberg, G.K. (1990) The structure of executive information systems, Johannesburg:University of the Witwatersrand (MBA). Stokes, S.L. (1999) At the crossroads of technology and management, Information Systems Management, Vol 16, No. 2, pp 7-11. Thomsen, E. (2003) BI's promised land, Intelligent Enterprise, Vol 6, No. 4, pp 21-25. Turban, E., Mclean, E. and Wetherbe, J. (1996) Information technology for management. John Wiley, Toronto.

www.ejise.com

75

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (57 76)
Typanski, R.E. (1999) Creating an effective information environment, Information Systems Management, Vol 16, No. 2, pp 32-39. Volonino, L. and Drinkard, G. (1989) Integrating EIS into the strategic plan: A case study of Fisher-Price: DSS-89 Transactions, Ninth International conference on Decision Support Systems, Edited by G.R. Widmeyer, Providence:College on Information Systems. Walstrom, K.A. and Wilson, R.l. (1997) An examination of executive information system (EIS) users. Information and Management, 32(2):75-83. Watson, H.J., Watson, R.T., Singh, S. and Holmes, D. (1995) Development practices for executive information systems: Findings of a field study, Decision Support Systems, Vol 14, No. 2, pp171-184. Yoon, Y. (1999) Discovering knowledge in corporate databases, Information Systems Management, Vol 16, No. 2, pp 64-71. Young, D., Carr, H.H. and Rainer, (Jr) R.K. (1999) Strategic implications of electronic linkages, Information Systems Management, Vol 16, No. 12, pp 32-40. Young, D. and Watson, H.J. (1995) Determinants of EIS acceptance, Information and Management, Vol 29, No.3, pp 153-164.

www.ejise.com

76

Academic Conferences Ltd

Post-Implementation Evaluation of Collaborative Technology: a Case Study in Business Education

Andriani Piki Royal Holloway University of London, UK
a.piki@rhul.ac.uk
Abstract: To be successful in their future careers students need to develop diverse skills and qualifications. Firstly, in addition to understanding the course content and the underlying theories, students need to explore the implications that emerge from their practical application and develop their critical thinking and analytical skills. Secondly, students need to gain experience and confidence in working effectively within multidisciplinary and multicultural groups that mirror the situation they are likely to face in their future work environment. Thirdly, they need to familiarise themselves with collaborative technologies (CTs) since these are increasingly used in the workplace to facilitate communication and collaboration between distant co-workers. To address these learning needs it is essential to incorporate CTs (such as videoconferencing systems) in the curriculum and provide wellorganized opportunities for students to gain hands-on experience. Nevertheless, what technologies are used does not make the difference between motivated and unmotivated students; it is how these technologies are used that matters. Whilst innovative technologies can be fascinating, they must be properly evaluated and adjusted to specific educational, individual, and group needs in order to be successfully adopted by students. This evaluation entails taking into consideration the context within which the technology will be used (appropriateness evaluation) and the social-psychological motives for user acceptance (evaluation of user satisfaction). This paper reports the findings from an interpretive case study in postgraduate business education where students were using a state-of-the-art videoconferencing system as part of their workshops and group discussion sessions. This setting provided a suitable social milieu for post-implementation evaluation of this collaborative technology. Qualitative methods were employed including participant observation, focus groups, and analysis of videoconferencing sessions captured on video. The findings indicate that computer-supported collaborative learning (CSCL) helps students become confident with using CTs, learn best practices for communicating and collaborating effectively in technology-mediated settings, and appreciate the impact that technology has on everyday social endeavours. The videoconferencing exercises also engaged students to actively participate in the learning process. Given the duality of technology presence (in educational and business contexts alike) the findings can inform the design of new pedagogical models that maximize the learning potential of CTs. Keywords: computer-supported collaborative learning (CSCL), videoconferencing; collaborative technology (CT), business education, post-implementation evaluation, video-ethnography, case study

1. Introduction
During the last decade a number of reasons have led to the escalating use of collaborative technologies (CTs) in education. Firstly, the increasing accessibility and enhanced functionalities of CTs open up new arenas for knowledge sharing and collaboration among students. CTs ranging from synchronous (such as video and audio-conferencing, shared online applications, etc) to asynchronous (including Web 2.0 tools such as blogs, forums, wikis, etc) provide innovative opportunities for collaborative learning thereby allowing students to develop both their interpersonal and technical skills. Another reason for promoting computer-supported collaborative learning (CSCL) activities is the increasing number of dispersed, multidisciplinary project teams in business organisations. In this respect CSCL can improve students ability to act as part of a group, coordinate their efforts, and apply their collective knowledge in problem-solving situations (Kreijns & Kirschner, 2001; Lehtinen, 2003). To accommodate these aspects, and help students improve their career prospects, educational practices need to be constantly re-adjusted and re-evaluated. Alavi (1994, p.159) argues that: Individuals need to learn at higher rates of effectiveness and efficiency than ever before because of rapidly growing bodies of relevant information and the escalation of knowledge and skill requirements for most jobs. This statement is more relevant today than it was fifteen years ago, and suggests the need for continuous development and evaluation of CSCL practices. Despite their potential, CTs are not yet uniformly integrated in the curriculum in higher education and many challenges remain to be addressed (Dohn, 2009). Many educators use technology simply for dissemination of instructional material and fail to create responsive, collaborative, and active learning environments. In other situations universities may acquire state-of-the-art technologies but lecturers may not design appropriate learning tasks that motivate students to engage with the technology, with the subject matter, and with each other. However, the successful adoption of CTs in education does ISSN 1566-6379 77 Academic Conferences Ltd
Reference this paper as: Piki, A. (2010) Post-Implementation Evaluation of Collaborative Technology: a Case Study in Business Education The Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010, (pp77 - 86), available online at www.ejise.com

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (77 86)

not simply depend on the available features and functionalities of the technology; it primarily depends on the pedagogical strategies and tasks employed (Lehtinen, 2003; Leidner & Jarvenpaa, 1995). Therefore, in addition to the technological aspects, the social and cognitive-psychological aspects of CSCL must be considered when designing, evaluating, and using educational technology (Garrison et al., 2000). The aim of this paper is to investigate how postgraduate business students appropriate a videoconferencing system and how they engage with each other and with technology as part of their learning. This investigation aims to address the following questions: Is the system appropriate taking into account the educational context within which it is used? (appropriateness evaluation) What are the perceived benefits and limitations of using the system and how do these affect student satisfaction and engagement? (evaluation of user satisfaction)

The paper initially lays out the theoretical framework in section 2, followed by an overview of the research context in section 3. The research methodology is then outlined in section 4 followed by the major findings and conclusions from the study in sections 5 and 6 respectively.

2. Theoretical insights
A definition of collaborative learning was provided by Dillenbourg (1999) and has been expanded here to accommodate the use of computer support resulting in the following definition: computersupported collaborative learning (CSCL) is the situation in which two or more people learn something together using technology. As an offspring of socio-constructivism, the major goal of collaborative learning is the construction of knowledge through teamwork and interaction with others. The key pedagogical assumption of collaborative learning is that knowledge is created as it is shared. Therefore, the more information students share the more they learn; participation and active contribution are considered critical for learning. It is also assumed that learners have prior knowledge they can contribute to the conversation and that they will participate if given optimal conditions and incentives (Leidner and Jarvenpaa, 1995). CTs can enhance the learning environment, support new modes of learning and instruction, and ultimately, encourage learners to actively participate in the learning process. Their increasing use has shifted the teachers role from being in the centre of instruction to becoming moderators or facilitators in the learning process. Their use also empowers students to become active participants rather than passive observers (de Freitas & Neumann, 2009; Dohn, 2009). Recent research is considering the possibilities of CTs to enable social interaction amongst students, between teachers and students, and within a wider community of learners (Chou & Min, 2009; Kreijns & Kirschner, 2001).

3. Research context
3.1 The videoconferencing system
The case study involves evaluation of the high-tech Collaborative Learning Laboratory (ColLab) which features an advanced videoconferencing system with technology-enhanced learning facilities (iCOM, 2008). The system (Figure 1), which was launched in November 2007, is available to both students and academic staff and it is used primarily for learning and research purposes. Since its launch, postgraduate business students registered in MSc in Business Information Systems at Royal Holloway University of London have been using ColLab to gain hands-on experience with the available tools and technologies built into the system. The system includes video cameras, high resolution plasma displays (used for videoconferencing and for sharing documents such as presentation slides with the remote sites), and a SmartBoard which allows remote participants to work on the same document simultaneously. The system is controlled by a touch-screen panel and students can log into the system using their usernames and passwords. ColLab can support videoconferencing with up to three remote locations. In addition to connecting with external sites there are facilities for connecting via videoconferencing with other rooms within the university campus hence allowing students to experiment and play with the technology. ColLab also offers recording capabilities (for recording the videoconference discussions) and archiving capabilities (for saving the documents created using the SmartBoard for future reference).

www.ejise.com

78

Academic Conferences Ltd

Andriani Piki

Figure 1: The videoconferencing system (ColLab)

3.2 The pedagogical context within which ColLab is used

The evaluation described in this paper draws from the 2008-09 delivery of the People and Technology course which is one of the core courses for students registered in MSc in Business Information Systems. The distinguishing aspect of this intensive Masters degree is that it brings together students from diverse academic and professional backgrounds including management, marketing and finance, computer science, and engineering amongst others. The aim of the People and Technology course as outlined on the Virtual Learning Environment (Moodle) is to increase students understanding of the principles of ICT design and implementation, with specific reference to advanced interactive systems, and to integrate the understanding with practical experience. To accommodate the diversity of students needs and interests a blended learning approach was employed. The course design incorporated classroom-based lectures, workshops, and presentations from invited speakers from the industry. During the workshops students were organised in groups and two of the groups had to engage (in turn) into collaborative tasks through videoconferencing. The aim was to facilitate their theoretical understanding of the role of technologies in human communication by sharing their views on how this communication can be enhanced, while at the same time gaining practical experience with technological tools. The two groups participating in each collaborative activity were located in two different rooms but within the same building for practical reasons. The learning tasks students had to participate in usually involved role play between the two groups and students were given a case script to frame the topic of their discussions. For instance, in one activity the first group took the role of a parent company and the second group was a newly-acquired child company and the two groups had to discuss and negotiate the new policies following the merger. In another example the first group represented the team that designed a system (e.g. a website) and the second group were their clients. In this example the two groups had to negotiate changes in the system functionality and resolve conflicts regarding the final system. These learning tasks intended to give students the opportunity to learn at multiple levels simultaneously: students could gain practical experience with the system, participate in, and manage, both intra-group and inter-group discussions, and at the same time apply the theories taught during the lecture in practical business problems. All students attended a videoconferencing training session at the beginning of the term and a user manual for ColLab was also created and uploaded on Moodle. The subsequent sections of this paper discuss the post-implementation, interpretive evaluation of this collaborative technology. In the following discussion the terms user and student are used interchangeably.

3.3 Participants
In total, 43 students (of which 17 were female and 26 were male) were enrolled in the course. The students in this cohort had diverse professional and academic backgrounds, different age groups (ranging from 20 to 45 with an average age of 24.3 years) and cultures (there were students from 16 different countries). The majority of students had prior work experience (68%). Many students were familiar with one-to-one desktop videoconferencing systems (e.g. Skype) but most students had no experience with advanced CTs such as the one used in this course which involved many-to-many videoconferencing. The novelty factor was taken into consideration in the study as it inevitably affects student satisfaction and can have implications for the appropriateness of the technology.
www.ejise.com

79

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (77 86)

4. Research methodology
4.1 The challenge of evaluating Collaborative Technologies (CTs)
The aspects that need to be considered when evaluating collaborative systems are more complex and varied than those needed when evaluating single-user systems. Group dynamics and collaborative factors need to be explored in addition to individual cognitive factors, usability (ease of use, effectiveness, efficiency, satisfaction), and contextual issues (Neale et al., 2004; Arrow et al., 2000; McGrath, 1984). Furthermore, collaborative systems evaluation does not only focus on humancomputer interaction; it also deals with human-human interaction as well as with the workflow changes that accompany the use of collaborative tools (Damianos et al., 1999). The variability in group composition and the wider social issues also impact system acceptance and inevitably add to the complexity of evaluating the system. Therefore, choosing a sound methodological framework is critical for capturing these multifaceted aspects. While experimental studies are widely employed and are straightforward for single-user applications, it is difficult to create a group in the laboratory that will reflect the social, motivational, economic, and political factors that are central to group performance. Interpretive field studies, on the other hand, can better capture the social and cognitive-psychological effects of CTs on everyday practices (Grudin, 1988). Nevertheless, quantitative, positivist studies still remain the dominant methodological paradigm in group research and exceed the number of interpretive field studies for various reasons (Arrow et al., 2000). On one hand, field studies often require longitudinal observation and therefore are more time-consuming and costly than laboratory-based or quantitative evaluations. On the other hand, in order to conduct a field study the system to be evaluated needs to have robust, working functionalities to allow researchers to observe how people use it in their everyday activities. However, this is not always possible but, even if there is such a possibility, the designers may be reluctant to release a version of the system which is under evaluation to be used by the intended end-users. Hence, the real-life environment within which evaluation will take place needs to be able to accommodate a technology which is still in an experimental, evaluation stage. Due to these factors researchers often use laboratory methods or exclusively quantitative metrics for technology evaluation (Arrow et al., 2000; Damianos et al., 1999; Cugini, et al., 1997). Experimental studies are conducted in controlled environments with time restrictions and many variables kept constant. Therefore, they less accurately identify dynamic issues embedded into natural environments such as cultural issues and learning curves and hence they fail to capture the wider context within which the system will be used (Cugini, et al., 1997). It can be argued that excluding people from their normal working environment can completely alter their working patterns and practices. Unlike an experimental study, an interpretive field study can provide deeper understanding of how well the system supports the various kinds of collaborative tasks. Interpretivist research studies explore a naturalistic setting in depth in an attempt to generate rich insights and make sense of the contextual aspects that affect the acceptance of collaborative technologies in the workplace (Orlikowski et al., 1995). These aspects cannot be replicated in an experimental or laboratory environment (Rosenberg, 2000). Furthermore, problems with technology often appear after the system is installed and embedded with the users everyday work activities; hence, there is a need for continuous re-evaluation. Within educational literature the number of post-implementation evaluations of CTs is limited. The postimplementation evaluation described in this paper is based on an observational field study which allows capturing the users experiences and perceptions in a naturalistic setting. A field study was deemed suitable for evaluating ColLab since it offers a complete set of functioning tools and it is already being used by students. Post-implementation evaluation is important for designers, researchers, and users in order to identify any external factors that may affect user-satisfaction and acceptance. It also helps to gain knowledge on which pitfalls to avoid in future applications of similar technologies.

4.2 Methods
In the course of the research a combination of qualitative data collection methods were employed. The primary method used was participant observation. Students were observed while using the system as part of their workshops. Some workshops were also video-recorded. Video-recording has the advantage that we can capture how the participants behave, how they use space, and most
www.ejise.com

80

Academic Conferences Ltd

Andriani Piki

importantly how they interact with the system and with each other. These insights are particularly important when it comes to studying interactive, collaborative technologies used by groups of people. Another advantage of video-recording is micro analysis: we can replay the video many times and capture details that might have not been evident during direct observation. However, videoethnography has many ethical implications which had to be considered throughout the research. For this reason participants (both students and the teaching team) were asked to sign an informed consent form and give full, partial, or no consent for their images or videos to be used for research purposes. In addition to direct observation and video-recording a total of nine focus groups were conducted with students at the end of the course. During the focus groups students were given the opportunity to elaborate on their perceptions and behaviours and talk about the issues which had strongly affected their attitude towards using ColLab. Students were asked to re-live their experience, talk about how they felt when using the system, and comment on their experiences and on certain behaviours which were observed during their interaction with the system. The focus groups were useful for eliciting details on how the users themselves viewed their experience with the system. The researcher also participated in group discussions and engaged in informal conversations with individual students in an attempt to grasp their perceptions and attitudes towards the system. Furthermore, to gain a holistic view, the researcher attended all the lectures and workshops in order to understand the students perspectives and identify with the metaphors they were using and with the references they were making to incidents and examples that took place throughout the course. The combination of observational and query data collection techniques facilitated the clarification of details and helped to understand why students were acting (or interacting) in a certain way and what affected their engagement with the system. Finally, as the best way to evaluate a system is to use it (Cugini et al., 1997, p. 15) the researcher used the system personally as part of the evaluation process.

4.3 Data analysis

The focus group discussions were recorded, fully transcribed, analysed, and coded in NVivo according to prominent themes. In particular, content analysis was initially used to filter the data and identify the most salient data that would form the basis for an in-depth analysis. The next stage involved thematic analysis to identify the prominent themes. These were coded and classified into thematic categories which included the students motivation and engagement with the system, collaborative learning patterns, individual learning preferences, and the lecturers role in engaging students to use the system (both in terms of educational approach, feedback, and design of learning tasks). The final stage involved systematically looking for code co-occurrence and relationships between the prominent thematic categories (Miles & Huberman, 1994). The data analysis also draws extensively on the informal discussions with students, from the observations that took place during the learning process as well as from the video recordings. The next section presents the main findings and illustrates how they relate to each dimension of evaluation.

5. Findings
The following sub-sections present the interpretation of the findings with the view to evaluate user satisfaction and the appropriateness of the videoconferencing system for the particular educational context within which it is used.

5.1 Appropriateness evaluation

Since the videoconferencing system is used within an educational context the aim of appropriateness evaluation was to assess whether the system contributes to learning and whether it is aligned with the course aims and objectives and the expected learning outcomes. Specifically, the focus was on exploring whether the system is robust, flexible, and accessible enough to accommodate the underlying educational goals (i.e. experiential learning with CTs and learning through collaboration). To assess the appropriateness of ColLab in the specific context we observed students in action focusing on how they engaged with the system throughout the academic term and later asked them to talk about what they learned by using videoconferencing as part of the course and how it has affected their learning experience in general. As outlined earlier, the pedagogical approach adopted in the particular postgraduate course reflected the view that to understand the principles of technology design and the impact that technology has in everyday work practices, it is important to get hands-on
www.ejise.com

81

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (77 86)

experience in addition to a theoretical understanding. Therefore, by collaborating through videoconferencing students had the opportunity to see things from the perspective of the user rather than considering the technology theories solely from the designer point of view. In this respect ColLab facilitated students learning by giving them a realistic, practical point of view and bringing to the forefront deeper issues that guided students to explore things further. The students reported that seeing things in perspective helped them to identify the practical relevance of what they learn during the lecture and what they read in the literature. This, in turn, helped them to develop their critical and analytical skills by reflecting on what they learn. Unavoidably, adjusting to the videoconferencing setting was not straightforward for every student. Participating in conversations and negotiations with non-collocated participants over a video link made students feel conscious of what they say and how they behave. The fact they had to ensure their messages were clearly transmitted and heard at the remote site deducted from the naturalness of the situation. During observations at the beginning of the year it was obvious that a few students were feeling uncomfortable or shy to talk over videoconferencing resulting in one or two students dominating the discussions. This feeling of uneasiness emerged partly due to the novelty of the situation (both in terms of many-to-many or between-group collaboration and in terms of technologymediation) and partly due to the fact that when two of the groups were participating the rest of the students were sitting in the classroom observing the interaction. Despite this initial nervousness, however, by the end of the course students appeared more confident to participate. This reinforces the value of experiential learning that is, students learn better by doing rather than by submissively acquiring a theoretical understanding as can be seen in the dialogue below: Student: We always knew the theories of how to work in a videoconferencing but actually sitting in front of a camera and speaking as a group is really hard [] Its like being on the stage in front of a hundred people, especially if there is an audience sitting behind and watching both groups, it makes it more harder. But it was really good. Interviewer: Do you think that if you use it more times it would be even better? Student: Yeah. The thing is we got an idea already on how to use it and now the next day we are going to be prepared. So, we know whats going to happen. Its going to be a lot easier. This excerpt shows that even though students were experiencing feelings of anxiety while using the technology (which is often documented in the literature) they could see the benefits in the long term. Furthermore, students seemed to be aware that during the videoconference they had to focus on many things at the same time such as making sure the remote audience can hear them and see them while also focusing on completing the actual task (e.g. negotiate the policies following a merger or agreeing on changes on the website design). By doing so, students started to appreciate the role of best practices (such as speaking slowly, pausing during the discussion, etc) in situations where two remote groups of people need to communicate and collaborate via videoconferencing: Student: With some of the techniques we discussed about doing these things, we understand how to make it easier to get these interactions going, like pausing during the discussion and things like that. This learning experience allowed students to learn how to manage themselves in computersupported collaborative settings. Most importantly they learned this in a safe environment before they move in a more demanding setting, the real business world. Further, as mentioned above, when two groups were using ColLab, their classmates were observing their interactions which allowed them to learn by watching how other students behave in the same situation. There was one incident in particular which was mentioned a number of times by students in the focus groups. When the audio was lost during one of the videoconferences, one of the remote participants wrote a message on a piece of paper saying We cant hear you and showed it to the others through the camera. Although this may be considered common practice (at least for those who widely use desktop videoconferencing), there were students who thought this was an innovative idea. Subsequently, during the focus groups when students were asked what they would do in case they could not hear the remote audience many of them referred to this incident. This example illustrates the emergent learning opportunities created by computer-supported collaborative learning practices. Overall, ColLab is considered robust for the educational purposes for which it was used and can accommodate the underlying learning goals. In some cases there were some technical problems with connectivity or being unable to hear the remote side but it was through these problems that students
www.ejise.com

82

Academic Conferences Ltd

Andriani Piki

actually learned how people manage to communicate despite the constraints posed by technology. The system was also flexible in the sense that it allowed students to experiment with it not only for the purposes of the course but also for pursuing personal interests or using it for their dissertations. Finally, ColLab was readily available outside the classroom hours and students could just book the room and use it either in groups or individually if they wanted to get more experience with it.

5.2 Evaluation of user satisfaction

The aim of evaluation of user satisfaction was to assess how students feel when they use ColLab, what their perceived benefits and limitations of using the system are, and how these affect their attitude and future use of the system. The evaluation draws from participant observation, the focus group discussions, as well as from informal encounters with students in an attempt to gain deeper insights into the extend to which they are satisfied with its use in their learning journey. The combination of research methods was invaluable because it made it possible to compare students actions (the observed behaviour) with what students said about their learning experiences and motivations. In verbalising their feelings and attitudes towards the system students focused more on the individual, personal benefits they got from participating in the CSCL tasks rather than what they could contribute in this collective effort. In particular, students reported that their overall experience was positive and they seemed to be very aware of both the short-term (i.e. facilitating their coursework and putting theories in practice) and the long-term benefits (i.e. developing new skills and preparing for their future careers) although more emphasis was placed on the latter. Additionally, students felt that CSCL tasks gave them the opportunity to experiment and gain familiarity with a state-of-the-art technology which they considered as beneficial in its own right. Given that some students had no prior experience with either group work or advanced videoconferencing systems, using ColLab as part of the course gave them the confidence to work effectively in groups and use similar systems in the future, as illustrated in the following excerpts: Student A: The workshops give you the chance of actually work in groups, interact and its an important experience to the real world. You will be never working alone, you will be interacting with people. Thats the importance of this kind of experience working together, achieving a common goal. Student B: There are so many companies that use videoconferencing so I think its going to be really useful in our future job [] We must know how to present our ideas and communicate with other workers. The focus groups also led us to some insights regarding the motivations and mixed factors that affected the students attitude towards using the system. Students seemed very enthusiastic and involved during the workshops and it was expected that they might want to extend the benefits by using the system in their own time. What we found instead was that although students found the videoconferencing exercises helpful in many ways none of them used it outside the workshops. When we investigated this further we found that the limited adoption of the system was related to the fact that the use of ColLab was not part of the assessment criteria. Students mentioned that if their participation was assessed they would get more involved. Although students go to the university to learn this does not necessarily mean that their primary motivating factor for engaging with the learning tasks is learning as such. Students may engage for getting higher marks or for gaining a competitive advantage. Thus, to enable students to actively engage in self-directed learning additional incentives need to be promoted such as making the links to other (assessed) learning tasks more obvious, praising or rewarding those who make an effort, emphasizing the practical skills they can develop by working with others and elucidating how these skills can help them in their future careers. Despite the fact that the wider adoption of the system was nonexistent students were very active during the in-class workshops although they were neither assessed nor mandatory. Students mentioned that they really benefited from the group discussions and that their experiences with ColLab during the workshops were very enjoyable. The question hence was, why did students attend the workshops but did not attempt to use ColLab as part of their self-study? This issue led us to investigate what motivated students to attend the workshops in the first place. We found out that during the workshops students knew they will have a specific task to complete and a given, predetermined case study to discuss which guided their interactions during the videoconference. Students explained that it is difficult to organise such tasks on their own. Another reason was the fact
www.ejise.com

83

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (77 86)

that they knew the tasks would be useful for facilitating their understanding of the theories and therefore would help them complete their individual assignment. Finally, many students explained that the feedback and guidance provided from the lecturer during the workshops was invaluable for their learning. This illustrates that student engagement with CTs is decidedly affected by the educational approach used. It also shows that well-organised, guided learning tasks make students more eager to participate. Further, when students were asked about their perceptions regarding the difference between face-toface communication and communication through videoconferencing the majority explained that the latter takes longer time, more effort, and requires more preparation. Some students also referred to the learning outcomes and stated that after having practical experience with ColLab they could appreciate how important it is to design systems that accommodate diverse user needs. Finally some mature students (who had prior work experience) mentioned that the experience they had during the workshops mirrors what happens in real-life situations and this created a feeling of fulfilment; this shows the value-added features of this CT. Overall, the findings show that student satisfaction and engagement depend on a combination of aspects and these have to be considered collectively when evaluating the effects of using CTs in education.

5.3 Limitations of the study

In interpretive studies, the researchers bias inevitably impacts the interpretation of the findings. This could be alleviated if more than one researchers collected information independently and then compared their interpretations. Another way to mitigate bias and increase the reliability of the findings is by using mixed methods and this should be addressed in future research. Also, although students became familiar with the researcher and provided their consent to be observed and video-recorded, their behaviours were likely to be influenced by the researchers presence. Quantitative methods could be employed in addition to the various qualitative methods to allow validation and triangulation of the findings. Another limitation was the solitary focus on students perspectives. Future research should systematically address the perspectives of teaching staff on the use of CTs in education since, as findings show, educators play a key role in the successful adoption of the system.

6. Discussion
Engaging students and preparing them for the ever-changing, complex world of work requires continuous re-adjustment and re-evaluation of educational technologies and educational practices alike. The paper evaluates the suitability of a videoconferencing system in postgraduate business education (appropriateness evaluation) and the students perceived motives towards using the system (evaluation of student satisfaction). The evaluation is based on an interpretive, real-life case study for capturing the users experiences and perceptions in a naturalistic setting. The end-goal of performing any type of evaluation is to identify potential areas of improvement. Arguably, the findings from this case study can inform the application of videoconferencing applications in educational contexts, particularly for subjects such as Business and IT. In particular, the findings suggest that using videoconferencing as part of a postgraduate course helped students to gain confidence with computer-mediated communication, learn best practices of how to collaborate in technology-mediated settings, and appreciate the impact of technology on everyday life. The videoconferencing exercises also acted as a means for engaging students to participate actively in the learning process by providing them appealing opportunities to put the subject-related theories in practice in addition to reading and attending lectures. The findings also show that student engagement and motivation to participate in CSCL tasks do not primarily depend on how attractive the technology is; rather they depend on a combination of individual, personal factors such as the benefits that students perceive they get from using the technology, and situational factors such as the type of the learning tasks and the quality of feedback and guidance from the lecturer. Using a system which is not robust enough can actually challenge students more and hence enhance the learning process; however, if educators do not design appropriate CSCL tasks that incorporate the system or if they do not provide sound incentives to encourage students to use it, then the system is less likely to be adopted. Technology alone is seldom enough for successful CSCL practices. We need appropriate learning tasks to be designed in order to engage students to participate and use the technology. Most importantly there is a need for new pedagogical models that make the most of the technology available today. 84

www.ejise.com

Academic Conferences Ltd

Andriani Piki

This study reinforces the view that bringing new technologies in a social educational setting is not the solitary ingredient for engaging students and contributing towards the learning outcomes. A more holistic approach needs to be pursued considering the individual, social, and contextual aspects intertwined in such complex situations. At a broader educational context this could mean that combining both formal and informal learning practices can possibly lead to an enhanced learning experience. For lecturers, the findings suggest that successful CSCL tasks require good organisation and regular feedback and guidance to students (even postgraduate students). For those responsible for educational policies, an issue that needs further investigation is the adjustment of current assessment strategies in order to reflect the value of CTs in learning. Further research is needed to continue exploring the potential uses of CTs in enhancing collaborative learning and to illuminate better understandings of the interplay and mutual influences between technology use and learning practices in educational contexts and beyond. Such enquiries alongside continuous re-evaluation of technology in naturalistic contexts may lead to enhanced CSCL experiences.

References
Alavi, M. (1994) Computer-Mediated Collaborative Learning: An Empirical Evaluation. MIS Quarterly, 18, 159174. Arrow, H., McGrath, J. E. & Berdahl, J. L. (2000) Small Groups as Complex Systems: Formation, Coordination, Development and Adaptation, Sage Publications. Chou, S.-W. & Min, H.-T. (2009) The impact of media on collaborative learning in virtual settings: The perspective of social construction. Computers & Education, 52, 417-431. Cugini, J., Damianos, L., Hirschman, L., Kozierok, R., Kurtz, J., Laskowski, S. & Scholtz, J. (1997) Methodology for Evaluation of Collaboration Systems. The Evaluation Working Group of The DARPA Intelligent Collaboration and Visualization Program. Damianos, L., Hirschman, L., Kozierok, R., Kurtz, J., Greenberg, A., Walls, K., Laskowski, S. & Scholtz, J. (1999) Evaluation for Collaborative Systems. ACM De Freitas, S. & Neumann, T. (2009) The use of 'exploratory learning' for supporting immersive learning in virtual environments. Computers & Education, 52, 343-352. Dillenbourg, P. (1999) What do you mean by 'collaborative learning'? In Dillenbourg, P. (Ed.) Collaborativelearning: Cognitive and Computational Approaches. Oxford, Elsevier. Dix, A., Finlay, J., Abowd, G. & Beale, R. (2004) Human-Computer Interaction, Pearson Education. Dohn, N. B. (2009) Web 2.0: Inherent tensions and evident challenges for education. International Journal of Computer-Supported Collaborative Learning, 4, 343-363. Garrison, D. R., Anderson, T. & Archer, W. (2000) Critical Inquiry in a Text-Based Environment: Computer Conferencing in Higher Education. The Internet and Higher Education, 2, 87-105. Grudin, J. (1988) Why CSCW Applications Fail: Problems in the Design and Evaluation of Organisational Interfaces. Proceedings of the ACM CSCW' 88 Conference on Computer Supported Cooperative Work, Perspectives on Evaluation, 85-93. iCOM, (2008) Centre for Research in Information Computing and Communication [Internet] Available at: st http://www.icom.rhul.ac.uk/ [Accessed 21 March 2009] Kreijns, K. & Kirschner, P. A. (2001) The Social Affordances of Computer Supported Collaborative Learning Environments. 31st ASEE/IEEE Frontiers in Education Conference. Reno, NV, USA, IEEE. Lehtinen, E. (2003) Computer-Supported Collaborative Learning: An approach to powerful learning environments. In De Corte, E., Verschaffel, L., Entwistle, N. & Van Merriboer, J. (Eds.) Unravelling basic componets and dimensions of powerful learning environments. Elsevier. Leidner, D. E. & Jarvenpaa, S. L. (1995) The Use of Information Technology to Enhance Management School Education: A Theoretical View. MIS Quarterly, 19, 265-291. McGrath, J. E. (1984) Groups: Interaction and performance. Englewoods Cliffs, NJ: Prentice-Hall. Miles, M. B., & Huberman, M. (1994). Qualitative Data Analysis: An Expanded Sourcebook (2nd ed.): Sage. Neale, D. C., Carroll, J. M. & Rosson, M. B. (2004) Evaluating computer-supported cooperative work: models and frameworks. Proceedings of the 2004 ACM conference on Computer supported cooperative work (CSCW04). Rosenberg, D. (2000) 3 Steps to Ethnography: A Discussion of Inter-Disciplinary Contributions. Artificial Intelligence and Society, Special Issue, 15.

www.ejise.com

85

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (77 86)

www.ejise.com

86

Academic Conferences Ltd

Radio Frequency Identification (RFID) Adoption in the South African Retail Sector: an Investigation of Perceptions Held by Members of the Retail Sector Regarding the Adoption Constraints
Chris Upfold and Haidi Liu Rhodes University, Grahamstown, South Africa
C.Upfold@ru.ac.za
Abstract: Radio Frequency Identification (RFID) technology is a method of identifying unique items using radio waves that communicate between RFID tags and readers without line-of-sight readability. Application areas include person identification, logistics, pharmaceutical, access control, security guard monitoring and asset management. One of the areas where RFID promises excellent potential is in the retail industry for the tracking of goods and products throughout the supply chain. There are concerns around numerous RFID adoption barriers. Decision makers in the South African retail sector seem to be adopting a wait-and-see approach. In an attempt to identify and explore these barriers, a literature review was conducted identifying 29 unique barriers to RFID adoption. A survey instrument, informed by these barriers, was constructed and administered to members of the retail sector in South Africa. The research reveals that the South African retail sector is aware of the benefits in adopting RFID technology, however, they have identified numerous adoption barriers that will need mitigation before they will commit to adopting RFID. The research confirms six main categories with several adoption barriers in each, needing to be addressed. The main categories include, RFID skills shortage, a lack of standardization, high costs of RFID devices, the difficulty of integrating with current legacy systems and a lack of familiarity with RFID systems. Keywords: RFID, diffusion of innovation, adoption barriers, business case, supply chain management

1. Introduction
RFID is a generic term for technologies that use radio waves to automatically identify individual items Sandip (2005:2). Although there are implementation variations, RFID makes use of a microchip with an in-built radio transmitter. The radio transmitter and the microchip together are called the RFID tag. Two types of tag exist; a remotely powered passive tag and a self powered active tag (Sweeney, 2005:20). RFID is gaining acceptance as it moves from being expensive and experimental to increasingly affordable and practically implementable. RFID technology is being applied in retail, supply chains, logistics, and other areas. At present, businesses are focused on using RFID to streamline data collection and data consistency; for example, tracking products through the manufacturing cycle and then locating them at warehouses and retailers. Using RFID, each product can be identified by physical location, manufacturing history and distribution path (Borriello, 2005). As RFID technology improves, so does the application of RFID broaden in scope. Nonetheless, problems associated with the implementation of RFID persist. These include cost variation, the lack of business case study, reader and tag collision, RFID privacy and security issues, radio frequency interference and lack of standards. Given the aforementioned RFID concerns, it is not surprising to find members of the South African retail sector with varying perceptions regarding the usability of RFID technology. This study was initiated to try and understand these perceptions so that a model describing RFID implementation barriers could be constructed.

2. RFID adoption; a brief review of the literature

While the implementation of RFID in the retail sector is a recent innovation and a departure from traditional barcode technology, RFID technology per se is not a new concept and according to Bhuptani et al., (2005: 25), can be traced back to World War II, when the British military needed to find a way to identify whether an approaching aircraft was friend or foe. There are numerous benefits to adopting RFID in the retail sector such as tracking and tracing goods and the containers that hold them, even in harsh environments, since RFID tags do not wear out and do not require line-of-sight to function (Sandip, 2005: 52). RFID can uniquely identify products, cases, and other items, which increases productivity and saves on labour costs in comparison to barcode technology (Sandip, 2005: 115; Lee et al., 2005). RFID virtually eliminates the need to have people locate items and manually scan barcodes. Unfortunately, as with most technology, RFID also has its limitations too. As
ISSN 1566-6379 87 Academic Conferences Ltd Reference this paper as: Upfold, C, and Liu, H. (2010) Radio Frequency Identification (RFID) Adoption in the South African Retail Sector: an Investigation of Perceptions Held by Members of the Retail Sector Regarding the Adoption Constraints The Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010, (pp87 - 96), available online at www.ejise.com

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (87 96) mentioned, RFID tags and transponders transfer information via radio waves. These radio waves can be subject to interference, mainly from metal and liquid products, especially when merchandise is packaged in metal cans or containers (Sandip, 2005: 60). These potential sources of interference must be recognised and accounted for during system planning. Unlike barcodes, it is quite possible for a bad or damaged tag at the item level within a batch of goods to go undetected when passing through the reader (Sandip, 2005: 60). RFID systems consist of more than just the tag. There are several components which introduce integration complexity (Finkenzeller 2003, Sandip 2005 and Shepard 2004). These components include: A programmable RFID tag or transponder for storing data (exception: read-only tags). An antenna to facilitate the reading and writing of data on the tag. In the case of a passive tag, the antenna assists in powering the tag. A reader that encodes or decodes the data in the tags integrated circuitry. In the case of passive RFID systems, the reader also supplies power to the tag. Software components that are required to communicate between the application and the hardware, such as tags and readers. These components include RFID system software, middleware and host applications.

Figure 1: Typical passive RFID tags with antennae highlighted (Bhuptani et al., 2005: 47)

Figure 2: Illustrates the various building blocks which constitute an RFID system (Sweeney, 2005: 78) One of the major factors restricting the development of RFID technology is the disunity of RFID standards (IDTechEx 2004). Clearly, organisations do not want to back a particular RFID standard which may be superseded by another. Currently, the two major standards are International Standards Organization (ISO) and EPCglobal (previously known as Auto-ID). Given the obvious benefits to implementing RFID as well as some impediments as mentioned above, currently, two of the worlds largest retailers; Wal-Mart and Metro in America have committed to adopting RFID technology within their supply chains. (Bhuptani et al., 2005: 31; Shepard, 2004:144). Wal-Mart in particular mandated that their top 100 suppliers use RFID tags on all product deliveries by 2005 (Wal-Mart, 2005). Despite some delay, the process was successfully accomplished and development is ongoing (Wal-Mart, 2006). At present in South Africa, some retailers such as Pick n Pay, Shoprite and Woolworths are aware of RFID and in some cases investigating the technology. There is some reluctance, possibly best described as a look and see approach to adopting RFID technology. In an attempt to understand what could be influencing the uptake of RFID technology within the South African retail sector, a survey was conducted using a purpose built instrument informed via a detailed literature review. The literature review included diffusion of innovation theory; the fundamental theory that explains how a new idea or innovation is spread within a social system, which consists of individuals, informal groups, organisations and subsystems (Rogers, 2003). For reasons of brevity, these theories are merely listed below. They include: Adoption of information technology innovation theory (Moore and Benbasat, 1991) 88
Academic Conferences Ltd

www.ejise.com

Chris Upfold and Haidi Liu

Theory of reasoned action (Fishbein and Ajzen, 1975) Social cognitive theory (Bandura, 1977; Bandura, 1986) Technology acceptance model (Venkatesh, Morris, Davis and Davis, 2003) Theory of planned behaviour (Ajzen, 1991) Model of personal computer utilization (Thompson, Higgins and Howell, 1991)

These theories can be grouped under individual-level and organisational-level technology adoption, each theory addressing particular adoption constraints with a view to understanding how individuals, organisations as well as groups may perceive the viability of adopting a particular innovation. Put together, individual-level technology adoption and organisational-level technology adoption (adoption of IT innovation theory and the diffusion of innovation model), represent a wealth of knowledge regarding the innovation adoption process. These theories provide evidence that a variety of factors influence whether potential adopters and consumers will accept or reject new technology within a social system (Baskerville and Pries-Heje, 2001). Following on from the investigation into diffusion of innovation theory, the literature review investigated adoption barriers. The authors and their respective concerns are listed in table 1 and 2 respectively. Table 1: Authors referred to in the literature review as referenced in table 2.
A: Montgomery (2006) B: Aberdeen Group (2005) C: Swanton (2005) D: VDC (2006) E: ATK and KSA (2004) F: Davison and Smith (2005) G: Wu, Nystrom, Lin and Yu (2006) H: Seymour, Lambert-Porter and Willuweit (2007) I: Cooke (2005) J: Walker (2004) K: Eckfeldt (2005) L: Commonwealth of Australia (2006) M: Leong, NG and Cole (2006) N: Asif and Mandviwalla (2005) O: Michael and McCathie (2005) P: Huber, Michael and McCathie (2007) Q: Angeles (2005) R: Juels (2006) S: Staake, Thiesse and Fleisch (2005)

Table 2: Summary of barriers to RFID adoption (1/3)

Authors and Research Group Barriers to RFID adoption (categories) A Authentication challenges Awaiting next generation of offerings Consumer privacy concerns Current technology in place is satisfactory/existing technologies will work faster and better Customers and suppliers won't use it Health challenges High cost of hardware/infrastructure High cost of software, integration, service, and support High cost of tags High degree of business process change required Implementation challenges Integration challenges X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X B C D E F G H I J K L M N O P Q R X S X

www.ejise.com

89

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (87 96) Table 2: Summary of barriers to RFID adoption (2/3)
Barriers to RFID adoption (categories) Lack of awareness Lack of application requirements/Not X X Authors and Research Group A X B C D X E F G H I J K L M N O P Q R S

applicable or relevant Lack of business case or unconvinced business case Lack of customer demands Lack of senior management support Lack of skilled personnel Lack of standards Need to fix data synchronisation first No compelling value proposition No identifiable business need Not suitable for product assortment Poor tag read rates/tag reader accuracy X X X X X X X

X X X X X X X X X X X X X X X X X X X

Table 2: Summary of barriers to RFID adoption (3/3)

Barriers to RFID adoption (categories) Products not optimal to RFID Security The data would swamp business Unclear on benefits of RFID Unclear ROI X X X X X Authors and Research Group A B C X X X X X D E F G H I J K L M N O P Q R S

3. Methodology
A quantitative research methodology was used for the study. A custom survey instrument consisting of 37 questions based on the research hypotheses was developed and piloted. The Wilcoxon Signed Rank test was chosen as an appropriate statistical tool for Likert scale type data used in the survey. Finally, issues of reliability and validity concerning data collection and analysis for the study were considered and addressed. The questionnaire was distributed via the Internet using an open source survey system called PHP Easy Survey Package. Only one respondent from each retail organisation was permitted to answer the questionnaire. This is because several organisations own more than one major retailer in South Africa. It was believed that major strategic IT decisions such as whether or not to adopt RFID would be made at the parent company level. As an example, Massmart Holdings owns Game, Dion, Makro, Builders Warehouse, Builders Express, Jumbo, Shield and Trade Department. Therefore, it was considered preferable to ask the CIO of Massmart Holdings to participate in the survey, rather than IT managers or IT personnel at the retail branch level. The final list consisted of 39 retailers who were approached telephonically and asked to complete the online survey. 33 completed surveys were collected.

4. Findings and discussion

RFID technology faces many challenges. Some are systematic, and others are as a result of negative perceptions. In order to implement RFID systems successfully, we need to understand some of the key barriers that hinder RFID adoption. Table 3 provides a summary of the technological constraints.

4.1 Technological constraints

www.ejise.com

90

Academic Conferences Ltd

Chris Upfold and Haidi Liu

Table 3: Technological constraints

Area of Constraints Hypothesis Set Factor Analysis of Perceived RFID Barriers Statistical Result: is or is not a barrier

Technological

1 2 3 4 5 6 7

Lack of technological usefulness and advantageousness Lack of a business case Lack of global standards Not suitable for product assortment Poor tag reader accuracy and rates Large amount of data would swamp the business Complexity of technology

No No Yes No Yes No No

81.8% of respondents believe RFID technology would provide additional value if deployed within the retail sector. 60.6% believe there is a convincing business case for RFID adoption in the retail sector. 66.6% of respondents believe that a lack of global standards is a hurdle for RFID adoption. 69.7% believe RFID technology is suitable for product assortment. 39.4% agree that poor reader accuracy is a barrier to RFID adoption while 44% were neutral and only 18% disagreed with this statement. 33.4% of respondents agree that poor RFID read rate is a barrier to RFID adoption while 54.5% were neutral and 12.1% disagree with that statement. 72.7% believe an RFID system does not generate too much data and 75.7% of respondents believe RFID systems are not too complex for users. Discussion: All but two of the technological constraints were not considered as barriers to RFID adoption. The majority of SA retailers do, however, consider poor tag reader accuracy and tag read rates as a drawback of the technology. This barrier will have a direct impact on product detection, and as a result, retailers are concerned that the problem could cause direct financial losses due to inaccurate data. Hence poor tag reader accuracy and read rates are a barrier to RFID adoption in the SA retail sector. There is also concern over the lack of global standards for RFID adoption. Unfortunately, there is no agreement amongst retailers on which standard to adopt in South Africa. There is furthermore a lack of regulation or guidance from the government on which standards should be followed when implementing RFID technology in South Africa. Uncertainty about the future direction of RFID standards is without doubt a factor causing retailers to hold back on the adoption in South Africa. Lack of global standards is seen as a barrier to RFID adoption.

4.2 Costs and Return on Investment (ROI)

Table 4: Cost and ROI Constraints

Area of Constraints Cost and ROI

Hypothesis Set 8 9 10 11

Factor Analysis of Perceived RFID Barriers The high cost of hardware and infrastructure The high cost of software, integration, service, and support The high cost of tags Unclear ROI

Statistical Result: is or is not a barrier Yes Yes Yes Yes

A total of 75.7% of respondents believe the high cost of RFID hardware and infrastructure is a barrier and 72.7% believe the high cost of RFID software, integration, service and support is a barrier. 72.7% believe the high cost of RFID tags to be a barrier and 69.7% believe uncertainty in return on investment is a barrier. Discussion: Cost and ROI are considered key determinants as to whether or not to adopt RFID technology. Currently, the high prices of RFID components including hardware, software and tags

www.ejise.com

91

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (87 96) make it hard to see an immediate return on investment. As a result, retailers will not rush in without first researching the best RFID strategy suitable for their requirements. Certainly, most SA retailers view RFID adoption from a business standpoint, not just a technological one; thus, examining the cost and return on investment is critical for adopting RFID technology.

4.3 Privacy and security constraints

Table 5: Privacy and security constraints
Area of Constraints Privacy and Security Hypothesis Set 12 13 Factor Analysis of Perceived RFID Barriers Customer privacy concerns Security concerns Statistical Result: is or is not a barrier Neutral Neutral

39.4% of respondents believe consumer privacy is a concern, however, 36.4% of respondents disagree with this statement, and 24.2% neither agree nor disagree. 30.3% of respondents believe RFID security is a concern, however, 30.3% of respondents disagree with this statement, and 39.4% neither agree nor disagree. Discussion: It is interesting that privacy and security issues remain neutral. Customer privacy concerns are mostly raised by privacy advocates abroad who do not have a major influence in South Africa. It is believed that the majority of customers in South Africa are not currently aware of the next generation of identification technology, such as RFID, let alone the privacy implications this may have on their lives. As a result of this lack of awareness, retailers are not under pressure to address this potential privacy concern. As general public awareness increases, South African retailers, like their counterparts in other countries will have to give this factor more attention. Hence, customer privacy should be a factor to consider in RFID adoption. A concern about security is somewhat related to privacy. It is about how to keep RFID information safe from hackers or intruders, rather than concentrating on securing customer information, hence, it is more focused internally in the business. It is believed that one of the main reasons SA retailers are not concerned about RFID security is that there are currently no major threats against RFID technology, particularly in South Africa. As RFID technology gains in popularity amongst individuals, retailers and in other marketplaces, security will become increasingly important, while individuals try to exploit this technology for their own benefits. As a result, security concerns will heighten over time.

4.4 Implementation constraints

Table 6: Implementation constraints
Area of Constraints Implementa tion Hypothesis Set 14 15 16 17 Factor Analysis of Perceived RFID Barriers Compatibility and integration with other technology Implementation challenges The need to fix data synchronisation first RFID authentication challenges Statistical Result: is or is not a barrier Yes Yes No Neutral

45.5% of respondents believe that the compatibility and integration of RFID technology with other technologies is a barrier. 69.7% believe that challenges relating to RFID implementation are a barrier. 54.5% believe that data synchronisation between RFID systems and other systems is a problem. 15.2% of respondents agree, 66.7% neither agree nor disagree and 18.2% disagree that a lack of authentication in RFID systems and tags is an adoption barrier. Discussion:Implementation constraints are clearly potential barriers to wide-scale deployment. As indicated in table 6, two out of four implementation factors are considered barriers to RFID adoption by SA retailers. How retailers view the difficulty of implementing RFID is a critical aspect to be measured by retailers that intended to mandate RFID technology in their business. There is no evidence indicating whether or not RFID authentication challenges are a barrier to RFID adoption in the retail sector.

www.ejise.com

92

Academic Conferences Ltd

Chris Upfold and Haidi Liu

4.5 Privacy and security concerns

Table 7: Privacy and security constraints
Area of Constr aints Privac y and Securit y Hypothesis Set Factor Analysis of Perceived RFID Barriers Statistical Result: is or is not a barrier Neutral Neutral

12 13

Customer privacy concerns Security concerns

39.4% of respondents believe consumer privacy is a concern, however, 36.4% of respondents disagree with this statement and 24.2% neither agree nor disagree. 30.3% believe RFID security is a concern, however, 30.3% disagree with this statement and 39.4% of respondents neither agree nor disagree. Discussion: It is interesting that SA retailers are undecided whether or not privacy and security concerns pose a barrier to RFID adoption. This is contrary to the findings from the literature review, which indicate that most researchers and research organisations see privacy and security issues as a key barrier. As previously discussed, when consumers and retailers gain a greater awareness of RFID technology, and the install base increases, so will security risks increase as exploiters realise on the value of data contained within these systems. Hence, it is believed that customer privacy concerns and security concerns should not be discarded as barriers, and must be included in the framework.

4.6 Organisational constraints

Table 8: Organisational constraints
Area of Constraint s Organisati onal Hypothesis Set Factor Analysis of Perceived RFID Barriers Statistical Result: is or is not a barrier Yes Yes Neutral

18 19 20

A high degree of business process change required Lack of awareness A lack of identifiable business needs

57.5% of respondents believe that a high degree of business process change required is a barrier. 60.6% believe that a lack of awareness in RFID technology is a barrier. 48.5% believe that a lack of identifiable business needs is not a barrier for RFID technology. 15.2% of respondents neither agree nor disagree, 30.3% of respondents agree and 6.1% of respondents strongly disagree that a lack of identifiable business need is an organisational barrier. Discussion: An examination of the impact of organisational constrains on RFID adoption revealed that a high degree of business process change and lack of awareness are barriers. It is also interesting to note that the answer to no identifiable business need is neutral, which means that there is no evidence to determine whether this factor is or is not a barrier.

4.7 People constraints

Table 9: People constraints
Area of Constr aints People Hypothesis Set Factor Analysis of Perceived RFID Barriers Statistical Result: is or is not a barrier Yes Neutral Yes

21 22 23

The unwillingness of the customer and supplier to use it Lack of senior management support Lack of skilled personnel

69.7% of respondents believe that a lack of willingness to use RFID technology is a barrier. 39.5% believe a lack of senior management support is a barrier while 24.2% neither agree nor disagree and 36.4% disagree with this statement. 63.6% believe that a lack of skilled RFID personnel is a barrier.

www.ejise.com

93

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (87 96) Discussion: The customers and suppliers lack of willingness to use the technology is a general concern for RFID adoption, and most retailers believe that this is a major adoption impediment. The reason is that retailers alone would not gain maximum benefits from a closed RFID system, since the scope of application of a closed system is limited within a single organisation. There are a multitude of benefits to be derived by an organisation integrating RFID across the entire supply chain. These benefits would positively impact risk and costs while increasing efficiency and success. Hence, an unwillingness of the customer and supplier to use the technology is a barrier to RFID adoption. Surprisingly the response from SA retailers on the issue of senior management support did not clearly indicate whether or not this factor is a barrier to RFID adoption. However, it is believed that a major drawback to wide-spread deployment of RFID systems is peoples overall attitude towards the technology. A possible reason for this research outcome could lie in the nature of the targeted respondents, as most respondents, if not all, were IT professionals, who are more likely to understand and support RFID adoption than other senior management who do not have the same insight. Furthermore, a lack of awareness, as identified earlier, indicates that in general, retailers have insufficient knowledge about RFID, and as a result, would not support its adoption. Hence a lack of senior management support has not been discarded from the framework and is considered a barrier to RFID adoption. RFID-knowledgeable personnel are hard to find. Many SA retailers, regardless of size, would discover they have no qualified RFID personnel. Without expert skills, retailers might end up spending too much time and money on an RFID project, possibly leading to its failure. Hence, SA retailers regard the lack of skilled personnel as a barrier to RFID adoption

4.8 Environment constraints

Table 10: Environment constraints
Area of Constraint s Environme nt Hypothesis Set Factor Analysis of Perceived RFID Barriers Statistical Result: is or is not a barrier No No

24 25

Social influence The effect of radio emissions on personal health

48.5% of respondents believe that social issues surrounding RFID technology is not a barrier while 27.3% neither agree nor disagree and 24.2% agree with this statement. 69.7% of respondents believe that the impact of RFID technology on human health is not a barrier while 27.3% neither agree nor disagree and only 3% of respondents agree with this statement. Discussion: Given the existing minimal install base of RFID in South Africa and the resultant lack of public knowledge and understanding of the technology, it is believed that environmental issues which include social influence as well as health concerns have not really been explored. There is evidence of these concerns in the international market. Within a South African context, these issues are not considered barriers to RFID adoption.

5. Summary
There are 16 barriers identified to be stumbling blocks to RFID adoption in the South African retail sector. These barriers are grouped according to areas of constraint and are illustrated in Table 11. in terms of an enhanced framework. The framework of RFID adoption barriers are sorted according to each category (area of constraints), rather than importance. This framework is an outline of the barriers impacting RFID adoption in the SA retail sector that need to be addressed. Table 11: Enhanced framework of the barriers to RFID adoption in the South African retail sector (1/2)
Area of Constraints Technological RFID adoption Barriers Lack of global standards Poor tag reader accuracy and rates Cost and ROI The high cost of hardware and infrastructure

www.ejise.com

94

Academic Conferences Ltd

Chris Upfold and Haidi Liu

The high cost of software, integration, service, and support The high cost of tags Unclear ROI Area of Constraints Privacy and Security RFID adoption Barriers Customer privacy concerns Security concerns Implementation Compatibility and integration with other technology Implementation challenges RFID authentication challenges Organisational A high degree of business process change required Lack of awareness People The unwillingness of the customer and supplier to use it Lack of senior management support Lack of skilled personnel

6. Conclusion
The potential benefits to implementing RFID are acknowledged by most managers within the South African retail sector. RFID could become the preferred supply chain management technology in future retail systems, however, there are adoption barriers that represent major obstacles that must be identified, understood, and as far as possible, overcome, in order for South African retailers to consider adopting RFID technology. The RFID adoption barriers identified by means of a survey include: Lack of global standards Poor tag reader accuracy and rates The high cost infrastructure of hardware and Implementation challenges RFID authentication challenges A high degree of business process change required Lack of awareness The unwillingness of the customer and supplier to use it Lack of senior management support Lack of skilled personnel

The high cost of software, integration, service, and support The high cost of tags Unclear ROI Customer privacy concerns Security concerns Compatibility and integration with other technology

As RFID technology is increasingly adopted within global markets by companies such as Wal-Mart, it is believed that many of the current adoption barriers will fall away. Costs will decrease through greater scales of economy resulting in improved ROI, technology will improve as equipment suppliers compete for market share and expertise will increase through wider implementation experiences. South African retailers have demonstrated an interest in RFID technology and as adoption barriers decrease, the decision to adopt RFID in the South Africa retail sector may well become increasingly compelling.

www.ejise.com

95

ISSN 1566-6379

Electronic Journal Information Systems Evaluation Volume 13 Issue 1 2010 (87 96)

7. References
Aberdeen Group. (2005) The RFID Benchmark Report: Finding the Technologys Tipping Point, Aberdeen Group. Boston, MA, Available online: http://www.aberdeen.com/summary/report/benchmark/RA_RFID_JF_2466.asp. Ajzen, I. (1991) The Theory of Planned Behavior, Organizational Behavior and Human Decision. Processes 50, 2: pp 179 211 Angeles, R. (2005) RFID Technologies: Supply-Chain Applications and Implementation Issues, Information Systems Management, Vol 22, 1, pp 51-65. Asif, F. and Mandviwalla, M. (2005) Integrating the Supply Chain with RFID: A Technical and Business Analysis, Communications of the Association for Information Systems, 15, 24: pp 393-426. ATK AND KSA. (2004) Connect the Dots Harnessing Collaborative Technologies to Deliver Better Value to Customers. A.T. Kearney and Kurt Salmon Associate. Available online: http://www.gmabrands.com/news/docs/connectthedots.pdf. Bandura, A. (1977) Self-Efficacy: Toward a Unifying Theory of Behavioral Change,. Psychology Review. 84: pp 191-215. Bandura, A. (1986) Social foundations of thought and action: A social cognitive theory (1e), New York: PrenticeHall. Baskerville, R. and Pries-Heje, J. (2001) A Multiple-theory analysis of a diffusion of information technology case, Information Systems Journal. 11, 3: pp 181-212 Bhuptani, M. And Moradpour, S. (2005) RFID Field Guide: Deploying Radio Frequency Identification (1e), USA: Pearson Education. Borriello, G. (2005) RFID: tagging the world: Introduction,Communications of the ACM. 48, 9: pp 34 - 37. Commonwealth Of Australia. (2006) Getting the most out of RFID: A starting guide to radio frequency identification for SMEs, Department of Communications, Information Technology and the Arts (DCITA), Australia. Available online: http://www.dcita.gov.au/__data/assets/pdf_file/41249/Getting_the_most_out_of_RFID.pdf. Cooke, J. A. (2005) Demand for RFID Professionals Exceeds Supply, Logistics Management, 44, pp 4: 21. Davison, j. and smith, s. e. (2005) Retail RFID Stirs Process Change and Improves Product Availability, Gartner. Available online http://www.gartner.com/DisplayDocument?id=482587. Eckfeldt, B. (2005) RFID: tagging the world: What does RFID do for the consumer?, Communications of the ACM. 48, 9: pp 77-79. Finkenzeller, K. (2003) RFID Handbook: Fundamentals and Applications in Contactless Smart Cards and Identification (2e), UK: John Wiley & Sons, Ltd. Fishbein, M., and Ajzen, I. (1975) Belief, Attitude, Intention and Behavior: An Introduction to Theory and Research (1e), MA: Addison-Wesley. Huber, N., Michael, K. and McCATHIE, L. (2007) Barriers to RFID adoption in the Supply Chain, RFID Eurasia, 2007 1st Annual, 5-6 September, Istanbul, Turkey. IDTECHEX. (2004) The Importance of RFID Standards, IDTechEx. Available online: http://www.idtechex.com/products/en/articles/00000029.asp Juels, A. (2006) RFID security and privacy: A research survey, IEEE Journal on Selected Areas in Communication, 24, 2. Lee, H. And Ozer, O. (2005) Unlocking the Value of RFID, Production and Operations Management. 16, 1: pp 40-64. Leong, K. S., Ng, M. L. And Cole, P.H. (2006) Positioning analysis of multiple antennas in a dense RFID reader environment, International Symposium on Applications and the Internet (SAINT) Workshop 4, January 23-27, 2006, Phoenix, Arizona. Luckett, D. (2004) The Supply Chain, BT Technology Journal. 22, 3: pp 50 - 55. Montgomery, N. (2006) RFID in UK Industry An Executive Summary, Benchmark Research Ltd. Available online: http://www.benchmark-research.co.uk/attachments/RFID_Microsoft_LR.pdf. Moore, G. C. and Benbasat, I. (1991) Development of an instrument to measure the perceptions of adopting an information technology innovation, Information Systems Research, 2, 3: pp 192-222. Sandip, L. (2005) RFID Sourcebook (1e), USA: Pearson Education. Swanton, B. (2005) RFID Adoption Presentation - Obstacles with regard to RFID adoption, International Electronics Manufacturing Initiative (iNEMI), October 4, 2005, Newark, CA: AMR Research. Thompson, R. L., Higgins, C. A., and Howell, J. M. (1994) Influence of Experience on Personal Computer Utilization: Testing a Conceptual Model, Journal of Management Information Systems. 11, 1: pp 167-187. Venkatesh, V., Morris, M. G., Davis, G. B., and Davis, F. D. (2003) User acceptance of information technology: Toward a unified view, MIS Quarterly. 27, 3: pp 425-478. Venture Development Corporation. (2006) RFID Business Planning Service RFID Vertical Market VI: Retail POS, Venture Development Corporation. Walker, J. (2004) Commentary: The year of living RFID, Forrester Research. Available online: http://www.news.com/Commentary-The-year-of-living-RFID/2030-1012_3-5147590.html. WAL-MART. (2006) Wal-Mart Continues RFID Expansion, United States: Wal-Mart. Available online: http://www.walmartfacts.com/articles/2596.aspx. Wu, N. C., Nystrom, M. A., Lin, T. R. And Yu, h. c. (2006) Challenges to global RFID adoption, Technovation. 26, 12: pp 1317-1323. www.ejise.com

96

Academic Conferences Ltd