Vous êtes sur la page 1sur 5

Psychiatry Research 237 (2016) 196200

Contents lists available at ScienceDirect

Psychiatry Research
journal homepage: www.elsevier.com/locate/psychres

Validation of the French version of the Acceptability E-scale (AES) for

mental E-health systems
Jean-Arthur Micoulaud-Franchi a,b, Alain Sauteraud a,b, Jrme Olive b, Patricia Sagaspe a,b,
Stphanie Bioulac a,b, Pierre Philip a,b,n
Clinique du Sommeil, Service d'Explorations Fonctionnelles du Systme Nerveux, CHU de Bordeaux, Place Amlie Raba-Lon, 33076 Bordeaux, France
USR CNRS 3413 SANPSY, Universit de Bordeaux, 33076 Bordeaux, France

art ic l e i nf o a b s t r a c t

Article history: Despite the increasing use of E-health systems for mental-health organizations, there is a lack of psy-
Received 9 November 2015 chometric tools to evaluate their acceptability by patients with mental disorders. Thus, this study aimed
Received in revised form to translate and validate a French version of the Acceptability E-scale (AES), a 6-item self-reported
9 January 2016
questionnaire that evaluates the extent to which patients nd E-health systems acceptable. A forward
Accepted 16 January 2016
Available online 19 January 2016
backward translation of the AES was performed. The psychometric properties of the French AES version,
with construct validity, internal structural validity and external validity (Pearson's coefcient between
Keywords: AES scores and depression symptoms on the Beck Depression Inventory II) were analyzed. In a sample of
Satisfaction 178 patients (mean age 46.51 years, SD 12.91 years), the validation process revealed satisfactory
psychometric properties: factor analysis revealed two factors: Satisfaction (3 items) and Usability (3
items) and Cronbach's alpha was 0.7. No signicant relation was found between AES scores and de-
Psychometrics pression symptoms. The French version of the AES revealed a two-factor scale that differs from the
original version. In line with the importance of acceptability in mental health and with a view to E-health
systems for patients with mental disorders, the use of the AES in psychiatry may provide important
information on acceptability (i.e., satisfaction and usability).
& 2016 Elsevier Ireland Ltd. All rights reserved.

1. Introduction decisions for diagnosis, prognosis and therapeutic follow-up

(Chouvarda et al., 2015). While the extent to which patients are
Patient-reported outcomes (PROs), including measures of sa- satised with E-health systems and nd them acceptable should
tisfaction, have gained increasing attention in the mental-health be evaluated, there is a lack of psychometric tools to evaluate it.
services (Boyer et al., 2009; Crow et al., 2002; Fitzpatrick, 1991). The Acceptability E-scale (AES) presents some advantages in
Indeed, patient satisfaction has become one of the important PROs the eld of patient-reported outcomes in E-health systems (Tari-
and a contributing outcome in the assessment of health care man et al., 2011). It is a generic and validated questionnaire that
quality (Zendjidjian et al., 2014b). Patient satisfaction with car- can accurately evaluate satisfaction with a broad range of E-health
egiving is a strong predictor of future behaviors, in particular systems (Carlson et al., 2001; Mullen et al., 2004; Taenzer et al.,
treatment adherence (Ware and Davies, 1983; Zendjidjian et al., 2000), a point that differentiates it from other questionnaires that
2014a; 2014b), and provides important information for improving may have no clear validation process and which focus only on
the quality of health care organizations (Cleary and McNeil, 1988). satisfaction with one precise E-health solution. It is also based on
Given the increased prevalence of chronic disease, in particular 6 items (with a 5-point Likert scale for each item), which is time-
of mental and brain disorders (Wykes et al., 2015), health care efcient and increases its usability in clinical practice. Further-
organizations now have to deal with the increased use of E-health more, it is a self-reported questionnaire, which is recognized as
systems (Chouvarda et al., 2015). Such systems facilitate: i) com- the most reliable method to measure satisfaction (Crow et al.,
munication and coordination between patient and healthcare 2002).
professionals (in primary care and hospital), and ii) medical Despite French being the 6th most widely spoken world lan-
guage with 220 million speakers (Organization International de la
francophonie, 2009), the AES has not previously been translated
Corresponding author at: Clinique du Sommeil, Service d'Explorations Fonc-
tionnelles du Systme Nerveux, CHU de Bordeaux, Place Amlie Raba-Lon, 33076
and validated in French. Thus, in order to better evaluate sa-
Bordeaux, France. tisfaction with E-health solution in native French speakers, this
E-mail address: pr.philip@free.fr (P. Philip). study sought to design and validate a French version. Translating

0165-1781/& 2016 Elsevier Ireland Ltd. All rights reserved.
J.-A. Micoulaud-Franchi et al. / Psychiatry Research 237 (2016) 196200 197

questionnaires may be dependent on cultural background (Brislin, were notied to the French psychiatrist (AS) for a second round of translation and
back-translation (KR). After this second round, the back-translated version and the
1970) so before using any translated questionnaire, a transcultural
original version was declared equivalent. The translated version of the AES was
validation should be undertaken according to specic rules and administered to 7 patients and showed good clarity and cultural acceptability. No
methods. In the present study we analyzed the psychometric further adaptations were required. The nal version of the French AES is shown in
properties of the French AES version in a sample of French patients Table 1.

that was interviewed with a virtual agent, which are the most
2.2.2. Assessments
innovative E-health solution to sustain a clinical evaluation (Philip Self-rated assessment. The Acceptability E-scale (AES) (Tariman et al., 2011)
et al., 2014). In this study, the virtual agents made a clinical eva- and the Beck Depression Inventory II (BDI-II) (Beck et al., 1996) were completed as
luation of depressive symptoms. part of the self-rated psychiatric assessment. The AES consists of 6 items rated by
the patients on a balanced ve-point Likert scale ranging from 1 to 5. The rating
was determined according to patients' experience concerning the evaluation with
2. Methods and materials the virtual agent. The score was obtained by computing the sum of the scores
obtained by items associated with it, from 1 to 5. The scale ranges from 6 to 30. The
BDI-II consists of 21 items rated by the patients according to their experience in the
2.1. Participants
preceding month. The BDI-II has previously been translated and validated in French
(Beck, 1996; Bourque and Beaudette, 1982). The scale ranges from 0 to 63.
Outpatients were recruited by psychiatrists in Bordeaux University Hospital
from November 2014 to June 2015 in a consecutive sample design. Inclusion criteria Psychiatric assessment. Patients were medically examined by an experi-
were: native French-speaking adult patients ( 418 years) with any type of psy-
enced psychiatrist, who judged whether MDD was present according to DSM-5
chiatric or sleep complaints. Exclusion criteria were: insufcient capacity to con-
criteria. However, no standardized interview was applied. The consulting psy-
sent to and understand and answer the self-report questionnaires.
chiatrist judged whether MDD was present and treatment was required. Therefore,
Gender, age, year of education and antidepressant treatment were noted. Pa-
the diagnostic standard for the present study was expert opinion. Only MDD was
tients were given a clinical interview with a virtual agent to evaluate Major De-
evaluated in the present study. The psychiatrists were blind to the results of the
pressive Disorder (MDD) according to DSM-5 criteria, which was the E-health so-
virtual agent.
lution that they had to evaluate with the AES. This virtual agent was adapted from
previously developed software able to self-conduct interactive face-to-face clinical
interviews (Philip et al., 2014). This software was based on four modules (Philip 2.3. Statistical analyses and hypotheses
et al., 2014): i) an interview manager module that conducts the whole interview
and manage the other modules, ii) a 3D rendering module that display the virtual Descriptive statistics of the obtained data included frequencies and percentages
agent and play animations, iii) a speech synthesizer module that create speech of of categorical variables together with means and standard deviations of continuous
the virtual agent and iv) a speech recognizer module that recognize the responses variables. For the validation process, we analyzed the psychometric properties of
of the patients. The software suite was installed on a computer (Windows 8-i7 the French AES version including construct validity, internal structural validity and
3770@3.4 GHz8 GB-NVidia 670 GTX) connected to a 40-in. display and to a Mi- external validity. Data analysis was performed using SPSS software (Version 18 for
crosoft Kinect sensor. In the present study the virtual agent was able to self-con- Mac, PASW Statistics), LISREL software (Scientic Software International, Inc.) and
duct interactive face-to-face clinical interviews to investigate each MDD DSM-5 WINSTEP Software (Wright et al., 2001). For all the tests, the accepted signicance
criteria (American Psychiatric Association, 2013). level was 5%.
Patients were invited to participate in the study during their routine clinical
evaluation. After receiving a detailed description of the study, participants gave
2.3.1. Construct validity
their informed consent. The study was conducted in accordance with the De-
A conrmatory factor analysis (CFA) was performed using the LISREL model to
claration of Helsinki and French Good Clinical Practices. The study was classied as
analyze the construct validity and to test the one factor structure of the original
a clinical trial by the US National Institutes of Health (ClinicalTrials.gov identier:
scale (Tariman et al., 2011). If the Root Mean Square Error of Approximation
NCT02544295). The present article of validation of the AES is part of a more general
(RMSEA) was not satisfactory (4 0.05), a principal component factor analysis (PCA)
study on the validation of based-virtual reality diagnosis for neuropsychiatric dis-
with Varimax rotation was performed to explore the structure of the French ver-
orders and sleep/wake disorders (PHENOVIRTPSY).
sion of the AES (Kaiser, 1958). This analysis determined the nal structure and the
number of independent factor of the French version of the AES. Items were in-
2.2. Procedure cluded in a factor if they revealed a loading greater than 0.4. The undimensionality
of each factor was assessed using Rasch analyses by computing the pattern of item
2.2.1. Translation of the AES goodness-of-t statistics (INFIT) for each factor (Wright et al., 2001). A value of
A forwardbackward translation was performed through a multistep method INFIT between 0.5 and 1.5 ensures that all items of the scale tend to measure the
(Brislin, 1970; Epstein et al., 2015). The original version was translated into French same concept (Wright et al., 2001).
by a French native psychiatrist (AS). The back-translation into English was under-
taken by a professional native English translator (KR) blind to the original version. 2.3.2. Internal structural validity
This back-translated version was submitted to a second professional native English Item-internal consistency (IIC) was assessed by correlating each item with its
translator (HK) who noted the differences with the original version. The differences related factor; correlations of at least 0.4 are recommended for supporting IIC

Table 1
English version/French version and factor analysis of the French version of the AES.

No. Factor Items French version Mean SDa Floor (%) Ceiling (%) Alphab INFITc

1 Usability How easy was this computer program for quel point avez-vous trouv ce programme in- 4.62 1.06 6.2 85.4 0.762 1.64
you to use? formatique facile d'utilization?
2 Usability How understandable were the questions? quel point les questions taient-elles 4.77 0.52 15.7 80.9 0.713 0.95
3 Satisfaction How much did you enjoy using this com- quel point avez-vous apprci l'utilization de ce 4.07 0.91 34.8 38.8 0.595 0.47
puter program? programme informatique?
4 Satisfaction How helpful was this computer program in A quel point ce programme informatique vous a-t-il 3.7 1.07 31.5 27.5 0.626 0.64
describing your symptoms and quality of t utile pour dcrire vos symptmes et votre qua-
life? lit de vie?
5 Usability Was the amount of time it took to com- Le temps consacr rpondre ce programme in- 4.57 0.89 14 75.3 0.644 0.80
plete this computer program acceptable? formatique tait-il acceptable?
6 Satisfaction How would you rate your overall satisfac- Comment valuer-vous votre satisfaction gnrale 4.07 0.93 38.8 38.2 0.577 0.69
tion with this computer program? de cet outil informatique?

Standard deviation.
Cronbach's alpha if item is deleted.
Rasch statistics.
198 J.-A. Micoulaud-Franchi et al. / Psychiatry Research 237 (2016) 196200

(Carey and Seibert, 1993). Item discriminant validity (IDV) was assessed by de- internal consistency reliability was satisfactory, except for item 1
termining whether items correlated better with the factor they were hypothesized
(as the deletion of this item increase the Cronbach's alpha coef-
to represent compared with the other factor (Campbell and Fiske, 1959).
Internal consistency reliability was assessed by Cronbach's alpha coefcient cient). Floor effects ranged from 27.5% to 85.4% and ceiling effects
and was recalculated after items were removed. To conrm consistency, a coef- ranged from 0% to 6.7% (Table 2).
cient of at least 0.7 was expected (Cronbach, 1951). It is recommended that the
delete of any of the 6 items did not increase the Cronbach's alpha coefcient. 3.2.3. External validity
Floor and ceiling effects were reported to assess the distribution of responses.
The rate of oor and ceiling effects was calculated as the proportion of individuals
The correlation between AES and BDI-II scores was not sig-
who obtained the lowest (1) and the highest (5) scores for any of the items. nicant (r (178) 0.12, p 0.11). The mean AES score in women
was 25.75 (SD 3.31) and in men was 25.88 (SD 3.71). No sta-
2.3.3. External validity tistical association was related to gender concerning the AES
To explore external validity, the relation between BDI-II and AES scores was scores (p 0.81).
investigated by computing Pearson's coefcients. No relations between symptoms
of depression and the AES score were hypothesized. Difference in AES scores be-
tween men and women was also investigated by t-test. No signicant difference
between men and women was hypothesized. 4. Discussion

The aim of this study was to translate and validate the French
3. Results version of the AES in order to make it available for evaluating
satisfaction with E-health systems in French-speaking patients.
3.1. Sample characteristics Transcultural validation supported the structural validity of the
instrument but revealed some differences with the original ver-
A total of 178 native French speakers were included. None of sion (Tariman et al., 2011).
the patients reported any difculties in understanding the items of In the original version, principal component factor analysis
the AES. found only one factor with all 6 items with a value of item load
The mean age was 46.51 years old (SD 12.91, range: [1964] equal to or greater than 0.4 (Tariman et al., 2011). With the French
years old), 57.3% (102) were women, the mean number of years of version, two factors were found: the rst factor is related to the
education was 13.36 (SD 3.07), 14.6% (26) were being treated question that evaluates the enjoyability, the helpfulness to de-
with antidepressant treatment, and 19.7% (35) had a diagnosis of scribe his/her own feeling and the satisfaction of the patient with
current MDD according to the psychiatrist's evaluation. Age was the E-health system, while the second factor is related to the
not signicantly different between patients with and without question that evaluates the usability, the understandability and the
MDD (p 0.30). However, there were signicantly more women acceptability of E-health systems. These two factors can be thus
with MDD than men with MDD (p 0.002). The mean BDI-II score labeled according to their constitutive items as Satisfaction for
was 10.73 (SD 12.52) and the mean AES score was 25.81 factor 1 (3 items: 3, 4 and 6) and Usability for factor 2 (3 items: 1,
(SD 3.48). 2 and 5). Interestingly, these two factors refer to the classical
Technology Acceptance Model (TAM). The TAM is an information
3.2. Validity systems theory that models the acceptability for a new technology
(Davis et al., 1989). The TAM posits that acceptability is concerned
3.2.1. Construct validity with the user's subjective evaluation of technological tools in-
The conrmatory factor analysis using LISREL model revealed cluding two factors: perceived usefulness (related to satisfaction)
that the RMSEA was unsatisfactory (RMSEA 0.089). The ex- and perceived ease of use (related to usability) (Davis et al.,
ploratory factor analysis using principal component factoring re- 1989).
vealed two factors: the rst with items 3, 4, and 6, and the second The difference between the construct of the original version
with items 1, 2, and 5. For each factor the value of item loads was and that of the French version may be due to differences with the
greater than 0.4. The overall pattern of item goodness-of-t sta- E-health systems used to validate the AES. In the original version,
tistics (INFIT) for each factor was satisfactory, except for item 1 the AES was used to evaluate a web-based questionnaire (Tariman
(Table 1). et al., 2011). In the French version, it was used to evaluate a virtual
agent (Philip et al., 2014). Two important kinds of parameters are
3.2.2. Internal structural validity controlled to build a virtual agent: a) the quality of each question
The item-internal consistency (IIC) was satisfactory for the two and more generally of the understandability of the verbal con-
factors: each item achieved the 0.40 standard threshold value. The tents; and b) the quality of the interaction involving non-verbal
item discriminant validity (IDV) was satisfactory as the correlation parameters. The latter are not involved in classical web-based
between items with their contributive factor was higher than questionnaires and are specic to virtual agents. Thus, the use of a
items with the other factor. Cronbach's alpha coefcient was 0.70 virtual agent to evaluate the AES probably favored the responses of
and ranged from 0.58 to 0.76 after items were deleted. The overall participants in two directions: the understandability of each

Table 2
Validation of the French version of the AES.

Factor (number of items) Mean (SD)a IICb minmax IDVc minmax Floor (%) Ceiling (%) Alphad INFITeminmax

Factor 1: satisfaction (3) 11.84 (2.53) 0.860.88 0.210.32 20.8 1.1 0.835 0.470.69
Factor 2: usability (3) 13.96 (1.72) 0.520.75 0.090.42 61.2 1.1 0.643 0.81.64
Global score 25.81 (3.48) 13.5 1.1

Standard deviation.
Item internal consistency (item-to-own dimension correlations).
Item discriminant validity (item-to-other dimensions correlations).
Cronbach's alpha.
Rasch statistics.
J.-A. Micoulaud-Franchi et al. / Psychiatry Research 237 (2016) 196200 199

question asked by the virtual agent, and the satisfaction with the version of the AES revealed a two-factor scale that differs from the
clinical interaction of the interview conducted by the virtual agent. original version. Its inclusion in research on E-health systems
Moreover, the understandability of the virtual agent used in this would provide critical information regarding patient acceptability
study was very high (with a proportion of individuals who ob- (i.e., satisfaction and usability) of these new ways of giving and
tained the highest scores for the items of factor usability higher organizing care for chronic disease. Since use of the French lan-
than 75%) and may explain the poor internal consistency of these guage is increasing (Organization International de la francophonie,
items. Further studies are needed to evaluate the specicity of the 2009) and given the increasing prevalence of mental and brain
satisfaction for a broad range of E-health systems and in particular disorders in developed countries (Wykes et al., 2015), the French
for the use of a virtual agent. In line with the Proximal Similarity version of the AES may prove to be of particular interest in this
Model (Campbell, 1986), the difference between the construct of future.
the original version and that of the French version may also be due
to differences in the medical conditions of patients who were
evaluated with the AES and the difference in the symptoms eval- Conict of interest
uated by the E-health tool. In the original version, patients were
suffering from cancer and quality of life was evaluated with the We report no conicts of interest.
E-health tool (Tariman et al., 2011). In the French version, patients
were consulting for psychiatric or sleep problems, and depressive
symptoms were evaluated with the E-health tool. Further studies Acknowledgments
are needed to evaluate the specicity of the satisfaction with
E-health systems in different medical conditions and different Special thanks to Muriel Bacarisse, Cdric Valtat, and Aurlien
kind of symptoms. Boiseau for selecting participants, collecting and monitoring data
Some limitations in the current study have to be considered. and for administrative, technical, and logistic support. We ac-
Firstly, the diagnostic standard for MDD in the present study was knowledge the ANR-PHENOVIRT.
not a structured clinical interview. Instead, MDD diagnosis was
based on expert opinion. This could prove to be a disadvantage
considering the accuracy of the assessment. Secondly, the present References
study with virtual agent was focused on MDD and the psychiatrist
evaluated only MDD criteria. Around 20% of the patients tested in
American Psychiatric Association, 2013. Diagnostic and Statistical Manual of Mental
this sample had a diagnosis of MDD, which is also within the range Disorder, 5th ed. American Psychiatric Association, Washington, DC.
previously described in France (Bromet et al., 2011). Interestingly, Attkisson, C.C., Zwick, R., 1982. The client satisfaction questionnaire. Psychometric
their symptoms of depression were not related to their AES score. properties and correlations with service utilization and psychotherapy out-
come. Eval. Program. Plan. 5, 233237.
These results suggest that the AES can be used with a large range Beck, A.T., Steer, R.A., Ball, R., Ranieri, W., 1996. Comparison of beck depression
of patients, including those with a mental disorder. However, the inventories -IA and -II in psychiatric outpatients. J. Personal. Assess. 67,
generalizability of the AES French version could be limited. Indeed, 588597.
Beck A.T., 1996. Inventaire de Dpression de Beck (2me dition): Manuel du BDI-II.
the validity of the AES in other mental disorders could not be les d. du Centre de psychologie applique (Ed.), Paris.
assured. Thus, the psychometric validities of the AES in patients Bourque, P., Beaudette, D., 1982. Etude psychomtrique du questionnaire de d-
with MDD and with other mental disorders should be evaluated pression de Beck auprs d'un chantillon d'tudiants universitaires franco-
phones. Revue can. des sci. du comportement 14, 211218.
specically in further studies. This is essential in view of the im-
Boyer, L., Baumstarck-Barrau, K., Cano, N., Zendjidjian, X., Belzeaux, R., Limousin, S.,
portance of satisfaction in mental health (Boyer et al., 2009) and Magalon, D., Samuelian, J.C., Lancon, C., Auquier, P., 2009. Assessment of psy-
the perspective of using E-health systems with patients with chiatric inpatient satisfaction: a systematic review of self-reported instruments.
mental disorders (Kimhy et al., 2012; Serre et al., 2015). Thirdly, in Eur. Psychiatry. 24, 540549.
Brislin, R., 1970. Back-translation for cross-cultural research. J. Cross Cult. Psychol. 1,
the context of E-health services, there was no gold-standard 185216.
questionnaire to examine the external validity of the AES. How- Bromet, E., Andrade, L.H., Hwang, I., Sampson, N.A., Alonso, J., de Girolamo, G., de
ever, there are some generic and validated questionnaires, like the Graaf, R., Demyttenaere, K., Hu, C., Iwata, N., Karam, A.N., Kaur, J., Kos-
tyuchenko, S., Lepine, J.P., Levinson, D., Matschinger, H., Mora, M.E., Browne, M.
Client Satisfaction Questionnaire CSQ-8, which is a self-report O., Posada-Villa, J., Viana, M.C., Williams, D.R., Kessler, R.C., 2011. Cross-national
questionnaire constructed to measure satisfaction with health, epidemiology of DSM-IV major depressive episode. BMC Med. 9, 90.
mental health, and human services received by individuals (Att- Campbell, D., 1986. Relabeling internal and external validity for applied social sci-
entists. New Dir. Program Eval. 31, 6777.
kisson and Zwick, 1982). The external validity of the AES with the Campbell, D.T., Fiske, D.W., 1959. Convergent and discriminant validation by the
CSQ-8 should be investigated in further studies. Fourthly, the use multitrait-multimethod matrix. Psychol. Bull. 56, 81105.
of PCA to explore the structure of the AES can be criticized. A CFA Carey, R.G., Seibert, J.H., 1993. A patient survey system to measure quality im-
provement: questionnaire reliability and validity. Med. Care 31, 834845.
is also a relevant statistical method when a theoretical model of
Carlson, L.E., Speca, M., Hagen, N., Taenzer, P., 2001. Computerized quality-of-life
relationships is available. The choice between PCA and CFA can screening in a cancer pain clinic. J. Palliat. Care 17, 4652.
thus be debatable. In this work, as the TAM model was not the Chouvarda, I.G., Goulis, D.G., Lambrinoudaki, I., Maglaveras, N., 2015. Connected
explicitly the theoretical model used by Tariman et al. (2011), we health and integrated care: Toward new models for chronic disease manage-
ment. Maturitas 82, 2227.
considered that our knowledge was insufcient to provide a robust Cleary, P.D., McNeil, B.J., 1988. Patient satisfaction as an indicator of quality care.
underlying structure. Moreover, an exploratory approach was Inquiry 25, 2536.
justied to propose an adaptive structure in the French context. Cronbach, L.J., 1951. Coefcient alpha and the internal consistency of tests. Psy-
chometrika 16, 297334.
Finally, the internal consistency of factor 2 was limited. Indeed, Crow, R., Gage, H., Hampson, S., Hart, J., Kimber, A., Storey, L., Thomas, H., 2002. The
item 1, which explores how easy was the use of the computer measurement of satisfaction with healthcare: implications for practice from a
program, showed ceiling effect, an increase of the internal con- systematic review of the literature. Health Technol. Assess. 6, 1244.
Davis, F., Bagozzi, R., Warshaw, P., 1989. User acceptance of computer technology: a
sistency when deleted, and an INFIT statistic above the satisfactory
comparison of two theoretical models. Manag. Sci. 35, 9821003.
range. Thus, the internal consistency of factor 2 should be in- Epstein, J., Santo, R.M., Guillemin, F., 2015. A review of guidelines for cross-cultural
vestigated in further studies. adaptation of questionnaires could not bring out a consensus. J. Clin. Epidemiol.
In conclusion, the French version of the AES is a psychome- 68, 435441.
Fitzpatrick, R., 1991. Surveys of patients satisfaction: I Important general con-
trically acceptable self-reported questionnaire for evaluating sa- siderations. BMJ 302, 887889.
tisfaction with and the usability of E-health systems. The French Kaiser, H., 1958. The varimax criterion for analytic rotation in factor analysis.
200 J.-A. Micoulaud-Franchi et al. / Psychiatry Research 237 (2016) 196200

Psychometrika 23, 187200. testing of the Acceptability E-scale for web-based patient-reported outcomes in
Kimhy, D., Myin-Germeys, I., Palmier-Claus, J., Swendsen, J., 2012. Mobile assess- cancer care. Appl. Nurs. Res. 24, 5358.
ment guide for research in schizophrenia and severe mental disorders. Schi- Ware Jr., J.E., Davies, A.R., 1983. Behavioral consequences of consumer dissatisfac-
zophr. Bull. 38, 386395. tion with medical care. Eval. Program Plan. 6, 291297.
Mullen, K.H., Berry, D.L., Zierler, B.K., 2004. Computerized symptom and quality-of- Wright, B., Linacre, J., 2001. Winsteps: A Rasch Model Computer Program. MESA
life assessment for patients with cancer part II: acceptability and usability. Press, Chicago.
Oncol. Nurs. Forum 31, E84E89. Wykes, T., Haro, J.M., Belli, S.R., Obradors-Tarrago, C., Arango, C., Ayuso-Mateos, J.L.,
Organisation International de la francophonie, 2009. Le franais, enjeux du XXIe Bitter, I., Brunn, M., Chevreul, K., Demotes-Mainard, J., Elfeddali, I., Evans-Lacko,
sicle. S., Fiorillo, A., Forsman, A.K., Hazo, J.B., Kuepper, R., Knappe, S., Leboyer, M.,
Philip, P., Bioulac, S., Sauteraud, A., Chaufton, C., Olive, J., 2014. Could a virtual Lewis, S.W., Linszen, D., Luciano, M., Maj, M., McDaid, D., Miret, M., Papp, S.,
human be used to explore excessive daytime sleepiness in patients? Presence: Park, A.L., Schumann, G., Thornicroft, G., van der Feltz-Cornelis, C., van Os, J.,
Teleoperators Virtual Environ. 23, 369376. Wahlbeck, K., Walker-Tilley, T., Wittchen, H.U., 2015. Mental health research
Serre, F., Fatseas, M., Swendsen, J., Auriacombe, M., 2015. Ecological momentary priorities for Europe. Lancet Psychiatry.
assessment in the investigation of craving and substance use in daily life: a Zendjidjian, X.Y., Baumstarck, K., Auquier, P., Loundou, A., Lancon, C., Boyer, L.,
systematic review. Drug Alcohol Depend 148, 120. 2014b. Satisfaction of hospitalized psychiatry patients: why should clinicians
Taenzer, P., Bultz, B.D., Carlson, L.E., Speca, M., DeGagne, T., Olson, K., Doll, R., care? Patient Preference Adherence 8, 575583.
Rosberger, Z., 2000. Impact of computerized quality of life screening on phy- Zendjidjian, X.Y., Auquier, P., Lancon, C., Loundou, A., Parola, N., Faugere, M., Boyer,
sician behaviour and patient satisfaction in lung cancer outpatients. Psy- L., 2014a. Determinants of patient satisfaction with hospital health care in
chooncology 9, 203213. psychiatry: results based on the SATISPSY-22 questionnaire. Patient Preference
Tariman, J.D., Berry, D.L., Halpenny, B., Wolpin, S., Schepp, K., 2011. Validation and Adherence 8, 14571464.