See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/272525671

Metrology in health: A pilot study

Article  in  Journal of Physics Conference Series · February 2015

DOI: 10.1088/1742-6596/588/1/012018

CITATIONS READS

2 471

2 authors:

Maria Ferreira Ana Matos

Instituto Português da Qualidade Universidade NOVA de Lisboa
22 PUBLICATIONS   36 CITATIONS    31 PUBLICATIONS   314 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Metrology on Health View project

All content following this page was uploaded by Maria Ferreira on 14 October 2015.

The user has requested enhancement of the downloaded file.

Home Search Collections Journals About Contact us My IOPscience

Metrology in health: a pilot study

This content has been downloaded from IOPscience. Please scroll down to see the full text.

2015 J. Phys.: Conf. Ser. 588 012018

(http://iopscience.iop.org/1742-6596/588/1/012018)

View the table of contents for this issue, or go to the journal homepage for more

Download details:

IP Address: 116.248.168.2
This content was downloaded on 08/09/2015 at 16:39

Please note that terms and conditions apply.

IMEKO IOP Publishing
Journal of Physics: Conference Series 588 (2015) 012018 doi:10.1088/1742-6596/588/1/012018

Metrology in health: a pilot study

M Ferreira1,2 and A Matos2
1
Portuguese Institute for Quality, Rua António Gião, 2, 2829-513 Caparica, Portugal
2
UNIDEMI, Faculdade de Ciencias e Tecnologia, 2829-516 Caparica, Portugal

E-mail: [email protected]

Abstract. The purpose of this paper is to identify and analyze some relevant issues which
arise when the concept of metrological traceability is applied to health care facilities.
Discussion is structured around the results that were obtained through a characterization and
comparative description of the practices applied in 45 different Portuguese health entities.
Following a qualitative exploratory approach, the information collected was the support for the
initial research hypotheses and the development of the questionnaire survey. It was also
applied a quantitative methodology that included a descriptive and inferential statistical
analysis of the experimental data set.

1. Introdution
Measurement, as a field of scientific and technical activity, encompass a set of activities aimed at
determining the quantities and registers a remarkable evolution throughout the centuries The scientific
context that serves these activities is called Metrology which includes a wide-range of activities with a
key role in all sectors of society, being a fundamental basis for sustainable development [1], [2], [3]
[4].
According to international recommendations, every system of measurement must use accurate and
reliable results [5]. Metrological traceability [3] and uncertainty measurement [3] are the main
elements that support this assurance. This means improving a chain of measurements from a single
measurable result in a commonly accepted reference, as well as the uncertainty value associated to this
measuring result [6 ]. Surely due to the inherent risk of life, these principles are strongly applied in
health systems. Measurements and measuring instruments are present in day-to-day life and are
fundamental processes in prevention, diagnosis and treatment of diseases. Measurement results are
vital in the medical field, which requires they are accurate, real and comparable over time, i.e., they
must exhibit repeatability and reproducibility [7]. Nevertheless, one must be aware that errors and
uncertainties of measuring instruments could be the origin of unsuitable clinical procedures, such as a
false diagnosis or inadequate treatments.
Clinical laboratories constitute a good example for the improvements achieved by metrology as
regards traceability in medicine. The Joint Committee for Traceability in Laboratory Medicine
(JCTLM) [8] [9], hosted by the International Bureau of Weights and Measures (BIPM), is the first
international committee focused on clinical metrology and constitutes a platform envisaging the
harmonization and dissemination of good practices within the framework of analytical chemistry [10].
Therefore, under the scope of the European Medical Devices Directives [11], new responsibilities
and technical requirements for the measuring instruments were put in place, for preventing the
existence of barriers as regards the free trade. trade. For one hand, although a wide range of
instruments is covered by the medical devices directive, this sphere of regulation allows each Member
State considering additional measures to protect public health and citizens. On the other hand, after

Content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence. Any further distribution
of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.
Published under licence by IOP Publishing Ltd 1
IMEKO IOP Publishing
Journal of Physics: Conference Series 588 (2015) 012018 doi:10.1088/1742-6596/588/1/012018

placing them in the market and putting them into service, no further regulated control (according to
EU policy) exists for those medical devices [12].
As the awareness of the benefits for having traceable measurement results increases, some
researchers and technical experts (in metrology and medicine), have been increasing their efforts and
interest for developing research within this scope. In recent years several pieces of research were
published focusing some relevant measuring instruments used by health professionals. As regards
specific diseases, some authors highlighted the key role of metrological traceability of
sphygmomanometers as a tool for reducing health care costs and improving patient’s care [13], [14],
[15]. Likewise, other researchers pointed out the traceability of clinical temperature’s measurement
devices, such as thermometers [16], [17], [18]. In what concerns the evaluation of some ophthalmic
diseases, the clinical metrology of tonometers' utilization has also been under metrological research
[19], [20], [21]. Work has been developed regarding pulmonary and respiratory diseases, proving how
important reproducibility is and its impact on diagnosis [22], [23].
Regardless the aforementioned work, much remains to be done. In fact, the various actors with
jurisdiction in these subjects, and in particular in some areas of activity, didn’t put in place enough
actions -packed for assuring metrological traceability in health measurements. Furthermore, the
growing concern with issues related to quality in health care made emerge new areas of activities in
this important field of society. Moreover, it is also considered that the interoperability between these
three disciplines is not contextualized within the current European regulatory framework. The lack of
information regarding the perception of metrology by health professionals is a topic requiring
evaluation.

2. Methodology
Taking into account the literature review, as well as the current framework of national health care
facilities, the applied methodology aimed to identify the main indicators of health professionals’
practices with their working instruments for measurement.
The Portuguese hospitals were the target of this work research, which was developed in 2011 and
included private and public hospitals (national health system). A survey was applied to the hospitals
included in the last database from Portuguese Directorate-General of Health (DGS) that was available
at the time [24]. However, at the time, private hospitals were not available in the aforementioned
national database. Hence, the universe of private hospitals was identified by regional random search.
According to that information and based on the real context of observational analysis, a
multimethod design [25] approach was applied: i) inductive and deductive reasoning methods [26], ii)
qualitative and quantitative research methodology [27].

Survey characterization
The issues in survey formulation were designed and developed based on the analysis of the National
and European metrological requirements in health care facilities. Therefore, the research hypotheses
and the development of the measurement instrument (survey) were performed through an exploratory
qualitative approach without a previous similar validated survey.
From May to September 2011, 122 questionnaires were sent by e-mail to Portuguese hospitals.
From those, 99 were addressed to public hospitals, including islands, and 35 responses were received.
The remaining 23 were sent to private hospitals, and 9 responses were received. All surveys were sent
by e-mail with a follow-up procedure and received through the same way. Thus, considering a
stratified sample, a response rate of 35,4 % was obtained for public hospitals and 39,1 % for private
hospitals, leading to an overall response rate of 36%, that were collected and analyzed.
A 'tree' questionnaire was developed, in which the choice of certain issues is determined by the
answers in previous questions. The ten questions included in the developed survey were distributed
between open and closed questions, including dichotomous (yes/no) and ordinal scales (Likert type).
Envisaging the assurance of survey's validation, a pretesting was conducted [28]. Apart from slight
changes, the questions were maintained after the pretesting.

2
IMEKO IOP Publishing
Journal of Physics: Conference Series 588 (2015) 012018 doi:10.1088/1742-6596/588/1/012018

The questionnaire was designed to characterize information regarding the following issues:
a) Relationship between qualification systems implemented by health care facilities and associated
metrological practices;
b) Metrological requirements that could influence the acquisition of medical instruments;
c) Budget for medical equipment;
d) Levels of literacy in health's metrology.

In a first stage a univariate analysis was adopted for characterizing the trend of responses in each
scale item. Descriptive statistics were utilized for computing the mean, standard deviation, minimum,
maximum and frequency response. The second stage focused the correlation between variables,
including a bivariate statistical analysis from the dataset. Finally, in the third stage, it were utilized
statistical inference, nonparametric tests, correspondence analysis and factor analysis.

3. Results
Data analysis was performed using the Statistical Package for Social Sciences (SPSS) version 19. A
value for Cronbach alpha of 0,74 was achieved, thus making acceptable the internal consistency [29].
Once more, it is important to stress that the questionnaire constitutes a new survey model, entirely
developed for this piece of research.

Quality systems of survey respondents

According to data analysis, 53,3 % of hospitals had a partial qualification, 25,0 % were accredited and
22,7 % of respondents referred that the hospitals were certified as a whole. In addition, the results’
overview regarding the quality management system of hospitals shows that more than 50,0 % of
public hospitals are qualified according to international standards. All responses from private hospitals
revealed a certification by ISO 9001[30]. Both typologies of hospitals prefer national certifiers
entities.

Procedure for acquiring equipment’s vs metrology

The importance given to metrological requirements within the procedures for equipment’s acquisition
was assessed. Exploratory Factorial Analysis (EFA) was utilized with Varimax rotation. Table 1
shows the grouped variables by factor: i) the maintenance conditions and the historical of the
equipment were named as post-sales factor (1st EFA factor); ii) the 2nd EFA factor was named as
legal requirements and it includes the compliance of requirements by producers requisites as well as
the hospitals' requirements for the notified bodies; iii) the price and supplier trust were grouped in the
3rd factor (pre-sales factor). The high value obtained for communalities shows that retained factors
are appropriate for describing the latent correlation structure. The measure of sampling adequacy was
validated by the result from KMO (Kaiser-Meyer-Olkin) and the Bartlett test [29].

Table nº 1: Factor Structure of Exploratory Factorial Analyses

Factor
Variable 1 2 3 Communality
Price -0,211 0,004 0,828 0,730
Supplier trust 0,286 0,054 0,752 0,650
Requirements of ISO 13485 -0,074 0,892 0,136 0,819
Maintenance conditions 0,801 0,265 0,131 0,729
Historical of the equipment 0,770 -0,050 -0,048 0,597
Requirements of the Notified Body 0,440 0,708 -0,129 0,711
Eigenvalues 1,926 1,302 1,009
Variance (%) 32,1 21,7 16,8

3
IMEKO IOP Publishing
Journal of Physics: Conference Series 588 (2015) 012018 doi:10.1088/1742-6596/588/1/012018

Through the utilization of cross analyses, the maintenance requirement was considered the most
relevant for private hospitals (Figure 1), whereas for the public hospitals (Figure 2), this issue is less
important than the supplier's confidence (59,0 %). For these hospitals, price was the most important
requirement (51,0 %) which constitutes a strong restriction for the enforcement of metrological
traceability.

Private Hospitals

Notified body 22% 56% 22%

2
Historical 33% 67%
3
Maintenance 11% 89%
4
ISO 13485 11% 33% 44% 11%

5
Trust 11% 56% 33%

Price 11% 78% 11%

Figure nº 1: Indicators from private hospitals

Official Hospitals

Notified body 6% 18% 38% 26% 12% 1

Historical 21% 29% 50% 2

Maintenance 6% 57% 37% 3

ISO 13485 3% 11% 31% 31%

4
23%

5
Trust 15% 59% 26%

Price 9% 41% 51%

Figure nº 2: Indicators from official hospitals

Metrological documentation that is included in the terms of reference (contract document) was
other aspect that was evaluated on the acquisition phase. Multiple Correspondence Analysis (MCA)
was utilized [29], with engeinvalue 1 as a retention criteria for components. According to values
shown in Table 2, orthogonal components explain 74,0 % of data variability. One component (first
dimension) was represented by the verification certificate and the measures report. The calibration
certificate and the conformity report have different scores, representing the 2nd and 3rd dimension.
The 1st dimension is influenced by respondents who have no preference for calibration certificates,
which are replaced by verification certificates and test reports. Opposite reasoning is applied to the
variables in the 2nd dimension. The analysis for the 3rd dimension indicates that respondents who
receive evaluation reports are influenced by receiving test reports. The chi-square χ2 (α = 0,05) was

4
IMEKO IOP Publishing
Journal of Physics: Conference Series 588 (2015) 012018 doi:10.1088/1742-6596/588/1/012018

utilized for analyzing the relation between those variables and the hospital’s typology (public vs
private). Since the null hypothesis H0 is rejected, one concludes that the acquisition of medical
measuring instruments with conformity report is dependent of hospital typology (Table nº 2). This
situation was also assessed using the odds ratio θ evaluation, whose values show that the choice of the
conformity report depends on the type of hospital, and this probability is more pronounced in private
hospitals (θ1 =0,650; θ2 =8,000 ).
Table 2: Survey MCA
Principal Components
Variables (dimensions)
1 2 3 χ2 p-value

calibration certificate 0,006 0,706 0,088 0,754 0,385

verification certificate 0,523 0,023 0,075 0,937 0,333
test report 0,582 0,000 0,090 0,075 0,784
conformity report (product) 0,287 0,026 0,498 6,929 0,008
no certificate 0,003 0,461 0,362 0,024 0,877
Eigenvalue 1,402 1,216 1,113
Variance (%) 28,0 24,3 22,3

Relevance of the traceability 0,315 0,339 0,000

of medical instruments
Metrological Legislation 0,644 0,007 0,542
External calibrations 0,237 0,306 0,000
Internal calibrations 0,032 0,497 0,049
Metrological knowledge for 0,477 0,642
measuring instruments
Interpretation of certificates 0,923 -0,057
vs tests reports
Relation adverse event vs error 0,924 -0,068
of instrument
Relation error of instrument 0,226 -0,842
vs indication results
Eigenvalue 1,198 1,129
Variânce (%) 49,6 28,3

Preventive maintenance 0,544 0,973

Curative maintenance 0,847 0,022
Calibrations 0,928 -0,306
Verifications 0,938 -,0282
Eigenvalue 2,755 1,121
α de Cronbach 0,849 0,144

5
IMEKO IOP Publishing
Journal of Physics: Conference Series 588 (2015) 012018 doi:10.1088/1742-6596/588/1/012018

It was also analyzed the relation between hospitals that acquire equipments with calibration
certificates and the related relevance as regards i) metrological knowledge and ii) technical
competence for certificates’ analyses. The opinion of relevant respondents pointed out that hospitals
that purchase certified equipment (calibrated measuring instruments) consider relevant their certified
interpretation (50,0 %). Furthermore, 46,0 % expressed their interest in metrological's knowledge
applied on clinical practice.

Metrology vs risk management vs patient safety

Another issue that was evaluated focuses the key role of metrology in the risk management and patient
safety. The MCA methodology (Table 2) was adopted as statistical approach . According to those
values, it was possible to characterize the 1st dimension through the association between the needs to
interpret the metrological certificates of equipment and the reflection of this interpretation on the
errors provided by the measuring instruments. These errors can be widespread by the diagnostics and
therapeutic procedures. Extrapolation of this interpretation for clinical practice, whose influence on
adverse effects can be significant, is also present in this dimension.
Within this framework, the metrological knowledge that health professionals may acquire
regarding their measuring instruments is very important and can influence the evaluation assigned to
measurement's error as well as the outcomes. The high score value of 1st dimension for the
interpretation of certificates vs tests reports and relation adverse event vs error of instrument show a
significant concerns revolve around patient safety. Based upon these results, we sustain that the
assessment of the risk associated to the utilization of medical equipment and patient safety might be
considered exogenous latent variables, with special relevance in health metrology.

Budget for medical equipment’s with measuring functions

It was also assessed the budget outcomes from hospitals that are applied to metrological operations
and maintenance, notably preventive and curative maintenance, calibrations and verifications of
measuring instruments. Having this in mind, the categorical variables were distributed by an interval
range, from less than 10 % to more than 30 % of hospital’s overall budget allocation. According with
these results, curative maintenance is the operation selected by 13,6 % of hospitals with a budget more
than 30,0 % whereas 75 % of hospitals allocated less than 10% of such overall budget.
It is worth mentioning that the number of medical equipments with measuring functions in the
respondents' hospitals was above 1100 units. On the other hand, only 2,3 % of respondents reported
more than fourty calibrated instruments by calibration period. Therefore, one can assume that the
results obtained for the budget that is applied by hospitals represents a worrying indicator and a poor
predictor for the metrology in the health services.
According the MCA analyses for such dataset (Table 2), the curative maintenance, the calibrations
and verifications are included in the same principal component, being all of them close to saturation
value. The preventive maintenance has the highest score for the second component. The value of
Cronbach α shows a good internal consistence (α = 0,849).

Metrological factors as a transversal tool in health care facilities

This issue is clearly cross-sectional, which led to its inclusion in several questions. For these purposes,
the following variables were evaluated: i) Relevance of traceability for medical instruments, ii)
Metrological Legislation, iii) External calibrations, iiii) Internal calibrations.
As the responses presented lack of harmonization, a binomial test was applied to the
aforementioned variables. The null hypothesis was rejected as regards metrological legislation, which
reveals that the proportion of hospitals that considers sufficient the present legislation in the field of
traceability of medical devices, is statistically similar to the proportion of those who do not consider
this factor as relevant. A MCA was applied for validating this conclusion (Table 2). Two dimensions
were retained, whose values lead to designating dimension 1 as the component focused on the law
(metrological traceability) and the second dimension as the traceability component. Among the

6
IMEKO IOP Publishing
Journal of Physics: Conference Series 588 (2015) 012018 doi:10.1088/1742-6596/588/1/012018

respondents that classify the traceability as relevant, 93.2% make use of external calibration. Among
the hospitals that perform internal calibrations, 63,6 % apply this procedure to evaluate the conditions
of metrological instruments, without support of external calibrations (no evidence of a traceability
reliably). The two approaches (external and internal calibrations) are practiced by only 34,1 % of
respondents. The Odds Ratio (θ) was assessed for studying the relationship between internal
calibration and hospital's typology. It was concluded that the probability of private hospitals to
practice internal calibrations is 1,5 higher than the public hospitals and that there is no dependency
relationship between both variables.

4. Discussion
This study showed that a large majority of respondents were representing organizations with
recognized qualifications, which constitutes a valuable piece of information for establishing their
profile.
Regarding the metrological issues, notably for public hospitals, the price of the product is the
prevailing requirement. However, it was also verified that there is a significant correlation between
trust in the supplier and the price of product to be delivered. This suggests that the score of the
variable price may be undervalued when remaining confidence in the entity provider.. Nevertheless, as
the economic downturn has raised major concerns for all sectors of activity, this component can
represent an even greater concern in acquisition procedures. Thereby, the criterion of price is in fact
the most important requirement to acquire equipment, which deserves a proper reflection.

Usually, the metrological documentation accompanying the equipment during its acquisition phase
does is not dependent neither of the type of hospitals nor of systems of quality management, which
reflects the reduced concern regarding the compliance to requirements of metrological equipment.
Hence, despite the statistical significance for matters regarding the maintenance and taking into
account the analysis performed on the metrological documentation mentioned on the previous items, it
is suggested that the protocol established by hospitals to acquire instruments measurement does not
reflect metrological concerns.
For both types of hospitals, the relationship between the adverse event and the instrument was
fairly valued, being an indicator of the perception of health professionals regarding the dimension of
risk management vs metrology.
In addition, entities that acquire equipment with calibration certificates are those who consider that
its interpretation is important, which requires an improved metrological knowledge. Hence, the
obtained results suggest that healthcare professionals have the perception that the metrological
performance of equipment is important and that can influence the outcome of the measurements in
their clinical practice. However, in most institutions, the metrological traceability is still concealed in
maintenance operations, thus contributing for the low level of best practices of metrology in medicine.
The research results also stressed that a greater confidence in supplier is proportional to a lower
overall budget for maintenance operations. The interpretation could be made under the framework of
health and metrological literacy, encompassing a weak frontier between maintenance and metrology
operations.
Therefore, considering the ratio of total number of equipments to the number of equipments
subjected to calibration’s operations (previous issue), one can assume that Portuguese hospitals has a
reduced number of instruments that are submited to procedures able to assure the quality control of
obtained results.
The hospitals tend to realize external calibrations, showing no tendency to perform internal
calibrations. Those entities that perform it internally are predominantly private, and perform those
operation for assuring the conditions of their metrological instruments. These entities have technicians
that received specific training for this field of work.
The knowledge and opinions of respondents as regards metrological legislation proved to be
incipient. It was also concluded that a trend for the application of metrological practices prevails in

7
IMEKO IOP Publishing
Journal of Physics: Conference Series 588 (2015) 012018 doi:10.1088/1742-6596/588/1/012018

those entities that revealed more knowledge and awareness for metrology issues, emphasizing the
implementation of procedures that promote the metrological traceability of measuring instruments.

5. Conclusion
Taking into account the obtained results, one concludes that metrological traceability is a subject that
is absent from day-to-day life of healthcare professionals, although it is known by them.
Some factors contributing for this perception were identified, which might enrich the scientific
knowledge regarding the fields of health and metrology.
Considering the quality in health care facilities, this piece of research aims to contribute for
introducing the metrology subject in hospitals’ framework.
Further work, focusing on the impact of measurements on Gross National Product (GDP) of
globalized economies, particularly regarding health systems, is clearly necessary. Hence, further
informations allowing updates and improvement developments on this important subject are obviously
wellcome.
At the European level, a continuous international collaboration of experts is needed for promoting
comprehensive Metrology within the framework of quality in health care. This is a major challenge.

References
[1] R White 2011 The meaning of measurement in metrology Accreditation and Quality Assurance
16(1):31-41
[2] D Kind and H Lubbig 2003 Metrology–the present meaning of a historical term Metrology 40:255-
7
[3] JCGM 200. 2012 International Vocabulary of Metrology Basic and General Concepts and
Associated Terms
[4] G Hratch, R Semerjian and J Watters 2000 Impact of measurement and standards infrastructure on
the national economy and international trade Measurement 27:179-96
[5] M Golze 2003 Why do we need traceability and uncertainty evaluation of measurement and test
results? Accreditation Quality Assurance 8:539-540
[6] E Desimoni and B Brunetti 2006 Considering uncertainty of measurement when assessing
compliance or non-compliance with reference values given in compositional specifications and
statutory limits: a proposal Accreditation and Quality Assurance 11(7):363-6
[7] P Ashford 2010 Traceability. Cell Tissue Bank 11:329–33
[8] W Hubert and L Vesper 2009 Traceability in laboratory medicine. Clinical Chemistry 55(6):
1067-75
[9] D Armbruster and R Richard 2007 The joint committee for traceability in laboratory medicine
(jctlm): a global approach to promote the standardisation of clinical laboratory test Results
Clinical Biochemestry Review 28(3): 105–14
[10] http://www.bipm.org/en/committees/jc/jctlm/
[11] http://ec.europa.eu/health/medical-devices/regulatory-framework/index_en.htm
[12] M Ferreira 2011The role of metrology in the field of medical devices International Journal of
Metrolology and Quality Engineering. 2:135–40
[13] C A'Court, R Stevens and R Sanders 2011 Type and accuracy of sphygmomanometers in primary
care: a cross-sectional observational study British Journal of General Practice 61(590):598-603
[14] E O’Brien, and A Neil 2007 Validation and reliability of blood pressure monitors Clínical
Hypertension and Vascular Diseases:Blood Pressure Monitoring in Cardiovascular Medicine
and Therapeutics pp 97-132. ISBN: 978-1-59259-978-3.
[15] M Turner, C Speechly and N Bignell 2007 Sphygmomanometer calibration why, how and how
often? Australian Family Physician 36(10):834-8
[16] I Juntaro 2008 Improving the realibility of temperature measurements taken with clínical infrared
ear thermometers Synthesiology-English Edition. I: 47-58

8
 IMEKO IOP Publishing
Journal of Physics: Conference Series 588 (2015) 012018 doi:10.1088/1742-6596/588/1/012018

[17] R Simpson, G Machin, H McEvoy and H., Rusby, H. (2006) Traceability and calibration in
temperature measurement: a clinical necessity. Journal of Medical Engineering & Technology
30(4):212-217
[18] W Chung and C Chen (2010 Evaluation of performance and uncertainty of infrared tympanic
thermometers Sensors (Basel). 10(4):3073-89
[19] S Sandhu, S Chattopadhyay, and G. Amariotakis 2009 The accuracy of continued clínical use of
goldmann applanation tonometers with known calibration errors Ophthalmology 116(1):9-13
[ 20] K Ogbuehi and T Almubrad 2008 Accuracy and reliability of the Keeler Pulsair EasyEye non-
contact tonometer Optometry and vision science 85(1):61-6
[21] S Choudhari, R George and M Baskaran 2009 Measurement of Goldmann applanation tonometer
calibration error Ophthalmology 116(1):3-8
[22] R Jensen, J Teeter, R England, H White and E Pickering 2007 Instrument accuracy and
reprodutibility in measurements of pulmonary function CHEST 132:388-95
[23] R Miller, J Hankinson and J Brusasco 2005 Standardisation of spirometry European Respiratory
Journal 26: 319–23
[24] DGS (2008 Elementos estatísticos: informação geral Direcção de Serviços de Epidemiologia e
Estatísticas de Saúde. Divisão de Estatísticas de Saúde. Lisboa. ISSN 0872-1114.
[25] A Tashakkori and C Teddlie Handbook of Mixed Methods in Social & Behavioral Research
Authors editors 2002 ISBN-13: 978-0761920731
[26] W Trochim and J Donnelly The Research Methods Knowledge Base 2006 Edition: 3ISBN-13:
978-1592602919
[27] J Brannen 2005 Mixing methods: the entry of qualitative and quantitative approaches into the
research process International Journal of Social Research Methodology 8(3):173-184
[28] R Ghiglione and B Matalon 1997 O Inquérito: Teoria e Prática (3ª ed.). Oeiras: Celta Editora
[29] J Maroco 2007 Statistical analysis using SPSS (3rd ed.). Walker Editions. ISBN: 978-972-618-
452-2.
[30] EN ISO 9001:2000 Quality management systems-requirements

View publication stats