BMJ  2006;333:19 (1 July), doi:10.1136/bmj.38874.499167.7C (published 22 June 2006)

Research

Patients' own assessments of quality of primary care compared with objective records based measures of technical quality of care: cross sectional study

¹ Department of Health and Human Sciences, University of Essex, Colchester CO4 3SQ, ² Public Health and Policy Research Unit, Queen Mary, University of London, London E1 2AT, ³ London School of Hygiene and Tropical Medicine, London WC1E 7HT, ⁴ Centre for Behavioural Aspects of Health and Disease, Glasgow Caledonian University, Glasgow G4 OBA

Abstract

Objective To investigate the relation between older patients' assessments of the quality of their primary care and measures of good clinical practice on the basis of data from administrative and clinical records.

Design Cross sectional population based study using the general practice assessment survey.

Setting 18 general practices in the Basildon primary care trust area, south east England.

Participants 3487 people aged 65 or more.

Main outcome measures Correlations between mean practice scores on the general practice assessment survey and three evidence based measures on survey of case records (monitoring for, and control of, hypertension, and vaccination against influenza).

Results 76% of people (3487/4563) responded to the general practice assessment survey. Correlations between patient assessed survey scores for technical quality and the objective records based measures of good clinical practice were 0.22 (95% confidence interval -0.28 to 0.62) for hypertension monitored, 0.30 (-0.19 to 0.67) for hypertension controlled, and -0.05 (-0.50 to 0.43) for influenza vaccination.

Conclusions Older patients' assessments are not a sufficient basis for assessing the technical quality of their primary care. For an overall assessment both patient based and records based measures are required.

Introduction

The complex, multidimensional nature of the quality of health care has long been recognised.^1-3 Since the outcomes movement of the 1980s much effort has gone into developing rigorous methods of assessment. Meanwhile the growth of the consumer movement has led to a view of patients as being potentially the best judges of the quality of their care.^{4 5} This is implicit in the introduction of the agenda about choice in the UK NHS.^{6 7}

Research in the United States suggests that patient reports can be used to identify health plans that offer care of higher clinical quality.⁸ The general practice assessment survey is a patient questionnaire developed in the United States and adapted for use in the United Kingdom.^{9 10} It has been tested in a variety of settings. More recently, the general practice assessment survey has been modified to the general practice assessment questionnaire (www.gpaq.info/). We used the general practice assessment survey to test whether older patients' assessments of the technical quality of their care in general practice were related to evidence based good clinical practice as indicated by data from medical records.

Participants and methods

We invited 23 general practices in Basildon to participate in the study. Sample size calculations suggested that 19 practices would allow for the detection of a correlation coefficient of 0.60 with 80% power at the 5% significance level, and 17 practices, a value of 0.63 or more.¹¹ (In the general practice assessment survey validation study correlations between survey assessments of technical care and four of the other domains were 0.63 or more.¹⁰) The sampling frame consisted of all patients of participating practices aged 65 or more, registered on 1 September 2000. We excluded patients identified by their doctor as too ill to participate.

Patient assessment of quality
The general practice assessment survey covers nine domains of patient assessed quality, including quality of care provided by practice nursing or reception staff, satisfaction with practice premises, and the technical quality of care. Each domain includes several items, which are weighted to provide an overall index. The technical quality domain includes items on medical knowledge, thoroughness of physical examination, arrangement of tests when needed, making the right diagnosis, and prescribing the right treatment. The general practice assessment survey also includes personal information and indicators of socioeconomic status, including car ownership and housing tenure. We used a postal version of the general practice assessment survey. The questionnaires were bar coded and anonymised. Previous studies suggested response rates of around 66% in the general population.¹⁰ Questionnaires were sent to 300 randomly selected people aged 65 years or more registered with each practice. For practices with fewer than 300 such people, we selected all registered people. After two weeks we sent non-respondents another copy of the questionnaire.

Record based measures of technical quality
We chose three indicators of the technical quality of clinical care using the following criteria: clinical guidelines were available on the care that should be provided to older people, based on evidence of benefit; adherence to guidelines could be determined from patient records; and the conditions concerned were sufficiently common for differences to be detectable between practices in adherence to guidelines.¹²

Two indicators were based on adherence to the British Hypertension Society guidelines.¹³ We extracted medical records with sampling fractions dependent on estimated numbers of patients aged 65-79 with hypertension: 50% for practices estimated to have more than 400 such patients; 75% for practices with 250-399; and 100% for all other practices. The sample size for this part of the study was 5473. We developed an algorithm for data extraction. Two trained research nurses established whether blood pressure had been measured within the past five years (hypertension monitored) and whether hypertension was controlled to the standards of the British Hypertension Society (hypertension controlled).

Both nurses audited one practice as part of their training, and interobserver agreement was estimated. One hundred per cent matches were found for 160 of 192 records audited ( = 0.89). All except one disagreement resulted from illegible handwriting. As a result one nurse ascertained hypertension measures for seven practices where the research nurses were confident about the doctors' (mainly computerised) record keeping. Both nurses undertook assessment for the 11 practices where records were mainly hand written. When the diagnosis of hypertension was uncertain, cases were discussed and agreement was reached.

The third indicator of technical quality of care was coverage of influenza vaccination. At the time of the survey the guideline recommended influenza vaccination for all patients aged 75 years or more.¹⁴ The research nurses extracted data on the vaccination status of patients aged 75 years or more (4961 people) in each of the practices using case notes, computer records, or practice vaccination registers. They recorded vaccination status on a structured assessment form.

Data analysis
We used prescribed scoring methods to estimate mean general practice assessment survey scores for each domain in each practice.^{9 10} We treated data from each practice as if sampled from infinite populations. Analysis of variance was used to assess variation in domain scores between and within practices.

Earlier studies suggested that general practice assessment survey scores were related to age and social class but not sex.¹⁵ We constructed four socioeconomic groups from two questionnaire variables as follows: access to car, owns or is buying home; access to car, renting home; no access to car, owns or is buying home; no access to car, renting home. This ordinal scale indicator has been found to correlate with reported incomes of elderly people in data from the 1998 health survey for England.¹⁶ For each domain we derived a regression equation, with, as independent variables, five age groups, four socioeconomic groups, and sex. We used these equations to produce practice scores adjusted for age, socioeconomic status, and sex.

We estimated the proportions of participants in the relevant age groups in each practice who had had their blood pressure monitored and influenza vaccination. For those with a diagnosis of hypertension, we also estimated the proportions whose blood pressure was controlled.

We produced two matrices of correlation coefficients between general practice assessment survey domain scores and records based measures of technical quality. One was unweighted. For the other we used weights inversely proportional to the variance of the estimated practice mean to account for the varying survey sample sizes. We used the method described by Bland to estimate confidence intervals for the coefficients.¹¹ Data manipulation was carried out using Excel, and statistical analysis was done using Stata 8.1.

Results

Nineteen of 23 practices agreed to participate. These varied in size, from eight single handed practices to two with five doctors. The numbers of people aged 65 or more in the practices ranged from 155 to 1695. The smallest practice had only 39 such patients and was excluded from further analysis.

The demographic structure of the practices was similar to that of the rest of England (table 1). In total, 3487 out of a possible 4563 patients in the 18 included practices responded to the general practice assessment survey (76% response rate). Response rates were lower in those aged 85 years or more (60% for those more than 85; 78% for those less than 85; ² =1, df = 62.2, P < 0.0001) but were not related to sex. Response rates by practice varied from 67% to 82% (² = 17, df = 41.1, P = 0.001).

View this table: [in this window] [in a new window]   Table 1 Respondents to general practice assessment survey questionnaire by age and sex compared with general population in England

 

Respondents were of lower socioeconomic status than the population aged 65 or more in the health survey for England.¹⁶ The proportion of respondents who owned or were buying their home and had access to a car was 37.7% compared with 47.8% in the health survey for England, and the proportion who lived in rented or part rented accommodation with no access to a car was 28.5% compared with 22.1% in the health survey for England. Data on satisfaction with practice nursing were missing for 35% of respondents (table 2). Several patients commented that they had not had sufficient contact with the nurse to be able to judge, and one practice had no nurse.

View this table: [in this window] [in a new window]   Table 2 General practice assessment survey results for each domain

 

Case notes were retrieved for 97.5% (n = 5336) of the 5473 patients eligible for hypertension monitoring. The number per practice ranged from 90 to 602. Overall, 4332 (81.2%) of patients had had their blood pressure measured within the past five years, with practice rates ranging from 51% to 95%. In total, 2166 people in the 18 practices had a diagnosis of hypertension (number per practice ranged from 24 to 232), and 1473 (68.0%) of these had a systolic blood pressure of less than 160 mm Hg (values per practice ranged from 39% to 79%). Overall, 94.4% (n = 4683) of the 4961 records sampled for investigation of influenza vaccination rates were retrieved (number per practice ranged from 66 to 753); 2282 (46.0%) had been vaccinated (proportions per practice ranged from 16% to 63%).

Adjustment of mean general practice assessment survey scores for age, sex, and socioeconomic status made little difference. The largest change in score for any of the 162 practice and domain combinations was less than 1%, with 97% of the changes less than 0.5% and 61% less than 0.2%. Unadjusted scores were therefore used in the analyses presented here. Table 2 shows mean general practice assessment survey scores, standard deviations, and ranges for each of the domains across practices. With a possible range of scores for each domain of 0 to100, overall mean scores ranged from 61.4 for doctors' knowledge to 76.9 for satisfaction with receptionists. Variation between practices for most domains was broadly similar, except for satisfaction with nursing, which was relatively low.

The figure gives scatter plots for general practice assessment survey scores for technical care against each of the records based indicators. Each circle represents a practice. One practice's technical quality score seems to be an outlier.

View larger version (26K): [in this window] [in a new window]   Scatter plots for general practice assessment survey scores for technical quality compared with three records based measures. Size of marker indicates confidence interval around survey score estimate

 

Table 3 gives weighted correlation coefficients with 95% confidence intervals for 17 practices, excluding this outlier. Strong and significant correlations were found between general practice assessment survey technical quality and three other general practice assessment survey domain scores: communication skills of doctors (0.85, 95% confidence interval 0.64 to 0.94), interpersonal skills of doctors (0.88, 0.69 to 0.95), and trust in doctors (0.87, 0.69 to 0.95). The correlations between technical quality and the records based measures were, however, much weaker and not significant (technical care and hypertension monitored 0.22, -0.28 to 0.62; technical care and hypertension controlled 0.30, -0.19 to 0.67; and technical care and influenza vaccination -0.05, -0.50 to 0.43). When the outlier was included, coefficients changed little (0.22, 0.30, -0.05, 0.52, -0.08, and 0.27 became 0.36, 0.23, -0.32, 0.51, -0.19, and 0.23, respectively) and none became significant.

Influenza vaccination rate was negatively related to continuity of care and access: -0.44, -0.75 to 0.03 and -0.46 -0.76 to 0.01, respectively. Some evidence was found for a correlation between monitoring and control of hypertension (0.52, 0.07 to 0.79; P = 0.028). However, the correlations between influenza vaccination rates and the measures of hypertension care were weak: hypertension monitored -0.08 (-0.53 to 0.40) and hypertension controlled 0.27 (-0.22 to 0.66).

Discussion

In this survey, older patients in primary care did not distinguish between technical quality of care and other aspects of doctor quality. We found evidence of variation between practices in all our measures of good clinical practice, but weak correlations between the technical care measure on the general practice assessment survey and three records based measures (hypertension monitored, hypertension controlled, influenza vaccination), and also between the records based measures themselves. Strong correlations were, however, shown between patients' own assessments of technical care using the general practice assessment survey and their assessments of other characteristics of their doctors, such as interpersonal skills, trust, and communication.

View this table: [in this window] [in a new window]   Pearson's correlation coefficients (95% confidence intervals) for each domain of general practice assessment survey, with general practice assessment survey "technical care" and three records based measures, unadjusted, but weighted by number sampled in each practice

 

The general practice assessment survey has been extensively tested for validity and reliability.^{9 10} Response rates in our study were good except in the very oldest age group (85 years or more), and general practice assessment survey scores are close to (within 5%) those found previously in the United Kingdom for patients from the full range of ages.¹⁰ Our three records based measures were based on explicit criteria, and we believe that they were reliably ascertained. Basildon is a medium sized town in south east England with fairly typical demography, and although the population has below average home ownership and access to a car, there are few pockets of serious deprivation. Recruitment of practices to the study was good. Therefore our findings are likely to be generalisable to other parts of the United Kingdom and potentially more widely.

One possible explanation for our results is that patients are right; technical quality of care offered by doctors in primary care may be closely related to communication skills, interpersonal skills, and trustworthiness and it is our records based indicators that are misleading. If so, patient assessments would be sufficient. A parallel qualitative study of patients who had been asked to complete the general practice assessment survey, however, found that "relatively few patients had enough knowledge about their own particular illnesses or about possible alternative treatments to make informed judgements."¹⁷ Also, in a study of 120 practices in England and Scotland,¹⁸ a practice performance index based on 16 subindicators from routine NHS data covering prescribing, prevention, and external quality was not significantly correlated with a questionnaire based patient enablement index. Alazri and Neal¹⁹ found a significant correlation of -0.27 between individual overall general practice assessment survey scores and HbA_1c levels in 106 patients (mean age 68) with type 2 diabetes. However, the correlation with the technical quality element of the score was, at -0.126, one of the lowest. These results are hard to reconcile with the proposition that patient questionnaires are all that is necessary for an assessment of the quality of clinical practice.

It seems more likely that when patients are asked about technical quality they base their judgments on those aspects of care that they feel better able to judge. These may be only weakly related to recognised standards of good clinical practice. The question then is how to construct a valid instrument for assessing clinical quality. We used only three measures. It is reassuring that the two related to hypertension were correlated with each other but a matter of concern that they were not correlated to influenza vaccination rates. An approach that covered more aspects of care would be more valid in principle, but a more comprehensive approach might possibly involve more controversial components and problems with collecting and interpreting data.

In an era of strong advocacy for quality of health care driven by choice and "assessment and accountability,"²⁰ valid and reliable indicators are needed. Our findings suggest that in general, older patients do not distinguish between technical quality and other aspects of doctor quality and that the domain of technical quality of care itself may have many subdimensions. We conclude that the evidence for relying solely on patient assessed measures of quality of primary care for older people is weak and that a broader range of measurements is needed.²¹ The challenge for schemes such as the UK Quality and Outcomes Framework²² will be to find measures that fulfil our criteria: based on carefully evaluated evidence, clear practice guidelines, and events or characteristics that are well defined, frequent enough, and recorded sufficiently accurately for values to be reliably estimated and compared at a practice level.

What is already known on this topic Different dimensions need to be taken into consideration when measuring quality of care in general practice Patients can appropriately assess some aspects of quality of health care What this study adds Some aspects of technical quality of primary care can be measured using practice records; these measures indicate substantial variations between practices Patients' own assessments of technical quality are not closely related to independently ascertained records based measures of technical quality Assessment of the technical quality of primary care should not rely on patient based assessments alone

---

We thank Basildon primary care trust and the general practitioners for their support; the participants for completing the questionnaires; research nurses Christine Mann and Pamela Milnes, who helped distribute the questionnaire and manage the database; and Martin McKee and the referees who commented on drafts of the paper.

Contributors: MR conceived the study, obtained funding, designed the protocol, recruited participants, and collected and analysed the data. AC helped design the study and interpreted and analysed the data. CS helped design the study and carried out statistical analysis. RH helped design the study, interpret data, and carry out statistical analysis. AC will act as guarantor. All authors helped write the manuscript.

Funding: North Thames regional health authority.

Competing interests: At the time of the study MR was employed first as director of public health for South Essex Health Authority and subsequently as director of the Essex Public Health network.

Ethical approval: This study was approved by south Essex research ethics committee.

References

1. Donabedian A. Exploration in quality assessment and monitoring. Vol I. Definition of quality and approaches to its assessment. University of Michigan, Ann Arbor: Health Administration Press, 1980.
2. Maxwell RJ. Quality assessment in health. BMJ 1984;288: 1470-2.[ISI][Medline]
3. Brook RH, McGlynn EA, Cleary PD. Quality of health care. Part 2: measuring quality of care. N Engl J Med 1996;335: 966-1001.[Free Full Text]
4. Shickle D. Public preferences for health care: prioritisation in the United Kingdom Bioethics 1997;11: 277-90.[Medline]
5. Rhodes P, Nocon A. User involvement and the NHS reforms. Health Expect 1998;1: 73-81.[CrossRef][Medline]
6. Department of Health. Choice, responsiveness and equity in the NHS. London: Stationery Office, 2003. (CM6709.)
7. Department of Health. Delivering investment in general practice: implementing the new GMS contract. London: Department of Health for England and Wales, 2003.
8. Schneider EC, Zaslavsky AM, Landon BE, Lied TR, Sheingold S, Cleary PD. National quality monitoring of Medicare health plans: the relationship between enrolee's reports and the quality of clinical care. Med Care 2001;39: 1313-25.[CrossRef][ISI][Medline]
9. Safran DG, Kosinski M, Tarlov AR, Rogers WH, Taira DH, Lieberman N, et al. The primary care assessment survey: tests of data quality and measurement performance. Med Care 1998;36: 728-39.[CrossRef][ISI][Medline]
10. Ramsay J, Campbell JL, Schroter S, Green J, Roland M. The general practice assessment survey (GPAS): tests of data quality and measurement properties. Fam Pract 2000;17: 372-9.[Abstract/Free Full Text]
11. Bland M. An introduction to medical statistics, 3rd edn. Oxford: Oxford University Press, 2000.
12. McColl A, Roderick P, Smith H, Wilkinson E, Moore M, Exworthy M, et al. Clinical governance in primary care groups: the feasibility of deriving evidence-based performance indicators. Qual Health Care 2000;9: 90-7.[Abstract/Free Full Text]
13. Sever P, Beevers G, Bulpitt C, Lever A, Ramsay L, Reid J, et al. Management guidelines in essential hypertension: report of the second working party of the British Hypertension Society. BMJ 1993;306: 983-7.[ISI][Medline]
14. Chief Medical Officer and Chief Nursing Officer. Influenza immunisation: extension of current policy to include all those aged 75 years and over. London: Department of Health for England and Wales, 1998. (PL/CMO/98/4.)
15. Campbell JL, Ramsay J, Green J. Age, gender, socioeconomic and ethnic differences in patients' assessments of primary health care. Qual Health Care 2001;10: 90-5.[Abstract/Free Full Text]
16. Erens B, Primatesta P. Health survey for England 1998. A survey carried out on behalf of the Department of Health. London: Stationery Office, 1999.
17. Chapple A, Campbell S, Rogers A, Roland M. Users' understanding of medical knowledge in general practice. Soc Sci Med 2002;54: 1215-24.[CrossRef][ISI][Medline]
18. Heaney DJ, Walker JJ, Howie JGR, Maxwell M, Freeman GK, Berrey PN, et al. The development of a routine NHS data-based index of performance in general practice (NHSPPI). Fam Pract 2002;19: 77-84.[Abstract/Free Full Text]
19. Alazri MH, Neal RD. The association between satisfaction with services provided in primary care and outcomes in type 2 diabetes mellitus. Diabet Med 2003;20: 486-90.[CrossRef][ISI][Medline]
20. Relman AS. Assessment and accountability: the third revolution in medical care. N Engl J Med 1988;319: 1220-2.[ISI][Medline]
21. Toon P. Towards a philosophy of general practice: a study of the virtuous practitioner. Occasional paper 78. London: Royal College of General Practitioners, 1999.
22. Roland M. Linking physicians' pay to the quality of care—a major experiment in the United Kingdom. N Engl J Med 2004;351: 1448-54.[Free Full Text]

(Accepted 9 May 2006)

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Relevant Articles

Patient involvement in health care will improve quality

Robert Hunter and Rosie Cameron
BMJ 2006 333: 147-148. [Extract] [Full Text]

Patients' views are not enough to judge quality of care
BMJ 2006 333: 0. [Full Text]

Can patients assess the quality of health care?

Angela Coulter
BMJ 2006 333: 1-2. [Extract] [Full Text] [PDF]

This article has been cited by other articles:

• Lee, D. S., Tu, J. V., Chong, A., Alter, D. A. (2008). Patient Satisfaction and Its Relationship With Quality and Outcomes of Care After Acute Myocardial Infarction. Circulation 118: 1938-1945 [Abstract] [Full text]  
• Jha, A. K., Orav, E. J., Zheng, J., Epstein, A. M. (2008). Patients' Perception of Hospital Care in the United States. NEJM 359: 1921-1931 [Abstract] [Full text]  
• Cheraghi-Sohi, S., Hole, A. R., Mead, N., McDonald, R., Whalley, D., Bower, P., Roland, M. (2008). What Patients Want From Primary Care Consultations: A Discrete Choice Experiment to Identify Patients' Priorities. Ann Fam Med 6: 107-115 [Abstract] [Full text]  
• Howell, E., Graham, C., Hoffman, A, Lowe, D, McKevitt, C., Reeves, R., Rudd, A G (2007). Comparison of patients' assessments of the quality of stroke care with audit findings. Qual Saf Health Care 16: 450-455 [Abstract] [Full text]  
• Hunter, R., Cameron, R. (2006). Patient involvement in health care will improve quality.. BMJ 333: 147-148 [Full text]  
• Coulter, A. (2006). Can patients assess the quality of health care?. BMJ 333: 1-2 [Full text]  

Rapid Responses:

Read all Rapid Responses

Questionable measures of quality

Sarah Nicholls, et al.
bmj.com, 7 Jul 2006 [Full text]

when interpersonal skills and technical care collide

Anne E Thompson
bmj.com, 14 Jul 2006 [Full text]