Informatics Section

Surgical informatics on the internet: any improvement?

J.O. Murphy K.J. Sweeney J.C. O’Mahony S.M. Johnston K. Conlon F.B.V. Keane S. Tierney
Department of Clinical Surgery, The Adelaide and Meath Hospital, incorporating the National Children’s Hospital, Tallaght, Dublin 24, Ireland.

Correspondence: Mr Sean Tierney, Dept of Clinical Surgery, AMNCH, Tallaght, Dublin 24, Ireland E-mail:sean.tierney@amnch.ie

Introduction Methods

Results Discussion References

Keywords: Internet, medical informatics, patient education Surg J R Coll Surg Edinb Irel., 1 June 2003, 177-179

Purpose: The Internet is a popular, but ungoverned, source of medical information. This study tracked the change in performance of commonly available search engines and the quality of medical data therein over a four-year period. Methods: We compared the accuracy of information on a commonly performed surgical procedure (vasectomy) using six standard search engines in a four-year period and with two recently developed search engines. The top 25 ranked sites cited by each search engine were scored for description of the procedure, post-operative instructions, complications and unproven associations. Results: There was no improvement in quality of individual sites over the study period. Additionally, the hit rate of search engines remained poor with 27 sites cited (40%) in 2002 either irrelevant or unavailable. Discussion: Few useful sites with accurate information on surgical procedures are available on the Internet and simple search strategies fail to identify site quality or relevancy. Conclusion: At present, the Internet cannot be recommended as a reliable resource for many aspects of health information for patients. The onus is on health-care providers to provide high quality sites and direct patients to these sources of reliable information

INTRODUCTION
Global accessibility and falling costs have resulted in the Internet becoming a major source of specialist information for the general public. However, the internet is essentially ungoverned and patients seeking medical information on the Internet may be exposed to large volumes of incomplete and often-erroneous data.^1,2

The worldwide “online” population increased from 245 million in 1999 to about 600 million in 2002.3 The growth of traf.c on the Internet has increased by 2 to 3 fold every year from the mid 1990s and this should be sustained for the next few years.4 Whether the quality of information has improved during this time is unclear.⁵ What is clear, however, is that the healthcare profession must regularly re-appraise the role of the Internet in patients’ decision-making process. Herein we investigate the change in quality of medical informatics on the Internet.

METHODS
To create a clinical relevant model a home personal computer (Dell computers, Ireland) with a standard telephone connection, 56K modem, a free Internet service provider (Eircom, Ireland) and Microsoft Internet Explorer^TM software (Microsoft, USA) were used. Information on vasectomy was sought, as it is a simple, elective surgical procedure performed on healthy patients, which has lifechanging consequences. It requires detailed information and counselling on which to establish informed consent.

Standard Internet search engines were used as they were felt to best represent patients’ search profiles. These search engines ranked web sites according to their ‘hit-popularity’, that is to say, the number of people accessing the site per day. Six popular search engines (Alta Vista, Excite, Hotbot, Lycos, Go ( previously Infoseek), and Yahoo) were queried with the term ‘vasectomy’ and the quality of information contained in the first 25 ranked websites was scored for completeness and accuracy of information based on 34 items in 5 categories (Table 1). Each item scored 2 points for complete information, 1 point for incomplete information, 0 points if the item was not mentioned and -1 point for incorrect information. Thus, the maximum a website could score was 68. Final scores are expressed as a percentage of the maximum score.

Website scores were measured at two time points (June 1999 and May 2002). Additionally, we incorporated two new search protocols into the 2002 assessment: Copernic 2001 Basic is a free composite search engine which accesses 80 search strategies, and Google. Google differs from the other search engines because it ranks web sites by means of linkpopularity (the number of links to a site from other sites). This strategy is believed to be a more accurate method of evaluating individual web sites.

Data are presented as median percentage of maximum score (interquartile range). Statistical analysis was performed using the Mann-Whitney U Test. A p value of < 0.05 was deemed statistically significant.

RESULTS
Ninety-eight (of a potential 150) individual relevant sites were identi.ed in 1999 (duplicated sites between search engines were ignored) and 66 individual websites were found in 2002. Increasing duplicity probably represents a homogenisation of the various search engines. Furthermore, there was a trend towards improvement in the individual search engines and this was signi.cant in the cases of Alta Vista (p=0.001), and Lycos (p=0.009) (Table 2).

Although Google obtained better results than most other search engines in 1999, it obtained similar scores to the same search engines in 2002 (Table 2). Not surprisingly, Copernic demonstrated no distinct advantage over the original search engines at any time point and there was no signi.cant difference in quality of information between either Google or Copernic. These .ndings concur with other studies demonstrating a lack of correlation between measures of site quality and linkpopularity.¹

Only 20 of the 98 original websites (20%) remained active from 1999 until 2002. These sites showed no improvement in quality (16 (2-35) vs. 14 (0-29); 1999 score vs. 2002 score; p=0.9).

In 2002, 40% of websites found using standard search engines were unavailable or irrelevant (e.g. discussing vasectomy reversal only or commercial sites advertising a clinic). This had increased from 28% unavailability or irrelevancy in 1999.

DISCUSSION
The results of this study are of concern to all healthcare workers. Despite improved methods (i.e. link-popularity), basic web search engines remain poor discriminators of quality medical information. Additionally, there are few websites that patients can be reasonably referred to for health information. A systematic review of 86 studies assessing the quality of health information on the Internet found that most studies concluded that quality is a problem on the Internet.⁵

Other mass media sources such as television and radio have been shown to be inadequate for medical information, with authors in the early 1970s .nding the majority of health information broadcasts on television to be inaccurate, misleading, or both.6 Collaboration between the media and the healthcare profession has resulted in higher quality medical information by these means and the same must now happen with the Internet to ensure accentuation of its content.

We were disappointed to observe a lack of improvement in information quality over a period that many have described as a revolution in global informatics.

We conclude that the Internet still needs improvement before it can be recommended as a reliable resource of health information for patients. The onus is on the medical profession to provide high quality sites and to direct patients to sources of reliable information.

REFERENCES
1. Meric F, Bernstam EV, Mirza NQ, Hunt KK, Ames FC, Ross ML, Kuerer HM, Pollock RE, Musen MA, Singletary SE. Breast cancer on the World Wide Web: cross sectional survey of quality of information and popularity of websites. BMJ 2002; 324: 577-581.
2. Libertiny G, Perkins JM, Magee TR, Galland RB. Varicose veins on the internet. Eur J Vasc Endovasc Surg 2000 Oct;20(4):386-9.
3. Global Reach. Evolution of online linguistic populations. http://www.global-reach.biz/globstats/evol.html, last accessed on 13-Mar-2003
4. Featherly K. Annual Internet growth still tripling. http: //www.computeruser.com/news/02/01/20/news1.html, last accessed on 13-Mar-2003
5. Eysenbach G, Powell J, MSc, Kuss O, Sa ER. Empirical Studies Assessing the Quality of Health Information for Consumers on the World Wide Web, A Systematic Review. JAMA 2002; 287: 2691-700.
6. Smith FA, Trivax G, Zuehlke DA, Lowinger P, Nghiem TL. Health information during a week of television. N Engl J Med 1972; 286(10): 516-20.

Copyright: 22 April 2003