Computer-based EBM
Marjorie Lazoff, MD
Emergency Medicine
Philadelphia, Pennsylvania
Medical Editor
Medical Computing Today

accepted for publication in Medical Computing Today August 1998
Originally published in edited form August 1998 in Medical Software Reviews

Learning About EBM - Major Online Resources - Fee-Based Databases - Free Databases
Journals & Newsletters - Megasearches and Megalists

Evidence-based medicine (EBM) can be found on hundreds of general sites, conference and training programs, software descriptions and files for downloading, extensive megalists, and online databases. Granted, the majority of EBM sites are text-intensive and tend toward unexciting first- or second-generation Web design. Yet these sites are also characterized by global cooperation and complementary resources, and by their lack of sensationalism and tactless marketing. From a business perspective, several sites illustrate the growing trend in medicine toward academic/government/commercial partnerships, while others are working examples of how to present free abstracted material while holding back the full text reference for a fee.

Technology is central to EBM's practice and growth; statistical analyses, advanced search techniques, global database management, and information dissemination to all physicians and other clinicians would be impractical or impossible without a computer. As our nation's health care system becomes computer-literate and computer-dependent, to what extent will -- and should -- our training and practice become evidence-based?

M edical research is considered by most physicians to be the cornerstone of modern clinical practice. Yet a formal literature search and analysis of relevant studies ("evidence") performed as one component of a structured clinical decision-making process and applied to a specific patient, is considered by many to be a subtly new paradigm of medical care. Rooted as much in 19th century philosophy as in the 1972 book, Effectiveness and Efficiency: Random Reflections on Health Services by British epidemiologist Arthur Cochrane, EBM was initially developed as a way to teach medical students and only later applied to clinical problem-solving. Today, most (not all) EBM enthusiasts are found in medical schools and academia, within departments of medical informatics, in leadership positions within scholarly medical journals, and managed care organizations and governments throughout the world … and on the Web.

EBM proponents believe that to practice the highest quality medicine a physician must know how to formulate a clinical inquiry based on a patient's problem(s), skillfully search the medical reference databases for relevant studies that meet rigorous statistical standards, and then appropriately apply or reject this data based on the actual patient's condition and needs. Such a physician demonstrates an organized, insightful use of research in daily practice, rather than relying on today's unstructured, common sense approach based on decades-old medical training and textbooks nearly as old, personal clinical experiences, and a smattering of information gleamed from quickly scanned journals and attendance at sun-and-sand conventions. But while good medical informatics skills are critical they are not sufficient to providing good care; identifying health care problems and understanding how to apply study results to individual patients requires medical training and clinical judgment as well. As David L. Sackett, MD, director of UK's Centre for Evidence-Based Medicine and a leading EBM advocate explains, "(g)ood doctors use both individual clinical expertise and the best available external evidence, and neither alone is enough."1

To standardize and expedite retrieval of medical "evidence," groups of EBM experts write analytic reviews based on biostatistical and/or meta-analyses of selected patient-centered studies (ideally, randomized controlled trials and outcomes research). Together, these systematic reviews-such as the Cochrane Database of Systematic Reviews and the Centre for Reviews and Dissertations' databases-comprise a unique clinical resource, especially when continually updated and distributed via print and electronic journals, software, online services, or on the Web. Medline is an important EBM reference as well, but with carefully constructed searches to "filter out" unacceptable, irrelevant, or unrelated studies.

Over the decade EBM has evolved top-down, not bottom-up as would a grassroots movement. Spurred on by each governments' national health services, EBM is most popular in England and Canada and to a lesser extent, Australia and other countries. In the U.S., EBM is supported by several specialty organizations, most notably ACP-ASIM and AAFP, and portions of the government, most notably the AHCPR. It is also linked to many academic centers through physician training, medical informatics, biostatistics, and clinical research, and a number of managed care companies through cost-cutting strategies and the purchaser-provider model, where the purchaser insists on good evidence that a service or procedure is safe and effective.

Many U.S. practitioners react with suspicion upon hearing EBM accolades and promises of improved quality care. Skeptics wonder whether EBM isn't just the same medical paradigm repackaged in technospeak by armchair academicians, and adopted by managed care managers as a way to legitimize practice guidelines with hidden business agendas. Teaching and practicing EBM on a wide scale will require a professional and national commitment of time, money and education reform-efforts that many feel are better spent directly on patient care, given all the gaps in our medical knowledge and the imperfect organization of our present databases. Even should investments in EBM pay off with closing gaps and well-indexed databases, there remain practical limitations inherent in all scientifically and statistically rigorous studies. Accurate data will always be hard to come by, and when gathered it's easy to be seduced into believing accurate data is omnipotent.

Yet there are several EBM-inspired observations to which most everyone agrees:

M ost online EBM instruction is thoughtfully designed but necessarily pedantic and requires significant time, attention, and practice. Few of us enjoy learning statistics or research principles in our spare time, but it's exciting to know that, should professional needs or personal inclinations require it, the Web is a free and convenient alternative for such instruction. Note too that at present there is no EBM database equivalent to Medline, so physicians must hunt down and search multiple databases, several of which are important but very small. One hopeful sign is that at least three free sites and one online service are attempting to create a searchable EBM megadatabase. Finally, it's difficult for physicians to know whether a non-peer reviewed systematic review or summary is authoritative, or if missed and/or more recent studies disprove or discount it. As a clinical tool, EBM databases at present complement rather than replace Medline.

Sections Learning About EBM

Many physicians find EBM terminology quite intimidating. Fortunately, several easy-to-use glossaries exist on line, among them the glossary of EBM terms, clinical practice terms, Cochrane terms and research terms. Consider bookmarking one or two for easy reference as you visit the sites below.

For those unfamiliar with EBM, a good starting point is a 1996 BMJ editorial by Dr. Sackett, Evidence-Based Medicine: What it is and what it isn't. SUNY Health Sciences: Evidence-Based Medicine Course offers a frames-based basic introduction on how to formulate a search inquiry using PICO (population, intervention, comparison and outcome), how to perform a Medline search using the popular Ovid interface, and with non-mathematical explanations of key EBM statistical terms. University of Sheffield's Internet Tutorial on Evidence-Based Practice is another good starting point for an introduction to EBM, basic Medline search instructions, and non-statistical analyses of clinical studies.

Dozens of online resources are dedicated to teaching how to search the medical literature. The classic reference is Users' Guides to Medical Literature -- non-frames or frames, a series of articles written by Dr. Sackett and colleagues and published in JAMA during Fall-Winter 1993-94. Use the turquoise image map on top of each page to navigate through the sections (top row) of each guide (bottom row). The Guide has also been summarized in query-list form with full JAMA citations.

Searching or filtering Medline for only those studies that meet EBM rigorous standards requires search commands that differ depending on the engine and interface. Oxford's Institute of Health Sciences has made available for downloading its Medline search filters for a number of popular engines. For other filters scroll down to the end of University of Hertfordshire's Database list.

Understanding EBM biostatistics is not an intuitively obvious process for most physicians. A highly regarded series of 10 articles by senior lecturer Trisha Greenhalgh of the University of Leeds/Royal Free Hospital's Unit for Evidence-based Practice and Policy on How to read a paper, from Summer 1997 in BMJ, is a relatively painless beginning. Toolbox has one of the better explanations of EBM statistics on line. See too Evidence-Based Drug Therapy: What do the numbers mean from the University of British Columbia's Therapeutics Initiative. For more details on number needed to treat (NNT), see BMJ's excellent 1996 article, The number needed to treat: a clinically useful nomogram in its proper context.

Beyond biostatistics, it is also important to understand the concept behind systematic reviews. Self-training guide and notes from the UK's National Health Service (NHS) are self-extracting Winzip or Powerpoint files that introduce the definition and value of systematic reviews, the work of the Cochrane Collaboration and NHS Centre for Reviews and Dissemination, and an in-depth explanation of odds-ratios. The March 1, 1997, Annals of Internal Medicine (AIM) editorial on Systematic Reviews: Critical Links in the Great Chain of Evidence and Systematic Reviews: Synthesis of Best Evidence for Clinical Decisions discuss systematic reviews and what they can and can't accomplish. See too AIM's article from April 1, 1997, Locating and Appraising Systematic Reviews. On a related topic, the Aug 21, 1997, issue of NEJM has an editorial on Promise and Problems of Meta-Analysis, which warns that the statistical elegance of meta analyses and their frequently accompanying diagrammatic representations may focus the reader away from critical appraisal of a systematic review. Getting research findings into practice: When to act on the evidence from the July 11, 1998, issue of BMJ, discusses theoretical considerations and gives practical advice to clinicians interested in learning how to incorporate evidence-based medicine into their practice.

Oxford University's Appraising the Quality of Research, and University of Leeds' The Systematic Literature Review Training Module, detail the relationship between research and systematic reviews from the reviewers' perspective. The Cochrane Handbook is also written for systematic reviewers, but provides the rest of us with insight into the organization and goals of Cochrane Collaborations. The Aggressive Research Intelligence Facility is a funded NHS project created to improve access and advice on reviewing research. Its Critical Appraisal Checklist is a series of questions used by their appraisers when assessing the quality of EBM reviews.

EBM concepts and tools overlap those of quality assessment and clinical decision-making. All articles published in AIM's October 15, 1997, supplemental issue on Measuring Quality, Outcomes, and Cost of Care Using Large Databases: The Sixth Regenstrief Conference are on line. Although some history and theory is introduced, this series of articles is primarily practical and futuristic. It considers EBM from the vantage of databases and electronic medical records, research, performance and outcome measures, and medical administration. Charles E. Phelps, PhD, summarizes the diffusion of information in health care in the April 20, 1998, issue of AMNews, and explores this in relation to regional variation in care, a key issue in quality assurance. Decision analysis is described in a 1993 article from Australian Health Informatics Newsletter, Analyzing the art of medicine using formal rules based on the information (probabilities), values, and preferences of the decision maker.

Those interested in teaching EBM will appreciate A New Approach to Teaching the Practice of Medicine and Teaching resources of scenarios and worksheets.

Sections Major Online Resources

Although cited elsewhere in this report, two EBM Web sites are so comprehensive that readers may wish to visit their home pages directly and review the sites' information, clinical and educational resources, and interactivity. Oxford's Centre for Evidence-Based Medicine (CEBM) is a straightforward, easily navigable site that contains some excellent educational resources, particularly in statistics, and less formal clinical reviews called Critically Appraised Topics or CATs. Posting a more attractive and complex site, Canada's Health Information Research Unit (HIRU) from McMaster University is North America's leading center for EBM, referred to here as Evidence-Based Health Informatics. The left-sided frame contains the site menu. As described above, its highly regarded Users' Guides are a popular reference among students and house staff. The Database section of relationships, people, and products are available in both Javascript and HTML. The Files section gives access and instructions to EBM FTP sites.

Both CEBM and HIRU are associated with their countries' respective Cochrane Centres, a dozen or so groups scattered throughout the world whose goal is to help coordinate and support The Cochrane Collaboration, EBM's premier working organization. Founded in 1992 by dozens of people from nine countries, the Collaboration was inspired by Arthur Cochrane's challenge to the health community two decades earlier to create systematic, up-to-date reviews of all relevant randomized controlled trials. The Collaboration's Nine Values provides a philosophic framework for its 40-odd Collaborative Review Groups (CRGs), covering topics ranging from, for example, acute respiratory infections to effective professional practice, diabetes, fertility regulation, incontinence, and stroke. Each CRG works through a highly structured process to systematically prepare and maintain reviews of selected clinical trials and research studies within its assigned medical topic; see the November 1995 issue of Evidence-Based Medicine's How members of the Cochrane Collaboration prepare and maintain systematic reviews of the effects of health care. An example of a systematic review is worth its thousand words.

Sections Fee-Based Databases

Gathering the 737+ systematic reviews together results in the Cochrane Database of Systematic Reviews (CDSR), the first of four databases comprising the Cochrane Library. As part of the Library, the CDSR is presented as either completed reviews, or as protocols of the background, objective and methods for reviews currently being prepared. While CDSR is a subscription database, the Abstracts of Cochrane Reviews are freely available on line, as described below.

The Library's second database is the Database of Abstracts of Reviews of Effectiveness (DARE). Prepared by the NHS's Centre for Reviews and Dissemination (CRD) at the University of York, a sibling organization of the UK Cochrane Center, DARE was created to complement the CDSR by providing over 1,500 previously published, quality assessed reviews in areas where there is currently no Cochrane review. Note that DARE is also freely available, along with two other non-Cochrane Library databases as described below, off the CRD Web site.

The remaining two Cochrane Library databases are bibliographies, including abstracts. The Cochrane Controlled Trials Register lists about 180,000 clinical studies considered (though not necessarily included) in the systematic reviews; unfortunately, the studies cited at the end of each systematic review are not linked to this Register, but each review includes the CRG's search. The Cochrane Review Methodology Database includes almost 700 articles on the science of research. The Library also contains the above-described Handbook on the science of reviewing research, a glossary of terms, and CRG contact information.

Update Software is presently the sole publisher of The Cochrane Library software products. The full Library is available on CD-ROM and diskette, for Windows PCs or installation on a LAN server. Price for an annual subscription, updated quarterly, is about $225; multiuser and organizational pricing starts at $400. This is a "read-only" reference demanding little by way of processor speed or RAM, and without the bookmarking, highlighting, or note features users have come to expect. The software even lacks typical Windows editing or file export functions, although articles can be printed out or saved as a text file. Fortunately the Library is accessible relatively fast directly from the CD drive, since installation requires at least 650 Mb of hard drive space, and must be reinstalled with each quarterly update.

The screen is divided into scrollable frames and chunky feature buttons reminiscent of reference software from the early 1990s -- but the interface is surprisingly intuitive and very functional, and users can optimize features or toggle among several screen layout. The search engine is likewise inelegant but sturdy, with a number of excellent search features, including the ability to save searches for future use. Thus, despite its dated interface and limited features, Update's The Cochrane Library software does the important things right: it provides for easy and intuitive database access and retrieval.

Update Software's The Cochrane Library Online is one of several EBM online services accessible through the Web. With price comparable to the CD-ROM and also including the full Library contents, it is a good alternative to software purchase for those with quick, unlimited Web access or for those using operating systems other than Windows. Also, unlike the CD software, editing and other browser functions are available on line. A straightforward text interface and good online help complement the service, but unfortunately it lacks the fine screen layout and strong search engines software users enjoy.

Subscriptions to CDSR (the first database, not the full Library) are available from several other services, including Australia's Health Communication Network and Canada's Synapse Publishing Inc.

Ovid's Evidence-Based Medicine Reviews is comprised of CDSR (not the full Cochrane Library) and Best Evidence (a database of study reviews from the ACP Journal Club and Evidence-Based Medicine); subscribers may also use its Medline service with preselected EBM search filters. Those who are familiar with and enjoy Ovid's intuitive, thoughtful interface may prefer this online service. One very nice touch: when available under Best Evidence and Medline, reviews are linked to the full text of the original article. Another interesting feature (not tested) is described as "expert topic searches," which recreate the exact searches originally used by the CRG, so users can perform their own Medline searches to update information. Sadly, Ovid's steep access and hourly fees limit the utility of all its databases.

Sections Free Databases

The Abstracts of Cochrane Reviews -- by menus or a scrollable page -- are available on line for free. All 737+ abstracts are presented under topics corresponding to each CRG. The abstracts are a wonderful online reference for a quick scan of the medical literature, but as a clinical or scholarly database it has limitations; the CDSR itself isn't comprehensive, and the abstracts omit key primary information such as the actual articles reviewed. Included with the abstracts are the titles of protocols, which are reviews-in-progress. Adding a free-text search engine would be a thoughtful touch. Although a number of sites link to the Abstracts, these two are the most conscientiously updated (quarterly).

The Centre for Review and Dissemination (CRD), a facility commissioned by the NHS with contributions from the University of York, produce and disseminate highly regarded reviews on the quality and cost-effectiveness of healthcare interventions. The CRD offers a trio of searchable databases, the most popular of which is a collection of over a thousand systematic reviews created to supplement the CDSR, Database of Abstracts of Reviews Of Effectiveness (DARE). This database is included with the fee-based Cochrane Library, as described above. The two other CRD databases available on line are the Economic Evaluation Database, which contains structured abstracts of economic and quality evaluations of health care interventions, and The Health Technology Assessment Database, which contains abstracts produced by the International Network of Agencies for Health Technology Assessment and other healthcare technology agencies. (A related database located at a separate site, UK's Health Technology Assessment Programme makes about 20 reports available as PDF files for free downloading.)

Wessex Institute of Public Health Medicine's approximately 50 Development and Evaluation Committee Reports are narrowly focused clinical summaries based on critical reviews of the literature. Writeups are conservative and thoughtful, and users are appropriately warned that each report, like any systematic review, is time-limited and that more recent research may invalidate it. Earlier reports are listed but not available on line. Ontario's Institute for Clinical Evaluative Sciences (ICES) is a nonprofit organization that conducts research into health care effectiveness, quality and efficiency. Over 100 summaries are grouped into 15 categories, and include working paper summaries and newsletter articles; alas, there is no search engine. Health Evidence Bulletin -- Wales on the University of Wales College of Medicine Web site is an ambitious developing database of systematic bulletin reviews presented in chart format. Maternal and early child health, oral health, and respiratory diseases are online at present.

General internal medicine fellows at McMaster University invented Critically Appraised Topics, or CATs to sharpen their critical appraisal skills and improve their abilities as bedside teachers of EBM. Also, see also Southwest Ontario Regional Academic Health Science Network (SORAHSN)'s Critically Appraised Papers. Unlike systematic reviews, CATs and CAPs are single study reviews and are not peer-reviewed.

EBM purists might insist that all evidence reports and clinical practice guidelines (CPGs) be based on systematic reviews or follow similar procedures, although in reality this is not usually the case. Still, a number of online resources are highly regarded and often included among EBM references. AHCPR'S Clinical Information is a menu of U.S. government resources, including Evidence Reports (colon cancer screening); 20-odd Clinical Practice Guidelines; NIH's Guide to Clinical Preventive Services, second edition; and Health Technology Assessment and Review of new technologies, from 1990-1996. NIH's CSAT Treatment Improvement Protocols (TIPs) are best practice guidelines for the treatment of substance abuse. CPG Infobase, a Canadian Medical Association database, plans a much-needed search engine for this extensive and well-maintained resource. NHS Executive posts three R&D Briefing Reports on procedural treatment of coronary ischemia, eradication of H pylori, and prenatal diagnostic testing. The Scottish Intercollegiate Guidelines Network (SIGN)'s evidence-based CPGs are accessible from the site's frame-based home page as PDF (all) and HTML files. For many more practice guidelines, see SORAHSN's Catalog and The New York Academy of Medicine's CPG List.

The University of British Columbia's Departments of Pharmacology and Therapeutics, and Family Practice, established the Therapeutics Initiative to provide physicians and pharmacists with up-to- date, evidence-based, practical information on drug therapy. The Initiative prides itself on being a thoroughly independent organization and on maintaining its database of bimonthly topical issues since 1994. This is a thoughtful reference reminiscent of The Medical Letter.

NHS's OxamWeb is an index of mental health references, divided into categories. Evidence-based summaries are noted. Future plans include site organization according to Medical Subject Heading (MeSH) terms and a search engine. Washington University's PedsCCM Evidence-Based Journal Club has a dozen or so completed reviews, with many more in progress. Scroll up and down for further information about its club and mailing list. Group Health Northwest's A Gathering of Evidence has about a dozen abstracts posted with commentary and suggested EBM resources.

UK's Clearing House on Health Outcomes Databases includes three outcomes databases. These are not systematic reviews, but rather library identifiers and terse summaries of the material covered in the article. Each database is entered using a search engine: the Outcomes Activities Database, Outcomes Database of Structured Abstracts, and Outcomes Literature Database. Since 1997, this last database has joined Nuffield Institute's HELMIS database, which is off line.

Sections Journals & Newsletters

The most aesthetically pleasing and organized of the four main EBM journals is NHS's Bandolier, which describes its summaries as bullet points of evidence-based medicine. The home page allows access to the month's current issue, and to archives of past issues, subject index, and table of NNTs. The Journal of Family Practice describes its Patient Oriented Evidence that Matters, or POEMs, as "critical appraisals of articles which have the potential to change practice methods." This page accesses all past and current POEMs. Less generous, the ACP Journal Club and Evidence-Based Medicine, co-published by the ACP and BMJ, are bimonthly journals that each place a handful of their summaries on line, although both sites contain archives of past summaries. Other journals and newsletters: University of York/NHS Centers for Reviews and Dissemination publications, a nice resource in its own right, links to Effective Health Care which posts all but the first volume of its three years of newsletters. Bullet summaries are on line, with most full-text articles available as PDF files. Effectiveness Matters is described as a complement to Effective Health Care. ICES's Infomed, the monthly newsletter from Ontario, also places summaries on line.

NHS Executive's Evidence-Based Purchasing's subject index is a nice shortcut to this online bimonthly newsletter, which summarizes updates on, among other databases, the Cochrane Library, NHS Centre for Reviews and Dissemination, Health Technology Assessment Programme, and Development and Evaluation Committee Reports.

UK's Primary Care Clinical Effectiveness Team for Gwent publishes Implementation and Change a bimonthly Web-based journal that applies evidence-based techniques to public health care delivery. A laudable effort, but the writing is imprecise, which weakens the authority of each issue's dozen summaries.

Sections Megasearches And Megalists

Internet Database of Evidence-Based Abstracts and Articles (IDEA) Topic List is a searchable database currently available only within the Oregon Health Sciences University network. Fortunately for the rest of us, a list of MeSH-indexed documents is available here. This is a useful resource, even without a search engine and with dated material (nothing beyond 1996 could be found). Curiously, no Cochrane Abstracts are included. More complete and current is a second searchable database, UK's TRIP. It includes Cochrane Abstracts, DARE, the major EBM journals, and excellent although not strictly EBM resources such as Silverplatter's MD Digests. Unfortunately, its free text search engine looks at titles only, so consider all synonyms and spelling variations.

There are a number of standout megalists, most of which are annotated and well-maintained. University of Sheffield's Netting the Evidence: A ScHARR (School of Health and Related Research) Introduction to Evidence-Based Practice on the Internet is perhaps the most extensive and best known, but see too Trawling the Net: A ScHARR Introduction to Free Databases of Interest to NHS Staff on the Internet. Ottawa Hospital's Evidence-based medicine and practice resources is another comprehensive, well-maintained list. University of Hertfordshire EBM maintains an excellent list of resources, including updated print (rarely Web) scholarly articles and many other Internet resources. The New York Academy of Medicine's EBM List is especially well-organized and has an extensive list of clinical practice guidelines. HIRU Quicklinks describes, then links, to online search engines, databases, and EBM-related resources. Evidence-based CPG Repository links to many online resources, although its discussion area has not been well received. OHSU's Michael Zacks, MD EBM List is no longer maintained, but has several excellent active links. Finally, the April 20, 1998, AMNews article on Finding and Assembling the Best Evidence includes a glossary with links to other online resources.

1 Sackett DL, et al. Evidence-Based Medicine: What it is and what it isn't. BMJ 1996; 312: 71-72
Return to article
Comments or questions for posting?
Archives of other articles
Introduction and List of other NetView articles