Reliability of citations of medRxiv preprints in articles published on COVID-19 in the world leading medical journals

Introduction

Preprints have been widely cited during the COVID-19 pandemics, even in the major medical journals. However, since subsequent publication of preprint is not always mentioned in preprint repositories, some may be inappropriately cited or quoted. Our objectives were to assess the reliability of preprint citations in articles on COVID-19, to the rate of publication of preprints cited in these articles and to compare, if relevant, the content of the preprints to their published version.

Methods

Articles published on COVID in 2020 in the BMJ, The Lancet, the JAMA and the NEJM were manually screened to identify all articles citing at least one preprint from medRxiv. We searched PubMed, Google and Google Scholar to assess if the preprint had been published in a peer-reviewed journal, and when. Published articles were screened to assess if the title, data or conclusions were identical to the preprint version.

Results

Among the 205 research articles on COVID published by the four major medical journals in 2020, 60 (29.3%) cited at least one medRxiv preprint. Among the 182 preprints cited, 124 were published in a peer-reviewed journal, with 51 (41.1%) before the citing article was published online and 73 (58.9%) later. There were differences in the title, the data or the conclusion between the preprint cited and the published version for nearly half of them. MedRxiv did not mentioned the publication for 53 (42.7%) of preprints.

Conclusions

More than a quarter of preprints citations were inappropriate since preprints were in fact already published at the time of publication of the citing article, often with a different content. Authors and editors should check the accuracy of the citations and of the quotations of preprints before publishing manuscripts that cite them.

OntoBioStat: Supporting Causal Diagram Design and Analysis

Suitable causal inference in biostatistics can be best achieved by knowledge representation thanks to causal diagrams or directed acyclic graphs. However, necessary and sufficient causes are not easily represented. Since existing ontologies do not fill this gap, we designed OntoBioStat in order to enable covariate selection support based on causal relation representations. OntoBioStat automatic ontological causal diagram construction and inferences are detailed in this study. OntoBioStat inferences are allowed by Semantic Web Rule Language rules and axioms. First, statements made by the users include outcome, exposure, covariate, and causal relation specification. Then, reasoning enable automatic construction using generic instances of Meta_Variable and Necessary_Variable classes. Finally, inferred classes highlighted potential bias such as confounder-like. Ontological causal diagram built with OntoBioStat was compared to a standard causal diagram (without OntoBioStat) in a theoretical study. It was found that confounding and bias were not completely identified by the standard causal diagram, and erroneous covariate sets were provided. Further research is needed in order to make OntoBioStat more usable.

Ontological Models Supporting Covariates Selection in Observational Studies

In the context of causal inference, biostatisticians use causal diagrams to select covariates in order to build multivariate models. These diagrams represent datasets variables and their relations but have some limitations (representing interactions, bidirectional causal relations). The MetBrAYN project aims at building an ontological-based process to tackle these issues. The knowledge acquired by the biostatistician during a methodological consultation for a research question will be represented in a general ontology. In order to aggregate various forms of knowledge the ontology will act as a wrapper. Ontology-based causal diagrams will be semi-automatically built. Founded on inference rules, the global system will help biostatisticians to curate it and to visualize recommended covariates for their research question.

Manual Evaluation of the Automatic Mapping of International Classification of Diseases (ICD)-11 (in French)

A lexical method was used to map ICD-11 to the terminologies included in the HeTOP server. About half of ICD-11 codes (47.76%) were mapped to at least one concept. The developed tool reached a global precision of 0.98 and a recall of 0.66. Lexical methods are powerful methods to map health terminologies. Supervised and manual mapping is still necessary to complete the mapping.

Identification of the Best Semantic Expansion to Query PubMed Through Automatic Performance Assessment of Four Search Strategies on All Medical Subject Heading Descriptors: Comparative Study

Background

With the continuous expansion of available biomedical data, efficient and effective information retrieval has become of utmost importance. Semantic expansion of queries using synonyms may improve information retrieval.

Objective

The aim of this study was to automatically construct and evaluate expanded PubMed queries of the form “preferred term”[MH] OR “preferred term”[TIAB] OR “synonym 1”[TIAB] OR “synonym 2”[TIAB] OR …, for each of the 28,313 Medical Subject Heading (MeSH) descriptors, by using different semantic expansion strategies. We sought to propose an innovative method that could automatically evaluate these strategies, based on the three main metrics used in information science (precision, recall, and F-measure).

Methods

Three semantic expansion strategies were assessed. They differed by the synonyms used to build the queries as follows: MeSH synonyms, Unified Medical Language System (UMLS) mappings, and custom mappings (Catalogue et Index des Sites Médicaux de langue Française [CISMeF]). The precision, recall, and F-measure metrics were automatically computed for the three strategies and for the standard automatic term mapping (ATM) of PubMed. The method to automatically compute the metrics involved computing the number of all relevant citations (A), using National Library of Medicine indexing as the gold standard (“preferred term”[MH]), the number of citations retrieved by the added terms (”synonym 1“[TIAB] OR ”synonym 2“[TIAB] OR …) (B), and the number of relevant citations retrieved by the added terms (combining the previous two queries with an “AND” operator) (C). It was possible to programmatically compute the metrics for each strategy using each of the 28,313 MeSH descriptors as a “preferred term,” corresponding to 239,724 different queries built and sent to the PubMed application program interface. The four search strategies were ranked and compared for each metric.

Results

ATM had the worst performance for all three metrics among the four strategies. The MeSH strategy had the best mean precision (51%, SD 23%). The UMLS strategy had the best recall and F-measure (41%, SD 31% and 36%, SD 24%, respectively). CISMeF had the second best recall and F-measure (40%, SD 31% and 35%, SD 24%, respectively). However, considering a cutoff of 5%, CISMeF had better precision than UMLS for 1180 descriptors, better recall for 793 descriptors, and better F-measure for 678 descriptors.

Conclusions

This study highlights the importance of using semantic expansion strategies to improve information retrieval. However, the performances of a given strategy, relatively to another, varied greatly depending on the MeSH descriptor. These results confirm there is no ideal search strategy for all descriptors. Different semantic expansions should be used depending on the descriptor and the user’s objectives. Thus, we developed an interface that allows users to input a descriptor and then proposes the best semantic expansion to maximize the three main metrics (precision, recall, and F-measure).

Word Embedding for the French Natural Language in Health Care: Comparative Study

Background

Word embedding technologies, a set of language modeling and feature learning techniques in natural language processing (NLP), are now used in a wide range of applications. However, no formal evaluation and comparison have been made on the ability of each of the 3 current most famous unsupervised implementations (Word2Vec, GloVe, and FastText) to keep track of the semantic similarities existing between words, when trained on the same dataset.

Objective

The aim of this study was to compare embedding methods trained on a corpus of French health-related documents produced in a professional context. The best method will then help us develop a new semantic annotator.

Methods

Unsupervised embedding models have been trained on 641,279 documents originating from the Rouen University Hospital. These data are not structured and cover a wide range of documents produced in a clinical setting (discharge summary, procedure reports, and prescriptions). In total, 4 rated evaluation tasks were defined (cosine similarity, odd one, analogy-based operations, and human formal evaluation) and applied on each model, as well as embedding visualization.

Results

Word2Vec had the highest score on 3 out of 4 rated tasks (analogy-based operations, odd one similarity, and human validation), particularly regarding the skip-gram architecture.

Conclusions

Although this implementation had the best rate for semantic properties conservation, each model has its own qualities and defects, such as the training time, which is very short for GloVe, or morphological similarity conservation observed with FastText. Models and test sets produced by this study will be the first to be publicly available through a graphical interface to help advance the French biomedical research.

The Adverse Drug Reactions from Patient Reports in Social Media Project: Five Major Challenges to Overcome to Operationalize Analysis and Efficiently Support Pharmacovigilance Process

Background

Adverse drug reactions (ADRs) are an important cause of morbidity and mortality. Classical Pharmacovigilance process is limited by underreporting which justifies the current interest in new knowledge sources such as social media. The Adverse Drug Reactions from Patient Reports in Social Media (ADR-PRISM) project aims to extract ADRs reported by patients in these media. We identified 5 major challenges to overcome to operationalize the analysis of patient posts: (1) variable quality of information on social media, (2) guarantee of data privacy, (3) response to pharmacovigilance expert expectations, (4) identification of relevant information within Web pages, and (5) robust and evolutive architecture.

Objective

This article aims to describe the current state of advancement of the ADR-PRISM project by focusing on the solutions we have chosen to address these 5 major challenges.

Methods

In this article, we propose methods and describe the advancement of this project on several aspects: (1) a quality driven approach for selecting relevant social media for the extraction of knowledge on potential ADRs, (2) an assessment of ethical issues and French regulation for the analysis of data on social media, (3) an analysis of pharmacovigilance expert requirements when reviewing patient posts on the Internet, (4) an extraction method based on natural language processing, pattern based matching, and selection of relevant medical concepts in reference terminologies, and (5) specifications of a component-based architecture for the monitoring system.

Results

Considering the 5 major challenges, we (1) selected a set of 21 validated criteria for selecting social media to support the extraction of potential ADRs, (2) proposed solutions to guarantee data privacy of patients posting on Internet, (3) took into account pharmacovigilance expert requirements with use case diagrams and scenarios, (4) built domain-specific knowledge resources embeding a lexicon, morphological rules, context rules, semantic rules, syntactic rules, and post-analysis processing, and (5) proposed a component-based architecture that allows storage of big data and accessibility to third-party applications through Web services.

Conclusions

We demonstrated the feasibility of implementing a component-based architecture that allows collection of patient posts on the Internet, near real-time processing of those posts including annotation, and storage in big data structures. In the next steps, we will evaluate the posts identified by the system in social media to clarify the interest and relevance of such approach to improve conventional pharmacovigilance processes based on spontaneous reporting.

Physician satisfaction with transition from CPOE to paper-based prescription

INTRODUCTION

In January 2015, Rouen University Hospital's information system experienced serious issues. It was necessary to rapidly switch from the computerized provider order entry (CPOE) system towards a paper-based order entry (PBOE) system. This was an opportunity to evaluate prescriber opinion on the two provider order entry (POE) systems.

METHODS

All residents were asked to fill an augmented version of the POE satisfaction and usage survey for both POE systems. The results were compared to identify the strengths and weaknesses of each system.

RESULTS

Fifty-one respondents had used the CPOE system and the PBOE system. Overall, satisfaction was higher with PBOE than CPOE (odds ratio (OR)=3.74; p<0.001). Usability (OR=4.00; p<0.001), reliability (OR=8.54; p<0.001), time consumption (OR=0.50; p<0.05 - survey statement was formulated negatively), and communication with nurses (OR=14.27; p<0.0001) reached statistically better agreement. The more experience with CPOE the more residents were disillusioned with the reliability (OR=6.55; p<0.01), the usability (OR=5.68; p<0.01) and the patient safety (OR=0.27; p<0.05 - survey statement was formulated negatively) of CPOE. Although safety issues were reported for both systems, the causes were different; PBOE imposed frequent rewriting of the order while CPOE lack of usability might be unsafe. Another important issue with both POE systems was time consumption.

CONCLUSION

Residents did not report any increase in safety issues with the rapid switch from CPOE to PBOE. They even seemed more satisfied with the rollback to paper, which remains a possible degraded mode in case of health information technology collapse.

Searching for rare diseases in PubMed: a blind comparison of Orphanet expert query and query based on terminological knowledge

BACKGROUND

Despite international initiatives like Orphanet, it remains difficult to find up-to-date information about rare diseases. The aim of this study is to propose an exhaustive set of queries for PubMed based on terminological knowledge and to evaluate it versus the queries based on expertise provided by the most frequently used resource in Europe: Orphanet.

METHODS

Four rare disease terminologies (MeSH, OMIM, HPO and HRDO) were manually mapped to each other permitting the automatic creation of expended terminological queries for rare diseases. For 30 rare diseases, 30 citations retrieved by Orphanet expert query and/or query based on terminological knowledge were assessed for relevance by two independent reviewers unaware of the query's origin. An adjudication procedure was used to resolve any discrepancy. Precision, relative recall and F-measure were all computed.

RESULTS

For each Orphanet rare disease (n = 8982), there was a corresponding terminological query, in contrast with only 2284 queries provided by Orphanet. Only 553 citations were evaluated due to queries with 0 or only a few hits. There were no significant differences between the Orpha query and terminological query in terms of precision, respectively 0.61 vs 0.52 (p = 0.13). Nevertheless, terminological queries retrieved more citations more often than Orpha queries (0.57 vs. 0.33; p = 0.01). Interestingly, Orpha queries seemed to retrieve older citations than terminological queries (p < 0.0001).

CONCLUSION

The terminological queries proposed in this study are now currently available for all rare diseases. They may be a useful tool for both precision or recall oriented literature search.

Influence of author's affiliation and funding sources on the results of cohort studies on occupational cancer

BACKGROUND

Reliability and credibility of research conducted by industry have been questioned, including in the field of occupational health.

METHODS

Cohort studies on occupational cancer published between 2000 and 2010 were compared according to their results, their conclusions, their funding, and the affiliation of their authors.

RESULTS

Overall, 510 articles were included. Studies published by authors with public affiliation or funded by public grants concluded that their study showed an excess of cancer more frequently (P = 0.01) than studies published by authors with private affiliation or funded by private grants (88% [95%CI = 85-91] vs. 73% [95%CI = 56-88] and 92% [95%CI = 86-97] vs. 71% [95%CI = 57-84], respectively). Discrepancies between statistical results and conclusion occurred more frequently in articles written by authors from the private sector than from the public sector (42% [IC95% = 26-60] vs. 23% [IC95% = 18-26], P = 0.02).

CONCLUSIONS

Industry affiliations of authors or industry support of studies are associated with the results of published studies on occupational cancer. The underlying mechanisms warrant further investigation.

Knowledge Representation and Management. From Ontology to Annotation. Findings from the Yearbook 2015 Section on Knowledge Representation and Management

OBJECTIVE

To summarize the best papers in the field of Knowledge Representation and Management (KRM).

METHODS

A comprehensive review of medical informatics literature was performed to select some of the most interesting papers of KRM published in 2014.

RESULTS

Four articles were selected, two focused on annotation and information retrieval using an ontology. The two others focused mainly on ontologies, one dealing with the usage of a temporal ontology in order to analyze the content of narrative document, one describing a methodology for building multilingual ontologies.

CONCLUSION

Semantic models began to show their efficiency, coupled with annotation tools.

[Limiting a Medline/PubMed query to the "best" articles using the JCR relative impact factor]

BACKGROUND

Medline/PubMed is the most frequently used medical bibliographic research database. The aim of this study was to propose a new generic method to limit any Medline/PubMed query based on the relative impact factor and the A & B categories of the SIGAPS score.

MATERIAL AND METHODS

The entire PubMed corpus was used for the feasibility study, then ten frequent diseases in terms of PubMed indexing and the citations of four Nobel prize winners. The relative impact factor (RIF) was calculated by medical specialty defined in Journal Citation Reports. The two queries, which included all the journals in category A (or A OR B), were added to any Medline/PubMed query as a central point of the feasibility study.

RESULTS

Limitation using the SIGAPS category A was larger than the when using the Core Clinical Journals (CCJ): 15.65% of PubMed corpus vs 8.64% for CCJ. The response time of this limit applied to the entire PubMed corpus was less than two seconds. For five diseases out of ten, limiting the citations with the RIF was more effective than with the CCJ. For the four Nobel prize winners, limiting the citations with the RIF was more effective than the CCJ.

CONCLUSION

The feasibility study to apply a new filter based on the relative impact factor on any Medline/PubMed query was positive.

Evaluating alignment quality between iconic language and reference terminologies using similarity metrics

BACKGROUND

Visualization of Concepts in Medicine (VCM) is a compositional iconic language that aims to ease information retrieval in Electronic Health Records (EHR), clinical guidelines or other medical documents. Using VCM language in medical applications requires alignment with medical reference terminologies. Alignment from Medical Subject Headings (MeSH) thesaurus and International Classification of Diseases - tenth revision (ICD10) to VCM are presented here. This study aim was to evaluate alignment quality between VCM and other terminologies using different measures of inter-alignment agreement before integration in EHR.

METHODS

For medical literature retrieval purposes and EHR browsing, the MeSH thesaurus and the ICD10, both organized hierarchically, were aligned to VCM language. Some MeSH to VCM alignments were performed automatically but others were performed manually and validated. ICD10 to VCM alignment was entirely manually performed. Inter-alignment agreement was assessed on ICD10 codes and MeSH descriptors, sharing the same Concept Unique Identifiers in the Unified Medical Language System (UMLS). Three metrics were used to compare two VCM icons: binary comparison, crude Dice Similarity Coefficient (DSCcrude), and semantic Dice Similarity Coefficient (DSCsemantic), based on Lin similarity. An analysis of discrepancies was performed.

RESULTS

MeSH to VCM alignment resulted in 10,783 relations: 1,830 of which were manually performed and 8,953 were automatically inherited. ICD10 to VCM alignment led to 19,852 relations. UMLS gathered 1,887 alignments between ICD10 and MeSH. Only 1,606 of them were used for this study. Inter-alignment agreement using only validated MeSH to VCM alignment was 74.2% [70.5-78.0]CI95%, DSCcrude was 0.93 [0.91-0.94]CI95%, and DSCsemantic was 0.96 [0.95-0.96]CI95%. Discrepancy analysis revealed that even if two thirds of errors came from the reviewers, UMLS was nevertheless responsible for one third.

CONCLUSIONS

This study has shown strong overall inter-alignment agreement between MeSH to VCM and ICD10 to VCM manual alignments. VCM icons have now been integrated into a guideline search engine (http://www.cismef.org) and a health terminologies portal (http://www.hetop.eu).

How are the different specialties represented in the major journals in general medicine?

Background

General practitioners and medical specialists mainly rely on one "general medical" journal to keep their medical knowledge up to date. Nevertheless, it is not known if these journals display the same overview of the medical knowledge in different specialties. The aims of this study were to measure the relative weight of the different specialties in the major journals of general medicine, to evaluate the trends in these weights over a ten-year period and to compare the journals.

Methods

The 14,091 articles published in The Lancet, the NEJM, the JAMA and the BMJ in 1997, 2002 and 2007 were analyzed. The relative weight of the medical specialities was determined by categorization of all the articles, using a categorization algorithm which inferred the medical specialties relevant to each article MEDLINE file from the MeSH terms used by the indexers of the US National Library of Medicine to describe each article.

Results

The 14,091 articles included in our study were indexed by 22,155 major MeSH terms, which were categorized into 81 different medical specialties. Cardiology and Neurology were in the first 3 specialties in the 4 journals. Five and 15 specialties were systematically ranked in the first 10 and first 20 in the four journals respectively. Among the first 30 specialties, 23 were common to the four journals. For each speciality, the trends over a 10-year period were different from one journal to another, with no consistency and no obvious explanatory factor.

Conclusions

Overall, the representation of many specialties in the four journals in general and internal medicine included in this study may differ, probably due to different editorial policies. Reading only one of these journals may provide a reliable but only partial overview.

Mapping the ATC classification to the UMLS metathesaurus: some pragmatic applications

ATC classification is a WHO international classification used to classify drugs. The aim of this paper is to evaluate two lexical methods in English and in French to map ATC to UMLS. Several applications have been impemented to illustrate the use of the ATC mapping in English and French: (a) MeSH translation in Norwegian, (b) Drug Information Portal, and (c) ATC to PubMed tool. Two lexical methods were used to map ATC to UMLS. The first approach used a French natural language processing tool to map French terms of ATC to the French terminologies of UMLS. The second approach used the MetaMap tool to map English terms of ATC to UMLS. The English MetaMap provides slightly more mappings than the French NLP tool (3,170 vs. 2,992). On the other hand, the French NLP tool provides a slightly better precision than MetaMap (88% vs. 86%). Using a manual mapping between ATC and MeSH, the union of the validated mappings between ATC and MeSH provides 2,824 mappings (68.7% of ATC codes of the fifth level). Lexical methods are powerful methods to map health terminologies to the UMLS Metathesaurus. Manual mapping is still necessary to complete the mapping.

Evaluation of multi-terminology super-concepts for information retrieval

BACKGROUND

Following a recent change in the indexing policy for French quality controlled health gateway CISMeF, multiple terminologies are now being used for indexing in addition to MeSH®.

OBJECTIVE

To evaluate precision and recall of super-concepts for information retrieval in a multi-terminology paradigm compared to MeSH-only.

METHODS

We evaluate the relevance of resources retrieved by multi-terminology super-concepts and MeSH-only super-concepts queries.

RESULTS

Recall was 8-14% higher for multi-terminology super-concepts compared to MeSH only super-concepts. Precision decreased from 0.66 for MeSH only super-concepts to 0.61 for multi-terminology super-concepts. Retrieval performance was found to vary significantly depending on the super-concepts (p<10^-4) and indexing methods (manual vs automatic; p<0.004).

CONCLUSION

A multi-terminology paradigm contributes to increase recall but lowers precision. Automated tools for indexing are not accurate enough to allow a very precise information retrieval.

Health multi-terminology portal: a semantic added-value for patient safety

Since the mid-90s, several quality-controlled health gateways were developed. In France, CISMeF is the leading health gateway. It indexes Internet resources from the main institutions, using the MeSH thesaurus and the Dublin Core metadata element set. Since 2005, the CISMeF Information System (IS) includes 24 health terminologies, classifications and thesauri for indexing and information retrieval. This work aims at creating a Health Multi-Terminology Portal (HMTP) and connect it to the CISMeF Terminology Database mainly for searching concepts and terms among all the health controlled vocabularies available in French (or in English and translated in French) and browsing it dynamically. To integrate the terminologies in the CISMeF IS, three steps are necessary: (1) designing a meta-model into which each terminology can be integrated, (2) developing a process to include terminologies into the HMTP, (3) building and integrating existing and new inter-terminology mappings into the HMTP. A total of 24 terminologies are included in the HMTP, with 575,300 concepts, 852,000 synonyms, 222,800 definitions and 1,180,000 relations. Heightteen of these terminologies are not included yet in the UMLS among them, some from the World Health Organization. Since January 2010, HMTP is daily used by CISMeF librarians to index in multi-terminology mode. A health multiterminology portal is a valuable tool helping the indexing and the retrieval of resources from a quality-controlled patient safety gateway. It can also be very useful for teaching or performing audits in terminology management.

An automated approach to map a French terminology to UMLS

BACKGROUND

CCAM is a French terminology for coding clinical procedures. CCAM is a multi-hierarchical structured classification for procedures used in France for reimbursement in health care, which is external to UMLS.

OBJECTIVE

The objective of this work is to describe a French lexical approach allowing mapping CCAM procedures to the UMLS Metathesaurus to achieve interoperability to multiple international terminologies. This approach used a preliminary step intended to take only the significant characters used to code CCAM corresponding to anatomical and actions axes.

RESULTS

According to the 7,926 CCAM codes used in this study, 5,212 possible matches (exact matching, single to multiple matching, partial matching) are found using the French CCAM to UMLS based mapping, 65% of the corresponding anatomical terms in the CCAM code are mapped to at least one UMLS Concept and 37% of the corresponding action terms in the CCAM code are mapped to at least one UMLS Concept. For all the exact matches found (n=200), 91% were rated by a human expert as narrower than the mapped UMLS Concepts, while only 3% were irrelevant.

Is a web-based concordance test feasible to assess therapeutic decision-making skills in a French context?

BACKGROUND

At the end of undergraduate medical curriculum, a written simulation-based examination is used in France to assess therapeutic decision-making skills and to rank students for the purpose of matching their training specialties. However, this examination based on a single assessment method remains a subject of debate.

AIM

To study the feasibility of a web-based Concordance test for therapeutic decision-making assessment.

METHODS

A 12 clinical-case Concordance test was developed based on objectives for the undergraduate training program. The test was administered on line to candidates with different levels of clinical experience. Fifteen therapeutic teachers constituted the reference panel. Data analysis included analysis of variance, post-hoc test, and Cronbach's alpha.

RESULTS

One hundred and seventy participants (113 students, 34 residents, 23 physicians) fully completed the free-access test on line with no technical problems. Differences between the mean scores for groups were significant (p < 0.001). Significant differences occurred between fourth year students and residents (p < 0.001), fourth year students and physicians (p = 0.001). No difference was found between residents and physicians. Reliability coefficient was 0.67.

CONCLUSION

A web-based Concordance test in the field of therapeutic decision-making was considered feasible in a French learning environment. Further research is warranted to determine its usefulness as a part of the National Examination.

French Infobutton: an academic and business perspective

Infobuttons are context-specific links between clinical information systems and other online information resources. The objective of this study is to describe a French Infobutton, which will be sold in the French-speaking Health Information market.

Cataloguing and displaying Web feeds from French language health sites: a Web 2.0 add-on to a health gateway

Among the numerous new functionalities of the Internet, commonly called Web 2.0, Web syndication illustrates the trend for better and faster information sharing. Web feeds (a.k.a RSS feeds), which were used mostly on weblogs at first, are now also widely used in academic, scientific and institutional websites such as PubMed. As very few French language feeds were listed or catalogued in the Health field by the year of 2007, it was decided to implement them in the quality-controlled health gateway CISMeF ([French] acronym for Catalogue and Index of French Language Health Resources on the Internet). Furthermore, making full use of the nature of Web syndication, a Web feed aggregator was put online in to provide a dynamic news gateway called "CISMeF actualités" (http://www.chu-rouen.fr/actualites/). This article describes the process to retrieve and implement the Web feeds in the catalogue and how its terminology was adjusted to describe this new content. It also describes how the aggregator was put online and the features of this news gateway. CISMeF actualités was built accordingly to the editorial policy of CISMeF. Only a part of the Web feeds of the catalogue were included to display the most authoritative sources. Web feeds were also grouped by medical specialties and by countries using the prior indexing of websites with MeSH terms and the so-called metaterms. CISMeF actualités now displays 131 Web feeds across 40 different medical specialities, coming from 5 different countries. It is one example, among many, that static hypertext links can now easily and beneficially be completed, or replaced, by dynamic display of Web content using syndication feeds.

Use of super-concepts to customize electronic medical records data display

Patient medical record systems (MRS) merely offer static applications, in which mostly unstructured text is linked to coded data. In these applications the more common presentation is a time oriented one, which does not allow easily for data and information retrieval. Concept oriented views based on supper-concepts (metaterms) initially defined in CISMeF to optimize Web medical search, was implemented in our MRS as specialties views. This work shows that these terminological tools are able to facilitate information retrieval.

Using knowledge for indexing health web resources in a quality-controlled gateway

OBJECTIVES

The aim of this study is to provide to indexers MeSH terms to be considered as major ones in a list of terms automatically extracted from a document.

MATERIAL AND METHODS

We propose a method combining symbolic knowledge - the UMLS Metathesaurus and Semantic Network - and statistical knowledge drawn from co-occurrences of terms in the CISMeF database (a French-language quality-controlled health gateway) using data mining measures. The method was tested on CISMeF corpus of 293 resources.

RESULTS

There was a proportion of 0.37+/-0.26 major terms in the processed records. The method produced lists of terms with a proportion of terms initially pointed out as major of 0.54+/-0.31.

DISCUSSION

The method we propose reduces the number of terms, which seem not useful for content description of resources, such as "check tags", but retains the most descriptive ones. Discarding these terms is accounted for by: 1) the removal by using semantic knowledge of associations of concepts bearing no real medical significance, 2) the removal by using statistical knowledge of nonstatistically significant associations of terms.

CONCLUSION

This method can assist effectively indexers in their daily work and will be soon applied in the CISMeF system.

Doc’UMVF: Two search tools to provide quality-controlled teaching resources in French to students and teachers

The UMVF project is a federation of medical teaching resources covering 32 medical schools in France. Today, the indexing of these resources is carried out manually by the CISMeF team at the University Hospital of Rouen. This indexing is based on MeSH thesaurus. We use a subset of SCORM metadata standard. This choice was defined in collaboration with the French Medical Virtual University consortium (French acronym: UMVF). Currently, with the UMVF searching tool (called Doc'UMVF), medical students can reach more than 3300 resources useful in their curriculum. Doc'UMVF is developed in close collaboration between the medical informatics laboratories of Rennes and Rouen. In this paper we present two complementary searching tools based on different methods and which are integrated and used to improve both the relevance and the coverage rate of the answers. A specific searching module has been built to retrieve specific resources concerning the National Medical Exam ENC ("Examen National Classant") is also available. Nevertheless, due to lack of time, numerous resources are not yet indexed. Therefore we have decided to use also automatic indexing method (Nomindex). This approach will be improved by further research works, resulting from Rouen and Geneva teams. After having built a searching meta-motor, our objective is to develop a meta-tool intended to index the whole set of digital pedagogical resources produced by the UMVF framework. This manual re-indexing will be carried out only for the most important resources (national references), with a more or less fine granularity.

On line clinical reasoning assessment with Script Concordance test in urology: results of a French pilot study

BACKGROUND

The Script Concordance test (SC) test is an assessment tool that measures the capacity to solve ill-defined problems, that is, reasoning in a context of uncertainty. This study assesses the feasibility, reliability and validity of the SC test made available on the Web to French urologists.

METHODS

A 97 items SC test was developed based on major educational objectives of French urology training programmes. A secure Web site was created with two sequential modules: a) The first one for the reference panel to elaborate the scoring system; b) The second for candidates with different levels of experience in urology: Board certified urologists, chief-residents, residents, medical students. All participants were recruited on a voluntary basis. Statistical analysis included descriptive statistics of the participants' scores and factorial analysis of variance (ANOVA) to study differences between groups' means. Reliability was evaluated with Cronbach's alpha coefficient.

RESULTS

The on line SC test has been operational since June 2004. Twenty-six faculty members constituted the reference panel. During the following 10 months, 207 participants took the test online (124 urologists, 29 chief-residents, 38 residents, 16 students). No technical problem was encountered. Forty-five percent of the participants completed the test partially only. Differences between the means scores for the 4 groups were statistically significant (P = 0.0123). The Bonferroni post-hoc correction indicated that significant differences were present between students and chief-residents, between students and urologists. There were no differences between chief-residents and urologists. Reliability coefficient was 0.734 for the total group of participants.

CONCLUSION

Feasibility of Web-based SC test was proved successful by the large number of participants who participated in a few months. This Web site has permitted to quickly confirm reliability of the SC test and develop strategy to improve construct validity of the test when applied in the field of urology. Nevertheless, optimisation of the SC test content, with a smaller number of items will be necessary. Virtual medical education initiative such as this SC test delivered on the Internet warrants consideration in the current context of national pre-residency certification examination in France.

A web-based teaching resource to prepare for final undergraduate examination: a French pilot study

Access to accurate and quality-controlled health information on the Internet for medical students is not an easy task. CISMeF is the search tool of a MeSH-indexed directory of medical Internet resources in French. Since 2004, a new French Pre-Residency Examination (PRE) is compulsory for all medical students in the 6th year of the curriculum. The goal of this study is to evaluate CISMeF as a tool to provide teaching resources available on the Internet covering PRE material. The CISMeF terminology and the PRE CISMeF module are described. To assess the CISMeF performance in covering PRE program, its precision (number of relevant resources/number of overall resources extracted by CISMeF) and coverage (number of PRE questions covered by at least one resource in the CISMeF gateway) were computed. The CISMeF module for the new French Pre-Residency Examination is efficient as it already covers 95.7% of the program with a precision of 82.2%. Our data demonstrates that CISMeF is acceptable to guide students' learning and should be a useful teaching resource for the preparation of the French Pre-Residency Examination.

A MEDLINE categorization algorithm

Background

Categorization is designed to enhance resource description by organizing content description so as to enable the reader to grasp quickly and easily what are the main topics discussed in it. The objective of this work is to propose a categorization algorithm to classify a set of scientific articles indexed with the MeSH thesaurus, and in particular those of the MEDLINE bibliographic database. In a large bibliographic database such as MEDLINE, finding materials of particular interest to a specialty group, or relevant to a particular audience, can be difficult. The categorization refines the retrieval of indexed material. In the CISMeF terminology, metaterms can be considered as super-concepts. They were primarily conceived to improve recall in the CISMeF quality-controlled health gateway.

Methods

The MEDLINE categorization algorithm (MCA) is based on semantic links existing between MeSH terms and metaterms on the one hand and between MeSH subheadings and metaterms on the other hand. These links are used to automatically infer a list of metaterms from any MeSH term/subheading indexing. Medical librarians manually select the semantic links.

Results

The MEDLINE categorization algorithm lists the medical specialties relevant to a MEDLINE file by decreasing order of their importance. The MEDLINE categorization algorithm is available on a Web site. It can run on any MEDLINE file in a batch mode. As an example, the top 3 medical specialties for the set of 60 articles published in BioMed Central Medical Informatics & Decision Making, which are currently indexed in MEDLINE are: information science, organization and administration and medical informatics.

Conclusion

We have presented a MEDLINE categorization algorithm in order to classify the medical specialties addressed in any MEDLINE file in the form of a ranked list of relevant specialties. The categorization method introduced in this paper is based on the manual indexing of resources with MeSH (terms/subheadings) pairs by NLM indexers. This algorithm may be used as a new bibliometric tool.

Online clinical reasoning assessment with the Script Concordance test: a feasibility study

Background

The script concordance (SC) test is an assessment tool that measures capacity to solve ill-defined problems, that is, reasoning in context of uncertainty. This tool has been used up to now mainly in medicine. The purpose of this pilot study is to assess the feasibility of the test delivered on the Web to French urologists.

Methods

The principle of SC test construction and the development of the Web site are described. A secure Web site was created with two sequential modules: (a) The first one for the reference panel (n = 26) with two sub-tasks: to validate the content of the test and to elaborate the scoring system; (b) The second for candidates with different levels of experience in Urology: Board certified urologists, residents, medical students (5 or 6^th year). Minimum expected number of participants is 150 for urologists, 100 for residents and 50 for medical students. Each candidate is provided with an individual access code to this Web site. He/she may complete the Script Concordance test several times during his/her curriculum.

Results

The Web site has been operational since April 2004. The reference panel validated the test in June of the same year during the annual seminar of the French Society of Urology. The Web site is available for the candidates since September 2004. In six months, 80% of the target figure for the urologists, 68% of the target figure for the residents and 20% of the target figure for the student passed the test online. During these six months, no technical problem was encountered.

Conclusion

The feasibility of the web-based SC test is successful as two-thirds of the expected number of participants was included within six months. Psychometric properties (validity, reliability) of the test will be evaluated on a large scale (N = 300). If positive, educational impact of this assessment tool will be useful to help urologists during their curriculum for the acquisition of clinical reasoning skills, which is crucial for professional competence.

Enhancing the MeSH thesaurus to retrieve French online health resources in a quality-controlled gateway

The amount of health information available on the Internet is considerable. In this context, several health gateways have been developed. Among them, CISMeF (Catalogue and Index of Health Resources in French) was designed to catalogue and index health resources in French. The goal of this article is to describe the various enhancements to the MeSH thesaurus developed by the CISMeF team to adapt this terminology to the broader field of health Internet resources instead of scientific articles for the medline bibliographic database. CISMeF uses two standard tools for organizing information: the MeSH thesaurus and several metadata element sets, in particular the Dublin Core metadata format. The heterogeneity of Internet health resources led the CISMeF team to enhance the MeSH thesaurus with the introduction of two new concepts, respectively, resource types and metaterms. CISMeF resource types are a generalization of the publication types of medline. A resource type describes the nature of the resource and MeSH keyword/qualifier pairs describe the subject of the resource. A metaterm is generally a medical specialty or a biological science, which has semantic links with one or more MeSH keywords, qualifiers and resource types. The CISMeF terminology is exploited for several tasks: resource indexing performed manually, resource categorization performed automatically, visualization and navigation through the concept hierarchies and information retrieval using the Doc'CISMeF search engine. The CISMeF health gateway uses several MeSH thesaurus enhancements to optimize information retrieval, hierarchy navigation and automatic indexing.

Usability study on two handheld computers to retrieve drug information

Objective :Performing a usability study on two handheld computers (personal digital assistant and tablet PC), as tools for retrieving drug information.

MATERIALS AND METHODS

A randomised crossover study was performed: 34 students in pharmacy and medicine used the two handheld tools in a randomised order, to answer a questionnaire containing 12 questions covering all the aspects of a drug database and a qualitative analysis on six different items to measure access to drug information. The availability of the drug information database Vidal on PDA and on tablet PC implied our choice of the database. Three main criteria for evaluation were chosen: success rates, time-on-task, and number of clicks.

RESULTS

There were no significant differences between the two groups neither on age, sex, medical discipline, study years nor previous computer practice. The success rate is significantly higher with the PDA for only one question. The PDA is significantly faster than the tablet PC on 7 of the 12 questions and generates fewer clicks for 3 questions. Compared to the tablet PC, it appears that the PDA is better in terms of clearness, navigability and usefulness for professional practice and it is the only tool which is significantly preferred to all other supports.

CONCLUSION

In this study with students, the PDA is significantly more effective quantitatively and qualitatively than the tablet PC to retrieve drug information.

Evaluation of French and English MeSH indexing systems with a parallel corpus

OBJECTIVE

This paper presents the evaluation of two MeSH indexing systems for French and English on a parallel corpus.

MATERIAL AND METHODS

We describe two automatic MeSH in-dexing systems - MTI for English, and MAIF for French. The French version of the evaluation resources has been manually indexed with MeSH keyword/qualifier pairs. This professional indexing is used as our gold standard in the evaluation of both systems on keyword retrieval.

RESULTS

The English system (MTI) obtains significantly better precision and recall (78% precision and 21% recall at rank 1, vs. 37%. precision and 6% recall for MAIF ). Moreover, the performance of both systems can be optimised by the break-age function used by the French system (MAIF), which selects an adaptive number of descriptors for each resource indexed.

CONCLUSION

MTI achieves better performance. However, both systems have features that can benefit each other.

Seal of transparency heritage in the CISMeF quality-controlled health gateway

Background

It is an absolute necessity to continually assess the quality of health information on the Internet. Quality-controlled subject gateways are Internet services which apply a selected set of targeted measures to support systematic resource discovery.

Methods

The CISMeF health gateway became a contributor to the MedCIRCLE project to evaluate 270 health information providers. The transparency heritage consists of using the evaluation performed on providers that are referenced in the CISMeF catalogue for evaluating the documents they publish, thus passing on the transparency label from the publishers to their documents.

Results

Each site rated in CISMeF has a record in the CISMeF database that generates an RDF into HTML file. The search tool Doc'CISMeF displays information originating from every publisher evaluated with a specific MedCIRCLE button, which is linked to the MedCIRCLE central repository. Starting with 270 websites, this trust heritage has led to 6,480 evaluated resources in CISMeF (49.8% of the 13,012 resources included in CISMeF).

Conclusion

With the MedCIRCLE project and transparency heritage, CISMeF became an explicit third party.

Using CISMeF MeSH “Encapsulated” terminology and a categorization algorithm for health resources

INTRODUCTION

CISMeF is a Quality Controlled Health Gateway using a terminology based on the Medical Subject Headings (MeSH) thesaurus that displays medical specialties (metaterms) and the relationships existing between them and MeSH terms.

OBJECTIVE

The need to classify the resources within the catalogue has led us to combine this type of semantic information with domain expert knowledge for health resources categorization purposes.

MATERIAL AND METHODS

A two-step categorization process consisting of mapping resource keywords to CISMeF metaterms and ranking metaterms by decreasing coverage in the resource has been developed. We evaluate this algorithm on a random set of 123 resources extracted from the CISMeF catalogue. Our gold standard for this evaluation is the manual classification provided by a domain expert, viz. a librarian of the team.

RESULTS

The CISMeF algorithm shows 81% precision and 93% recall, and 62% of the resources were assigned a "fully relevant" or "fairly relevant" categorization according to strict standards.

DISCUSSION

A thorough analysis of the results has enabled us to find gaps in the knowledge modeling of the CISMeF terminology. The necessary adjustments having been made, the algorithm is currently used in CISMeF for resource categorization.

Level of evidence as a future gold standard for the content quality of health resources on the internet

OBJECTIVE

An assessment of the quality of health information on the Internet is an absolute necessity. In this study 'sensitive' information was defined as information found in documents published on the Internet, which could be used in a medical decision. For sensitive information, the main criterion chosen for the quality of the information was an indication of the level of evidence. A survey was conducted using the CISMeF health catalogue to assess how often a score of the level of evidence is mentioned in the information accessible on the Internet in French-language health resources.

METHODS

Since 1999, members of the CISMeF team have systematically been searching for all documents containing 'sensitive' information and verifying whether the level of evidence was explicitly indicated as a score at least once in the document.

RESULTS

As of June 2001, 10,190 resources were included in CISMeF; including 2964 textual 'sensitive' resources (29.1%). Out of all these resources, only 4.7% (95% confidence interval: 4.0 - 5.5%) indicated the level of evidence. A statistically significant difference in the prevalence of indicating the level of evidence according to resource types (e.g., 18.1% for guidelines compared to 0.0% for teaching material), year of publication (almost three times greater in 1997-2001 compared with 1990-1996) and publishers was observed.

CONCLUSION

As the number of people accessing the growing amount of information on the Internet is increasing daily, publishers have an ethical obligation to inform their readers about the validity of 'sensitive' information their sites contain. However, the vast majority of the French language Internet resources that were surveyed do not mention a score of the level of evidence for their sensitive information.

MedCIRCLE: collaboration for Internet rating, certification, labelling and evaluation of health information on the World-Wide-Web

We describe MedCIRCLE, an EU-funded semantic web project to implement the first steps towards a global, collaborative rating and guidance system for health information proposed in the MedCERTAIN project. In MedCIRCLE, three European gateway sites for consumer health information will implement the metadata vocabulary HIDDEL (Health Information Disclosure, Description and Evaluation Language). HIDDEL allows portals and gateways to make the results of their evaluations accessible as XML/RDF. The three participating national portals are: AQUMED (Agency for Quality in Medicine) patienten-information, de, COMB (Official Medical College of Barcelona) and CISMeF, a quality-controlled health gateway developed at Rouen University Hospital. Other health subject gateways, accreditation, or rating services are invited to join the collaboration simply by implementing HIDDEL on their gateways. Widespread implementation HIDDEL will allow intelligent agents or client-side software to harvest statements and opinions about the trustworthiness of other websites, assisting users in selecting trustworthy websites. The MedCIRCLE project builds on, expands and continues work on rating health information on the Internet piloted within the MedCERTAIN project. While MedCERTAIN provided the core technologies and software for rating and "trustmarking" health information, MedCIRCLE is built around these technologies and involves a wider medical community to assess health information, demonstrating the power of collaborative and interoperable evaluations in a semantic web environment. MedCIRCLE is a project with the overall objective to develop and promote technologies able to guide consumers to trustworthy health information on the Internet, to establish a global web of trust for networked health information, and to empower consumers to positively select high quality health information on the web. Other aims include refinement and expansion of HIDDEL, to become a standard vocabulary and interchange format for self- and third-party ratings of health information.

VUMeF: extending the French involvement in the UMLS Metathesaurus

A considerable number of robust vocabularies and thesauri have been developed for the healthcare and biomedical domain. No single vocabulary, however; provides complete coverage of the information needs from a public health perspective. The results of an investigation of vocabulary sources for the development of a comprehensive controlled vocabulary for the public health domain at the Centers for Disease Control and Prevention (CDC) is presented.

Reading factor: a bibliometric tool to manage a virtual library

Among the many bibliometric criteria used to evaluate biomedical journals, the impact factor is the most commonly used. Despite its limitations, it quantifies the influence of a journal on secondary publications. It does not however evaluate the practical usefulness of primary documents. Usefulness is field-related and varies greatly among specialities. We introduce a new bibliographic criterion, the "reading factor", and define it as the ratio between the number of electronic consultations of a particular journal (i.e., number of clicks on a hyper-link) and the mean number of electronic consultations of all the journals studied (itself calculated by dividing the total number of electronic accesses by the number of journals in the database). We describe its observed distribution, relative to that of the impact factor, based on electronic consultation records from our University Hospital medical digital library, where full-text electronic versions of 45 major biomedical journals have been available since December 1997. From this analysis we found no correlation between the 1999 reading factor and the 1998 impact factor of these 45 journals, and we observed a dramatic change in the hierarchy of journals upon using the reading factor as the yardstick rather than the impact factor. Moreover, we describe how using the reading factor has helped in managing the collection of our University Hospital's virtual library. The selection of journals to be discarded from the virtual library for the year 2001 was based on journals' RF values and this process will repeated over the coming years. The reading factor also permits a cost-analysis of a virtual library.

CONCLUSION

The measurement of the reading factor is highly automated, practical and efficient. It appears as a new tool for electronic collection management by librarians, well fitting with economical data.

Doc'CISMEF: a search tool based on "encapsulated" MeSH thesaurus

In the year 2000, the Internet became a major source of health information for the health professional and the Netizen. The objective of Doc'CISMeF (D'C) was to create a powerful generic search tool based on an structured information model which â encapsulates' the MeSH thesaurus to index and retrieve quality health resources on the Internet. To index resources, D'C uses four sections in its information model: 'meta-term', keyword, subheading, and resource type. Two search options are available: simple and advanced. The simple search requires the end-user to input a single term or expression. If this term belongs to the D'C information structure model, it will be exploded. If not, a full-text search is performed. In the advanced search, complex searches are possible combining Boolean operators with meta-terms, keywords, subheadings and resource types. D'C uses two standard tools for organising information: the MeSH thesaurus and the Dublin Core metadata format. Resources included in D'C are described according to the following elements: title, author or creator, subject and keywords, description, publisher, date, resource type, format, identifier, and language.

An assessment of the visibility of MeSH-indexed medical web catalogs through search engines

Manually indexed Internet health catalogs such as CliniWeb or CISMeF provide resources for retrieving high-quality health information. Users of these quality-controlled subject gateways are most often referred to them by general search engines such as Google, AltaVista, etc. This raises several questions, among which the following: what is the relative visibility of medical Internet catalogs through search engines? This study addresses this issue by measuring and comparing the visibility of six major, MeSH-indexed health catalogs through four different search engines (AltaVista, Google, Lycos, Northern Light) in two languages (English and French). Over half a million queries were sent to the search engines; for most of these search engines, according to our measures at the time the queries were sent, the most visible catalog for English MeSH terms was CliniWeb and the most visible one for French MeSH terms was CISMeF.

Automatic conceptual indexing of French pharmaceutical theses

French pharmaceutical theses are rarely quoted. If the main obstacles originate from language or access barriers, proper indexation could also be blamed. Manually extracted key-words don't necessary come from a structured thesaurus. In the following work, this manual indexing method is compared to an automated one, "Nomindex", based on UMLS. The automated method is improved by the addition of a relevance scoring system. The first indexing step consists of downloading, adapting and indexing theses in electronic format. Results will then be analyzed and sorted by relevance, through the comparison of classic statistical indices (noise, silence and relevance). It was assumed that the manually obtained key-words were always relevant. The silence of manual indexing is nevertheless high: seven new key-words are proposed by Nomindex, which results are mixed (10% of silence, but 50% of noise). These results are promising on the first experiment on pharmaceutical document without lexicon improvement. The indexing, if it is currently insufficient for a real life use, could easily be improved by specific updates of the lexicon.

A preliminary study to assess a French code of ethics for health teaching resources on the Internet

BACKGROUND

Constant assessment of the quality of health information on the Internet is an absolute necessity as peer review is often lacking in this media.

OBJECTIVE

To develop a simple and easy French Code of Ethics, which will enable medical students to judge quality of health information in teaching material available on the Internet.

DESIGN

Three medical informaticians selected ten criteria from previously established codes of ethics from Europe and the USA. This instrument was tested on a sample of 30 health Internet teaching resources.

RESULTS

For the panel of experts, chance corrected inter-observer agreement (kappa) for quality rating ranged from k = -0.19 and k = 0.33, demonstrating poor agreement among the raters.

CONCLUSION

If negative results of this preliminary study are confirmed by further research, this finding may detrimentally affect projects in Europe to accredit or certify Internet health resources.

Cost effectiveness of a medical digital library

The rapid increase in the price of electronic journals has made the optimization of collection management an urgent task. As there is currently no standard procedure for the evaluation of this problem, we applied the Reading Factor (RF), an electronically computed indicator used for consultation of individual articles. The aim of our study was to assess the cost effective impact of modifications in our digital library (i.e. change of access from the Intranet to the Internet or change in editorial policy). The digital OVID library at Rouen University Hospital continues to be cost-effective in comparison with the interlibrary loan costs. Moreover, when electronic versions are offered alongside a limited amount of interlibrary loans, a reduction in library costs was observed.

A search tool based on 'encapsulated' MeSH thesaurus to retrieve quality health resources on the internet

In the year 2001, the Internet has become a major source of health information for the health professional and the Netizen. The objective of Doc' CISMeF (D'C) was to create a powerful generic search tool based on a structured information model which 'encapsulates' the MeSH thesaurus to index and retrieve quality health resources on the Internet. To index resources, D'C uses four sections in its information model: 'meta-term', keyword, subheading, and resource type. Two search options are available: simple and advanced. The simple search requires the end-user to input a single term or expression. If this term belongs to the D'C information structure model, it will be exploded. If not, a full-text search is performed. In the advanced search, complex searches are possible combining Boolean operators with meta-terms, keywords, subheadings and resource types. D'C uses two standard tools for organising information: the MeSH thesaurus and the Dublin Core metadata format. Resources included in D'C are described according to the following elements: title, author or creator, subject and keywords, description, publishers, date, resource type, format, identifier, and language.