Multimodal medical case retrieval using the Dezert-Smarandache theory

Most medical images are now digitized and stored with semantic information, leading to medical case databases. They may be used for aid to diagnosis, by retrieving similar cases to those in examination. But the information are often incomplete, uncertain and sometimes conflicting, so difficult to use. In this paper, we present a Case Based Reasoning (CBR) system for medical case retrieval, derived from the Dezert-Smarandache theory, which is well suited to handle those problems. We introduce a case retrieval specific frame of discernment θ, which associates each element of θ with a case in the database; we take advantage of the flexibility offered by the DSmT's hybrid models to finely model the database. The system is designed so that heterogeneous sources of information can be integrated in the system: in particular images, indexed by their digital content, and symbolic information. The method is evaluated on two classified databases: one for diabetic retinopathy follow-up (DRD) and one for screening mammography (DDSM). On these databases, results are promising: the retrieval precision at five reaches 81.8% on DRD and 84.8% on DDSM.


INTRODUCTION
In medicine, the knowledge of experts is a mixture of textbook knowledge and experience through real life clinical cases.Consequently, there is a growing interest in case-based reasoning (CBR), introduced in the early 1980s, for the development of medical decision support systems .The underlying idea of CBR is the assumption that analogous problems have similar solutions, an idea backed up by physicians [1] ' experience.In CBR, the basic process of interpreting a new situation revolves around the retrieval of relevant cases in a case database.The retrieved cases are then used to help interpreting the new one.
We propose in this article a CBR system for the retrieval of medical cases made up of a series of images with contextual information: a class of CBR problems which has hardly been treated.The proposed system is applied to a Diabetic Retinopathy (DR) multimedia database built up in our laboratory; to diagnose DR, physicians analyze series of multimodal photographs together with contextual information such as the patient age, sex and medical history.To show that the method is generic, we also applied it to DDSM, a public access database for screening mammography; to screen mammography, physicians analyze two views of each breast, with associated contextual information.
When designing a CBR system to retrieve such cases, several problems arise.We have to aggregate heterogeneous sources of evidence (images, contextual information) and to manage missing information.These sources may be uncertain and conflicting.As a consequence, we applied the Dezert-Smarandache Theory (DSmT) of plausible and paradoxical reasoning, proposed in recent years , which is well suited to [2] fuse uncertain, highly conflicting and imprecise sources of evidence.

Databases
the diabetic retinopathy (DR) database contains retinal images of diabetic patients, with associated Diabetic retinopathy database: anonymous information on the pathology.Diabetes is a metabolic disorder characterized by sustained inappropriate high blood sugar levels.This progressively affects blood vessels in many organs, which may lead to serious renal, cardiovascular, cerebral and also retinal complications.Different lesions appear on the damaged vessels, which may lead to blindness.The database is made up of 63 patient files containing 1045 photographs altogether.Images have a definition of 1280 pixels/line for 1008 lines/image.They are lossless compressed images.Patients have been recruited at Brest University Hospital since June 2003 and images were acquired by experts using a Topcon Retinal Digital Camera (TRC-50IA) connected to a computer.An example of an image series is given in .figure 1 The contextual information available is the patients age and sex and structured medical information (about the general clinical context, the ' diabetes context, eye symptoms and maculopathy).Thus, at most, patients records are made up of 10 images per eye (see ) and of 13 figure 1 contextual attributes; 12.1 of these images and 40.5 of these contextual attribute values are missing.The disease severity level, according to % % ICDRS classification , was determined by experts for each patient.The distribution of the disease severity among the above-mentioned 63 [3] patients is given in .table I the DDSM project has built a mammographic image database for research on Digital Database for Screening Mammography (DDSM): [4] breast cancer screening.It is made up of 2277 patient files.Each one includes two images of each breast, associated with some patient information (age at time of study, rating for abnormalities, American College of Radiology breast density rating and keyword description of abnormalities) and imaging information.The following contextual attributes are taken into account in the system: the age at time of study breast density rating Images have a varying definition, of about 2000 pixels/line for 5000 lines/image.An example of image sequence is given in .Each figure 2 patient file has been graded by a physician.Patients are then classified in three groups: normal, benign and cancer.The distribution of grades among the patients is given in .table I

Including images in the retrieval system
To include images in the proposed retrieval system, we have to define a distance measure between images and to cluster images acquired at a given imaging modality into a finite number of groups.For this purpose, we follow the usual steps of Content-Based Image Retrieval (CBIR) : 1) building a signature for each image (i.e.extracting a feature vector summarizing their numerical content), and 2) defining a distance [5] measure between two signatures.Thus, measuring the distance between two images comes down to measuring the distance between two signatures.We can then cluster similar image signatures according to the defined distance measure.
In previous studies, we proposed to compute a signature for images from their wavelet transform (WT) .These signatures model the [6] distribution of the WT coefficients in each subband of the decomposition.The associated distance measure computes the divergence d [6] between these distributions.We used these signature and distance measure to cluster similar images.
Any clustering algorithm can be used, provided that the distance measure between feature vectors can be specified.We used FCM (Fuzzy ) , one of the most common algorithms, and replaced the Euclidian distance by .C-Means [7] d

Dezert-Smarandache Theory
The Dezert-Smarandache Theory (DSmT) allows to combine any type of independent sources of information represented in term of belief functions.It is more general than probabilistic fusion or Dempster-Shafer theory (DST).It is particularly well suited to fuse uncertain, highly conflicting and imprecise sources of evidence .
[2] let , , be a set of hypotheses under consideration for a fusion problem; is called the frame of discernment.The fusion model: In DST, these hypotheses are assumed incompatibles (constrained model ( )), while in DSmT they are not (free model ( )): if blue , red , we may model objects that are both blue and red (magenta).Thus, in DSmT, a belief mass ( ) is assigned to each element of the " " "} m A A hyper-power set ( ), i.e. the set of all composite propositions built from elements of with and operators, such that ( ) 0 and ( ) 1; is called a generalized basic belief assignment (gbba).Nevertheless, it is possible to introduce constraints in the model (hybrid model ( )): we can specify pairs of incompatible hypotheses ( , ), i.e. each subset of must have a null mass, noted θ to fuse information in the DSmT framework, the user specifies a gbba for each source The fusion operators and the decision functions: m j of evidence , 1.. .Then, these gbba are fused into a global gbba , according to a given rule of combination.Several rules have been S j = N m f proposed to combine mass functions, including the hybrid rule of combination or the PCR (Proportional Conflict Redistribution) rules . [2] once the fused gbba has been computed, a decision function (such as the credibility, the plausibility or the The decision functions: m f pignistic probability) is used to evaluate the probability of each hypothesis.The pignistic probability , a compromise between the other BetP two functions, is used since it provides the best system performance: ( ) 4 and Dezert-Smarandache Theory Based Retrieval let be a case placed as a query by the user.We want to rank the cases in the database by decreasing order of Outline of the method: c q relevance for .In that purpose, for each case in the database, 1.. , we estimate the degree of relevance of for , noted ( , ); c q c i i = M c i c q DR c i c q then we rank the cases by decreasing order of ( , ), and the top five results are returned to the user.To compute ( , ), we see c i DR c i c q DR c i c q each case descriptor as a source of evidence: the degrees of relevance are first estimated for each case feature , 1.. , noted ( , ) F j j = N DR j c i c q (as described in section II-D.3), they are then translated into gbba (see section II-D.4) which are fused in the DSmT framework.Finally, ( , DR c i ) is estimated from the fused gbba by the pignistic probability (see ).The procedure is summed up in .c q equation 1 figure 5 the following frame of discernment is used for the fusion problem: , where is the hypothesis is The model: . .The cardinal of ( ) is hyper-exponential in (it is majored by 2 ).As a consequence, from computational considerations, it is necessary to include constraints in the model.These constraints are also justified by logical considerations.Indeed, it is not possible for the case to be similar to both cases and if c q c a c b and are dissimilar, In those cases, we set ( ) .To build the model ( ), we first define a non-oriented graph ( ), that we call compatibility graph; each vertex represents an hypothesis and each edge a couple of compatible hypothesis.To build , we link v ∈ V e ∈ E G c the hypothesis associated with each case to the hypothesis associated with its nearest neighbors at the same severity level.The distance c i l measure used to find the nearest neighbors is simply a linear combination of heterogeneous distance functions (one for each case feature -F j see section II-B for images), managing missing values .We noticed that the complexity of the fusion operators mainly depends on the cardinality of the largest clique in (a clique is a set of vertices such that for every two vertices in , there exists an edge connecting the G c V V two).The number is closely related to the cardinality of the largest clique in and consequently to the complexity of the fusion operation.l G c Finally, the hybrid model ( ) is built from by identifying all the cliques in (see ).
ℳ θ G c G c figure 4 to compute the degree of relevance ( ) that a case is relevant for , given the feature , we first The degree of relevance: DR j c , c i q c i c q F j define a finite number of states for .Then we compare the membership degree of ( ( )) and of ( ( )) to each state .If is a f jk F j c i α jk c i c q α jk c q f jk F j discrete variable, we associate a state with each possible value for and is a Boolean function.If is an image attribute, ( ) is the F j α jk    Robustness with respect to missing values.Note that cases are returned at random when no attributes are inputted (0 on the X axis).

F j α jk c membership degree ofFig. 1 Fig. 2
Fig. 1 Photograph series of a patient eye Images (a), (b) and (c) are photographs obtained by applying different color filters.Images (d) to (j) form a temporal angiographic series: a contrast product is injected and photographs are taken at different stages (early (d), intermediate (e) (i) and late (j)).-

Fig. 3
Fig. 3 Considering the frame of discernment , , , the figure represents, from left to right, the Venn diagram of the constrained, the free and a θ = {θ 1 θ 2 θ 3 } hybrid model (in which is incompatible with the other two hypotheses).A non-empty intersection between circles means the corresponding θ 3 hypotheses are compatible.

Fig. 4
Fig. 4 Building the model ( ) from the compatibility graph.An example of compatibility graph is shown on figure (a).Hypotheses associated with ℳ θ cases at different severity levels are represented with different colors.

Fig. 5
Fig. 5Outline of the method