Devices & Hardware

NovaMedSearch: A multimodal search engine for medical case-based retrieval

NovaMedSearch: A multimodal search engine for medical case-based retrieval
of 2
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Related Documents
  NovaMedSearch: A multimodal search engine for medicalcase-based retrieval André Mourãoa.mourao@campus.fct.unl.ptFlávio Departamento de InformáticaFaculdade de Ciências e TecnologiaUniversidade Nova de LisboaCaparica, Portugal ABSTRACT Medical information retrieval systems help support healthcare experts in diagnostic and treatment decisions throughthe management of large amounts of clinical data. However,the heterogeneity and the ever growing amount of data pro-duced in medical environments poses several challenges.In this paper, we propose a multimodal search interfacefor medical articles to provide better analysis tools for diag-nosis and medical case retrieval. The underlying frameworkentered the Medical ImageCLEF 2013 challenge and is basedon state of the art information retrieval, image retrieval anddata fusion techniques. Keywords search, image retrieval, search interfaces, medical retrieval 1. INTRODUCTION Search engines are often the main aids for case-based re-trieval for medical personnel activities including diagnostic.Textual search is the norm, but images can often provideadditional information that is difficult to convey in textualqueries. In medical information retrieval, additional infor-mation regarding the modality classification of the imagescan be conveyed from the query images (e.g. charts vs. X-rays) and information that is hard to writeup (e.g. the po-sition of a mass on a MRI). In this demo, we present NovaMedSearch, a medical search engine that integrates the twosearch modalities: text and image. Our goal is to providean intuitive and simplified way of supporting multimodalqueries in medical search. 2. SEARCH ENGINE Nova MedSearch is a multimodal (text and image) medi-cal search engine that can retrieve either similar images orrelated medical cases. The user can upload his own imagesto build their query or use existing sample images in their Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee. OAIR  ’13 May 22-24, 2013, Lisbon, PortugalCopyright 2013 CID 978-2-905450-09-8 Figure 1: Interface mockup. queries. The results are displayed in an ranked list with ba-sic information (e.g. title, keywords, images (if available))and a link to the corresponding article details. The inter-face takes into account both the relevancy of the images andtext similarity. Previous work exists on multimodal searchengines such as MedGIFT [3] (text or image based searchfor medical articles) and img(Anaktisi) [6] (image based re-trieval on multiple (i.e. medical) datasets)) two previouslyexisting systems.Our interface aims at simplifying use of images and textsimultaneous, both in queries and in the presentation of theresults. For instance, for custom queries we use native dragand drop support in modern browsers for image uploading.We have also implemented a guided query expansion sys- 223  Figure 2: Framework overview. tem that interactively provides auto-complete suggestionsand expansion feedback sourced from a SKOS version of theMeSH indexing terms. A mockup of the diferent componentsof the interface is in Figure 1. The query box (a) mockupcontains a textbox to enter the text queries, emphasising theguided expansion interface. In the example, we see the termsthat will also match when searching for the word ¨osteoporo-sis¨. There is also an area to drop query images. The articlesearch result mockup (b) contains a sample result for thequery example above. Besides general article information(title with link to full article, date, venue and abstract), wedisplay the images in the article and their modalities (e.g.x-ray, MIR). The image search result mockup (c) containsthe best match for the query and information regarding therelated article (including other images). 2.1 Framework The framework (see Figure 2) behind the search engine isbased on three major components: image processing, natu-ral language processing and multimodal retrieval. The sys-tem is described in detail on our working notes of the medicalImageCLEF 2013 [4]. The image processing component isbased on the analysis and retrieval of images using CEDD,FCTH, [2], Local Binary Pattern histograms [1] and colorhistograms of segmented image. We extract the features forthe images and the dataset and store them on a FLANNindex [5]. The retrieval is based on the  L 2  distance betweenthe features of the query images and the features present onthe dataset. Modality classification is also performed usingthe same features and an SVM. The text is indexed usingLucene and Lucene-SKOS is used to add the query expan-sion feature. The combination of the image and text resultsis based on a late fusion approach. The text and imagesare analyzed separately and the results are combined in arank-based CombSUM variant. We found that the late fu-sion of the results was useful for heterogeneous queries (e.gonly text, only 1 image, text and 3 images), as the combi-nation of the image and text search can be ignored if thequery does not contain images. 2.2 Technologies The Nova MedSearch web application is developed in Javaand C ++  and modern HTML5 browser technologies. On theserver side, the application is based on Apache Solr, Drop-wizard REST framework to support the retrieval webser-vices for images and text. The underlying operating systemis Ubuntu 12.04 LTS. 2.3 Demo In the demo, we display a fully functioning web basedinterface for our medical search engine. The users are ableto make textual queries, image queries and combined queries(text or images). The users can upload custom images andwe provide a set of sample combined queries. The results areeither images or cases (medical articles). The type of queriesfollow the same structure of the Medical ImageCLEF. 3. CONCLUSIONS Our system combines a powerful framework based on stateof the art image and text processing algorithms with a sim-ple yet powerful multimodal search interface to provide avaluable tool to the retrieval of medical data. 4. REFERENCES [1] T. Ahonen, A. Hadid, and M. Pietik¨ainen. Facedescription with local binary patterns: application toface recognition.  IEEE transactions on pattern analysis and machine intelligence  , 28(12):2037–41, Dec. 2006.[2] S. A. Chatzichristofis, K. Zagoris, Y. S. Boutalis, andN. Papamarkos. Accurate Image Retrieval Based onCompact Composite Descriptors and RelevanceFeedback Information.  International Journal of Pattern Recognition and Artificial Intelligence (IJPRAI) ,24(2):207 – 244, 2010.[3] MedGIFT Group. medSearch - Medical search engineby HES-SO Valais, 2009.[4] A. Mour˜ao, F. Martins, and J. Magalh˜aes. NovaSearchon medical ImageCLEF 2013. In  Working Notes of CLEF 2013  , pages 1–10, 2013.[5] M. Muja and D. Lowe. Fast approximate nearestneighbors with automatic algorithm configuration. In International Conference on Computer Vision Theory and Application VISSAPP’09) , volume 340, pages331–340. INSTICC Press, 2009.[6] K. Zagoris, S. A. Chatzichristos, N. Papamarkos, andY. S. Boutalis. img(Anaktisi): A Web Content BasedImage Retrieval System. In  2009 Second International Workshop on Similarity Search and Applications  , pages154–155. IEEE, Aug. 2009. 224
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks