by Aakash Agrawal, Postdoctoral fellow, Cognitive neuroimaging lab, Collège de France & Aniruddha Mundhada, MD, Dhruv Diagnostics



Content based Image retrieval systems or CBIR has gained a lot of attention recently mainly because of two factors:  1) exciting developments in the field of computer vision over the last 10 years and the availability of large and steadily growing amounts of visual and multimedia data, with approximately 2 GB of data along with metadata is produced for every whole slide scan (WSI) and 2) the development of the Internet underline the need to create thematic access methods that offer more than simple text-based queries or requests based on matching exact database fields. Many programs and tools have been developed to formulate and execute queries based on the visual or audio content and to help browsing large multimedia repositories. However, no general breakthrough has been achieved with respect to large, varied databases with documents of differing sorts and with varying characteristics. Future work and research is needed in respect to speed, semantic descriptors or objective image interpretations to make CBIR more robust and practical.


CBIR is a computer vision technique that can be used to query a desired output from a collection of digital image databases. This technique is widely applied across multiple domains including medical image diagnosis. The CBIR technique doesn’t depend on any labels and instead relies on visual similarity judgement between a given pair of images. This is a desirable property especially for datasets where it is either difficult to assign semantically meaningful labels (e.g., patterns), or can only be provided by domain experts (pathologists).


The process of examining tissues, organs, fluids, etc. involves visually inspecting them under a microscope. This process is tedious, time consuming, and prone to errors. Recently, pathological labs are adopting latest technologies to digitize the slides and viewing it on a computer. This has significantly reduced the effort involved with using a microscope. There is a potential to further improve the experience of pathologists by exploiting the advances in computer vision. Over the last decade, the computer image classification accuracy has improved manyfold, mainly due to availability of large numbers of labelled images, and computational power. However, the neural networks trained on classification tasks cannot be generalized easily to detect malicious cells due to huge diversity within each disease condition. Further, the data from rare disease conditions are sparse, which affects the model’s performance on those categories.



The clinical use of CBIR needs to address a need that an average pathologist cannot be expected to cover based on existing infrastructure or knowledge gained over training. One aspect of it is the accuracy for recall that supersedes that of an average pathologist; indeed, it should be magnitudes better than a pathologist’s interpretations, as then only may it be expected to be used in clinical practice.


The application of AI can be used to index the vast number of images and then categorically sort them based on feature points. This involves clever engineering because the algorithm that sorts these need to do so in a way clinically useful. The repository so created can be used to search for and browse to match new images; this will benefit the pathologist and clinician together. Another way that these algorithms can screen for images and look for common patterns in the archive- is to cross reference the current image from other pathology centers and help the pathologist make a correct decision and diagnosis.


Briefly, CBIR works by first extracting the features of an image, and then comparing it with features of other images in the dataset. Classical approaches used low-level image properties such as color histograms, texture frequency, shape etc. as image features, which yielded poor results. However, the advent of deep neural networks has enabled extraction of a more complex set of features, which has drastically improved CBIR performance on standard benchmark challenges. The next step of comparison is typically performed by estimating the distance (Euclidean, cosine, Mahalanobis, etc.)  between a pair of feature vectors. While conceptually simple, there are quite a few challenges that need to be addressed such as 1) real-time performance for a very large database (> 1 million images), 2) identifying the region of interest for analysis, 3) Scale of feature extraction (global vs local patterns), 4) Identifying the desired set of features (e.g., shape vs color), 4) computational cost (memory, GPUs).


Existing examples of CBIR focus on small subsets of classes- for example 4 histopathological classes of cancer. One of the examples is the Luigi dataset that has image patches from representative sections of the cancer tissue from the TCGA dataset.


There is active research on how to make CBIR more clinically relevant to pathologists by grouping images by semantics along with visually similar images. Yottixel, an image search algorithm recently proposed, aims to do that for a very large histopathological database.


Other well-known CBIR is the Google SMILY database for histopathology, IRMA, YottaLook, NHANES II, FIRE (flexible image retrieval engine), and few in radiology as well- RadLex (Radiology Lexicon) and ASSERT (for HRCT images in radiology).



The widespread use into mainstream laboratory medicine for CBIR will be to overcome the two following hurdles:


The business model should address the cost/benefit of CBIR. This could be delivered as a subscription, with access purchased for a period of time, say yearly. The access would be billed to every pathology diagnosis which uses the repository, with the cost added to the final bill of pathology services; and with reimbursement approved by existing insurance companies.


The other hurdle is acceptance into the normal workflow of pathologists as a companion diagnostic tool. This will only happen when CBIR proves its worth as a tool that helps to add to information useful in the diagnosis and management of patients compared to information that is missing from viewing through a conventional microscope.  The accuracy of such systems can be further enhanced by having a feedback system for points relevant for final classification and this has increased relevance for final results. This seems to be a good strategy for higher level multidimensional data where the features are numerous.



Initially, computational requirements required weeks to at least minutes of accessing the gigabytes of data in the form of whole slide images, which due to advancements in high performance computing has brought down the retrieval time from minutes to seconds making CBIR a practical proposition.


Other limitations include varied pathological categories with varied clinical management, bad quality of images, improper segmentation of images, noise, and differences in the modalities used.


One more aspect that has been seen is that being better than a pathologist only in diagnosis is not enough. It must add value to the already existing knowledge in a laboratory, in the form of therapeutics or management and prognosis of the patient.


The validation of CBIR is to use the pathologist’s vast amounts of experience to correctly classify and validate the diagnosis. This is a silver lining in itself as it means that this technology is to supplement the work of pathologists, and not to replace them.



CBIR is a rich, useful tool to query for images in a visual-specific field like pathology, where language query is not a very appropriate tool and will lead to a subjective bias. Also, it will be a pathologist-centric model where the pathologist makes the final decision as compared to a machine. The increasing research in the medical domain points to the importance of it in mainstream use.


The main limitations of variation in reporting due to subjectivity can also potentially be tackled by use of CBIR; and its benefits such as feature extraction from noisy images, algorithm training, and business efficiencies will ease the ushering in of CBIR as a companion diagnostic tool.



  1. Building SMILY, a Human-Centric, Similar-Image Search Tool for Pathology ; Accessed 20 February 2021
  2. Kalra, S., Tizhoosh, H.R., Choi, C., Shah, S., Diamandis, P., Campbell, C.J. and Pantanowitz, L., 2020. Yottixel–an image search engine for large archives of histopathology whole slide images. Medical Image Analysis, 65, p.101757.
  3. Camelyon Grand Challenges Website:
  4. Satish, B., 2016. A Methodical Study of Content Based Medical Image Retrieval in Current Days. Global Journal of Computer Science and Technology.
  5. Tavolara, T.E., Niazi, M.K.K., Arole, V., Chen, W., Frankel, W. and Gurcan, M.N., 2019. A modular cGAN classification framework: Application to colorectal tumor detection. Scientific reports, 9(1), pp.1-8.