Semantic Features: We used OsiriX[1] to review CT studies and to identify lesions with a Region-of-Interest (ROI), and a custom-developed plug-in, called image Physician Annotation Device (iPAD),[2] to annotate them with semantic features. Users are prompted to describe lesions according to 12 categories of semantic description, enabling a thorough description of each lesion. As the user types, iPad checks on-the-fly for direct mapping to 161 terms from an augmented list of RadLex terms. The resulting coded annotation is stored in a file compliant with the AIM (Annotation and Image Markup) standard, established by the National Cancer Institute’s Cancer Bioinformatics Grid (CaBIG),[3,4] and uploaded to a database.

Texture Features: We computed multiple features for each lesion based on its pixels, including (a) gray-level histogram-based: 14 features including the histogram itself , the low frequency coefficients of its 3-level Haar wavelet transform, the abscissa of its peak, and its variance,[5] and (b) Gabor features:[6] mean of the energy in the frequency domain over 4 scales and 8 orientations in each of 32 bins, for a total of 46 features.

Boundary Features: We computed features for boundary sharpness by bi-linearly interpolating profiles along radial line segments that are automatically drawn at many angles from the center outwards towards a dilated version of the ROI boundary. We characterized the difference in intensity between the lesion and the surrounding liver and the sharpness of the margin along this profile using parameters derived by fitting a sigmoid function to each intensity profile, and averaging the parameters across all profiles.

Similarity Metric: All image features described above were used to create a feature vector describing each lesion. Similarity between any pair of images was computed as the negation of a weighted sum of differences between corresponding elements of two feature vectors. Weights were derived using a machine-learning method that maximizes the retrieval performance, as measured by Normalized Discounted Cumulative Gain (NDCG: see Evaluation Metrics, below) compared to a separate Similarity Standard (see below).

Patient Data: We selected 81 portal venous phase CT images from patients demonstrating liver lesions, including 25 cysts, 24 metastases, 14 hemangiomas, 7 HCCs, 6 focal nodular hyperplasias, 3 abscesses, 1 laceration, and 1 focal fat deposit. One radiologist used OsiriX and iPAD to annotate each lesion as described above, followed by completion of each feature vector using automatically derived texture and boundary features.

Similarity Standard: We created a Web-based tool that randomly selects an image from the database and asks the user to rate 5 lesion characteristics (heterogeneity, boundary shape, margin, density, and rim) on a 5-point scale, and computes visual similarity based on the difference between ratings of each characteristic. Five readers used this tool to rate the 81 CT images of lesions. Characteristic-combining weights were derived using a bootstrap approach, optimizing the weights to match results in a subset of the database for which a degree of general visual similarity was agreed upon by a consensus of two radiologists.

Evaluation Metrics: To evaluate performance, we used Precision-Recall (P-R), which plots the number of similar images (defined as having an average similarity score in the Similarity Standard of 3.0 or greater) retrieved divided by the total number of images retrieved vs. the number of similar images retrieved divided by the total number of similar images in the database.[7] As P-R does not account for graded truth, as in our 5-point characteristic similarity scale, nor does it account for the order of retrieved images in a ranked list, we also used Normalized Discounted Cumulative Gain (NDCG) as a function of the number of images, K, desired and retrieved.[8] At a given K, higher NDCG(K) means more lesions similar to the query image are ranked ahead of dissimilar ones, with NDCG(K)=1 implying perfect retrieval of K images. Both analyses were conducted by withholding each lesion in turn from the database as the query image, training the weights used in the Similarity Metric described above using the remaining 80 images together with the Similarity Standard, and retrieving images in order of their similarity to the query images using the same metric with the trained weights.

Figure 1: Precision-recall plot, showing average performance (circles; error bars indicate 1 s.d.) in our set of 81 portal venous liver CT images containing various types of lesions. Note excellent performance with average precision > 90% at all values of recall.

Figure 2: Normalized Discounted Cumulative Gain plot showing average gain (circles; error bars indicate 1 s.d.) as a function of the number, K, of top-ranked retrieved images using our 81-lesion database. Note that the average gain is greater than 90% for all queries retrieving 3 or more images.

Figure 3: Retrieval example with small cyst as query lesion, with integer rankings and similarity scores in parentheses above each ranked image. The image matrix shows the 11 images with the most similar lesions (top two rows) and the 12 least similar ones (bottom 2 rows) from our 81-lesion database.