Weekly Paper Review: How AI is helping to detect Macula Edema and Subretinal Fluid in OCT scans.

7 min readJun 19, 2020

This week I read the paper titled: “Mobile Artificial Intelligence Technology for Detecting Macula Edema and Subretinal Fluid on OCT Scans: Initial Results from the DATUM alpha Study” by Stephen G. Odaibo, M.D., M.S.(Math), M.S.(Comp. Sci.) et al., (2019).

The main objective of the study was to evaluate the feasibility of cloud-based mobile Artificial Intelligence (AI) for detection of retinal diseases. In addition, the authors sought to evaluate the accuracy of such system for detection of Subretinal Fluid (SRF) and Macula Edema (ME) on OCT scans.

Preamble

The number of retina specialists in the world is not an encouraging one. For example in the US, there are under 3,000 retina specialists as at July 1st, 2018 as reported by US census Bereau.

I was shocked to see that in Nigeria there were less than ten (10) fellowship-trained retina specialists in practice. Note that Nigeria’s population is over 190 million.

Was that shocking?

Bolivia had just one (1) fellowship-trained retina specialist. Personally, I think the issue is not about getting more people trained to become retina specialists.

This means that there needs to be another solution to supplement the available retina specialists for a wider reach. Fortunately, AI can help as a supplement to the retinal care system and the healthcare system as a whole.

Lest I forget, I will be speaking at ODSC webinar Nigeria Team on Friday, June 26th, 2020 at 4:00pm (GMT +1) on the topic: “AI-Empowered solutions for quality assessment and control in healthcare delivery”.

You can register here: http://meetu.ps/e/J4Mbt/Hm0pJ/d

Now, let us start checking exciting areas of the paper.

Problem Statement

A key problem the study aimed to solve was the means and mechanism through which AI-driven diagnostic capability were distributed in areas where it was needed the most (it is observed that retina specialists are clustered in urban areas with few in the rural and non-coastal states).

To deal with this challenge, the first mobile app for eye care providers called Fluid Intelligence was used as case study.

Methods

About the mobile app

Fluid Intelligence is an assist device for diagnosing the presence or absence of Macula edema or subretinal fluid.

What is Macula edema?
Macular edema is the build-up of fluid in the macula, an area in the center of the retina. The retina is the light-sensitive tissue at the back of the eye and the macula is the part of the retina responsible for sharp, straight-ahead vision. Fluid buildup causes the macula to swell and thicken, which distorts vision. (NIH Website)
What is subretinal fluid?
Subretinal fluid corresponds to the accumulation of a clear or lipid-rich exudate (serous fluid) in the subretinal space, i.e., between the neurosensory retina (NSR) and the underlying retinal pigment epithelium (RPE), in the absence of retinal breaks, tears, or traction (Kanski et al. 2011). It represents a breakdown of the normal anatomical arrangement of the retina and its supporting tissues, i.e., the RPE, Bruch’s membrane, and the choroid.

In order to solve the issue of distribution to areas where it is needed. The app platform provides an opportunity for cloud-based AI distribution method. The iOS operating system version of the app was used on iPhone 6 services.

From within the app, the iPhone camera modality was used by study investigators to take pictures of OCT scans of the retina displayed on computer monitor.

Participants

The study investigators were board-certified ophthalmologists who have completed retina fellowships (retina specialists). The study investigators were allowed to choose OCT scans that satisfied these exclusion criteria. The following types of OCT images were excluded from the study:

poor quality images
macula holes
dense epiretinal membranes
macula retinoschisis
vitreomacular traction syndrome, and
subtle or ambiguous images.

Anonymized images were stored in the cloud and assessed for compliance with eligibility criteria. An image eligibility check was done on all the centers’ submissions.

Analysis

In summary, the board-certified ophthalmologists noted the presence or absence of macula edema or subretinal fluid, then compared their assessment to that obtained from Fluid Intelligence, a mobile AI app that detects SRF and ME on OCT scans.

The Approach

Study investigators selected images retrospectively (with consideration of past events or situations), and in consecutive fashion. They made sure to balance the number of OCT images with subretinal fluid (SRF) or macula edema (ME) termed as “wet” against images without SRF or ME termed as “dry”.

I think the “wet” class is the positive class, that is, the class of images with SRF or ME

Upon selecting an image, an assessment of the image was made by the investigator, after which the investigator used the app to determine the AI’s assessment. If the AI app’s assessment of a “wet” image corresponded to the retina specialist’s, it was deemed a true positive.

The above statement agrees with my “thinking” . Again, the “wet” is the positive class.

If the AI app was in agreement with the retina specialist on a “dry” image, it was deemed a true negative.

If the AI app assessed an image as “wet” but the retina specialist assessed it as “dry,” it was deemed a false positive. And if the AI app assessed an image as “dry” but the retina specialist assessed it as “wet” it was deemed a false negative.

Furthermore, a model was trained on a data set consisting of 1 million OCT images after augmentation. The training was done with a deep learning model with an imageNet-trained inception base. Basically, transfer learning was completed after which the trained model was hosted in the cloud.

I hope you still remember the intuition behind transfer learning?
Transfer learning is a machine learning technique where a model trained on one task is re-purposed on a second related task.

Finally, a front-end app for iOS was developed and deployed into the App Store. The front-end was connected to the cloud service to serve the trained machine learning model during inference. An intermediate noSQL data base archived the images sent to the cloud server for inference.

Results

At the time the paper was submitted, five centers had completed their initial studies. This consists of a total of 283 OCT scans of which 155 had either ME or SRF (“wet”) and 128 did not (“dry”). The sensitivity ranged from 82.5% to 97% with a weighted average of 89.3%. The specificity ranged from 52% to 100% with a weighted average of 81.23%. The figures below shows sensitivities and specificities of individual centers.

I hope you remember what sensitivity and specificity measures again?

Sensitivity (also called the true positive rate, the epidemiological/clinical sensitivity, the recall, or probability of detection in some fields) measures the proportion of actual positives that are correctly identified as such (Wikipedia). For this paper, that is the percentage of images that were correctly identified as “wet”.

This means that the mobile app did a great job identifying the images with SRF or ME.

Specificity (also called the true negative rate) measures the proportion of actual negatives that are correctly identified as such (Wikipedia). For the case of the paper, that is the percentage of images that were correctly identified as “dry”.

The app didn’t do badly in some centers. Two (2) centers had 100% specificity. The difference we see here could be due to choice of images.

Note: The high sensitivities across the different centers and the tight band of 14.5 percentage points (from 82.5% to 97%) between centers, shows that the with confidence, the AI app can detect a problem where one exist.

I hope the results from this paper was quite understandable?

Link to the paper:

https://arxiv.org/pdf/1902.02905.pdf

Discussion

Recall that, one key question the study sought to answer was discovering and developing the most effective and efficient mechanisms by which AI-driven diagnostic capability will be distributed.

It is known that a mobile AI app modality can achieve this effectiveness and efficiency in the distribution of AI diagnostic cap

ability. The results of the study showed that mobile AI running in the Cloud is an effective and efficient means of distributing AI diagnostic capability.

In addition, one of the advantages of running the machine learning algorithm in the cloud is the ability for periodically enhancing the algorithm’s training and updating the AI seamlessly.

Limitations & Conclusion

As what looked like a limitation, the authors observed that False positives were more likely in images that had features commonly associated with exudative macular degeneration; features such as large confluent druse or pigment epithelial detachments.

They suggested that, these false positives can be “trained out” with increase in the size and feature diversity of our training data set. The authors suggested that further studies will be in the line of training on larger and more feature diverse data sets.

Great read right?

Make sure you give this review a clap if you learnt from this paper and loved this!

Thank you for reading through this. Make sure you register for the upcoming webinar as stated above.

References

U.S. Census Bureau. U.s. census clock. https://www.census.gov/popclock/, 2018 (accessed July 1st 2018).
NIH Website: https://www.nei.nih.gov/learn-about-eye-health/eye-conditions-and-diseases/macular-edema#:~:text=Macular%20edema%20is%20the%20build,and%20thicken%2C%20which%20distorts%20vision.
Kanski JJ, Bowling B, Nischal K, Pearson A (2011) Clinical ophthalmology: a systematic approach, 7th edn. Elsevier/Saunders, Edinburgh/New York
Wikipedia: https://en.wikipedia.org/wiki/Sensitivity_and_specificity