Image analysis and machine learning for detecting malaria

Malaria is caused by protozoan parasites of the genus Plasmodium that are transmitted through the bites of infected female Anopheles mosquitoes and that infect the red blood cells. Most deaths occur among children in Africa, where a child dies almost every minute from malaria, and where malaria is a leading cause of childhood neuro-disability. According to the World Malaria Report 2016, an estimated 3.2 billion people in 95 countries and territories are at risk of being infected with malaria and developing disease, and 1.2 billion are at high risk (>1 in 1000 chance of getting malaria in a year). There were about 214 million cases of malaria globally in 2016 and about 438,000 malaria deaths. The burden was heaviest in the African region, where an estimated 92% of all malaria deaths occurred, and in children aged under 5 years, who accounted for more than two thirds of all deaths (see also the malaria death rates from an earlier WHO report in Fig 1). Typical symptoms of malaria include fever, fatigue, headaches, and, in severe cases, seizures and coma, leading to death.

Hundreds of millions of blood films are examined every year for malaria, which involves manual counting of parasites and infected red blood cells by a trained microscopist. Accurate parasite counts are essential not only for malaria diagnosis. They are also important for testing for drug-resistance, measuring drug-effectiveness, and classifying disease severity. However, microscopic diagnostics is not standardized and depends heavily on the experience and skill of the microscopist. It is common for microscopists in low-resource settings to work in isolation, with no rigorous system in place that can ensure the maintenance of their skills and thus diagnostic quality. This leads to incorrect diagnostic decisions in the field. For false-negative cases, this leads to unnecessary use of antibiotics, a second consultation, lost days of work, and in some cases progression into severe malaria. For false-positive cases, a misdiagnosis entails unnecessary use of anti-malaria drugs and suffering from their potential side effects, such as nausea, abdominal pain, diarrhea, and sometimes severe complications.

This sober analysis of malaria diagnosis has prompted efforts to perform malaria diagnosis automatically. Automatic parasite counting has several advantages compared with manual counting: (1) it provides a more reliable and standardized interpretation of blood films, (2) it allows more patients to be served by reducing the workload of the malaria field workers, and (3) it can reduce diagnostic costs. Several key processing steps are typically required to quantify parasitemia automatically. First, digital blood slide images need to be acquired, which often requires preprocessing to normalize for lighting or staining variations. In a second step, blood cells or parasites need to be detected. For blood cells, this typically implies cell segmentation to identify individual cells in cell clumps to obtain accurate cell counts. In a third step, after cell detection and segmentation, features are computed to describe the typical visual appearance of infected and uninfected blood cells. In a final classification step, a classifier, who has been trained on an independent and typically manually annotated training set, then discriminates between infected and uninfected cells. Once the number of infected and uninfected cells is known, computation of parasitemia is a straightforward mathematical equation, which includes clinical parameters such as hematocrit value, for example.

The prospects of automating malaria diagnosis with its obvious advantages has attracted many researchers, especially in the last decade. The publications reflect all the major developments we have seen in the areas of automatic pattern recognition and machine learning in the last years. Our article will give an overview of the articles that have been published, using the processing steps mentioned above as a framework and guide. This is not the first survey article on the subject. In fact, several survey articles have already been published before, which bear testimony to both the importance of automated malaria diagnosis and the research dynamics and rapid system development. We refer readers in particular to the following surveys for additional information about the background of automatic malaria diagnosis and the image processing and machine learning methods used for automated microscopy diagnosis of malaria.^{,  ,}  In addition, more specific surveys have been published on cell features for malaria parasite detection, on malaria diagnosis, on malaria diagnostic tools, and on alternatives to conventional microscopy. The purpose of our article is not to replace these surveys, but rather to complement them and to provide the latest update of the state of the art in image analysis and machine learning for malaria diagnosis as it presents itself at the end of the year 2017. With about 170 literature citations, we have collected more references compared with the other surveys. We had the goal to include also maybe lesser known publications to provide a historical documentation of the work done. In addition, we included a section on deep learning, which is the latest development in malaria diagnosis and which arguably has the potential to render many of the old approaches obsolete, similar to the development in other imaging application areas. There have also been many developments in hardware for automatic malaria diagnosis, which are however out of the scope of this article and deserve a separate article^{,  ,  ,  ,}  . Nevertheless, we devote a section to rapid diagnostic tests (RDTs) for malaria diagnosis because they are also widely used in the field. The bulk of our articles have been collected from the Journal of Microscopy, Malaria Journal, and PLOS ONE, including a few articles from Nature and others. We have also collected publications from Institute of Electrical and Electronics Engineers (IEEE) conferences and other proceedings published by Springer and Elsevier. Furthermore, we have organized the articles into sections for preprocessing, cell detection and segmentation, feature computation, and classification. We have also added a separate section about deep learning and an extensive section about mobile smartphone applications for malaria diagnosis. A discussion of the latest developments and our conclusion mark the end of this article.

There are 5 Plasmodium species that cause malaria in human: Plasmodium falciparum, Plasmodium vivax, Plasmodium malariae, Plasmodium ovale, and Plasmodium knowlesi. The 2 most common species are P. falciparum and P. vivax. P. falciparum is the most severe form and is responsible for most malaria-related deaths globally.

P. falciparum is the most prevalent malaria parasite in sub-Saharan Africa, accounting for 99% of estimated malaria cases in 2016. Outside of Africa, P. vivax is the predominant parasite in the WHO Region of the Americas, representing 64% of malaria cases, and is above 30% in the WHO Southeast Asia and 40% in the Eastern Mediterranean regions.

Each of these parasite species goes through stages during their development cycle (48 hours), which gives the parasites a different visual appearance that can be observed under the microscope. In chronologic order, these stages are the ring stage, trophozoite stage, schizont stage, and gametocyte stage. Fig 2 shows typical examples of all stages for each species.

In nonsevere malaria, mostly the young stages (<24 hours old) of P. falciparum are present in the peripheral blood, whereas for severe malaria all stages can be present in the peripheral blood. For P. falciparum, the trophozoite-infected red blood cells disappear from the peripheral blood circulation by attachment to the walls of capillaries inside vital organs, which is a process called sequestration. If the capillaries are blocked for newly infected cells by already attached cells, more mature parasite stages (trophozoites and schizonts) will be visible in the peripheral blood, which indicates a severe infection and a bad prognosis.

For P. falciparum, ring stages have a visible cytoplasm and 1 or 2 small chromatin dots. The infected blood cells are not enlarged but can feature multiple infections. P. falciparum trophozoites are rarely seen in peripheral blood smears. The cytoplasm of mature trophozoites tends to be more dense than younger rings, trophozoites can appear round in shape with brown malarial pigment inside, (Centers for Disease Control and Prevention (CDC)). P. falciparum schizonts are also seldomly seen in peripheral blood. They are displaying more than 2 and up to 32 nuclei (merozoites) with dark brown pigment clumped in the middle. Gametocytes of P. falciparum have a crescent or sausage shape, and can be seen in the blood smear 1 week after a parasite infection. The chromatin is visible as a single mass or is diffuse. For more information about P. falciparum morphology, see for example References ^,  . Similar observations can be made for the stages of the other parasite species. For example, for P. vivax, host cells are often enlarged and have irregular shape. Trophozoites are amoeboid in shape with malaria pigment seen, and for severe infections multiple infections of single blood cells are not uncommon. For P. malariae, host cells are not enlarged. Trophozoites have a strong tendency to form a band with malarial pigment scattered along across the diameter of infected red blood cells. Multiple infections are extremely rare for P. malariae. On the other hand, for P. ovale, host cells are slightly enlarged and have an oval shape with tufted ends, often fimbriated. Parasites are slightly enlarged and trophozoites are amoeboid in shape with malarial pigment. Multiple infections of a single cell are more common than for P. vivax. For P. knowlesi, infected red blood cells do not appear enlarged. The parasite erythocytic cycle is only 24 hours, which is shorter than P. falciparum's cycle (48 hours) and much shorter than P. malariae's cycle (72 hours), which will lead to the same stage seen in peripheral blood every day at a given time. The morphology of P. knowlesi parasites is similar to P. malariae. Trophozoites can feature malarial pigment spread inside, band form may be seen like P. malariae, but their cytoplasm is more irregular, and multiple parasites infecting 1 single red blood cell can be seen like in P. falciparum.

Fig 3 shows 2 examples of different parasite stages in the same thin blood slide image. In the first slide image, P. falciparum trophozoites and gametocytes can be seen together with white blood cells. The latter are larger and have a pronounced nucleus compared with the many red blood cells in the image. In the second image, P. falciparum ring stages are together with schizonts. In addition, other objects such as parasite outside cells and staining noise are visible in both images. Staining noise in particular can be confused with parasites by an unexperienced microscopist.

Malaria is a curable disease, with drugs available for treatment, including drugs that can help prevent malaria infections in travelers to malaria-prone regions. However, there exists no effective vaccine against malaria yet, although this is an area of active research and field studies. Once infected, malaria is a rapidly progressing disease, with a serious risk of developing into severe and cerebral malaria with neurologic symptoms for P. falciparum infections. Therefore, a timely diagnosis of malaria is very important. Although malaria can be diagnosed in many different ways, there is room for improvement for current malaria diagnostic tests including reducing cost, increasing specificity, and improving ease of use. Because automated malaria diagnosis for resource-poor settings is the main topic of this survey, we have devoted 2 subsections to light microscopy and RDTs, which are by far the 2 most heavily used diagnostic means in these areas. We also briefly discuss the other options for malaria diagnosis, although they are arguably less suited for the conditions in remote malaria regions. For more information about malaria diagnosis, we refer readers to the surveys in , and the following references: ^{,  ,}  .

Detecting the presence of parasites is the key to malaria diagnosis. In addition, identifying the parasite species and presence of potentially mixed infections is important, as well as the observation of the stage development of P. falciparum parasites in relation to the severity of the disease. Counting parasites for determining the level of parasitemia is not only important for identifying an infection and measuring its severity, it also allows monitoring patients by measuring drug efficacy and potential drug resistance.

More than 100 years ago, Giemsas stain (1902) was applied for the first time for the diagnosis of malaria. Since then, it received increased attention. Because of its low cost, its high sensitivity, and specificity, it is currently widely used in microscopical malaria examinations. However, Giemsa staining requires multiple reagents, experienced personal, and is labor-intensive and time-consuming (it typically requires at least 45 minutes to stain a slide).

Other stains have been used, too, like Field stain that significantly reduces the staining time, although it requires drying of samples before and during staining. However there are also disadvantages with Field's stain, especially in under-resourced health centers in which the stain might be used. Poor blood preparations often result in the generation of artifacts commonly mistaken for malaria parasites, such as bacteria, fungi, stain precipitation, dirt, and cell debris. These can frequently cause false-positive readings.

Another stain is Leishman's stain (1901), which has a high sensitivity, is cheap, and relatively easy to perform. Among the other stains being used is, for example, the Wright-Giemsa stain, which is a combination of Wright and Giemsa stain, and where the former facilitates the differentiation of blood cell types.

In 1970s, Sodeman et al. investigated the effect of fluorochrome staining in identifying the malaria parasites at low-level infection. It has been shown that fluorochrome staining is more sensitive and less time-consuming than Romanowsky and Giemsa staining methods^{,  ,}  but requires considerable practice and training, and suffers from artifacts including photobleaching and phototoxicity.^,  Moreover, fluorescence microscopes are more expensive than standard light microscopes, which is a factor in tropical resource-poor regions where malaria is endemic.^{,  ,} 

Table I shows the blood smear types and staining techniques used for the approaches published in the literature. Clearly, the vast majority of publications has been for thin smears. Certainly, 1 reason for this lies in the fact that thin smears allow to determine the parasite species and stages more easily, in addition to the parasitemia. So, in some sense, thin smears are more versatile and contain more information. Another important reason is probably that the presence of red blood cells gives the problem of parasite detection more structure, and makes the problem easier to a certain degree, as parasites need to be detected only inside cells. For thick films, parasite detection may be harder because of noise and staining artifacts that can lead to false positives. Nevertheless, because of the importance of thick smears for practical malaria diagnosis, it is very likely that more approaches for thick films will be implemented in the future. However, if convincing optical hardware solutions are found to scan multiple fields in thin smears and achieve a sensitivity comparable with thick smears, then this may be a moot point.^, 

Table I also shows that the majority of approaches, for both thin and thick smears, have adopted the most popular stain in practice, Giemsa. Although stains like Leishman provide very good results for malaria parasites, Giemsa stain has proved to be the best all-round stain for the routine diagnosis of malaria. It has the disadvantage of being relatively expensive, but this is outweighed by its stability over time and its consistent staining quality over a wide range of temperatures.

This section provides the core information of our survey, namely a compilation of references that should cover the vast majority of articles ever published on automated microscopy for malaria diagnosis, with the bulk of the articles published in the last 10 years. The work that has been done in this area is quite diverse. Nevertheless, a system for automated cell microscopy usually implements a sequence of key processing steps that can serve as a guideline. Therefore, each of the following subsections will focus on 1 specific aspect of the processing pipeline.

The first step is usually the acquisition of digital images of blood smears, which largely depends on the equipment and materials being use. The Image acquisition section breaks down the different approaches for the different types of microscopy, blood slides (thin or thick), and staining.

Following image acquisition, most systems perform one or several preprocessing methods to remove noise and to normalize lighting and color variations inherent in the image acquisition and staining process. The Preprocessing section sorts the publications according to the preprocessing methods implemented.

The next step usually involves the detection and segmentation (outlining) of individual blood cells and maybe other objects that can be visible in a blood slide image, such as parasites or platelets. The section titled Red blood cell detection and segmentation gives an overview of all the segmentation methods that have been used for microscopic malaria diagnosis.

For most articles, cell segmentation is followed by the computation of a set of features, which describe the visual appearance of the segmented objects in a mathematical succinct way. The section titled Feature extraction and selection presents the different features and potential feature selection strategies that can be found in the literature.

In the last step, a mathematical discrimination method that classifies the segmented objects into different classes based on the computed features is implemented. For example, labeling each red blood cell as either infected or uninfected is a key classification task performed in this step, which then allows to compute the parasitemia. The section titled Parasite identification and labeling lists all the classification methods used in the literature for malaria diagnosis.

Later in the article, in the section titled Deep Learning, we will present references for the latest classification trend, deep learning, which skips the feature computation step and sometimes even the segmentation step. Furthermore, in the section titled Mobile Smatphones for Malaria Diagnosis, we will discuss how smartphones can be used for microscopic malaria diagnosis and list the systems that have already been implemented and published.