UK +44 (0)1865 849841
Malaysia +60 3 2117 5193

Image Based Risk Assessment in Cancer

Pathologists identify and interpret the changes that characterise diseases in cells and tissues, both for the studying/understanding disease processes in general and obtaining clinically relevant information for individual patients. Historically, by examining biopsy specimens, pathologists identified whether a lesion was neoplastic, inflammatory, or some other broad category. As medicine evolved, the task evolved into identifying more specific classifications. For example, if it was not sufficient to make the diagnosis of cancer; it was necessary to identify the specific subtype and grade of cancer in order to inform treatment decisions that were becoming increasingly sophisticated.

At the same time, general histologic features were identified that correlated with clinical behaviour, such as:

  • the degree of disorganisation of histological pattern
  • variability in size and shape of cellular and subcellular structures
  • patterns of local invasion
  • mitotic activity, etc.

This is essentially what a pathologist looks for when he or she “grades” a tumour to provide a histological analysis of a specimen.

Even in this age of modern molecular biology with gene and protein arrays, morphology remains a major contributor to diagnose disease and provide treatment guidelines. Both specific identification of the lesion and histological gradings are informative with respect to patient management. We can think of this information as a feature set for prediction. In other words, we wish to predict the behaviour of cancer and to predict how this behaviour can be modified optimally by an appropriate treatment. Survival predictions help optimise treatments.

The Convolutional Neural Network

Even before machine learning and deep learning, image analysis software was available that could identify edges, boundaries, geometric shapes, and density distributions, as well as texture analysis, as aids for the pathologist. Subsequently, machine learning of various types such as random forests and support vector machines were able to learn from the features and identify lesions.

With the advent of deep learning with multiple connected processing layers, it was no longer necessary to extract such features from an image before using those features as representations of the image to input into the algorithm. The most widely used deep learning model is the convolutional neural network (CNN). A CNN can do feature extraction automatically and analyse that information directly. All that a CNN needs is a (suitably labelled) training set of the images themselves.

Rather than working with the (large number) of pixels in the image directly, the initial layers of the CNN will generate increasingly complex representations of those images which, though much smaller than the original, capture important features that the rest of the network uses for analysis and classification. Such networks can output diagnostic information that, in many published examples, rival the judgment of trained pathologists.

However, the behaviour of a cancer is also dependent on many other risk factors obtained from the patient history, physical, and laboratory tests. The latter are often evaluated via multifactorial regression models designed to predict patient outcomes. It would clearly be of importance to combine image-based information with traditional survival models to gain a comprehensive insight into the disease.


Recently, a number of studies have appeared that involve a computational approach for learning patient outcomes from digital pathology images by combining traditional deep learning algorithms with a standard risk hazard model. For a number of reasons, including computational ease and the ability to deal with non-linearity, logarithmic models are used, in particular, the Cox proportional hazard model. We will discuss two approaches; DeepConvSurv (1) and Survival Convolutional Neural Networks (SCNN) (2). Both look at image patches from regions of interest within the biopsy specimen.

DeepConvSurv is basically a convolutional neural net except that it uses a Cox function as the loss function that is used to train the network. In DeepConvSurv a large number of semi-randomly collected image patches undergo dimensional reduction using K-means clustering. Aggregated clusters then run through the connected neural network with boosting via the Cox’s negative log function. Finally, the aggregated patch risk data is equated to patient risk. This, however, is only a loose approximation, since patient risk also includes size and location of a tumour, all modified by age, sex, and co-morbidity, and in modern medicine, genomic and proteomic features.

Survival Convolutional Neural Networks

SCCN is more sophisticated, in that it explicitly utilises genomic data as well as image morphology. It requires annotation of the image dataset which is then run through a full CNN. The genomic data is inputted directly into the fully connected layers, bypassing the convolutional layers. The network is trained in the usual fashion with a standard loss function, except that the outputs are then entered into a final Cox approximation layer. In effect, SCCN is taking advantage of the universal approximation theorem, namely that a single layer with an appropriate activation factor can approximate any arbitrary function. Here the function is the Cox partial hazard function. By providing a risk-based heat map overlying histological features in their publication, the authors documented that the algorithm is detecting meaningful risk elements.

Looking ahead, it is likely that deep learning algorithms will become better able to integrate morphological data with the whole spectrum of available patient information. This will allow optimal tailoring of therapy to the specific disease as manifested in a specific patient, thus enabling truly personalised precision medicine. However, we will have to overcome the limiting factor in deep learning, which is the need for gigantic sets of labelled (annotated) images. Hybrid networks such as CUNET (3) incorporating both K-means and convolutional nets in a fully integrated manner, and GAN (generative adversarial networks) are two important steps in this direction.

Stanley Cohen
is a member of the Digital Pathology Association and the Board of International Academy of Digital Pathology. As well as this, he is a Professor of Pathology at the University of Pennsylvania and Jefferson University.


Take a look at the agenda for the upcoming Digital Pathology & AI Congress, to be held in London on the 6th – 7th December 2018. 

  • Zhu, X. et al. WISA: Making survival prediction from whole slide histopathological images. IEEE Conference on Computer Vision and Pattern Recognition, 7234-7242, 2016
  • Mobadersany P. et al Predicting cancer outcomes from histology and genomics using convolutional networks., 2018
  • Dong, L. et al.  CUNet: A compact unsupervised network for image classification. J. IEEE Trans. Multimedia, arXiv:1607.01577v1 [cs.CV], 2016
  • Hou, L. et al.  Unsupervised histopathology image synthesis. arXiv:1712.05021v1 [cs.CV]. 2017

Leave a Reply

Subscribe to Our Newsletter

Get free reports and resources from our world class speakers.
  • This field is for validation purposes and should be left unchanged.

Life Sciences Twitter Feed