Auto-delineation of oropharyngeal clinical target volumes using 3D convolutional neural networks

Clinical target volumes (CTVs) are essential volumes of interest used in radiation therapy treatment planning. These volumes provide coverage to the observable gross tumor volume (GTV) as well as any suspected microscopic disease and pathways of tumor spread such as regional lymph nodes (International Commission on Radiation Units and Measurements (ICRU) 1999). The task of delineating head-and-neck CTVs is unique in that radiation oncologists manually contour several target volumes throughout the head-and-neck region to prescribe volume-dependent doses based on the risk of disease recurrence. Several reports show that this process can be time consuming and subject to high inter- and intra-observer variability (Multi-Institutional Target Delineation in Oncology 2011, Hong et al 2012). The inherent variability in CTV delineation is quite problematic since patient outcomes depend heavily on accurate segmentation of target volumes. In addition, this variability hinders the ability to methodologically assess the quality of radiation therapy treatment plans and is considered a large source of uncertainty (Van Herk 2004). Therefore, there is a need for an accurate and consistent tool to automatically segment these volumes.

The majority of published works researching the auto-segmentation of CTVs in the head-and-neck have been focused on auto-segmenting low-risk lymph node regions using atlas-based approaches (Han et al 2008, Gorthi et al 2009, Chen et al 2010, Stapleford et al 2010, Sjöberg et al 2013, Yang et al 2014). Atlas-based auto-segmentation techniques are limited due to large inter-subject anatomical variations which typically do not generalize well because of the small number of subjects included in the contouring atlas. Taking into consideration these anatomical variations in the head-and-neck region, Han et al proposed the use of shape information to improve the registration between test and atlas subjects (Han et al 2008). Similarly, Yang et al used a two-step registration approach (rigid, then deformable) to auto-segment low-risk CTVs for tonsil cancer patients (Yang et al 2014). Both of these studies reported large variability in overlap between the predicted segmentation and the physician manually-contoured ground-truth volumes.

In contrast to atlas-based registration techniques, deep learning allows for the use of larger image datasets to be used for training. It is believed that the inclusion of more subjects allows for better model generalization and these approaches have quickly become state-of-the-art for image segmentation. Convolutional neural networks have been developed to auto-delineate nasopharyngeal and rectal cancer CTVs (Men et al 2017a, Men et al 2017b) but many more treatment sites could benefit from these deep learning algorithms. In a previous study, we developed a deep learning architecture to auto-delineate high-risk CTVs for oropharyngeal patients and have since shown the benefits of using an auto-delineation tool in comparison to multi-observer manual delineations (Cardenas et al 2018). However, we still lack a way to auto-segment CTVs as a whole volume.

The aim of this study is to provide a tool to auto-segment clinically acceptable CTVs for oropharyngeal cancer patients. In a previous study (Cardenas et al 2017b) we showed that the clinical decision of determining what to cover and to what extent on per CTV basis varies widely from patient to patient; therefore, we developed our models to predict a single CTV which is representative of the union of all CTVs (high-risk, intermediate-risk, low-risk) used for treatment in our clinic. The resulting models from the current study combined with our previously developed models for high-risk CTV auto-delineation could aid in providing radiation oncologists with a two target treatment plan. The proposed methodology and subsequent results are promising, and could have great impact in reducing inter- and intra-observer variability when designing head-and-neck radiotherapy plans.

Patient dataset

Data preparation

Simulation CT image and radiation therapy structures DICOM files were exported for each patient. The images and physician contoured structures were visually inspected and the CTV levels were combined into a single structure (CTVall). This was particularly important since the number of CTV levels used varied between cases (median: 4, range: 2–6). CTV levels were combined into a single structure because previously (Cardenas et al 2017b) we found the target delineation of intermediate- and low-risk CTVs to be more subject to variability in delineations. Using a single CTV structure allowed for a reduction in the inherent uncertainty found in our clinical data. In addition, we combined all physician contours identifying gross disease (primary and nodal) into a single structure (GTVall). Lastly, we used a thresholding tool in RayStation v6 (RaySearch Laboratories, Stockholm, Sweden) to define the patient's body contour (External).

The CT images, CTVall, GTVall, and External contours were converted into 3D matrices in Python (version 3.6.3) using the pydicom module. We then transform the intensity of our CT images using our clinic's head-and-neck CT window (−350, 350 Hounsfield Units) to have values from 0 to 1 (i.e.  −350  =  0 and 350  =  1). The image and contour matrices were resampled to have a slice thickness of 3.0 mm, and pixel spacing of 1.0 mm, and set a matrix size of # of slices  ×  512  ×  512. Since the extent of the simulation CT scan varied widely between cases, we manually identified two anatomical markers in the cranial and caudal directions (figure 1). Cranially, we identified the most caudal slice where the sphenoid bone is fused to the basilar part of the occipital bone. Caudally, we identified the most cranial slice where the sternum was observed.

Zoom In Zoom Out Reset image size

After visually identifying the cranial and caudal slices for each patient, the CT image and contour matrices were cropped to only include slices between these anatomical markers plus a 10 mm margin on both cranial and caudal directions. These markers were chosen because all patients' low-risk CTV contours were contained in between these slices. After reducing the image space in the slice direction, we sought to remove rows and columns in our matrices that did not contain the patient's body. We found that using a region of interest of size 340  ×  400 centered about the center-of-mass of the External contour provided appropriate coverage for all patients. This allowed for a reduction of voxels in the axial space of approximate 48%. Lastly, we resized all images and contour matrices from number of slices (S_Cr  −  S_Cd)  ×  340  ×  400 to 60  ×  140  ×  200. Finally we split our dataset into training (210 patients) and test (75 patients) cohorts.

Two-channel U-Net architecture

Hyper-parameter search

Along with the standard 3D U-Net, our architecture has a down-convolutional and an up-convolutional path, utilizes batch normalization before each ReLU, doubles the number of features before each max pooling layer, and uses shortcut connections of equal resolution to provide high-resolution features to the up-convolutional path, but we choose to identify the remaining optimal parameters for our segmentation task. Parameter selection was determined using a grid-search approach and parameters investigated included the number of resolution steps, number of root features, convolution kernel size, dropout ratio, max pooling kernel size and stride size, and weighted cross-entropy cost values for the segmentation class. The cost values were used to account for the class imbalance in positive (segmentation) and background voxels. The voxel-wise weighted cross-entropy loss is defined in equation (1) as:

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {\rm Loss}(T,P)=\sum\limits_{x}{\sum\limits_{y}{\sum\limits_{z}{C*{{P}_{x,y,z}}*-\log \left( {\rm Sigmoid}\left( {{T}_{x,y,z}} \right) \right)+\left(1-{{P}_{x,y,z}} \right)*-\log \left(1-{\rm Sigmoid}\left({{T}_{x,y,z}} \right) \right)}}}\nonumber \end{align} \tag{ 1 }$$

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {\rm Sigmoid}\left(n \right)=~\frac{1}{1+{{e}^{-n}}}\nonumber \end{align} \tag{ 2 }$$

where C is the cost applied to the segmentation class, ${{P}_{x,y,z}}$ is the predicted voxel's class probability, and ${{T}_{x,y,z}}$ is the ground-truth segmentation value (1  =  CTVall, 0  =  background) for the voxel in the x-, y-, and z-planes. A value of C  >  1 reduces the overall false negative count, hence increasing the sensitivity of the segmentation model.

To search for the optimal parameters, we use 3-fold cross-validation within our training dataset. Predicted segmentations on the cross-validation sets are then scored by using a metric (equation (3)) based on the dice similarity coefficient (DSC) (equation (4)), which provides a measure of the overlap between the predicted segmentation and ground-truth, and the false negative dice (FND) (equation (5)), which provides a measure of under-treatment in our segmentation task. These metrics are defined as:

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {\rm Score}(y,\hat{y})~=~{\rm DSC}({{y}_{i}},{{\hat{y}}_{i}})~-~w~\times ~{\rm FND}({{y}_{i}},{{\hat{y}}_{i}})\nonumber \end{align} \tag{ 3 }$$

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {\rm DSC}\left({{y}_{i}},{{{\hat{y}}}_{i}} \right)=\frac{2\times \left({{{\hat{y}}}_{i}}\cap {{y}_{i}} \right)}{\left| {{y}_{i}} \right|+\left| {{{\hat{y}}}_{i}} \right|}\nonumber \end{align} \tag{ 4 }$$

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {\rm FND}\left({{y}_{i}},{{{\hat{y}}}_{i}} \right)=\frac{2\times \left({{\overline{{\hat{y}}}}_{i}}\cap {{y}_{i}} \right)}{\left| {{y}_{i}} \right|+\left| {{{\hat{y}}}_{i}} \right|}\nonumber \end{align} \tag{ 5 }$$

where ${{y}_{i}}\cap {{\hat{y}}_{i}}$ is the number of voxels where both ${{y}_{i}},{{\hat{y}}_{i}}$ are equal to 1 and $\left| {{y}_{i}} \right|,\left| {{{\hat{y}}}_{i}} \right|$ is sum of positive voxels for each vector. For the FND, ${{\overline{{\hat{y}}}}_{i}}\cap {{y}_{i}}$ is the number of voxels were the background (${{\overline{{\hat{y}}}}_{i}}$) prediction vector and the ground-truth are equal to 1. The DSC has values from 0 to 1, where a value of 1 means that there is perfect overlap between the two volumes. The FND has values from 0 to 2, where a value of 0 means that the predicted volume fully overlapped the ground-truth. We chose to include FND in determining the best model parameters since most radiation oncologists would often prefer to over-contour (over-treat) than to miss any microscopic disease. A weight factor, $w$, with a value of 1.5 chosen for this task, was used to penalize models with large under-segmented regions. The model parameters with the highest mean Score value in the cross-validation sets was chosen to train our final model. Based on this optimal architecture, we trained these parameters using a cost value of 1, 2, and 5 resulting in three volumes, CTV_Tight, CTV_Moderate, and CTV_Wide, respectively.

Training

Volumetric comparison

The auto-delineated CTVs were compared to the physician ground-truth volumes by calculating differences in volumes (ΔV), true positive fraction (TFP), DSC, FND, false positive dice (FPD), mean surface distance (MSD, equation (8)) and the Hausdorff distance (HD, equation (9)). The TFP represents the proportion of the ground-truth volume that is overlapped by the auto-delineated volumes (equation (6)). The FPD provides a measure of over-contouring or over-treatment (equation (7)). These metrics are defined as follows,

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {\rm TPF}\left({{y}_{i}},{{{\hat{y}}}_{i}} \right)=\frac{\left({{{\hat{y}}}_{i}}\cap {{y}_{i}} \right)}{\left| {{y}_{i}} \right|}\nonumber \end{align} \tag{ 6 }$$

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {\rm FND}\left({{y}_{i}},{{{\hat{y}}}_{i}} \right)=\frac{2\times \left({{{\hat{y}}}_{i}}\cap {{{\hat{y}}}_{i}} \right)}{\left| {{y}_{i}} \right|+\left| {{{\hat{y}}}_{i}} \right|}\nonumber \end{align} \tag{ 7 }$$

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {\rm MSD}=\frac{1}{2}({{\bar{d}}_{a,b}}+{{\bar{d}}_{b,a}})\nonumber \end{align} \tag{ 8 }$$

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {\rm HD}=\max ({{d}_{a,b}}\cup {{d}_{a,b}})\nonumber \end{align} \tag{ 9 }$$

where d_a,b is a vector containing all minimum Euclidian distances from each surface voxel on volume a to volume b.

Comparison to other methods

After manually determining the cranial and caudal extent of the image, preparing the image set and predicting on a new patients was completed in under a minute for all test cases. Box plots with the cross-validation results (DSC, FND, and Score) observed during hyper-parameter selection can be found figure 2. As can be appreciated by this figure, the resulting auto-delineations varied depending on the choice of hyper-parameters. Auto-delineated CTVs had better overlap to the ground-truth when training deeper networks (p  <  0.001, Wilcoxon rank sum test). When comparing convolution kernel size we found, in general, that using 3  ×  3  ×  3 kernels resulted in better agreement than using a 5  ×  5  ×  5 kernel (p  <  0.001, Wilcoxon rank sum test). Furthermore, choosing a higher dropout ratio resulted in improved predictions (p  =  0.011, Wilcoxon rank sum test). As expected, increasing the cost value resulted in a reduction in average FND, as it provides an improvement in model sensitivity. The optimal architecture based on our overlap Score (equation (3)) is shown in figure 3. The test patient set's volumes, differences in volume, TPF, DSC, FND, FPD, MSD, and HD value distributions are shown in figures 4–7 for the tight, moderate and wide auto-delineated CTVs. A summary of these distributions, including distribution statistics, is provided in tables 1–3. Individual volumetric values for each patient can be found in the supplementary materials table S1 (available online at stacks.iop.org/PMB/63/215026/mmedia).

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

The test set predictions showed good overlap agreement with the ground-truth volumes; the average TFPs were 0.839, 0.852, and 0.940 for the tight, moderate, and wide auto-delineations, respectively. In terms of DSC, the percent of cases having DSC  >  0.75 were 96%, 96%, and 69% for the tight, moderate, and wide auto-delineations, respectively. When considering MSD, the percent of cases having MSD  ⩽  3.0 mm were 37%, 36%, and 1% for the tight, moderate, and wide auto-delineations, respectively. The percentage of patients meeting MSD  ⩽  5.0 mm were 97%, 97%, and 80% for tight, moderate, and wide auto-delineations, respectively. Motivated by this finding, we investigated the use of 5 mm uniform margin expansions (as it's usually done to create planning target volumes (PTV)) which were applied to the auto-delineated CTVs to account for delineation uncertainty. Using this 'PTV' (auto-delineated CTV  +  5 mm) increased the coverage of ground-truth CTVs such that 92%, 95%, and 97% of patients had a TFP of at least 0.90 (90% of volume). This showed an improvement from 17%, 17% and 89% of patients when considering the tight, moderate, and wide auto-delineated CTVs, respectively, alone.

On a case-by-case visual inspection of the predicted CTVs, we noticed that there was larger disagreement between the neural network's prediction and the ground-truth in the lower nodal region. This was expected as it is typical in clinical practice for physicians to treat these low-risk regions more differently than regions with observable (GTV) disease (Hong et al 2012, Cardenas et al 2017b). A visual comparison between auto-delineated and physician target volumes are provided for five patients in the supplementary materials (figures S1–S5).

We further compared our approach's results to those calculated for an atlas-based and 2D fully convolutional network approach, as described in the Methods section. The DSC and MSD results (mean  ±  standard deviation) are provided in table 4. Both the mean DSC and MSD values were better for our 3D convolutional network than the other two methods.

This is the first study to show that is possible to automate oropharyngeal CTV delineation using a convolutional neural network. The two-channel 3D U-Net network was able to identify physician contouring patterns (figure 8) and we found that the predicted segmentations had high overlap agreement with the physician contoured volumes. More importantly, we implemented a DSC/FND score during cross-validation to identify parameters that provided segmentations with the least missed volumes. This is important in radiation therapy since undertreating microscopic disease can lead to loco-regional recurrences.

Zoom In Zoom Out Reset image size

While the CTV_wide volumes were able to produce the highest overlap (TFP) between the auto-delineated and ground-truth targets, their volumes were generally much larger than the ground-truth and in some cases not appropriate for clinical use. Besides the TFP and FND metrics, CTV_tight and CTV_moderate volumes resulted in better volumetric agreement than the CTV_wide volumes. A large majority of CTV_tight and CTV_moderate volumes had MSDs to the ground-truth targets which were less than or equal to 5 mm. Some of this contouring uncertainty could be included within the PTV margin (which accounts for geometrical uncertainties in patient setup accuracy, organ motion, and the delineation process of GTVs and CTVs (Van Herk 2004)). As an initial assessment of this, we evaluated the additional ground-truth volume coverage provided by applying a 5 mm margin expansion to the auto-delineated targets, and found that 92% and 95% of patients had an overlap of at least 90% of their ground-truth CTV volume. A limitation of this comparison is that it assumes that other sources of uncertainty found in radiation therapy are reduced to zero. Although this is not strictly true, the introduction of image-guided radiation therapy has allowed for the reduction of some of these uncertainties (Juan-Senabre et al 2011, Djordjevic et al 2014) leaving inter-observer variability in tumor and target delineations the leading source of uncertainty for head and neck cancers (Rasch et al 2005, Beadle and Anderson 2018). This initial assessment indicates that there is merit in a future, more detailed evaluation of the use of PTV margins to account for uncertainties in the automatic contouring process.

Our approach appears to outperform other published segmentation techniques such as atlas-based and the FCN-8 architecture. It should be noted, however, that comparisons of this type are difficult and, while these approaches were optimized for this segmentation task, further optimization of those techniques may improve the results from those approaches.

In a blinded survey of a subset of these patients, we asked two head and neck expert radiation oncologists to rate the most appropriate delineation amongst the ground-truth and auto-delineated volumes. For both physicians, four out of five times they preferred the auto-delineated volumes over the ground-truth; however, one preferred the CTV_tight volumes in 3 out of 5 cases whereas the other physician preferred the CTV_moderate volumes with the same rate. The physician manual contours were only chosen in 1 out of 5 cases for both physicians but on different cases. It remains unclear which volume is most appropriate for clinical use and this may be dependent of physician preference.

The importance of accurate CTV delineation has been a topic widely discussed in the head-and-neck radiation oncology community (Beadle and Anderson 2018). While many contouring guidelines exist (Grégoire et al 2014, Grégoire et al 2017, Lee et al 2017), it remains unknown how many physicians follow these recommendations. Hong et al showed large heterogeneity in CTV contouring and clinical practice amongst head-and-neck experts when they were asked to contour an identical oropharyngeal case (Hong et al 2012). This inter-observer variability has been suggested to be the largest source of uncertainty in radiotherapy treatment planning (Burnet et al 2004, Segedin and Petric 2016). In the study Hong et al, the ratio between the maximum and minimum volumes manually delineated on a single oropharyngeal case was 18.3, whereas the median ratio in maximum and minimum volumes (measured for each patient) in our study was 1.15 (range: 1.00–1.65), 1.15 (range: 1.00–1.64), and 1.45 (range: 1.05–2.22) for the tight, moderate, and wide auto-delineated volumes, respectively. Similar maximum–minimum ratios (range: 1.72–3.41) were reported by Peng et al (Peng et al 2018) on manual delineations from ten radiation oncologists on a single nasopharynx case.

Several publications have shown that physician expertise and patient load are highly correlated with patient outcomes (Wuthrick et al 2015, Boero et al 2016), meaning that more experienced physicians who see a larger number of cancer cases per year tend to provide patients with better treatment outcomes. The ground-truth segmentations used in this study come from a large group of sub-specialized head-and-neck radiation oncologists and undergo thorough peer-review prior to treatment (Cardenas et al 2017b). This QA process in our clinical workflow aids in reducing physician errors and possible tumor misses.

There are a few limitations to our study. The delineations used to train and test our models come from a single institution and may only reflect clinical practice and treatment decision patterns between the physicians in our clinic. While the physician CTV delineations used in this study were subject to a rigorous peer-review QA session prior to treatment, the inter-observer variability in these delineations remains to be determined and it is acknowledged that this variability could impact the performance of the predicted delineations. Our models use only imaging information (CT image and GTV contours) and lack patient-specific clinical information that is considered when designing a patient's treatment plan; further implementation of clinical information in CTV auto-delineation models may improve auto-delineation predictions (Cardenas et al 2017a), but this was not part of the scope of this study. Lastly, our model requires manual GTV delineations as one of the input channels; this limitation could be addressed by implementing tumor lesion auto-segmentation algorithms (Street et al 2007, Yang et al 2015).

The auto-delineation of CTVs could aid in reducing delineation variability which remains one of the largest sources of uncertainty in radiation therapy (Segedin and Petric 2016). Creating a tool that automatically provides physicians with target volumes would be a significant contribution to our field. This would allow for the collection of better clinical data which would be particularly useful in clinical trials and multi-institutional studies where heterogeneity in clinical practices is largest amongst practitioners.

Using CTVs previously segmented and used in our clinic to deliver patient treatments, the trained convolutional neural network is able to segment the union of CTVs with high overlap and close average surface distances (DSC  >  0.75 on 96% of cases for tight and moderate auto-delineation models and 97% of cases having MSD  ⩽  5.0 mm). We found that applying a 5 mm uniform margin expansion to the auto-delineated CTVs would cover at least 90% of the physician CTV volumes for a large majority of patients; however, determination of appropriate margin expansions for auto-delineated CTVs merits further investigation.