CAMELYON17 Banner
Join

Task

In clinical practice several lymph nodes are surgically removed after which these nodes are processed in the pathology laboratory. Usually, for each patient many lymph node slides are prepared. However, the size of the CAMELYON17 data set would grow beyond reasonable size if we shared all the slides that are prepared per patient in practice. So in this challenge we forged artificial patients. There are 5 slides provided for each patient and each slide corresponds to exactly one node. Please note that in the process of assembling the artificial patients, we have (heavily) mixed slides from several real patients. Therefore, one should never assume that the 5 slides of an artificial CAMELYON17 patient were derived from the same real patient.

The task in this challenge is to determine a pN-stage for every patient in the test dataset. To compose a pN-stage, the number of positive lymph nodes (i.e. nodes with a metastasis) are counted. There are two categories of lymph node metastasis:

  • Macro-metastases: metastases greater than 2.0 mm.
  • Micro-metastases: metastases greater than 0.2 mm or more than 200 cells, but smaller than 2.0 mm.

A separate category, called isolated tumour cells (ITC), is strictly not a metastasis, but is rather defined as: single tumour cells or a cluster of tumour cells smaller than 0.2 mm or less than 200 cells. Lymph nodes containing only ITC are therefore not counted as positive lymph nodes. However, pathologists are required to report on ITC when no macro-metastases or micro-metastases were found in a patient's lymph nodes.

In this challenge, we are using a simplified version of the pN-staging system in breast cancer. The task is to automatically determine per patient which of the following pN-stages applies.

Pathologic lymph node classification (pN-stage) in CAMELYON17:

  • pN0: No micro-metastases or macro-metastases or ITCs found.
  • pN0(i+): Only ITCs found.
  • pN1mi: Micro-metastases found, but no macro-metastases found.
  • pN1: Metastases found in 1–3 lymph nodes, of which at least one is a macro-metastasis.
  • pN2: Metastases found in 4–9 lymph nodes, of which at least one is a macro-metastasis.

By official definition, the absence of ITCs must be confirmed by cytokeratin immunohistochemistry staining. When ITC absence is confirmed in all lymph nodes, a patient is labelled as pN0(i-). However, in this challenge we only provide H&E-stained slides, which means that pN0(i-) will not be part of the outcome metric.

If micro-metastases are found in several lymph nodes, and no macro-metastases are found, that particular case is classified as pN1mi.

For example the patient below, with 5 lymph nodes, should be classified as pN1:

  • Node 1: contains no metastases and no ITC.
  • Node 2: contains one macrometastasis and one micro-metastasis.
  • Node 3: contains two micro-metastases.
  • Node 4: contains one micrometastasis and ITC.
  • Node 5: contains only ITC.

A complete list of all possible pN-stages with official definitions, as provided by the Union for International Cancer Control (UICC) and the American Joint Committee on Cancer (AJCC), can be found in this document (second page, right-side).

We simplified the pN-staging system in this challenge for several reasons:

  • Many of the official pN-(sub)stages, like pN1b, specifically deal with a lymph node’s location of origin (e.g. internal mammary node) and whether the lymph node could be detected by imaging or palpation.
  • Some pN-stages require molecular techniques (RT-PCR).
  • Including pN3-stage patients would require us to provide patients with 10 or more lymph nodes, this would increase the amount of data that participants would have to download to an unacceptable level.

Evaluation

For the evaluation of the results we use five class quadratic weighted kappa where the classes are the pN-stages. In case of a tie, we rank the submissions by the kappa score calculated on the individual pN stages in reversed order (starting with pN2).

For example if submission A and B reached equal Kappa score, A will be put in front of B if the five class quadratic weighted kappa score of submission A is better than the Kappa score of submission B calculated only on the pN2 stage patients of the ground truth. If it is still a tie we calculate it for pN1 only, and so on.