Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?


Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 02.04.15 in Vol 1, No 1 (2015): Jan-Jun

This paper is in the following e-collection/theme issue:

    Original Paper

    Evaluation of a Web-Based App Demonstrating an Exclusionary Algorithmic Approach to TNM Cancer Staging

    Brigham and Women's Hospital, Division of Endocrinology, Diabetes and Hypertension, Boston, MA, United States

    Corresponding Author:

    Matthew Kim, MD

    Brigham and Women's Hospital

    Division of Endocrinology, Diabetes and Hypertension

    221 Longwood Avenue


    Boston, MA, 02115

    United States

    Phone: 1 617 732 5666

    Fax:1 617 732 5764



    Background: TNM staging plays a critical role in the evaluation and management of a range of different types of cancers. The conventional combinatorial approach to the determination of an anatomic stage relies on the identification of distinct tumor (T), node (N), and metastasis (M) classifications to generate a TNM grouping. This process is inherently inefficient due to the need for scrupulous review of the criteria specified for each classification to ensure accurate assignment. An exclusionary approach to TNM staging based on sequential constraint of options may serve to minimize the number of classifications that need to be reviewed to accurately determine an anatomic stage.

    Objective: Our aim was to evaluate the usability and utility of a Web-based app configured to demonstrate an exclusionary approach to TNM staging.

    Methods: Internal medicine residents, surgery residents, and oncology fellows engaged in clinical training were asked to evaluate a Web-based app developed as an instructional aid incorporating (1) an exclusionary algorithm that polls tabulated classifications and sorts them into ranked order based on frequency counts, (2) reconfiguration of classification criteria to generate disambiguated yes/no questions that function as selection and exclusion prompts, and (3) a selectable grid of TNM groupings that provides dynamic graphic demonstration of the effects of sequentially selecting or excluding specific classifications. Subjects were asked to evaluate the performance of this app after completing exercises simulating the staging of different types of cancers encountered during training.

    Results: Survey responses indicated high levels of agreement with statements supporting the usability and utility of this app. Subjects reported that its user interface provided a clear display with intuitive controls and that the exclusionary approach to TNM staging it demonstrated represented an efficient process of assignment that helped to clarify distinctions between tumor, node, and metastasis classifications. High overall usefulness ratings were bolstered by supplementary comments suggesting that this app might be readily adopted for use in clinical practice.

    Conclusions: A Web-based app that utilizes an exclusionary algorithm to prompt the assignment of tumor, node, and metastasis classifications may serve as an effective instructional aid demonstrating an efficient and informative approach to TNM staging.

    JMIR Cancer 2015;1(1):e3




    The tumor/node/metastasis (TNM) staging system collaboratively developed and maintained by the American Joint Committee on Cancer and the International Union for Cancer Control plays a critical role in the evaluation and management of patients diagnosed with a range of different types of cancers [1]. Accurate staging based on assessment of the extent of anatomic spread of cancer at the time of diagnosis helps to determine prognosis based on correlated survival rates. Staging also helps to guide the planning of treatment, facilitates communication between providers working in different disciplines, and serves as the basis for identifying patients who may be eligible for enrollment in clinical trials [2].

    Criteria for stage assignments have been established for 47 different types of cancers. Determination of a patient’s stage is based on the classification of three principal components that may be assessed at the point of diagnosis to determine a clinical stage, or after definitive surgery to determine a pathologic stage. Assignment of a tumor (T) classification ranging from T0-T4(a-d) is based on assessment of the size and extent of contiguous spread of the primary tumor. Assignment of a node (N) classification ranging from N0-N3(a-c) is based on assessment of the extent of spread of tumor to regional draining lymph nodes. Assignment of a metastasis (M) classification ranging from M0-M1(a-b) is based on assessment of the presence or absence of distant metastases (Table 1).

    Additional prognostic factors that have proven to be significant in the staging of specific types of cancers include tumor grade, tumor location, mitotic rate, risk factors, histologic scores, and biochemical tumor marker levels. Compiled groupings of T, N, M, and prognostic factor classifications are sorted into tabular arrays that are stratified to define stages characterized as anatomic stages or prognostic groups ranging from 0-IV(A-C) in order of declining prognosis (Table 2).

    Table 1. T, N, and M classifications for cancer of the lung.
    View this table
    Table 2. Anatomic stage/prognostic groups for cancer of the lung.
    View this table

    The conventional approach to staging involves (1) selecting appropriate T, N, M, and prognostic factor classifications, (2) combining these classifications to generate a TNM grouping, and (3) locating this TNM grouping in the array of possible combinations to determine a corresponding stage. This combinatorial approach is inherently inefficient due to the fact that successive tumor and node classifications for different types of cancers are not always graded or mutually exclusive. Assignment of a T1 classification may be based on measurement of the diameter of the primary tumor, while assignment of a T2 classification may be based on identification of a pattern of local invasion. Assignment of an N2 classification may be based on confirmation of spread of tumor to a specific group of regional draining lymph nodes, while assignment of an N3 classification may be based on tabulation of the number of involved lymph nodes. As a result, accurate staging relies on scrupulous review of the criteria for each T, N, and prognostic factor classification to ensure that correct assignments are made.

    An alternative approach to staging that seeks to optimize the efficiency of the process is predicated on the notion that unambiguous selection or exclusion of a T, N, M, or prognostic factor classification may serve to constrain the number of subsequent classifications that need to be reviewed to identify a correct TNM grouping. If a specific T, N, M, or prognostic factor classification can be selected based on review of its criteria, then any TNM groupings that do not include that classification can be excluded from further consideration. Alternatively, if a specific classification can be excluded without reservation, then any TNM groupings that include that classification can be excluded from further consideration. The set of TNM groupings that remain as viable options after a specific classification has been selected or excluded will most often encompass a restricted subset of classifications. In its elaboration, this exclusionary approach may effectively serve to minimize the number of classifications that need to be reviewed to accurately determine a patient’s stage.


    A Web-based app configured to demonstrate this approach to staging was developed as an instructional aid for trainees. A version coded in ActionScript 3.0 was iteratively adapted to incorporate the following key features [3]:

    • an exclusionary algorithm that cycles through a sequence of (1) polling the set of TNM groupings listed for each anatomic stage to tabulate the number of times that each T, N, M, or prognostic factor classification is listed, (2) sorting the tabulated classifications in ascending order of frequency prioritized based on the extent of spread (M >N > T) and level of classification (N3 > N2 > N1> N0), (3) prompting selection or exclusion of the first ranked classification, and (4) excluding nullified TNM groupings from the set based on the response
    • a grid of anatomic stage listings with corresponding TNM groupings incorporating selectable T, N, M, and prognostic factor classifications
    • reconfiguration of the criteria specified for T, N, M, and prognostic factor classifications to generate yes/no questions phrased to (1) itemize the components of complex definitions, (2) disambiguate definitions that incorporate combined Boolean AND + OR conditions, and (3) minimize negative definitions (Table 3)
    Table 3. Reconfiguration of classification criteria for cancer of the lung.
    View this table

    Users are prompted to select a tumor type to begin a staging exercise. Some tumor types require selection of secondary options which may include identification of a tumor subtype, anatomic location, age limit, or phase of staging (clinical vs pathologic). Selection of a tumor type shifts to a display that includes a grid of anatomic stage listings and a prompted yes/no question corresponding to the first ranked T, N, M, or prognostic factor classification (Figure 1).

    Clicking a Yes or No button to answer the question will select or exclude the classification. If a specific classification is known at the point of entry, it can be directly selected by clicking on a corresponding entry in the grid of anatomic stage listings. Selection or exclusion of a classification triggers fading of nullified TNM groupings and cycling of the exclusionary algorithm. Subsequent prompted yes/no questions will appear in sequence until the set of TNM groupings has been narrowed to delimit a single TNM grouping and/or a specific anatomic stage. If a specific anatomic stage has been delimited with multiple TNM groupings that persist as viable options, a “Complete” button can be clicked to prompt further selection and exclusion. When a single TNM grouping has been identified, a terminal display lists the anatomic stage and T, N, M, and prognostic factor classifications with their corresponding criteria. Forward and back arrows can be clicked to scroll through the sequence of classifications with highlighting of selected answers and concordant enhancement and fading of associated TNM groupings.

    Figure 1. Clicking a response to the prompted question triggers fading of nullified TNM groupings. Forward and back arrows can be clicked to scroll through the sequence of selected and excluded classifications.
    View this figure


    An evaluation study was conducted to assess the usability and utility of this app. Participating subjects recruited from regional clinical training programs by email solicitation included 14 internal medicine residents, 9 surgery residents, and 7 oncology fellows. Institutional review board approval was obtained prior to recruitment and enrollment. Subjects were provided with open access to a fully functional version of the app along with a set of basic operating instructions. They were asked to evaluate its performance during staging exercises that included assessment of a newly diagnosed cancer, restating of recurrent cancer, and review of an incorrectly staged cancer. Subjects were asked to complete these exercises with an eye towards assessment of how they might use this app to (1) simulate the staging of different types of cancers encountered during training and (2) study for board exams that test knowledge of TNM staging criteria. After completing the staging exercises, subjects were asked to complete a Web-based survey about the app focused on rating its ease of use, clarity of presentation, perceived accuracy, and potential for adoption in different clinical and educational settings. The survey consisted of 12 statements phrased to support the usability and utility of the app. Selectable responses were arrayed on a 5-point scale with options labeled “Strongly disagree”, “Disagree”, “Undecided”, “Agree”, and “Strongly agree”.

    Statistical Analysis

    Median Likert scores and 25th and 75th quartiles were calculated for each survey item assigning a value of 1 to responses scored as “Strongly disagree”, 3 to items scored as “Undecided”, and 5 to items scored as “Strongly agree”. Subjects were also asked to rate the overall usefulness of the app on a scale ranging from 1 (“Not at all useful”) to 10 (“Essential”) and were provided with the option of entering free text comments about what they did or did not like about the app.


    All 30 of the enrolled subjects completed the entire survey. Subjects expressed a high level of agreement with each of the 12 statements. No “Strongly disagree” responses were registered. The median level of agreement was “Agree” for 7 of 12 statements and “Strongly agree” for the remainder with minimal dispersion (Table 4).

    Table 4. Survey responses.
    View this table

    The median overall usefulness rating was 9 (25-75 interquartile range 8-9.75). Eighteen subjects elected to enter free text comments that ranged from general impressions of the app to specific criticisms of its navigability and functionality. While most of the general impressions were favorable (“Great application”, “Very useful clinically”), a few subjects expressed reservations about the limited scope of the app (eg, “It would be nice to have links to the appropriate staging guidelines or references”, “It would be great to see some of the hematologic malignancies like myeloma and lymphoma added”). The majority of the specific criticisms focused on problems that subjects experienced when trying to use the browser back button to navigate between screens. This problem is commonly encountered with the first use of platform-independent apps that run in browser plug-ins [4]. Internal navigation controls were moved to more intuitive locations in subsequent iterations as a result. A few of the subjects found staging exercises that began with prompted questions about the presence of distant metastases to be disconcerting at first, but on reflection they expressed an understanding of the logic of this approach.


    Principal Findings

    Studies conducted to assess the validity of TNM staging after the most recent revision of specified criteria have demonstrated high levels of correlation between accurately assigned anatomic stages and overall survival for a range of different cancer types [5-14]. In light of the critical role that TNM staging has come to play in the treatment of cancer, it is curious that there do not appear to have been any published studies investigating the approaches that providers adopt to assign anatomic stages. Most of the studies evaluating TNM staging have focused on assessing rates of completion and accuracy of assignment without any examination of the process itself. Inventories of tumor registries have revealed that providers treating patients with specific types of cancers do not always assign anatomic stages or track the information needed to retrospectively confirm accurate assignments [15-17]. Studies that have compared assigned anatomic stages to adjudicated anatomic stages have shown that the accuracy of assignment may vary based on the expertise levels of providers and the specific types of cancers under consideration [18-21].

    Resources that have been developed to assist providers engaged in the task of assigning anatomic stages include printed worksheets, encoded spreadsheets, wizards incorporated in electronic medical records, and an array of apps developed to run on smartphones and tablets [1,22-28]. While the controls and interfaces that they present vary to an extent, these resources universally implement combinatorial approaches to staging that rely on the selection of discrete T, N, M, and prognostic factor classifications to determine an anatomic stage. While the automated linkage of TNM groupings to anatomic stages provided by coded apps may ensure greater accuracy of assignment that obviates the need to refer to tabular arrays, users still need to review the criteria for each T, N, and prognostic group classification to ensure that correct groupings have been identified.


    This evaluation study demonstrated the perceived utility of a Web-based app configured to demonstrate an exclusionary approach to TNM staging. Subjects recruited from a pool of target users found that it was easy to use, and they deemed the approach to assignment that it employed to be informative, efficient, accurate, and reliable. It was interesting to note that while this app was originally developed as an educational resource, the statement that elicited the greatest number of “Disagree” responses focused on its potential use as an instructional aid to help prepare for board certification and re-certification exams. By way of contrast, statements suggesting that it could be used in clinical practice by providers with varying degrees of expertise elicited greater numbers of “Strongly agree” responses. This feedback may guide further development and investigation that may focus on evaluating the performance and acceptance of this app when it is deployed for use in simulated cancer staging exercises and real-time clinical practice.

    Conflicts of Interest

    None declared.


    1. AJCC cancer staging manual. New York: Springer; 2009.
    2. National Comprehensive Cancer Network. NCCN Guidelines. 2014.   URL: [accessed 2014-10-28] [WebCite Cache]
    3. Kim M. TNM Staging. 2013.   URL: [accessed 2015-03-27] [WebCite Cache]
    4. Carlos S. 8 reasons to avoid Flash (or Silverlight) like the plague when designing a website. 2007.   URL: [accessed 2014-10-28] [WebCite Cache]
    5. Loh KC, Greenspan FS, Gee L, Miller TR, Yeo PP. Pathological tumor-node-metastasis (pTNM) staging for papillary and follicular thyroid carcinomas: a retrospective analysis of 700 patients. J Clin Endocrinol Metab 1997 Nov;82(11):3553-3562. [CrossRef] [Medline]
    6. Jonklaas J, Sarlis NJ, Litofsky D, Ain KB, Bigos ST, Brierley JD, et al. Outcomes of patients with differentiated thyroid carcinoma following initial therapy. Thyroid 2006 Dec;16(12):1229-1242. [CrossRef] [Medline]
    7. Tilki D, Reich O, Karakiewicz PI, Novara G, Kassouf W, Ergün S, et al. Validation of the AJCC TNM substaging of pT2 bladder cancer: deep muscle invasion is associated with significantly worse outcome. Eur Urol 2010 Jul;58(1):112-117. [CrossRef] [Medline]
    8. Kwan ML, Haque R, Lee VS, Joanie CWL, Avila CC, Clancy HA, et al. Validation of AJCC TNM staging for breast tumors diagnosed before 2004 in cancer registries. Cancer Causes Control 2012 Sep;23(9):1587-1591 [FREE Full text] [CrossRef] [Medline]
    9. McGhan LJ, Pockaj BA, Gray RJ, Bagaria SP, Wasif N. Validation of the updated 7th edition AJCC TNM staging criteria for gastric adenocarcinoma. J Gastrointest Surg 2012 Jan;16(1):53-61; discussion 61. [CrossRef] [Medline]
    10. Park JS, Choi GS, Hasegawa S, Sakai Y, Huh JW, Kim HR, et al. Validation of the seventh edition of the American Joint Committee on cancer tumor node-staging system in patients with colorectal carcinoma in comparison with sixth classification. J Surg Oncol 2012 Nov;106(6):674-679. [CrossRef] [Medline]
    11. Bergman P, Brodin D, Lewensohn R, de PL. Validation of the 7th TNM classification for non-small cell lung cancer: a retrospective analysis on prognostic implications for operated node-negative cases. Acta Oncol 2013 Aug;52(6):1189-1194. [CrossRef] [Medline]
    12. Fibla JJ, Cassivi SD, Decker PA, Allen MS, Darling GE, Landreneau RJ, ACOSOG Z0030 Study Group. Validation of the lung cancer staging system revisions using a large prospective clinical trial database (ACOSOG Z0030). Eur J Cardiothorac Surg 2013 May;43(5):911-914. [CrossRef] [Medline]
    13. Kee KM, Wang JH, Lin CY, Wang CC, Cheng YF, Lu SN. Validation of the 7th edition TNM staging system for hepatocellular carcinoma: an analysis of 8,828 patients in a single medical center. Dig Dis Sci 2013 Sep;58(9):2721-2728. [CrossRef] [Medline]
    14. Yoshida Y, Murayama T, Sato Y, Suzuki Y, Saito H, Tanaka N. Validation of 7th TNM staging system for lung cancer, based on surgical outcomes. Asian Cardiovasc Thorac Ann 2013 Dec;21(6):693-699. [CrossRef] [Medline]
    15. Liu WL, Kasl S, Flannery JT, Lindo A, Dubrow R. The accuracy of prostate cancer staging in a population-based tumor registry and its impact on the black-white stage difference (Connecticut, United States). Cancer Causes Control 1995 Sep;6(5):425-430. [Medline]
    16. Schouten LJ, Langendijk JA, Jager JJ, van den Brandt PA. Validity of the stage of lung cancer in records of the Maastricht cancer registry, The Netherlands. Lung Cancer 1997 May;17(1):115-122. [Medline]
    17. Klassen AC, Curriero F, Kulldorff M, Alberg AJ, Platz EA, Neloms ST. Missing stage and grade in Maryland prostate cancer surveillance data, 1992-1997. Am J Prev Med 2006 Feb;30(2 Suppl):S77-S87. [CrossRef] [Medline]
    18. McGowan L, Lesher LP, Norris HJ, Barnett M. Misstaging of ovarian cancer. Obstet Gynecol 1985 Apr;65(4):568-572. [Medline]
    19. Fanning J, Gangestad A, Andrews SJ. National Cancer Data Base/Surveillance Epidemiology and End Results: potential insensitive-measure bias. Gynecol Oncol 2000 Jun;77(3):450-453. [CrossRef] [Medline]
    20. Teramoto N, Nishimura R, Mandai K, Matsumoto T, Nogawa T, Hiura M. The importance of precise pT diagnosis for prognostic prediction of uterine cervical cancer--a single institutional report at a Japanese comprehensive cancer hospital. Virchows Arch 2009 Oct;455(4):307-313. [CrossRef] [Medline]
    21. Reese AC, Sadetsky N, Carroll PR, Cooperberg MR. Inaccuracies in assignment of clinical stage for localized prostate cancer. Cancer 2011 Jan 15;117(2):283-289 [FREE Full text] [CrossRef] [Medline]
    22. Sanyal S. An Excel™-based e-Staging Tool for Tumors. In: Stanford Medicine X 2012 Proceedings. Stanford University; 2012 Presented at: Stanford Medicine X; September 28-30, 2012; Stanford, CA.
    23. Bender E. Lung Cancer Stage. 2010.   URL: [accessed 2014-10-28] [WebCite Cache]
    24. Mayrhofer B. Melanoma Staging Calculator. 2011.   URL: [accessed 2014-10-28] [WebCite Cache]
    25. Caceres G. TNM Breast and Bone. 2010.   URL: [accessed 2014-10-28] [WebCite Cache]
    26. Cancer Staging. 2013.   URL: [accessed 2014-10-28] [WebCite Cache]
    27. Guy W. Head & Neck TNM Staging. 2013.   URL: [accessed 2014-10-28] [WebCite Cache]
    28. Kulkarni K. TNM Cancer Staging. 2013.   URL: [accessed 2014-10-28] [WebCite Cache]


    TNM: tumour/node/metastasis

    Edited by G Eysenbach; submitted 25.11.14; peer-reviewed by X Jiang; comments to author 14.03.15; revised version received 17.03.15; accepted 17.03.15; published 02.04.15

    ©Matthew Kim. Originally published in JMIR Cancer (, 02.04.2015.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Cancer, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.