Int J Biol Sci 2019; 15(1):221-228. doi:10.7150/ijbs.28720
Development and Validation of a Prognostic Nomogram in AFP-negative hepatocellular carcinoma
1. Department of Laboratory Medicine, State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-sen University Cancer Center, Guangzhou, China;
2. Department of Information section, State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-sen University Cancer Center, Guangzhou, China;
3. Department of Experimental Research, State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-sen University Cancer Center, Guangzhou, China;
4. Department of Pathology, State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-sen University Cancer Center, Guangzhou, China.
#Authors contributed equally to this work.
*Authors contributed equally to this work.
Wang X, Mao M, He Z, Zhang L, Li H, Lin J, He Y, Dai S, Hu W, Liu W. Development and Validation of a Prognostic Nomogram in AFP-negative hepatocellular carcinoma. Int J Biol Sci 2019; 15(1):221-228. doi:10.7150/ijbs.28720. Available from http://www.ijbs.com/v15p0221.htm
The aim of this study is to establish and validate an effective prognostic nomogram in patients with AFP-negative hepatocellular carcinoma (HCC). The nomogram was based on a primary cohort that consisted of 419 patients with clinicopathologically diagnosed with HCC, all the data was gathered from 2008 to 2014 in Sun Yat-sen University Cancer Center. All the model factors were determined by univariate and multivariate Cox hazard analysis. The concordance index (C-index) and calibration curve were used to determine the predictive accuracy and discriminative ability of the nomogram, and compared with the TNM staging systems on HCC. Internal validation was assessed. An independent validation cohort contained 150 continuous patients from 2014 to 2015. Independent factors for overall survival (OS) were body mass index (BMI), tumor stage, distant metastases, HBs Ag, lactate dehydrogenase (LDH), gamma-glutamyl transpeptidase (GGT), and albumin (ALB), which were all contained into the nomogram. The calibration curve for probability of OS showed good agreement between prediction by nomogram and actual observation. The C-index of nomogram was 0.807 (95% CI: 0.770-0.844), which was superior to the C-index of AJCC TNM Stage (0.697). The AUC was 0.809(95%CI: 0.762-0.857). In the validation cohort, the nomogram still gave good discrimination (C-index: 0.866, 95% CI: 00.796-0.936; AUC: 0.832, 95%CI: 0.747-0.917) and good calibration. Decision curve analysis demonstrated that the nomogram was clinically useful. Moreover, patients were divided into three distinct risk groups for OS by the nomogram: low risk group, middle risk group and a high risk group, respectively. The proposed nomogram presents more accurate and useful prognostic prediction for patients with AFP-negative HCC.
Keywords: hepatocellular carcinoma, prognosis, nomogram, liver function
Hepatocellular carcinoma (HCC) is a leading cause of cancer deaths global, which is the fifth most common malignancy and the third leading cause of cancer-related death . As with many cancers, HCC in early stage has better prognosis compared to advanced stage disease [1, 2]. Despite imaging technology has greatly improved the detection of HCC; biochemical analyses are still indispensable for its diagnosis, especially for the early stage of disease. Alpha fetal protein (AFP) is still the only available blood test for detection and surveillance of HCC; However, is limited by its poor sensitivity and has proven to be a less than ideal surrogate for monitoring treatment response of HCC , such as elevated serum AFP was only observed in 60-70% of overall HCC patients, while the proportion was merely 33-65% regarding patients harboring HCCs of <3 cm in diameter [4, 5]. Therefore, inefficient diagnosis of early stage HCC remains a primary causal factor of the high mortality and poor prognosis.
Biochemical parameters of liver function test (LFT) is responsible for metabolism and excretion of various endogenous and foreign substances.In patients with liver-specific diseases, accurate assessment of liver function is critical for the selection of treatment options. Hepatitis and cirrhosis, for example, are associated with an increased risk of liver failure after partial liver resection, especially after neo-adjuvant chemotherapy, or in living donor liver transplantation .
Currently, the American Joint Committee on Cancer (AJCC) TNM classification, based on pathological information, and the treatment regimens for HCC patients were established according to the staging system . Large variations are reported in the clinical outcomes, even patients with the same stage receiving similar treatment strategies .This findings indicate that the present staging system is inadequate for predicting recurrence and does not reflect the biological heterogeneity of HCC patients. However, many other risk factors, such as age, sex, alcohol status, body mass index (BMI), HBs Ag, LFT, coagulation tests have been demonstrated to influence recurrence in HCC patients  and should be considered for predicting individualized prognosis, the same as the AFP-negative HCC. Therefore, a comprehensive, easy-to-use tool that estimates individual risk by incorporating TNM stage and LFT factors could serve as a valuable decision-making tool for clinicians.
Therefore, the aim of our study was to develop and validate a nomogram that combined both clinicopathologic factors, LFT factors and other tumor markers for the prognosis in patients with AFP-negative HCC [11-13]. We also performed a test to determine whether this model provides a more accurate prediction of prognosis when compared with TNM staging systems .
The primary cohort of our study comprised 419 patients with histologically diagnosed HCC who were retrospectively reviewed from the information system from April 2008 to April 2014. All the patients had undergone surgical resection with curative intent. Only the first records of hospitalizations were retained, and the levels of LFT factors and tumor markers were investigated before treatment. However, cases with concomitant diseases which might influence serum LFT levels (i.e., diabetes, hyperlipidemia, or metabolic syndrome) were excluded. In addition, patients with other types of tumors were also excluded. The stage of tumor was evaluated using the American Joint Committee on Cancer Staging system (AJCC, 2002; Greene). From April 2014 to January 2015, 150 consecutive patients were enrolled in independent validation cohort using the same inclusion and exclusion criteria as that in the primary cohort. The patients in the validation cohort were restaged according to the seventh AJCC TNM staging manual. The study was approved by the ethics committees in Sun Yat-sen University Cancer Center (SYSUCC, Guangdong, China). It was conducted in accordance with the ethical standards of the World Medical Association Declaration of Helsinki. The authenticity of this article has been validated by uploading the raw data onto the Research Data Deposit public platform (http://www.researchdata.org.cn), with the approval RDD Number as RDDA2018000808.
Patients received routine tests at the first visit in our hospital. Serum samples were collected and clotted at room temperature, then centrifuged at 3500 r/min for 10 min, which could be used to estimate the level of serum biomarkers, including alanine aminotransferase (ALT), aspartate aminotransferase (AST), prothrombin time (PT), lactate dehydrogenase (LDH), gamma-glutamyl transpeptidase (GGT), albumin (ALB), carcino-embryonic antigen (CEA) and CA199. The Baseline clinical data, including age, gender, preoperative histologic grade, and HBV infection were extracted from the Electronic Medical Record (EMR) system. Informed consent was obtained from each patient prior to use of serum and plasma. All patients provided written informed consent. The Institute Research Ethics Committee of the Sun Yat-Sen University Cancer Center, Guangzhou, China approved this study.
All HCC patients were advised to receive regular follow-ups after completion of the primary therapy according to clinical guidelines. Patients were generally followed up every 3 months in the first 2 years and annually thereafter for patients without evidence of recurrence in the following 3 to 5 years. Patients who did not visit our hospital as scheduled were telephoned for follow-ups to obtain the treatment information and living status (performed by The Medical Information Unit in our Cancer Center). The last follow-up occurred in June 2016. The outcome of our study was overall survival (OS). OS was defined as the time from the diagnosis of HCC to the date of the last follow-up or death.
Statistical analysis was done using SPSS 16.0 (IBM, Chicago, IL, USA) and R software (version 3.1.4; http://www.Rproject.org). Categorical variables were classified based on clinical findings. All the optimal cut-off points in our study were evaluated by reference ranges and continuous variables were transformed to categorical variables. Univariate and multivariate regression analysis was used to analyze the risk factors, to predict prognosis selected for the derivation of prediction models.
Risk factors which based on the multivariable logistic analysis were applied to develop a diagnostic model for AFP-negative HCC by using the primary cohort. Backward step down selection was performed as the final stopping rule with the Akaike information criterion. The nomogram was constructed for predicting 3 and 5years OS. Nomogram validation consisted of discrimination and calibration by using the primary and validation set. To quantify the discrimination performance of the nomogram, Harrell's C-index was evaluated. In brief, a C-index value greater than 0.75 is considered to represent relatively good discrimination. Calibration was performed by observing survival probability with Kaplan-Meier estimating. In the validation cohort, according to the established nomogram, the C-index and calibration curve were derived based on the regression analysis. The decision curve was also plotted for the model of nomogram and TNM staging system [16, 17]. The total points of each patient were calculated according to the established Cox regression model, 2 groups of patients with high and low risk of prognosis (based on the total points) were delineated using maximally selected rank statistics as implemented in the maxstat package. Survival curves were depicted by the Kaplan-Meier method, and using the dichotomized risk group as a factor, finally, compared using the log-rank test. A two tailed P value < 0.05 was considered statistically significant.
For nomogram construction and validation, we assigned three quarters of the patients to the primary set and one quarter to the validation set. The clinic pathologic characteristics of the training and validation sets were evaluated. The characteristics of the 419 consecutive AFP-negative HCC patients in the primary cohort and 150 patients in the validation cohort are showed in Table 1. For the primary cohorts, there were 105 patients died in cancer for 5 years.
Patient demographics and clinical characteristics
|Primary cohort (419)||Validation cohort (150)||P value|
|Characteristic||No. (%)||No. (%)|
All the available information's, including clinic pathologic characteristics and biomarkers, were included for univariate and multivariate analysis (Table 2). In univariate analyses, there were significant correlation between BMI, TNM stage, tumor stage, node stage, and distant metastases, HBs Ag, AST, LDH, GGT, ALB, PT, CEA, CA199 and OS. The multivariate analyses were then performed to identify factors distinguished in univariate analyses. Result showed that BMI, tumor stage, distant metastases, HBs Ag, LDH, GGT and ALB, were independent risk factors for prognosis of AFP-negative HCC.
Development and Validation of the Prediction Model
A nomogram was constructed to predict 3- and 5- year OS, on the basis of the identified prognostic factors (Figure 1). The validation of nomogram was consisted of discrimination and calibration by using the validation set. Discrimination was performed by using a concordance index (C-index) and ROC curves. Calibration was evaluated by comparing the means of predicted survival with estimating of predicted with observed Kaplan-Meier survival, with the x-axes are actual survival estimated by the nomogram, the y-axes are observed survival calculated by the Kaplan-Meier method. The C-index for OS prediction was 0.807 (95% CI: 0.770-0.844). The AUC (ROC curve) was 0.809(95%CI: 0.762-0.857), with sensitivity 85.00%, specificity 68.00%, PPV 48% and NPV 93% (Figure 2A). The calibration plot for the probability of OS at 3 or 5 year after therapy showed an optimal agreement between the prediction by nomogram and actual observation (Figure 3).
Validation of the Predictive Accuracy of Nomograms for OS
In the validation cohort, The C-index for OS prediction was up to 0.866 (95% CI: 0.796-0.936). The AUC was 0.832 (95%CI: 0.747-0.917), with sensitivity 89.50%, specificity 68.00%, PPV 30.00% and NPV 98.00% (Figure 2B). The calibration plot for the probability of OS at 3- year after therapy showed an optimal agreement between the prediction by nomogram and actual observation (Figure 3).
Univariate and multivariate cox hazards analysis of the primary cohort
|Univariate analysis||Multivariate analysis|
|Characteristic||HR (95% CI)||P||HR (95% CI)||P|
|<57 vs. ≥57||1.099(0.748-1.614)||0.632|
|Male vs. Female||1.112(0.609-2.029)||0.729|
|<18.5 vs. 18.5-25 vs. ≥25||0.704(0.531-0.934)||0.015||0.677(0.498-0.921)||0.013|
|No vs. Yes||0.712(0.423-1.197)||0.199|
|No vs. Yes||0.933(0.623-1.396)||0.735|
|0-1 vs. 2||o.547(0.076-3.924)||0.549|
|Ⅰ vs. Ⅱ vs. Ⅲ-Ⅳ||2.273(1.813-2.851)||<0.001|
|T1 vs. T2 vs. T3-T4||2.348(1.869-2.951)||<0.001||1.697(1.130-2.197)||<0.001|
|N0 vs. N1||2.653(1.419-4.959)||0.002||1.903(0.952-3.805)||0.069|
|M0 vs. M1||2.711(1.413-5.205)||0.003||2.047(1.016-4.125)||0.045|
|Negative vs. Positive||0.514(0.328-0.808)||0.004||0.489(0.300-0.799)||0.004|
|<40 vs. ≥40||1.981(1.344-2.920)||0.001||0.799(0.485-1.316)||0.379|
|<50 vs. ≥50||1.225(0.827-1.813)||0.311|
|<250 vs. ≥250||3.690(2.486-5.477)||<0.001||2.561(1.642-3.994)||<0.001|
|<60 vs. ≥60||3.206(2.106-4.878)||<0.001||2.551(1.569-4.149)||<0.001|
|<28 vs. 28-35 vs. ≥35||0.315(0.222-0.447)||<0.001||2.549(1.647-3.946)||<0.001|
|<34.2 vs. 34.2-51.3 ≥51.3||1.471(0.870-2.487)||0.150|
|<13.5 vs. ≥13.5||1.930(1.196-3.114)||0.007||0.938(0.538-1.636)||0.821|
|<5 vs. ≥5||2.041(1.293-3.220)||0.002||1.266(0.766-2.092)||0.358|
|<35 vs. ≥35||1.768(1.195-2.617)||0.004||1.194(0.763-1.870)||0.438|
Nomogram, including BMI, tumor stage, distant metastases, HBs Ag, LDH, GGT and ALB, for three and five years overall survival (OS) in patients with AFP-negative HCC. The nomogram is valued to obtain the probability of three and five years survival by adding up the points identified on the points scale for each variable.(Click on the image to enlarge.)
ROC curve of the nomogram in the primary and validation cohort. A. The AUC for OS was 0.809 in the primary cohort. B. The AUC for OS was 0.832 in the validation cohort.(Click on the image to enlarge.)
Calibration curve of the nomogram in the primary and validation cohort, with the x-axes are actual survival estimated by the nomogram, the y-axes are observed survival calculated by the Kaplan-Meier method. A. Three-year OS in the primary cohort. B. Five-year survival OS in the primary cohort. C. Three-year OS in the validation cohort.(Click on the image to enlarge.)
Decision curve analysis
The decision curve analysis for the nomogram and TNM staging systems is showed in Figure 4. The decision curve presented that if the threshold probability of a patient is > 10%, the developed nomogram and TNM staging system in predicting OS is more benefit than all patients dead scheme or none patients dead scheme. Furthermore, the net benefit was comparable; the nomogram in predicting OS is more benefit than that of TNM staging system in this range.
Decision curve analysis for overall survival. Black line: All patients dead. Gray line: None patients dead. Black dashed line: Model of nomogram. Red dashed line: Model of TNM staging system(Click on the image to enlarge.)
Comparison of Predictive Accuracy for OS Between Nomogram and TNM Stage Systems
Based on the nomogram we developed in this study, patients were subdivided into a low-risk group, and a high-risk group, which showed good prognostic classification for HCC patients both in primary cohort and validation cohort. In the primary cohort, there were 232 patients in the low-risk group, 89 patients in the middle-risk group, while 73 patients in the high-risk group. The OS between the 3 risk groups were (37.76 ± 14.44) months, (31.39 ± 17.93) and (25.63 ± 18.82) months (p < 0.001). Also, in the validation cohort, there were 121 patients in the low-risk group, 17 patients in the middle-risk group, while 6 patients in the high-risk group. The OS between the 3 risk groups were (22.45 ± 8.96) months, (14.59 ± 9.19) months and (9.83 ± 10.18) months (p <0.001) (Figure 5).
Furthermore, the discrimination of the nomogram and that of the AJCC TNM Stage have been compared. In the primary cohort, the C-index of nomogram was 0.807 (95% CI: 0.770-0.844), which was superior than the C-index of AJCC TNM Stage(0.697, 95% CI: 0.649-0.745, P < 0.001),or that of other biomarkers (Table 3).
The C-index of Eight Significant Risk Factors in the Primary Cohort
Kaplan-Meier survival curves of nomogram. A. In the primary cohort. B. In the validation cohort.(Click on the image to enlarge.)
The accurate tumor prognosis after definitive treatment is important. Based on the conventional TNM stage system, there are varies controversies, that the system only take the anatomical extent of the disease into account and not think about the liver biological function heterogeneity of AFP-negative HCC, which could not completely reflect the accurate prognosis. Nomograms have been developed and shown to be more accurate than the conventional staging systems for predicting prognosis in some cancers [18-20]. Thus, we planned to develop and validate a prognostic nomogram, which included serum/plasma liver biological function biomarker, had better predictive accuracy than those of the traditional TNM stage system.
For construction of the nomogram, clinic pathologic factors, biomarkers into the TNM staging system and liver biological function have been examined by Multivariate analysis . This method could choose the predictors for the OS of AFP-negative HCC. The nomogram were combined with tumor stage, metastasis stage, BMI, LDH level, GGT level, ALB level and HBs Ag status. The nomogram performed well in predicting overall survival, which showed adequate discrimination in the primary cohort (C-index, 0.807; AUC: 0.809), which was then surprisingly improved in the validation cohort (C-index, 0.866; AUC: 0.832). When compared with the conventional TNM stage system(C-index, 0.697, P < 0.01) or other biomarkers, the nomogram showed better predictive accuracy for OS. Furthermore, both in primary cohorts and validation cohort, patients were divided into three distinct risk groups for OS based on the nomogram, which could effectively discriminate the survival outcomes. Thus, the nomogram, which composed of the clinical and biomarkers we already have, could be used to a more convenient biomarker for the prediction of OS and treatment strategies guidance for AFP-negative HCC [22, 23].
AFP is the best tumour marker of HCC, and it is used for the clinical diagnosis of liver cancer screening, prognostic judgement and recurrence monitoring . However, recent studies reported that the sensitivity of AFP for the diagnosis of HCC is only 40-65%, and the specificity is 76-96%. Notably, AFP expression in many cases of liver cancer is not elevated or even expressed . Our studies have intended to identify a diagnostic factor for AFP-negative HCC.
TNM stage system is the most common staging system of HCC, which is composed of tumor stage, node stage and metastasis stage, could be served as treatment guideline and the independent risk factor for the prognosis of AFP . Unexpectedly, the prognosis of HCC is not only related to tumor factors (TNM stage), but also to the liver function of patients, only the integration of TNM stage may introduce sampling bias to the treatment and the prognosis . Therefore, we recommended that the TNM stage integrate LFTs for the OS prediction. In this study, tumor diameter and tumor number (tumor stage, C-index, 0.701) which reflected the invasiveness of HCC, were significantly associated with prognosis on multivariate analysis. Furthermore, the tumor metastasis was a strong risk factor(C-index, 0.529). The prognosis of the patients with tumor metastasis was significantly poorer. The LFTs, including LDH (C-index, 0.630), GGT (C-index, 0.649) and ALB (C-index, 0.574) have been suggested to be independent risk factors for prognosis of AFP-negative HCC. Furthermore, the basic condition of patients, such as BMI (C-index, 0.560) and HBs Ag (C-index, 0.561) also included in the nomogram. But tumor markers have not been included in the nomogram. The decision curve showed that the nomogram in predicting OS is more benefit than that of TNM staging system in all range .
Several limitations in our study. First, the nomogram was established based on data obtained from one institution in China. Second, in the validation cohort the follow-up time was shorter, and close monitoring and five-year follow-up data are still required for patients in the validation cohort. The third limitation is more patients needed both in the primary and the validation cohort.
In summary, we developed and validated nomograms to predict the three- and five-year OS for AFP-negative HCC. The proposed nomogram in this study provided statistically significantly better discrimination than the current TNM stage, and it offers a useful tool for prognosis. To generalize the use of this nomogram in other groups, additional validation with data from other institutions is required.
HCC: hepatocellular carcinoma; AFP: alpha-fetoprotein; LFTs: Liver function tests; BMI: body mass index; AST: aspartate aminotransferase; ALT: alanine aminotransferase; LDH: lactate dehydrogenase; GGT: gamma-glutamyl transpeptidase; ALB: albumin; TBIL: total bilirubin; PT: prothrombin time; CEA: carcino-embryonic antigen; CA199: carbohydrate antigen 199; HR: hazard ratio; 95% CI: 95% confidence interval; OS: overall survival.
We thank the staff of the biochemical laboratory of Sun Yat-sen University Cancer Center who provided various biochemical markers, and all of the staff who supported our study.
Ethics approval and consent to participate
Written informed consent for the use of plasma and serum samples was obtained from all HCC patients. This study was approved by the Institute Research Ethics Committee of the Sun Yat-Sen University Cancer Center, Guangzhou, China.
Availability of data and material
Due to ethical restrictions, the raw data underlying this paper are available upon request to the corresponding author or the Research Data Deposit public platform (http://www.researchdata.org.cn, with the approval RDD Number as RDDA2018000808).
XPW, MJM, ZLH and LZ. contributed equally to this manuscript. XPW, MJM. and ZLH designed the experiments; LZ, HLL. and JHL collected data; MJM ,YH and WM H analyzed data; SQD, XPW and MJM provided research materials and methods; XPW and WLL wrote the manuscript. All authors read and approved the final manuscript.
The authors have declared that no competing interest exists.
1. Ferlay J, Shin HR, Bray F, Forman D, Mathers C, Parkin DM. Estimates of worldwide burden of cancer in 2008: GLOBOCAN 2008. INT J CANCER. 2010;127(12):2893-2917
2. Bruix J, Sherman M. Management of hepatocellular carcinoma. HEPATOLOGY. 2005;42(5):1208-1236
3. Johnson PJ. Role of alpha-fetoprotein in the diagnosis and management of hepatocellular carcinoma. J Gastroenterol Hepatol. 1999;14(Suppl):S32-S36
4. She S, Xiang Y, Yang M, Ding X, Liu X, Ma L, Liu Q, Liu B, Lu Z, Li S. et al. C-reactive protein is a biomarker of AFP-negative HBV-related hepatocellular carcinoma. INT J ONCOL. 2015;47(2):543-554
5. Taketa K. Alpha-fetoprotein: reevaluation in hepatology. HEPATOLOGY. 1990;12(6):1420-1432
6. van den Broek MA, Olde DS, Dejong CH, Lang H, Malago M, Jalan R, Saner FH. Liver failure after partial hepatic resection: definition, pathophysiology, risk factors and treatment. LIVER INT. 2008;28(6):767-780
7. El-Serag HB. Hepatocellular carcinoma. N Engl J Med. 2011;365(12):1118-1127
8. Zhang G, Li R, Zhao X, Meng S, Ye J, Zhao L. Validation of the American Joint Committee on Cancer eighth edition staging system in patients undergoing hepatectomy for hepatocellular carcinoma: a US population-based study. J SURG RES. 2018;222:55-68
9. Mao YP, Xie FY, Liu LZ, Sun Y, Li L, Tang LL, Liao XB, Xu HY, Chen L, Lai SZ. et al. Re-evaluation of 6th edition of AJCC staging system for nasopharyngeal carcinoma and proposed improvement based on magnetic resonance imaging. Int J Radiat Oncol Biol Phys. 2009;73(5):1326-1334
10. Wang XP, Mao MJ, He ZL, Zhang L, Chi PD, Su JR, Dai SQ, Liu WL. A retrospective discussion of the prognostic value of combining prothrombin time(PT) and fibrinogen(Fbg) in patients with Hepatocellular carcinoma. J CANCER. 2017;8(11):2079-2087
11. Iasonos A, Schrag D, Raj GV, Panageas KS. How to build and interpret a nomogram for cancer prognosis. J CLIN ONCOL. 2008;26(8):1364-1370
12. Sternberg CN. Are nomograms better than currently available stage groupings for bladder cancer?. J CLIN ONCOL. 2006;24(24):3819-3820
13. Karakiewicz PI, Briganti A, Chun FK, Trinh QD, Perrotte P, Ficarra V, Cindolo L, De la Taille A, Tostain J, Mulders PF. et al. Multi-institutional validation of a new renal cancer-specific survival nomogram. J CLIN ONCOL. 2007;25(11):1316-1322
14. Thirunavukarasu P, Talati C, Munjal S, Attwood K, Edge SB, Francescutti V. Effect of Incorporation of Pretreatment Serum Carcinoembryonic Antigen Levels Into AJCC Staging for Colon Cancer on 5-Year Survival. JAMA SURG. 2015;150(8):747-755
15. Harrell FJ, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. STAT MED. 1996;15(4):361-387
16. Vickers AJ, Cronin AM, Elkin EB, Gonen M. Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Med Inform Decis Mak. 2008;8:53
17. Pauker SG, Kassirer JP. The threshold approach to clinical decision making. N Engl J Med. 1980;302(20):1109-1117
18. Tang LQ, Li CF, Li J, Chen WH, Chen QY, Yuan LX, Lai XP, He Y, Xu YX, Hu DP. et al. Establishment and Validation of Prognostic Nomograms for Endemic Nasopharyngeal Carcinoma. J Natl Cancer Inst. 2016:108 (1)
19. Huang YQ, Liang CH, He L, Tian J, Liang CS, Chen X, Ma ZL, Liu ZY. Development and Validation of a Radiomics Nomogram for Preoperative Prediction of Lymph Node Metastasis in Colorectal Cancer. J CLIN ONCOL. 2016;34(18):2157-2164
20. Wang Y, Li J, Xia Y, Gong R, Wang K, Yan Z, Wan X, Liu G, Wu D, Shi L. et al. Prognostic nomogram for intrahepatic cholangiocarcinoma after partial hepatectomy. J CLIN ONCOL. 2013;31(9):1188-1195
21. Harrell FJ, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. STAT MED. 1996;15(4):361-387
22. Sternberg CN. Are nomograms better than currently available stage groupings for bladder cancer?. J CLIN ONCOL. 2006;24(24):3819-3820
23. Mariani L, Miceli R, Kattan MW, Brennan MF, Colecchia M, Fiore M, Casali PG, Gronchi A. Validation and adaptation of a nomogram for predicting the survival of patients with extremity soft tissue sarcoma using a three-grade system. CANCER-AM CANCER SOC. 2005;103(2):402-408
24. Song P, Tobe RG, Inagaki Y, Kokudo N, Hasegawa K, Sugawara Y, Tang W. The management of hepatocellular carcinoma around the world: a comparison of guidelines from 2001 to 2011. LIVER INT. 2012;32(7):1053-1063
25. Asrih M, Lenglet S, Mach F, Montecucco F. Alpha-fetoprotein: a controversial prognostic biomarker for small hepatocellular carcinoma. World J Gastroenterol. 2013;19(3):328-330
26. Sherman M. Staging for hepatocellular carcinoma: complex and confusing. GASTROENTEROLOGY. 2014;146(7):1599-1602
27. Abdel-Rahman O. Assessment of the discriminating value of the 8th AJCC stage grouping for hepatocellular carcinoma. HPB (Oxford). 2018;20(1):41-48
28. Vickers AJ, Jang K, Sargent D, Lilja H, Kattan MW. Systematic review of statistical methods used in molecular marker studies in cancer. CANCER-AM CANCER SOC. 2008;112(8):1862-1868
Corresponding author: Wanli Liu, Department of Laboratory Medicine, Sun Yat-sen University Cancer Center, Guangzhou, China(e-mail: liuwanli126com). Tel./Fax: +86-20-8734-3438