是否有显著差异性进行Nemenyi检验

Question

请问对两个模型（逻辑回归和随机森林）在多个不平衡率的数据集中各个性能度量指标（训练集的acc、auc、precision、recall、f1已经得出具体数值）是否有显著差异性进行Nemenyi检验的Python代码应该怎么写？
以下是现在已完成的在其中一个数据集中得出两个模型的性能指标的代码：

# 初始化随机森林分类器
lg=LogisticRegression(random_state=42)
rf = RandomForestClassifier(n_estimators=100, random_state=42)
# 交叉验证
skfolds=StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
accuracy_score_list1,precision_score_list1,recall_score_list1,f1_score_list1, auc_score_list1=[], [], [], [], []
accuracy_score_list2,precision_score_list2,recall_score_list2,f1_score_list2, auc_score_list2=[], [], [], [], []

# 拆分数据集
for train_index,test_index in skfolds.split(X,y):
    X_train=X.iloc[train_index]
    y_train=y.iloc[train_index]
    X_test=X.iloc[test_index]
    y_test=y.iloc[test_index]
#随机过采样，预测    
    X_resampled,y_resampled=ros.fit_resample(X_train,y_train)
    lg.fit(X_resampled,y_resampled)
    y_pred1=lg.predict(X_test)
    
    rf_classifier=rf.fit(X_resampled,y_resampled)
    y_pred2=rf_classifier.predict(X_test)
    
#评估 
    #lg
    from sklearn import metrics
    y_score1=lg.predict_proba(X_test)[:,1]
    fpr1, tpr1,threshold1=metrics.roc_curve(y_test,y_score1)
    roc_auc1=metrics.auc(fpr1,tpr1)
    
    AccuracyScore1=accuracy_score(y_test,y_pred1)
    PrecisionScore1=precision_score(y_test,y_pred1)
    RecallScore1=recall_score(y_test,y_pred1)
    F1Score1=f1_score(y_test, y_pred1)
    
    accuracy_score_list1.append(AccuracyScore1)
    precision_score_list1.append(PrecisionScore1)
    recall_score_list1.append(RecallScore1)
    f1_score_list1.append(F1Score1)
    auc_score_list1.append(roc_auc1)
    
    #rf
    y_score2=rf.predict_proba(X_test)[:,1]
    fpr2, tpr2,threshold2=metrics.roc_curve(y_test,y_score2)
    roc_auc2=metrics.auc(fpr2,tpr2)
    
    AccuracyScore2=accuracy_score(y_test,y_pred2)
    PrecisionScore2=precision_score(y_test,y_pred2)
    RecallScore2=recall_score(y_test,y_pred2)
    F1Score2=f1_score(y_test, y_pred2)
    
    accuracy_score_list2.append(AccuracyScore2)
    precision_score_list2.append(PrecisionScore2)
    recall_score_list2.append(RecallScore2)
    f1_score_list2.append(F1Score2)
    auc_score_list2.append(roc_auc2)

print('Accuracy1:%0.3f(+/-%0.3f)'%(np.average(accuracy_score_list1),np.std(accuracy_score_list1)*2))
print('Precision1:%0.3f(+/-%0.3f)'%(np.average(precision_score_list1),np.std(precision_score_list1)*2))
print('Recall1:%0.3f(+/-%0.3f)'%(np.average(recall_score_list1),np.std(recall_score_list1)*2))
print('F1_score1:%0.3f(+/-%0.3f)'%(np.average(f1_score_list1),np.std(f1_score_list1)*2))
print('AUC1:%0.3f(+/-%0.3f)'%(np.average(auc_score_list1),np.std(auc_score_list1)*2))
print('Classification Report:\n', classification_report(y_test, y_pred1))

print('Accuracy2:%0.3f(+/-%0.3f)'%(np.average(accuracy_score_list2),np.std(accuracy_score_list2)*2))
print('Precision2:%0.3f(+/-%0.3f)'%(np.average(precision_score_list2),np.std(precision_score_list2)*2))
print('Recall2:%0.3f(+/-%0.3f)'%(np.average(recall_score_list2),np.std(recall_score_list2)*2))
print('F1_score2:%0.3f(+/-%0.3f)'%(np.average(f1_score_list2),np.std(f1_score_list2)*2))
print('AUC2:%0.3f(+/-%0.3f)'%(np.average(auc_score_list2),np.std(auc_score_list2)*2))
print('Classification Report:\n', classification_report(y_test, y_pred2))

Answer 1

以下内容由CHATGPT及阿里嘎多学长共同生成、有用望采纳：

---

问题概括

这个问题的目的是评估两个不同的机器学习模型（逻辑回归和随机森林）在多个不平衡数据集上的性能度量指标（准确率、AUC、精确度、召回率、F1分数）是否存在显著差异，并使用Nemenyi检验来进行统计分析。

问题出现原因

问题出现的原因是在进行模型性能比较时，需要一个统计方法来确定两个模型的性能差异是否具有统计学意义。Nemenyi检验是一种常用的多重比较方法，适用于这种情况。

问题解决方案

为了解决这个问题，我们需要使用Python中的统计或科学计算库来执行Nemenyi检验。一个常用的库是scipy，它提供了进行Nemenyi检验的函数。

提供代码

以下是使用scipy进行Nemenyi检验的示例代码，这段代码基于您提供的代码片段进行了扩展：

import numpy as np
from scipy import stats
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score
from sklearn.model_selection import StratifiedKFold
from imblearn.over_sampling import RandomOverSampler
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier

# 初始化模型
lg = LogisticRegression(random_state=42)
rf = RandomForestClassifier(n_estimators=100, random_state=42)

# 交叉验证设置
skfolds = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

# 性能指标列表
performance_scores = {
    'lg': {'accuracy': [], 'precision': [], 'recall': [], 'f1': [], 'auc': []},
    'rf': {'accuracy': [], 'precision': [], 'recall': [], 'f1': [], 'auc': []}
}

# 交叉验证过程
for train_index, test_index in skfolds.split(X, y):
    X_train, X_test = X.iloc[train_index], X.iloc[test_index]
    y_train, y_test = y.iloc[train_index], y.iloc[test_index]
    
    # 随机过采样
    ros = RandomOverSampler(random_state=42)
    X_resampled, y_resampled = ros.fit_resample(X_train, y_train)
    
    # 逻辑回归模型训练和评估
    lg.fit(X_resampled, y_resampled)
    y_pred1 = lg.predict(X_test)
    y_score1 = lg.predict_proba(X_test)[:, 1]
    fpr1, tpr1, _ = metrics.roc_curve(y_test, y_score1)
    roc_auc1 = metrics.auc(fpr1, tpr1)
    
    performance_scores['lg']['accuracy'].append(accuracy_score(y_test, y_pred1))
    performance_scores['lg']['precision'].append(precision_score(y_test, y_pred1))
    performance_scores['lg']['recall'].append(recall_score(y_test, y_pred1))
    performance_scores['lg']['f1'].append(f1_score(y_test, y_pred1))
    performance_scores['lg']['auc'].append(roc_auc1)
    
    # 随机森林模型训练和评估
    rf.fit(X_resampled, y_resampled)
    y_pred2 = rf.predict(X_test)
    y_score2 = rf.predict_proba(X_test)[:, 1]
    fpr2, tpr2, _ = metrics.roc_curve(y_test, y_score2)
    roc_auc2 = metrics.auc(fpr2, tpr2)
    
    performance_scores['rf']['accuracy'].append(accuracy_score(y_test, y_pred2))
    performance_scores['rf']['precision'].append(precision_score(y_test, y_pred2))
    performance_scores['rf']['recall'].append(recall_score(y_test, y_pred2))
    performance_scores['rf']['f1'].append(f1_score(y_test, y_pred2))
    performance_scores['rf']['auc'].append(roc_auc2)

# 进行Nemenyi检验
for metric in performance_scores['lg']:
    # 将两个模型的该性能指标得分转换为数组
    scores_lg = np.array(performance_scores['lg'][metric])
    scores_rf = np.array(performance_scores['rf'][metric])
    
    # 执行Friedman检验
    friedman_stat, p = stats.friedmanchisquare(scores_lg, scores_rf)
    
    # 如果Friedman检验显著，执行Nemenyi后验检验
    if p < 0.05:
        nemenyi_result = stats.multiscale(scores_lg, scores_rf, method='nemenyi')
        print(f"Nemenyi Test for {metric}: {nemenyi_result}")
    else:
        print(f"No significant difference found by Friedman test for {metric}.")

示例代码运行方式

这段代码需要在具有Python环境的机器上运行，并且需要安装scikit-learn, imbalanced-learn, numpy, 和 scipy 这些库。可以使用pip命令安装所需的库：

pip install scikit-learn imbalanced-learn numpy scipy

运行代码前，请确保您的数据集X和目标变量y已经准备好，并且已经导入到您的Python环境中。

示例代码预期运行结果

代码将输出每个性能指标的Nemenyi检验结果，包括是否有显著差异以及相关的统计数据。如果Friedman检验的p值小于0.05，那么将执行Nemenyi后验检验，并输出检验结果。如果p值大于或等于0.05，则表明两个模型在该性能指标上没有显著差异。