In statistical machine learning research, the AUC (Area Under ROC Curve) measure based on $K$-fold cross-validation is always used to measure the classification algorithm performance. However, the point estimation obviously does not consider the variance information. For this reason, the commonly used symmetrical confidence interval (interval estimation) of AUC measure constructed by the $K$-fold cross-validated $t$ distribution based on the normal assumption is proposed. But these symmetrical confidence intervals always exhibit low degrees of confidence or long interval lengths. This may easily result in liberal statistical inference results. Through the theoretical analysis of AUC measure, we find that the real distribution of AUC measure is actually asymmetrical. In this case, it is obviously inappropriate to use symmetrical distribution to approximate asymmetrical distribution. Therefore, for the two-class classification problem, this paper proposes a new asymmetrical confidence interval based on $K$-fold cross-validated Beta distribution. Simulated and real data experiments show the superiority of the proposed confidence interval compared to the traditional symmetrical confidence interval based on $K$-fold cross-validated $t$ distribution.