Machine learning systems used in Clinical Decision Support Systems (CDSS) require further external validation, calibration analysis, assessment of bias and fairness. In this course, the main concepts of machine learning evaluation adopted in CDSS will be explained. Furthermore, decision curve analysis along with human-centred CDSS that need to be explainable will be discussed. Finally, privacy concerns of deep learning models and potential adversarial attacks will be presented along with the vision for a new generation of explainable and privacy-preserved CDSS.