Doctoral Dissertation Defense: Francesca McFadden

Title:  Application of Unsupervised Learning Methods to Trusted Decision Making with Machine–Learned Models

Abstract 
With the increasing use of artificial intelligence and machine-learning, many industries have established workflows for model predictions paired with human oversight or alternative algorithmic solutions. Human- machine teaming is the area of research in which human perception and insight are augmented by artificial intelligence or machine-learned models. With increased interest in bringing machine-learned intelligence to more applications, there is an opportunity to ensure that models are "teamed" with appropriately. Specifically, there is an increased need to ensure appropriate trust in the application of the predictions for end-user application or downstream system use.

Model competence methods extend the current application of model reported confidence to ensure that models are being trusted on inputs for which they have appropriate prediction capability and corresponding training data.  Unsupervised learning, namely clustering methods, is applied to enhance current approaches to model competence estimation. A recommended implementation of competence estimation to enable real time integration is addressed. When models are not deemed competent, there is an indication that a human decision, algorithm, heuristic, policy, or alternative competent machine-learned model should be integrated as an intervention.

Ensemble learning combines or selects among the predictions of multiple models. An approach to improve ensemble classification voting schemes through pruning of models not deemed competent in decision is described and demonstrated. The strategy to incorporate competence scores showed better performance compared to a common highest confidence approach while maintaining explainability in the models used for the ensemble prediction.  

Exploration of the application of a similar strategy to ensemble regression required introduction of a point-wise competence estimate for regression models. An approach for integration of competence estimation in ensemble regression is described and demonstrated. The approach showed reduced errors compared to an equal-weighting approach, with maintained explainability in how models were weighted and pruned.

Video call link: https://meet.google.com/aek-pard-uos