Abstract: Machine learning models are commonly used in applications with an objective of prediction. The complicated algorithms of many of these models, however, make them difficult to interpret. Methods have been proposed to provide insight into these "black-box" models, but there is little research that focuses on the situation when functional data are used as model inputs. In this work, we propose an explainable pipeline for training machine learning models with functional data that (1) accounts for the vertical and horizontal variability in the functional data, (2) provides insight into the functional variability in the data that is globally important to the model for prediction, and (3) is model agnostic. We refer to the pipeline as the Variable importance Explainable Elastic Shape Analysis (VEESA) pipeline. The VEESA pipeline makes use of the previously developed techniques of joint functional principal components analysis (jfPCA) and permutation feature importance (PFI). We apply the VEESA pipeline to an example related to national security to demonstrate the approach in a scenario where transparency in predictive modeling is essential.