Artificial intelligence (AI) is a concept, whose meaning and perception has changed considerably over the last decades. Starting off with individual and purely theoretical research efforts in the 1950s, AI has grown into a fully developed research field of modern times and may arguably emerge as one of the most important technological advancements of mankind. Despite these rapid technological advancements, some key questions revolving around the matter of transparency, interpretability and explainability of an AI’s decision-making remain unanswered. Thus, a young research field coined with the general term Explainable AI (XAI) has emerged from increasingly strict requirements for AI to be used in safety critical or ethically sensitive domains. An important research branch of XAI is to develop methods that help to facilitate a deeper understanding for the learned knowledge of artificial neural systems. In this book, a series of scientific studies are presented that shed light on how to adopt an empirical neuroscience inspired approach to investigate a neural network’s learned representation in the same spirit as neuroscientific studies of the brain.