This open access book provides a robust exposition of the mathematical foundations of data representation, focusing on two essential pillars of dimensionality reduction methods, namely geometry in general and Riemannian geometry in particular, and category theory. Presenting a list of examples consisting of both geometric objects and empirical datasets, this book provides insights into the different effects of dimensionality reduction techniques on data representation and visualization, with the aim of guiding the reader in understanding the expected results specific to each method in such scenarios. As a showcase, the dimensionality reduction method of “Uniform Manifold Approximation and Projection” (UMAP) has been used in this book, as it is built on theoretical foundations from all the areas we want to highlight here. Thus, this book also aims to systematically present the details of constructing a metric representation of a locally distorted metric space, which is essentially the problem that UMAP is trying to address, from a more general perspective. Explaining how UMAP fits into this broader framework, while critically evaluating the underlying ideas, this book finally introduces an alternative algorithm to UMAP. This algorithm, called IsUMap, retains many of the positive features of UMAP, while improving on some of its drawbacks.