The increasing complexity of modern data necessitates flexible statistical approaches capable of uncovering various latent structures, often intricately linked to population heterogeneity. This dissertation explores probabilistic models, particularly latent variable models and hierarchical models, within a Bayesian framework across various settings. It highlights their effectiveness in extracting meaningful patterns and representations from data, while also deepening our theoretical and computational understanding of these models.

The first chapter develops models and theoretical insights for hierarchical topic models, characterized by latent tree-structured topic hierarchies that yield a rich structure formed by multiple topic polytopes sharing faces. The second chapter builds on these geometric insights, extending them to continuous convolutional kernels. In particular, it sheds light on identifiability in general nonparametric mixtures of such distributions, where each component is nearly supported on a low-dimensional affine subspace. The third chapter revisits topic models from a different angle, exploring connections between Latent Dirichlet Allocation and mixtures of product multinomial models via tensor decomposition of the Dirichlet distribution. The fourth chapter examines general hierarchical models in grouped-data settings and extends strong identifiability theory from mixture models to these settings, establishing inverse bounds tailored to specific asymptotic regimes. The fifth chapter develops a nonparametric spatio-temporal model for dynamic velocity fields, with an emphasis on scalable inference. Finally, the last chapter addresses Bayesian methods in sequential decision-making and provides regret guarantees for Thompson sampling in sparse linear contextual bandit problems by analyzing the posterior under dynamic environments.

A recurring theme of this dissertation is the asymptotic analysis of the posterior distribution of model parameters. Parameter learning is significantly more challenging than density estimation for latent variable models, often with complex dependencies across components. Nevertheless, such analysis enhances interpretability and informs the performance of downstream tasks that incorporate these models. It also reveals structural properties that can be leveraged for efficient inference. Overall, this dissertation bridges theoretical and practical aspects of Bayesian modeling and emphasizes its potential to extract interpretable structures from complex datasets.