Modular invariance plays an important role in the study of two-dimensional CFTs. Most famously, it exhibits the universality of CFT spectra at high energies, but there are numerous other applications and generalizations, including in the context of holography. In recent years, a combination of exact results in supersymmetric CFTs and developments in AdS/CFT have sparked renewed interest in possible generalizations of modularity to CFTs in dimensions greater than two. We briefly survey these developments for both supersymmetric and non-supersymmetric CFTs. We then zoom in on the simplest 4d SCFT, a free N=1 chiral multiplet, and its superconformal index. This object is known to obey an exotic type of modular property, which has been called "modular factorization". We provide a Hamiltonian derivation of this property by identifying an underlying factorization of the CFT Hilbert space. In the process, both the physics and geometry behind the exotic modular property will become transparent. As an application, we show that modular factorization can be used to efficiently deduce all lens space partition functions of a dimensionally reduced 3d theory. We end with comments on generalizations to more interesting 4d SCFTs, and more generally d-dimensional (S)CFTs.