In metals, orbital motions of conduction electrons on the Fermi surface are quantized in magnetic fields, which is manifested by quantum oscillations in electrical resistivity. This Landau quantization is generally absent in insulators. Here, we report a notable exception in an insulator — ytterbium dodecaboride (YbB12). The resistivity of YbB12, which is of a much larger magnitude than the resistivity in metals, exhibits distinct quantum oscillations. These unconventional oscillations arise from the insulating bulk, even though the temperature dependence of the oscillation amplitude follows the conventional Fermi liquid theory of metals with a large effective mass. Quantum oscillations in the magnetic torque are also observed, albeit with a lighter effective mass. Our result reveals a mysterious dual nature of the ground state in YbB12: it is both a charge insulator and a strongly correlated metal.