Mean Field Games (MFGs) model equilibria in games with a continuum of weakly interacting players as limiting systems of symmetric $n$-player games. We consider the finite-state, infinite-horizon problem with two cost criteria: discounted and ergodic. Assuming Markovian controls, we prove convergence of each $n$-player equilibria to their respective mean field limits through the so-called Master Equation approach. Convergence requires regularity results for the discounted and ergodic Master Equations, so we introduce several linearized systems of ODEs to allow this. We prove convergence by constructing an approximate system to the $n$-player game through the Master Equation. Then under stationary distributions associated with jump processes arising from the $n$-player and approximate $n$-player systems, taking the discount factor $r>0$ small enough allows for some results in the discounted setting to transfer over to the ergodic setting. Speaker(s): Ethan Zell (UM)