Why can you assume $r_{\rm out}$ is infinite?

This is really only a good approximation in the saturation regime for a FET. $r_{\rm out}$ is effectively the reciprocal of the slope of the $I_d$ vs $V_{ds}$ curve. In the saturation regime, the curve is very flat (no more current for more applied $V_{ds}$), with a slope near zero, and the output resistance is very large.