The input and output impedances matter for the properties of the two-terminal network when connected to other circuits at the input and the output (which is usually what you are doing!). Whether you want small or large, or some particular value, depends entirely on what you are doing with the amplifier. For example, if you connect a load to the output, you want to make sure the amplifier can provide enough current-- that will determine requirements on output impedance. Or you might not want to draw too much current from a supply, so you'd want a high-enough input impedance. Sometimes you want to match input and output impedance to maximize power transfer. Or you might just want to match the impedance of the circuit, or cable, you are connecting up to, in order to avoid reflections.