Qualitatively, what causes the difference in band gap in conductors vs insulators vs semiconductors?

Quantitively the band gap width is determined from quantum mechanics, i.e., solving for the wavefunctions describing the system of multiple-electron atoms in the crystal. Qualitatively, you can think about the atoms in the crystal as having electrons residing in ``shells'' or ``orbitals'' (these come from solving the Schrödinger equation - I hope this picture is familiar to you from previous courses). Each shell with principal quantum number $n$ can hold a maximum of $2n^2$ electrons in subshells with different $l$ values ($l=0,...n-1$). According to the Pauli principle, you can have up to two electrons, with opposite spins, in each state. When a shell is fully occupied by electrons, it's ``closed''; further electrons at higher energies added to the atom are called ``valence'' electrons. You can think of a solid as a bunch of positive centers (nuclei) with closed shells of electrons, plus valence electrons. According to quantum mechanics, states that the valence electrons can occupy are determined from the solution to the Schrödinger equation, for which the Hamiltonian includes potentials due to interactions between atoms. These interactions will tend to modify the allowed energy levels such that valence electrons can occupy a band of energies, the so-called ``valence band''. There will also exist allowed, unoccupied-in-the-ground-state energy levels. Electrons in excited states occupying these could be delocalized i.e., could be found quite far from their ``home'' nuclei. Such states make up the ``conduction band''. Where the valence and conduction bands lie in energy depends on the specific nature of the atoms involved. The difference in energy between the conduction and valence bands is known as the ``band gap''.

Depending on the specific structure of the atoms in the crystal (note you can have compounds as well as elements), the conduction band can overlap with the valence band. In this case, an electron can easily be promoted from the valence to the conduction band with only a bit of extra energy (even just thermal energy) and go wandering around, easily influenced by external electric fields. Such materials are good conductors. If there's a big difference between the valence and conduction bands, then it takes a lot of energy to kick an electron up to the conduction band, and that material will be an insulator. A semiconductor corresponds to the case where there's a non-zero band gap, but a small one.

Why do different materials have different band gaps? It depends on their specific atomic structure. For example, if a material has a partially-filled shell, the next-state-up might be not very far away (in the same shell), and so the conduction band will be very close to the valence band- perhaps even so close that inter-atomic interactions will create energy levels easily accessible in the ground state- and you'll get a conductor. As another example, if there are no valence electrons (such as for noble elements like He, Ne, Ar, etc.), and just closed shells, the next highest energy state an electron can occupy could be quite far up (all the way to the next shell), so you will have large band gap and insulating properties. And another example: Si and Ge have subshells filled, but not full shells; the conduction band is nearby, but not right on top of the valence band- and you have a semiconductor. (BTW, note that you can have compounds as well as elements, with complicated electronic band structure.)