What is the fastest switch time? What limits it?

The fastest digital speeds (for a clock in a synchronous circuit) are something like THz ($10^{12}$ Hz), which corresponds to $\sim$picosecond transitions. (More normal digital electronics clock speeds are GHz, so have $\sim$nanosecond transitions.)

The switch time for a particular gate is limited by stray capacitance (and possibly other non-ideal performance of the transistor components). For a synchronous circuit with a continuously-maintained rate of transitions, I think the main limitation is heat dissipation. The higher the clock speed and rate of transitions in the circuit, the more power is dissipated and the hotter the circuit runs (and solid state components tend to be quite sensitive to temperature).