Why does the $V_{\rm out}$ of a high-pass filter have a discontinuous jump given a square wave input?

For a high-pass filter responding in the low frequency regime, the transfer function is approximately $\vert H\vert \sim j \omega RC$, which is equivalent to a (scaled) derivative function. Looking at it mathematically, the output signal will look like the derivative (the slope) of the input square wave. So for the rising edge, the output will be a sharp positive peak. The output will be zero where the square wave is flat, and it will be a sharp negative peak for the square wave's falling edge.

Looking at it physically: in the low frequency regime, the capacitor charges all the way up early in the cycle, and then passes no more current. When the voltage swings down, it discharges fully and charges up in the opposite direction, then stops passing current again. So you'll get only spikes of current (and corresponding spikes of voltage across the resistor) at the square wave rising and falling edges.