In the quiz, was the scale accelerating or was the system? Why was there an upward force, besides tension that didn't matter? Or was saying ``There is an upward force'' a bad hint?

In part c, the whole system is moving down together with acceleration $a$. The upward force on the system (acting on the scale) is given in the problem. ``There is an upward force'' is not a hint-- it's telling you that there's an upward force you need to include in your analysis of the problem.

This is another problem that can be approached with the how-to. First consider the mass: what are the forces on it? Answer: a normal force $N$ upwards from the scale, and gravity $mg$. Now consider the scale: what are the forces on it? Answer: a normal force $N$ downwards from the mass, gravity $Mg$, and the given force $F$ upwards.

Then parts b and c can be done by applying Newton's second law to each system. In part b, acceleration is 0. In part c, acceleration is $a$ downwards. By writing down the second law equations, you should get enough equations to solve for the unknown $F$.