How do you know which radius to use?

Well, this depends on the problem. But in many cases, consider where the force is acting: the radius from that point to the pivot (typically, where the axis of rotation is) is the $\vec{r}$ in $\vec{\tau} = \vec{r} \times \vec{F}$, and so this is the relevant radius. In today's yo-yo problem, the tension acts at $R/3$ away from the center of the yo-yo, so that's the radius to use.