Duddits,
Smooth run from the motor tab followed by jittery run when under the TX control means your control loops are reacting (overreacting) to some mechanical noise in the environment.
Usual suspects for this kind of thing are . . . poorly mounted control board (room to wobble or shake freely, or not solidly mounted), poor vibration isolation (some frames don't need it, some do), Overtuned PID's, poorly balanced motors and poorly balanced props.
To know which it is, we'll need more info. See what you can find, and if the issue doesn't pop out at you, post some pics of your build and we can dig deeper.