The oscillations are from coupling of a few of the axes. For example, giving a pitch output with the cyclos implies the rear cyclos spin up, front spins down. Because of the axis of rotation of the cyclos, this also induces a rolling moment from the single front cyclo. So then you get differential left/right cyclo output as a response to counter the roll disturbance. Rinse, repeat, and you end up in this cyclic oscillation. This can also result simply from your thrust vectors not being perfectly vertical. For example in roll, if your thrust vectors are actually slightly backward, a differential roll output will cause an unwanted yaw. The fix to all of this is less agressive P gain, and more D gain to dampen the self-oscillations out.
As for the pitch angle... I think your rear 2 cyclos are creating a pitch up torque. Then the front cyclo is lifting the nose up as well. This will lead to a natural tendency to pitch up that the controller cannot compensate for fully. Basically, the torque from the rear cyclos is already doing ~half of the work lifting the nose as you'd expect the single front cyclo to be doing. So a few things to try...one or a combination of these might fix the problem:
- Shift your battery toward the nose to weigh down the nose and put more load on the front cyclo
- Multiple thro_des by ~0.5 in the control mixer so that it does not spool up as much as the rear 2 cyclos
- Double the i_limit parameter in the code. This will allow the integrator term in the pitch controller to spool up higher and maybe compensate for the extreme nose up tendency from the rear cyclo torque. Beware that the hover trim throttle value for that nose rotor may very well be 0.. or negative, meaning that to counter the rear cyclos' torque, the nose rotor actually has to thrust up, making the rear cyclos have to do all of the heavy lifting and then some more..
- Reverse the direction of the rear cyclos so that they are torquing the nose down, not up. This will require more thrust from the nose rotor to prevent a nose-down tendency, which could be compensated by placing the nose cyclo farther forward for more leverage. This will be more similar to my dual cyclo configuration