Most of the ft models from what ive seen lack proper airofoil and instead rely on power from the motors to drag them through the air, if you can't get enough pull on these models you'll struggle, many things can affect that pull.
Are you flying from a cliff edge or hillside? If you are instead go to the middle of a field in no wind and see if it makes a difference.
Taking smaller models off a hill etc will be a handful because of the turbulence over powering the motor.
Generally the smaller the model The quicker it is, the harder it is to fly and the easier it is for the wind to spoil your day.
Test glides before each flying day are imperative.
When the zagi first came out I had issues trying to get its balance right but once it was it was so easy to fly and I'm thinking in the back of my mind the reason I'm doing an arrow at the moment is to relive some of that fun.
The previous advice of lowering your throws is going to be your friend here, start small throws and increase them if you can't keep it in the air. Do them mechanically first (hole closest to centre in servo horn and furthest away from centre on surface horn. Then you can play on the transmitter. If it won't change enough move the surface horn end towards the centre 1 hole, reset your transmitter and try again.
As an avid wing flyer, I agree with @Kananga, wings, when property setup, are no more difficult to fly than any other plane.
They are very sensitive to your setup, keep your CG forward (@ 25% of wing area) and keep your throws low. About half of throws used on a plane with a tail. Like all planes, wings will fly better when you get the plane trimmed. When trimmed, slowly turn the throws up to suit your flying stile, mild or wild.
It's really hard to judge without looking at some video. BUT something I would suggest is dialing down your rates even more. Maybe a 3 position switch for 3 different rates. Perhaps make your low rates even lower. Could also bump up the expo a little bit. Just something super easy while you work out the kinks. Do some glide tests too and see if it glides ok, battery attached, but no power.
Minis can be a whole different experience...I don't have a ton of experience with them either. But one common thing I've heard is weight. Keep your glue usage with them to as little as possible. Use real light servos, lightweight motors, light batteries, etc.
I agree. Video would have helped people help me easier with some early issues if I'd had it sooner, but I'm learning to record and upload footage for the first time as well as flying planes, so by the time I had the means and technique to record videos reliably and upload them, which was quite recently, I was already proficient with a trainer type plane(Hobby Zone Champ and had two similar but slightly different prototypes of my own designs flying reliably. I'm still learning, and when I start to branch out in completely different directions with my builds and designs, I'm sure I'll have some times where something doesn't work the way I want it too right on the first try, and then I'll be glad I have video footage to share.
You seem to not do well with anything that has elevons. I don't do well with elevons yet either, but part of that is the precision they require in building. I'll be working more on that soon. Other than that, planes that need a lot of elevator authority to climb and very little aileron input to roll, usually have sizeable elevons which further this bias a bit. For these reasons, I have some low aspect delta type planes that I designed to have a RET control scheme, to lead me in sort of gently. They are still quite sensitive and would likely still make your sketchy list. Simulator flight helped me out a lot. Perhaps get a free sim such as Multiflight and fly a bunch of stuff you have trouble with, like flying wings, elevon jets, low-wing warbirds, etc. I have heard the FT flyer is very docile for a plane with elevons. It is very responsive, but can fly slow. I have not built or flown one myself, but I'm sure those who have would agree.