OGnapkin
You haven't actually defined what is happening. Is the CofG is the right place?
Small twists and bends in the wing or fuselage can cause flight characteristics that can be speed dependent.
Trim it to fly satisfactorily at one speed but any faster or slower and it may want to turn, bank, climb or dive in any combination! Furthermore it is likely to get even worse worse at or near the stall.
A good visual inspection checking to see that the wings look
exactly the same left and right, and from all angles, is a good starting point.
With a bit of practise you can do quite a bit of aerodynamic analysis with the good old Mk1 eyeball.