If they are the same 9x5 prop that I use on my plane you would have had to drill/reem out the hole to fit it on the prop adapter. If that is the case, they definitely could be that out of balance. The flight control board is going crazy because they don't handle that amount of vibration well. I would try 9x4.7 props that come with a little ring to fit the shaft size that you are using.
To balance, you can use an app on a phone to test each motor and ad tape to dynamically balance. I would start with the props alone first as they can be way off. I made a quick prop balancer out of old hard drive magnets and a cut down nail until my real one came. Not the most accurate but it got me close enough to fly with.
The tricopter doesn't care which way the motors are spinning, the tail rotor will correct for the drift. As long as the thrust is downward, it will fly.