It is all in your gain adjustment. Some test you can do is to run in GPS then go to Atti mode and see the difference. In Atti it will drift with the wind, go back to GPS and it will try to stay in one spot. Do you an 8 channel radio? You maybe able to set it to adjust the gains in flight. There are so many variables as each fight vehicle is so different, so the gains are dependent on the unit it is in. Another test is to get it set with GPS and do fast forward, fast reverse passes then stop. Does it try to return to the same height it was before it lost height in fast flight? If so, it is working. Check the Manuel on how to set the gains. What has worked for others with the same QAV 500 frame as mine, is to much for mine. Motors, props, esc's all seem to make a difference. Props seem to make a huge difference, at least that is my understanding. The more flexible the prop, the less precise control. Experience has shown me that the location of the GPS antenna is very important. The closer to the center the better it flies.
Other thought, did you do the Digital Compass Calibration per instruction Manuel? Might help.
Here is a thread on another forum, you may find useful information you can apply.
http://fpvlab.com/forums/showthread.php?5349-The-new-QAV500-FPV-Quadcopter-frame-by-fpvmanuals
Start close to the end for best information.
