Hmmmm. I have been playing with quads on INAV the last week-ish with 4 different boards of two different makes. ( 2 ea of dRonin Seppuku and SpeedyBee F405-wing mini) I do notice that it does take a few seconds to settle down after moving after first flashed, But after calibration, Things seem OK. Which version of INAV, what OS are you running the configurator on, what FC, and does your GPS have a compass?
If GPS has a compass, it could be magnetic/electrical interference with the magnetometer.
Which tab in the configurator? Initial Setup Screen?