Main concern with batteries (besides polarity) is ensuring that + and - don't touch each other, creating a short circuit. The thing with bullet connectors is that it's not easy to completely cover them such that no metal is exposed. You can put heat shrink around all the way to the end of the female connector, but you can't quite extend it beyond the end without the heat shrink blocking the hole.
EDIT: A bag of 10 pairs of XT30U or XT60U (depending on battery size) costs less than $10. Seems to be a relatively small price to pay for safety when it comes to batteries. Imagine the cost of just a small mistake with the polarity or a short that fries your battery, or worse starts a fire.