Please tell the model of your charger.
Most multi chargers require that you charge with the same wires as you use to your ESC.
You normally need a lot of "adapter" cables to charge your different batteries.
Some very simple chargers use only the balancing leads to charge - but only small currents as you have.
But you might have a connection break somewhere - it is not uncommon to have a "cold solder" in the charger or the battery cable.
Found your 4-battery charger - YOU NEED adapter cables.
The connector is called mini or micro Deans and you will probably not find them at HobbyKing any longer.
You have alligators and bananas so you will not be without flying ;-)
You can get to the positive pin on your battery easy.
With some care not to shorten anything it is possible to get e steel needle through the black wire and connect the minus alligator being extremely careful not to short the positive (red) alligator to the negative (black) one.