The best way to know whether a prop is too big is to buy an in-line amp-meter and test it out. You can get a basic amp-meter that goes up to 30 amps from Hobbyking for about $5. Higher-end models are available too.
But what if you don't want to do trial-and-error with prop sizes? If you can find a spec sheet for the motor, that's the best way to determine the biggest prop your motor can swing. The spec sheet should say the battery size 3S, 4S; prop size; and current drawn. The manufacturer won't bother publishing specs for props that over-draw the motor, so as long as your ESC is big enough (including at least a 20% margin, and more if you aren't cooling it well) you can run any prop on the spec sheet, or smaller. You might also be able to run bigger ones, but you'll want to test with an amp-meter to be sure. For example, I have a 1400 kV motor that is only specced to run up to 8x4.5, but I am running an 8x6 on it just because I had a bag of them laying around. The amp meter shows that the 8x6 pulls 20.5 amps at WOT on a motor that is rated for 21 amps, so it's right on the line, but as long as I don't run the motor flat out for too long, I feel okay about it. But if I didn't have an amp meter, I would never have gone above the spec sheet's recommendations.
If you can't find a spec sheet, another way to go is to use eCalc, WebOCalc, or something similar, to estimate current draw. These calculators aren't perfect, but they get you in the ballpark.
Because motors can have different internal construction and quality levels, it's not necessarily safe to assume that two ostensibly similar motors can handle similar prop sizes. If you know about the details of motor construction, you can do a decent job of comparing two motors to see whether one's specs are truly similar to the other's. But just because two motors are roughly the same size and kv (for example) is not enough to say for sure that they can handle the same prop sizes.
