Propellers make thrust by grabbing a mass of air, accelerating it, and pushing it to towards the rear (of the aircraft).
The actual thrust is the product of these two things:
Thrust (in newtons) = (mass of air per second in kilograms) x (change in air velocity going through the prop in metres/second).
Note that this equation only works if you use sane units, i.e. SI units. Mass in kilograms, velocity change in metres/second. Thrust in newtons.
Anyhow: since thrust is the product of two things, you can make a given amount of thrust either by accelerating a large mass of air per second quite gently, or by violently accelerating a much smaller amount of air.
The first of those corresponds to turning a large propeller at relatively low rpm. The second corresponds to screaming a small propeller at high rpm.
Even if you carefully pick the propeller pitch to produce the same pitch speed from both propellers, guess which of those two processes takes more power from the motor?
Not sure? Ask yourself why helicopters always have a huge rotor. A big rotor is hard to engineer, it takes up lots of room, it requires enormous gearbox reduction ratios, it's heavy, it's bulky. There must be a good reason why helicopter designers go to all the trouble and expense and inconvenience. They would not be using huge rotors if small ones worked as well, right?
Yup, you got it. It takes a lot more power to get "x" watts of thrust at "y" pitch speed from a small propeller. Small props are inefficient. Big props are more efficient. When picking a propeller for your model, the first goal should always be to make the propeller as large as possible, provided you can find a motor with low enough Kv to swing it properly, and provided you can get a sufficiently high pitch speed from that big, slow-turning propeller. And provided the torque from the big propeller doesn't make your model hard to fly when you open or close the throttle.
In practice, several powerful WWII fighter aircraft ended up with propeller diameters that were nearly 30% of the wingspan. If the prop was bigger than that, torque reaction and landing gear length became problematic.
I found roughly the same practical limit in my powerful aerobatic electric RC planes. Prop diameter is best kept under 30% of the wingspan.
If the wingspan is huge (as on a high-aspect-ratio model), and the model a gentle flyer (an electric-assist glider, say), then a much smaller propeller will work just fine.
But in general, use the biggest propeller possible.
What about the other limit, the one you asked about? (i.e. using a very small prop?) Well, it amounts to throwing away efficiency. The smaller the prop, the more efficiency you're throwing away. And you are likely to also end up with far too much pitch speed to suit the model. This is like strapping a motorcycle drivetrain to a farm tractor. Not a great match.
The guys who build and fly rubber-powered models know all about this. They don't have excess power to waste. FAI F1D rubber-powered indoor models have enormous propellers that turn absurdly slowly, because that is the most efficient way to convert the limited amount of power from the rubber, into forward propulsion of the model. Take a look:
By the way, when you throw away efficiency by using a too-small propeller, you need more watts to make the same thrust. This means you need a bigger and heavier motor, and a bigger and heavier battery. So you also end up with a model that's heavier than it needs to be.
IMO, this is the single worst thing about using direct-drive outrunner motors, particularly in small sizes, on slow-flying model aircraft. Most of the time, the motor has too high a Kv to spin a properly sized propeller. You end up forced to use a too-small propeller turning at too-high rpm, producing poor efficiency and too much pitch speed to suit the model.
Two decades ago, the solution was to use a gearbox between motor and propeller. Gearboxes in electric RC planes have virtually gone extinct, but for slow-flying models, they were actually a much better solution than the little high-rpm outrunner motors we have today.
-Gnobuddy