how can a “prop in slot” design be optimized?
An interesting question but by far the greatest "optimisation" is to not use a prop in slot at all! It disrupts the airflow to and from the prop thus reducing its efficiency.
A tractor prop is best then a pusher but it should be well clear any aerodynamic surface and finally a prop in slot. Of course even worse still is a ducted fan.
If you
have to use a prop in slot then to improve efficiency you have to consider how disrupted will be the airflow leading to the prop and how obstructed will it be away from it. Remember air flowing over even a flat smooth surface is slowed down by the effect of the air's viscosity.
In general a prop is most effected by almost any form of turbulence in the airstream immediately in front of the prop leading edge. As a pro blade passes through any turbulence it effects the airflow over the whole width of the blade. It follows that the prop in a conventional slot will be disturbed over the whole length of each prop blade
twice for every revolution.
There can be good reason to use a prop in slot but efficiency is not one of them.
I have many electric powered planes. Tractor, pusher and EDF but no prop in slot.