Get the boat up to wide open throttle with the trim out as far as it can go without cavitating. Bring it out one click at a time, until it cavitates, then click down until it stops. Then look at your Max RPM.
Get the length, width, gear ratio, hp, etc. Go through the calculator to see what it comes up with.
http://www.mercurymarine.com/propellers/prop-selector/