forgotpwdagain9

Last resort these days, thankfully; 19th century naturalists would shoot everything that moved...

Do you have an algorithm in place to evaluate "controversial" classifications? e.g. flag this image if the spread of user-submitted results is broader than two standard deviations (or whatever would be the appropriate measure in any given project)

Are conflicts then resolved manually (by you guys) or allowed to gather more user-submitted results or (some third option I haven't thought of)?

Is there a set number of times a given image is presented to users for evaluation, or do you just let them all run, so to speak, for as long as you can, before it's time to gather up and analyze results?

Thanks in advance!