How do we choose among strategic experts?
Expertsourcing is a key ingredient for strategivc decision making in applications such as technology forecasting, financial markets, healthcare services etc. To maximize the efficacy of strategic planning, it is often desirable to seek advice from multiple experts and combine their inputs intelligently. We consider an expertsourcing problem where for each task, an optimal subset of experts needs to be selected so that the aggregated opinion guarantees a target level of accuracy. The problem poses two important challenges: (1) aggregated outcome depends on unknown heterogeneous qualities and (2) experts could be strategic about the cost. We develop what we call the “assured accuracy bandit framework” for the proposed problem and derive an adaptive exploration separated algorithm “Constrained Confidence Bound.” We provide a formal lower bound on the number of times any algorithm must select a sub-optimal set and we discover remarkably that the lower bound matches our upper bound up to a constant factor.
Reference:
Shweta Jain, Sujit Gujar, Satyanath Bhat, Onno Zoeter. A quality assuring, cost optimal multi-armed bandit mechanism for expertsourcing. Artificial Intelligence. 254 (2018): 44-63.