Researchers at the Allen Institute for Artificial Intelligence, the University of Washington, and the Hebrew University of Jerusalem have developed GENIE. It is a leaderboard for human in loop evaluation of text generation. The leaderboard provides model predictions to crowdsourcing platforms like Amazon Mechanical Turk. At the platform, human annotators will analyze them according to the predefined, dataset-based rules for conciseness, fluency, correctness, etc.
As per reports, to avoid deterring researchers from submitting citing limited resources, GENIE plans to keep the submission price around $100. They plan to explore more payment options in the future.