AI Minecraft Challenge: Measuring Artificial Intelligence Through Gameplay

Key Takeaways

• Researchers have developed a new way of testing AI using Minecraft.
• Current AI benchmarks are limited when it comes to testing problem-solving ability.
• However, the creators of “MinePlanner argue that future models will need to deal with ‘messy’ problems.

School teachers know play is often the best way to get students to think for themselves. From kindergarten onwards, educators use games to assess original thinking and problem-solving skills. Now, AI developers want to apply a similar strategy.

Researchers at the University of the Witwatersrand in South Africa recently proposed a new benchmark test that uses Minecraft to assess AI performance. Named MinePlanner, the currently unpublished benchmark anticipates the next stage of AI development, in which models will be expected to do more than understand a question and provide a reasonable answer based on their training data. 

Minecraft as an Educational Tool

Speaking to New Scientist, lead researcher on the MinePlanner project Steven James observed that future AI models will need to tackle messy problems and he hopes to test how well they deal with unfamiliar scenarios by getting them to play Minecraft.

MinePlanner consists of 15 construction problems, each with an easy, medium, and hard setting, for a total of 45 tasks. To complete each task, the AI may need to take multiple steps. For example, to place a block at a certain height, it may first need to build stairs.

For the same reason that Minecraft has become a popular educational tool  for teaching children 3-dimensional problem-solving skills, James thinks it is an ideal test environment for AI models that will be deployed in unpredictable real-world scenarios.

Testing For General Intelligence

To oversimplify a complex process, AI development involves training models on massive data sets and then testing their ability to answer questions and solve problems. To improve the scores, developers then adjust their model weights, following the logic that a higher score on key benchmarks will lead to better performance in real-world applications like chatbots.

The problem with the standard approach, however, is that it doesn’t test AI models’ ability to deal with new information, only how well they can recall information from their training data. 

The Need For Better Benchmarks

Current AI benchmarks are the equivalent of making students sit a test consisting only of questions from past papers. Someone with a perfect photographic memory could answer every question correctly but still not understand why an answer is correct or how to replicate the result.

While AI has come leaps and bounds in a short space of time, contemporary models still perform badly in areas where they can’t simply regurgitate training data. When tests do assess AI’s capacity for original thinking, the results speak volumes.

The authors of the industry standard benchmark Massive Multitask Language Understanding (MMLU ) test found that AI models fared the worst on tasks related to calculation-heavy subjects such as physics and mathematics and subjects related to human values like law and morality.  For example, when OpenAI’s GPT-3 was prompted with multiple choice questions in a few-shot MMLU test, it scored around 30% in elementary mathematics questions, barely above the 25% baseline score that would result from random guesses.

Of course, AI has evolved since then. But as models improve, new testing methodologies could be required.

If the idea of assessing AI performance with video games seems unusually whimsical for the technology, that’s because whimsy belongs to a higher order or intelligence than what has been demonstrated by existing models.  Consider, for example, that while play is observed across the animal kingdom, only mammals and some birds engage in more complex games or puzzles, and the tasks proposed by James et al are the kind that would even challenge the creativity of a young child. 

About the Author

James Morales

Although his background is in crypto and FinTech news, these days, James likes to roam across CCN’s editorial breadth, focusing mostly on digital technology. Having always been fascinated by the latest innovations, he uses his platform as a journalist to explore how new technologies work, why they matter and how they might shape our future.

See more

[email protected] LinkedIn Twitter