Neckerworld is a computer vision game designed to teach students about human and computer vision systems, object detection and identification, visual field construction, autonomous movement and strategy.
All players and resources in the game are cubes. The cube players are guided solely through autonomous computer vision programs. No human manual control or input is permitted during gameplay.
To successfully play the game requires a competent program to do object detection and identification, playing field knowledge representation and strategic decision making.
The Necker Cube Illusion (first published as a rhomboid in 1832 by Swiss crystallographer Louis Albert Necker) is a well known example of a perceptual illusion where the orientation of the cube is unstable and easily flips between two states. Serious study and research into why we perceive multiple states leads to a set of questions about the very nature of our vision.
Further, cubes are very simple objects which can be used to illustrate many other aspects of human and computer vision, such as the effects of contours, shading, texture, occlusion, size-constancy and more.
Components within the Neckerworld programs allow an instructor to easily construct many of these examples and allows students to learn a lot about visual perception.
Visual Systems are very different. They are uniquely engineered to serve the needs of the organism. They have evolved over 100's of millions of years. The human visual system serves humans well, but not zebras, owls, rats, bees, or spiders.
Three guiding principles of the human visual system:
Consquently, game points are scored through three activities:
For a quick demo, check out this video: https://youtu.be/79xiX2FlELk
There are five types of players in the game.
The male, female and enby players are controlled remotely and autonomously via programs created by the game participants. The predators and resources are provided by and controlled by the gaming field server.
Game points are earned in three different ways and determine which player wins the game.
Energy points are the vital resource that keeps a cube alive during the game. Each second of play uses up energy points as does moving around the playing field. Energy points come from feeding on the resource cubes.
A player (Male, Female or Enby) earns game points:
Game play is completely autonomous and a winning program will have fast object detection, an internal representation of the field, a movement strategy that avoids field obstacles and excellent decision making on what targets will yield the most points. A prototype player program is provided (see client/nwplay.py) that is a starter program for competition. The player programs are remote from the game server and connected by a messaging protocol. Consequently, autonomous player programs can be developed on any compute platform desired using favorite systems and tools.
The Neckerworld consists of a playing field server and one or more remote client players. The server places predators and food resources on the field and controls their activity and movement. The client player programs exchange messages (JSON files) with the server:
The client players and programs can be on any computer running any operating system with any programming language.
A crucial element of this game is the proper identification of cube objects and positions. Each client player needs to scan the environment and identify friends, foes, mates and energy (food) resources. This requires that each received cube's-eye image is run through an object detection and identification program to determine what's in the visual field.
Current solutions for this type of problem use a neural network trained on the visual objects of interest. This repository provides two pre-trained neural network models that may be used for game play. The models run out-of-the-box with a Tensorflow-2 programming environment.
Training a neural network on cube objects is a non-trivial undertaking and requires substantial computer resources. A YOLOv5 model using PyTorch for training is documented in the "train" directory. This runs well on an NVidia GPU system using CUDA. There are multiple utilities for setting up the required image and label files with annotations and bounding boxes. Image samples and a label dataset suitable for training is contained within the training directory and provides an accelerated start to developing a preferred neural network model.
The server has only been tested on Ubuntu 20.04.6 LTS and Ubuntu 22.04.3 LTS. The desktop edition is required as both the server code and player will open a graphical window to function.
The following packages are needed to compile and run these programs:
Get the source code at github.com/hankster/Neckerworld
All programs and materials in this repository are under the MIT License. The MIT license is a simple open source license agreement that grants users the right to reuse code for any purpose, including inside proprietary software, which sets it apart from a Copyleft license.
See: https://mit-license.org and https://en.wikipedia.org/wiki/MIT_License for more information.
Discussion and citations about Neckerworld:
This game is still in an early stage of development and many aspects of the server and client need to be improved. The author is very interested in feedback and comments related to interest and applicability of this work. There is, to the author's knowledge, only one similar effort to build a game where programmed automata are the only players, and that is a racing car game. Future efforts to continue and enhance this game concept will, in part, depend on the feedback received.