Serpent.AI Logo

A new version of Serpent.AI has been released on PyPI and is available for general public use. This is a major release. It includes new features, improvements, tweaks, fixes, internal API changes and a new Jupyter Notebook.

Existing users may upgrade using pip install --upgrade SerpentAI followed by serpent update. If this is your first foray into a game agent framework like Serpent.AI, you are in luck as it has never been easier to get started! The official wiki contains all the information you need to install and write your first agent.

Never heard of Serpent.AI before? No problem, here's a tl;dr: Serpent.AI allows you to turn video games you own (e.g. Steam games, DRM-free executables, browser games) into fully-equipped sandboxes for machine learning and artificial intelligence experiments. Neat, uh?

Release Highlights

Feature: Gameplay Recording

The blockbuster feature! This release of Serpent.AI includes cross-platform gameplay recording of any video game. One simple command is all it takes:

serpent record Cuphead SerpentCupheadGameAgent

That's it! Run it and start playing the game. Every keyboard and mouse action you perform in-game will be captured and associated with a sequence of frames that led to and followed it. When you are done with your session, focus out of the game and the collected data will be neatly packed in a single HDF5 file. The dataset is optimized in such a way that 10 minutes of gameplay will on average be around 100MB; pretty awesome considering the volume of information contained within!

Working with reinforcement learning? You can optionally register a reward function to put the frame sequence through and the scores will also be appended to the dataset!

Want to see it in action? Here is a video demonstrating how you can go from zero to recorded gameplay dataset with Serpent.AI:

The main goal of this feature is to trivialize the creation of high quality player-driven input datasets. The contained information should be enough to power a wide gamut of machine learning applications; from static supervised learning approaches to state-of-the-art reinforcement learning techniques.

Curious as to how the cross-platform input capture works? A separate library exclusively dedicated to this task was developed in parallel with this feature. You can browse the source code on GitHub: sneakysnek

Simpler Installation & Updates

Perhaps the biggest obstacle to adoption that was observed during the 5 month beta period was the installation process. Some of the dependencies are difficult to get working and don't necessarily power features that see a lot of use. As the number of features grew and some metrics were gathered in terms of what people actually use, it started making sense to modularize the installation of the framework.

As of 2018.1.0, Serpent.AI is much easier to install, only requiring a Redis server, a pip install and a serpent setup. The modules that now require additional setup are as follows:

  • OCR: Powers the OCR functionality. Requires a working Tesseract install.
  • GUI: Powers the visual debugger. Requires a working Kivy install. Painful.
  • ML: Powers the packaged, off-the-shelf machine learning features. Requires a working Tensorflow install and Keras.

Optional modules can be installed by following the wiki instructions and running:

serpent setup <module_name>

As far as updates are concerned, anyone using 2018.1.0 and up will be able to update to the latest Serpent.AI version with a simple serpent update. This command handles more than a mere pip install: It will also check if new packages are now required on your specific platform and check your configuration file for missing keys and add default entries as needed.

Bye-Bye Beta

Serpent.AI entered its beta period in September 2017 and is now exiting it about 5 months later. While some big features might still show up in the future, the core is now considered solid enough to drive most experiments to success. The focus will shift from the development of major features to improving the documentation and training material and building many experiments to showcase the capabilities of the framework. A big heartfelt thank you to everyone that gave feedback during beta. You helped shape Serpent.AI into what it is today!

Other Goodies

Region-Based Context Frame Captures

The label of registered screen regions in your game plugins can now be passed to context frame capture operations. This is especially useful for users wanting to create a context classifier for web browser games; where the game area is generally a smaller subset of the full window. You can scope your context frame captures to a screen region like this:

serpent capture context <game> 1 <context_label> <screen_region_label>

GameFrame Object Timestamps

GameFrame objects now contain a timestamp property that represents the exact time.time() where the image data was captured by the FrameGrabber process. Beyond helping with specific timing-based operations, another direct benefit of this change is that all capture frames are now ordered in the directory they are saved in by timestamp instead of a random UUID4.


Captured game frames ordered by timestamp

There is more! You can consult the full CHANGELOG here

That's it! You can learn more by visiting the GitHub repository and/or reading the Wiki.

Enjoyed the post?

Awesome! For more content, feel free to: