Auto Maple

Auto Maple is an intelligent Python program that plays MapleStory, a 2D side-scrolling MMORPG, using simulated key presses, TensorFlow machine learning, OpenCV template matching, and other computer vision techniques.

Minimap

Auto Maple uses OpenCV template matching to determine the bounds of the minimap as well as the various elements within it, allowing it to accurately track the player's in-game position. If record_layout is set to True, Auto Maple will record the player's previous positions in a quadtree-based Layout object, which is periodically saved to a file in the "layouts" directory. Every time a new routine is loaded, its corresponding layout file, if it exists, will also be loaded. This Layout object uses the A* search algorithm on its stored points to calculate the shortest path from the player to any target location, which can dramatically improve the accuracy and speed at which routines are executed.

Click here to view the above routine.

Command Books

_{The above video shows Auto Maple consistently performing a mechanically advanced ability combination.}

Designed with modularity in mind, Auto Maple can operate any character in the game as long as it is provided with a list of in-game actions, or a "command book". A command book is a Python file that contains multiple classes, one for each in-game ability, that tells the program what keys it should press and when to press them. Once a command book is imported, its classes are automatically compiled into a dictionary that Auto Maple can then use to interpret commands within routines. Commands have access to all of Auto Maple's global variables, which can allow them to actively change their behavior based on the player's position and the state of the game.

Routines

Click here to view the entire routine.

A routine is a user-created CSV file that tells Auto Maple where to move and what commands to use at each location. A custom compiler within Auto Maple parses through the selected routine and converts it into a list of Component objects that can then be executed by the program. An error message is printed for every line that contains invalid parameters, and those lines are ignored during the conversion. Below is a summary of the most commonly used routine components: Point stores the commands directly below it and will execute them in that order once the character is within move_tolerance of the specified location. There are also a couple optional keyword arguments: adjust fine-tunes the character's position to be within adjust_tolerance of the target location before executing any commands. frequency tells the Point how often to execute. If set to N, this Point will execute once every N iterations. skip tells the Point whether to run on the first iteration or not. If set to True and frequency is N, this Point will execute on the N-1th iteration. Label acts as a reference point that can help organize the routine into sections as well as create loops. Jump jumps to the given label from anywhere in the routine. Setting updates the specified setting to the given value. It can be placed anywhere in the routine, so different parts of the same routine can have different settings. All editable settings can be found at the bottom of settings.py.

Runes

Auto Maple has the ability to automatically solve "runes", or in-game arrow key puzzles. It first uses OpenCV's color filtration and Canny edge detection algorithms to isolate the arrow keys and reduce as much background noise as possible. Then, it runs multiple inferences on the preprocessed frames using a custom-trained TensorFlow model until two inferences agree. Because of this preprocessing, Auto Maple is extremely accurate at solving runes in all kinds of (often colorful and chaotic) environments.

Video Demonstration

Click below to see Auto Maple in action:

Setup

1. Download and install Python3.
2. Download and install the latest version of CUDA Toolkit.
3. Download and unzip the latest Auto Maple release.
4. Download the TensorFlow model and unzip the "models" folder into Auto Maple's "assets" directory.
5. Inside Auto Maple's main directory, open a command prompt and run:

   python -m pip install -r requirements.txt

6. Lastly, create a desktop shortcut by running python setup.py. This shortcut uses absolute paths, so feel free to move it wherever you want. However, if you move Auto Maple's main directory, you will need to run python setup.py again to generate a new shortcut.

Reflection

From working on this project, I truly learned a lot. Auto Maple not only allowed me to apply the knowledge I learned in lectures to a challenging problem, but it also introduced me to many exciting and complex concepts such as machine learning and Canny edge detection. However, perhaps more importantly, working on Auto Maple has given me a deeper appreciation of human problem solving and a clearer understanding of just how hard it is for a computer to emulate that.

Early in this project, when I was still trying to get the character to move to locations on the minimap, I resorted to letting the program blindly move horizontally and vertically until it reached its target. This was very inefficient and often resulted in the character getting stuck. Later, I realized that memory was the main aspect Auto Maple was missing: human gamers remember where they've been, which places are safe to walk on, and which places aren't. This gave me the idea to create a Layout class to help the program chart a path in advance based on where it has already been, much like a human.

In the end, Auto Maple is still far from perfect. It can't predict and prevent missteps like we can. It can't reproduce the fluid actions of a practiced gamer. But acknowledging these shortcomings inspires me to continue learning and experimenting with new ways to make programs faster and smarter.