<jupyter_start><jupyter_text>Unity ML Agents - 3DBallIn this example, we are going to use a evolutionary algorithm to train a dense neural network for the Unity's [3DBall environment](https://github.com/Unity-Technologies/ml-agents/blob/main/docs/Learning-Environment-Examples.md). The challenge of this environment is to move the agent in a way that doesn't drop the ball. ![Environment](3DBall-Figure1.png)  Install dependencies and environmentFirst, you need to install the libraries to interact with the Unity environment:<jupyter_code>!pip3 install mlagents==0.27.0
!pip3 install gym_unity<jupyter_output><empty_output><jupyter_text>For the environment, there are two options:- Use the pre-built version (Windows): [3DBall Windows Build](https://drive.google.com/file/d/150UDRhKwpFWaKiehBYE3Qk7xfLE2GGTN/view?usp=sharing), download and extract.- Build manually:  - Install Unity and clone the ML Agents repository: [Installation Instructions](https://github.com/Unity-Technologies/ml-agents/blob/main/docs/Installation.md)  - Open the repository project in Unity ("Project" folder)  - Open the "3DBall" scene, and keep only one copy of the environment (delete "3DBall (1-11)", keeping only the first "3DBall"), as the interface only works with scenes with an agent.  - Build the scene: [Using an Environment Executable](https://github.com/Unity-Technologies/ml-agents/blob/main/docs/Learning-Environment-Executable.md), follow the instructions in "Building the 3DBall environment"Once the environment is built, change the path in the block below to the environment path:<jupyter_code>args = {"env_path": "D:\\Github\\ml-agents\\builds\\3dball_single\\UnityEnvironment.exe"}<jupyter_output><empty_output><jupyter_text>EvaluatorOur agent will be a dense neural network, which will have the weights encoded by the individual. The class that will be our fitness function is defined in the file ["unity3dball_evaluator.py"](unity3dball_evaluator.py).<jupyter_code>import evolvepy as ep
import numpy as np
from matplotlib import pyplot as plt
import unity3dball_evaluator as unity3dball<jupyter_output><empty_output><jupyter_text>We create the evaluator:<jupyter_code>evaluator = ep.evaluator.ProcessEvaluator(unity3dball.Unity3DBallEvaluator, args = args)<jupyter_output><empty_output><jupyter_text>And the descriptor of our individual. Let's use a network with 2 hidden layers, each with 20 units.:<jupyter_code>import nn
descriptor = nn.create_descriptor(input_size=8, output_size=2, units=[20,20])<jupyter_output><empty_output><jupyter_text>Generatorlet's start by defining our population size. As we are going to parallelize the evaluations, it is interesting to use a size multiple of the number of processors, using all the computational power:<jupyter_code>import multiprocessing as mp

population_size = (100//mp.cpu_count())*mp.cpu_count()
population_size<jupyter_output><empty_output><jupyter_text>Our generator will have elitism, keeping the best 5% individuals of the generation, and random predation, exchanging the worst 75% individuals for better individuals (in the other 25%):<jupyter_code>first = ep.generator.Layer()
sort = ep.generator.Sort()

first.next = sort

concat = ep.generator.Concatenate()

predation = ep.generator.RandomPredation(int(0.75*population_size))
combine = ep.generator.CombineLayer(ep.generator.selection.tournament, ep.generator.crossover.one_point)
mutation = ep.generator.mutation.NumericMutationLayer(ep.generator.mutation.sum_mutation, 1.0, 0.9, (-0.5, 0.5))
filter0 = ep.generator.FilterFirsts(int(np.floor(0.95*population_size)))

sort.next = predation
predation.next = combine
combine.next = mutation
mutation.next = filter0
filter0.next = concat

filter1 = ep.generator.FilterFirsts(int(np.ceil(0.05*population_size)))

sort.next = filter1
filter1.next = concat

generator = ep.generator.Generator(first_layer=first, last_layer=concat, descriptor=descriptor)<jupyter_output><empty_output><jupyter_text>EvolverFinally, we define our Evolver. We gonna use dynamic mutation to avoid getting stuck at a local maximum:<jupyter_code>dyn_mut = ep.callbacks.DynamicMutation([mutation.name], refinement_patience=5, exploration_patience=5, refinement_steps=5)

evolver = ep.Evolver(generator, evaluator, population_size, [dyn_mut])<jupyter_output><empty_output><jupyter_text>Evaluation and resultsLet's evolve for 70 generations:<jupyter_code>hist, last_pop = evolver.evolve(70)<jupyter_output><empty_output><jupyter_text>And see the results:<jupyter_code>plt.plot(hist.max(axis=1))
plt.xlabel("Generation")
plt.ylabel("Fitness")
plt.title("Evolution History")
plt.show()<jupyter_output><empty_output><jupyter_text>We can see that the agent manages to improve in the environment, reaching the maximum possible fitness of 100:![](3DBall-Figure2.jpg) Let's see how the best individual fared in the environment:<jupyter_code>best = last_pop[np.argmax(hist[-1])]

test_evaluator = unity3dball.Unity3DBallEvaluator(show=True, args = args)
test_evaluator.evaluate([best])<jupyter_output><empty_output>