<jupyter_start><jupyter_text>Navigation---In this notebook, you will learn how to use the Unity ML-Agents environment for the first project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893). 1. Start the EnvironmentWe begin by importing some necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/).<jupyter_code>from agents.python.unityagents import UnityEnvironment
import numpy as np<jupyter_output><empty_output><jupyter_text>Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.- **Mac**: `"path/to/Banana.app"`- **Windows** (x86): `"path/to/Banana_Windows_x86/Banana.exe"`- **Windows** (x86_64): `"path/to/Banana_Windows_x86_64/Banana.exe"`- **Linux** (x86): `"path/to/Banana_Linux/Banana.x86"`- **Linux** (x86_64): `"path/to/Banana_Linux/Banana.x86_64"`- **Linux** (x86, headless): `"path/to/Banana_Linux_NoVis/Banana.x86"`- **Linux** (x86_64, headless): `"path/to/Banana_Linux_NoVis/Banana.x86_64"`For instance, if you are using a Mac, then you downloaded `Banana.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:```env = UnityEnvironment(file_name="Banana.app")```<jupyter_code>env = UnityEnvironment(file_name="Banana.app")<jupyter_output>INFO:unityagents:
'Academy' started successfully!
Unity Academy name: Academy
        Number of Brains: 1
        Number of External Brains : 1
        Lesson number : 0
        Reset Parameters :
		
Unity brain name: BananaBrain
        Number of Visual Observations (per agent): 0
        Vector Observation space type: continuous
        Vector Observation space size (per agent): 37
        Number of stacked Vector Observation: 1
        Vector Action space type: discrete
        Vector Action space size (per agent): 4
        Vector Action descriptions: , , ,<jupyter_text>Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python.<jupyter_code># get the default brain
brain_name = env.brain_names[0]
brain = env.brains[brain_name]<jupyter_output><empty_output><jupyter_text>2. Examine the State and Action SpacesThe simulation contains a single agent that navigates a large environment.  At each time step, it has four actions at its disposal:- `0` - walk forward - `1` - walk backward- `2` - turn left- `3` - turn rightThe state space has `37` dimensions and contains the agent's velocity, along with ray-based perception of objects around agent's forward direction.  A reward of `+1` is provided for collecting a yellow banana, and a reward of `-1` is provided for collecting a blue banana. Run the code cell below to print some information about the environment.<jupyter_code># reset the environment
env_info = env.reset(train_mode=True)[brain_name]

# number of agents in the environment
print('Number of agents:', len(env_info.agents))

# number of actions
action_size = brain.vector_action_space_size
print('Number of actions:', action_size)

# examine the state space 
state = env_info.vector_observations[0]
print('States look like:', state)
state_size = len(state)
print('States have length:', state_size)<jupyter_output>Number of agents: 1
Number of actions: 4
States look like: [1.         0.         0.         0.         0.84408134 0.
 0.         1.         0.         0.0748472  0.         1.
 0.         0.         0.25755    1.         0.         0.
 0.         0.74177343 0.         1.         0.         0.
 0.25854847 0.         0.         1.         0.         0.09355672
 0.         1.         0.         0.         0.31969345 0.
 0.        ]
States have length: 37<jupyter_text>3. Take Random Actions in the EnvironmentIn the next code cell, you will learn how to use the Python API to control the agent and receive feedback from the environment.Once this cell is executed, you will watch the agent's performance, if it selects an action (uniformly) at random with each time step.  A window should pop up that allows you to observe the agent, as it moves through the environment.  Of course, as part of the project, you'll have to change the code so that the agent is able to use its experience to gradually choose better actions when interacting with the environment!<jupyter_code>import torch
import torch.nn as nn
import torch.nn.functional as F
import random

class DQN(nn.Module):
    def __init__(self):
        super().__init__()
        layer1 = nn.Linear(37, 1024)
        layer2 = nn.Linear(1024, 256)
        layer3 = nn.Linear(256, 64)
        self.output_layer = nn.Linear(64, 4)
        
        self.hidden_layers = nn.ModuleList([layer1, layer2, layer3])
        # weight initialization
        for l in self.hidden_layers:
            nn.init.kaiming_normal_(l.weight, nonlinearity='relu')
            nn.init.constant_(l.bias, 0)
        nn.init.xavier_normal_(self.output_layer.weight)
        nn.init.constant_(self.output_layer.bias, 0)
        
    def forward(self, state):
        x = state
        for l in self.hidden_layers:
            x = F.relu(l(x))
        out = self.output_layer(x)
        
        return out
from collections import deque, namedtuple

class ReplayBuffer():
    def __init__(self, sample_size, buffer_size=int(1e4)):
        self.buffer = deque(maxlen=buffer_size)
        self.sample_size = sample_size
        self.experience = namedtuple('experience', field_names=('state', 'action', 'reward', 'next_state', 'done'))
        
    def add(self, state, action, reward, next_state, done):
        exp = self.experience(state, action, reward, next_state, done)
        self.buffer.append(exp)
        
    def sample(self):
        experiences = random.sample(self.buffer, self.sample_size)
        
        # divide into batches
        states = torch.from_numpy(np.vstack([e.state for e in experiences])).float().to(device)
        actions = torch.from_numpy(np.vstack([e.action for e in experiences])).long().to(device)
        rewards = torch.from_numpy(np.vstack([e.reward for e in experiences])).float().to(device)
        next_states = torch.from_numpy(np.vstack([e.next_state for e in experiences])).float().to(device)
        dones = torch.from_numpy(np.vstack([e.done for e in experiences]).astype(np.uint8)).float().to(device)
        
        return states, actions, rewards, next_states, dones
    
    def __len__(self):
        return len(self.buffer)
from copy import deepcopy
from torch.torchvision.transforms to 
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

class Agent():
    def __init__(self, alpha=1e-3, gamma=0.99, epsilon=0.1, tau=1e-3, action_size=action_size, batch_size=32):
        self.Q = DQN().to(device)
        self.Q_target = deepcopy(self.Q).to(device)
        self.buffer = ReplayBuffer(sample_size=batch_size)
        self.t = 0
        self.action_size = action_size
        self.alpha = alpha
        self.gamma = gamma
        self.epsilon = epsilon
        self.tau = tau
        self.optimizer = torch.optim.Adam(self.Q.parameters(), lr=self.alpha)
        
    def act(self, state):
        state = torch.from_numpy(state).float().unsqueeze(0).to(device)
        self.Q.eval()
        with torch.no_grad():
            action_values = self.Q(state)
        self.Q.train()
        
        if random.random() < self.epsilon:
            return random.choice(range(self.action_size))
        else:
            return np.argmax(action_values.cpu().numpy())
        
    def step(self, state, action, reward, next_state, done):
        self.buffer.add(state, action, reward, next_state, done)
        
        self.t += 1
        if self.t > 100 + self.buffer.sample_size:
            experiences = self.buffer.sample()
            self._update_net(experiences)
            
    def _update_net(self, experiences):
        states, actions, rewards, next_states, dones = experiences
        
        targets = rewards + (1 - dones) * self.gamma * self.Q_target(next_states).detach().max(1)[0].reshape((-1, 1))
        predictions = self.Q(states).gather(1, actions)
        
        loss = F.mse_loss(predictions, targets)
        
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()
        
        self._moving_average()
    
    def _moving_average(self):
        for target_param, param in zip(self.Q_target.parameters(), self.Q.parameters()):
            target_param.data.copy_((1 - self.tau) * target_param.data + self.tau * param.data)
agent = Agent()
for i in range(int(1e3)):
    env_info = env.reset(train_mode=True)[brain_name]
    state = env_info.vector_observations[0]
    score = 0 
    while True:
        action = agent.act(state)                      # select an action
        env_info = env.step(action)[brain_name]        # send the action to the environment
        next_state = env_info.vector_observations[0]   # get the next state
        reward = env_info.rewards[0]                   # get the reward
        done = env_info.local_done[0]                  # see if episode has finished
        agent.step(state, action, reward, next_state, done)
        score += reward                                # update the score
        state = next_state                             # roll over the state to next time step
        if done:                                       # exit loop if episode finished
            break
    
    print("Episode {} Score: {}".format(i, score))<jupyter_output>Episode 0 Score: -1.0
Episode 1 Score: 3.0
Episode 2 Score: 0.0
Episode 3 Score: 1.0
Episode 4 Score: 0.0
Episode 5 Score: 0.0
Episode 6 Score: -1.0
Episode 7 Score: -3.0
Episode 8 Score: -1.0
Episode 9 Score: 2.0
Episode 10 Score: -2.0
Episode 11 Score: -1.0
Episode 12 Score: -2.0
Episode 13 Score: 0.0
Episode 14 Score: -2.0
Episode 15 Score: 2.0
Episode 16 Score: -4.0
Episode 17 Score: -2.0
Episode 18 Score: -3.0
Episode 19 Score: -3.0
Episode 20 Score: 3.0
Episode 21 Score: -4.0
Episode 22 Score: -2.0
Episode 23 Score: 0.0
Episode 24 Score: 1.0
Episode 25 Score: -2.0
Episode 26 Score: 7.0
Episode 27 Score: -3.0
Episode 28 Score: -3.0
Episode 29 Score: 2.0
Episode 30 Score: -1.0
Episode 31 Score: 1.0
Episode 32 Score: -4.0
Episode 33 Score: 0.0
Episode 34 Score: -2.0
Episode 35 Score: 3.0
Episode 36 Score: 4.0
Episode 37 Score: 4.0
Episode 38 Score: 2.0
Episode 39 Score: -2.0
Episode 40 Score: 8.0
Episode 41 Score: 3.0
Episode 42 Score: 6.0
Episode 43 Score: 3.0
Episode 44 Score: -1.[...]<jupyter_text>When finished, you can close the environment.<jupyter_code>env.close()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup