<jupyter_start><jupyter_text>`Introduction `_ ||`Tensors `_ ||**Autograd** ||`Building Models `_ ||`TensorBoard Support `_ ||`Training Models `_ ||`Model Understanding `_The Fundamentals of Autograd============================Follow along with the video below or on `youtube `__... raw:: html           PyTorch’s *Autograd* feature is part of what make PyTorch flexible andfast for building machine learning projects. It allows for the rapid andeasy computation of multiple partial derivatives (also referred to as*gradients)* over a complex computation. This operation is central tobackpropagation-based neural network learning.The power of autograd comes from the fact that it traces yourcomputation dynamically *at runtime,* meaning that if your model hasdecision branches, or loops whose lengths are not known until runtime,the computation will still be traced correctly, and you’ll get correctgradients to drive learning. This, combined with the fact that yourmodels are built in Python, offers far more flexibility than frameworksthat rely on static analysis of a more rigidly-structured model forcomputing gradients.What Do We Need Autograd For?----------------------------- A machine learning model is a *function*, with inputs and outputs. Forthis discussion, we’ll treat the inputs a as an *i*-dimensional vector$\vec{x}$, with elements $x_{i}$. We can then express themodel, *M*, as a vector-valued function of the input: $\vec{y} =\vec{M}(\vec{x})$. (We treat the value of M’s output asa vector because in general, a model may have any number of outputs.)Since we’ll mostly be discussing autograd in the context of training,our output of interest will be the model’s loss. The *loss function*L($\vec{y}$) = L($\vec{M}$\ ($\vec{x}$)) is asingle-valued scalar function of the model’s output. This functionexpresses how far off our model’s prediction was from a particularinput’s *ideal* output. *Note: After this point, we will often omit thevector sign where it should be contextually clear - e.g.,* $y$instead of $\vec y$.In training a model, we want to minimize the loss. In the idealized caseof a perfect model, that means adjusting its learning weights - that is,the adjustable parameters of the function - such that loss is zero forall inputs. In the real world, it means an iterative process of nudgingthe learning weights until we see that we get a tolerable loss for awide variety of inputs.How do we decide how far and in which direction to nudge the weights? Wewant to *minimize* the loss, which means making its first derivativewith respect to the input equal to 0:$\frac{\partial L}{\partial x} = 0$.Recall, though, that the loss is not *directly* derived from the input,but a function of the model’s output (which is a function of the inputdirectly), $\frac{\partial L}{\partial x}$ =$\frac{\partial {L({\vec y})}}{\partial x}$. By the chain rule ofdifferential calculus, we have$\frac{\partial {L({\vec y})}}{\partial x}$ =$\frac{\partial L}{\partial y}\frac{\partial y}{\partial x}$ =$\frac{\partial L}{\partial y}\frac{\partial M(x)}{\partial x}$.$\frac{\partial M(x)}{\partial x}$ is where things get complex.The partial derivatives of the model’s outputs with respect to itsinputs, if we were to expand the expression using the chain rule again,would involve many local partial derivatives over every multipliedlearning weight, every activation function, and every other mathematicaltransformation in the model. The full expression for each such partialderivative is the sum of the products of the local gradient of *everypossible path* through the computation graph that ends with the variablewhose gradient we are trying to measure.In particular, the gradients over the learning weights are of interestto us - they tell us *what direction to change each weight* to get theloss function closer to zero.Since the number of such local derivatives (each corresponding to aseparate path through the model’s computation graph) will tend to go upexponentially with the depth of a neural network, so does the complexityin computing them. This is where autograd comes in: It tracks thehistory of every computation. Every computed tensor in your PyTorchmodel carries a history of its input tensors and the function used tocreate it. Combined with the fact that PyTorch functions meant to act ontensors each have a built-in implementation for computing their ownderivatives, this greatly speeds the computation of the localderivatives needed for learning.A Simple Example----------------That was a lot of theory - but what does it look like to use autograd inpractice?Let’s start with a straightforward example. First, we’ll do some importsto let us graph our results:<jupyter_code># %matplotlib inline

import torch

import matplotlib.pyplot as plt
import matplotlib.ticker as ticker
import math<jupyter_output><empty_output><jupyter_text>Next, we’ll create an input tensor full of evenly spaced values on theinterval $[0, 2{\pi}]$, and specify ``requires_grad=True``. (Likemost functions that create tensors, ``torch.linspace()`` accepts anoptional ``requires_grad`` option.) Setting this flag means that inevery computation that follows, autograd will be accumulating thehistory of the computation in the output tensors of that computation.<jupyter_code>a = torch.linspace(0., 2. * math.pi, steps=25, requires_grad=True)
print(a)<jupyter_output><empty_output><jupyter_text>Next, we’ll perform a computation, and plot its output in terms of itsinputs:<jupyter_code>b = torch.sin(a)
plt.plot(a.detach(), b.detach())<jupyter_output><empty_output><jupyter_text>Let’s have a closer look at the tensor ``b``. When we print it, we seean indicator that it is tracking its computation history:<jupyter_code>print(b)<jupyter_output><empty_output><jupyter_text>This ``grad_fn`` gives us a hint that when we execute thebackpropagation step and compute gradients, we’ll need to compute thederivative of $sin(x)$ for all this tensor’s inputs.Let’s perform some more computations:<jupyter_code>c = 2 * b
print(c)

d = c + 1
print(d)<jupyter_output><empty_output><jupyter_text>Finally, let’s compute a single-element output. When you call``.backward()`` on a tensor with no arguments, it expects the callingtensor to contain only a single element, as is the case when computing aloss function.<jupyter_code>out = d.sum()
print(out)<jupyter_output><empty_output><jupyter_text>Each ``grad_fn`` stored with our tensors allows you to walk thecomputation all the way back to its inputs with its ``next_functions``property. We can see below that drilling down on this property on ``d``shows us the gradient functions for all the prior tensors. Note that``a.grad_fn`` is reported as ``None``, indicating that this was an inputto the function with no history of its own.<jupyter_code>print('d:')
print(d.grad_fn)
print(d.grad_fn.next_functions)
print(d.grad_fn.next_functions[0][0].next_functions)
print(d.grad_fn.next_functions[0][0].next_functions[0][0].next_functions)
print(d.grad_fn.next_functions[0][0].next_functions[0][0].next_functions[0][0].next_functions)
print('\nc:')
print(c.grad_fn)
print('\nb:')
print(b.grad_fn)
print('\na:')
print(a.grad_fn)<jupyter_output><empty_output><jupyter_text>With all this machinery in place, how do we get derivatives out? Youcall the ``backward()`` method on the output, and check the input’s``grad`` property to inspect the gradients:<jupyter_code>out.backward()
print(a.grad)
plt.plot(a.detach(), a.grad.detach())<jupyter_output><empty_output><jupyter_text>Recall the computation steps we took to get here:::   a = torch.linspace(0., 2. * math.pi, steps=25, requires_grad=True)   b = torch.sin(a)   c = 2 * b   d = c + 1   out = d.sum()Adding a constant, as we did to compute ``d``, does not change thederivative. That leaves $c = 2 * b = 2 * sin(a)$, the derivativeof which should be $2 * cos(a)$. Looking at the graph above,that’s just what we see.Be aware than only *leaf nodes* of the computation have their gradientscomputed. If you tried, for example, ``print(c.grad)`` you’d get back``None``. In this simple example, only the input is a leaf node, so onlyit has gradients computed.Autograd in Training--------------------We’ve had a brief look at how autograd works, but how does it look whenit’s used for its intended purpose? Let’s define a small model andexamine how it changes after a single training batch. First, define afew constants, our model, and some stand-ins for inputs and outputs:<jupyter_code>BATCH_SIZE = 16
DIM_IN = 1000
HIDDEN_SIZE = 100
DIM_OUT = 10

class TinyModel(torch.nn.Module):

    def __init__(self):
        super(TinyModel, self).__init__()
        
        self.layer1 = torch.nn.Linear(1000, 100)
        self.relu = torch.nn.ReLU()
        self.layer2 = torch.nn.Linear(100, 10)
    
    def forward(self, x):
        x = self.layer1(x)
        x = self.relu(x)
        x = self.layer2(x)
        return x
    
some_input = torch.randn(BATCH_SIZE, DIM_IN, requires_grad=False)
ideal_output = torch.randn(BATCH_SIZE, DIM_OUT, requires_grad=False)

model = TinyModel()<jupyter_output><empty_output><jupyter_text>One thing you might notice is that we never specify``requires_grad=True`` for the model’s layers. Within a subclass of``torch.nn.Module``, it’s assumed that we want to track gradients on thelayers’ weights for learning.If we look at the layers of the model, we can examine the values of theweights, and verify that no gradients have been computed yet:<jupyter_code>print(model.layer2.weight[0][0:10]) # just a small slice
print(model.layer2.weight.grad)<jupyter_output><empty_output><jupyter_text>Let’s see how this changes when we run through one training batch. For aloss function, we’ll just use the square of the Euclidean distancebetween our ``prediction`` and the ``ideal_output``, and we’ll use abasic stochastic gradient descent optimizer.<jupyter_code>optimizer = torch.optim.SGD(model.parameters(), lr=0.001)

prediction = model(some_input)

loss = (ideal_output - prediction).pow(2).sum()
print(loss)<jupyter_output><empty_output><jupyter_text>Now, let’s call ``loss.backward()`` and see what happens:<jupyter_code>loss.backward()
print(model.layer2.weight[0][0:10])
print(model.layer2.weight.grad[0][0:10])<jupyter_output><empty_output><jupyter_text>We can see that the gradients have been computed for each learningweight, but the weights remain unchanged, because we haven’t run theoptimizer yet. The optimizer is responsible for updating model weightsbased on the computed gradients.<jupyter_code>optimizer.step()
print(model.layer2.weight[0][0:10])
print(model.layer2.weight.grad[0][0:10])<jupyter_output><empty_output><jupyter_text>You should see that ``layer2``\ ’s weights have changed.One important thing about the process: After calling``optimizer.step()``, you need to call ``optimizer.zero_grad()``, orelse every time you run ``loss.backward()``, the gradients on thelearning weights will accumulate:<jupyter_code>print(model.layer2.weight.grad[0][0:10])

for i in range(0, 5):
    prediction = model(some_input)
    loss = (ideal_output - prediction).pow(2).sum()
    loss.backward()
    
print(model.layer2.weight.grad[0][0:10])

optimizer.zero_grad()

print(model.layer2.weight.grad[0][0:10])<jupyter_output><empty_output><jupyter_text>After running the cell above, you should see that after running``loss.backward()`` multiple times, the magnitudes of most of thegradients will be much larger. Failing to zero the gradients beforerunning your next training batch will cause the gradients to blow up inthis manner, causing incorrect and unpredictable learning results.Turning Autograd Off and On---------------------------There are situations where you will need fine-grained control overwhether autograd is enabled. There are multiple ways to do this,depending on the situation.The simplest is to change the ``requires_grad`` flag on a tensordirectly:<jupyter_code>a = torch.ones(2, 3, requires_grad=True)
print(a)

b1 = 2 * a
print(b1)

a.requires_grad = False
b2 = 2 * a
print(b2)<jupyter_output><empty_output><jupyter_text>In the cell above, we see that ``b1`` has a ``grad_fn`` (i.e., a tracedcomputation history), which is what we expect, since it was derived froma tensor, ``a``, that had autograd turned on. When we turn off autogradexplicitly with ``a.requires_grad = False``, computation history is nolonger tracked, as we see when we compute ``b2``.If you only need autograd turned off temporarily, a better way is to usethe ``torch.no_grad()``:<jupyter_code>a = torch.ones(2, 3, requires_grad=True) * 2
b = torch.ones(2, 3, requires_grad=True) * 3

c1 = a + b
print(c1)

with torch.no_grad():
    c2 = a + b

print(c2)

c3 = a * b
print(c3)<jupyter_output><empty_output><jupyter_text>``torch.no_grad()`` can also be used as a function or method dectorator:<jupyter_code>def add_tensors1(x, y):
    return x + y

@torch.no_grad()
def add_tensors2(x, y):
    return x + y


a = torch.ones(2, 3, requires_grad=True) * 2
b = torch.ones(2, 3, requires_grad=True) * 3

c1 = add_tensors1(a, b)
print(c1)

c2 = add_tensors2(a, b)
print(c2)<jupyter_output><empty_output><jupyter_text>There’s a corresponding context manager, ``torch.enable_grad()``, forturning autograd on when it isn’t already. It may also be used as adecorator.Finally, you may have a tensor that requires gradient tracking, but youwant a copy that does not. For this we have the ``Tensor`` object’s``detach()`` method - it creates a copy of the tensor that is *detached*from the computation history:<jupyter_code>x = torch.rand(5, requires_grad=True)
y = x.detach()

print(x)
print(y)<jupyter_output><empty_output><jupyter_text>We did this above when we wanted to graph some of our tensors. This isbecause ``matplotlib`` expects a NumPy array as input, and the implicitconversion from a PyTorch tensor to a NumPy array is not enabled fortensors with requires_grad=True. Making a detached copy lets us moveforward.Autograd and In-place Operations~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~In every example in this notebook so far, we’ve used variables tocapture the intermediate values of a computation. Autograd needs theseintermediate values to perform gradient computations. *For this reason,you must be careful about using in-place operations when usingautograd.* Doing so can destroy information you need to computederivatives in the ``backward()`` call. PyTorch will even stop you ifyou attempt an in-place operation on leaf variable that requiresautograd, as shown below.NoteThe following code cell throws a runtime error. This is expected.::   a = torch.linspace(0., 2. * math.pi, steps=25, requires_grad=True)   torch.sin_(a) Autograd Profiler-----------------Autograd tracks every step of your computation in detail. Such acomputation history, combined with timing information, would make ahandy profiler - and autograd has that feature baked in. Here’s a quickexample usage:<jupyter_code>device = torch.device('cpu')
run_on_gpu = False
if torch.cuda.is_available():
    device = torch.device('cuda')
    run_on_gpu = True
    
x = torch.randn(2, 3, requires_grad=True)
y = torch.rand(2, 3, requires_grad=True)
z = torch.ones(2, 3, requires_grad=True)

with torch.autograd.profiler.profile(use_cuda=run_on_gpu) as prf:
    for _ in range(1000):
        z = (z / x) * y
        
print(prf.key_averages().table(sort_by='self_cpu_time_total'))<jupyter_output><empty_output><jupyter_text>The profiler can also label individual sub-blocks of code, break out thedata by input tensor shape, and export data as a Chrome tracing toolsfile. For full details of the API, see the`documentation `__.Advanced Topic: More Autograd Detail and the High-Level API-----------------------------------------------------------If you have a function with an n-dimensional input and m-dimensionaloutput, $\vec{y}=f(\vec{x})$, the complete gradient is a matrix ofthe derivative of every output with respect to every input, called the*Jacobian:*\begin{align}J     =     \left(\begin{array}{ccc}     \frac{\partial y_{1}}{\partial x_{1}} & \cdots & \frac{\partial y_{1}}{\partial x_{n}}\\     \vdots & \ddots & \vdots\\     \frac{\partial y_{m}}{\partial x_{1}} & \cdots & \frac{\partial y_{m}}{\partial x_{n}}     \end{array}\right)\end{align}If you have a second function, $l=g\left(\vec{y}\right)$ thattakes m-dimensional input (that is, the same dimensionality as theoutput above), and returns a scalar output, you can express itsgradients with respect to $\vec{y}$ as a column vector,$v=\left(\begin{array}{ccc}\frac{\partial l}{\partial y_{1}} & \cdots & \frac{\partial l}{\partial y_{m}}\end{array}\right)^{T}$- which is really just a one-column Jacobian.More concretely, imagine the first function as your PyTorch model (withpotentially many inputs and many outputs) and the second function as aloss function (with the model’s output as input, and the loss value asthe scalar output).If we multiply the first function’s Jacobian by the gradient of thesecond function, and apply the chain rule, we get:\begin{align}J^{T}\cdot v=\left(\begin{array}{ccc}   \frac{\partial y_{1}}{\partial x_{1}} & \cdots & \frac{\partial y_{m}}{\partial x_{1}}\\   \vdots & \ddots & \vdots\\   \frac{\partial y_{1}}{\partial x_{n}} & \cdots & \frac{\partial y_{m}}{\partial x_{n}}   \end{array}\right)\left(\begin{array}{c}   \frac{\partial l}{\partial y_{1}}\\   \vdots\\   \frac{\partial l}{\partial y_{m}}   \end{array}\right)=\left(\begin{array}{c}   \frac{\partial l}{\partial x_{1}}\\   \vdots\\   \frac{\partial l}{\partial x_{n}}   \end{array}\right)\end{align}Note: You could also use the equivalent operation $v^{T}\cdot J$,and get back a row vector.The resulting column vector is the *gradient of the second function withrespect to the inputs of the first* - or in the case of our model andloss function, the gradient of the loss with respect to the modelinputs.**``torch.autograd`` is an engine for computing these products.** Thisis how we accumulate the gradients over the learning weights during thebackward pass.For this reason, the ``backward()`` call can *also* take an optionalvector input. This vector represents a set of gradients over the tensor,which are multiplied by the Jacobian of the autograd-traced tensor thatprecedes it. Let’s try a specific example with a small vector:<jupyter_code>x = torch.randn(3, requires_grad=True)

y = x * 2
while y.data.norm() < 1000:
    y = y * 2

print(y)<jupyter_output><empty_output><jupyter_text>If we tried to call ``y.backward()`` now, we’d get a runtime error and amessage that gradients can only be *implicitly* computed for scalaroutputs. For a multi-dimensional output, autograd expects us to providegradients for those three outputs that it can multiply into theJacobian:<jupyter_code>v = torch.tensor([0.1, 1.0, 0.0001], dtype=torch.float) # stand-in for gradients
y.backward(v)

print(x.grad)<jupyter_output><empty_output><jupyter_text>(Note that the output gradients are all related to powers of two - whichwe’d expect from a repeated doubling operation.)The High-Level API~~~~~~~~~~~~~~~~~~There is an API on autograd that gives you direct access to importantdifferential matrix and vector operations. In particular, it allows youto calculate the Jacobian and the *Hessian* matrices of a particularfunction for particular inputs. (The Hessian is like the Jacobian, butexpresses all partial *second* derivatives.) It also provides methodsfor taking vector products with these matrices.Let’s take the Jacobian of a simple function, evaluated for a 2single-element inputs:<jupyter_code>def exp_adder(x, y):
    return 2 * x.exp() + 3 * y

inputs = (torch.rand(1), torch.rand(1)) # arguments for the function
print(inputs)
torch.autograd.functional.jacobian(exp_adder, inputs)<jupyter_output><empty_output><jupyter_text>If you look closely, the first output should equal $2e^x$ (sincethe derivative of $e^x$ is $e^x$), and the second valueshould be 3.You can, of course, do this with higher-order tensors:<jupyter_code>inputs = (torch.rand(3), torch.rand(3)) # arguments for the function
print(inputs)
torch.autograd.functional.jacobian(exp_adder, inputs)<jupyter_output><empty_output><jupyter_text>The ``torch.autograd.functional.hessian()`` method works identically(assuming your function is twice differentiable), but returns a matrixof all second derivatives.There is also a function to directly compute the vector-Jacobianproduct, if you provide the vector:<jupyter_code>def do_some_doubling(x):
    y = x * 2
    while y.data.norm() < 1000:
        y = y * 2
    return y

inputs = torch.randn(3)
my_gradients = torch.tensor([0.1, 1.0, 0.0001])
torch.autograd.functional.vjp(do_some_doubling, inputs, v=my_gradients)<jupyter_output><empty_output>