<jupyter_start><jupyter_text>Parameter ManagementOnce we have chosen an architectureand set our hyperparameters,we proceed to the training loop,where our goal is to find parameter valuesthat minimize our loss function.After training, we will need these parametersin order to make future predictions.Additionally, we will sometimes wishto extract the parameterseither to reuse them in some other context,to save our model to disk so thatit may be executed in other software,or for examination in the hope ofgaining scientific understanding.Most of the time, we will be ableto ignore the nitty-gritty detailsof how parameters are declaredand manipulated, relying on deep learning frameworksto do the heavy lifting.However, when we move away fromstacked architectures with standard layers,we will sometimes need to get into the weedsof declaring and manipulating parameters.In this section, we cover the following:* Accessing parameters for debugging, diagnostics, and visualizations.* Parameter initialization.* Sharing parameters across different model components.We start by focusing on an MLP with one hidden layer.<jupyter_code>import tensorflow as tf

net = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(4, activation=tf.nn.relu),
    tf.keras.layers.Dense(1),
])

X = tf.random.uniform((2, 4))
net(X)<jupyter_output><empty_output><jupyter_text>Parameter AccessLet us start with how to access parametersfrom the models that you already know.When a model is defined via the `Sequential` class,we can first access any layer by indexinginto the model as though it were a list.Each layer's parameters are convenientlylocated in its attribute.We can inspect the parameters of the second fully-connected layer as follows.<jupyter_code>print(net.layers[2].weights)<jupyter_output>[<tf.Variable 'dense_1/kernel:0' shape=(4, 1) dtype=float32, numpy=
array([[-0.70263696],
       [-0.22071278],
       [ 0.70914674],
       [ 0.6964009 ]], dtype=float32)>, <tf.Variable 'dense_1/bias:0' shape=(1,) dtype=float32, numpy=array([0.], dtype=float32)>]<jupyter_text>The output tells us a few important things.First, this fully-connected layercontains two parameters,corresponding to that layer'sweights and biases, respectively.Both are stored as single precision floats (float32).Note that the names of the parametersallow us to uniquely identifyeach layer's parameters,even in a network containing hundreds of layers. Targeted ParametersNote that each parameter is representedas an instance of the parameter class.To do anything useful with the parameters,we first need to access the underlying numerical values.There are several ways to do this.Some are simpler while others are more general.The following code extracts the biasfrom the second neural network layer, which returns a parameter class instance, and further accesses that parameter's value.<jupyter_code>print(type(net.layers[2].weights[1]))
print(net.layers[2].weights[1])
print(tf.convert_to_tensor(net.layers[2].weights[1]))<jupyter_output><class 'tensorflow.python.ops.resource_variable_ops.ResourceVariable'>
<tf.Variable 'dense_1/bias:0' shape=(1,) dtype=float32, numpy=array([0.], dtype=float32)>
tf.Tensor([0.], shape=(1,), dtype=float32)<jupyter_text>All Parameters at OnceWhen we need to perform operations on all parameters,accessing them one-by-one can grow tedious.The situation can grow especially unwieldywhen we work with more complex blocks (e.g., nested blocks),since we would need to recursethrough the entire tree to extracteach sub-block's parameters. Below we demonstrate accessing the parameters of the first fully-connected layer vs. accessing all layers.<jupyter_code>print(net.layers[1].weights)
print(net.get_weights())<jupyter_output>[<tf.Variable 'dense/kernel:0' shape=(4, 4) dtype=float32, numpy=
array([[ 0.7081526 , -0.01244313,  0.55799276, -0.6492791 ],
       [ 0.22770828, -0.770811  , -0.54992485, -0.34173292],
       [-0.5914638 ,  0.4005019 , -0.13743204,  0.28752023],
       [-0.7954682 ,  0.18603677,  0.1703785 , -0.03160411]],
      dtype=float32)>, <tf.Variable 'dense/bias:0' shape=(4,) dtype=float32, numpy=array([0., 0., 0., 0.], dtype=float32)>]
[array([[ 0.7081526 , -0.01244313,  0.55799276, -0.6492791 ],
       [ 0.22770828, -0.770811  , -0.54992485, -0.34173292],
       [-0.5914638 ,  0.4005019 , -0.13743204,  0.28752023],
       [-0.7954682 ,  0.18603677,  0.1703785 , -0.03160411]],
      dtype=float32), array([0., 0., 0., 0.], dtype=float32), array([[-0.70263696],
       [-0.22071278],
       [ 0.70914674],
       [ 0.6964009 ]], dtype=float32), array([0.], dtype=float32)]<jupyter_text>This provides us with another way of accessing the parameters of the network as follows.<jupyter_code>net.get_weights()[1]<jupyter_output><empty_output><jupyter_text>Collecting Parameters from Nested BlocksLet us see how the parameter naming conventions workif we nest multiple blocks inside each other.For that we first define a function that produces blocks(a block factory, so to speak) and thencombine these inside yet larger blocks.<jupyter_code>def block1(name):
    return tf.keras.Sequential([
        tf.keras.layers.Flatten(),
        tf.keras.layers.Dense(4, activation=tf.nn.relu)],
        name=name)

def block2():
    net = tf.keras.Sequential()
    for i in range(4):
        # Nested here
        net.add(block1(name=f'block-{i}'))
    return net

rgnet = tf.keras.Sequential()
rgnet.add(block2())
rgnet.add(tf.keras.layers.Dense(1))
rgnet(X)<jupyter_output><empty_output><jupyter_text>Now that we have designed the network,let us see how it is organized.<jupyter_code>print(rgnet.summary())<jupyter_output>Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
sequential_2 (Sequential)    (2, 4)                    80        
_________________________________________________________________
dense_6 (Dense)              (2, 1)                    5         
=================================================================
Total params: 85
Trainable params: 85
Non-trainable params: 0
_________________________________________________________________
None<jupyter_text>Since the layers are hierarchically nested,we can also access them as thoughindexing through nested lists.For instance, we can access the first major block,within it the second sub-block,and within that the bias of the first layer,with as follows.<jupyter_code>rgnet.layers[0].layers[1].layers[1].weights[1]<jupyter_output><empty_output><jupyter_text>Parameter InitializationNow that we know how to access the parameters,let us look at how to initialize them properly.We discussed the need for proper initialization in :numref:`sec_numerical_stability`.The deep learning framework provides default random initializations to its layers.However, we often want to initialize our weightsaccording to various other protocols. The framework provides most commonlyused protocols, and also allows to create a custom initializer. By default, Keras initializes weight matrices uniformly by drawing from a range that is computed according to the input and output dimension, and the bias parameters are all set to zero.TensorFlow provides a variety of initialization methods both in the root module and the `keras.initializers` module.  Built-in InitializationLet us begin by calling on built-in initializers.The code below initializes all weight parametersas Gaussian random variableswith standard deviation 0.01, while bias parameters cleared to zero.<jupyter_code>net = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(
        4, activation=tf.nn.relu,
        kernel_initializer=tf.random_normal_initializer(mean=0, stddev=0.01),
        bias_initializer=tf.zeros_initializer()),
    tf.keras.layers.Dense(1)])

net(X)
net.weights[0], net.weights[1]<jupyter_output><empty_output><jupyter_text>We can also initialize all the parametersto a given constant value (say, 1).<jupyter_code>net = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(
        4, activation=tf.nn.relu,
        kernel_initializer=tf.keras.initializers.Constant(1),
        bias_initializer=tf.zeros_initializer()),
    tf.keras.layers.Dense(1),
])

net(X)
net.weights[0], net.weights[1]<jupyter_output><empty_output><jupyter_text>We can also apply different initializers for certain blocks.For example, below we initialize the first layerwith the Xavier initializerand initialize the second layerto a constant value of 42.<jupyter_code>net = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(
        4,
        activation=tf.nn.relu,
        kernel_initializer=tf.keras.initializers.GlorotUniform()),
    tf.keras.layers.Dense(
        1, kernel_initializer=tf.keras.initializers.Constant(1)),
])

net(X)
print(net.layers[1].weights[0])
print(net.layers[2].weights[0])<jupyter_output><tf.Variable 'dense_11/kernel:0' shape=(4, 4) dtype=float32, numpy=
array([[ 0.77454144, -0.42678392,  0.18757135,  0.2861511 ],
       [-0.12093848, -0.63353795, -0.55993074, -0.26746225],
       [ 0.50490373,  0.38413602, -0.83991975,  0.181912  ],
       [-0.08994305, -0.03767806,  0.82064635, -0.33605212]],
      dtype=float32)>
<tf.Variable 'dense_12/kernel:0' shape=(4, 1) dtype=float32, numpy=
array([[1.],
       [1.],
       [1.],
       [1.]], dtype=float32)><jupyter_text>Custom InitializationSometimes, the initialization methods we needare not provided by the deep learning framework.In the example below, we define an initializerfor any weight parameter $w$ using the following strange distribution:$$\begin{aligned}    w \sim \begin{cases}        U(5, 10) & \text{ with probability } \frac{1}{4} \\            0    & \text{ with probability } \frac{1}{2} \\        U(-10, -5) & \text{ with probability } \frac{1}{4}    \end{cases}\end{aligned}$$ Here we define a subclass of `Initializer` and implement the `__call__`function that return a desired tensor given the shape and data type.<jupyter_code>class MyInit(tf.keras.initializers.Initializer):
    def __call__(self, shape, dtype=None):
        data=tf.random.uniform(shape, -10, 10, dtype=dtype)
        factor=(tf.abs(data) >= 5)
        factor=tf.cast(factor, tf.float32)
        return data * factor        

net = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(
        4,
        activation=tf.nn.relu,
        kernel_initializer=MyInit()),
    tf.keras.layers.Dense(1),
])

net(X)
print(net.layers[1].weights[0])<jupyter_output><tf.Variable 'dense_13/kernel:0' shape=(4, 4) dtype=float32, numpy=
array([[ 5.889845 ,  8.331539 ,  0.       , -0.       ],
       [-0.       , -0.       , -0.       , -9.657394 ],
       [-0.       , -0.       , -0.       , -0.       ],
       [-0.       , -5.579841 , -5.4117727, -7.462876 ]], dtype=float32)><jupyter_text>Note that we always have the optionof setting parameters directly.<jupyter_code>net.layers[1].weights[0][:].assign(net.layers[1].weights[0] + 1)
net.layers[1].weights[0][0, 0].assign(42)
net.layers[1].weights[0]<jupyter_output><empty_output><jupyter_text>Tied ParametersOften, we want to share parameters across multiple layers.Let us see how to do this elegantly.In the following we allocate a dense layerand then use its parameters specificallyto set those of another layer.<jupyter_code># tf.keras behaves a bit differently. It removes the duplicate layer
# automatically
shared = tf.keras.layers.Dense(4, activation=tf.nn.relu)
net = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(),
    shared,
    shared,
    tf.keras.layers.Dense(1),
])

net(X)
# Check whether the parameters are different
print(len(net.layers) == 3)<jupyter_output>True
StarcoderdataJupyterStructuredCleanDedup