We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Maybe there is a bug in chainer.optimizer.weight_decay(decay).
In the [optimizer.py](https://github.com/pfnet/chainer/blob/30807b4f9f40177f592099663b86a6bb3eb1f867/chainer/optimizer.py) file,  `weight_decay` is implemented as following:

``` python
 def weight_decay(self, decay):
     for p, g, _ in self.tuples:
         g -= decay * p
```

However, the gradient is decayed, not the weight. So if we update the weights `param -= lr * grad`, because the grad becomes smaller, the param (weight) becomes larger, which is not we want. 

As far as I know the regularized gradient decent is `p = p * (1-decay) - lr * grad`, so I think the weight should be decayed not the gradient.


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `chainer/optimizer.py`
Content:
```
1 import math
2 
3 import numpy
4 
5 from chainer import cuda
6 
7 
8 # TODO(delta2323): Make it public function and move it to common directory.
9 
10 
11 def _sqnorm(x):
12     if isinstance(x, cuda.GPUArray):
13         with cuda.using_device(x):
14             return float(cuda.gpuarray.dot(x, x).get())
15     x = x.ravel()
16     return float(x.dot(x))
17 
18 
19 class Optimizer(object):
20 
21     """Base class of all numerical optimizers.
22 
23     Optimizer is set up with references to parameters and gradients, and
24     then on every call of :meth:`update`, it updates parameters based on
25     corresponding gradients. Optimizer implementations must override
26     :meth:`update_one` method, which updates one parameter array using the
27     corresponding gradient array.
28 
29     Optimizer can optionally use state for each parameter/gradient pair. It is
30     initialized by :meth:`init_state` method at set up.
31 
32     Attributes:
33         t (int): Number of update steps. It can be used in :meth:`update_one`
34             implementation, where :attr:`t` is incremented beforehand.
35 
36     """
37 
38     def setup(self, params_grads):
39         """Prepares states for all given parameter/gradient pairs.
40 
41         Args:
42             params_grads: Tuple (pair) of two tuples. The first element is a
43                 tuple of parameter arrays, and the second is a tuple of
44                 corresponding gradient arrays.
45                 Return value of :meth:`FunctionSet.collect_parameters` method
46                 can be used.
47 
48         """
49         self.t = 0
50         self.tuples = []
51         for p, g in zip(*params_grads):
52             with cuda.using_device(p):
53                 state = self.init_state(p, g)
54                 self.tuples.append((p, g, state))
55 
56     def init_state(self, param, grad):
57         """Returns the initial state for given parameter and gradient.
58 
59         Default implementation delegates the procedure to
60         :meth:`init_state_cpu` or :meth:`init_state_gpu` depending on the type
61         of ``param``.
62 
63         Args:
64             param: Parameter array.
65             grad: Gradient array corresponding to ``param``.
66 
67         Returns:
68             Initial state value.
69 
70             .. warning::
71 
72                 Note that, on every call of :meth:`update_one`, the state value
73                 is passed by value and then the method updates its content, so
74                 the state must be a reference. Especiallly, one cannot use a
75                 value of built-in numeric type. If the state is one scalar
76                 value, it is recommended to use scalar array, i.e.
77                 :class:`~numpy.ndarray` with shape ``()``.
78 
79         """
80         if isinstance(param, cuda.GPUArray):
81             return self.init_state_gpu(param, grad)
82         return self.init_state_cpu(param, grad)
83 
84     def init_state_cpu(self, param, grad):
85         """Returns the initial state for given parameter and gradient on GPU.
86 
87         Args:
88             param (~numpy.ndarray): Parameter array.
89             grad  (~numpy.ndarray): Gradient array.
90 
91         Returns:
92             Initial state value.
93 
94         .. seealso:: :meth:`init_state`, :meth:`init_state_gpu`
95 
96         """
97         return None
98 
99     def init_state_gpu(self, param, grad):
100         """Returns the initial state for given parameter and gradient on CPU.
101 
102         Args:
103             param (~pycuda.gpuarray.GPUArray): Parameter array.
104             grad  (~pycuda.gpuarray.GPUArray): Gradient array.
105 
106         Returns:
107             Initial state value.
108 
109         .. seealso:: :meth:`init_state`, :meth:`init_state_gpu`
110 
111         """
112         return None
113 
114     def zero_grads(self):
115         """Fills all gradient arrays by zeros.
116 
117         This method should be call before backprop takes place, since
118         gradients are accumulated on backprop.
119 
120         """
121         for _, g, _ in self.tuples:
122             if isinstance(g, cuda.GPUArray):
123                 with cuda.using_device(g):
124                     g.fill(0)
125             else:
126                 g.fill(0)
127 
128     def compute_grads_norm(self):
129         """Computes the norm of whole gradients.
130 
131         Returns:
132             float: L2 norm of whole gradients, i.e. square root of sum of
133             square of all gradient elements.
134 
135         .. warning::
136 
137             This method returns a CPU-computed value, which means that this
138             method synchronizes between CPU and GPU if at least one of the
139             gradients reside on the GPU.
140 
141         """
142         # TODO(beam2d): Make it asynchronous to CPU when gradients exist on GPU
143         sqnorm = 0
144         for _, g, _ in self.tuples:
145             sqnorm += _sqnorm(g)
146         return math.sqrt(sqnorm)
147 
148     def clip_grads(self, maxnorm):
149         """Clips the norm of whole gradients up to given threshold.
150 
151         Args:
152             maxnorm (float): Threshold of gradient L2 norm.
153 
154         .. seealso::
155 
156             :meth:`compute_grads_norm`
157                 It uses this method to compute the gradient norm to be clipped.
158 
159         """
160         norm = self.compute_grads_norm()
161         if norm > maxnorm:
162             ratio = maxnorm / norm
163             for _, g, _ in self.tuples:
164                 with cuda.using_device(g):
165                     g *= ratio
166 
167     def weight_decay(self, decay):
168         """Applies weight decay to the parameter/gradient pairs.
169 
170         Args:
171             decay (float): Coefficient of weight decay
172 
173         """
174         for p, g, _ in self.tuples:
175             if isinstance(p, cuda.GPUArray):
176                 with cuda.using_device(p):
177                     cuda.elementwise('float* g, const float* p, float decay',
178                                      'g[i] -= decay * p[i]',
179                                      'weight_decay')(g, p, decay)
180             else:
181                 g -= decay * p
182 
183     def accumulate_grads(self, grads):
184         """Accumulates gradients from other source.
185 
186         This method just adds given gradient arrays to gradients that this
187         optimizer holds. It is typically used in data-parallel optimization,
188         where gradients for different shards are computed in parallel and
189         aggregated by this method. This method correctly treats multiple GPU
190         devices.
191 
192         Args:
193             grads (Iterable): Iterable of gradient arrays to be accumulated.
194 
195         """
196         for (_, g_dst, _), g_src in zip(self.tuples, grads):
197             if isinstance(g_dst, numpy.ndarray):
198                 g_dst += cuda.to_cpu(g_src)
199                 continue
200 
201             with cuda.using_device(g_dst):
202                 if (isinstance(g_src, cuda.GPUArray) and
203                         g_dst.gpudata.device != g_src.gpudata.device):
204                     g_dst += cuda.copy(g_src, out_device=g_src.gpudata.device)
205                 else:
206                     g_dst += cuda.to_gpu(g_src)
207 
208     def update(self):
209         """Updates all parameters and states using corresponding gradients.
210 
211         This method iteratively calls :meth:`update_one` for each parameter/
212         gradient/state tuple. Beforehand, :attr:`t` attribute is incremented.
213 
214         """
215         self.t += 1
216         for p, g, s in self.tuples:
217             with cuda.using_device(p):
218                 self.update_one(p, g, s)
219 
220     def update_one(self, param, grad, state):
221         """Updates a parameter array and its state using given gradient.
222 
223         The default implementation delegates the procedure to
224         :meth:`update_one_cpu` or :meth:`update_one_gpu` depending on the type
225         of the parameter array. Optimizer implmentation must override these
226         type-specific methods or this :meth:`update_one` method directly.
227 
228         Args:
229             param: Parameter array.
230             grad:  Gradient array.
231             state: State value.
232 
233         .. seealso:: :meth:`update_one_cpu`, :meth:`update_one_gpu`
234 
235         """
236         if isinstance(param, cuda.GPUArray):
237             self.update_one_gpu(param, grad, state)
238         else:
239             self.update_one_cpu(param, grad, state)
240 
241     def update_one_cpu(self, param, grad, state):
242         """Updates a parameter array and its state using given gradient on CPU.
243 
244         Args:
245             param (~numpy.ndarray): Parameter array.
246             grad  (~numpy.ndarray): Gradient array.
247             state: State value.
248 
249         .. seealso:: :meth:`update_one`, :meth:`update_one_gpu`
250 
251         """
252         raise NotImplementedError()
253 
254     def update_one_gpu(self, param, grad, state):
255         """Updates a parameter array and its state using given gradient on GPU.
256 
257         Args:
258             param (~pycuda.gpuarray.GPUArray): Parameter array.
259             grad  (~pycuda.gpuarray.GPUArray): Gradient array.
260             state: State value.
261 
262         .. seealso:: :meth:`update_one`, :meth:`update_one_cpu`
263 
264         """
265         raise NotImplementedError()
266
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/chainer/optimizer.py b/chainer/optimizer.py
--- a/chainer/optimizer.py
+++ b/chainer/optimizer.py
@@ -175,10 +175,10 @@
             if isinstance(p, cuda.GPUArray):
                 with cuda.using_device(p):
                     cuda.elementwise('float* g, const float* p, float decay',
-                                     'g[i] -= decay * p[i]',
+                                     'g[i] += decay * p[i]',
                                      'weight_decay')(g, p, decay)
             else:
-                g -= decay * p
+                g += decay * p
 
     def accumulate_grads(self, grads):
         """Accumulates gradients from other source.

{"golden_diff": "diff --git a/chainer/optimizer.py b/chainer/optimizer.py\n--- a/chainer/optimizer.py\n+++ b/chainer/optimizer.py\n@@ -175,10 +175,10 @@\n             if isinstance(p, cuda.GPUArray):\n                 with cuda.using_device(p):\n                     cuda.elementwise('float* g, const float* p, float decay',\n-                                     'g[i] -= decay * p[i]',\n+                                     'g[i] += decay * p[i]',\n                                      'weight_decay')(g, p, decay)\n             else:\n-                g -= decay * p\n+                g += decay * p\n \n     def accumulate_grads(self, grads):\n         \"\"\"Accumulates gradients from other source.\n", "issue": "Maybe there is a bug in chainer.optimizer.weight_decay(decay).\nIn the [optimizer.py](https://github.com/pfnet/chainer/blob/30807b4f9f40177f592099663b86a6bb3eb1f867/chainer/optimizer.py) file,  `weight_decay` is implemented as following:\n\n``` python\n def weight_decay(self, decay):\n     for p, g, _ in self.tuples:\n         g -= decay * p\n```\n\nHowever, the gradient is decayed, not the weight. So if we update the weights `param -= lr * grad`, because the grad becomes smaller, the param (weight) becomes larger, which is not we want. \n\nAs far as I know the regularized gradient decent is `p = p * (1-decay) - lr * grad`, so I think the weight should be decayed not the gradient.\n\n", "before_files": [{"content": "import math\n\nimport numpy\n\nfrom chainer import cuda\n\n\n# TODO(delta2323): Make it public function and move it to common directory.\n\n\ndef _sqnorm(x):\n    if isinstance(x, cuda.GPUArray):\n        with cuda.using_device(x):\n            return float(cuda.gpuarray.dot(x, x).get())\n    x = x.ravel()\n    return float(x.dot(x))\n\n\nclass Optimizer(object):\n\n    \"\"\"Base class of all numerical optimizers.\n\n    Optimizer is set up with references to parameters and gradients, and\n    then on every call of :meth:`update`, it updates parameters based on\n    corresponding gradients. Optimizer implementations must override\n    :meth:`update_one` method, which updates one parameter array using the\n    corresponding gradient array.\n\n    Optimizer can optionally use state for each parameter/gradient pair. It is\n    initialized by :meth:`init_state` method at set up.\n\n    Attributes:\n        t (int): Number of update steps. It can be used in :meth:`update_one`\n            implementation, where :attr:`t` is incremented beforehand.\n\n    \"\"\"\n\n    def setup(self, params_grads):\n        \"\"\"Prepares states for all given parameter/gradient pairs.\n\n        Args:\n            params_grads: Tuple (pair) of two tuples. The first element is a\n                tuple of parameter arrays, and the second is a tuple of\n                corresponding gradient arrays.\n                Return value of :meth:`FunctionSet.collect_parameters` method\n                can be used.\n\n        \"\"\"\n        self.t = 0\n        self.tuples = []\n        for p, g in zip(*params_grads):\n            with cuda.using_device(p):\n                state = self.init_state(p, g)\n                self.tuples.append((p, g, state))\n\n    def init_state(self, param, grad):\n        \"\"\"Returns the initial state for given parameter and gradient.\n\n        Default implementation delegates the procedure to\n        :meth:`init_state_cpu` or :meth:`init_state_gpu` depending on the type\n        of ``param``.\n\n        Args:\n            param: Parameter array.\n            grad: Gradient array corresponding to ``param``.\n\n        Returns:\n            Initial state value.\n\n            .. warning::\n\n                Note that, on every call of :meth:`update_one`, the state value\n                is passed by value and then the method updates its content, so\n                the state must be a reference. Especiallly, one cannot use a\n                value of built-in numeric type. If the state is one scalar\n                value, it is recommended to use scalar array, i.e.\n                :class:`~numpy.ndarray` with shape ``()``.\n\n        \"\"\"\n        if isinstance(param, cuda.GPUArray):\n            return self.init_state_gpu(param, grad)\n        return self.init_state_cpu(param, grad)\n\n    def init_state_cpu(self, param, grad):\n        \"\"\"Returns the initial state for given parameter and gradient on GPU.\n\n        Args:\n            param (~numpy.ndarray): Parameter array.\n            grad  (~numpy.ndarray): Gradient array.\n\n        Returns:\n            Initial state value.\n\n        .. seealso:: :meth:`init_state`, :meth:`init_state_gpu`\n\n        \"\"\"\n        return None\n\n    def init_state_gpu(self, param, grad):\n        \"\"\"Returns the initial state for given parameter and gradient on CPU.\n\n        Args:\n            param (~pycuda.gpuarray.GPUArray): Parameter array.\n            grad  (~pycuda.gpuarray.GPUArray): Gradient array.\n\n        Returns:\n            Initial state value.\n\n        .. seealso:: :meth:`init_state`, :meth:`init_state_gpu`\n\n        \"\"\"\n        return None\n\n    def zero_grads(self):\n        \"\"\"Fills all gradient arrays by zeros.\n\n        This method should be call before backprop takes place, since\n        gradients are accumulated on backprop.\n\n        \"\"\"\n        for _, g, _ in self.tuples:\n            if isinstance(g, cuda.GPUArray):\n                with cuda.using_device(g):\n                    g.fill(0)\n            else:\n                g.fill(0)\n\n    def compute_grads_norm(self):\n        \"\"\"Computes the norm of whole gradients.\n\n        Returns:\n            float: L2 norm of whole gradients, i.e. square root of sum of\n            square of all gradient elements.\n\n        .. warning::\n\n            This method returns a CPU-computed value, which means that this\n            method synchronizes between CPU and GPU if at least one of the\n            gradients reside on the GPU.\n\n        \"\"\"\n        # TODO(beam2d): Make it asynchronous to CPU when gradients exist on GPU\n        sqnorm = 0\n        for _, g, _ in self.tuples:\n            sqnorm += _sqnorm(g)\n        return math.sqrt(sqnorm)\n\n    def clip_grads(self, maxnorm):\n        \"\"\"Clips the norm of whole gradients up to given threshold.\n\n        Args:\n            maxnorm (float): Threshold of gradient L2 norm.\n\n        .. seealso::\n\n            :meth:`compute_grads_norm`\n                It uses this method to compute the gradient norm to be clipped.\n\n        \"\"\"\n        norm = self.compute_grads_norm()\n        if norm > maxnorm:\n            ratio = maxnorm / norm\n            for _, g, _ in self.tuples:\n                with cuda.using_device(g):\n                    g *= ratio\n\n    def weight_decay(self, decay):\n        \"\"\"Applies weight decay to the parameter/gradient pairs.\n\n        Args:\n            decay (float): Coefficient of weight decay\n\n        \"\"\"\n        for p, g, _ in self.tuples:\n            if isinstance(p, cuda.GPUArray):\n                with cuda.using_device(p):\n                    cuda.elementwise('float* g, const float* p, float decay',\n                                     'g[i] -= decay * p[i]',\n                                     'weight_decay')(g, p, decay)\n            else:\n                g -= decay * p\n\n    def accumulate_grads(self, grads):\n        \"\"\"Accumulates gradients from other source.\n\n        This method just adds given gradient arrays to gradients that this\n        optimizer holds. It is typically used in data-parallel optimization,\n        where gradients for different shards are computed in parallel and\n        aggregated by this method. This method correctly treats multiple GPU\n        devices.\n\n        Args:\n            grads (Iterable): Iterable of gradient arrays to be accumulated.\n\n        \"\"\"\n        for (_, g_dst, _), g_src in zip(self.tuples, grads):\n            if isinstance(g_dst, numpy.ndarray):\n                g_dst += cuda.to_cpu(g_src)\n                continue\n\n            with cuda.using_device(g_dst):\n                if (isinstance(g_src, cuda.GPUArray) and\n                        g_dst.gpudata.device != g_src.gpudata.device):\n                    g_dst += cuda.copy(g_src, out_device=g_src.gpudata.device)\n                else:\n                    g_dst += cuda.to_gpu(g_src)\n\n    def update(self):\n        \"\"\"Updates all parameters and states using corresponding gradients.\n\n        This method iteratively calls :meth:`update_one` for each parameter/\n        gradient/state tuple. Beforehand, :attr:`t` attribute is incremented.\n\n        \"\"\"\n        self.t += 1\n        for p, g, s in self.tuples:\n            with cuda.using_device(p):\n                self.update_one(p, g, s)\n\n    def update_one(self, param, grad, state):\n        \"\"\"Updates a parameter array and its state using given gradient.\n\n        The default implementation delegates the procedure to\n        :meth:`update_one_cpu` or :meth:`update_one_gpu` depending on the type\n        of the parameter array. Optimizer implmentation must override these\n        type-specific methods or this :meth:`update_one` method directly.\n\n        Args:\n            param: Parameter array.\n            grad:  Gradient array.\n            state: State value.\n\n        .. seealso:: :meth:`update_one_cpu`, :meth:`update_one_gpu`\n\n        \"\"\"\n        if isinstance(param, cuda.GPUArray):\n            self.update_one_gpu(param, grad, state)\n        else:\n            self.update_one_cpu(param, grad, state)\n\n    def update_one_cpu(self, param, grad, state):\n        \"\"\"Updates a parameter array and its state using given gradient on CPU.\n\n        Args:\n            param (~numpy.ndarray): Parameter array.\n            grad  (~numpy.ndarray): Gradient array.\n            state: State value.\n\n        .. seealso:: :meth:`update_one`, :meth:`update_one_gpu`\n\n        \"\"\"\n        raise NotImplementedError()\n\n    def update_one_gpu(self, param, grad, state):\n        \"\"\"Updates a parameter array and its state using given gradient on GPU.\n\n        Args:\n            param (~pycuda.gpuarray.GPUArray): Parameter array.\n            grad  (~pycuda.gpuarray.GPUArray): Gradient array.\n            state: State value.\n\n        .. seealso:: :meth:`update_one`, :meth:`update_one_cpu`\n\n        \"\"\"\n        raise NotImplementedError()\n", "path": "chainer/optimizer.py"}], "after_files": [{"content": "import math\n\nimport numpy\n\nfrom chainer import cuda\n\n\n# TODO(delta2323): Make it public function and move it to common directory.\n\n\ndef _sqnorm(x):\n    if isinstance(x, cuda.GPUArray):\n        with cuda.using_device(x):\n            return float(cuda.gpuarray.dot(x, x).get())\n    x = x.ravel()\n    return float(x.dot(x))\n\n\nclass Optimizer(object):\n\n    \"\"\"Base class of all numerical optimizers.\n\n    Optimizer is set up with references to parameters and gradients, and\n    then on every call of :meth:`update`, it updates parameters based on\n    corresponding gradients. Optimizer implementations must override\n    :meth:`update_one` method, which updates one parameter array using the\n    corresponding gradient array.\n\n    Optimizer can optionally use state for each parameter/gradient pair. It is\n    initialized by :meth:`init_state` method at set up.\n\n    Attributes:\n        t (int): Number of update steps. It can be used in :meth:`update_one`\n            implementation, where :attr:`t` is incremented beforehand.\n\n    \"\"\"\n\n    def setup(self, params_grads):\n        \"\"\"Prepares states for all given parameter/gradient pairs.\n\n        Args:\n            params_grads: Tuple (pair) of two tuples. The first element is a\n                tuple of parameter arrays, and the second is a tuple of\n                corresponding gradient arrays.\n                Return value of :meth:`FunctionSet.collect_parameters` method\n                can be used.\n\n        \"\"\"\n        self.t = 0\n        self.tuples = []\n        for p, g in zip(*params_grads):\n            with cuda.using_device(p):\n                state = self.init_state(p, g)\n                self.tuples.append((p, g, state))\n\n    def init_state(self, param, grad):\n        \"\"\"Returns the initial state for given parameter and gradient.\n\n        Default implementation delegates the procedure to\n        :meth:`init_state_cpu` or :meth:`init_state_gpu` depending on the type\n        of ``param``.\n\n        Args:\n            param: Parameter array.\n            grad: Gradient array corresponding to ``param``.\n\n        Returns:\n            Initial state value.\n\n            .. warning::\n\n                Note that, on every call of :meth:`update_one`, the state value\n                is passed by value and then the method updates its content, so\n                the state must be a reference. Especiallly, one cannot use a\n                value of built-in numeric type. If the state is one scalar\n                value, it is recommended to use scalar array, i.e.\n                :class:`~numpy.ndarray` with shape ``()``.\n\n        \"\"\"\n        if isinstance(param, cuda.GPUArray):\n            return self.init_state_gpu(param, grad)\n        return self.init_state_cpu(param, grad)\n\n    def init_state_cpu(self, param, grad):\n        \"\"\"Returns the initial state for given parameter and gradient on GPU.\n\n        Args:\n            param (~numpy.ndarray): Parameter array.\n            grad  (~numpy.ndarray): Gradient array.\n\n        Returns:\n            Initial state value.\n\n        .. seealso:: :meth:`init_state`, :meth:`init_state_gpu`\n\n        \"\"\"\n        return None\n\n    def init_state_gpu(self, param, grad):\n        \"\"\"Returns the initial state for given parameter and gradient on CPU.\n\n        Args:\n            param (~pycuda.gpuarray.GPUArray): Parameter array.\n            grad  (~pycuda.gpuarray.GPUArray): Gradient array.\n\n        Returns:\n            Initial state value.\n\n        .. seealso:: :meth:`init_state`, :meth:`init_state_gpu`\n\n        \"\"\"\n        return None\n\n    def zero_grads(self):\n        \"\"\"Fills all gradient arrays by zeros.\n\n        This method should be call before backprop takes place, since\n        gradients are accumulated on backprop.\n\n        \"\"\"\n        for _, g, _ in self.tuples:\n            if isinstance(g, cuda.GPUArray):\n                with cuda.using_device(g):\n                    g.fill(0)\n            else:\n                g.fill(0)\n\n    def compute_grads_norm(self):\n        \"\"\"Computes the norm of whole gradients.\n\n        Returns:\n            float: L2 norm of whole gradients, i.e. square root of sum of\n            square of all gradient elements.\n\n        .. warning::\n\n            This method returns a CPU-computed value, which means that this\n            method synchronizes between CPU and GPU if at least one of the\n            gradients reside on the GPU.\n\n        \"\"\"\n        # TODO(beam2d): Make it asynchronous to CPU when gradients exist on GPU\n        sqnorm = 0\n        for _, g, _ in self.tuples:\n            sqnorm += _sqnorm(g)\n        return math.sqrt(sqnorm)\n\n    def clip_grads(self, maxnorm):\n        \"\"\"Clips the norm of whole gradients up to given threshold.\n\n        Args:\n            maxnorm (float): Threshold of gradient L2 norm.\n\n        .. seealso::\n\n            :meth:`compute_grads_norm`\n                It uses this method to compute the gradient norm to be clipped.\n\n        \"\"\"\n        norm = self.compute_grads_norm()\n        if norm > maxnorm:\n            ratio = maxnorm / norm\n            for _, g, _ in self.tuples:\n                with cuda.using_device(g):\n                    g *= ratio\n\n    def weight_decay(self, decay):\n        \"\"\"Applies weight decay to the parameter/gradient pairs.\n\n        Args:\n            decay (float): Coefficient of weight decay\n\n        \"\"\"\n        for p, g, _ in self.tuples:\n            if isinstance(p, cuda.GPUArray):\n                with cuda.using_device(p):\n                    cuda.elementwise('float* g, const float* p, float decay',\n                                     'g[i] += decay * p[i]',\n                                     'weight_decay')(g, p, decay)\n            else:\n                g += decay * p\n\n    def accumulate_grads(self, grads):\n        \"\"\"Accumulates gradients from other source.\n\n        This method just adds given gradient arrays to gradients that this\n        optimizer holds. It is typically used in data-parallel optimization,\n        where gradients for different shards are computed in parallel and\n        aggregated by this method. This method correctly treats multiple GPU\n        devices.\n\n        Args:\n            grads (Iterable): Iterable of gradient arrays to be accumulated.\n\n        \"\"\"\n        for (_, g_dst, _), g_src in zip(self.tuples, grads):\n            if isinstance(g_dst, numpy.ndarray):\n                g_dst += cuda.to_cpu(g_src)\n                continue\n\n            with cuda.using_device(g_dst):\n                if (isinstance(g_src, cuda.GPUArray) and\n                        g_dst.gpudata.device != g_src.gpudata.device):\n                    g_dst += cuda.copy(g_src, out_device=g_src.gpudata.device)\n                else:\n                    g_dst += cuda.to_gpu(g_src)\n\n    def update(self):\n        \"\"\"Updates all parameters and states using corresponding gradients.\n\n        This method iteratively calls :meth:`update_one` for each parameter/\n        gradient/state tuple. Beforehand, :attr:`t` attribute is incremented.\n\n        \"\"\"\n        self.t += 1\n        for p, g, s in self.tuples:\n            with cuda.using_device(p):\n                self.update_one(p, g, s)\n\n    def update_one(self, param, grad, state):\n        \"\"\"Updates a parameter array and its state using given gradient.\n\n        The default implementation delegates the procedure to\n        :meth:`update_one_cpu` or :meth:`update_one_gpu` depending on the type\n        of the parameter array. Optimizer implmentation must override these\n        type-specific methods or this :meth:`update_one` method directly.\n\n        Args:\n            param: Parameter array.\n            grad:  Gradient array.\n            state: State value.\n\n        .. seealso:: :meth:`update_one_cpu`, :meth:`update_one_gpu`\n\n        \"\"\"\n        if isinstance(param, cuda.GPUArray):\n            self.update_one_gpu(param, grad, state)\n        else:\n            self.update_one_cpu(param, grad, state)\n\n    def update_one_cpu(self, param, grad, state):\n        \"\"\"Updates a parameter array and its state using given gradient on CPU.\n\n        Args:\n            param (~numpy.ndarray): Parameter array.\n            grad  (~numpy.ndarray): Gradient array.\n            state: State value.\n\n        .. seealso:: :meth:`update_one`, :meth:`update_one_gpu`\n\n        \"\"\"\n        raise NotImplementedError()\n\n    def update_one_gpu(self, param, grad, state):\n        \"\"\"Updates a parameter array and its state using given gradient on GPU.\n\n        Args:\n            param (~pycuda.gpuarray.GPUArray): Parameter array.\n            grad  (~pycuda.gpuarray.GPUArray): Gradient array.\n            state: State value.\n\n        .. seealso:: :meth:`update_one`, :meth:`update_one_cpu`\n\n        \"\"\"\n        raise NotImplementedError()\n", "path": "chainer/optimizer.py"}]}