We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[Bug] get_fantasy_model does not work for SGPR with InducingPointKernel
# 🐛 Bug

Not sure if this should be considered a bug or a feature request, but gpytorch's implementation of SGPR using the InducingPointKernel kernel seems to not support get_fantasy_model.

## To reproduce
I am including the smallest mwe (or should I say mnwe) here. Note that I get the same behaviour by taking the [example tutorial for SGPR](https://gpytorch.readthedocs.io/en/latest/examples/05_Scalable_GP_Regression_Multidimensional/SGPR_Example_CUDA.html) and add a get_fantasy_model added at the end. I can post that too if required, but it is longer and might clutter the ticket.

**Code snippet to reproduce**
```python
import gpytorch
import torch
from gpytorch.kernels import ScaleKernel, RBFKernel, InducingPointKernel
from gpytorch.distributions import MultivariateNormal
from gpytorch.likelihoods import GaussianLikelihood
from gpytorch.means import ConstantMean

class GPRegressionModel(gpytorch.models.ExactGP):
    def __init__(self, train_x, train_y, likelihood):
        super(GPRegressionModel, self).__init__(train_x, train_y, likelihood)
        self.mean_module = ConstantMean()
        self.base_covar_module = ScaleKernel(RBFKernel())
        self.covar_module = InducingPointKernel(self.base_covar_module, inducing_points=train_x[:500, :], likelihood=likelihood)

    def forward(self, x):
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        return MultivariateNormal(mean_x, covar_x)

train_X = torch.randn((100,5)).to("cpu")
train_y = torch.randn((100)).to("cpu")

likelihood = GaussianLikelihood()

model = GPRegressionModel(train_X, train_y, likelihood)
model.train()
model.eval()

test_pred = model(torch.randn((1,5)).to("cpu"))

model = model.get_fantasy_model(torch.randn((1,5)).to("cpu"), torch.randn((1)).to("cpu"))
```

**Stack trace/error message**
```
Traceback (most recent call last):
  File "mwe_sgpr_fantasy.py", line 31, in <module>
    model = model.get_fantasy_model(torch.randn((1,5)).to("cpu"), torch.randn((1)).to("cpu"))
  File "/home/user/miniconda3/lib/python3.7/site-packages/gpytorch/models/exact_gp.py", line 173, in get_fantasy_model
    new_model = deepcopy(self)
  File "/home/user/miniconda3/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/user/miniconda3/lib/python3.7/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/user/miniconda3/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/user/miniconda3/lib/python3.7/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/user/miniconda3/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/user/miniconda3/lib/python3.7/copy.py", line 306, in _reconstruct
    value = deepcopy(value, memo)
  File "/home/user/miniconda3/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/user/miniconda3/lib/python3.7/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/user/miniconda3/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/user/miniconda3/lib/python3.7/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/user/miniconda3/lib/python3.7/copy.py", line 161, in deepcopy
    y = copier(memo)
  File "/home/user/miniconda3/lib/python3.7/site-packages/torch/tensor.py", line 23, in __deepcopy__
    raise RuntimeError("Only Tensors created explicitly by the user "
RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment
```

## Expected Behavior

I would expect a fantasized model to be returned efficiently.

## System information

**Please complete the following information:**
- GPyTorch Version 0.3.3


- PyTorch Version 1.1.0
- Ubuntu 18.04

## Additional context
It seems that during the update, the `new_model = deepcopy(self)` tries to copy `self._inducing_inv_root` but detects that it is trainable by autograd and balks. I guess gpytorch made this design choice because of the goal of optimizing the inducing points as a hyperparameter, but as a tradeoff it does not allow for efficient updates.

So far I tried to replace the inducing points with a non-trainable version by setting `requires_grad` to `False`, but it seems to not help. I would guess that [any of these tensors multiplications](https://github.com/cornellius-gp/gpytorch/blob/master/gpytorch/kernels/inducing_point_kernel.py#L45-L47) in the implementation of `_inducing_inv_root` could end up reactivating autograd, and I am afraid that without more knowledge of gpytorch's internals patching them one-by-one might end up in a long whack-a-mole.


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `gpytorch/kernels/inducing_point_kernel.py`
Content:
```
1 #!/usr/bin/env python3
2 
3 import math
4 import torch
5 from .kernel import Kernel
6 from ..lazy import delazify, DiagLazyTensor, MatmulLazyTensor, RootLazyTensor, PsdSumLazyTensor
7 from ..distributions import MultivariateNormal
8 from ..mlls import InducingPointKernelAddedLossTerm
9 from ..utils.cholesky import psd_safe_cholesky
10 
11 
12 class InducingPointKernel(Kernel):
13     def __init__(self, base_kernel, inducing_points, likelihood, active_dims=None):
14         super(InducingPointKernel, self).__init__(active_dims=active_dims)
15         self.base_kernel = base_kernel
16         self.likelihood = likelihood
17 
18         if inducing_points.ndimension() == 1:
19             inducing_points = inducing_points.unsqueeze(-1)
20         if inducing_points.ndimension() != 2:
21             raise RuntimeError("Inducing points should be 2 dimensional")
22         self.register_parameter(name="inducing_points", parameter=torch.nn.Parameter(inducing_points))
23         self.register_added_loss_term("inducing_point_loss_term")
24 
25     def train(self, mode=True):
26         if hasattr(self, "_cached_kernel_mat"):
27             del self._cached_kernel_mat
28         return super(InducingPointKernel, self).train(mode)
29 
30     @property
31     def _inducing_mat(self):
32         if not self.training and hasattr(self, "_cached_kernel_mat"):
33             return self._cached_kernel_mat
34         else:
35             res = delazify(self.base_kernel(self.inducing_points, self.inducing_points))
36             if not self.training:
37                 self._cached_kernel_mat = res
38             return res
39 
40     @property
41     def _inducing_inv_root(self):
42         if not self.training and hasattr(self, "_cached_kernel_inv_root"):
43             return self._cached_kernel_inv_root
44         else:
45             chol = psd_safe_cholesky(self._inducing_mat, upper=True)
46             eye = torch.eye(chol.size(-1), device=chol.device, dtype=chol.dtype)
47             inv_root = torch.triangular_solve(eye, chol)[0]
48 
49             res = inv_root
50             if not self.training:
51                 self._cached_kernel_inv_root = res
52             return res
53 
54     def _get_covariance(self, x1, x2):
55         k_ux1 = delazify(self.base_kernel(x1, self.inducing_points))
56         if torch.equal(x1, x2):
57             covar = RootLazyTensor(k_ux1.matmul(self._inducing_inv_root))
58 
59             # Diagonal correction for predictive posterior
60             correction = (self.base_kernel(x1, x2, diag=True) - covar.diag()).clamp(0, math.inf)
61             covar = PsdSumLazyTensor(covar, DiagLazyTensor(correction))
62         else:
63             k_ux2 = delazify(self.base_kernel(x2, self.inducing_points))
64             covar = MatmulLazyTensor(
65                 k_ux1.matmul(self._inducing_inv_root), k_ux2.matmul(self._inducing_inv_root).transpose(-1, -2)
66             )
67 
68         return covar
69 
70     def _covar_diag(self, inputs):
71         if inputs.ndimension() == 1:
72             inputs = inputs.unsqueeze(1)
73 
74         # Get diagonal of covar
75         covar_diag = delazify(self.base_kernel(inputs, diag=True))
76         return DiagLazyTensor(covar_diag)
77 
78     def forward(self, x1, x2, diag=False, **kwargs):
79         covar = self._get_covariance(x1, x2)
80 
81         if self.training:
82             if not torch.equal(x1, x2):
83                 raise RuntimeError("x1 should equal x2 in training mode")
84             zero_mean = torch.zeros_like(x1.select(-1, 0))
85             new_added_loss_term = InducingPointKernelAddedLossTerm(
86                 MultivariateNormal(zero_mean, self._covar_diag(x1)),
87                 MultivariateNormal(zero_mean, covar),
88                 self.likelihood,
89             )
90             self.update_added_loss_term("inducing_point_loss_term", new_added_loss_term)
91 
92         if diag:
93             return covar.diag()
94         else:
95             return covar
96 
97     def num_outputs_per_input(self, x1, x2):
98         return self.base_kernel.num_outputs_per_input(x1, x2)
99
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/gpytorch/kernels/inducing_point_kernel.py b/gpytorch/kernels/inducing_point_kernel.py
--- a/gpytorch/kernels/inducing_point_kernel.py
+++ b/gpytorch/kernels/inducing_point_kernel.py
@@ -2,6 +2,7 @@
 
 import math
 import torch
+import copy
 from .kernel import Kernel
 from ..lazy import delazify, DiagLazyTensor, MatmulLazyTensor, RootLazyTensor, PsdSumLazyTensor
 from ..distributions import MultivariateNormal
@@ -96,3 +97,33 @@
 
     def num_outputs_per_input(self, x1, x2):
         return self.base_kernel.num_outputs_per_input(x1, x2)
+
+    def __deepcopy__(self, memo):
+        replace_inv_root = False
+        replace_kernel_mat = False
+
+        if hasattr(self, "_cached_kernel_inv_root"):
+            replace_inv_root = True
+            kernel_inv_root = self._cached_kernel_inv_root
+            self._cached_kernel_inv_root = None
+        if hasattr(self, "_cached_kernel_mat"):
+            replace_kernel_mat = True
+            kernel_mat = self._cached_kernel_mat
+            self._cached_kernel_mat = None
+
+        deepcopy_method = self.__deepcopy__
+        self.__deepcopy__ = None
+        cp = copy.deepcopy(self, memo)
+
+        self.__deepcopy__ = deepcopy_method
+        cp.__deepcopy__ = deepcopy_method
+
+        if replace_inv_root:
+            self._cached_kernel_inv_root = kernel_inv_root
+            cp._cached_kernel_inv_root = kernel_inv_root
+
+        if replace_kernel_mat:
+            self._cached_kernel_mat = kernel_mat
+            cp._cached_kernel_mat = kernel_mat
+
+        return cp

{"golden_diff": "diff --git a/gpytorch/kernels/inducing_point_kernel.py b/gpytorch/kernels/inducing_point_kernel.py\n--- a/gpytorch/kernels/inducing_point_kernel.py\n+++ b/gpytorch/kernels/inducing_point_kernel.py\n@@ -2,6 +2,7 @@\n \n import math\n import torch\n+import copy\n from .kernel import Kernel\n from ..lazy import delazify, DiagLazyTensor, MatmulLazyTensor, RootLazyTensor, PsdSumLazyTensor\n from ..distributions import MultivariateNormal\n@@ -96,3 +97,33 @@\n \n     def num_outputs_per_input(self, x1, x2):\n         return self.base_kernel.num_outputs_per_input(x1, x2)\n+\n+    def __deepcopy__(self, memo):\n+        replace_inv_root = False\n+        replace_kernel_mat = False\n+\n+        if hasattr(self, \"_cached_kernel_inv_root\"):\n+            replace_inv_root = True\n+            kernel_inv_root = self._cached_kernel_inv_root\n+            self._cached_kernel_inv_root = None\n+        if hasattr(self, \"_cached_kernel_mat\"):\n+            replace_kernel_mat = True\n+            kernel_mat = self._cached_kernel_mat\n+            self._cached_kernel_mat = None\n+\n+        deepcopy_method = self.__deepcopy__\n+        self.__deepcopy__ = None\n+        cp = copy.deepcopy(self, memo)\n+\n+        self.__deepcopy__ = deepcopy_method\n+        cp.__deepcopy__ = deepcopy_method\n+\n+        if replace_inv_root:\n+            self._cached_kernel_inv_root = kernel_inv_root\n+            cp._cached_kernel_inv_root = kernel_inv_root\n+\n+        if replace_kernel_mat:\n+            self._cached_kernel_mat = kernel_mat\n+            cp._cached_kernel_mat = kernel_mat\n+\n+        return cp\n", "issue": "[Bug] get_fantasy_model does not work for SGPR with InducingPointKernel\n# \ud83d\udc1b Bug\r\n\r\nNot sure if this should be considered a bug or a feature request, but gpytorch's implementation of SGPR using the InducingPointKernel kernel seems to not support get_fantasy_model.\r\n\r\n## To reproduce\r\nI am including the smallest mwe (or should I say mnwe) here. Note that I get the same behaviour by taking the [example tutorial for SGPR](https://gpytorch.readthedocs.io/en/latest/examples/05_Scalable_GP_Regression_Multidimensional/SGPR_Example_CUDA.html) and add a get_fantasy_model added at the end. I can post that too if required, but it is longer and might clutter the ticket.\r\n\r\n**Code snippet to reproduce**\r\n```python\r\nimport gpytorch\r\nimport torch\r\nfrom gpytorch.kernels import ScaleKernel, RBFKernel, InducingPointKernel\r\nfrom gpytorch.distributions import MultivariateNormal\r\nfrom gpytorch.likelihoods import GaussianLikelihood\r\nfrom gpytorch.means import ConstantMean\r\n\r\nclass GPRegressionModel(gpytorch.models.ExactGP):\r\n    def __init__(self, train_x, train_y, likelihood):\r\n        super(GPRegressionModel, self).__init__(train_x, train_y, likelihood)\r\n        self.mean_module = ConstantMean()\r\n        self.base_covar_module = ScaleKernel(RBFKernel())\r\n        self.covar_module = InducingPointKernel(self.base_covar_module, inducing_points=train_x[:500, :], likelihood=likelihood)\r\n\r\n    def forward(self, x):\r\n        mean_x = self.mean_module(x)\r\n        covar_x = self.covar_module(x)\r\n        return MultivariateNormal(mean_x, covar_x)\r\n\r\ntrain_X = torch.randn((100,5)).to(\"cpu\")\r\ntrain_y = torch.randn((100)).to(\"cpu\")\r\n\r\nlikelihood = GaussianLikelihood()\r\n\r\nmodel = GPRegressionModel(train_X, train_y, likelihood)\r\nmodel.train()\r\nmodel.eval()\r\n\r\ntest_pred = model(torch.randn((1,5)).to(\"cpu\"))\r\n\r\nmodel = model.get_fantasy_model(torch.randn((1,5)).to(\"cpu\"), torch.randn((1)).to(\"cpu\"))\r\n```\r\n\r\n**Stack trace/error message**\r\n```\r\nTraceback (most recent call last):\r\n  File \"mwe_sgpr_fantasy.py\", line 31, in <module>\r\n    model = model.get_fantasy_model(torch.randn((1,5)).to(\"cpu\"), torch.randn((1)).to(\"cpu\"))\r\n  File \"/home/user/miniconda3/lib/python3.7/site-packages/gpytorch/models/exact_gp.py\", line 173, in get_fantasy_model\r\n    new_model = deepcopy(self)\r\n  File \"/home/user/miniconda3/lib/python3.7/copy.py\", line 180, in deepcopy\r\n    y = _reconstruct(x, memo, *rv)\r\n  File \"/home/user/miniconda3/lib/python3.7/copy.py\", line 280, in _reconstruct\r\n    state = deepcopy(state, memo)\r\n  File \"/home/user/miniconda3/lib/python3.7/copy.py\", line 150, in deepcopy\r\n    y = copier(x, memo)\r\n  File \"/home/user/miniconda3/lib/python3.7/copy.py\", line 240, in _deepcopy_dict\r\n    y[deepcopy(key, memo)] = deepcopy(value, memo)\r\n  File \"/home/user/miniconda3/lib/python3.7/copy.py\", line 180, in deepcopy\r\n    y = _reconstruct(x, memo, *rv)\r\n  File \"/home/user/miniconda3/lib/python3.7/copy.py\", line 306, in _reconstruct\r\n    value = deepcopy(value, memo)\r\n  File \"/home/user/miniconda3/lib/python3.7/copy.py\", line 180, in deepcopy\r\n    y = _reconstruct(x, memo, *rv)\r\n  File \"/home/user/miniconda3/lib/python3.7/copy.py\", line 280, in _reconstruct\r\n    state = deepcopy(state, memo)\r\n  File \"/home/user/miniconda3/lib/python3.7/copy.py\", line 150, in deepcopy\r\n    y = copier(x, memo)\r\n  File \"/home/user/miniconda3/lib/python3.7/copy.py\", line 240, in _deepcopy_dict\r\n    y[deepcopy(key, memo)] = deepcopy(value, memo)\r\n  File \"/home/user/miniconda3/lib/python3.7/copy.py\", line 161, in deepcopy\r\n    y = copier(memo)\r\n  File \"/home/user/miniconda3/lib/python3.7/site-packages/torch/tensor.py\", line 23, in __deepcopy__\r\n    raise RuntimeError(\"Only Tensors created explicitly by the user \"\r\nRuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment\r\n```\r\n\r\n## Expected Behavior\r\n\r\nI would expect a fantasized model to be returned efficiently.\r\n\r\n## System information\r\n\r\n**Please complete the following information:**\r\n- GPyTorch Version 0.3.3\r\n\r\n\r\n- PyTorch Version 1.1.0\r\n- Ubuntu 18.04\r\n\r\n## Additional context\r\nIt seems that during the update, the `new_model = deepcopy(self)` tries to copy `self._inducing_inv_root` but detects that it is trainable by autograd and balks. I guess gpytorch made this design choice because of the goal of optimizing the inducing points as a hyperparameter, but as a tradeoff it does not allow for efficient updates.\r\n\r\nSo far I tried to replace the inducing points with a non-trainable version by setting `requires_grad` to `False`, but it seems to not help. I would guess that [any of these tensors multiplications](https://github.com/cornellius-gp/gpytorch/blob/master/gpytorch/kernels/inducing_point_kernel.py#L45-L47) in the implementation of `_inducing_inv_root` could end up reactivating autograd, and I am afraid that without more knowledge of gpytorch's internals patching them one-by-one might end up in a long whack-a-mole.\r\n\n", "before_files": [{"content": "#!/usr/bin/env python3\n\nimport math\nimport torch\nfrom .kernel import Kernel\nfrom ..lazy import delazify, DiagLazyTensor, MatmulLazyTensor, RootLazyTensor, PsdSumLazyTensor\nfrom ..distributions import MultivariateNormal\nfrom ..mlls import InducingPointKernelAddedLossTerm\nfrom ..utils.cholesky import psd_safe_cholesky\n\n\nclass InducingPointKernel(Kernel):\n    def __init__(self, base_kernel, inducing_points, likelihood, active_dims=None):\n        super(InducingPointKernel, self).__init__(active_dims=active_dims)\n        self.base_kernel = base_kernel\n        self.likelihood = likelihood\n\n        if inducing_points.ndimension() == 1:\n            inducing_points = inducing_points.unsqueeze(-1)\n        if inducing_points.ndimension() != 2:\n            raise RuntimeError(\"Inducing points should be 2 dimensional\")\n        self.register_parameter(name=\"inducing_points\", parameter=torch.nn.Parameter(inducing_points))\n        self.register_added_loss_term(\"inducing_point_loss_term\")\n\n    def train(self, mode=True):\n        if hasattr(self, \"_cached_kernel_mat\"):\n            del self._cached_kernel_mat\n        return super(InducingPointKernel, self).train(mode)\n\n    @property\n    def _inducing_mat(self):\n        if not self.training and hasattr(self, \"_cached_kernel_mat\"):\n            return self._cached_kernel_mat\n        else:\n            res = delazify(self.base_kernel(self.inducing_points, self.inducing_points))\n            if not self.training:\n                self._cached_kernel_mat = res\n            return res\n\n    @property\n    def _inducing_inv_root(self):\n        if not self.training and hasattr(self, \"_cached_kernel_inv_root\"):\n            return self._cached_kernel_inv_root\n        else:\n            chol = psd_safe_cholesky(self._inducing_mat, upper=True)\n            eye = torch.eye(chol.size(-1), device=chol.device, dtype=chol.dtype)\n            inv_root = torch.triangular_solve(eye, chol)[0]\n\n            res = inv_root\n            if not self.training:\n                self._cached_kernel_inv_root = res\n            return res\n\n    def _get_covariance(self, x1, x2):\n        k_ux1 = delazify(self.base_kernel(x1, self.inducing_points))\n        if torch.equal(x1, x2):\n            covar = RootLazyTensor(k_ux1.matmul(self._inducing_inv_root))\n\n            # Diagonal correction for predictive posterior\n            correction = (self.base_kernel(x1, x2, diag=True) - covar.diag()).clamp(0, math.inf)\n            covar = PsdSumLazyTensor(covar, DiagLazyTensor(correction))\n        else:\n            k_ux2 = delazify(self.base_kernel(x2, self.inducing_points))\n            covar = MatmulLazyTensor(\n                k_ux1.matmul(self._inducing_inv_root), k_ux2.matmul(self._inducing_inv_root).transpose(-1, -2)\n            )\n\n        return covar\n\n    def _covar_diag(self, inputs):\n        if inputs.ndimension() == 1:\n            inputs = inputs.unsqueeze(1)\n\n        # Get diagonal of covar\n        covar_diag = delazify(self.base_kernel(inputs, diag=True))\n        return DiagLazyTensor(covar_diag)\n\n    def forward(self, x1, x2, diag=False, **kwargs):\n        covar = self._get_covariance(x1, x2)\n\n        if self.training:\n            if not torch.equal(x1, x2):\n                raise RuntimeError(\"x1 should equal x2 in training mode\")\n            zero_mean = torch.zeros_like(x1.select(-1, 0))\n            new_added_loss_term = InducingPointKernelAddedLossTerm(\n                MultivariateNormal(zero_mean, self._covar_diag(x1)),\n                MultivariateNormal(zero_mean, covar),\n                self.likelihood,\n            )\n            self.update_added_loss_term(\"inducing_point_loss_term\", new_added_loss_term)\n\n        if diag:\n            return covar.diag()\n        else:\n            return covar\n\n    def num_outputs_per_input(self, x1, x2):\n        return self.base_kernel.num_outputs_per_input(x1, x2)\n", "path": "gpytorch/kernels/inducing_point_kernel.py"}], "after_files": [{"content": "#!/usr/bin/env python3\n\nimport math\nimport torch\nimport copy\nfrom .kernel import Kernel\nfrom ..lazy import delazify, DiagLazyTensor, MatmulLazyTensor, RootLazyTensor, PsdSumLazyTensor\nfrom ..distributions import MultivariateNormal\nfrom ..mlls import InducingPointKernelAddedLossTerm\nfrom ..utils.cholesky import psd_safe_cholesky\n\n\nclass InducingPointKernel(Kernel):\n    def __init__(self, base_kernel, inducing_points, likelihood, active_dims=None):\n        super(InducingPointKernel, self).__init__(active_dims=active_dims)\n        self.base_kernel = base_kernel\n        self.likelihood = likelihood\n\n        if inducing_points.ndimension() == 1:\n            inducing_points = inducing_points.unsqueeze(-1)\n        if inducing_points.ndimension() != 2:\n            raise RuntimeError(\"Inducing points should be 2 dimensional\")\n        self.register_parameter(name=\"inducing_points\", parameter=torch.nn.Parameter(inducing_points))\n        self.register_added_loss_term(\"inducing_point_loss_term\")\n\n    def train(self, mode=True):\n        if hasattr(self, \"_cached_kernel_mat\"):\n            del self._cached_kernel_mat\n        return super(InducingPointKernel, self).train(mode)\n\n    @property\n    def _inducing_mat(self):\n        if not self.training and hasattr(self, \"_cached_kernel_mat\"):\n            return self._cached_kernel_mat\n        else:\n            res = delazify(self.base_kernel(self.inducing_points, self.inducing_points))\n            if not self.training:\n                self._cached_kernel_mat = res\n            return res\n\n    @property\n    def _inducing_inv_root(self):\n        if not self.training and hasattr(self, \"_cached_kernel_inv_root\"):\n            return self._cached_kernel_inv_root\n        else:\n            chol = psd_safe_cholesky(self._inducing_mat, upper=True)\n            eye = torch.eye(chol.size(-1), device=chol.device, dtype=chol.dtype)\n            inv_root = torch.triangular_solve(eye, chol)[0]\n\n            res = inv_root\n            if not self.training:\n                self._cached_kernel_inv_root = res\n            return res\n\n    def _get_covariance(self, x1, x2):\n        k_ux1 = delazify(self.base_kernel(x1, self.inducing_points))\n        if torch.equal(x1, x2):\n            covar = RootLazyTensor(k_ux1.matmul(self._inducing_inv_root))\n\n            # Diagonal correction for predictive posterior\n            correction = (self.base_kernel(x1, x2, diag=True) - covar.diag()).clamp(0, math.inf)\n            covar = PsdSumLazyTensor(covar, DiagLazyTensor(correction))\n        else:\n            k_ux2 = delazify(self.base_kernel(x2, self.inducing_points))\n            covar = MatmulLazyTensor(\n                k_ux1.matmul(self._inducing_inv_root), k_ux2.matmul(self._inducing_inv_root).transpose(-1, -2)\n            )\n\n        return covar\n\n    def _covar_diag(self, inputs):\n        if inputs.ndimension() == 1:\n            inputs = inputs.unsqueeze(1)\n\n        # Get diagonal of covar\n        covar_diag = delazify(self.base_kernel(inputs, diag=True))\n        return DiagLazyTensor(covar_diag)\n\n    def forward(self, x1, x2, diag=False, **kwargs):\n        covar = self._get_covariance(x1, x2)\n\n        if self.training:\n            if not torch.equal(x1, x2):\n                raise RuntimeError(\"x1 should equal x2 in training mode\")\n            zero_mean = torch.zeros_like(x1.select(-1, 0))\n            new_added_loss_term = InducingPointKernelAddedLossTerm(\n                MultivariateNormal(zero_mean, self._covar_diag(x1)),\n                MultivariateNormal(zero_mean, covar),\n                self.likelihood,\n            )\n            self.update_added_loss_term(\"inducing_point_loss_term\", new_added_loss_term)\n\n        if diag:\n            return covar.diag()\n        else:\n            return covar\n\n    def num_outputs_per_input(self, x1, x2):\n        return self.base_kernel.num_outputs_per_input(x1, x2)\n\n    def __deepcopy__(self, memo):\n        replace_inv_root = False\n        replace_kernel_mat = False\n\n        if hasattr(self, \"_cached_kernel_inv_root\"):\n            replace_inv_root = True\n            kernel_inv_root = self._cached_kernel_inv_root\n            self._cached_kernel_inv_root = None\n        if hasattr(self, \"_cached_kernel_mat\"):\n            replace_kernel_mat = True\n            kernel_mat = self._cached_kernel_mat\n            self._cached_kernel_mat = None\n\n        deepcopy_method = self.__deepcopy__\n        self.__deepcopy__ = None\n        cp = copy.deepcopy(self, memo)\n\n        self.__deepcopy__ = deepcopy_method\n        cp.__deepcopy__ = deepcopy_method\n\n        if replace_inv_root:\n            self._cached_kernel_inv_root = kernel_inv_root\n            cp._cached_kernel_inv_root = kernel_inv_root\n\n        if replace_kernel_mat:\n            self._cached_kernel_mat = kernel_mat\n            cp._cached_kernel_mat = kernel_mat\n\n        return cp\n", "path": "gpytorch/kernels/inducing_point_kernel.py"}]}