We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Loading LORA weights in `diffusers` with a `peft` backend increases in latency as more paths are added to `PYTHONPATH`
### System Info

```
accelerate==0.21.0
diffusers==0.26.3
peft==0.9.0
safetensors==0.3.3
tokenizers==0.15.2
torch==2.2.1
transformers==4.36.2
```

### Who can help?

@sayakpaul 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder
- [X] My own task or dataset (give details below)

### Reproduction

```python
from diffusers import DiffusionPipeline
import time
import torch
import sys
import os
import shutil

pipe_id = "stabilityai/stable-diffusion-xl-base-1.0"
pipe = DiffusionPipeline.from_pretrained(pipe_id, torch_dtype=torch.float16).to("cuda")

loras = [
    {
        "adapter_name": "anime"
        "location": "./anime",
        "weight_name": "anime.safetensors",
        "token": "my_anime"
    },
]

def run_dynamic_lora_inference(lora):
    start_load_time = time.time()
    pipe.load_lora_weights(lora["location"], weight_name=lora["weight_name"], adapter_name=lora["adapter_name"])
    end_load_time = time.time()
    prompt = f"Illustration of a dog in the style of {lora["token"]}"

    start_fuse_time = time.time()
    pipe.fuse_lora()
    end_fuse_time = time.time()

    start_set_adapter_time = time.time()
    pipe.set_adapters(lora_name)
    end_set_adapter_time = time.time()

    start_inference_time = time.time()
    image = pipe(
        prompt, num_inference_steps=30, generator=torch.manual_seed(0)
    ).images[0]
    end_inference_time = time.time()

    start_unfuse_time = time.time()
    pipe.unfuse_lora()
    end_unfuse_time = time.time()

    start_unload_time = time.time()
    pipe.unload_lora_weights()
    end_unload_time = time.time()

    image.save(f"./{lora_name}.png")

    print("Load time:", end_load_time - start_load_time)
    print("Fuse time:", end_fuse_time - start_fuse_time)
    print("Set adapter time", end_set_adapter_time - start_set_adapter_time)
    print("Inference time:", end_inference_time - start_inference_time)
    print("Unfuse time:", end_unfuse_time - start_unfuse_time)
    print("Unload time:", end_unload_time - start_unload_time)

def add_to_python_path():
    root_path = "./folders"
    shutil.rmtree(root_path)
    os.mkdir(root_path)

    folders =  [f"folder_{x}" for x in range(0, 10000)]
    for folder in folders:
        os.mkdir(os.path.join(root_path, folder))
        sys.path.append(os.path.join(root_path, folder))

def main():
    for lora in loras:
        run_dynamic_lora_inference(lora)

main()
```

Flamegraph: 
![profile](https://github.com/huggingface/peft/assets/54301847/63b4d841-e740-4cb8-af51-96d011a8bbae)



### Expected behavior

I run a system with a somewhat large `PYTHONPATH` that we can't truncate, and we are currently blocked from upgrading `diffusers` to any version that uses `peft` for LORA inference. 

It's loosely based on this post: https://huggingface.co/blog/lora-adapters-dynamic-loading

We've observed a behavior where the time taken for `load_lora_weights` increases significantly with more paths added to `PYTHONPATH`. This can be reproduced in the example provided - with 10,000 folders added to PYTHONPATH, we get the following latencies:

```
Load time: 291.78441095352173
Fuse time: 0.12406659126281738
Set adapter time 0.06171250343322754
Inference time: 9.685987710952759
Unfuse time: 0.08063459396362305
Unload time: 0.15737533569335938
```

Benchmarking against 1, 10, 100, 1000, 10000 and 50000 entries in the `PYTHONPATH`, we get a pretty astounding increase in load latency: 

![image](https://github.com/huggingface/peft/assets/54301847/cd8894d5-3c6c-47fd-b8ca-ead34a32e36d)

Even at 100 entries, we're looking at an extra 4 seconds per `load` call which is a pretty significant increase.

We looked briefly at it and came to the conclusion that it's something to do with the way `peft` runs module imports, particularly repetitive calls to import modules, where the imports are not cached, eg, `importlib.util.find_spec` doesn't cache imports. 

Instead of this behaviour, we'd expect that `load_lora_weights` retains a relatively constant load time, regardless of the length of our python path. 

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `src/peft/import_utils.py`
Content:
```
1 # Copyright 2023-present the HuggingFace Inc. team.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
4 # you may not use this file except in compliance with the License.
5 # You may obtain a copy of the License at
6 #
7 #     http://www.apache.org/licenses/LICENSE-2.0
8 #
9 # Unless required by applicable law or agreed to in writing, software
10 # distributed under the License is distributed on an "AS IS" BASIS,
11 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
14 import importlib
15 import importlib.metadata as importlib_metadata
16 from functools import lru_cache
17 
18 import packaging.version
19 
20 
21 def is_bnb_available() -> bool:
22     return importlib.util.find_spec("bitsandbytes") is not None
23 
24 
25 def is_bnb_4bit_available() -> bool:
26     if not is_bnb_available():
27         return False
28 
29     import bitsandbytes as bnb
30 
31     return hasattr(bnb.nn, "Linear4bit")
32 
33 
34 def is_auto_gptq_available():
35     if importlib.util.find_spec("auto_gptq") is not None:
36         AUTOGPTQ_MINIMUM_VERSION = packaging.version.parse("0.5.0")
37         version_autogptq = packaging.version.parse(importlib_metadata.version("auto_gptq"))
38         if AUTOGPTQ_MINIMUM_VERSION <= version_autogptq:
39             return True
40         else:
41             raise ImportError(
42                 f"Found an incompatible version of auto-gptq. Found version {version_autogptq}, "
43                 f"but only versions above {AUTOGPTQ_MINIMUM_VERSION} are supported"
44             )
45 
46 
47 def is_optimum_available() -> bool:
48     return importlib.util.find_spec("optimum") is not None
49 
50 
51 @lru_cache
52 def is_torch_tpu_available(check_device=True):
53     "Checks if `torch_xla` is installed and potentially if a TPU is in the environment"
54     if importlib.util.find_spec("torch_xla") is not None:
55         if check_device:
56             # We need to check if `xla_device` can be found, will raise a RuntimeError if not
57             try:
58                 import torch_xla.core.xla_model as xm
59 
60                 _ = xm.xla_device()
61                 return True
62             except RuntimeError:
63                 return False
64         return True
65     return False
66 
67 
68 def is_aqlm_available():
69     return importlib.util.find_spec("aqlm") is not None
70 
71 
72 def is_auto_awq_available():
73     return importlib.util.find_spec("awq") is not None
74
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/src/peft/import_utils.py b/src/peft/import_utils.py
--- a/src/peft/import_utils.py
+++ b/src/peft/import_utils.py
@@ -18,10 +18,12 @@
 import packaging.version
 
 
+@lru_cache
 def is_bnb_available() -> bool:
     return importlib.util.find_spec("bitsandbytes") is not None
 
 
+@lru_cache
 def is_bnb_4bit_available() -> bool:
     if not is_bnb_available():
         return False
@@ -31,6 +33,7 @@
     return hasattr(bnb.nn, "Linear4bit")
 
 
+@lru_cache
 def is_auto_gptq_available():
     if importlib.util.find_spec("auto_gptq") is not None:
         AUTOGPTQ_MINIMUM_VERSION = packaging.version.parse("0.5.0")
@@ -44,6 +47,7 @@
             )
 
 
+@lru_cache
 def is_optimum_available() -> bool:
     return importlib.util.find_spec("optimum") is not None
 
@@ -65,9 +69,11 @@
     return False
 
 
+@lru_cache
 def is_aqlm_available():
     return importlib.util.find_spec("aqlm") is not None
 
 
+@lru_cache
 def is_auto_awq_available():
     return importlib.util.find_spec("awq") is not None

{"golden_diff": "diff --git a/src/peft/import_utils.py b/src/peft/import_utils.py\n--- a/src/peft/import_utils.py\n+++ b/src/peft/import_utils.py\n@@ -18,10 +18,12 @@\n import packaging.version\n \n \n+@lru_cache\n def is_bnb_available() -> bool:\n     return importlib.util.find_spec(\"bitsandbytes\") is not None\n \n \n+@lru_cache\n def is_bnb_4bit_available() -> bool:\n     if not is_bnb_available():\n         return False\n@@ -31,6 +33,7 @@\n     return hasattr(bnb.nn, \"Linear4bit\")\n \n \n+@lru_cache\n def is_auto_gptq_available():\n     if importlib.util.find_spec(\"auto_gptq\") is not None:\n         AUTOGPTQ_MINIMUM_VERSION = packaging.version.parse(\"0.5.0\")\n@@ -44,6 +47,7 @@\n             )\n \n \n+@lru_cache\n def is_optimum_available() -> bool:\n     return importlib.util.find_spec(\"optimum\") is not None\n \n@@ -65,9 +69,11 @@\n     return False\n \n \n+@lru_cache\n def is_aqlm_available():\n     return importlib.util.find_spec(\"aqlm\") is not None\n \n \n+@lru_cache\n def is_auto_awq_available():\n     return importlib.util.find_spec(\"awq\") is not None\n", "issue": "Loading LORA weights in `diffusers` with a `peft` backend increases in latency as more paths are added to `PYTHONPATH`\n### System Info\r\n\r\n```\r\naccelerate==0.21.0\r\ndiffusers==0.26.3\r\npeft==0.9.0\r\nsafetensors==0.3.3\r\ntokenizers==0.15.2\r\ntorch==2.2.1\r\ntransformers==4.36.2\r\n```\r\n\r\n### Who can help?\r\n\r\n@sayakpaul \r\n\r\n### Information\r\n\r\n- [ ] The official example scripts\r\n- [X] My own modified scripts\r\n\r\n### Tasks\r\n\r\n- [ ] An officially supported task in the `examples` folder\r\n- [X] My own task or dataset (give details below)\r\n\r\n### Reproduction\r\n\r\n```python\r\nfrom diffusers import DiffusionPipeline\r\nimport time\r\nimport torch\r\nimport sys\r\nimport os\r\nimport shutil\r\n\r\npipe_id = \"stabilityai/stable-diffusion-xl-base-1.0\"\r\npipe = DiffusionPipeline.from_pretrained(pipe_id, torch_dtype=torch.float16).to(\"cuda\")\r\n\r\nloras = [\r\n    {\r\n        \"adapter_name\": \"anime\"\r\n        \"location\": \"./anime\",\r\n        \"weight_name\": \"anime.safetensors\",\r\n        \"token\": \"my_anime\"\r\n    },\r\n]\r\n\r\ndef run_dynamic_lora_inference(lora):\r\n    start_load_time = time.time()\r\n    pipe.load_lora_weights(lora[\"location\"], weight_name=lora[\"weight_name\"], adapter_name=lora[\"adapter_name\"])\r\n    end_load_time = time.time()\r\n    prompt = f\"Illustration of a dog in the style of {lora[\"token\"]}\"\r\n\r\n    start_fuse_time = time.time()\r\n    pipe.fuse_lora()\r\n    end_fuse_time = time.time()\r\n\r\n    start_set_adapter_time = time.time()\r\n    pipe.set_adapters(lora_name)\r\n    end_set_adapter_time = time.time()\r\n\r\n    start_inference_time = time.time()\r\n    image = pipe(\r\n        prompt, num_inference_steps=30, generator=torch.manual_seed(0)\r\n    ).images[0]\r\n    end_inference_time = time.time()\r\n\r\n    start_unfuse_time = time.time()\r\n    pipe.unfuse_lora()\r\n    end_unfuse_time = time.time()\r\n\r\n    start_unload_time = time.time()\r\n    pipe.unload_lora_weights()\r\n    end_unload_time = time.time()\r\n\r\n    image.save(f\"./{lora_name}.png\")\r\n\r\n    print(\"Load time:\", end_load_time - start_load_time)\r\n    print(\"Fuse time:\", end_fuse_time - start_fuse_time)\r\n    print(\"Set adapter time\", end_set_adapter_time - start_set_adapter_time)\r\n    print(\"Inference time:\", end_inference_time - start_inference_time)\r\n    print(\"Unfuse time:\", end_unfuse_time - start_unfuse_time)\r\n    print(\"Unload time:\", end_unload_time - start_unload_time)\r\n\r\ndef add_to_python_path():\r\n    root_path = \"./folders\"\r\n    shutil.rmtree(root_path)\r\n    os.mkdir(root_path)\r\n\r\n    folders =  [f\"folder_{x}\" for x in range(0, 10000)]\r\n    for folder in folders:\r\n        os.mkdir(os.path.join(root_path, folder))\r\n        sys.path.append(os.path.join(root_path, folder))\r\n\r\ndef main():\r\n    for lora in loras:\r\n        run_dynamic_lora_inference(lora)\r\n\r\nmain()\r\n```\r\n\r\nFlamegraph: \r\n![profile](https://github.com/huggingface/peft/assets/54301847/63b4d841-e740-4cb8-af51-96d011a8bbae)\r\n\r\n\r\n\r\n### Expected behavior\r\n\r\nI run a system with a somewhat large `PYTHONPATH` that we can't truncate, and we are currently blocked from upgrading `diffusers` to any version that uses `peft` for LORA inference. \r\n\r\nIt's loosely based on this post: https://huggingface.co/blog/lora-adapters-dynamic-loading\r\n\r\nWe've observed a behavior where the time taken for `load_lora_weights` increases significantly with more paths added to `PYTHONPATH`. This can be reproduced in the example provided - with 10,000 folders added to PYTHONPATH, we get the following latencies:\r\n\r\n```\r\nLoad time: 291.78441095352173\r\nFuse time: 0.12406659126281738\r\nSet adapter time 0.06171250343322754\r\nInference time: 9.685987710952759\r\nUnfuse time: 0.08063459396362305\r\nUnload time: 0.15737533569335938\r\n```\r\n\r\nBenchmarking against 1, 10, 100, 1000, 10000 and 50000 entries in the `PYTHONPATH`, we get a pretty astounding increase in load latency: \r\n\r\n![image](https://github.com/huggingface/peft/assets/54301847/cd8894d5-3c6c-47fd-b8ca-ead34a32e36d)\r\n\r\nEven at 100 entries, we're looking at an extra 4 seconds per `load` call which is a pretty significant increase.\r\n\r\nWe looked briefly at it and came to the conclusion that it's something to do with the way `peft` runs module imports, particularly repetitive calls to import modules, where the imports are not cached, eg, `importlib.util.find_spec` doesn't cache imports. \r\n\r\nInstead of this behaviour, we'd expect that `load_lora_weights` retains a relatively constant load time, regardless of the length of our python path. \n", "before_files": [{"content": "# Copyright 2023-present the HuggingFace Inc. team.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport importlib\nimport importlib.metadata as importlib_metadata\nfrom functools import lru_cache\n\nimport packaging.version\n\n\ndef is_bnb_available() -> bool:\n    return importlib.util.find_spec(\"bitsandbytes\") is not None\n\n\ndef is_bnb_4bit_available() -> bool:\n    if not is_bnb_available():\n        return False\n\n    import bitsandbytes as bnb\n\n    return hasattr(bnb.nn, \"Linear4bit\")\n\n\ndef is_auto_gptq_available():\n    if importlib.util.find_spec(\"auto_gptq\") is not None:\n        AUTOGPTQ_MINIMUM_VERSION = packaging.version.parse(\"0.5.0\")\n        version_autogptq = packaging.version.parse(importlib_metadata.version(\"auto_gptq\"))\n        if AUTOGPTQ_MINIMUM_VERSION <= version_autogptq:\n            return True\n        else:\n            raise ImportError(\n                f\"Found an incompatible version of auto-gptq. Found version {version_autogptq}, \"\n                f\"but only versions above {AUTOGPTQ_MINIMUM_VERSION} are supported\"\n            )\n\n\ndef is_optimum_available() -> bool:\n    return importlib.util.find_spec(\"optimum\") is not None\n\n\n@lru_cache\ndef is_torch_tpu_available(check_device=True):\n    \"Checks if `torch_xla` is installed and potentially if a TPU is in the environment\"\n    if importlib.util.find_spec(\"torch_xla\") is not None:\n        if check_device:\n            # We need to check if `xla_device` can be found, will raise a RuntimeError if not\n            try:\n                import torch_xla.core.xla_model as xm\n\n                _ = xm.xla_device()\n                return True\n            except RuntimeError:\n                return False\n        return True\n    return False\n\n\ndef is_aqlm_available():\n    return importlib.util.find_spec(\"aqlm\") is not None\n\n\ndef is_auto_awq_available():\n    return importlib.util.find_spec(\"awq\") is not None\n", "path": "src/peft/import_utils.py"}], "after_files": [{"content": "# Copyright 2023-present the HuggingFace Inc. team.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport importlib\nimport importlib.metadata as importlib_metadata\nfrom functools import lru_cache\n\nimport packaging.version\n\n\n@lru_cache\ndef is_bnb_available() -> bool:\n    return importlib.util.find_spec(\"bitsandbytes\") is not None\n\n\n@lru_cache\ndef is_bnb_4bit_available() -> bool:\n    if not is_bnb_available():\n        return False\n\n    import bitsandbytes as bnb\n\n    return hasattr(bnb.nn, \"Linear4bit\")\n\n\n@lru_cache\ndef is_auto_gptq_available():\n    if importlib.util.find_spec(\"auto_gptq\") is not None:\n        AUTOGPTQ_MINIMUM_VERSION = packaging.version.parse(\"0.5.0\")\n        version_autogptq = packaging.version.parse(importlib_metadata.version(\"auto_gptq\"))\n        if AUTOGPTQ_MINIMUM_VERSION <= version_autogptq:\n            return True\n        else:\n            raise ImportError(\n                f\"Found an incompatible version of auto-gptq. Found version {version_autogptq}, \"\n                f\"but only versions above {AUTOGPTQ_MINIMUM_VERSION} are supported\"\n            )\n\n\n@lru_cache\ndef is_optimum_available() -> bool:\n    return importlib.util.find_spec(\"optimum\") is not None\n\n\n@lru_cache\ndef is_torch_tpu_available(check_device=True):\n    \"Checks if `torch_xla` is installed and potentially if a TPU is in the environment\"\n    if importlib.util.find_spec(\"torch_xla\") is not None:\n        if check_device:\n            # We need to check if `xla_device` can be found, will raise a RuntimeError if not\n            try:\n                import torch_xla.core.xla_model as xm\n\n                _ = xm.xla_device()\n                return True\n            except RuntimeError:\n                return False\n        return True\n    return False\n\n\n@lru_cache\ndef is_aqlm_available():\n    return importlib.util.find_spec(\"aqlm\") is not None\n\n\n@lru_cache\ndef is_auto_awq_available():\n    return importlib.util.find_spec(\"awq\") is not None\n", "path": "src/peft/import_utils.py"}]}