We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[BUG] configure_tensorflow raises TypeError: expected string or bytes-like object in Horovod
**Describe the bug**
While trying to integrate Horovod's KerasEstimator w/ NVTabular, I'm seeing this error:
```
[1,0]<stderr>:/nvtabular/nvtabular/loader/tf_utils.py:53: UserWarning: TensorFlow runtime already initialized, may not be enough memory for cudf
[1,0]<stderr>:  warnings.warn(
[1,0]<stderr>:Traceback (most recent call last):
[1,0]<stderr>:  File "/nvtabular/nvtabular/loader/tf_utils.py", line 57, in configure_tensorflow
[1,0]<stderr>:    tf.config.experimental.set_virtual_device_configuration(
[1,0]<stderr>:  File "/opt/conda/envs/rapids/lib/python3.8/site-packages/tensorflow/python/framework/config.py", line 871, in set_logical_device_configuration
[1,0]<stderr>:    context.context().set_logical_device_configuration(device, logical_devices)
[1,0]<stderr>:  File "/opt/conda/envs/rapids/lib/python3.8/site-packages/tensorflow/python/eager/context.py", line 1629, in set_logical_device_configuration
[1,0]<stderr>:    raise RuntimeError(
[1,0]<stderr>:RuntimeError: Virtual devices cannot be modified after being initialized[1,0]<stderr>:
[1,0]<stderr>:
[1,0]<stderr>:During handling of the above exception, another exception occurred:
[1,0]<stderr>:
[1,0]<stderr>:Traceback (most recent call last):
[1,0]<stderr>:  File "/opt/conda/envs/rapids/lib/python3.8/runpy.py", line 194, in _run_module_as_main
[1,0]<stderr>:    return _run_code(code, main_globals, None,
[1,0]<stderr>:  File "/opt/conda/envs/rapids/lib/python3.8/runpy.py", line 87, in _run_code
[1,0]<stderr>:    exec(code, run_globals)
[1,0]<stderr>:  File "/horovod/horovod/spark/task/mpirun_exec_fn.py", line 52, in <module>
[1,0]<stderr>:    main(codec.loads_base64(sys.argv[1]), codec.loads_base64(sys.argv[2]))
[1,0]<stderr>:  File "/horovod/horovod/spark/task/mpirun_exec_fn.py", line 45, in main
[1,0]<stderr>:    task_exec(driver_addresses, settings, 'OMPI_COMM_WORLD_RANK', 'OMPI_COMM_WORLD_LOCAL_RANK')
[1,0]<stderr>:  File "/horovod/horovod/spark/task/__init__.py", line 61, in task_exec
[1,0]<stderr>:    result = fn(*args, **kwargs)
[1,0]<stderr>:  File "/horovod/horovod/spark/keras/remote.py", line 261, in train
[1,0]<stderr>:    history = fit(model, dm, steps_per_epoch,
[1,0]<stderr>:  File "/horovod/horovod/spark/keras/util.py", line 41, in fn
[1,0]<stderr>:    train_data = data_module.train_data(train_reader)
[1,0]<stderr>:  File "/horovod/horovod/spark/keras/datamodule.py", line 110, in train_data
[1,0]<stderr>:    from nvtabular.loader.tensorflow import KerasSequenceLoader
[1,0]<stderr>:  File "/nvtabular/nvtabular/loader/tensorflow.py", line 28, in <module>
[1,0]<stderr>:    from_dlpack = configure_tensorflow()
[1,0]<stderr>:  File "/nvtabular/nvtabular/loader/tf_utils.py", line 63, in configure_tensorflow
[1,0]<stderr>:    warnings.warn(e)
[1,0]<stderr>:TypeError: expected string or bytes-like object
```

This is resulting in a fatal error instead of a non-fatal warning.  My current workaround is to just use `warnings.warn(str(e))` here, although I'm not sure why the `RuntimeError` isn't automatically cast to string (when it normally would cast to string).

**Steps/Code to reproduce bug**
Unfortunately, since this involves integration with horovod, so the setup to reproduce is non-trivial.

**Expected behavior**
A non-fatal warning, and continued execution.

**Environment details (please complete the following information):**
 - Environment location: Docker on Linux with single GPU.
 - Method of NVTabular install: custom docker image based on `nvcr.io/nvidia/merlin/merlin-tensorflow-training:22.05`  

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `nvtabular/loader/tf_utils.py`
Content:
```
1 #
2 # Copyright (c) 2021, NVIDIA CORPORATION.
3 #
4 # Licensed under the Apache License, Version 2.0 (the "License");
5 # you may not use this file except in compliance with the License.
6 # You may obtain a copy of the License at
7 #
8 #     http://www.apache.org/licenses/LICENSE-2.0
9 #
10 # Unless required by applicable law or agreed to in writing, software
11 # distributed under the License is distributed on an "AS IS" BASIS,
12 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 # See the License for the specific language governing permissions and
14 # limitations under the License.
15 #
16 
17 import os
18 import warnings
19 
20 import tensorflow as tf
21 from packaging import version
22 from tensorflow.python.feature_column import feature_column_v2 as fc
23 
24 from merlin.core.dispatch import HAS_GPU
25 from merlin.core.utils import device_mem_size
26 
27 
28 def configure_tensorflow(memory_allocation=None, device=None):
29     total_gpu_mem_mb = device_mem_size(kind="total", cpu=(not HAS_GPU)) / (1024 ** 2)
30 
31     if memory_allocation is None:
32         memory_allocation = os.environ.get("TF_MEMORY_ALLOCATION", 0.5)
33 
34     if float(memory_allocation) < 1:
35         memory_allocation = total_gpu_mem_mb * float(memory_allocation)
36     memory_allocation = int(memory_allocation)
37     assert memory_allocation < total_gpu_mem_mb
38 
39     # TODO: what will this look like in any sort
40     # of distributed set up?
41     if device is None:
42         device = int(os.environ.get("TF_VISIBLE_DEVICE", 0))
43     tf_devices = tf.config.list_physical_devices("GPU")
44     if HAS_GPU and len(tf_devices) == 0:
45         raise ImportError("TensorFlow is not configured for GPU")
46     if HAS_GPU:
47         try:
48             tf.config.set_logical_device_configuration(
49                 tf_devices[device],
50                 [tf.config.LogicalDeviceConfiguration(memory_limit=memory_allocation)],
51             )
52         except RuntimeError:
53             warnings.warn(
54                 "TensorFlow runtime already initialized, may not be enough memory for cudf"
55             )
56         try:
57             tf.config.experimental.set_virtual_device_configuration(
58                 tf_devices[device],
59                 [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=memory_allocation)],
60             )
61         except RuntimeError as e:
62             # Virtual devices must be set before GPUs have been initialized
63             warnings.warn(e)
64 
65     # versions using TF earlier than 2.3.0 need to use extension
66     # library for dlpack support to avoid memory leak issue
67     __TF_DLPACK_STABLE_VERSION = "2.3.0"
68     if version.parse(tf.__version__) < version.parse(__TF_DLPACK_STABLE_VERSION):
69         try:
70             from tfdlpack import from_dlpack
71         except ModuleNotFoundError as e:
72             message = "If using TensorFlow < 2.3.0, you must install tfdlpack-gpu extension library"
73             raise ModuleNotFoundError(message) from e
74 
75     else:
76         from tensorflow.experimental.dlpack import from_dlpack
77 
78     return from_dlpack
79 
80 
81 def _get_parents(column):
82     """
83     recursive function for finding the feature columns
84     that supply inputs for a given `column`. If there are
85     none, returns the column. Uses sets so is not
86     deterministic.
87     """
88     if isinstance(column.parents[0], str):
89         return set([column])
90     parents = set()
91     for parent in column.parents:
92         parents |= _get_parents(parent)
93     return parents
94 
95 
96 def get_dataset_schema_from_feature_columns(feature_columns):
97     """
98     maps from a list of TensorFlow `feature_column`s to
99     lists giving the categorical and continuous feature
100     names for a dataset. Useful for constructing NVTabular
101     Workflows from feature columns
102     """
103     base_columns = set()
104     for column in feature_columns:
105         base_columns |= _get_parents(column)
106 
107     cat_names, cont_names = [], []
108     for column in base_columns:
109         if isinstance(column, fc.CategoricalColumn):
110             cat_names.append(column.name)
111         else:
112             cont_names.append(column.name)
113 
114     cat_names = sorted(cat_names)
115     cont_names = sorted(cont_names)
116     return cat_names, cont_names
117
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/nvtabular/loader/tf_utils.py b/nvtabular/loader/tf_utils.py
--- a/nvtabular/loader/tf_utils.py
+++ b/nvtabular/loader/tf_utils.py
@@ -60,7 +60,7 @@
             )
         except RuntimeError as e:
             # Virtual devices must be set before GPUs have been initialized
-            warnings.warn(e)
+            warnings.warn(str(e))
 
     # versions using TF earlier than 2.3.0 need to use extension
     # library for dlpack support to avoid memory leak issue

{"golden_diff": "diff --git a/nvtabular/loader/tf_utils.py b/nvtabular/loader/tf_utils.py\n--- a/nvtabular/loader/tf_utils.py\n+++ b/nvtabular/loader/tf_utils.py\n@@ -60,7 +60,7 @@\n             )\n         except RuntimeError as e:\n             # Virtual devices must be set before GPUs have been initialized\n-            warnings.warn(e)\n+            warnings.warn(str(e))\n \n     # versions using TF earlier than 2.3.0 need to use extension\n     # library for dlpack support to avoid memory leak issue\n", "issue": "[BUG] configure_tensorflow raises TypeError: expected string or bytes-like object in Horovod\n**Describe the bug**\r\nWhile trying to integrate Horovod's KerasEstimator w/ NVTabular, I'm seeing this error:\r\n```\r\n[1,0]<stderr>:/nvtabular/nvtabular/loader/tf_utils.py:53: UserWarning: TensorFlow runtime already initialized, may not be enough memory for cudf\r\n[1,0]<stderr>:  warnings.warn(\r\n[1,0]<stderr>:Traceback (most recent call last):\r\n[1,0]<stderr>:  File \"/nvtabular/nvtabular/loader/tf_utils.py\", line 57, in configure_tensorflow\r\n[1,0]<stderr>:    tf.config.experimental.set_virtual_device_configuration(\r\n[1,0]<stderr>:  File \"/opt/conda/envs/rapids/lib/python3.8/site-packages/tensorflow/python/framework/config.py\", line 871, in set_logical_device_configuration\r\n[1,0]<stderr>:    context.context().set_logical_device_configuration(device, logical_devices)\r\n[1,0]<stderr>:  File \"/opt/conda/envs/rapids/lib/python3.8/site-packages/tensorflow/python/eager/context.py\", line 1629, in set_logical_device_configuration\r\n[1,0]<stderr>:    raise RuntimeError(\r\n[1,0]<stderr>:RuntimeError: Virtual devices cannot be modified after being initialized[1,0]<stderr>:\r\n[1,0]<stderr>:\r\n[1,0]<stderr>:During handling of the above exception, another exception occurred:\r\n[1,0]<stderr>:\r\n[1,0]<stderr>:Traceback (most recent call last):\r\n[1,0]<stderr>:  File \"/opt/conda/envs/rapids/lib/python3.8/runpy.py\", line 194, in _run_module_as_main\r\n[1,0]<stderr>:    return _run_code(code, main_globals, None,\r\n[1,0]<stderr>:  File \"/opt/conda/envs/rapids/lib/python3.8/runpy.py\", line 87, in _run_code\r\n[1,0]<stderr>:    exec(code, run_globals)\r\n[1,0]<stderr>:  File \"/horovod/horovod/spark/task/mpirun_exec_fn.py\", line 52, in <module>\r\n[1,0]<stderr>:    main(codec.loads_base64(sys.argv[1]), codec.loads_base64(sys.argv[2]))\r\n[1,0]<stderr>:  File \"/horovod/horovod/spark/task/mpirun_exec_fn.py\", line 45, in main\r\n[1,0]<stderr>:    task_exec(driver_addresses, settings, 'OMPI_COMM_WORLD_RANK', 'OMPI_COMM_WORLD_LOCAL_RANK')\r\n[1,0]<stderr>:  File \"/horovod/horovod/spark/task/__init__.py\", line 61, in task_exec\r\n[1,0]<stderr>:    result = fn(*args, **kwargs)\r\n[1,0]<stderr>:  File \"/horovod/horovod/spark/keras/remote.py\", line 261, in train\r\n[1,0]<stderr>:    history = fit(model, dm, steps_per_epoch,\r\n[1,0]<stderr>:  File \"/horovod/horovod/spark/keras/util.py\", line 41, in fn\r\n[1,0]<stderr>:    train_data = data_module.train_data(train_reader)\r\n[1,0]<stderr>:  File \"/horovod/horovod/spark/keras/datamodule.py\", line 110, in train_data\r\n[1,0]<stderr>:    from nvtabular.loader.tensorflow import KerasSequenceLoader\r\n[1,0]<stderr>:  File \"/nvtabular/nvtabular/loader/tensorflow.py\", line 28, in <module>\r\n[1,0]<stderr>:    from_dlpack = configure_tensorflow()\r\n[1,0]<stderr>:  File \"/nvtabular/nvtabular/loader/tf_utils.py\", line 63, in configure_tensorflow\r\n[1,0]<stderr>:    warnings.warn(e)\r\n[1,0]<stderr>:TypeError: expected string or bytes-like object\r\n```\r\n\r\nThis is resulting in a fatal error instead of a non-fatal warning.  My current workaround is to just use `warnings.warn(str(e))` here, although I'm not sure why the `RuntimeError` isn't automatically cast to string (when it normally would cast to string).\r\n\r\n**Steps/Code to reproduce bug**\r\nUnfortunately, since this involves integration with horovod, so the setup to reproduce is non-trivial.\r\n\r\n**Expected behavior**\r\nA non-fatal warning, and continued execution.\r\n\r\n**Environment details (please complete the following information):**\r\n - Environment location: Docker on Linux with single GPU.\r\n - Method of NVTabular install: custom docker image based on `nvcr.io/nvidia/merlin/merlin-tensorflow-training:22.05`  \n", "before_files": [{"content": "#\n# Copyright (c) 2021, NVIDIA CORPORATION.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n\nimport os\nimport warnings\n\nimport tensorflow as tf\nfrom packaging import version\nfrom tensorflow.python.feature_column import feature_column_v2 as fc\n\nfrom merlin.core.dispatch import HAS_GPU\nfrom merlin.core.utils import device_mem_size\n\n\ndef configure_tensorflow(memory_allocation=None, device=None):\n    total_gpu_mem_mb = device_mem_size(kind=\"total\", cpu=(not HAS_GPU)) / (1024 ** 2)\n\n    if memory_allocation is None:\n        memory_allocation = os.environ.get(\"TF_MEMORY_ALLOCATION\", 0.5)\n\n    if float(memory_allocation) < 1:\n        memory_allocation = total_gpu_mem_mb * float(memory_allocation)\n    memory_allocation = int(memory_allocation)\n    assert memory_allocation < total_gpu_mem_mb\n\n    # TODO: what will this look like in any sort\n    # of distributed set up?\n    if device is None:\n        device = int(os.environ.get(\"TF_VISIBLE_DEVICE\", 0))\n    tf_devices = tf.config.list_physical_devices(\"GPU\")\n    if HAS_GPU and len(tf_devices) == 0:\n        raise ImportError(\"TensorFlow is not configured for GPU\")\n    if HAS_GPU:\n        try:\n            tf.config.set_logical_device_configuration(\n                tf_devices[device],\n                [tf.config.LogicalDeviceConfiguration(memory_limit=memory_allocation)],\n            )\n        except RuntimeError:\n            warnings.warn(\n                \"TensorFlow runtime already initialized, may not be enough memory for cudf\"\n            )\n        try:\n            tf.config.experimental.set_virtual_device_configuration(\n                tf_devices[device],\n                [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=memory_allocation)],\n            )\n        except RuntimeError as e:\n            # Virtual devices must be set before GPUs have been initialized\n            warnings.warn(e)\n\n    # versions using TF earlier than 2.3.0 need to use extension\n    # library for dlpack support to avoid memory leak issue\n    __TF_DLPACK_STABLE_VERSION = \"2.3.0\"\n    if version.parse(tf.__version__) < version.parse(__TF_DLPACK_STABLE_VERSION):\n        try:\n            from tfdlpack import from_dlpack\n        except ModuleNotFoundError as e:\n            message = \"If using TensorFlow < 2.3.0, you must install tfdlpack-gpu extension library\"\n            raise ModuleNotFoundError(message) from e\n\n    else:\n        from tensorflow.experimental.dlpack import from_dlpack\n\n    return from_dlpack\n\n\ndef _get_parents(column):\n    \"\"\"\n    recursive function for finding the feature columns\n    that supply inputs for a given `column`. If there are\n    none, returns the column. Uses sets so is not\n    deterministic.\n    \"\"\"\n    if isinstance(column.parents[0], str):\n        return set([column])\n    parents = set()\n    for parent in column.parents:\n        parents |= _get_parents(parent)\n    return parents\n\n\ndef get_dataset_schema_from_feature_columns(feature_columns):\n    \"\"\"\n    maps from a list of TensorFlow `feature_column`s to\n    lists giving the categorical and continuous feature\n    names for a dataset. Useful for constructing NVTabular\n    Workflows from feature columns\n    \"\"\"\n    base_columns = set()\n    for column in feature_columns:\n        base_columns |= _get_parents(column)\n\n    cat_names, cont_names = [], []\n    for column in base_columns:\n        if isinstance(column, fc.CategoricalColumn):\n            cat_names.append(column.name)\n        else:\n            cont_names.append(column.name)\n\n    cat_names = sorted(cat_names)\n    cont_names = sorted(cont_names)\n    return cat_names, cont_names\n", "path": "nvtabular/loader/tf_utils.py"}], "after_files": [{"content": "#\n# Copyright (c) 2021, NVIDIA CORPORATION.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n\nimport os\nimport warnings\n\nimport tensorflow as tf\nfrom packaging import version\nfrom tensorflow.python.feature_column import feature_column_v2 as fc\n\nfrom merlin.core.dispatch import HAS_GPU\nfrom merlin.core.utils import device_mem_size\n\n\ndef configure_tensorflow(memory_allocation=None, device=None):\n    total_gpu_mem_mb = device_mem_size(kind=\"total\", cpu=(not HAS_GPU)) / (1024 ** 2)\n\n    if memory_allocation is None:\n        memory_allocation = os.environ.get(\"TF_MEMORY_ALLOCATION\", 0.5)\n\n    if float(memory_allocation) < 1:\n        memory_allocation = total_gpu_mem_mb * float(memory_allocation)\n    memory_allocation = int(memory_allocation)\n    assert memory_allocation < total_gpu_mem_mb\n\n    # TODO: what will this look like in any sort\n    # of distributed set up?\n    if device is None:\n        device = int(os.environ.get(\"TF_VISIBLE_DEVICE\", 0))\n    tf_devices = tf.config.list_physical_devices(\"GPU\")\n    if HAS_GPU and len(tf_devices) == 0:\n        raise ImportError(\"TensorFlow is not configured for GPU\")\n    if HAS_GPU:\n        try:\n            tf.config.set_logical_device_configuration(\n                tf_devices[device],\n                [tf.config.LogicalDeviceConfiguration(memory_limit=memory_allocation)],\n            )\n        except RuntimeError:\n            warnings.warn(\n                \"TensorFlow runtime already initialized, may not be enough memory for cudf\"\n            )\n        try:\n            tf.config.experimental.set_virtual_device_configuration(\n                tf_devices[device],\n                [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=memory_allocation)],\n            )\n        except RuntimeError as e:\n            # Virtual devices must be set before GPUs have been initialized\n            warnings.warn(str(e))\n\n    # versions using TF earlier than 2.3.0 need to use extension\n    # library for dlpack support to avoid memory leak issue\n    __TF_DLPACK_STABLE_VERSION = \"2.3.0\"\n    if version.parse(tf.__version__) < version.parse(__TF_DLPACK_STABLE_VERSION):\n        try:\n            from tfdlpack import from_dlpack\n        except ModuleNotFoundError as e:\n            message = \"If using TensorFlow < 2.3.0, you must install tfdlpack-gpu extension library\"\n            raise ModuleNotFoundError(message) from e\n\n    else:\n        from tensorflow.experimental.dlpack import from_dlpack\n\n    return from_dlpack\n\n\ndef _get_parents(column):\n    \"\"\"\n    recursive function for finding the feature columns\n    that supply inputs for a given `column`. If there are\n    none, returns the column. Uses sets so is not\n    deterministic.\n    \"\"\"\n    if isinstance(column.parents[0], str):\n        return set([column])\n    parents = set()\n    for parent in column.parents:\n        parents |= _get_parents(parent)\n    return parents\n\n\ndef get_dataset_schema_from_feature_columns(feature_columns):\n    \"\"\"\n    maps from a list of TensorFlow `feature_column`s to\n    lists giving the categorical and continuous feature\n    names for a dataset. Useful for constructing NVTabular\n    Workflows from feature columns\n    \"\"\"\n    base_columns = set()\n    for column in feature_columns:\n        base_columns |= _get_parents(column)\n\n    cat_names, cont_names = [], []\n    for column in base_columns:\n        if isinstance(column, fc.CategoricalColumn):\n            cat_names.append(column.name)\n        else:\n            cont_names.append(column.name)\n\n    cat_names = sorted(cat_names)\n    cont_names = sorted(cont_names)\n    return cat_names, cont_names\n", "path": "nvtabular/loader/tf_utils.py"}]}