We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
ESET connector causing runaway Redis memory consumption despite REDIS__TRIMMING=1000000
## Description

When enabling the `external-import/eset` connector with defaults (from `external-import/eset/docker-compose.yml`), it causes redis to rapidly run away with consuming all memory. I have `REDIS__TRIMMING=1000000` set in my `docker-compose.yml` and it doesn't seem to make a difference for this particular connector, but seems to be working for everything else.

The `connector-eset` section of my `docker-compose.yml` is lifted almost verbatim from the example provided in the connector's directory:
```yaml
  connector-eset:
    image: opencti/connector-eset:5.12.32
    environment:
      - OPENCTI_URL=http://opencti-url
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${ESET_CONNECTOR_ID}
      - "CONNECTOR_NAME=ESET Cyber Threat Intelligence"
      - CONNECTOR_SCOPE=eset
      - CONNECTOR_CONFIDENCE_LEVEL=30 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=error
      - ESET_API_URL=eti.eset.com
      - ESET_USERNAME=${ESET_CONNECTOR_USER}
      - ESET_PASSWORD=${ESET_CONNECTOR_PASSWORD}
      - ESET_COLLECTIONS=ei.misp,ei.cc,ei.urls
      - ESET_IMPORT_APT_REPORTS=true
      - ESET_IMPORT_START_DATE=2022-04-01
      - ESET_CREATE_OBSERVABLES=true
      - ESET_INTERVAL=60 # Required, in minutes
    restart: always
```

## Environment

1. OS (where OpenCTI server runs): Amazon Linux 2023
2. OpenCTI version: 5.12.32
3. OpenCTI client: connector
4. Other environment details: Using Docker CE and `docker-compose`

## Reproducible Steps

Steps to create the smallest reproducible scenario:
1. Add `connector-eset` in my `docker-compose.yml`
2. Update `.env` with proper credentials for access
3. `docker-compose up -d`
4. Wait for awhile and eventually redis grows to consume all RAM (in my case, it got to around 12GB in usage)

## Expected Output

Redis shouldn't consume all memory and `REDIS__TRIMMING=....` should be able to keep it within a reasonably predictable ceiling. In this particular case, `redis` resident memory size seems to stay under 2GB when trimming is set to `1000000`.

## Actual Output

`redis` memory consumption grows without restraint until the system runs out of memory and the OOM reaper has to kill something.

## Additional information

Here is a `docker-compose.yml` that seems to be working well for me with `REDIS__TRIMMING=2000000`: https://github.com/ckane/opencti-docker/blob/tf-main/docker-compose.yml

In the deployment I am trying to use it in, I reduced this to `1000000` to see if it would fix the problem, but it doesn't seem to have any impact at all.

In this case, `redis` memory consumption stays under control, but if I add `connector-eset` to it, then when the `eset` connector starts ingesting intel from their feed, `redis` consumption will grow rapidly until it exhausts all RAM in the system.

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `external-import/eset/src/eset.py`
Content:
```
1 import base64
2 import datetime
3 import json
4 import os
5 import sys
6 import time
7 
8 import cabby
9 import eti_api
10 import pytz
11 import stix2
12 import yaml
13 from dateutil.parser import parse
14 from pycti import OpenCTIConnectorHelper, Report, get_config_variable
15 
16 TMP_DIR = "TMP"
17 
18 
19 class Eset:
20     def __init__(self):
21         # Instantiate the connector helper from config
22         config_file_path = os.path.dirname(os.path.abspath(__file__)) + "/config.yml"
23         config = (
24             yaml.load(open(config_file_path), Loader=yaml.FullLoader)
25             if os.path.isfile(config_file_path)
26             else {}
27         )
28         self.helper = OpenCTIConnectorHelper(config)
29         # Extra config
30         self.eset_api_url = get_config_variable(
31             "ESET_API_URL", ["eset", "api_url"], config
32         )
33         self.eset_username = get_config_variable(
34             "ESET_USERNAME", ["eset", "username"], config
35         )
36         self.eset_password = get_config_variable(
37             "ESET_PASSWORD", ["eset", "password"], config
38         )
39         self.eset_collections = get_config_variable(
40             "ESET_COLLECTIONS", ["eset", "collections"], config
41         )
42         self.eset_import_apt_reports = get_config_variable(
43             "ESET_IMPORT_APT_REPORTS",
44             ["eset", "import_apt_reports"],
45             config,
46             False,
47             True,
48         )
49         self.eset_import_start_date = get_config_variable(
50             "ESET_IMPORT_START_DATE",
51             ["eset", "import_start_date"],
52             config,
53         )
54         self.eset_create_observables = get_config_variable(
55             "ESET_CREATE_OBSERVABLES",
56             ["eset", "create_observables"],
57             config,
58         )
59         self.eset_interval = get_config_variable(
60             "ESET_INTERVAL", ["eset", "interval"], config, True
61         )
62         self.update_existing_data = get_config_variable(
63             "CONNECTOR_UPDATE_EXISTING_DATA",
64             ["connector", "update_existing_data"],
65             config,
66         )
67         self.identity = self.helper.api.identity.create(
68             type="Organization",
69             name="ESET",
70             description="ESET, s.r.o., is a software company specializing in cybersecurity.",
71         )
72         self.added_after = int(parse(self.eset_import_start_date).timestamp())
73         # Init variables
74         self.cache = {}
75         if self.eset_collections is not None:
76             self.eset_collections = self.eset_collections.split(",")
77 
78         # Create temporary dir and initialize logging.
79         if sys.version_info.major == 2:  # Compatibility with Python 2.7.
80             if not os.path.isdir(TMP_DIR):
81                 os.makedirs(TMP_DIR)
82         else:
83             os.makedirs(TMP_DIR, exist_ok=True)
84 
85     def get_interval(self):
86         return int(self.eset_interval) * 60
87 
88     def _download_all_report_stuff(self, connection, report, base_path):
89         """Download xml, pdf and adds (if available) from given *report* into paths starting with *base_path*."""
90         for fmt in ["pdf", "xml", "adds"]:
91             ext = fmt if fmt != "adds" else "zip"
92             connection.get_report(report, fmt, file_path="{}.{}".format(base_path, ext))
93 
94     def _import_reports(self, work_id, start_epoch):
95         connection = eti_api.Connection(
96             username=self.eset_username,
97             password=self.eset_password,
98             host="eti.eset.com",
99         )
100         from_date = datetime.datetime.utcfromtimestamp(start_epoch).astimezone(pytz.utc)
101         i = 0
102         for report in connection.list_reports(
103             type="all", datetimefrom=from_date.isoformat()
104         ):
105             bundle_objects = []
106             if report["status"] != "finished":
107                 self.helper.log_info("Finished")
108                 continue  # Skip not generated reports.
109             i += 1
110             file_path = os.path.join(TMP_DIR, "{}_{:02d}".format("all", i))
111             self._download_all_report_stuff(connection, report, file_path)
112             if os.path.isfile(file_path + ".pdf"):
113                 name = report["filename"].replace(".pdf", "")
114                 date = parse(report["date"])
115                 with open(file_path + ".pdf", "rb") as f:
116                     file_data_encoded = base64.b64encode(f.read())
117                 file = {
118                     "name": report["filename"],
119                     "data": file_data_encoded.decode("utf-8"),
120                     "mime_type": "application/pdf",
121                     "no_trigger_import": True,
122                 }
123                 stix_report = stix2.Report(
124                     id=Report.generate_id(name, date),
125                     name=name,
126                     report_types=["APT Report"],
127                     description=name,
128                     published=date,
129                     labels=["apt", "eset"],
130                     confidence=self.helper.connect_confidence_level,
131                     created_by_ref=self.identity["standard_id"],
132                     object_refs=[self.identity["standard_id"]],
133                     allow_custom=True,
134                     x_opencti_files=[file],
135                     object_marking_refs=[stix2.TLP_AMBER.get("id")],
136                 )
137                 bundle_objects.append(stix_report)
138                 try:
139                     self.helper.log_debug("Objects to be sent " + str(bundle_objects))
140                     self.helper.send_stix2_bundle(
141                         self.helper.stix2_create_bundle(bundle_objects),
142                         update=self.update_existing_data,
143                         bypass_split=True,
144                         work_id=work_id,
145                     )
146                 except Exception as e:
147                     self.helper.log_info("Failed to process report " + name)
148                     self.helper.log_info("ERROR: " + str(e))
149                 os.remove(file_path + ".pdf")
150                 if os.path.isfile(file_path + ".xml"):
151                     os.remove(file_path + ".xml")
152                 if os.path.isfile(file_path + ".zip"):
153                     os.remove(file_path + ".zip")
154 
155     def _import_collection(self, collection, work_id, start_epoch):
156         object_types_with_confidence = [
157             "attack-pattern",
158             "course-of-action",
159             "threat-actor",
160             "intrusion-set",
161             "campaign",
162             "malware",
163             "tool",
164             "vulnerability",
165             "report",
166             "relationship",
167             "indicator",
168         ]
169         client = cabby.create_client(
170             self.eset_api_url, discovery_path="/taxiiservice/discovery", use_https=True
171         )
172         client.set_auth(username=self.eset_username, password=self.eset_password)
173         no_more_result = False
174         end_epoch = start_epoch + 3600
175         while no_more_result is False:
176             self.helper.log_info(
177                 "Iterating with collection="
178                 + str(collection)
179                 + ", start_epoch="
180                 + str(start_epoch)
181                 + ", end_epoch="
182                 + str(end_epoch)
183             )
184             begin_date = datetime.datetime.utcfromtimestamp(start_epoch).astimezone(
185                 pytz.utc
186             )
187             end_date = datetime.datetime.utcfromtimestamp(end_epoch).astimezone(
188                 pytz.utc
189             )
190             try:
191                 for item in client.poll(
192                     collection + " (stix2)", begin_date=begin_date, end_date=end_date
193                 ):
194                     if not item.content:  # Skip empty packages.
195                         continue
196                     parsed_content = json.loads(item.content)
197                     objects = []
198                     for object in parsed_content["objects"]:
199                         if "confidence" in object_types_with_confidence:
200                             if "confidence" not in object:
201                                 object["confidence"] = int(
202                                     self.helper.connect_confidence_level
203                                 )
204                         if object["type"] == "indicator":
205                             object["name"] = object["pattern"]
206                             object["pattern_type"] = "stix"
207                             object["pattern"] = (
208                                 object["pattern"]
209                                 .replace("SHA1", "'SHA-1'")
210                                 .replace("SHA256", "'SHA-256'")
211                             )
212                             if self.eset_create_observables:
213                                 object["x_opencti_create_observables"] = (
214                                     self.eset_create_observables
215                                 )
216                         objects.append(object)
217                     parsed_content["objects"] = objects
218                     self.helper.send_stix2_bundle(
219                         json.dumps(parsed_content),
220                         update=self.update_existing_data,
221                         work_id=work_id,
222                     )
223             except Exception as e:
224                 self.helper.log_error(str(e))
225             if end_epoch > int(time.time()):
226                 no_more_result = True
227             else:
228                 start_epoch = end_epoch
229                 end_epoch = start_epoch + 3600
230 
231     def run(self):
232         while True:
233             try:
234                 self.helper.log_info("Synchronizing with ESET API...")
235                 timestamp = int(time.time())
236                 now = datetime.datetime.utcfromtimestamp(timestamp)
237                 friendly_name = "ESET run @ " + now.strftime("%Y-%m-%d %H:%M:%S")
238                 work_id = self.helper.api.work.initiate_work(
239                     self.helper.connect_id, friendly_name
240                 )
241                 current_state = self.helper.get_state()
242                 if current_state is None:
243                     self.helper.set_state({"last_run": self.added_after})
244                 # Get collections
245                 current_state = self.helper.get_state()
246 
247                 if self.eset_collections is not None:
248                     for collection in self.eset_collections:
249                         self._import_collection(
250                             collection, work_id, current_state["last_run"]
251                         )
252                 if self.eset_import_apt_reports:
253                     self._import_reports(work_id, current_state["last_run"])
254                 self.helper.set_state({"last_run": timestamp})
255                 message = "End of synchronization"
256                 self.helper.api.work.to_processed(work_id, message)
257                 self.helper.log_info(message)
258 
259                 if self.helper.connect_run_and_terminate:
260                     self.helper.log_info("Connector stop")
261                     self.helper.force_ping()
262                     sys.exit(0)
263 
264                 time.sleep(self.get_interval())
265             except (KeyboardInterrupt, SystemExit):
266                 self.helper.log_info("Connector stop")
267                 sys.exit(0)
268 
269             except Exception as e:
270                 self.helper.log_error(str(e))
271 
272                 if self.helper.connect_run_and_terminate:
273                     self.helper.log_info("Connector stop")
274                     sys.exit(0)
275 
276                 time.sleep(60)
277 
278 
279 if __name__ == "__main__":
280     try:
281         esetConnector = Eset()
282         esetConnector.run()
283     except Exception as e:
284         print(e)
285         time.sleep(10)
286         sys.exit(0)
287
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/external-import/eset/src/eset.py b/external-import/eset/src/eset.py
--- a/external-import/eset/src/eset.py
+++ b/external-import/eset/src/eset.py
@@ -11,7 +11,13 @@
 import stix2
 import yaml
 from dateutil.parser import parse
-from pycti import OpenCTIConnectorHelper, Report, get_config_variable
+from pycti import (
+    Indicator,
+    Malware,
+    OpenCTIConnectorHelper,
+    Report,
+    get_config_variable,
+)
 
 TMP_DIR = "TMP"
 
@@ -195,12 +201,55 @@
                         continue
                     parsed_content = json.loads(item.content)
                     objects = []
+                    id_remaps = {}
+                    removed_ids = set()
                     for object in parsed_content["objects"]:
                         if "confidence" in object_types_with_confidence:
                             if "confidence" not in object:
                                 object["confidence"] = int(
                                     self.helper.connect_confidence_level
                                 )
+                        # Don't consume identity entities w/ "customer" as the name.
+                        # ESET uses this to indicate country targeting, and consuming
+                        # these causes problems due to dedupe.
+                        # TODO: Convert these & relevant relationship refs to country
+                        # locations.
+                        if (
+                            object["type"] == "identity"
+                            and "name" in object
+                            and object["name"] == "customer"
+                        ) or object["type"] == "observed-data":
+                            removed_ids.add(object["id"])
+                            continue
+
+                        # Malware STIX IDs need to be manually recomputed so they're
+                        # deterministic by malware name
+                        if object["type"] == "malware" and "name" in object:
+                            new_id = Malware.generate_id(object["name"])
+                            if object["id"] in id_remaps:
+                                new_id = id_remaps[object["id"]]
+                            else:
+                                id_remaps[object["id"]] = new_id
+                            object["id"] = new_id
+
+                        # If we remapped a STIX id earlier to a pycti one, we need  to
+                        # reflect that properly in any relevant relationship too
+                        if object["type"] == "relationship":
+                            if "source_ref" in object:
+                                if object["source_ref"] in removed_ids:
+                                    continue  # skip relationship if either ref is in removed_ids
+                                if object["source_ref"] in id_remaps:
+                                    object["source_ref"] = id_remaps[
+                                        object["source_ref"]
+                                    ]
+                            if "target_ref" in object:
+                                if object["target_ref"] in removed_ids:
+                                    continue  # skip relationship if either ref is in removed_ids
+                                if object["target_ref"] in id_remaps:
+                                    object["target_ref"] = id_remaps[
+                                        object["target_ref"]
+                                    ]
+
                         if object["type"] == "indicator":
                             object["name"] = object["pattern"]
                             object["pattern_type"] = "stix"
@@ -209,6 +258,12 @@
                                 .replace("SHA1", "'SHA-1'")
                                 .replace("SHA256", "'SHA-256'")
                             )
+                            new_id = Indicator.generate_id(object["pattern"])
+                            if object["id"] in id_remaps:
+                                new_id = id_remaps[object["id"]]
+                            else:
+                                id_remaps[object["id"]] = new_id
+                            object["id"] = new_id
                             if self.eset_create_observables:
                                 object["x_opencti_create_observables"] = (
                                     self.eset_create_observables

{"golden_diff": "diff --git a/external-import/eset/src/eset.py b/external-import/eset/src/eset.py\n--- a/external-import/eset/src/eset.py\n+++ b/external-import/eset/src/eset.py\n@@ -11,7 +11,13 @@\n import stix2\n import yaml\n from dateutil.parser import parse\n-from pycti import OpenCTIConnectorHelper, Report, get_config_variable\n+from pycti import (\n+    Indicator,\n+    Malware,\n+    OpenCTIConnectorHelper,\n+    Report,\n+    get_config_variable,\n+)\n \n TMP_DIR = \"TMP\"\n \n@@ -195,12 +201,55 @@\n                         continue\n                     parsed_content = json.loads(item.content)\n                     objects = []\n+                    id_remaps = {}\n+                    removed_ids = set()\n                     for object in parsed_content[\"objects\"]:\n                         if \"confidence\" in object_types_with_confidence:\n                             if \"confidence\" not in object:\n                                 object[\"confidence\"] = int(\n                                     self.helper.connect_confidence_level\n                                 )\n+                        # Don't consume identity entities w/ \"customer\" as the name.\n+                        # ESET uses this to indicate country targeting, and consuming\n+                        # these causes problems due to dedupe.\n+                        # TODO: Convert these & relevant relationship refs to country\n+                        # locations.\n+                        if (\n+                            object[\"type\"] == \"identity\"\n+                            and \"name\" in object\n+                            and object[\"name\"] == \"customer\"\n+                        ) or object[\"type\"] == \"observed-data\":\n+                            removed_ids.add(object[\"id\"])\n+                            continue\n+\n+                        # Malware STIX IDs need to be manually recomputed so they're\n+                        # deterministic by malware name\n+                        if object[\"type\"] == \"malware\" and \"name\" in object:\n+                            new_id = Malware.generate_id(object[\"name\"])\n+                            if object[\"id\"] in id_remaps:\n+                                new_id = id_remaps[object[\"id\"]]\n+                            else:\n+                                id_remaps[object[\"id\"]] = new_id\n+                            object[\"id\"] = new_id\n+\n+                        # If we remapped a STIX id earlier to a pycti one, we need  to\n+                        # reflect that properly in any relevant relationship too\n+                        if object[\"type\"] == \"relationship\":\n+                            if \"source_ref\" in object:\n+                                if object[\"source_ref\"] in removed_ids:\n+                                    continue  # skip relationship if either ref is in removed_ids\n+                                if object[\"source_ref\"] in id_remaps:\n+                                    object[\"source_ref\"] = id_remaps[\n+                                        object[\"source_ref\"]\n+                                    ]\n+                            if \"target_ref\" in object:\n+                                if object[\"target_ref\"] in removed_ids:\n+                                    continue  # skip relationship if either ref is in removed_ids\n+                                if object[\"target_ref\"] in id_remaps:\n+                                    object[\"target_ref\"] = id_remaps[\n+                                        object[\"target_ref\"]\n+                                    ]\n+\n                         if object[\"type\"] == \"indicator\":\n                             object[\"name\"] = object[\"pattern\"]\n                             object[\"pattern_type\"] = \"stix\"\n@@ -209,6 +258,12 @@\n                                 .replace(\"SHA1\", \"'SHA-1'\")\n                                 .replace(\"SHA256\", \"'SHA-256'\")\n                             )\n+                            new_id = Indicator.generate_id(object[\"pattern\"])\n+                            if object[\"id\"] in id_remaps:\n+                                new_id = id_remaps[object[\"id\"]]\n+                            else:\n+                                id_remaps[object[\"id\"]] = new_id\n+                            object[\"id\"] = new_id\n                             if self.eset_create_observables:\n                                 object[\"x_opencti_create_observables\"] = (\n                                     self.eset_create_observables\n", "issue": "ESET connector causing runaway Redis memory consumption despite REDIS__TRIMMING=1000000\n## Description\r\n\r\nWhen enabling the `external-import/eset` connector with defaults (from `external-import/eset/docker-compose.yml`), it causes redis to rapidly run away with consuming all memory. I have `REDIS__TRIMMING=1000000` set in my `docker-compose.yml` and it doesn't seem to make a difference for this particular connector, but seems to be working for everything else.\r\n\r\nThe `connector-eset` section of my `docker-compose.yml` is lifted almost verbatim from the example provided in the connector's directory:\r\n```yaml\r\n  connector-eset:\r\n    image: opencti/connector-eset:5.12.32\r\n    environment:\r\n      - OPENCTI_URL=http://opencti-url\r\n      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}\r\n      - CONNECTOR_ID=${ESET_CONNECTOR_ID}\r\n      - \"CONNECTOR_NAME=ESET Cyber Threat Intelligence\"\r\n      - CONNECTOR_SCOPE=eset\r\n      - CONNECTOR_CONFIDENCE_LEVEL=30 # From 0 (Unknown) to 100 (Fully trusted)\r\n      - CONNECTOR_LOG_LEVEL=error\r\n      - ESET_API_URL=eti.eset.com\r\n      - ESET_USERNAME=${ESET_CONNECTOR_USER}\r\n      - ESET_PASSWORD=${ESET_CONNECTOR_PASSWORD}\r\n      - ESET_COLLECTIONS=ei.misp,ei.cc,ei.urls\r\n      - ESET_IMPORT_APT_REPORTS=true\r\n      - ESET_IMPORT_START_DATE=2022-04-01\r\n      - ESET_CREATE_OBSERVABLES=true\r\n      - ESET_INTERVAL=60 # Required, in minutes\r\n    restart: always\r\n```\r\n\r\n## Environment\r\n\r\n1. OS (where OpenCTI server runs): Amazon Linux 2023\r\n2. OpenCTI version: 5.12.32\r\n3. OpenCTI client: connector\r\n4. Other environment details: Using Docker CE and `docker-compose`\r\n\r\n## Reproducible Steps\r\n\r\nSteps to create the smallest reproducible scenario:\r\n1. Add `connector-eset` in my `docker-compose.yml`\r\n2. Update `.env` with proper credentials for access\r\n3. `docker-compose up -d`\r\n4. Wait for awhile and eventually redis grows to consume all RAM (in my case, it got to around 12GB in usage)\r\n\r\n## Expected Output\r\n\r\nRedis shouldn't consume all memory and `REDIS__TRIMMING=....` should be able to keep it within a reasonably predictable ceiling. In this particular case, `redis` resident memory size seems to stay under 2GB when trimming is set to `1000000`.\r\n\r\n## Actual Output\r\n\r\n`redis` memory consumption grows without restraint until the system runs out of memory and the OOM reaper has to kill something.\r\n\r\n## Additional information\r\n\r\nHere is a `docker-compose.yml` that seems to be working well for me with `REDIS__TRIMMING=2000000`: https://github.com/ckane/opencti-docker/blob/tf-main/docker-compose.yml\r\n\r\nIn the deployment I am trying to use it in, I reduced this to `1000000` to see if it would fix the problem, but it doesn't seem to have any impact at all.\r\n\r\nIn this case, `redis` memory consumption stays under control, but if I add `connector-eset` to it, then when the `eset` connector starts ingesting intel from their feed, `redis` consumption will grow rapidly until it exhausts all RAM in the system.\n", "before_files": [{"content": "import base64\nimport datetime\nimport json\nimport os\nimport sys\nimport time\n\nimport cabby\nimport eti_api\nimport pytz\nimport stix2\nimport yaml\nfrom dateutil.parser import parse\nfrom pycti import OpenCTIConnectorHelper, Report, get_config_variable\n\nTMP_DIR = \"TMP\"\n\n\nclass Eset:\n    def __init__(self):\n        # Instantiate the connector helper from config\n        config_file_path = os.path.dirname(os.path.abspath(__file__)) + \"/config.yml\"\n        config = (\n            yaml.load(open(config_file_path), Loader=yaml.FullLoader)\n            if os.path.isfile(config_file_path)\n            else {}\n        )\n        self.helper = OpenCTIConnectorHelper(config)\n        # Extra config\n        self.eset_api_url = get_config_variable(\n            \"ESET_API_URL\", [\"eset\", \"api_url\"], config\n        )\n        self.eset_username = get_config_variable(\n            \"ESET_USERNAME\", [\"eset\", \"username\"], config\n        )\n        self.eset_password = get_config_variable(\n            \"ESET_PASSWORD\", [\"eset\", \"password\"], config\n        )\n        self.eset_collections = get_config_variable(\n            \"ESET_COLLECTIONS\", [\"eset\", \"collections\"], config\n        )\n        self.eset_import_apt_reports = get_config_variable(\n            \"ESET_IMPORT_APT_REPORTS\",\n            [\"eset\", \"import_apt_reports\"],\n            config,\n            False,\n            True,\n        )\n        self.eset_import_start_date = get_config_variable(\n            \"ESET_IMPORT_START_DATE\",\n            [\"eset\", \"import_start_date\"],\n            config,\n        )\n        self.eset_create_observables = get_config_variable(\n            \"ESET_CREATE_OBSERVABLES\",\n            [\"eset\", \"create_observables\"],\n            config,\n        )\n        self.eset_interval = get_config_variable(\n            \"ESET_INTERVAL\", [\"eset\", \"interval\"], config, True\n        )\n        self.update_existing_data = get_config_variable(\n            \"CONNECTOR_UPDATE_EXISTING_DATA\",\n            [\"connector\", \"update_existing_data\"],\n            config,\n        )\n        self.identity = self.helper.api.identity.create(\n            type=\"Organization\",\n            name=\"ESET\",\n            description=\"ESET, s.r.o., is a software company specializing in cybersecurity.\",\n        )\n        self.added_after = int(parse(self.eset_import_start_date).timestamp())\n        # Init variables\n        self.cache = {}\n        if self.eset_collections is not None:\n            self.eset_collections = self.eset_collections.split(\",\")\n\n        # Create temporary dir and initialize logging.\n        if sys.version_info.major == 2:  # Compatibility with Python 2.7.\n            if not os.path.isdir(TMP_DIR):\n                os.makedirs(TMP_DIR)\n        else:\n            os.makedirs(TMP_DIR, exist_ok=True)\n\n    def get_interval(self):\n        return int(self.eset_interval) * 60\n\n    def _download_all_report_stuff(self, connection, report, base_path):\n        \"\"\"Download xml, pdf and adds (if available) from given *report* into paths starting with *base_path*.\"\"\"\n        for fmt in [\"pdf\", \"xml\", \"adds\"]:\n            ext = fmt if fmt != \"adds\" else \"zip\"\n            connection.get_report(report, fmt, file_path=\"{}.{}\".format(base_path, ext))\n\n    def _import_reports(self, work_id, start_epoch):\n        connection = eti_api.Connection(\n            username=self.eset_username,\n            password=self.eset_password,\n            host=\"eti.eset.com\",\n        )\n        from_date = datetime.datetime.utcfromtimestamp(start_epoch).astimezone(pytz.utc)\n        i = 0\n        for report in connection.list_reports(\n            type=\"all\", datetimefrom=from_date.isoformat()\n        ):\n            bundle_objects = []\n            if report[\"status\"] != \"finished\":\n                self.helper.log_info(\"Finished\")\n                continue  # Skip not generated reports.\n            i += 1\n            file_path = os.path.join(TMP_DIR, \"{}_{:02d}\".format(\"all\", i))\n            self._download_all_report_stuff(connection, report, file_path)\n            if os.path.isfile(file_path + \".pdf\"):\n                name = report[\"filename\"].replace(\".pdf\", \"\")\n                date = parse(report[\"date\"])\n                with open(file_path + \".pdf\", \"rb\") as f:\n                    file_data_encoded = base64.b64encode(f.read())\n                file = {\n                    \"name\": report[\"filename\"],\n                    \"data\": file_data_encoded.decode(\"utf-8\"),\n                    \"mime_type\": \"application/pdf\",\n                    \"no_trigger_import\": True,\n                }\n                stix_report = stix2.Report(\n                    id=Report.generate_id(name, date),\n                    name=name,\n                    report_types=[\"APT Report\"],\n                    description=name,\n                    published=date,\n                    labels=[\"apt\", \"eset\"],\n                    confidence=self.helper.connect_confidence_level,\n                    created_by_ref=self.identity[\"standard_id\"],\n                    object_refs=[self.identity[\"standard_id\"]],\n                    allow_custom=True,\n                    x_opencti_files=[file],\n                    object_marking_refs=[stix2.TLP_AMBER.get(\"id\")],\n                )\n                bundle_objects.append(stix_report)\n                try:\n                    self.helper.log_debug(\"Objects to be sent \" + str(bundle_objects))\n                    self.helper.send_stix2_bundle(\n                        self.helper.stix2_create_bundle(bundle_objects),\n                        update=self.update_existing_data,\n                        bypass_split=True,\n                        work_id=work_id,\n                    )\n                except Exception as e:\n                    self.helper.log_info(\"Failed to process report \" + name)\n                    self.helper.log_info(\"ERROR: \" + str(e))\n                os.remove(file_path + \".pdf\")\n                if os.path.isfile(file_path + \".xml\"):\n                    os.remove(file_path + \".xml\")\n                if os.path.isfile(file_path + \".zip\"):\n                    os.remove(file_path + \".zip\")\n\n    def _import_collection(self, collection, work_id, start_epoch):\n        object_types_with_confidence = [\n            \"attack-pattern\",\n            \"course-of-action\",\n            \"threat-actor\",\n            \"intrusion-set\",\n            \"campaign\",\n            \"malware\",\n            \"tool\",\n            \"vulnerability\",\n            \"report\",\n            \"relationship\",\n            \"indicator\",\n        ]\n        client = cabby.create_client(\n            self.eset_api_url, discovery_path=\"/taxiiservice/discovery\", use_https=True\n        )\n        client.set_auth(username=self.eset_username, password=self.eset_password)\n        no_more_result = False\n        end_epoch = start_epoch + 3600\n        while no_more_result is False:\n            self.helper.log_info(\n                \"Iterating with collection=\"\n                + str(collection)\n                + \", start_epoch=\"\n                + str(start_epoch)\n                + \", end_epoch=\"\n                + str(end_epoch)\n            )\n            begin_date = datetime.datetime.utcfromtimestamp(start_epoch).astimezone(\n                pytz.utc\n            )\n            end_date = datetime.datetime.utcfromtimestamp(end_epoch).astimezone(\n                pytz.utc\n            )\n            try:\n                for item in client.poll(\n                    collection + \" (stix2)\", begin_date=begin_date, end_date=end_date\n                ):\n                    if not item.content:  # Skip empty packages.\n                        continue\n                    parsed_content = json.loads(item.content)\n                    objects = []\n                    for object in parsed_content[\"objects\"]:\n                        if \"confidence\" in object_types_with_confidence:\n                            if \"confidence\" not in object:\n                                object[\"confidence\"] = int(\n                                    self.helper.connect_confidence_level\n                                )\n                        if object[\"type\"] == \"indicator\":\n                            object[\"name\"] = object[\"pattern\"]\n                            object[\"pattern_type\"] = \"stix\"\n                            object[\"pattern\"] = (\n                                object[\"pattern\"]\n                                .replace(\"SHA1\", \"'SHA-1'\")\n                                .replace(\"SHA256\", \"'SHA-256'\")\n                            )\n                            if self.eset_create_observables:\n                                object[\"x_opencti_create_observables\"] = (\n                                    self.eset_create_observables\n                                )\n                        objects.append(object)\n                    parsed_content[\"objects\"] = objects\n                    self.helper.send_stix2_bundle(\n                        json.dumps(parsed_content),\n                        update=self.update_existing_data,\n                        work_id=work_id,\n                    )\n            except Exception as e:\n                self.helper.log_error(str(e))\n            if end_epoch > int(time.time()):\n                no_more_result = True\n            else:\n                start_epoch = end_epoch\n                end_epoch = start_epoch + 3600\n\n    def run(self):\n        while True:\n            try:\n                self.helper.log_info(\"Synchronizing with ESET API...\")\n                timestamp = int(time.time())\n                now = datetime.datetime.utcfromtimestamp(timestamp)\n                friendly_name = \"ESET run @ \" + now.strftime(\"%Y-%m-%d %H:%M:%S\")\n                work_id = self.helper.api.work.initiate_work(\n                    self.helper.connect_id, friendly_name\n                )\n                current_state = self.helper.get_state()\n                if current_state is None:\n                    self.helper.set_state({\"last_run\": self.added_after})\n                # Get collections\n                current_state = self.helper.get_state()\n\n                if self.eset_collections is not None:\n                    for collection in self.eset_collections:\n                        self._import_collection(\n                            collection, work_id, current_state[\"last_run\"]\n                        )\n                if self.eset_import_apt_reports:\n                    self._import_reports(work_id, current_state[\"last_run\"])\n                self.helper.set_state({\"last_run\": timestamp})\n                message = \"End of synchronization\"\n                self.helper.api.work.to_processed(work_id, message)\n                self.helper.log_info(message)\n\n                if self.helper.connect_run_and_terminate:\n                    self.helper.log_info(\"Connector stop\")\n                    self.helper.force_ping()\n                    sys.exit(0)\n\n                time.sleep(self.get_interval())\n            except (KeyboardInterrupt, SystemExit):\n                self.helper.log_info(\"Connector stop\")\n                sys.exit(0)\n\n            except Exception as e:\n                self.helper.log_error(str(e))\n\n                if self.helper.connect_run_and_terminate:\n                    self.helper.log_info(\"Connector stop\")\n                    sys.exit(0)\n\n                time.sleep(60)\n\n\nif __name__ == \"__main__\":\n    try:\n        esetConnector = Eset()\n        esetConnector.run()\n    except Exception as e:\n        print(e)\n        time.sleep(10)\n        sys.exit(0)\n", "path": "external-import/eset/src/eset.py"}], "after_files": [{"content": "import base64\nimport datetime\nimport json\nimport os\nimport sys\nimport time\n\nimport cabby\nimport eti_api\nimport pytz\nimport stix2\nimport yaml\nfrom dateutil.parser import parse\nfrom pycti import (\n    Indicator,\n    Malware,\n    OpenCTIConnectorHelper,\n    Report,\n    get_config_variable,\n)\n\nTMP_DIR = \"TMP\"\n\n\nclass Eset:\n    def __init__(self):\n        # Instantiate the connector helper from config\n        config_file_path = os.path.dirname(os.path.abspath(__file__)) + \"/config.yml\"\n        config = (\n            yaml.load(open(config_file_path), Loader=yaml.FullLoader)\n            if os.path.isfile(config_file_path)\n            else {}\n        )\n        self.helper = OpenCTIConnectorHelper(config)\n        # Extra config\n        self.eset_api_url = get_config_variable(\n            \"ESET_API_URL\", [\"eset\", \"api_url\"], config\n        )\n        self.eset_username = get_config_variable(\n            \"ESET_USERNAME\", [\"eset\", \"username\"], config\n        )\n        self.eset_password = get_config_variable(\n            \"ESET_PASSWORD\", [\"eset\", \"password\"], config\n        )\n        self.eset_collections = get_config_variable(\n            \"ESET_COLLECTIONS\", [\"eset\", \"collections\"], config\n        )\n        self.eset_import_apt_reports = get_config_variable(\n            \"ESET_IMPORT_APT_REPORTS\",\n            [\"eset\", \"import_apt_reports\"],\n            config,\n            False,\n            True,\n        )\n        self.eset_import_start_date = get_config_variable(\n            \"ESET_IMPORT_START_DATE\",\n            [\"eset\", \"import_start_date\"],\n            config,\n        )\n        self.eset_create_observables = get_config_variable(\n            \"ESET_CREATE_OBSERVABLES\",\n            [\"eset\", \"create_observables\"],\n            config,\n        )\n        self.eset_interval = get_config_variable(\n            \"ESET_INTERVAL\", [\"eset\", \"interval\"], config, True\n        )\n        self.update_existing_data = get_config_variable(\n            \"CONNECTOR_UPDATE_EXISTING_DATA\",\n            [\"connector\", \"update_existing_data\"],\n            config,\n        )\n        self.identity = self.helper.api.identity.create(\n            type=\"Organization\",\n            name=\"ESET\",\n            description=\"ESET, s.r.o., is a software company specializing in cybersecurity.\",\n        )\n        self.added_after = int(parse(self.eset_import_start_date).timestamp())\n        # Init variables\n        self.cache = {}\n        if self.eset_collections is not None:\n            self.eset_collections = self.eset_collections.split(\",\")\n\n        # Create temporary dir and initialize logging.\n        if sys.version_info.major == 2:  # Compatibility with Python 2.7.\n            if not os.path.isdir(TMP_DIR):\n                os.makedirs(TMP_DIR)\n        else:\n            os.makedirs(TMP_DIR, exist_ok=True)\n\n    def get_interval(self):\n        return int(self.eset_interval) * 60\n\n    def _download_all_report_stuff(self, connection, report, base_path):\n        \"\"\"Download xml, pdf and adds (if available) from given *report* into paths starting with *base_path*.\"\"\"\n        for fmt in [\"pdf\", \"xml\", \"adds\"]:\n            ext = fmt if fmt != \"adds\" else \"zip\"\n            connection.get_report(report, fmt, file_path=\"{}.{}\".format(base_path, ext))\n\n    def _import_reports(self, work_id, start_epoch):\n        connection = eti_api.Connection(\n            username=self.eset_username,\n            password=self.eset_password,\n            host=\"eti.eset.com\",\n        )\n        from_date = datetime.datetime.utcfromtimestamp(start_epoch).astimezone(pytz.utc)\n        i = 0\n        for report in connection.list_reports(\n            type=\"all\", datetimefrom=from_date.isoformat()\n        ):\n            bundle_objects = []\n            if report[\"status\"] != \"finished\":\n                self.helper.log_info(\"Finished\")\n                continue  # Skip not generated reports.\n            i += 1\n            file_path = os.path.join(TMP_DIR, \"{}_{:02d}\".format(\"all\", i))\n            self._download_all_report_stuff(connection, report, file_path)\n            if os.path.isfile(file_path + \".pdf\"):\n                name = report[\"filename\"].replace(\".pdf\", \"\")\n                date = parse(report[\"date\"])\n                with open(file_path + \".pdf\", \"rb\") as f:\n                    file_data_encoded = base64.b64encode(f.read())\n                file = {\n                    \"name\": report[\"filename\"],\n                    \"data\": file_data_encoded.decode(\"utf-8\"),\n                    \"mime_type\": \"application/pdf\",\n                    \"no_trigger_import\": True,\n                }\n                stix_report = stix2.Report(\n                    id=Report.generate_id(name, date),\n                    name=name,\n                    report_types=[\"APT Report\"],\n                    description=name,\n                    published=date,\n                    labels=[\"apt\", \"eset\"],\n                    confidence=self.helper.connect_confidence_level,\n                    created_by_ref=self.identity[\"standard_id\"],\n                    object_refs=[self.identity[\"standard_id\"]],\n                    allow_custom=True,\n                    x_opencti_files=[file],\n                    object_marking_refs=[stix2.TLP_AMBER.get(\"id\")],\n                )\n                bundle_objects.append(stix_report)\n                try:\n                    self.helper.log_debug(\"Objects to be sent \" + str(bundle_objects))\n                    self.helper.send_stix2_bundle(\n                        self.helper.stix2_create_bundle(bundle_objects),\n                        update=self.update_existing_data,\n                        bypass_split=True,\n                        work_id=work_id,\n                    )\n                except Exception as e:\n                    self.helper.log_info(\"Failed to process report \" + name)\n                    self.helper.log_info(\"ERROR: \" + str(e))\n                os.remove(file_path + \".pdf\")\n                if os.path.isfile(file_path + \".xml\"):\n                    os.remove(file_path + \".xml\")\n                if os.path.isfile(file_path + \".zip\"):\n                    os.remove(file_path + \".zip\")\n\n    def _import_collection(self, collection, work_id, start_epoch):\n        object_types_with_confidence = [\n            \"attack-pattern\",\n            \"course-of-action\",\n            \"threat-actor\",\n            \"intrusion-set\",\n            \"campaign\",\n            \"malware\",\n            \"tool\",\n            \"vulnerability\",\n            \"report\",\n            \"relationship\",\n            \"indicator\",\n        ]\n        client = cabby.create_client(\n            self.eset_api_url, discovery_path=\"/taxiiservice/discovery\", use_https=True\n        )\n        client.set_auth(username=self.eset_username, password=self.eset_password)\n        no_more_result = False\n        end_epoch = start_epoch + 3600\n        while no_more_result is False:\n            self.helper.log_info(\n                \"Iterating with collection=\"\n                + str(collection)\n                + \", start_epoch=\"\n                + str(start_epoch)\n                + \", end_epoch=\"\n                + str(end_epoch)\n            )\n            begin_date = datetime.datetime.utcfromtimestamp(start_epoch).astimezone(\n                pytz.utc\n            )\n            end_date = datetime.datetime.utcfromtimestamp(end_epoch).astimezone(\n                pytz.utc\n            )\n            try:\n                for item in client.poll(\n                    collection + \" (stix2)\", begin_date=begin_date, end_date=end_date\n                ):\n                    if not item.content:  # Skip empty packages.\n                        continue\n                    parsed_content = json.loads(item.content)\n                    objects = []\n                    id_remaps = {}\n                    removed_ids = set()\n                    for object in parsed_content[\"objects\"]:\n                        if \"confidence\" in object_types_with_confidence:\n                            if \"confidence\" not in object:\n                                object[\"confidence\"] = int(\n                                    self.helper.connect_confidence_level\n                                )\n                        # Don't consume identity entities w/ \"customer\" as the name.\n                        # ESET uses this to indicate country targeting, and consuming\n                        # these causes problems due to dedupe.\n                        # TODO: Convert these & relevant relationship refs to country\n                        # locations.\n                        if (\n                            object[\"type\"] == \"identity\"\n                            and \"name\" in object\n                            and object[\"name\"] == \"customer\"\n                        ) or object[\"type\"] == \"observed-data\":\n                            removed_ids.add(object[\"id\"])\n                            continue\n\n                        # Malware STIX IDs need to be manually recomputed so they're\n                        # deterministic by malware name\n                        if object[\"type\"] == \"malware\" and \"name\" in object:\n                            new_id = Malware.generate_id(object[\"name\"])\n                            if object[\"id\"] in id_remaps:\n                                new_id = id_remaps[object[\"id\"]]\n                            else:\n                                id_remaps[object[\"id\"]] = new_id\n                            object[\"id\"] = new_id\n\n                        # If we remapped a STIX id earlier to a pycti one, we need  to\n                        # reflect that properly in any relevant relationship too\n                        if object[\"type\"] == \"relationship\":\n                            if \"source_ref\" in object:\n                                if object[\"source_ref\"] in removed_ids:\n                                    continue  # skip relationship if either ref is in removed_ids\n                                if object[\"source_ref\"] in id_remaps:\n                                    object[\"source_ref\"] = id_remaps[\n                                        object[\"source_ref\"]\n                                    ]\n                            if \"target_ref\" in object:\n                                if object[\"target_ref\"] in removed_ids:\n                                    continue  # skip relationship if either ref is in removed_ids\n                                if object[\"target_ref\"] in id_remaps:\n                                    object[\"target_ref\"] = id_remaps[\n                                        object[\"target_ref\"]\n                                    ]\n\n                        if object[\"type\"] == \"indicator\":\n                            object[\"name\"] = object[\"pattern\"]\n                            object[\"pattern_type\"] = \"stix\"\n                            object[\"pattern\"] = (\n                                object[\"pattern\"]\n                                .replace(\"SHA1\", \"'SHA-1'\")\n                                .replace(\"SHA256\", \"'SHA-256'\")\n                            )\n                            new_id = Indicator.generate_id(object[\"pattern\"])\n                            if object[\"id\"] in id_remaps:\n                                new_id = id_remaps[object[\"id\"]]\n                            else:\n                                id_remaps[object[\"id\"]] = new_id\n                            object[\"id\"] = new_id\n                            if self.eset_create_observables:\n                                object[\"x_opencti_create_observables\"] = (\n                                    self.eset_create_observables\n                                )\n                        objects.append(object)\n                    parsed_content[\"objects\"] = objects\n                    self.helper.send_stix2_bundle(\n                        json.dumps(parsed_content),\n                        update=self.update_existing_data,\n                        work_id=work_id,\n                    )\n            except Exception as e:\n                self.helper.log_error(str(e))\n            if end_epoch > int(time.time()):\n                no_more_result = True\n            else:\n                start_epoch = end_epoch\n                end_epoch = start_epoch + 3600\n\n    def run(self):\n        while True:\n            try:\n                self.helper.log_info(\"Synchronizing with ESET API...\")\n                timestamp = int(time.time())\n                now = datetime.datetime.utcfromtimestamp(timestamp)\n                friendly_name = \"ESET run @ \" + now.strftime(\"%Y-%m-%d %H:%M:%S\")\n                work_id = self.helper.api.work.initiate_work(\n                    self.helper.connect_id, friendly_name\n                )\n                current_state = self.helper.get_state()\n                if current_state is None:\n                    self.helper.set_state({\"last_run\": self.added_after})\n                # Get collections\n                current_state = self.helper.get_state()\n\n                if self.eset_collections is not None:\n                    for collection in self.eset_collections:\n                        self._import_collection(\n                            collection, work_id, current_state[\"last_run\"]\n                        )\n                if self.eset_import_apt_reports:\n                    self._import_reports(work_id, current_state[\"last_run\"])\n                self.helper.set_state({\"last_run\": timestamp})\n                message = \"End of synchronization\"\n                self.helper.api.work.to_processed(work_id, message)\n                self.helper.log_info(message)\n\n                if self.helper.connect_run_and_terminate:\n                    self.helper.log_info(\"Connector stop\")\n                    self.helper.force_ping()\n                    sys.exit(0)\n\n                time.sleep(self.get_interval())\n            except (KeyboardInterrupt, SystemExit):\n                self.helper.log_info(\"Connector stop\")\n                sys.exit(0)\n\n            except Exception as e:\n                self.helper.log_error(str(e))\n\n                if self.helper.connect_run_and_terminate:\n                    self.helper.log_info(\"Connector stop\")\n                    sys.exit(0)\n\n                time.sleep(60)\n\n\nif __name__ == \"__main__\":\n    try:\n        esetConnector = Eset()\n        esetConnector.run()\n    except Exception as e:\n        print(e)\n        time.sleep(10)\n        sys.exit(0)\n", "path": "external-import/eset/src/eset.py"}]}