# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: conda_python3
#     language: python
#     name: conda_python3
# ---

# # Generate conversation IDs for other EMIS issues
#
# Produce a sample of 20 conversation IDs where:
# - EMIS is the sending practice, and the failure is due to 'request not acknowledged'
# - EMIS is the sending practice, and the failure is due to 'Large messages rejected due to timeout duration reached of overall transfer'
#

import pandas as pd
import numpy as np

transfers = pd.read_parquet("s3://prm-gp2gp-transfer-data-dev/14-day-cutoff/v4/2021/6/transfers.parquet")

emis_sending = transfers["sending_supplier"] == "EMIS"

# ## EMIS is the sending practice, and the failure is due to 'request not acknowledged'

request_not_acknowledged = transfers["failure_reason"] == "Request not Acknowledged"
transfers[emis_sending & request_not_acknowledged].sample(20)

# Looking at these example in the SPINE logs, most of these seem to be the cases where the only message in the converation was the initial request for GP2GP. (i.e the request was never responded to).
#
# However, there seem to be a handful of other cases mixed in. The common theme for these cases is that the sender responded to the request but only after a very long time.
# - `9383F0E8-7BFE-4100-ADF6-16EDDDE157BA` - did eventually do a full transfer, but request was only responded to *after about a month*
# - `D89D0284-E042-486F-9C38-48620F37C2B2` - as above
# - `C87B9F90-9D2E-43D1-81DD-02114C0F0FF5` - did eventually respond with an error code around a month later
# - `338E9790-D341-11EB-ACC7-9B0E52FFDA17` - did eventually send an EHR and ack the request, but only after several weeks 
# - `3FD20898-960C-492D-A328-C7DC26E38D32` - did eventually send an EHR and ack the request, but only after a couple of weeks
#
# Finally this conversation was a bit difficult to classify:
# - `16B6FB75-40EA-4E71-9D8C-42E0331E1BCF` - record was sent on time and looks to be integrated, but possibly interpreted incorrectly as request acknowledgement appeared to be missing from the SPINE logs

# ## EMIS is the sending practice, and the failure is due to transfer timeout

#Error code 25: Large messages rejected due to timeout duration reached of overall transfer
transfer_timeout = transfers["final_error_codes"].apply(lambda errors: 25 in errors)
transfers[emis_sending & transfer_timeout].sample(20)

# Looking at the SPINE logs for these conversation, this sample presents an interesting cross section of issues.
#
# There are threee failure patterns that appeared more than once.
#
# Requestor sent a COPC continue but large fragment COPCs appear to have never been sent
# - `D4BF64A6-4D57-4E5E-AD02-DD17CBFD2203`
# - `633A3CB6-BA1F-41D8-928D-D28F459787AF`
# - `46DB3D51-06D0-4938-A9C4-4344471F1373`
# - `CF26FA43-89AA-4D24-BEE9-08C2963BA830`
# - `82E4DEE5-A36A-46F6-A33B-DB767BEF48DA`
# - `40E02F24-81AA-4305-8165-B7E79A02552A`
# - `E504669F-F8F0-4D8B-94DA-0A28C3D8AEA9`
# - `72AF37B9-4D6B-405A-ADA5-F86308835183` (EHR was resent a couple of weeks after initial request but stalled again in the same way)
#
# Started sending, messages stop, then some days later the requestor reports a timeout. No more messages are sent.
# - `4AB02368-C3A8-491A-BF80-66D5916B6E70`
# - `F793CB12-F42E-4F3C-B9FC-4BEA28579EBA`
# - `1B7DFC3B-61A8-43A1-A6D0-9B4E09770C9E`
# - `763F58D6-AB76-4F8C-A4EE-9D6E6C91043C`
# - `CCD1C6F9-198C-4BBE-8779-AE2DFAADDA13`
# - `AB28FF0A-9FFD-4A2C-A9A4-A3607099A3D8`
# - `6E5D186A-48D6-4806-8987-21AF72081948` (also had a duplicate ack of request by sender half way though transfer)
#
# Transfer seemed to run into spine errors / large message re-assembly failure, several days later the requestor reports timeout.
# - `99D0B87D-CC06-4F97-B0D5-56CCEBBB6F6C`
# - `FAE4DC12-44DF-4FF0-8732-81E03164CF25`
#
# There were also the following cases that were unique in our sample:  
#
# `E5E294EE-7F93-4C74-8AE0-B4ACA4E4E06E`- Started sending, messages stop, around a week later the requestor reports a timeout. However, the sender then retransmits the EHR extract and the process appears to start again, running to completion. By the SLA definition this was interpreted as being an integration on time, but in reality the records was transferred after we expect the paper records to be printed.
#
# `925A2F8B-E037-45AC-B11F-D706A9CE325A` - The EHR is sent but the requestor (also EMIS) appeared to not issue a COPC continue message… and then reports a timeout error about a day later. Then around several weeks later, the EHR is resent and this is integrated successfully (but again, this is very late - so we expect the paper to have been printed already)
#
# `5741F734-7A60-49EF-ADE9-8C8ED4F6BABE` - transfer seems too stall with no error code, around a week later the EHR is resent multiple times, some of these are acknowledged as a duplicate by the requester, but the transfer appears to stall again and a couple of days later the requestor reports a timeout.

# ## EMIS is the sending practice, and the failure is due to transfer timeout (simpler examples)

single_positive_sender_ack = transfers["sender_error_codes"].apply(lambda errors: tuple(np.isnan(e) for e in errors)) == (True,)
only_transfer_timeout = transfers["final_error_codes"].apply(set) == {25}
no_intermediate_error_codes = transfers["intermediate_error_codes"].apply(len) == 0
transfers[emis_sending & only_transfer_timeout & no_intermediate_error_codes & single_positive_sender_ack].sample(20)

# ## Appendix - error code 25 only appears in final error codes column

transfers["final_error_codes"].explode().value_counts()

transfers["sender_error_codes"].explode().value_counts()

transfers["intermediate_error_codes"].explode().value_counts()

StarcoderdataJupyterScriptsDedupFiltered