<jupyter_start><jupyter_text>Creating data set for splitting that into Train, Validation and Test<jupyter_code>X = pd.DataFrame(np.random.randint(100, size = (100, 2)), columns = ['x', 'y'])
X.index = [pd.Timestamp('20200312010000') + pd.Timedelta(minutes = i * 2) for i in range(X.shape[0])]
X.shape<jupyter_output><empty_output><jupyter_text>Creating labels for the data set<jupyter_code>y = pd.Series(np.random.randint(2, size = X.shape[0]), index = X.index)
y<jupyter_output><empty_output><jupyter_text>Splitting data setHere the data set will be splitted into 3 sections popularly known as train, validation and test. Test is optional though.The idea is that according to temporal order, train data comes first, after that there may be a gap which will be defined by the parameter 'dist' and then comes validation data and again there is a same amount of gap and then comes test data.  Initialization parameters for SplitDataset classdata: pandas dataframe or series or numpy array of dimension samples X other dimensions with rank 2 or higherlabel: pandas dataframe or series or numpy array with dimension samples X classes with rank 2index : numpy array, we can explicitly insert indices of the data set and labelstest_ratio : float, ratio of test datanp_seed : int, numpy seed used for random shufflesame_ratio : bool, should be only true if the labels are classification labels and we want to keep the test and validation ratio same for all classes (it doesnt have any use in sequence split)make_onehot : bool, if True, the labels will be converted into one hot encoded format<jupyter_code># defining object as splitter
splitter = msd.SplitDataset(X, y, test_ratio = .1)<jupyter_output><empty_output><jupyter_text>Parameters for sequence_split functionseq_len : int, length of each sequenceval_ratio : float, validation data set ratio, deafult is .15data_stride : int, indicates the number of shift between two adjacent example data to prepare the data set, default is 1label_shift : temporal difference between the label and the corresponding data's last time step, default is 1split_method : {'train_val', 'multiple_train_val'}, default is 'multiple_train_val'sec : int or pandas dataframe with columns naming 'start' and 'stop', number of sections for multiple_train_val method splitting (basically train_val splitting is multiple_train_val method with sec = 1), deafult is 1dist : int, number of data to be leftover between two adjacent sec, default is 0<jupyter_code># applying sequence split
data = splitter.sequence_split(seq_len = 10, val_ratio = .2, data_stride = 2, label_shift = 3, 
                               split_method = 'train_val', sec = 1, dist = 1)<jupyter_output><empty_output><jupyter_text>Checking output of split function<jupyter_code>data.keys()
data['train'].keys()
data['validation']['index']
# Accumulating data and label in order to understand the splitting easily
X['label'] = y
X.reset_index().values
for see in sees:
    print('\n############## %s #############\n'%see)
    for i in range(data[see]['data'].shape[0]):
        print(data[see]['data'][i], data[see]['label'][i], data[see]['index'][i])<jupyter_output>############## train #############

[[12 97]
 [28 88]
 [99 36]
 [28 52]
 [58 63]
 [56 85]
 [ 5 92]
 [87 18]
 [70 46]
 [30  8]] [1] 2020-03-12 01:20:00
[[99 36]
 [28 52]
 [58 63]
 [56 85]
 [ 5 92]
 [87 18]
 [70 46]
 [30  8]
 [36 85]
 [54 64]] [0] 2020-03-12 01:24:00
[[58 63]
 [56 85]
 [ 5 92]
 [87 18]
 [70 46]
 [30  8]
 [36 85]
 [54 64]
 [26 81]
 [83  7]] [0] 2020-03-12 01:28:00
[[ 5 92]
 [87 18]
 [70 46]
 [30  8]
 [36 85]
 [54 64]
 [26 81]
 [83  7]
 [ 5 13]
 [53 65]] [0] 2020-03-12 01:32:00
[[70 46]
 [30  8]
 [36 85]
 [54 64]
 [26 81]
 [83  7]
 [ 5 13]
 [53 65]
 [72 50]
 [ 3 71]] [0] 2020-03-12 01:36:00
[[36 85]
 [54 64]
 [26 81]
 [83  7]
 [ 5 13]
 [53 65]
 [72 50]
 [ 3 71]
 [70 19]
 [12 67]] [1] 2020-03-12 01:40:00
[[26 81]
 [83  7]
 [ 5 13]
 [53 65]
 [72 50]
 [ 3 71]
 [70 19]
 [12 67]
 [90 75]
 [46 51]] [1] 2020-03-12 01:44:00
[[ 5 13]
 [53 65]
 [72 50]
 [ 3 71]
 [70 19]
 [12 67]
 [90 75]
 [46 51]
 [40 98]
 [39  6]] [1] 2020-03-12 01:48:00
[[72 50]
 [ 3 71]
 [70 19]
 [12 67]
 [90 75[...]
StarcoderdataJupyterStructuredCleanDedup