<jupyter_start><jupyter_text>Assembling the full dataset<jupyter_code>dset = extract_dset(PATH_FULL + 'pairs.txt')
construct_analogy_dset(dset, PATH_FULL + 'tetrads_sub_all.txt', PATH_FULL + 'tetrads_nonsub_all.txt')<jupyter_output><empty_output><jupyter_text>Assembling the dataset with OOV words removed<jupyter_code>model = KeyedVectors.load_word2vec_format('models/morphoseg/model.txt')
# removing OOV words
clean_dset(PATH_FULL + 'pairs.txt', PATH_OOV_REMOVED + 'pairs_cleaned.txt', model)
dset = extract_dset(PATH_OOV_REMOVED + 'pairs_cleaned.txt')
construct_analogy_dset(dset, PATH_OOV_REMOVED + 'tetrads_sub.txt', PATH_OOV_REMOVED + 'tetrads_nonsub.txt')
# in nonsub
# REMOVE AFTER TESTING

filein = open('final_analogies/tetrads_nonsub.txt', 'r')
l = filein.readline()
cnt = 0
while l != '':
    cnt += 1
    l = filein.readline()
    
filein.close()

print(cnt)
# in sub
# REMOVE AFTER TESTING

filein = open('final_analogies/tetrads_sub.txt', 'r')
l = filein.readline()
cnt = 0
while l != '':
    cnt += 1
    l = filein.readline()
    
filein.close()

print(cnt)
1576 + 794
cnts = []
for rel_id in dset:
    cnts.append(sum([len(dset[rel_id][k]) for k in dset[rel_id]]))
    
sum([x * (x-1)/2 for x in cnts])
# train/test split
sub = extract_analogy_dset(PATH_OOV_REMOVED + 'tetrads_sub.txt')
nonsub = extract_analogy_dset(PATH_OOV_REMOVED + 'tetrads_nonsub.txt')
train_test_split(sub, PATH_OOV_REMOVED + 'train_sub.txt', PATH_OOV_REMOVED + 'test_sub.txt', p=0.75)
train_test_split(nonsub, PATH_OOV_REMOVED + 'train_nonsub.txt', PATH_OOV_REMOVED + 'test_nonsub.txt', p=0.75)<jupyter_output>Id 0 train 5 test 2
Id 1 train 265 test 89
Id 3 train 4 test 2
Id 4 train 39 test 13
Id 5 train 2 test 1
Id 9 train 3 test 2
Id 10 train 102 test 34
Id 11 train 128 test 43
Id 12 train 142 test 48
Id 13 train 263 test 88
Id 18 train 33 test 12
Id 21 train 90 test 30
Id 22 train 102 test 34
Id 0 train 168 test 56
Id 1 train 60 test 21
Id 2 train 21 test 7
Id 3 train 6 test 3
Id 4 train 134 test 45
Id 5 train 5 test 2
Id 6 train 21 test 7
Id 7 train 4 test 2
Id 8 train 4 test 2
Id 9 train 169 test 57
StarcoderdataJupyterStructuredCleanDedup