<jupyter_start><jupyter_text>Creating a Kobe Twitter Botby <NAME>*Since: August 24th, 2020*Using gpt-2-simple, I refined OpenAI's Generative Pre-trained Transformer 2 (GPT) by training the model on Kobe's Tweets. Using TWINT, I scraped all of Kobe's tweets since he first came onto Twitter until his infamous last tweet about <NAME>.<jupyter_code>%tensorflow_version 1.x
!pip install -q gpt-2-simple
import gpt_2_simple as gpt2
from datetime import datetime
from google.colab import files<jupyter_output>TensorFlow 1.x selected.
  Building wheel for gpt-2-simple (setup.py) ... [?25l[?25hdone
WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.<jupyter_text>GPUVerifying Which GPU is in use - Tesla T4 is better than the P100<jupyter_code>!nvidia-smi<jupyter_output>Tue Aug 25 17:10:36 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.57       Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |
| N/A   36C    P8    28W / 149W |      0MiB / 11441MiB |      0%      Default |
|                               |                      |                 ERR! |
+-------------------------------+----------------------+----------------------+
                                                                               
+-------[...]<jupyter_text>Downloading GPT-2There are three released sizes of GPT-2:* `124M` (default): the "small" model, 500MB on disk.* `355M`: the "medium" model, 1.5GB on disk.* `774M`: the "large" model, cannot currently be finetuned with Colaboratory but can be used to generate text from the pretrained model (see later in Notebook)* `1558M`: the "extra large", true model. Will not work if a K80 GPU is attached to the notebook. (like `774M`, it cannot be finetuned).Larger models have more knowledge, but take longer to finetune and longer to generate text.<jupyter_code>gpt2.download_gpt2(model_name="124M")<jupyter_output>Fetching checkpoint: 1.05Mit [00:00, 190Mit/s]                                                      
Fetching encoder.json: 1.05Mit [00:00, 62.5Mit/s]                                                   
Fetching hparams.json: 1.05Mit [00:00, 433Mit/s]                                                    
Fetching model.ckpt.data-00000-of-00001: 498Mit [00:06, 74.1Mit/s]                                  
Fetching model.ckpt.index: 1.05Mit [00:00, 440Mit/s]                                                
Fetching model.ckpt.meta: 1.05Mit [00:00, 113Mit/s]                                                 
Fetching vocab.bpe: 1.05Mit [00:00, 127Mit/s]<jupyter_text>Mounting Google DriveThe best way to get input text to-be-trained into the Colaboratory VM, and to get the trained model *out* of Colaboratory, is to route it through Google Drive *first*.Running this cell (which will only work in Colaboratory) will mount your personal Google Drive in the VM, which later cells can use to get data in/out. (it will ask for an auth code; that auth is not saved anywhere)<jupyter_code>gpt2.mount_gdrive()<jupyter_output>Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly&response_type=code

Enter your authorization code:
··········
Mounted at /content/drive<jupyter_text>Uploading a Text File to be Trained to ColaboratoryIn the Colaboratory Notebook sidebar on the left of the screen, select *Files*. From there you can upload files:![alt text](https://i.imgur.com/TGcZT4h.png)Upload **any smaller text file**  (<10 MB) and update the file name in the cell below, then run the cell.<jupyter_code>file_name = "kobe.txt"<jupyter_output><empty_output><jupyter_text>If your text file is larger than 10MB, it is recommended to upload that file to Google Drive first, then copy that file from Google Drive to the Colaboratory VM.<jupyter_code>gpt2.copy_file_from_gdrive(file_name)<jupyter_output><empty_output><jupyter_text>Finetune GPT-2The next cell will start the actual finetuning of GPT-2. It creates a persistent TensorFlow session which stores the training config, then runs the training for the specified number of `steps`. (to have the finetuning run indefinitely, set `steps = -1`)The model checkpoints will be saved in `/checkpoint/run1` by default. The checkpoints are saved every 500 steps (can be changed) and when the cell is stopped.The training might time out after 4ish hours; make sure you end training and save the results so you don't lose them!**IMPORTANT NOTE:** If you want to rerun this cell, **restart the VM first** (Runtime -> Restart Runtime). You will need to rerun imports but not recopy files.Other optional-but-helpful parameters for `gpt2.finetune`:*  **`restore_from`**: Set to `fresh` to start training from the base GPT-2, or set to `latest` to restart training from an existing checkpoint.* **`sample_every`**: Number of steps to print example output* **`print_every`**: Number of steps to print training progress.* **`learning_rate`**:  Learning rate for the training. (default `1e-4`, can lower to `1e-5` if you have <1MB input data)*  **`run_name`**: subfolder within `checkpoint` to save the model. This is useful if you want to work with multiple models (will also need to specify  `run_name` when loading the model)* **`overwrite`**: Set to `True` if you want to continue finetuning an existing model (w/ `restore_from='latest'`) without creating duplicate copies.<jupyter_code>sess = gpt2.start_tf_sess()

gpt2.finetune(sess,
              dataset=file_name,
              model_name='124M',
              steps=1000,
              restore_from='latest',
              run_name='run1',
              print_every=10,
              sample_every=200,
              save_every=500,
              overwrite=True
              )<jupyter_output>WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/gpt_2_simple/src/sample.py:17: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
Loading checkpoint checkpoint/run1/model-706
INFO:tensorflow:Restoring parameters from checkpoint/run1/model-706<jupyter_text>After the model is trained, you can copy the checkpoint folder to your own Google Drive.If you want to download it to your personal computer, it's strongly recommended you copy it there first, then download from Google Drive. The checkpoint folder is copied as a `.rar` compressed file; you can download it and uncompress it locally.<jupyter_code>gpt2.copy_checkpoint_to_gdrive(run_name='run1')<jupyter_output><empty_output><jupyter_text>You're done! Feel free to go to the **Generate Text From The Trained Model** section to generate text based on your retrained model.  Load a Trained Model CheckpointRunning the next cell will copy the `.rar` checkpoint file from your Google Drive into the Colaboratory VM.<jupyter_code>gpt2.copy_checkpoint_from_gdrive(run_name='run1')<jupyter_output><empty_output><jupyter_text>The next cell will allow you to load the retrained model checkpoint + metadata necessary to generate text.**IMPORTANT NOTE:** If you want to rerun this cell, **restart the VM first** (Runtime -> Restart Runtime). You will need to rerun imports but not recopy files.<jupyter_code>sess = gpt2.start_tf_sess()
gpt2.load_gpt2(sess, run_name='run1')<jupyter_output>Loading checkpoint checkpoint/run1/model-848
INFO:tensorflow:Restoring parameters from checkpoint/run1/model-848<jupyter_text>Generate Text From The Trained ModelAfter you've trained the model or loaded a retrained model from checkpoint, you can now generate text. `generate` generates a single text from the loaded model.<jupyter_code>gpt2.generate(sess, run_name='run1')<jupyter_output>@PhilJackson11 I'm thinking of writing a novel about retiring from basketball and founding a company…  http://instagram.com/p/l6yTNrUVjd/ 
 @PhilJackson11 I've been thinking about writing a novel for a while now but didnt know I was writing it lol I'm not a bad kid hey are you ready to play yet?
 Thinking about writing a novelisation of #ThePunies #GameSetTellerforyou #TeamUSA #LakerNation
 A #ThankYouMLK50 march is about more than just seeing shoes fitted. It's about being represented in the highest level. It's about being a part of the change. Change is coming. Watch Episode 4 here:   http://es.pn/2J1XR7H 
 Lesson 1: Love the hate. #HeroVillain #KOBE11
 I love the trash talk. #ambien #mambaout
 This is why I partner with Lenovo #LegacyandtheQueen #Kobe11
 #TMT:  http://www.youtube.com/watch?v=t8ZtgNuZDM ” by far the craziest thing I've done! I have no idea how I walked after that #real***t
 @PhilJackson11 ist really only funny 8to9 years but it will get better. I'm thinking 9.5 to 1[...]<jupyter_text>If you're creating an API based on your model and need to pass the generated text elsewhere, you can do `text = gpt2.generate(sess, return_as_list=True)[0]`You can also pass in a `prefix` to the generate function to force the text to start with a given character sequence and generate text from there (good if you add an indicator when the text starts).You can also generate multiple texts at a time by specifing `nsamples`. Unique to GPT-2, you can pass a `batch_size` to generate multiple samples in parallel, giving a massive speedup (in Colaboratory, set a maximum of 20 for `batch_size`).Other optional-but-helpful parameters for `gpt2.generate` and friends:*  **`length`**: Number of tokens to generate (default 1023, the maximum)* **`temperature`**: The higher the temperature, the crazier the text (default 0.7, recommended to keep between 0.7 and 1.0)* **`top_k`**: Limits the generated guesses to the top *k* guesses (default 0 which disables the behavior; if the generated output is super crazy, you may want to set `top_k=40`)* **`top_p`**: Nucleus sampling: limits the generated guesses to a cumulative probability. (gets good results on a dataset with `top_p=0.9`)* **`truncate`**: Truncates the input text until a given sequence, excluding that sequence (e.g. if `truncate=''`, the returned text will include everything before the first ``). It may be useful to combine this with a smaller `length` if the input texts are short.*  **`include_prefix`**: If using `truncate` and `include_prefix=False`, the specified `prefix` will not be included in the returned text.<jupyter_code>gpt2.generate(sess,
              length=250,
              temperature=0.7,
              prefix="@KingJames",
              nsamples=5,
              batch_size=5
              )<jupyter_output>@KingJames #countonkobe
 Watching @TurnerSportsEJ and the crew do this #allstarchallenge is hilarious Not a good case for nba players being the best athletes!  HA
 #QueenMamba ladyvb24 Celebrate the one you love #myvalentine happy valentines day to all #blessings  http://instagram.com/p/kbG3a8RNqV/ 
 Happy birthday Mr Russell. Thank you for all of your wisdom and the amount of time you have taken to…  http://instagram.com/p/kWHezyRNlF/ 
 On my Nike set with the champ rsherman_25 #differentanimalSamebeast  http://instagram.com/p/kQdrA8xNiv/ 
 Major shout to @sagekotsenburg @Jme_Anderson #snowboardGold #usa #SochiStomped
 Tonight, watch @BillClinton, @AllysonFelix, @TheRealMattKemp & myself discuss why kids in sports is so important. http://es.pn/KidsAndSports 
 What a game! #seriously
====================
@KingJames #GOAT
 Catch the short film I Just Metagy feat. Demi Lovato and Dina is dancing! It’s Disney’s masterpiece and one of my favorite videos. https://youtu.be/oVK6YReKeDM ” (  h[...]<jupyter_text>For bulk generation, you can generate a large amount of text to a file and sort out the samples locally on your computer. The next cell will generate a generated text file with a unique timestamp.You can rerun the cells as many times as you want for even more generated texts!<jupyter_code>gen_file = 'gpt2_gentext_{:%Y%m%d_%H%M%S}.txt'.format(datetime.utcnow())

gpt2.generate_to_file(sess,
                      destination_path=gen_file,
                      length=500,
                      temperature=0.7,
                      nsamples=100,
                      batch_size=20
                      )
# may have to run twice to get file to download
files.download(gen_file)<jupyter_output><empty_output><jupyter_text>EtcIf the notebook has errors (e.g. GPU Sync Fail), force-kill the Colaboratory virtual machine and restart it with the command below:<jupyter_code>!kill -9 -1<jupyter_output><empty_output><jupyter_text>LICENSEMIT LicenseCopyright (c) 2019 <NAME>Permission is hereby granted, free of charge, to any person obtaining a copyof this software and associated documentation files (the "Software"), to dealin the Software without restriction, including without limitation the rightsto use, copy, modify, merge, publish, distribute, sublicense, and/or sellcopies of the Software, and to permit persons to whom the Software isfurnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in allcopies or substantial portions of the Software.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS ORIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THEAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHERLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THESOFTWARE.<jupyter_code><jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup