<jupyter_start><jupyter_text>pytti: python text to image---This is a closed beta. Leak it if you must, information wants to be free.[pytti is made possible by supporters like you.](https://www.patreon.com/sportsracer48) [Thank you.](https://www.youtube.com/watch?v=TexDW6nEhgU)<jupyter_code># @title Licensed under the MIT License
# Copyleft (c) 2021 <NAME>

# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:

# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.

# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.<jupyter_output><empty_output><jupyter_text>Instructions `scenes:` Descriptions of scenes you want generated, separated by `||`. Each scene can contain multiple prompts, separated by `|`.*Example:* `Winter sunrise | icy landscape || Winter day | snowy skyline || Winter sunset | chilly air || Winter night | clear sky` would go through several winter scenes.**Advanced:** weight prompts with `description:weight`. Higher `weight` values will be prioritized by the optimizer, and negative `weight` values will remove the description from the image. The default weight is $1$. Weights can also be functions of $t$ to change over the course of an animation.*Example scene:* `blue sky:10|martian landscape|red sky:-1` would try to turn the martian sky blue.**Advanced:** stop prompts once the image matches them sufficiently with `description:weight:stop`. `stop` should be between $0$ and $1$ for positive prompts, or between $-1$ and $0$ for negative prompts. Lower `stop` values will have more effect on the image (remember that $-1<-0.5<0$). A prompt with a negative `weight` will often go haywire without a stop. Stops can also be functions of $t$ to change over the course of an animation.*Example scene:* `Feathered dinosaurs|birds:1:0.87|scales:-1:-.9|text:-1:-.9` Would try to make feathered dinosaurs, lightly like birds, without scales or text, but without making 'anti-scales' or 'anti-text.'**NEW:****Advanced:** Use `description:weight_mask description` with a text prompt as `mask`. The prompt will only be applied to areas of the image that match `mask description` according to CLIP.*Example scene:* `Khaleesi Daenerys Targaryen | mother of dragons | dragon:3_baby` would only apply the weight `dragon` to parts of the image that match `baby`, thus turning the babies that `mother` tends to make into dragons (hopefully).**Advanced:** Use `description:weight_[mask]` with a URL or path to an image, or a path to a .mp4 video to use as a `mask`. The prompt will only be applied to the masked (white) areas of the mask image. Use `description:weight_[-mask]` to apply the prompt to the black areas instead.*Example scene:* `sunlight:3_[mask.mp4]|midnight:3_[-mask.mp4]` Would apply `sunlight` in the white areas of `mask.mp4`, and `midnight` in the black areas.**Legacy:** Directional weights will still work as before, but they aren't as good as masks.**Advanced:** Use `[path or url]` as a prompt to add a semantic image prompt. This will be read by CLIP and understood as a near perfect text description of the image.*Example scene:* `[artist signature.png]:-1:-.95|[https://i.redd.it/ewpeykozy7e71.png]:3|fractal clouds|hole in the sky`---`scene_prefix:` text prepended to the beginning of each scene.*Example:* `Trending on Arstation|``scene_suffix:` text appended to the end of each scene.*Example:* ` by <NAME>urney``interpolation_steps:` number of steps to spend smoothly transitioning from the last scene at the start of each scene. $200$ is a good default. Set to $0$ to disable.`steps_per_scene:` total number of steps to spend rendering each scene. Should be at least `interpolation_steps`. This will indirectly control the total length of an animation.---**NEW**: `direct_image_prompts:` paths or urls of images that you want your image to look like in a literal sense, along with `weight_mask` and `stop` values, separated by `|`.Apply masks to direct image prompts with `path or url of image:weight_path or url of mask` For video masks it must be a path to an mp4 file.**Legacy** latent image prompts are no more. They are now rolled into direct image prompts.---`init_image:` path or url of start image. Works well for creating a central focus.`direct_init_weight:` Defaults to $0$. Use the initial image as a direct image prompt. Equivalent to adding `init_image:direct_init_weight` as a `direct_image_prompt`. Supports weights, masks, and stops.`semantic_init_weight:` Defaults to $0$. Defaults to $0$. Use the initial image as a semantic image prompt. Equivalent to adding `[init_image]:direct_init_weight` as a prompt to each scene in `scenes`. Supports weights, masks, and stops. **IMPORTANT** since this is a semantic prompt, you still need to put the mask in `[` `]` to denote it as a path or url, otherwise it will be read as text instead of a file.---`width`, `height:` image size. Set one of these $-1$ to derive it from the aspect ratio of the init image.`pixel_size:` integer image scale factor. Makes the image bigger. Set to $1$ for VQGAN or face VRAM issues.`smoothing_weight:` makes the image smoother. Defaults to $0$ (no smoothing). Can also be negative for that deep fried look.`image_model:` select how your image will be represented.`vqgan_model:` select your VQGAN version (only for `image_model: VQGAN`)`random_initial_palette:` if checked, palettes will start out with random colors. Otherwise they will start out as grayscale. (only for `image_model: Limited Palette`)`palette_size:` number of colors in each palette. (only for `image_model: Limited Palette`)`palettes:` total number of palettes. The image will have `palette_size*palettes` colors total. (only for `image_model: Limited Palette`)`gamma:` relative gamma value. Higher values make the image darker and higher contrast, lower values make the image lighter and lower contrast. (only for `image_model: Limited Palette`). $1$ is a good default.`hdr_weight:` how strongly the optimizer will maintain the `gamma`. Set to $0$ to disable. (only for `image_model: Limited Palette`)`palette_normalization_weight:` how strongly the optimizer will maintain the palettes' presence in the image. Prevents the image from losing palettes. (only for `image_model: Limited Palette`)`show_palette:` check this box to see the palette each time the image is displayed. (only for `image_model: Limited Palette`)`target_pallete:` path or url of an image which the model will use to make the palette it uses.`lock_pallete:` force the model to use the initial palette (most useful from restore, but will force a grayscale image or a wonky palette otherwise).---`animation_mode:` select animation mode or disable animation.`sampling_mode:` how pixels are sampled during animation. `nearest` will keep the image sharp, but may look bad. `bilinear` will smooth the image out, and `bicubic` is untested :)`infill_mode:` select how new pixels should be filled if they come in from the edge.* mirror: reflect image over boundary* wrap: pull pixels from opposite side* black: fill with black * smear: sample closest pixel in image`pre_animation_steps:` number of steps to run before animation starts, to begin with a stable image. $250$ is a good default.`steps_per_frame:` number of steps between each image move. $50$ is a good default.`frames_per_second:` number of frames to render each second. Controls how $t$ is scaled.`direct_stabilization_weight: ` keeps the current frame as a direct image prompt. For `Video Source` this will use the current frame of the video as a direct image prompt. For `2D` and `3D` this will use the shifted version of the previous frame. Also supports masks: `weight_mask.mp4`.`semantic_stabilization_weight: ` keeps the current frame as a semantic image prompt. For `Video Source` this will use the current frame of the video as a direct image prompt. For `2D` and `3D` this will use the shifted version of the previous frame. Also supports masks: `weight_[mask.mp4]` or `weight_mask phrase`.`depth_stabilization_weight: ` keeps the depth model output somewhat consistent at a *VERY* steep performance cost. For `Video Source` this will use the current frame of the video as a semantic image prompt. For `2D` and `3D` this will use the shifted version of the previous frame. Also supports masks: `weight_mask.mp4`.`edge_stabilization_weight: ` keeps the images contours somewhat consistent at very little performance cost. For `Video Source` this will use the current frame of the video as a direct image prompt with a sobel filter. For `2D` and `3D` this will use the shifted version of the previous frame. Also supports masks: `weight_mask.mp4`.`flow_stabilization_weight: ` used for `animation_mode: 3D` and `Video Source` to prevent flickering. Comes with a slight performance cost for `Video Source`, and a great one for `3D`, due to implementation differences. Also supports masks: `weight_mask.mp4`. For video source, the mask should select the part of the frame you want to move, and the rest will be treated as a still background.---`video_path: ` path to mp4 file for `Video Source``frame_stride` advance this many frames in the video for each output frame. This is surprisingly useful. Set to $1$ to render each frame. Video masks will also step at this rate.`reencode_each_frame: ` check this box to use each video frame as an `init_image` instead of warping each output frame into the init for the next. Cuts will still be detected and trigger a reencode.`flow_long_term_samples: ` Sample multiple frames into the past for consistent interpolation even with disocclusion, as described by [<NAME>, <NAME>, and <NAME> (2016)](https://arxiv.org/abs/1604.08610). Each sample is twice as far back in the past as the last, so the earliest sampled frame is $2^{\text{long_term_flow_samples}}$ frames in the past. Set to $0$ to disable.---`translate_x:` horizontal image motion as a function of time $t$ in seconds.`translate_y:` vertical image motion as a function of time $t$ in seconds.`translate_z_3d:` forward image motion as a function of time $t$ in seconds. (only for `animation_mode:3D`)`rotate_3d:` image rotation as a quaternion $\left[r,x,y,z\right]$ as a function of time $t$ in seconds. (only for `animation_mode:3D`)`rotate_2d:` image rotation in degrees as a function of time $t$ in seconds. (only for `animation_mode:2D`)`zoom_x_2d:` horizontal image zoom as a function of time $t$ in seconds. (only for `animation_mode:2D`)`zoom_y_2d:` vertical image zoom as a function of time $t$ in seconds. (only for `animation_mode:2D`)`lock_camera:` check this box to prevent all scrolling or drifting. Makes for more stable 3D rotations. (only for `animation_mode:3D`)`field_of_view:` vertical field of view in degrees. (only for `animation_mode:3D`)`near_plane:` closest depth distance in pixels. (only for `animation_mode:3D`)`far_plane:` farthest depth distance in pixels. (only for `animation_mode:3D`)---`file_namespace:` output directory name.`allow_overwrite:` check to overwrite existing files in `file_namespace`.`display_every:` how many steps between each time the image is displayed in the notebook.`clear_every:` how many steps between each time notebook console is cleared.`display_scale:` image display scale in notebook. $1$ will show the image at full size. Does not affect saved images.`save_every:` how many steps between each time the image is saved. Set to `steps_per_frame` for consistent animation.`backups:` number of backups to keep (only the oldest backups are deleted). Large images make very large backups, so be warned. Set to `all` to save all backups. These are used for the `flow_long_term_samples` so be sure that this is at least $2^{\text{flow_long_term_samples}}+1$ for `Video Source` mode.`show_graphs:` check this to see graphs of the loss values each time the image is displayed. Disable this for local runtimes.`approximate_vram_usage:` currently broken. Don't believe its lies.---`ViTB32, ViTB16, RN50, RN50x4:` select your CLIP models. These take a lot of VRAM.`learning_rate:` how quickly the image changes.`reset_lr_each_frame:` the optimizer will adaptively change the learning rate, so this will thwart it.`seed:` pseudorandom seed.---`cutouts:` number of cutouts. Reduce this to use less VRAM at the cost of quality and speed.`cut_pow:` should be positive. Large values shrink cutouts, making the image more detailed, small values expand the cutouts, making it more coherent. $1$ is a good default. $3$ or higher can cause crashes.`cutout_border:` should be between $0$ and $1$. Allows cutouts to poke out over the edges of the image by this fraction of the image size, allowing better detail around the edges of the image. Set to $0$ to disable. $0.25$ is a good default.`border_mode:` how to fill cutouts that stick out over the edge of the image. Match with `infill_mode` for consistent infill.* clamp: move cutouts back onto image* mirror: reflect image over boundary* wrap: pull pixels from opposite side* black: fill with black * smear: sample closest pixel in image Step 1: SetupRun the cells in this section once for each runtime, or after a factory reset.<jupyter_code>#@title 1.1 Mount google drive (optional)
#@markdown Mounting your drive is optional but recommended. You can even restore from google randomly
#@markdown kicking you out if you mount your drive.
from google.colab import drive
drive._mount('/content/drive')
!mkdir -p /content/drive/MyDrive/pytti_test
%cd /content/drive/MyDrive/pytti_test
#@title 1.2 NVIDIA-SMI (optional)
#@markdown View information about your runtime GPU.
#@markdown Google will connect you to an industrial strength GPU, which is needed to run
#@markdown this notebook. You can also disable error checking on your GPU to get some
#@markdown more VRAM, at a marginal cost to stability. You will have to restart the runtime after
#@markdown disabling it.
enable_error_checking = False#@param {type:"boolean"}
if enable_error_checking:
  !nvidia-smi
else:
  !nvidia-smi
  !nvidia-smi -i 0 -e 0
#@title 1.3 Install everything else
#@markdown Run this cell on a fresh runtime to install the libraries and modules.
from os.path import exists as path_exists
if path_exists('/content/drive/MyDrive/pytti_test'):
  %cd /content/drive/MyDrive/pytti_test

try:
  from adjustText import adjust_text
  import pytti, torch
  everything_installed = True
except ModuleNotFoundError:
  everything_installed = False
def install_everything():
  !pip install tensorflow==1.15.2
  !pip install transformers                                        &> /dev/null                              
  !pip install PyGLM                                               &> /dev/null
  !pip install ftfy regex tqdm omegaconf pytorch-lightning         &> /dev/null
  !pip install kornia                                              &> /dev/null
  !pip install einops                                              &> /dev/null
  !pip install imageio-ffmpeg                                      &> /dev/null
  !pip install adjustText exrex bunch                              &> /dev/null
  !pip install matplotlib-label-lines                              &> /dev/null
  !git clone https://github.com/openai/CLIP.git                    &> /dev/null
  !git clone https://github.com/CompVis/taming-transformers.git    &> /dev/null
  if not path_exists('./pytti'):
    !git clone --branch p5 https://github.com/sportsracer48/pytti.git        &> /dev/null
  else:
    !rm -r pytti
    !git clone --branch p5 https://github.com/sportsracer48/pytti.git
  !git clone https://github.com/shariqfarooq123/AdaBins.git        &> /dev/null
  !git clone https://github.com/zacjiang/GMA.git                   &> /dev/null
  !mkdir -p AdaBins/pretrained
  if not path_exists('AdaBins/pretrained/AdaBins_nyu.pt'):
   !gdown https://drive.google.com/uc?id=1lvyZZbC9NLcS8a__YPcUP7rDiIpbRpoF
   if not path_exists('AdaBins_nyu.pt'):
     !gdown https://drive.google.com/uc?id=1zgGJrkFkJbRouqMaWArXE4WF_rhj-pxW
   !mv AdaBins_nyu.pt AdaBins/pretrained/AdaBins_nyu.pt
  
  from pytti.Notebook import change_tqdm_color
  change_tqdm_color()
  !mkdir -p images_out
  !mkdir -p videos

force_install = True #@param{type:"boolean"}
if not everything_installed or force_install:
  install_everything()
elif everything_installed:
  from pytti.Notebook import change_tqdm_color
  change_tqdm_color()<jupyter_output><empty_output><jupyter_text>Step 2: Run it!Edit the parameters, or load saved parameters, then run the model.<jupyter_code>#@title #2.1 Parameters:
#@markdown ---
from os.path import exists as path_exists
if path_exists('/content/drive/MyDrive/pytti_test'):
  %cd /content/drive/MyDrive/pytti_test
  drive_mounted = True
else:
  drive_mounted = False
try:
  from pytti.Notebook import change_tqdm_color, get_last_file
except ModuleNotFoundError:
  if drive_mounted:
    #THIS IS NOT AN ERROR. This is the code that would
    #make an error if something were wrong.
    raise RuntimeError('ERROR: please run setup (step 1.3).')
  else:
    #THIS IS NOT AN ERROR. This is the code that would
    #make an error if something were wrong.
    raise RuntimeError('WARNING: drive is not mounted.\nERROR: please run setup (step 1.3).')
change_tqdm_color()

import glob, json, random, re, math
try:
  from bunch import Bunch
except ModuleNotFoundError:
  if drive_mounted:
    #THIS IS NOT AN ERROR. This is the code that would
    #make an error if something were wrong.
    raise RuntimeError('ERROR: please run setup (step 1.3).')
  else:
    #THIS IS NOT AN ERROR. This is the code that would
    #make an error if something were wrong.
    raise RuntimeError('WARNING: drive is not mounted.\nERROR: please run setup (step 1.3).')

#these are used to make the defaults look pretty
model_default = None
random_seed = None
all  = math.inf
derive_from_init_aspect_ratio = -1

def define_parameters():
  locals_before = locals().copy()
  #@markdown ###Prompts:
  
  scenes = "a sunless garden when the flowers are dead"#@param{type:"string"}
  scene_prefix = ""#@param{type:"string"}
  scene_suffix = ""#@param{type:"string"}
  interpolation_steps = 0#@param{type:"number"}
  steps_per_scene =  60100#@param{type:"raw"}
  #@markdown ---
  #@markdown ###Image Prompts:
  direct_image_prompts   = ""#@param{type:"string"}
  #@markdown ---
  #@markdown ###Initial image:
  init_image = ""#@param{type:"string"}
  direct_init_weight =  ""#@param{type:"string"}
  semantic_init_weight = ""#@param{type:"string"}
  #@markdown ---
  #@markdown ###Image:
  #@markdown Use `image_model` to select how the model will encode the image
  image_model = "VQGAN" #@param ["VQGAN", "Limited Palette", "Unlimited Palette"]

  #@markdown image_model | description | strengths | weaknesses
  #@markdown --- | -- | -- | --
  #@markdown  VQGAN | classic VQGAN image | smooth images | limited datasets, slow, VRAM intesnsive 
  #@markdown  Limited Palette | pytti differentiable palette | fast,  VRAM scales with `palettes` | pixel images
  #@markdown  Unlimited Palette | simple RGB optimization | fast, VRAM efficient | pixel images
  
  #@markdown The output image resolution will be `width` $\times$ `pixel_size` by height $\times$ `pixel_size` pixels.
  #@markdown The easiest way to run out of VRAM is to select `image_model` VQGAN without reducing
  #@markdown `pixel_size` to $1$.

  #@markdown For `animation_mode: 3D` the minimum resoultion is about 450 by 400 pixels.
  width =  180#@param {type:"raw"}
  height =  112#@param {type:"raw"}
  pixel_size = 4#@param{type:"number"}
  smoothing_weight =  0.02#@param{type:"number"}
  #@markdown `VQGAN` specific settings:
  vqgan_model = "imagenet" #@param ["imagenet", "coco", "wikiart", "sflckr", "openimages"]
  #@markdown `Limited Palette` specific settings:
  random_initial_palette = False#@param{type:"boolean"}
  palette_size = 6#@param{type:"number"}
  palettes   = 9#@param{type:"number"}
  gamma = 1#@param{type:"number"}
  hdr_weight = 0.01#@param{type:"number"}
  palette_normalization_weight = 0.2#@param{type:"number"}
  show_palette = False #@param{type:"boolean"}
  target_palette = ""#@param{type:"string"}
  lock_palette = False #@param{type:"boolean"}
  #@markdown ---
  #@markdown ###Animation:
  animation_mode = "3D" #@param ["off","2D", "3D", "Video Source"]
  sampling_mode = "bicubic" #@param ["bilinear","nearest","bicubic"]
  infill_mode = "wrap" #@param ["mirror","wrap","black","smear"]
  pre_animation_steps =  100#@param{type:"number"}
  steps_per_frame =  50#@param{type:"number"}
  frames_per_second =  12#@param{type:"number"}
  #@markdown ---
  #@markdown ###Stabilization Weights:
  direct_stabilization_weight = ""#@param{type:"string"}
  semantic_stabilization_weight = ""#@param{type:"string"}
  depth_stabilization_weight = ""#@param{type:"string"}
  edge_stabilization_weight = ""#@param{type:"string"}
  #@markdown `flow_stabilization_weight` is used for `animation_mode: 3D` and `Video Source`
  flow_stabilization_weight = ""#@param{type:"string"}
  #@markdown ---
  #@markdown ###Video Tracking:
  #@markdown Only for `animation_mode: Video Source`.
  video_path = ""#@param{type:"string"}
  frame_stride = 1#@param{type:"number"}
  reencode_each_frame = True #@param{type:"boolean"}
  flow_long_term_samples = 1#@param{type:"number"}
  #@markdown ---
  #@markdown ###Image Motion:
  translate_x    = "-1700*sin(radians(1.5))" #@param{type:"string"}
  translate_y    = "0" #@param{type:"string"}
  #@markdown `..._3d` is only used in 3D mode.
  translate_z_3d = "(50+10*t)*sin(t/10*pi)**2" #@param{type:"string"}
  #@markdown `rotate_3d` *must* be a `[w,x,y,z]` rotation (unit) quaternion. Use `rotate_3d: [1,0,0,0]` for no rotation.
  #@markdown [Learn more about rotation quaternions here](https://eater.net/quaternions).
  rotate_3d      = "[cos(radians(1.5)), 0, -sin(radians(1.5))/sqrt(2), sin(radians(1.5))/sqrt(2)]"#@param{type:"string"}
  #@markdown `..._2d` is only used in 2D mode.
  rotate_2d      = "5" #@param{type:"string"}
  zoom_x_2d      = "0" #@param{type:"string"}
  zoom_y_2d      = "0" #@param{type:"string"}
  #@markdown  3D camera (only used in 3D mode):
  lock_camera   = True#@param{type:"boolean"}
  field_of_view = 60#@param{type:"number"}
  near_plane    = 1#@param{type:"number"}
  far_plane     = 10000#@param{type:"number"}

  #@markdown ---
  #@markdown ###Output:
  file_namespace = "wilde1"#@param{type:"string"}
  if file_namespace == '':
    file_namespace = 'out'
  allow_overwrite = False#@param{type:"boolean"}
  base_name = file_namespace
  if not allow_overwrite and path_exists(f'images_out/{file_namespace}'):
    _, i = get_last_file(f'images_out/{file_namespace}', 
                         f'^(?P<pre>{re.escape(file_namespace)}\\(?)(?P<index>\\d*)(?P<post>\\)?_1\\.png)$')
    if i == 0:
      print(f"WARNING: file_namespace {file_namespace} already has images from run 0")
    elif i is not None:
      print(f"WARNING: file_namespace {file_namespace} already has images from runs 0 through {i}")
  elif glob.glob(f'images_out/{file_namespace}/{base_name}_*.png'):
    print(f"WARNING: file_namespace {file_namespace} has images which will be overwritten")
  try:
    del i
    del _
  except NameError:
    pass
  del base_name
  display_every = steps_per_frame #@param{type:"raw"}
  clear_every = 0 #@param{type:"raw"}
  display_scale = 1#@param{type:"number"}
  save_every = steps_per_frame #@param{type:"raw"}
  backups =  2**(flow_long_term_samples+1)+1#this is used for video transfer, so don't lower it if that's what you're doing#@param {type:"raw"}
  show_graphs = False #@param{type:"boolean"}
  approximate_vram_usage = False#@param{type:"boolean"}

  #@markdown ---
  #@markdown ###Model:
  #@markdown Quality settings from Dribnet's CLIPIT (https://github.com/dribnet/clipit).
  #@markdown Selecting too many will use up all your VRAM and slow down the model.
  #@markdown I usually use ViTB32, ViTB16, and RN50 if I get a A100, otherwise I just use ViT32B.

  #@markdown quality | CLIP models
  #@markdown --- | --
  #@markdown  draft | ViTB32 
  #@markdown  normal | ViTB32, ViTB16 
  #@markdown  high | ViTB32, ViTB16, RN50
  #@markdown  best | ViTB32, ViTB16, RN50x4
  ViTB32 = True #@param{type:"boolean"}
  ViTB16 = True #@param{type:"boolean"}
  RN50 = False #@param{type:"boolean"}
  RN50x4 = False #@param{type:"boolean"}
  #@markdown the default learning rate is `0.1` for all the VQGAN models
  #@markdown except openimages, which is `0.15`. For the palette modes the
  #@markdown default is `0.02`. 
  learning_rate =  model_default#@param{type:"raw"}
  reset_lr_each_frame = True#@param{type:"boolean"}
  seed = random_seed #@param{type:"raw"}
  #@markdown **Cutouts**:

  #@markdown [Cutouts are how CLIP sees the image.](https://twitter.com/remi_durant/status/1460607677801897990)
  cutouts =  40#@param{type:"number"}
  cut_pow =  2#@param {type:"number"}
  cutout_border =  .25#@param {type:"number"}
  #@markdown NOTE: prompt masks (`promt:weight_[mask.png]`) will not work right on '`wrap`' or '`mirror`' mode.
  border_mode = "clamp" #@param ["clamp","mirror","wrap","black","smear"]
  
  if seed is None:
    seed = random.randint(-0x8000_0000_0000_0000, 0xffff_ffff_ffff_ffff)
  locals_after = locals().copy()
  for k in locals_before.keys():
    del locals_after[k]
  del locals_after['locals_before']
  return locals_after

params = Bunch(define_parameters())
print("SETTINGS:")
print(json.dumps(params))
#@title 2.2 Load settings (optional)
#@markdown copy the `SETTINGS:` output from the **Parameters** cell (tripple click to select the whole
#@markdown line from `{'scenes'...` to `}`) and paste them in a note to save them for later.

#@markdown Paste them here in the future to load those settings again. Running this cell with blank settings won't do anything.
from os.path import exists as path_exists
if path_exists('/content/drive/MyDrive/pytti_test'):
  %cd /content/drive/MyDrive/pytti_test
  drive_mounted = True
else:
  drive_mounted = False
try:
  from pytti.Notebook import *
except ModuleNotFoundError:
  if drive_mounted:
    #THIS IS NOT AN ERROR. This is the code that would
    #make an error if something were wrong.
    raise RuntimeError('ERROR: please run setup (step 1.3).')
  else:
    #THIS IS NOT AN ERROR. This is the code that would
    #make an error if something were wrong.
    raise RuntimeError('WARNING: drive is not mounted.\nERROR: please run setup (step 1.3).')
change_tqdm_color()
  
import json, random
try:
  from bunch import Bunch
except ModuleNotFoundError:
  if drive_mounted:
    #THIS IS NOT AN ERROR. This is the code that would
    #make an error if something were wrong.
    raise RuntimeError('ERROR: please run setup (step 1.3).')
  else:
    #THIS IS NOT AN ERROR. This is the code that would
    #make an error if something were wrong.
    raise RuntimeError('WARNING: drive is not mounted.\nERROR: please run setup (step 1.3).')

settings = ""#@param{type:"string"}
#@markdown Check `random_seed` to overwrite the seed from the settings with a random one for some variation.
random_seed = False #@param{type:"boolean"}

if settings != '':
  params = load_settings(settings, random_seed)
#@title 2.3 Run it!
#@markdown pytti is 1000% percent better code than VQLIPSE, so have a look at the code. 
#@markdown You just might understand what's going on.
import torch

from os.path import exists as path_exists
if path_exists('/content/drive/MyDrive/pytti_test'):
  %cd /content/drive/MyDrive/pytti_test
  drive_mounted = True
else:
  drive_mounted = False
try:
  from pytti.Notebook import *
except ModuleNotFoundError:
  if drive_mounted:
    #THIS IS NOT AN ERROR. This is the code that would
    #make an error if something were wrong.
    raise RuntimeError('ERROR: please run setup (step 1.3).')
  else:
    #THIS IS NOT AN ERROR. This is the code that would
    #make an error if something were wrong.
    raise RuntimeError('WARNING: drive is not mounted.\nERROR: please run setup (step 1.3).')
change_tqdm_color()
import sys
sys.path.append('./AdaBins')

try:
  from pytti import Perceptor
except ModuleNotFoundError:
  if drive_mounted:
    #THIS IS NOT AN ERROR. This is the code that would
    #make an error if something were wrong.
    raise RuntimeError('ERROR: please run setup (step 1.3).')
  else:
    #THIS IS NOT AN ERROR. This is the code that would
    #make an error if something were wrong.
    raise RuntimeError('WARNING: drive is not mounted.\nERROR: please run setup (step 1.3).')
print("Loading pytti...")
from pytti.Image import PixelImage, RGBImage, VQGANImage
from pytti.ImageGuide import DirectImageGuide
from pytti.Perceptor.Embedder import HDMultiClipEmbedder
from pytti.Perceptor.Prompt import parse_prompt
from pytti.LossAug import TVLoss, HSVLoss, OpticalFlowLoss, TargetFlowLoss
from pytti.Transforms import zoom_2d, zoom_3d, apply_flow
from pytti import *
from pytti.LossAug.DepthLoss import init_AdaBins
print("pytti loaded.")

import torch, gc, glob, subprocess, warnings, re, math, json
import numpy as np
from IPython import display
from PIL import Image, ImageEnhance

from torchvision.transforms import functional as TF

#display settings, because usability counts
#warnings.filterwarnings("error", category=UserWarning)
%matplotlib inline 
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
import pandas as pd
plt.style.use('bmh')
pd.options.display.max_columns = None
pd.options.display.width = 175

latest = -1
#@markdown check `batch_mode` to run batch settings
batch_mode = False #@param{type:"boolean"}
if batch_mode:
  try:
    batch_list
  except NameError:
    raise RuntimeError("ERROR: no batch settings. Please run 'batch settings' cell at the bottom of the page to use batch mode.")
else:
  try:
    params
  except NameError:
    raise RuntimeError("ERROR: no parameters. Please run parameters (step 2.1).")
#@markdown check `restore` to restore from a previous run
restore = False#@param{type:"boolean"}
#@markdown check `reencode` if you are restoring with a modified image or modified image settings
reencode = False#@param{type:"boolean"}
#@markdown which run to restore
restore_run = latest #@param{type:"raw"}
if restore and restore_run == latest:
  _, restore_run = get_last_file(f'backup/{params.file_namespace}', 
                           f'^(?P<pre>{re.escape(params.file_namespace)}\\(?)(?P<index>\\d*)(?P<post>\\)?_\\d+\\.bak)$')

def do_run():
  clear_rotoscopers()#what a silly name
  vram_profiling(params.approximate_vram_usage)
  reset_vram_usage()
  global CLIP_MODEL_NAMES
  #@markdown which frame to restore from
  restore_frame =  latest#@param{type:"raw"}

  #set up seed for deterministic RNG
  if params.seed is not None:
    torch.manual_seed(params.seed)

  #load CLIP
  load_clip(params)
  embedder = HDMultiClipEmbedder(cutn=params.cutouts, 
                                 cut_pow = params.cut_pow, 
                                 padding = params.cutout_border,
                                 border_mode = params.border_mode)
  
  #load scenes
  with vram_usage_mode('Text Prompts'):
    print('Loading prompts...')
    prompts = [[parse_prompt(embedder, p.strip()) 
              for p in (params.scene_prefix + stage + params.scene_suffix).strip().split('|') if p.strip()]
              for stage in params.scenes.split('||') if stage]
    print('Prompts loaded.')

  #load init image
  if params.init_image != '':
    init_image_pil = Image.open(fetch(params.init_image)).convert('RGB')
    init_size = init_image_pil.size
    #automatic aspect ratio matching
    if params.width == -1:
      params.width = int(params.height*init_size[0]/init_size[1])
    if params.height == -1:
      params.height = int(params.width*init_size[1]/init_size[0])
  else:
    init_image_pil = None

  #video source
  if params.animation_mode == "Video Source":
    print(f'loading {params.video_path}...')
    video_frames = get_frames(params.video_path)
    params.pre_animation_steps = max(params.steps_per_frame, params.pre_animation_steps)
    if init_image_pil is None:
      init_image_pil = Image.fromarray(video_frames.get_data(0)).convert('RGB')
      #enhancer = ImageEnhance.Contrast(init_image_pil)
      #init_image_pil = enhancer.enhance(2)
      init_size = init_image_pil.size
      if params.width == -1:
        params.width = int(params.height*init_size[0]/init_size[1])
      if params.height == -1:
        params.height = int(params.width*init_size[1]/init_size[0])

  #set up image
  if params.image_model == "Limited Palette":
    img = PixelImage(*format_params(params,
                     'width', 'height', 'pixel_size', 
                     'palette_size', 'palettes', 'gamma', 
                     'hdr_weight', 'palette_normalization_weight'))
    img.encode_random(random_pallet = params.random_initial_palette)
    if params.target_palette.strip() != '':
      img.set_pallet_target(Image.open(fetch(params.target_palette)).convert('RGB'))
    else:
      img.lock_pallet(params.lock_palette)
  elif params.image_model == "Unlimited Palette":
    img = RGBImage(params.width, params.height, params.pixel_size)
    img.encode_random()
  elif params.image_model == "VQGAN":
    VQGANImage.init_vqgan(params.vqgan_model)
    img = VQGANImage(params.width, params.height, params.pixel_size)
    img.encode_random()

  loss_augs = []

  if init_image_pil is not None:
    if not restore:
      print("Encoding image...")
      img.encode_image(init_image_pil)
      print("Encoded Image:")
      display.display(img.decode_image())
    #set up init image prompt
    init_augs = ['direct_init_weight']
    init_augs = [build_loss(x,params[x],f'init image ({params.init_image})', img, init_image_pil) 
                  for x in init_augs if params[x] not in ['','0']]
    loss_augs.extend(init_augs)
    if params.semantic_init_weight not in ['','0']:
      semantic_init_prompt = parse_prompt(embedder, 
                                    f"init image [{params.init_image}]:{params.semantic_init_weight}", 
                                    init_image_pil)
      prompts[0].append(semantic_init_prompt)
    else:
      semantic_init_prompt = None
  else:
    init_augs, semantic_init_prompt = [], None

  #other image prompts

  loss_augs.extend(type(img).get_preferred_loss().TargetImage(p.strip(), img.image_shape, is_path = True) 
                   for p in params.direct_image_prompts.split('|') if p.strip())

  #stabilization

  stabilization_augs = ['direct_stabilization_weight',
                        'depth_stabilization_weight',
                        'edge_stabilization_weight']
  stabilization_augs = [build_loss(x,params[x],'stabilization',
                                   img, init_image_pil) 
                        for x in stabilization_augs if params[x] not in ['','0']]
  loss_augs.extend(stabilization_augs)
  
  if params.semantic_stabilization_weight not in ['0','']:
    last_frame_semantic = parse_prompt(embedder, 
                                       f"stabilization:{params.semantic_stabilization_weight}", 
                                       init_image_pil if init_image_pil else img.decode_image())
    last_frame_semantic.set_enabled(init_image_pil is not None)
    for scene in prompts:
      scene.append(last_frame_semantic)
  else:
    last_frame_semantic = None

  #optical flow
  if params.animation_mode == 'Video Source':
    if params.flow_stabilization_weight == '':
      params.flow_stabilization_weight = '0'
    optical_flows = [OpticalFlowLoss.TargetImage(f"optical flow stabilization (frame {-2**i}):{params.flow_stabilization_weight}", 
                                                 img.image_shape) 
                     for i in range(params.flow_long_term_samples + 1)]
    for optical_flow in optical_flows:
      optical_flow.set_enabled(False)
    loss_augs.extend(optical_flows)
  elif params.animation_mode == '3D' and params.flow_stabilization_weight not in ['0','']:
    optical_flows = [TargetFlowLoss.TargetImage(f"optical flow stabilization:{params.flow_stabilization_weight}", 
                                                img.image_shape)]
    for optical_flow in optical_flows:
      optical_flow.set_enabled(False)
    loss_augs.extend(optical_flows)
  else:
    optical_flows = []
  #other loss augs
  if params.smoothing_weight != 0:
    loss_augs.append(TVLoss(weight = params.smoothing_weight))
  
  #set up filespace
  subprocess.run(['mkdir','-p',f'images_out/{params.file_namespace}'])
  subprocess.run(['mkdir','-p',f'backup/{params.file_namespace}'])
  if restore:
    base_name = params.file_namespace if restore_run == 0 else f'{params.file_namespace}({restore_run})'
  elif not params.allow_overwrite:
    #finds the next available base_name to save files with. Why did I do this with regex? 
    _, i = get_next_file(f'images_out/{params.file_namespace}', 
                         f'^(?P<pre>{re.escape(params.file_namespace)}\\(?)(?P<index>\\d*)(?P<post>\\)?_1\\.png)$',
                         [f"{params.file_namespace}_1.png",f"{params.file_namespace}(1)_1.png"])
    base_name = params.file_namespace if i == 0 else f'{params.file_namespace}({i})'
  else:
    base_name = params.file_namespace

  #restore
  if restore:
    if not reencode:
      if restore_frame == latest:
        filename, restore_frame = get_last_file(f'backup/{params.file_namespace}', 
                                                f'^(?P<pre>{re.escape(base_name)}_)(?P<index>\\d*)(?P<post>\\.bak)$')
      else: 
        filename = f'{base_name}_{restore_frame}.bak'
      print("restoring from", filename)
      img.load_state_dict(torch.load(f'backup/{params.file_namespace}/{filename}'))
    else:#reencode
      if restore_frame == latest:
        filename, restore_frame = get_last_file(f'images_out/{params.file_namespace}', 
                                                f'^(?P<pre>{re.escape(base_name)}_)(?P<index>\\d*)(?P<post>\\.png)$')
      else: 
        filename = f'{base_name}_{restore_frame}.png'
      print("restoring from", filename)
      img.encode_image(Image.open(f'images_out/{params.file_namespace}/{filename}').convert('RGB'))
    i = restore_frame*params.save_every
  else:
    i = 0

  #graphs
  if params.show_graphs:
    fig, axs = plt.subplots(4, 1, figsize=(21,13))
    axs  = np.asarray(axs).flatten()
    #fig.facecolor = (0,0,0)
  else:
    fig, axs = None, None

  #make the main model object
  model = DirectImageGuide(img, embedder, lr = params.learning_rate)

  #Update is called each step.
  def update(i, stage_i):
    #display
    if params.clear_every > 0 and i > 0 and i % params.clear_every == 0:
      display.clear_output()
    if params.display_every > 0 and i % params.display_every == 0:
      print(f"Step {i} losses:")
      if model.dataframe:
        print(model.dataframe[0].iloc[-1])
      if params.approximate_vram_usage:
        print("VRAM Usage:")
        print_vram_usage()
      display_width = int(img.image_shape[0]*params.display_scale)
      display_height = int(img.image_shape[1]*params.display_scale)
      if stage_i > 0 and params.show_graphs:
        model.plot_losses(axs)
        im = img.decode_image()
        sidebyside = make_hbox(im.resize((display_width, display_height), Image.LANCZOS), fig)
        display.display(sidebyside)
      else:
        im = img.decode_image()
        display.display(im.resize((display_width, display_height), Image.LANCZOS))
      if params.show_palette and isinstance(img, PixelImage):
        print('Palette:')
        display.display(img.render_pallet())
    #save
    if i > 0 and params.save_every > 0 and i % params.save_every == 0:
      try:
        im
      except NameError:
        im = img.decode_image()
      n = i//params.save_every
      filename = f"images_out/{params.file_namespace}/{base_name}_{n}.png"
      im.save(filename)
      if params.backups > 0:
        filename = f"backup/{params.file_namespace}/{base_name}_{n}.bak"
        torch.save(img.state_dict(), filename)
        if n > params.backups:
          subprocess.run(['rm', f"backup/{params.file_namespace}/{base_name}_{n-params.backups}.bak"])
    #animate
    t = (i - params.pre_animation_steps)/(params.steps_per_frame*params.frames_per_second)
    set_t(t)
    if i >= params.pre_animation_steps:
      if (i - params.pre_animation_steps) % params.steps_per_frame == 0:
        print(f"Time: {t:.4f} seconds")
        update_rotoscopers(((i - params.pre_animation_steps)//params.steps_per_frame+1)*params.frame_stride)
        if params.reset_lr_each_frame:
          model.set_optim(None)
        if params.animation_mode == "2D":
          tx, ty = parametric_eval(params.translate_x), parametric_eval(params.translate_y)
          theta = parametric_eval(params.rotate_2d)
          zx, zy = parametric_eval(params.zoom_x_2d), parametric_eval(params.zoom_y_2d)
          next_step_pil = zoom_2d(img, 
                                  (tx,ty), (zx,zy), theta, 
                                  border_mode = params.infill_mode, sampling_mode = params.sampling_mode)
        elif params.animation_mode == "3D":
          try:
            im
          except NameError:
            im = img.decode_image()
          with vram_usage_mode('Optical Flow Loss'):
            flow, next_step_pil = zoom_3d(img, 
                                        (params.translate_x,params.translate_y,params.translate_z_3d), params.rotate_3d, 
                                        params.field_of_view, params.near_plane, params.far_plane,
                                        border_mode = params.infill_mode, sampling_mode = params.sampling_mode,
                                        stabilize = params.lock_camera)
            freeze_vram_usage()
            
          for optical_flow in optical_flows:
            optical_flow.set_last_step(im)
            optical_flow.set_target_flow(flow)
            optical_flow.set_enabled(True)
        elif params.animation_mode == "Video Source":
          frame_n = min((i - params.pre_animation_steps)*params.frame_stride//params.steps_per_frame, len(video_frames) - 1)
          next_frame_n = min(frame_n + params.frame_stride, len(video_frames) - 1)
          next_step_pil = Image.fromarray(video_frames.get_data(next_frame_n)).convert('RGB').resize(img.image_shape, Image.LANCZOS)
          for j, optical_flow in enumerate(optical_flows):
            old_frame_n = frame_n - (2**j - 1)*params.frame_stride
            save_n = i//params.save_every - (2**j - 1)
            if old_frame_n < 0 or save_n < 1:
              break
            current_step_pil = Image.fromarray(video_frames.get_data(old_frame_n)).convert('RGB').resize(img.image_shape, Image.LANCZOS)
            filename = f"backup/{params.file_namespace}/{base_name}_{save_n}.bak"
            filename = None if j == 0 else filename
            flow_im, mask_tensor = optical_flow.set_flow(current_step_pil, next_step_pil, 
                                                        img, filename, 
                                                        params.infill_mode, params.sampling_mode)
            optical_flow.set_enabled(True)
            #first flow is previous frame
            if j == 0:
              mask_accum = mask_tensor.detach()
              valid = mask_tensor.mean()
              print("valid pixels:", valid)
              if params.reencode_each_frame or valid < .03:
                if isinstance(img, PixelImage) and valid >= .03:
                  img.lock_pallet()
                  img.encode_image(next_step_pil, smart_encode = False)
                  img.lock_pallet(params.lock_palette)
                else:
                  img.encode_image(next_step_pil)
                reencoded = True
              else:
                reencoded = False
            else:
              with torch.no_grad():
                optical_flow.set_mask((mask_tensor - mask_accum).clamp(0,1))
                mask_accum.add_(mask_tensor)
        if params.animation_mode != 'off':
          for aug in stabilization_augs:
            aug.set_comp(next_step_pil)
            aug.set_enabled(True)
          if last_frame_semantic is not None:
            last_frame_semantic.set_image(embedder, next_step_pil)
            last_frame_semantic.set_enabled(True)
          for aug in init_augs:
            aug.set_enabled(False)
          if semantic_init_prompt is not None:
            semantic_init_prompt.set_enabled(False)
            
      
  model.update = update
  
  print(f"Settings saved to images_out/{params.file_namespace}/{base_name}_settings.txt")
  save_settings(params, f"images_out/{params.file_namespace}/{base_name}_settings.txt")

  skip_prompts = i // params.steps_per_scene
  skip_steps   = i %  params.steps_per_scene
  last_scene = prompts[0] if skip_prompts == 0 else prompts[skip_prompts - 1]
  for scene in prompts[skip_prompts:]:
    print("Running prompt:", ' | '.join(map(str,scene)))
    i += model.run_steps(params.steps_per_scene-skip_steps, 
                         scene, last_scene, loss_augs, 
                         interp_steps = params.interpolation_steps,
                         i_offset = i, skipped_steps = skip_steps)
    skip_steps = 0
    model.clear_dataframe()
    last_scene = scene
  if fig:
    del fig, axs

#if __name__ == '__main__':
try:
  gc.collect()
  torch.cuda.empty_cache()
  if batch_mode:
    if restore:
      settings_list = batch_list[restore_run:]
    else:
      settings_list = batch_list
      namespace = batch_list[0]['file_namespace']
      subprocess.run(['mkdir','-p',f'images_out/{namespace}'])
      save_batch(batch_list, f'images_out/{namespace}/{namespace}_batch settings.txt')
      print(f"Batch settings saved to images_out/{namespace}/{namespace}_batch settings.txt")
    for settings in settings_list:
      setting_string = json.dumps(settings)
      print("SETTINGS:")
      print(setting_string)
      params = load_settings(setting_string)
      if params.animation_mode == '3D':
        init_AdaBins()
      params.allow_overwrite = False
      do_run()
      restore = False
      reencode = False
      gc.collect()
      torch.cuda.empty_cache()
  else:
    if params.animation_mode == '3D':
      pass
      #init_AdaBins()
    do_run()
    print("Complete.")
    gc.collect()
    torch.cuda.empty_cache()
except KeyboardInterrupt:
  pass
except RuntimeError:
  print_vram_usage()
  raise
      
#print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))<jupyter_output><empty_output><jupyter_text>Step 3: Render videoYou can dowload from the notebook, but it's faster to download from your drive.<jupyter_code>#@title 3.1 Render video
from os.path import exists as path_exists
if path_exists('/content/drive/MyDrive/pytti_test'):
  %cd /content/drive/MyDrive/pytti_test
  drive_mounted = True
else:
  drive_mounted = False
try:
  from pytti.Notebook import change_tqdm_color
except ModuleNotFoundError:
  if drive_mounted:
    #THIS IS NOT AN ERROR. This is the code that would
    #make an error if something were wrong.
    raise RuntimeError('ERROR: please run setup (step 1.3).')
  else:
    #THIS IS NOT AN ERROR. This is the code that would
    #make an error if something were wrong.
    raise RuntimeError('WARNING: drive is not mounted.\nERROR: please run setup (step 1.3).')
change_tqdm_color()
  
from tqdm.notebook import tqdm
import numpy as np
from os.path import exists as path_exists
from subprocess import Popen, PIPE
from PIL import Image, ImageFile
from os.path import splitext as split_file
import glob
from pytti.Notebook import get_last_file

ImageFile.LOAD_TRUNCATED_IMAGES = True

try:
  params
except NameError:
  raise RuntimeError("ERROR: no parameters. Please run parameters (step 2.1).")

if not path_exists(f"images_out/{params.file_namespace}"):
  if path_exists(f"/content/drive/MyDrive"):
    #THIS IS NOT AN ERROR. This is the code that would
    #make an error if something were wrong.
    raise RuntimeError(f"ERROR: file_namespace: {params.file_namespace} does not exist.")
  else:
    #THIS IS NOT AN ERROR. This is the code that would
    #make an error if something were wrong.
    raise RuntimeError(f"WARNING: Drive is not mounted.\nERROR: file_namespace: {params.file_namespace} does not exist.")

#@markdown The first run executed in `file_namespace` is number $0$, the second is number $1$, etc.

latest = -1
run_number = latest#@param{type:"raw"}
if run_number == -1:
  _, i = get_last_file(f'images_out/{params.file_namespace}', 
                       f'^(?P<pre>{re.escape(params.file_namespace)}\\(?)(?P<index>\\d*)(?P<post>\\)?_1\\.png)$')
  run_number = i
base_name = params.file_namespace if run_number == 0 else (params.file_namespace+f"({run_number})")
tqdm.write(f'Generating video from {params.file_namespace}/{base_name}_*.png')

all_frames = glob.glob(f'images_out/{params.file_namespace}/{base_name}_*.png')
all_frames.sort(key = lambda s: int(split_file(s)[0].split('_')[-1]))
print(f'found {len(all_frames)} frames matching images_out/{params.file_namespace}/{base_name}_*.png')

start_frame = 0#@param{type:"number"}
all_frames = all_frames[start_frame:]

fps =  params.frames_per_second#@param{type:"raw"}

total_frames = len(all_frames)

if total_frames == 0:
  #THIS IS NOT AN ERROR. This is the code that would
  #make an error if something were wrong.
  raise RuntimeError(f"ERROR: no frames to render in images_out/{params.file_namespace}")

frames = []

for filename in tqdm(all_frames):
  frames.append(Image.open(filename))

p = Popen(['ffmpeg', '-y', '-f', 'image2pipe', '-vcodec', 'png', '-r', str(fps), '-i', '-', '-vcodec', 'libx264', '-r', str(fps), '-pix_fmt', 'yuv420p', '-crf', '1', '-preset', 'veryslow', f"videos/{base_name}.mp4"], stdin=PIPE)
for im in tqdm(frames):
  im.save(p.stdin, 'PNG')
p.stdin.close()

print("Encoding video...")
p.wait()
print("Video complete.")
#@title 3.1 Render video (concatenate all runs)
from os.path import exists as path_exists
if path_exists('/content/drive/MyDrive/pytti_test'):
  %cd /content/drive/MyDrive/pytti_test
  drive_mounted = True
else:
  drive_mounted = False
try:
  from pytti.Notebook import change_tqdm_color
except ModuleNotFoundError:
  if drive_mounted:
    #THIS IS NOT AN ERROR. This is the code that would
    #make an error if something were wrong.
    raise RuntimeError('ERROR: please run setup (step 1.3).')
  else:
    #THIS IS NOT AN ERROR. This is the code that would
    #make an error if something were wrong.
    raise RuntimeError('WARNING: drive is not mounted.\nERROR: please run setup (step 1.3).')
change_tqdm_color()
  
from tqdm.notebook import tqdm
import numpy as np
from os.path import exists as path_exists
from subprocess import Popen, PIPE
from PIL import Image, ImageFile
from os.path import splitext as split_file
import glob
from pytti.Notebook import get_last_file

ImageFile.LOAD_TRUNCATED_IMAGES = True

try:
  params
except NameError:
  raise RuntimeError("ERROR: no parameters. Please run parameters (step 2.1).")

if not path_exists(f"images_out/{params.file_namespace}"):
  if path_exists(f"/content/drive/MyDrive"):
    raise RuntimeError(f"ERROR: file_namespace: {params.file_namespace} does not exist.")
  else:
    raise RuntimeError(f"WARNING: Drive is not mounted.\nERROR: file_namespace: {params.file_namespace} does not exist.")

#@markdown The first run executed in `file_namespace` is number $0$, the second is number $1$, etc.

latest = -1
run_number = latest
if run_number == -1:
  _, i = get_last_file(f'images_out/{params.file_namespace}', 
                       f'^(?P<pre>{re.escape(params.file_namespace)}\\(?)(?P<index>\\d*)(?P<post>\\)?_1\\.png)$')
  run_number = i

all_frames = []
for i in range(run_number+1):
  base_name = params.file_namespace if i == 0 else (params.file_namespace+f"({i})")
  frames = glob.glob(f'images_out/{params.file_namespace}/{base_name}_*.png')
  frames.sort(key = lambda s: int(split_file(s)[0].split('_')[-1]))
  all_frames.extend(frames)

start_frame = 0#@param{type:"number"}
all_frames = all_frames[start_frame:]

fps =  params.frames_per_second#@param{type:"raw"}

total_frames = len(all_frames)

if total_frames == 0:
  #THIS IS NOT AN ERROR. This is the code that would
  #make an error if something were wrong.
  raise RuntimeError(f"ERROR: no frames to render in images_out/{params.file_namespace}")

frames = []

for filename in tqdm(all_frames):
  frames.append(Image.open(filename))

p = Popen(['ffmpeg', '-y', '-f', 'image2pipe', '-vcodec', 'png', '-r', str(fps), '-i', '-', '-vcodec', 'libx264', '-r', str(fps), '-pix_fmt', 'yuv420p', '-crf', '1', '-preset', 'veryslow', f"videos/{base_name}.mp4"], stdin=PIPE)
for im in tqdm(frames):
  im.save(p.stdin, 'PNG')
p.stdin.close()

print("Encoding video...")
p.wait()
print("Video complete.")
#@title 3.2 Download the last exported video
from os.path import exists as path_exists
if path_exists('/content/drive/MyDrive/pytti_test'):
  %cd /content/drive/MyDrive/pytti_test

try:
  from pytti.Notebook import get_last_file
except ModuleNotFoundError:
  if drive_mounted:
    #THIS IS NOT AN ERROR. This is the code that would
    #make an error if something were wrong.
    raise RuntimeError('ERROR: please run setup (step 1.3).')
  else:
    #THIS IS NOT AN ERROR. This is the code that would
    #make an error if something were wrong.
    raise RuntimeError('WARNING: drive is not mounted.\nERROR: please run setup (step 1.3).')

try:
  params
except NameError:
  #THIS IS NOT AN ERROR. This is the code that would
  #make an error if something were wrong.
  raise RuntimeError("ERROR: please run parameters (step 2.1).")

from google.colab import files
try:
  base_name = params.file_namespace if run_number == 0 else (params.file_namespace+f"({run_number})")
  filename = f'{base_name}.mp4'
except NameError:
  filename, i = get_last_file(f'videos', 
                       f'^(?P<pre>{re.escape(params.file_namespace)}\\(?)(?P<index>\\d*)(?P<post>\\)?\\.mp4)$')

if path_exists(f'videos/{filename}'):
  files.download(f"videos/{filename}")
else:
  if path_exists(f"/content/drive/MyDrive"):
    #THIS IS NOT AN ERROR. This is the code that would
    #make an error if something were wrong.
    raise RuntimeError(f"ERROR: video videos/{filename} does not exist.")
  else:
    #THIS IS NOT AN ERROR. This is the code that would
    #make an error if something were wrong.
    raise RuntimeError(f"WARNING: Drive is not mounted.\nERROR: video videos/{filename} does not exist.")<jupyter_output><empty_output><jupyter_text>Batch SetingsWARNING: If you use google colab (even with pro and pro+) GPUs for long enought google will throttle your account. Be careful with batch runs if you don't want to get kicked.<jupyter_code>#@title batch settings
from os.path import exists as path_exists
if path_exists('/content/drive/MyDrive/pytti_test'):
  %cd /content/drive/MyDrive/pytti_test
  drive_mounted = True
else:
  drive_mounted = False
try:
  from pytti.Notebook import change_tqdm_color, save_batch
except ModuleNotFoundError:
  if drive_mounted:
    raise RuntimeError('ERROR: please run setup (step 1).')
  else:
    raise RuntimeError('WARNING: drive is not mounted.\nERROR: please run setup (step 1).')
change_tqdm_color()

try:
  import exrex, random, glob
except ModuleNotFoundError:
  if drive_mounted:
    raise RuntimeError('ERROR: please run setup (step 1).')
  else:
    raise RuntimeError('WARNING: drive is not mounted.\nERROR: please run setup (step 1).')
from numpy import arange
import itertools

def all_matches(s):
  return list(exrex.generate(s))

def dict_product(dictionary):
  return [dict(zip(dictionary, x)) for x in itertools.product(*dictionary.values())]

#these are used to make the defaults look pretty
model_default = None
random_seed = None

def define_parameters():
  locals_before = locals().copy()
  scenes = ["list","your","runs"] #@param{type:"raw"}
  scene_prefix = ["all "," permutations "," are run "] #@param{type:"raw"}
  scene_suffix = [" that", " makes", " 27" ] #@param{type:"raw"}
  interpolation_steps = [0] #@param{type:"raw"}
  steps_per_scene = [300] #@param{type:"raw"}
  direct_image_prompts = [""] #@param{type:"raw"}
  init_image = [""] #@param{type:"raw"}
  direct_init_weight = [""] #@param{type:"raw"}
  semantic_init_weight = [""] #@param{type:"raw"}
  image_model = ["Limited Palette"] #@param{type:"raw"}
  width = [180] #@param{type:"raw"}
  height = [112] #@param{type:"raw"}
  pixel_size = [4] #@param{type:"raw"}
  smoothing_weight = [0.05] #@param{type:"raw"}
  vqgan_model = ["sflckr"] #@param{type:"raw"}
  random_initial_palette = [False] #@param{type:"raw"}
  palette_size = [9] #@param{type:"raw"}
  palettes = [8] #@param{type:"raw"}
  gamma = [1] #@param{type:"raw"}
  hdr_weight = [1.0] #@param{type:"raw"}
  palette_normalization_weight = [1.0] #@param{type:"raw"}
  show_palette = [False] #@param{type:"raw"}
  target_palette = [""] #@param{type:"raw"}
  lock_palette = [False] #@param{type:"raw"}
  animation_mode = ["off"] #@param{type:"raw"}
  sampling_mode = ["bicubic"] #@param{type:"raw"}
  infill_mode = ["wrap"] #@param{type:"raw"}
  pre_animation_steps = [100] #@param{type:"raw"}
  steps_per_frame = [50] #@param{type:"raw"}
  frames_per_second = [12] #@param{type:"raw"}
  direct_stabilization_weight = [""] #@param{type:"raw"}
  semantic_stabilization_weight = [""] #@param{type:"raw"}
  depth_stabilization_weight = [""] #@param{type:"raw"}
  edge_stabilization_weight = [""] #@param{type:"raw"}
  flow_stabilization_weight = [""] #@param{type:"raw"}
  video_path = [""] #@param{type:"raw"}
  frame_stride = [1] #@param{type:"raw"}
  reencode_each_frame = [True] #@param{type:"raw"}
  flow_long_term_samples = [0] #@param{type:"raw"}
  translate_x = ["0"] #@param{type:"raw"}
  translate_y = ["0"] #@param{type:"raw"}
  translate_z_3d = ["0"] #@param{type:"raw"}
  rotate_3d = ["[1,0,0,0]"] #@param{type:"raw"}
  rotate_2d = ["0"] #@param{type:"raw"}
  zoom_x_2d = ["0"] #@param{type:"raw"}
  zoom_y_2d = ["0"] #@param{type:"raw"}
  lock_camera = [True] #@param{type:"raw"}
  field_of_view = [60] #@param{type:"raw"}
  near_plane = [1] #@param{type:"raw"}
  far_plane = [10000] #@param{type:"raw"}
  file_namespace = ["Basic Batch"] #@param{type:"raw"}
  allow_overwrite = [False]
  display_every = [50] #@param{type:"raw"}
  clear_every = [0] #@param{type:"raw"}
  display_scale = [1] #@param{type:"raw"}
  save_every = [50] #@param{type:"raw"}
  backups = [2] #@param{type:"raw"}
  show_graphs = [False] #@param{type:"raw"}
  approximate_vram_usage = [False] #@param{type:"raw"}
  ViTB32 = [True] #@param{type:"raw"}
  ViTB16 = [False] #@param{type:"raw"}
  RN50 = [False] #@param{type:"raw"}
  RN50x4 = [False] #@param{type:"raw"}
  learning_rate = [None] #@param{type:"raw"}
  reset_lr_each_frame = [True] #@param{type:"raw"}
  seed = [None] #@param{type:"raw"}
  cutouts = [40] #@param{type:"raw"}
  cut_pow = [2] #@param{type:"raw"}
  cutout_border = [0.25] #@param{type:"raw"}
  border_mode = ["clamp"] #@param{type:"raw"}
  locals_after = locals().copy()
  for k in locals_before.keys():
    del locals_after[k]
  del locals_after['locals_before']
  return locals_after

param_dict = define_parameters()
batch_list = dict_product(param_dict)
namespace = batch_list[0]['file_namespace']
if glob.glob(f'images_out/{namespace}/*.png'):
  print(f"WARNING: images_out/{namespace} contains images. Batch indicies may not match filenames unless restoring.")<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup