# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3 (ipykernel)
#     language: python
#     name: python3
# ---

# + [markdown] slideshow={"slide_type": "slide"}
# # Mathematical Preliminaries

# + [markdown] slideshow={"slide_type": "-"} tags=["remove-cell"]
# **DIVE into Deep Learning**
# ___

# + init_cell=true slideshow={"slide_type": "notes"}
from IPython import display
from manim import *

# + [markdown] slideshow={"slide_type": "-"}
# The following is a lecture series that introduces the basic theory of deep learning.

# + hideCode=true hidePrompt=true init_cell=true slideshow={"slide_type": "-"} tags=["hide-input"]
display.IFrame(src="https://www.youtube.com/embed/videoseries?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi", width="100%", height=450)

# + [markdown] slideshow={"slide_type": "notes"}
# Understanding the lectures requires some background on vector calculus and probability theory. $\def\abs#1{\left\lvert #1 \right\rvert}
# \def\Set#1{\left\{ #1 \right\}}
# \def\mc#1{\mathcal{#1}}
# \def\M#1{\boldsymbol{#1}}
# \def\R#1{\mathsf{#1}}
# \def\RM#1{\boldsymbol{\mathsf{#1}}}
# \def\op#1{\operatorname{#1}}
# \def\E{\op{E}}
# \def\d{\mathrm{\mathstrut d}}
# $

# + [markdown] slideshow={"slide_type": "subslide"}
# **What to know about vector calculus?**

# + code_folding=[] hideCode=true hidePrompt=true init_cell=true slideshow={"slide_type": "-"} tags=["hide-input"] language="html"
# <b>Notation:</b><br>
# <video width="100%" height="450" controls>
#       <source src="https://www.cs.cityu.edu.hk/~ccha23/dl/Notation.mp4" type="video/mp4">
# </video>

# + [markdown] cell_style="split" slideshow={"slide_type": "subslide"}
# - [Vectors](https://en.wikipedia.org/wiki/Vector_(mathematics_and_physics)) in lowercase boldface font: 
#
# $$\M{x}=\begin{bmatrix}x_1 \\ x_2 \\ \vdots \end{bmatrix}$$

# + [markdown] cell_style="split" slideshow={"slide_type": "-"}
# - [Matrices](https://en.wikipedia.org/wiki/Matrix_(mathematics)) in uppercase boldface font:
#
# $$\M{W}=\begin{bmatrix}w_{11} & w_{12} & \cdots \\
# w_{21} & \ddots &  \\
# \vdots &  &  \end{bmatrix}$$

# + [markdown] slideshow={"slide_type": "-"}
# - [Matrix multiplication](https://en.wikipedia.org/wiki/Matrix_multiplication):
#
# $$\M{W}\M{x} = \begin{bmatrix}w_{11} & w_{12} & \cdots \\
# w_{21} & \ddots &  \\
# \vdots &  &  \end{bmatrix}
# \begin{bmatrix}x_1 \\ x_2 \\ \vdots \end{bmatrix}
# =
# \begin{bmatrix}w_{11}x_1 + w_{12}x_2 + \cdots \\ w_{21}x_1+\cdots \\ \vdots \end{bmatrix}
# $$

# + [markdown] slideshow={"slide_type": "subslide"}
# **What to know about Probability Theory?**

# + code_folding=[] hideCode=true hidePrompt=true init_cell=true slideshow={"slide_type": "-"} tags=["hide-input"] language="html"
# <b>Distribution:</b><br>
# <video width="100%" height="450" controls>
#       <source src="https://www.cs.cityu.edu.hk/~ccha23/dl/Distribution.mp4" type="video/mp4">
# </video>

# + [markdown] slideshow={"slide_type": "subslide"}
# [Joint distribution](https://en.wikipedia.org/wiki/Joint_probability_distribution#Mixed_case):  
#
# $$p_{\RM{x}\R{y}}(\M{x},y)= \underbrace{p_{\R{y}|\RM{x}}(y|\M{x})}_{\underbrace{\Pr}_{\text{probability measure}\kern-3em}\Set{\R{y}=y|\RM{x}=\M{x}}} \cdot \underbrace{p_{\RM{x}}(\M{x})}_{(\underbrace{\partial_{x_1}}_{\text{partial derivative w.r.t. $x_1$}\kern-5em} \partial_{x_2}\cdots)\Pr\Set{\RM{x} \leq \M{x}}\kern-4em}\kern1em \text{where}$$    
# - $p_{\R{y}|\RM{x}}(y|\M{x})$ is the *probability mass function [(pmf)](https://en.wikipedia.org/wiki/Probability_mass_function)* of $\R{y}=y\in \mc{Y}$ [conditioned](https://en.wikipedia.org/wiki/Conditional_probability_distribution) on $\RM{x}=\M{x}\in \mc{X}$, and
# - $p_{\RM{x}}(\M{x})$ is the *(multivariate) probability density function [(pdf)](https://en.wikipedia.org/wiki/Probability_density_function#Densities_associated_with_multiple_variables)* of $\RM{x}=\M{x}\in \mc{X}$.

# + code_folding=[] hideCode=true hidePrompt=true init_cell=true slideshow={"slide_type": "subslide"} tags=["hide-input"] language="html"
# <b>Expectation:<br>
# <video width="100%" height="450" controls>
#       <source src="https://www.cs.cityu.edu.hk/~ccha23/dl/Expectation.mp4" type="video/mp4">
# </video>

# + [markdown] slideshow={"slide_type": "subslide"}
# For any function $g$ of $(\RM{x},y)$, the expectations are:  
#   
# $$
# \begin{align}
# \E[g(\RM{x},\R{y})|\RM{x}]&=\sum_{y\in \mc{Y}} g(\RM{x},y)\cdot p_{\R{y}|\RM{x}}(y|\RM{x})\tag{conditional exp.}
# \\
# \E[g(\RM{x},\R{y})] &=\int_{\mc{X}} \underbrace{\sum_{y\in \mc{Y}} g(\RM{x},y)\cdot \underbrace{p_{\RM{x},\R{y}}(\M{x},y)}_{p_{\R{y}|\RM{x}}(y|\M{x}) p_{\R{x}}(\M{x})}\kern-1.7em}_{\E[g(\RM{x},\R{y})|\RM{x}]}\kern1.4em\,\d \M{x} \tag{exp.}\\
# &= \E[\E[g(\RM{x},\R{y})|\RM{x}]] \tag{iterated exp.}
# \end{align}
# $$

# + [markdown] slideshow={"slide_type": "subslide"}
# All the videos were created using the python library [mathematical animation engine (`manim`)](https://github.com/3b1b/manim).

# + slideshow={"slide_type": "-"}
from manim import *


# + [markdown] slideshow={"slide_type": "subslide"}
# **Exercise** Run the following cell and see the effect when changing
#
# - Mobjects `Text('Hello, World!')` to 
#   - `MathTex(r'E=mc^2')` or
#   - `Circle()` or
#   - `Square()`...
# - Animation objects `Write` to 
#   - `FadeIn` or 
#   - `GrowFromCenter`...

# + [markdown] slideshow={"slide_type": "notes"}
# You may take a look at the documentation [here](https://docs.manim.community/en/v0.2.0/index.html) and a more detailed [tutorial](https://talkingphysics.wordpress.com/2019/01/08/getting-started-animating-with-manim-and-python-3-7/) here to create more complicated animations. Since rending videos take time, create a separate notebook for more intensive work without affecting the autograding of this notebook.

# + hideCode=false slideshow={"slide_type": "-"}
# %%manim -ql --progress_bar=none --disable_caching --flush_cache -v ERROR HelloWorld
class HelloWorld(Scene):
    def construct(self):
        self.play(Write(Text('Hello, World!')))

# + [markdown] slideshow={"slide_type": "fragment"} tags=[]
# To show all the options available to `manim`:

# + tags=["hide-output"]
# %manim render --help

StarcoderdataJupyterScriptsDedupFiltered