# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # List of concepts in AI, ML, Probablity, Statistics
# <EMAIL> / teleported.in

# ### Specificity and Sensitivity
#
# Ref:
# * https://www.youtube.com/watch?v=21Igj5Pr6u4
# * https://www.youtube.com/watch?v=vtYDyGGeQyo

# ### Cross Validation

# ### Data Leakage

# ### Bias Variance Tradeoff
#     * An algorithm which is highly Biased does not learn from training data
#     * An algorithm which is high on variance is very sensitive to the training data
#     * One way or other, they do not have the capacity top adapt to new data
#     * This is Bias Variance tradeoff
#     * Tuning an ML Algorith means reaching a sweet spot

# ### Confusion Matrix

# ### Naive Bayes explanation
#
# Ref:
# * http://stackoverflow.com/questions/10059594/a-simple-explanation-of-naive-bayes-classification

# ### Regression vs Classification
#
# In regression, we try to predict a real number (continuous)  
# In classification, we try to predict a category (from a set of possible finite categories. Discrete)

# ### Clustering
# * Purpose is to discover sub population of the population
# * Useful to identify outliers (Classification can not do that)
#

# ### Decision Boundary
# For Classification, the bounary within the problem space (the space of all possible feature values) which our classifier learns which separates one class from another

# ### Attribute Value Pair
# * The way we represent data in machine learning
# * We need to represent observations numerically
# * You can be very creative in designing how to represent your data in Machine Learning
# * Highly unlikely one will come up with a new ML algorithm. But one can show creativity in how to represent data.
# * One can achieve gain in performance by engineering the right representation of data.
# * Each individual obsevation represented with a set of attributes, and each attribute has a number assigned to it:
# ```python
# person = {height_inch: 70, weight_kg: 85, age_years: 45}
# ```
#
# * Unordered _bag-of-features_
# * Any data needs to be converted to this form before learning takes place
# * Attributes can be categorised into:
#     * Categorical: Red, Blue, Yellow 
#         * Discrete list of possible values - no ordering
#         * Can only hold one value at a time (out of the many categories)
#         * Mutually exclusive
#         * Categories turn to numbers when you code it up
#         * But not really numbers, can only do == and !=
#         * Tagging process is important, needs to be controlled
#     * Ordinal: high, medium, low 
#         * Like categorical, but has natural ordering to it
#         * Encoded as numbers, can test for ordering in addition to equality check
#         * ==, !=, >, <, >=, <= etc.
#     * Numerical: 1, 2, 3 
#         * Meaningful to add, multiply, computer mean/variance
#         * Usually you want to normalize them
#         * Because raw incoming data can take any range, need to control it
#         ```python
#         x` = (x - mean)/st.dev # Will make the values center around 0, with variance of 1
#         x` = (x-min)/(max-min) # Makes sense if there are definite max and min values
#         ```
#         * You bring the data to the same scale, so that all units are roughly comparable to each other
#         * They are sensitive to outliers - unusually large or small values. Must be handled before normalization. Normalization will get confused by outliers.
#         * BUT - Some outliers might actually be valid data due to skewed distribution
#             * Systematic extreme values
#             * Affects regression, kNN, NB. Not DT's
#             * Fix: log(x) for positive numbers or atan(x) for pos and neg numbers, then normalise
#             * Can also use Cumulative Distribution Function (Use rank instead of values)
#         * Monotonic vs Non-monotonic: 
#             * Monotonic e.g.: higher net worth -> lower lending risk
#             * Non-monotonic: Age -> winning marathon
#             * Fix: Quantization (Turn numeric to Categorical or Ordinal)
#                 * Can be unsupervised, overlapping
#     * Time series: series of tweets
# * Picking attribute (Detecting preditor in image) - one idea that works is similarity
#     * If you have several instances/observations and they have the same class
# * Blurring helps Machine Learning (recognising numbers, letters)
#
# Ref:
# * https://www.youtube.com/playlist?list=PLBv09BD7ez_4Z5ap8fJOzn-Ezz7WpWI5-

# ### Case study: Identifying a Zebra
# * In this domain, using pixel won't work - a single pixel no longer carries any meaning
#     * A zebra can be put upside down and still be zebra
# * **Segment** the image into regions - goes into image and identifies homogenous regions - regions which has similar local properties
#     * Algorithms: BlobWorld, Normalized Cuts
#     * Caveat: Segmentation algorithms not so much accurate on still pics, good on videos
#     * Hope that the anomalies would be systematic, so that it happens in every photo
# * Compute features describing the region
#     * position
#     * relative area
#     * circumference
#     * convexity
#     * orientation
#     * color frequency
#     * texture filters etc.
#    
# ### Case study: text classification - spam no spam
# * Input: string of characters
# * Idea: Words carry meaning
# * Naive way: Words as value (as many values as words in email, categorical)
# * For robustness, instead of tying a word to a position in the email, tie it to a word in the vocabulary
# * May use frequency or tf-idf weights
# * Use binary values for attributes
#
# ### Case study: Music Classification
# * Naive representation
#     * sample at regular intervals
#     * X(t) = amplitude at t
# * Else (better)
#     * Music is a time series
#     * Decompose it into constituent base frequencies f (Fourier transform)
#     * Find weight of each f
#     * The weights of the base frequency can be used to classify music - it is insensitive to shifts and volume increase

# ### Word frequency, TF-IDF weights
# * Word Frequency
#     * How many times the word occured in the document
# * TF-IDF
#     * TF: Term Frequency, the number of times the word occured in the document
#     * IDF: Inverse Document Frequency, words unique to this document, but rarely found in corpus of other documents are important for this document
#     * Score = #times the word appeares in this document * log(# total documents / # other documents the term appears in)
#     * Example: If the word 'automobile' appeared 10 times in this document, if there are 100 documents in the corpus, and if 20 other documents have the word 'automobile' in them, then:
#         * `Score = 10 * log(100/20) = 6.98`

# ### Robustness of an algorithm
# A small change to input should result in small change in output (and not huge)

# ### Learning Algorithms and Predictors
# * Learning Algorithms create Predictors (functions) that can be then deployed in production
# * Learning Algorithms use training data (sample inputs and expected outputs) to create the Predictor
# * For Classification: Predictor takes the form of a decision boundary. The function creating the decision boundary _is_ the Predictor. And the learning algorithm creates it (the function).

# ### Supervised vs Unsupervised
#
# * Supervised
#     * Trying to predict a specific quantity
#     * Have training examples with lables
#     * Can measure accuracy directly
#     
# * Unsupervised
#     * Trying to understand data
#     * Looking for structure or unusual patterns
#     * Not looking for something specific
#     * Does not require labled data
#     * Evaluation usually indirect or qualitative
# * Semi supervised
#     * Use Unsupervised methods to improve supervised algos
#     * Usually few labled examples + lots of unlabled

# ### Types of learning
# 1. Classification (Supervised)
#     * Multiclass Classification
#         * Classes mutually exclusive
#         * NB, kNN, DT, Logistics
#         * Assumes that there are not other classes than specified (i.e. classes are predefined)
#     * Binary Classification
#         * SVM is fundamentally a binary classifier
#         * One vs Rest can be done
#             * {a} vs {not b}, {b} vs {not b}
#         * Classes may overlap
#             * Instances can be both a and b
#             * Can be in none of the classes
#         * E.g. SVM, Logistic (softmax), Perceptron
# 2. Regression (Supervised)
# 3. Clustering (Unsupervised)
# 4. Dimentionality Reduction (Unsupervised)
#
#
#
# * Linear Classifier: Draws straignt decision boundaries

# ### Classification accuracy and imbalanced classes
# * Example: Look at a paper and predict if it will lead to a Nobel Prize
#     * Always say 'no', will lead to 99.9% accuracy classifier
#     * This is an example of unbalanced classes, where one class dominates
#     * Question to ask: What is the right error metric?
#     * Give relative weights to false positives, false negatives
# * Accuracy/Error rate poor metric (above)
# * Want:  cost(Misses) > cost (FA)
#     * What is FA?

# ### Generative vs Discriminative learning
# * Generative approach
#     * Probabilistic "model" of each class
#     * Decision boundary
#         * Where one model becomes more likely
#     * Natural use of unlabeled data, can also use labeled data
#     * Supervised or Unsupervised
#     * All Generative are Probablistic, but all Probablistic are not Generative, i.e. for some, you can calculate the probablity without first creating a model
# * Discriminative
#     * Focus on decision boundary
#     * More powerful when we have lots of examples
#     * Not designed to use unlabeled data
#     * Only Supervised tasks

# ### Outliers
# * Isolated instances in your data that don't look right
# * E.g. extreme values for one or more attribute value
# * Can be detected using confidence intervals
# * Remove or threshold

# ### Hyper Plane

# In geometry a hyperplane is a subspace of one dimension less than its ambient space. If a space is 3-dimensional then its hyperplanes are the 2-dimensional planes, while if the space is 2-dimensional, its hyperplanes are the 1-dimensional lines. This notion can be used in any general space in which the concept of the dimension of a subspace is defined.

StarcoderdataJupyterScriptsDedupFiltered