<jupyter_start><jupyter_text>Elegant Exception Handling Eyal Trabelsi - thanks for coming out, I hope you enjoyed all the amazing talks by now.- today I am going to talk to you about Elegent Exception Handling.  About Me 🙈 - Software Engineer at Salesforce 👷 - Big passion for python, data and performance optimisations 🐍🤖 - Online at [medium.com/@Eyaltra](https://medium.com/@Eyaltra) | [@eyaltra](https://twitter.com/eyaltra) 🌐  Restaurant Recommendation 🍔 - Get user info 👨👩 - Retrieve relevant restaurants 🍕🍗🍩🍔 - Pick the best ones 🍗🍔<jupyter_code>! pip install typeguard rollbar returns tenacity > /dev/null 2>&1
import contextlib
import json
import logging
import pathlib
import os
from typing import Union

import requests
from typeguard import typechecked
# Naive code snippets

def get_relevant_restaurants(user):
    base_url = "https://en.wikipedia.org/wiki"
    return requests.get(f"{base_url}/{user}").content

def get_user(path):
    with open(path, 'r') as json_file:
        return json.load(json_file)["user"]

def pick_best_restaurants(restaurants):
    pass
def get_restaurant_recommendation(path):
    user = get_user(path)
    candidates = get_relevant_restaurants(user)
    return pick_best_restaurants(candidates)
    
get_restaurant_recommendation("MY_AMAZING_FILE.json")<jupyter_output><empty_output><jupyter_text>We Can Proud of Ourselves 💃 - Implemented restaurant recommendation 💪 - Clean code 💄   Why Exception Handling? 🤨 - Hardware can fail 🌲 - Software often fail 🚪 - [How complex systems fail](https://www.researchgate.net/publication/228797158_How_complex_systems_fail) 🧩   Unexceptable 😡  Lesson 1: We want to build a fault tolerant system.  Types of errors- error that can be detected at compile time- errors that can be deteled at run time- errors that can be infered- reproducieable erros- non reproduceable errors  We want our code to be safe 👷  Exception Handling to the Rescue 👨‍🚒 - Detect errors 🕵 - Do something about them 🔒  Naive Approach 👶 - Log all exceptions 📝 - Ignore all exceptions 🙈<jupyter_code>def get_restaurant_recommendation(path):
    try:
        user = get_user(path)
        candidates = get_relevant_restaurants(user)
        pick_best_restaurants(candidates)
    except BaseException:
        logging.error("VERY UNINFORMATIVE INFORMATION")
        raise BaseException("VERY UNINFORMATIVE INFORMATION")<jupyter_output><empty_output><jupyter_text>Are we done ?! 🍰  - Code is very clean 🧹 - Code seems safe 🔒  The lurking problems  🐲 - Unintentional exceptions being caught 😧 - KeyboardInterrupt as we want the user to be able to kill the program.- SynatxError as we want our code to be valid. - Exceptions are not distinguishable 😵 - the invoker of this function can't really destinguise between the diffrent types of errors and allow to recover from certain expected issues.- For example, if we have flaky internet i would like to retry, but if a file is actually missing I dont. - Not safe 🔓 - generaly it’s better for a program to fail fast and crash than to silence the error and continue running the program. - The bugs that inevitably happen later on will be harder to debug since they are far removed from the original cause. - Just because programmers often ignore error messages doesn’t mean the program should stop emitting them.  - Unfortunately very common 😱  Types of exception handling- EAFP (it’s easier to ask for forgiveness than permission) - LBYL (Look before you leap)- Each has its own pros and cons (whether the thread-safty or readability)- but both are legitimate in python as oppose to other languages.  Take 2: Exception Handling 🎬 - Catch relevant exceptions only ☝ - Recover when possible 🔧<jupyter_code>def get_restaurant_recommendation(path):
    try:
        user = get_config(path)["user"]
    except FileNotFoundException:
        logging.error("VERY UNINFORMATIVE INFORMATION")
        raise
    except JSONDecodeError:
        logging.error("VERY UNINFORMATIVE INFORMATION")
        raise
    candidates = get_relevant_restaurants(user)
    pick_best_restaurants(candidates)<jupyter_output><empty_output><jupyter_text>- When a file does not exists or when the file is not a valid json we  raise FileNotFoundException and JSONDecodeError and log it away- We will reraise the same exact exception that occured instead raising a generic exception and allow the invoker to handle them diffrently.- Altough this code is far from pretty is much safer, we added deafault patiserie and  the invoker of this function can destinguise between the diffrent types of errors and handle them in a diffrent manner if needed.  Lesson 2: Catch relevant exceptions only.   Our code is bad 😭 - Dominated by exception handling - Business logic is not clear  Lesson 3:  Error handling should not obscures business logic - Error handling is important, but we should strives to make our job easier.- as the zen of python state "If the implementation is hard to explain, it's a bad idea."  Take 3: Exception Handling 🎬  A Bit of Mackup 💄 - Sharing exception blocks. - Use else clause. - Use dictionary built-in method  This<jupyter_code>def get_restaurant_recommendation(path):
    try:
        user = get_user(path)
    except (FileNotFoundException, JSONDecodeError):
        logging.error("VERY UNINFORMATIVE INFORMATION")
        raise
    else:
        candidates = get_relevant_restaurants(user)
        pick_best_restaurants(candidates)<jupyter_output><empty_output><jupyter_text>Becomes<jupyter_code>def get_restaurant_recommendation(path):
    try:
        user = get_user(path)
    except (FileNotFoundException, JSONDecodeError):
        logging.error("VERY UNINFORMATIVE INFORMATION")
        raise
    else:
        candidates = get_relevant_restaurants(user)
        pick_best_restaurants(candidates)<jupyter_output><empty_output><jupyter_text>- First since we handle both FileNotFoundException and JSONDecodeError in the same manner they can "share the except block"  as except clause may name multiple exceptions as a parenthesized tuple.- Secondly we can use else  clause which occur when the try block executed and did not raise an exception.- Thirdly, we use dictionary builtin function get which allow us to define default values.              Lesson 4: Frequent flows probably have clean existing solution.  Suppressing Exceptions 🤫 - There is another common flow for exception handling- i want to cover which is suppressing exceptions using suppress- supported from python>=3.5<jupyter_code>def run_unstopable_animation():
    pass<jupyter_output><empty_output><jupyter_text>**This**<jupyter_code>try:
    os.remove('somefile.pyc')
except FileNotFoundError:
    pass
try:
    run_unstopable_animation()
except KeyboardInterrupt:
    pass<jupyter_output><empty_output><jupyter_text>**Becomes**<jupyter_code>from contextlib import suppress

with suppress(FileNotFoundError):
    os.remove('somefile.pyc')
from contextlib import suppress

with suppress(KeyboardInterrupt):
    run_unstopable_animation()<jupyter_output><empty_output><jupyter_text>Reasons for errors- The obious one is that something exceptional happened.- As a control flow mechanism.- Can be triggered due to a bug in our code.  Still dominated by exception handling code😟  Take 3: Exception Handling 🎬 - Separate business logic from exception handling code ✂ - Handled exceptions in other layer 📚 ![](https://vignette.wikia.nocookie.net/memepediadankmemes/images/8/80/Acb.jpg/revision/latest/scale-to-width-down/340?cb=20180822064733)  This<jupyter_code>def get_user(path):
    with open(path, 'r') as json_file:
        return json.load(json_file)\
                   .get("user", "default_user")
    
def get_restaurant_recommendation(path):
    try:
        user = get_user(path)
    except (FileNotFoundException, JSONDecodeError):
        logging.error("VERY UNINFORMATIVE INFORMATION")
        raise
    else:
        candidates = get_relevant_restaurants(user)
        pick_best_restaurants(candidates)<jupyter_output><empty_output><jupyter_text>Becomes<jupyter_code>def get_user(path):
    with open(path, 'r') as json_file:
        try:
            user = json.load(json_file)\
                       .get("user", "default_user")
        except (FileNotFoundException, JSONDecodeError):
            logging.error("VERY UNINFORMATIVE INFORMATION")
            raise
        else:
            return user

def get_restaurant_recommendation(path):
    user = get_user(path)                
    candidates = get_relevant_restaurants(user)
    pick_best_restaurants(candidates)<jupyter_output><empty_output><jupyter_text>Lesson 5: Push the exception handling down/up in the abstraction level  Are we completly safe now? 👷  Silent Errors 🔇 - Does not crash code 🤯 - Delivers incorrect results 😠 - Makes matter worse 🤬  Contract 📜 - Output/Input types - Output/Input values - Postconditions/Preconditions - Side-effects/Invariants  Vanilla Exceptions 🍧 - All the validations are supported ✅  - Happens in runtime ✅ but not in compilation time ❌ - Not clean❌  Why not assertions ? ❌ - Raises the wrong exception type 😮  - Can be compiled away 😥  Type Hints 🔍 - Support validating input/output types ✅ - Doesn't support other validation ❌ - Support both Runtime ✅ / Compile time ✅ - Clean and elegant ✅  mypy for compile time<jupyter_code>def get_user(path: Union[str, pathlib.PurePath]) -> str:
    with open(path, 'r') as json_file:
        try:
            data = json.load(json_file)
        except (FileNotFoundException, JSONDecodeError):
            logging.error("VERY INFORMATIVE INFORMATION")
            raise
        else:
            return data.get("user","default_user")<jupyter_output><empty_output><jupyter_text>typeguard for runtime<jupyter_code>@typechecked
def get_user(path: Union[str, pathlib.PurePath]) -> str:
    with open(path, 'r') as json_file:
        try:
            data = json.load(json_file)
        except (FileNotFoundException, JSONDecodeError):
            logging.error("VERY INFORMATIVE INFORMATION")
            raise
        else:
            return data.get("user","default_user")<jupyter_output><empty_output><jupyter_text>Contract Testing Libraries 📜 - All the validations are supported ✅  - Happens in runtime ✅ but not in compilation time ❌ - Clean and elegant ✅ - No mature/maintained option ❌ - [icontract](https://github.com/Parquery/icontract)- not matured 🍼 - [contracts](https://github.com/deadpixi/contracts)- not maintained 🤕  Lesson 6: Protect your code from silent errors.  There are still problems lurking 🐉<jupyter_code>def get_relevant_restaurants(user):
    base_url = "cool_restaurants.com"
    resp = requests.get(f"{base_url}/{user}")
    resp.raise_for_status()
    return resp.content<jupyter_output><empty_output><jupyter_text>Unstable environment 🤪 - Your network might be down 😑 - The server might be down 😣 - The server might be too busy and you will face a timeout 😭<jupyter_code>def get_relevant_restaurants(user):
    base_url = "cool_restaurants.com"
    
    allowed_retries = 5
    for i in range(allowed_retries):
        try:
            resp = requests.get(f"{base_url}/{user}")
            resp.raise_for_status()
        except (requests.ConnectionError):
            if i == allowed_retries:
                raise
        else:
            return resp.content<jupyter_output><empty_output><jupyter_text>There must be better way 😇 - Decorators extend our function capabilities beyond its core intent🎊 - Context Managers wrap around enter and exit logic over a given resource 🌉 - Common usecases already implemented 💪<jupyter_code>from functools import wraps
def retry(exceptions, allowed_retries=5):
    def callable(func):
        @wraps(func)
        def wrapped(*args, **kwargs):
            for i in range(allowed_retries):
                try:
                    res = func()
                except exceptions:
                    continue
                else:
                    return res       
        return wrapped
    return callable
@retry(exceptions=requests.ConnectionError)
def get_relevant_restaurants(country):
    base_url = "cool_restaurants.com"
    resp = requests.get(f"{base_url}/{user}")
    resp.raise_for_status()
    return resp.content
import tenacity

@tenacity.retry(retry=tenacity.retry_if_exception_type(ConnectionError))
def get_relevant_restaurants(user):
    base_url = "cool_restaurants.com"
    resp = requests.get(f"{base_url}/{user}")
    resp.raise_for_status()
    return resp.content<jupyter_output><empty_output><jupyter_text>Useful usecases 🧠- **Decorator**: [ratelimit](https://github.com/tomasbasham/ratelimit), [Retry](https://github.com/jd/tenacity), [logger.catch](https://github.com/Delgan/loguru)- **Context manager**: [Database Connections](), [Transactions](), [Temporary Files]() and [Output Redirections]() - important note retry can be handled in the request itself by [writing an adapter](https://findwork.dev/blog/advanced-usage-python-requests-timeouts-retries-hooks/), but for the example sake i wont use it.   Lesson 7: Use decorators and context managers  Python hooks 🎣 - Python has builtin hooks for various events  - [sys.excepthook](https://dev.to/joshuaschlichting/catching-every-single-exception-with-python-40o3) for uncaught exceptions - Doesn't require modifying existing code.  sys's excepthook example! 🎣 - Uncaught exception print traceback to STDERR before closing - Unacceptable in production environment - Graceful exit by notify incident system<jupyter_code>import sys
import rollbar

rollbar.init("Super Secret Token")

def rollbar_except_hook(exc_type, exc_value, traceback):
    rollbar.report_exc_info((exc_type, exc_value, traceback))
    sys.__excepthook__(exc_type, exc_value, traceback)
    
sys.excepthook = rollbar_except_hook<jupyter_output>WARNING:rollbar:Rollbar already initialized. Ignoring re-init.<jupyter_text>Useful usecases 🧠 - [Format Diffrently](https://dev.to/joshuaschlichting/catching-every-single-exception-with-python-40o3)- [Redirect To Incident System](https://sweetcode.io/using-rollbar-capturing-logging-python-exceptions/)- [Multi Threading Behaviour](https://www.scrygroup.com/tutorial/2018-02-06/python-excepthook-logging/)- [Search Stackoverflow](https://gist.github.com/WoLpH/88e3222ac57d9c3bff113ff83afddda4/) 😛😛😛 - We can format the exceptions diffrently, to provide more/less information.- We can redirect Exceptions to an incident system like rollbar or pager-duty.- Since threading/multiprocessing have their own unhandled exception machinery.  that is a bit customized so no unhandled exception exists at the top level.  we might want to overide it to support KeyboardInterupt for example.- Search Stackoverflow for the exception that was being raise  Lesson 8: Python has some useful builtin hooks  Exceptions Components 📑 - Exception message 💬 - Exception type 🍅🍇🍆 - Exception cause 🤯  Exception Types 🍅🍇🍆 - Helps distinguish between different exceptions - Helps emphasis our intent - Builtin and Custom exceptions  Builtin Exception Types 🍅🍇🍆 - Dozens of built-in exceptions - Well documented, we can use some stackoverflow magic  - Should use builtin exceptions   Custom Exception Types 🍅🍇🍆 - Emphasis our intent  - Distinguish between different exceptions. Lets say we have ValueError and we want to recover in diffrent way between TooBig/TooSmall. - Group different exceptions. - Wrapping third party apis. - when we wrap third party api we minimize our dependecy on it. for example uppon recovery shouldn't have to import exceptions from your dependecies for example requests.exceptions- Also the users that use your library does not need/want to know about the implementation details.          Wrapping third party example 👀 - get_restaurant_recommendation can raise requests.ConnectionError - Recovering in get_restaurant_recommendation  - Require import requests for exceptions  Lesson 9: Pick the right exception types and messages.  Exception __ cause __ 🤯 - __ cause __ indicates the reason of the error  - We can overide the __ cause __ to replace exception  raised **Python default behavior** - When a modern Python developer catches an exception and raises a new one to replace it, they can enjoy seeing the complete information for both tracebacks. - This is very helpful for debugging, and is a win for everybody.<jupyter_code>try:
    1/0
except ZeroDivisionError:
    raise<jupyter_output><empty_output><jupyter_text>**Replace exception type with both traces**<jupyter_code>try:
    1/0
except ZeroDivisionError as e:
    raise Exception from e<jupyter_output><empty_output><jupyter_text>**Replace exception type with only one trace**<jupyter_code>try:
    1/0
except ZeroDivisionError as e:
    raise Exception from None<jupyter_output><empty_output><jupyter_text>Lesson 10: Replace exceptions using __cause__.  Still not perfect 💯 - Hard to tell what exceptions can be thrown  - Hard to tell where exceptions will be handled - No static analysis  Functional Exception Handling for the rescue 🚔 - Use success/failure container values - functions are typed, and safe - Railway oriented programming - [returns library](https://github.com/dry-python/returnsresult-container) ![](https://image.slidesharecdn.com/railway-oriented-programming-slideshare-140312155941-phpapp01/95/railway-oriented-programming-75-638.jpg?cb=1427456657)<jupyter_code>def pick_best_restaurants(user: str, candidates: List[str]) -> List[str]:    
    validate_user(user)
    best_candicates = heapq.nlargest(5, valid_candidates)
    update_df(user, best_candicates)
    send_email()
def get_user(path):
    with open(path, 'r') as json_file:
        try:
            user = json.load(json_file)\
                       .get("user", "default_user")
        except (FileNotFoundException, JSONDecodeError):
            logging.error("VERY UNINFORMATIVE INFORMATION")
            raise
        else:            
            try:
                send_email(user)
            else :
                
            return user<jupyter_output><empty_output><jupyter_text>Lesson 11: For complicated usecases consider the functional way.  Lesson 12: Split nested try catch blocks to separated try blocks - Avoid nested exception handling - Split to multiple exception blocks - zen of python state "Flat is better than nested".  Not all programs made equal  👯 - Extremely reliable ✈ ✨ - Highly reliable 🚘 -  Reliable 💳 -  Dodgy 📱 - Crap 💩  Lesson 13: We want to build a fault tolerant to a certain degree  Sensative information 🕵 - Messages will be spread far and wide 🇫🇷🇺🇸🇫🇷  through logging, reporting, and monitoring software. - Personal data privacy 🕵 In a world where regulation around personal data is constantly getting stricter,  - Never reveal your weaknesses, bad actors are everywhere 👺 - You can never be too careful 🤓<jupyter_code>def login(user):
    raise CommonPasswordException(f"password: {password} is too common")<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup