<jupyter_start><jupyter_text>Speed Up Your Data Science Project Using Persistent Caching Tools from Pythagoras Package Introductory TutorialWork with large dataframes could be slow. This notebook demonstrates how Pythagoras can help you win extra seconds and minutes (sometimes - hours) every time you need to load a large csv file or to execute a complex data-processing function.<jupyter_code>import numpy as np
import pandas as pd
import sys
import time
import logging

np.random.seed(42)<jupyter_output><empty_output><jupyter_text>How To Cache Function Outputs  A slow function Let's assume we have an important function that takes some unpleasantly long time to run:<jupyter_code># Let's create a sample DataFrame to experiment with

a_dataframe = pd.DataFrame(
    data = {
        'COL_1': [1.1, 2.2, 3.3]
        ,'COL_2': [4.4, 5.5, None]
        ,'COL_3': [7.7, 8.8, 9.9]
        ,'COL_4': [None, 11.11, 12.12]
    })   
a_dataframe
# Now, let's create a slow function.
# In real life, such slow function could be a part of 
# feature-engineering pipeline

def slowly_process_dataframe(df:pd.core.frame.DataFrame, a:float):
    result = df + a
    time.sleep(3)
    return result
%%time
demo_result = slowly_process_dataframe(a_dataframe,3.14)
demo_result<jupyter_output><empty_output><jupyter_text>A slow function + Pythagoras It took 3+ seconds to execute **slowly_process_dataframe()** in the cell above.Let's see how Pythagoras helps speed this up:<jupyter_code>import pythagoras # this is the library which will provide us with the
                  # advanced caching tools

demo_cache_obj = pythagoras.PickleCache(
    cache_dir = "./cache_files"   # Here Pythagoras will store cashed data, 
                                  # if/when it needs to.
    ,input_dir = "."  # From here Pythagoras will read .csv files, 
                      # if/when asked.
    )
print(demo_cache_obj) # print the status of the cache 
# This time we add a decorator while creating our function 
# everything else is exactly the same as above
@demo_cache_obj
def slowly_process_dataframe(df:pd.core.frame.DataFrame, a:float):
    result = df + a
    time.sleep(3)
    return result
%%time
demo_result = slowly_process_dataframe(a_dataframe,3.14)
demo_result<jupyter_output><empty_output><jupyter_text>The first call above took 3+ seconds to execute. Let's call the fucntion egain with exactly the same parameters:<jupyter_code>%%time
demo_result = slowly_process_dataframe(a_dataframe,3.14)
demo_result<jupyter_output><empty_output><jupyter_text>The second and all subsequent calls to **slowly_process_dataframe()** are now much faster: we went down from 3 seconds to just 11 milliseconds.  Calling the same function with different parameters<jupyter_code>a_dataframe
a_dataframe.iat[1,1] = -100
a_dataframe
%%time
demo_result = slowly_process_dataframe(a_dataframe,3.14)
demo_result
%%time
demo_result = slowly_process_dataframe(a_dataframe,3.14)
demo_result<jupyter_output><empty_output><jupyter_text>The first time we called the function with sligtly different parameters (one value in the input dataframe was changed), it again took 3 seconds to execute. However, the second attempt to call the function with the same new parameters was way faster.<jupyter_code>print(demo_cache_obj)  # print the status of the cache<jupyter_output>PickleCache in directory <./cache_files> contains 2 files, with total size 2 Kb. There are 275 Gb of free space available in the directory. Cache files are expected to have <.pkl> extension. Input files should be located in <.> folder, which contains 220 files, with total size 942 Kb. Cache READER is ACTIVE: cached versions of objects are loaded from disk if they are available there. Cache WRITER is ACTIVE: new objects get saved to disk as they are created. Names of cache files can not be longer than 250 characters. The following KEY-VALUE pairs will determine caching behaviour if they are present among function parameters or attributes of the parameters on any nested level: {'random_state': None}; a presence of an attribute or a parameter with the KEY name will disable caching, unless the parameter/attribute is set to VALUE. Logger pythagoras.PickleCache.140652536762672.demo_cache_obj (WARNING)<jupyter_text>How does it work?The first time we ran **slowly_process_dataframe()**, Pythagoras cached the output of the function.The next time the function was called, Pythagoras re-used that output, without actually executing the original fucntion. The output is stored as a pickle file, so it will save you time even when you run your notebook again next month.If we pass some other arguments to the functions, the same process will repeat.For each combination of different values, passed as function arguments, Pythagoras will create a new cache file.If you have enough disk space, it will save you a lot of time.> Pythagoras works not only with simple types of function arguments > (such as int or str),  but also with many others,> including DataFrames, dicts, lists, sets, > and all their possible combinations.> **Most of other persistent caching libraries can not do it today, >  they only work with limited group of basic argument types.**> If some existing or new (created by you) datatype> is not supported by Pythagoras out-of-the-box,> it provides a simple extensibility mechanism that allows you to > add support for any new type with just a few lines of code.> **Other persistent caching libraries can not easily recognize new types today.**> How can Pythagoras offer such a powerful flexibility > while almost all the other caching libraries only support > a limited number of datatypes for function arguments?> Most of the existing caching libraries in Python use> easy-to-implement algorithms that only allow to work > with immutable values. Pythagoras uses different approach > that can work with both mutable and immutable values.> This allows Pythagoras to cache functions whose parameters > can be of virtually any existing type/class.So, the next time you need to do complex feature engineeringby transforming a large dataframe into another, even larger dataframe,put your feature engineering code into a function and decorate it with PickleCache object.  Important noteThis approach only works with functions that create their output using exclusively the input argumets, without accessing any outside data.If a function reads from global variables, or files, or Internet, or uses current time, etc.,such function is not compatible with PickleCache  What is under the hood? For every function, which we modified with PickleCach decorator, Pythagoras creates a sub-folder within the main cache_dir folder.The name of sub-folder consists of a keyword "Func" plus the function's name plus its digital fingerprint. For example, for our slow function above, the subfolder was named **Func\_\_slowly\_process\_dataframe\_\_bdca2**<jupyter_code>!ls cache_files/<jupyter_output>[34mFunc__slowly_process_dataframe__59c21[m[m<jupyter_text>Inside each subfolder, Pythagoras puts .pkl files with different versions of function output.The name of each file consists of function's digital signature, plus slim human-readable summary representation of function arguments,plus digital fingerprint of function arguments.For example, for our slow function the subfolder now includes 2 files:<jupyter_code>!ls cache_files/Func__slowly_process_dataframe__bdca2<jupyter_output>ls: cache_files/Func__slowly_process_dataframe__bdca2: No such file or directory<jupyter_text>Slim human-readable parameter representations in the file-names allow to visually inpect the caching directory and help you better understan behavior of your PickleCache-enabled code. Digital fingerprints in the file-names allow PickleCache to uniquely identify and distinguish different parameter values, passed to the function when it was called.  A fast function + Pythagoras But what if a function is fast and an extra step of saving its output into a .pkl file actually slows down the process insted of speeding it up?Pythagoras will still do what we told it to do, but it will give us a warning:<jupyter_code>@demo_cache_obj
def fast_function(x,y):
    return x+" "+y
%%time
result = fast_function("Message for","Bob")
%%time
result = fast_function("Message for","Bob")
print(demo_cache_obj)  # print the status of the cache<jupyter_output>PickleCache in directory <./cache_files> contains 3 files, with total size 2 Kb. There are 275 Gb of free space available in the directory. Cache files are expected to have <.pkl> extension. Input files should be located in <.> folder, which contains 221 files, with total size 942 Kb. Cache READER is ACTIVE: cached versions of objects are loaded from disk if they are available there. Cache WRITER is ACTIVE: new objects get saved to disk as they are created. Names of cache files can not be longer than 250 characters. The following KEY-VALUE pairs will determine caching behaviour if they are present among function parameters or attributes of the parameters on any nested level: {'random_state': None}; a presence of an attribute or a parameter with the KEY name will disable caching, unless the parameter/attribute is set to VALUE. Logger pythagoras.PickleCache.140652536762672.demo_cache_obj (WARNING)<jupyter_text>How To Cache .read_csv()  Reading *.csv* files can be slow. Let's create one and play with it:<jupyter_code>large_dataframe = pd.DataFrame(data = 1000*np.random.rand(7000,7000))
large_dataframe.to_csv("example.csv", index=False)
%%time
new_dataframe = pd.read_csv("example.csv")
new_dataframe.shape<jupyter_output><empty_output><jupyter_text>Now, let's try a version of **.read_csv()** offered by Pythagoras:<jupyter_code>%%time
new_dataframe = demo_cache_obj.read_csv("example.csv")
%%time
new_dataframe = demo_cache_obj.read_csv("example.csv")
new_dataframe.shape<jupyter_output><empty_output><jupyter_text>The second and all subsequent calls to Pythagoras' **.read_csv()** are much faster: we went down from 10.4 seconds to 96 milliseconds - it's 108-x faster. For larger .csv files the difference is even more drastic.   How does it work? The first time we ran **PickleCache.read_csv()**, Pythagoras cached the output of the function.The next time  **PickleCache.read_csv()** was called with the same filename as an argument,Pythagoras re-used that output, without actually reading the original **.csv**.The cached output is stored as a pickle file.If we modify the **.csv** file outside of our notebook, or if you add more argumentsto **PickleCache.read_csv()** call, the same process will repeat.For each new version of the **.csv** file, and for each combination of additional parameters,Pythagoras will create a new cache file.If you have enough disk space, it will save you a lot of your time.Pickle can be loaded much faster than .csv,this approach saves substantial time while working with large .csv files.**PickleCache.read_csv()** accepts all *keyword arguments*which **pandas.read_csv()** accepts,such as *sep*, *names*, *index_col*, *dtype*, *na_values*, etc.You can use them to fine-tune behaviour of **read_csv()**  What is under the hood? For every csv file, which we read with **PickleCach.read_csv()** function ,Pythagoras creates a sub-folder within the main cache_dir folder. The name of sub-folder consists of a keyword "Data" plus the original file's name plus a digital fingerprint of the filename.For instance, for our file **example.csv** the subfolder was named **Data\_\_example.csv\_\_44e43**<jupyter_code>!ls cache_files<jupyter_output>[34mData__example.csv__37f98[m[m              [34mFunc__slowly_process_dataframe__59c21[m[m
[34mFunc__fast_function__08fda[m[m<jupyter_text>Inside each subfolder Pythagoras puts .pkl files with different versions of data from the original .csv file.The name of each .pkl file consists of digital fingerprint of the original .csv filename, plus the size of the original .csv file, plus datetime of its modification, plus encoded information about additional arguments, that were passed to the function.For instance, for our **example.csv** file, the subfolder should include only one file now:<jupyter_code>!ls cache_files/Data__example.csv__44e43<jupyter_output>ls: cache_files/Data__example.csv__44e43: No such file or directory<jupyter_text>But let's call **read_csv()** with an extra argument, and then look into our subfolder again:<jupyter_code>new_dataframe = demo_cache_obj.read_csv(
    "example.csv", usecols=[2,3,4,5,6,7,8,9,10,11,12,12,14,15,16,17,18])
new_dataframe.shape
!ls cache_files/Data__example.csv__44e43<jupyter_output>ls: cache_files/Data__example.csv__44e43: No such file or directory
StarcoderdataJupyterStructuredCleanDedup