<jupyter_start><jupyter_text>How to use `DataAssistants`* A `DataAssistant` enables you to quickly profile your data by providing a thin API over a pre-constructed `RuleBasedProfiler` configuration.* As a result of the profiling, you get back a result object consisting of     * `Metrics` that describe the current state of the data    * `Expectations` that are able to alert you if the data deviates from the expected state in the future.     * `DataAssistant` results can also be plotted to help you understand their data visually.* There are multiple `DataAssistants` centered around a theme (volume, nullity etc), and this notebook walks you through an example `VolumeDataAssistant` to show the capabilities and potential of this new interface.  What is a `VolumeDataAssistant`?* The `VolumeDataAssistant` allows you to automatically build a set of Expectations that alerts you if the volume of records significantly deviates from the norm. More specfically, the `VolumeDataAssistant` profiles the data and outputs an `ExpectationSuite` containing 2 `Expecation` types * `expect_table_row_count_to_be_between`* `expect_column_unique_value_count_to_be_between`with automatically selected values for upper and lower bound. The ranges are selected using a bootstrapping step on the sample `Batches`. This allows the `DataAssistant` to account for outliers, allowing it to obtain a more accurate estimate of the true ranges by taking into account the underlying distribution.<jupyter_code>import great_expectations as ge
from great_expectations.core.yaml_handler import YAMLHandler
from great_expectations.core.batch import BatchRequest
from great_expectations.core import ExpectationSuite
from great_expectations.core.expectation_configuration import ExpectationConfiguration
from great_expectations.validator.validator import Validator
from great_expectations.rule_based_profiler.data_assistant import (
    DataAssistant,
    VolumeDataAssistant,
)
from great_expectations.rule_based_profiler.types.data_assistant_result import (
    VolumeDataAssistantResult,
)
from typing import List
yaml = YAMLHandler()<jupyter_output>/Users/work/Development/ENVs/supercon_ge/lib/python3.8/site-packages/snowflake/connector/options.py:94: UserWarning: You have an incompatible version of 'pyarrow' installed (7.0.0), please install a version that adheres to: 'pyarrow<3.1.0,>=3.0.0; extra == "pandas"'
  warn_incompatible_dep(<jupyter_text>Set-up: Adding `taxi_data` `Datasource`* Add `taxi_data` as a new `Datasource`* We are using an `InferredAssetFilesystemDataConnector` to connect to data in the `test_sets/taxi_yellow_tripdata_samples` folder and get one `DataAsset` (`yellow_tripdata_sample`) that has 36 Batches, corresponding to one batch per month from 2018-2020.<jupyter_code>data_context: ge.DataContext = ge.get_context()
data_path: str = "../../../../test_sets/taxi_yellow_tripdata_samples"

datasource_config: dict = {
    "name": "taxi_data_all_years",
    "class_name": "Datasource",
    "module_name": "great_expectations.datasource",
    "execution_engine": {
        "module_name": "great_expectations.execution_engine",
        "class_name": "PandasExecutionEngine",
    },
    "data_connectors": {
        "inferred_data_connector_all_years": {
            "class_name": "InferredAssetFilesystemDataConnector",
            "base_directory": data_path,
            "default_regex": {
                "group_names": ["data_asset_name", "year", "month"],
                "pattern": "(yellow_tripdata_sample)_(2018|2019|2020)-(\\d.*)\\.csv",
            },
        },
    },
}

data_context.test_yaml_config(yaml.dump(datasource_config))
# add_datasource only if it doesn't already exist in our configuration
try:
    data_context.get_datasource(datasource_config["name"])
except ValueError:
    data_context.add_datasource(**datasource_config)<jupyter_output><empty_output><jupyter_text>Configure `BatchRequest` In this example, we will be using a `BatchRequest` that will return all 36 batches of data from the `taxi_data` dataset.  We will refer to the `Datasource` and `DataConnector` configured in the previous step.<jupyter_code>multi_batch_all_years_batch_request: BatchRequest = BatchRequest(
    datasource_name="taxi_data_all_years",
    data_connector_name="inferred_data_connector_all_years",
    data_asset_name="yellow_tripdata_sample",
)
batch_request: BatchRequest = multi_batch_all_years_batch_request<jupyter_output><empty_output><jupyter_text>Run the `VolumeDataAssistant` * The `VolumeDataAssistant` can be run directly from the `DataContext` by specifying `assistants` and `volume`, and passing in the `BatchRequest` from the previous step.<jupyter_code>result: VolumeDataAssistantResult = data_context.assistants.volume.run(batch_request=batch_request)<jupyter_output><empty_output><jupyter_text>Explore `DataAssistantResult` by plotting The resulting `DataAssistantResult` can be best explored by plotting. For each `Domain` considered (`Table` and `Column` in our case), the plots will display the value for each `Batch` (36 in total).<jupyter_code>result.plot_metrics()<jupyter_output><empty_output><jupyter_text>An additional layer of information that can be retrieved from the `DataAssistantResult` is the `prescriptive` information, which corresponds to the range values of the `Expectations` that result from the `DataAssistant` run. For example the `vendor_id` plot will show that the range of distinct `vendor_id` values ranged from 2-3 across all of our `Batches`, as indicated by the blue band around the plotted values. These values correspond to the `max_value` and `min_value` for the resulting `Expectation`, `expect_column_unique_value_count_to_be_between`.<jupyter_code>result.plot_expectations_and_metrics()<jupyter_output><empty_output><jupyter_text>Save `ExpectationSuite` Finally, we can save the `ExpectationConfiguration` objext resulting from the `DataAssistant` in our `ExpectationSuite` and then use the `DataContext`'s `save_expectation_suite()` method to pass in our `ExpectationSuite`, updated with the `DataAssistant`.<jupyter_code>suite: ExpectationSuite = ExpectationSuite(expectation_suite_name="taxi_data_suite")
resulting_configurations: List[ExpectationConfiguration] = suite.add_expectation_configurations(expectation_configurations=result.expectation_configurations)
data_context.save_expectation_suite(expectation_suite=suite)<jupyter_output><empty_output><jupyter_text>Optional: Clean-up Directory As part of running this notebook, the `DataAssistant` will create a number of ExpectationSuite configurations in the `great_expectations/expectations/tmp` directory. Optionally run the following cell to clean up the directory.<jupyter_code>#import shutil, os
#shutil.rmtree("great_expectations/expectations/tmp")
#os.remove("great_expectations/expectations/.ge_store_backend_id")
#os.remove("great_expectations/expectations/taxi_data_suite.json")<jupyter_output><empty_output>