python/examples/basic/Getting_Started.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
">### 🚩 *Create a free WhyLabs account to get more value out of whylogs!*<br> \n",
">*Did you know you can store, visualize, and monitor whylogs profiles with the [WhyLabs Observability Platform](https://whylabs.ai/whylogs-free-signup?utm_source=whylogs-Github&utm_medium=whylogs-example&utm_campaign=Getting_Started)? Sign up for a [free WhyLabs account](https://whylabs.ai/whylogs-free-signup?utm_source=whylogs-Github&utm_medium=whylogs-example&utm_campaign=Getting_Started) to leverage the power of whylogs and WhyLabs together!*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Getting Started"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[](https://colab.research.google.com/github/whylabs/whylogs/blob/mainline/python/examples/basic/Getting_Started.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"whylogs provides a standard to log any kind of data.\n",
"\n",
"With whylogs, we will show how to log data, generating statistical summaries called *profiles*. These profiles can be used in a number of ways, like:\n",
"\n",
"* Data Visualization\n",
"* Data Validation\n",
"* Tracking changes in your datasets"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Table of Content"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this example, we'll explore the basics of logging data with whylogs:\n",
"\n",
"- Installing whylogs\n",
"- Profiling data\n",
"- Interacting with the profile\n",
"- Writing/Reading profiles to/from disk"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Installing whylogs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"whylogs is made available as a Python package. You can get the latest version from PyPI with `pip install whylogs`:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# Note: you may need to restart the kernel to use updated packages.\n",
"%pip install whylogs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Minimal requirements:\n",
"\n",
"- Python 3.7+ up to Python 3.10\n",
"- Windows, Linux x86_64, and MacOS 10+"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Loading a Pandas DataFrame"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Before showing how we can log data, we first need the data itself. Let's create a simple Pandas DataFrame:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"data = {\n",
" \"animal\": [\"cat\", \"hawk\", \"snake\", \"cat\"],\n",
" \"legs\": [4, 2, 0, 4],\n",
" \"weight\": [4.3, 1.8, 1.3, 4.1],\n",
"}\n",
"\n",
"df = pd.DataFrame(data)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Profiling with whylogs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To obtain a profile of your data, you can simply use whylogs' `log` call, and navigate through the result to a specific profile with `profile()`:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"import whylogs as why\n",
"\n",
"results = why.log(df)\n",
"profile = results.profile()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Analyzing Profiles"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Once you're done logging the data, you can generate a `Profile View` and inspect it in a Pandas Dataframe format:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>cardinality/est</th>\n",
" <th>cardinality/lower_1</th>\n",
" <th>cardinality/upper_1</th>\n",
" <th>counts/inf</th>\n",
" <th>counts/n</th>\n",
" <th>counts/nan</th>\n",
" <th>counts/null</th>\n",
" <th>distribution/max</th>\n",
" <th>distribution/mean</th>\n",
" <th>distribution/median</th>\n",
" <th>...</th>\n",
" <th>frequent_items/frequent_strings</th>\n",
" <th>type</th>\n",
" <th>types/boolean</th>\n",
" <th>types/fractional</th>\n",
" <th>types/integral</th>\n",
" <th>types/object</th>\n",
" <th>types/string</th>\n",
" <th>types/tensor</th>\n",
" <th>ints/max</th>\n",
" <th>ints/min</th>\n",
" </tr>\n",
" <tr>\n",
" <th>column</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>animal</th>\n",
" <td>3.0</td>\n",
" <td>3.0</td>\n",
" <td>3.00015</td>\n",
" <td>0</td>\n",
" <td>4</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>NaN</td>\n",
" <td>0.000</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>[FrequentItem(value='cat', est=2, upper=2, low...</td>\n",
" <td>SummaryType.COLUMN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>4</td>\n",
" <td>0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>legs</th>\n",
" <td>3.0</td>\n",
" <td>3.0</td>\n",
" <td>3.00015</td>\n",
" <td>0</td>\n",
" <td>4</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>4.0</td>\n",
" <td>2.500</td>\n",
" <td>4.0</td>\n",
" <td>...</td>\n",
" <td>[FrequentItem(value='4', est=2, upper=2, lower...</td>\n",
" <td>SummaryType.COLUMN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>4</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>4.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>weight</th>\n",
" <td>4.0</td>\n",
" <td>4.0</td>\n",
" <td>4.00020</td>\n",
" <td>0</td>\n",
" <td>4</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>4.3</td>\n",
" <td>2.875</td>\n",
" <td>4.1</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>SummaryType.COLUMN</td>\n",
" <td>0</td>\n",
" <td>4</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>3 rows × 31 columns</p>\n",
"</div>"
],
"text/plain": [
" cardinality/est cardinality/lower_1 cardinality/upper_1 counts/inf \\\n",
"column \n",
"animal 3.0 3.0 3.00015 0 \n",
"legs 3.0 3.0 3.00015 0 \n",
"weight 4.0 4.0 4.00020 0 \n",
"\n",
" counts/n counts/nan counts/null distribution/max \\\n",
"column \n",
"animal 4 0 0 NaN \n",
"legs 4 0 0 4.0 \n",
"weight 4 0 0 4.3 \n",
"\n",
" distribution/mean distribution/median ... \\\n",
"column ... \n",
"animal 0.000 NaN ... \n",
"legs 2.500 4.0 ... \n",
"weight 2.875 4.1 ... \n",
"\n",
" frequent_items/frequent_strings type \\\n",
"column \n",
"animal [FrequentItem(value='cat', est=2, upper=2, low... SummaryType.COLUMN \n",
"legs [FrequentItem(value='4', est=2, upper=2, lower... SummaryType.COLUMN \n",
"weight NaN SummaryType.COLUMN \n",
"\n",
" types/boolean types/fractional types/integral types/object \\\n",
"column \n",
"animal 0 0 0 0 \n",
"legs 0 0 4 0 \n",
"weight 0 4 0 0 \n",
"\n",
" types/string types/tensor ints/max ints/min \n",
"column \n",
"animal 4 0 NaN NaN \n",
"legs 0 0 4.0 0.0 \n",
"weight 0 0 NaN NaN \n",
"\n",
"[3 rows x 31 columns]"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"prof_view = profile.view()\n",
"prof_df = prof_view.to_pandas()\n",
"\n",
"prof_df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This will provide you with valuable statistics on a column (feature) basis, such as:\n",
"\n",
"- Counters, such as number of samples and null values\n",
"- Inferred types, such as integral, fractional and boolean\n",
"- Estimated Cardinality\n",
"- Frequent Items\n",
"- Distribution Metrics: min,max, median, quantile values"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Writing to Disk"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can also store your profile in disk for further inspection:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"why.write(profile, \"profile.bin\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This will create a profile binary file in your local filesystem."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Reading from Disk"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can read the profile back into memory with:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"n_prof = why.read(\"profile.bin\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> Note: `write` expects a profile as parameter, while `read` returns a `Profile View`. That means that you can use the loaded profile for visualization purposes and merging, but not for further tracking and updates."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## What's Next?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There's a lot you can do with the profiles you just created. Keep getting your hands dirty with the following examples!"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"- Basic\n",
" - [Visualizing Profiles](https://whylogs.readthedocs.io/en/stable/examples/basic/Notebook_Profile_Visualizer.html) - Compare profiles to detect distribution shifts, visualize histograms and bar charts and explore your data\n",
" - [Logging Data](https://whylogs.readthedocs.io/en/stable/examples/basic/Logging_Different_Data.html) - See the different ways you can log your data with whylogs\n",
" - [Inspecting Profiles](https://whylogs.readthedocs.io/en/stable/examples/basic/Inspecting_Profiles.html) - A deeper dive on the metrics generated by whylogs\n",
" - [Schema Configuration for Tracking Metrics](https://whylogs.readthedocs.io/en/stable/examples/basic/Schema_Configuration.html) - Configure tracking metrics according to data type or column features\n",
" - [Data Constraints](https://whylogs.readthedocs.io/en/stable/examples/advanced/Metric_Constraints.html) - Set constraints to your data to ensure its quality\n",
" - [Merging Profiles](https://whylogs.readthedocs.io/en/stable/examples/basic/Merging_Profiles.html) - Merge your profiles logged across different computing instances, time periods or data segments\n",
"- Integrations\n",
" - [WhyLabs](https://whylogs.readthedocs.io/en/stable/examples/integrations/writers/Writing_to_WhyLabs.html) - Monitor your profiles continuously with the WhyLabs Observability Platform\n",
" - [Pyspark](https://whylogs.readthedocs.io/en/stable/examples/integrations/Pyspark_Profiling.html) - Use whylogs with pyspark\n",
" - [Writing Profiles](https://whylogs.readthedocs.io/en/stable/examples/integrations/writers/Writing_Profiles.html) - See different ways and locations to output your profiles\n",
" - [Flask](https://whylogs.readthedocs.io/en/stable/examples/integrations/flask_streaming/flask_with_whylogs.html) - See how you can create a Flask app with whylogs and WhyLabs integration\n",
" - [Feature Stores](https://whylogs.readthedocs.io/en/stable/examples/integrations/Feature_Stores_and_whylogs.html) - Learn how to log features from your Feature Store with feast and whylogs\n",
" - [BigQuery](https://whylogs.readthedocs.io/en/stable/examples/integrations/BigQuery_Example.html) - Profile data queried from a Google BigQuery table\n",
" - [MLflow](https://whylogs.readthedocs.io/en/stable/examples/integrations/Mlflow_Logging.html) - Log your whylogs profiles to an MLflow environment\n",
"\n",
"Or go to the [examples page](https://whylogs.readthedocs.io/en/stable/examples.html) for the complete list of examples!"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
},
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "5dd5901cadfd4b29c2aaf95ecd29c0c3b10829ad94dcfe59437dbee391154aea"
}
}
},
"nbformat": 4,
"nbformat_minor": 2
}