python/examples/integrations/Pyspark_Profiling.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
">### 🚩 *Create a free WhyLabs account to get more value out of whylogs!*<br> \n",
">*Did you know you can store, visualize, and monitor whylogs profiles with the [WhyLabs Observability Platform](https://whylabs.ai/whylogs-free-signup?utm_source=whylogs-Github&utm_medium=whylogs-example&utm_campaign=Pyspark_Profiling)? Sign up for a [free WhyLabs account](https://whylabs.ai/whylogs-free-signup?utm_source=whylogs-Github&utm_medium=whylogs-example&utm_campaign=Pyspark_Profiling) to leverage the power of whylogs and WhyLabs together!*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# PySpark Integration\n",
"[](https://colab.research.google.com/github/whylabs/whylogs/blob/mainline/python/examples/integrations/Pyspark_Profiling.ipynb)\n",
"\n",
"\n",
"Hi! Perhaps you're already feeling confident with our library, but you really wish there was an easy way to plug our profiling into your existing PySpark jobs. Well, glad you've made it here, because this is what we are going to cover in this example notebook 😃\n",
"\n",
"If you wish to have other insights on how to use whylogs, feel free to check our [other existing examples](https://github.com/whylabs/whylogs/tree/mainline/python/examples), as they might be extremely useful!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Installing the extra dependency\n",
"\n",
"As we want to enable users to have exactly what they need to use from whylogs, the `pyspark` integration comes as an extra dependency. In order to have it available, simply uncomment and run the following cell:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# Note: you may need to restart the kernel to use updated packages.\n",
"%pip install 'whylogs[spark]'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Initializing a SparkSession\n",
"\n",
"Here we will initialize a SparkSession. I'm also setting the `pyarrow` execution config, because it makes our methods even more performant. \n",
"\n",
">**IMPORTANT**: Make sure you have Spark 3.0+ available in your environment, as our implementation relies on it for a smoother integration"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"from pyspark.sql import SparkSession"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"spark = SparkSession.builder.appName('whylogs-testing').getOrCreate()\n",
"arrow_config_key = \"spark.sql.execution.arrow.pyspark.enabled\"\n",
"spark.conf.set(arrow_config_key, \"true\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Reading the data\n",
"\n",
"For the sake of simplicity (and computational efforts, so you can run this notebook from your local machine), we will read the Wine Quality dataset, available in this URL: \"http://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-red.csv\"."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"from pyspark import SparkFiles\n",
"\n",
"data_url = \"http://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-red.csv\"\n",
"spark.sparkContext.addFile(data_url)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
" \r"
]
}
],
"source": [
"spark_dataframe = spark.read.option(\"delimiter\", \";\").option(\"inferSchema\", \"true\").csv(SparkFiles.get(\"winequality-red.csv\"), header=True)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"-RECORD 0----------------------\n",
" fixed acidity | 7.4 \n",
" volatile acidity | 0.7 \n",
" citric acid | 0.0 \n",
" residual sugar | 1.9 \n",
" chlorides | 0.076 \n",
" free sulfur dioxide | 11.0 \n",
" total sulfur dioxide | 34.0 \n",
" density | 0.9978 \n",
" pH | 3.51 \n",
" sulphates | 0.56 \n",
" alcohol | 9.4 \n",
" quality | 5 \n",
"only showing top 1 row\n",
"\n"
]
}
],
"source": [
"spark_dataframe.show(n=1, vertical=True)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"root\n",
" |-- fixed acidity: double (nullable = true)\n",
" |-- volatile acidity: double (nullable = true)\n",
" |-- citric acid: double (nullable = true)\n",
" |-- residual sugar: double (nullable = true)\n",
" |-- chlorides: double (nullable = true)\n",
" |-- free sulfur dioxide: double (nullable = true)\n",
" |-- total sulfur dioxide: double (nullable = true)\n",
" |-- density: double (nullable = true)\n",
" |-- pH: double (nullable = true)\n",
" |-- sulphates: double (nullable = true)\n",
" |-- alcohol: double (nullable = true)\n",
" |-- quality: integer (nullable = true)\n",
"\n"
]
}
],
"source": [
"spark_dataframe.printSchema()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Profiling the data with whylogs\n",
"\n",
"Now that we have a Spark DataFrame in place, let's see how easy it is to profile our data with whylogs."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
" \r"
]
}
],
"source": [
"from whylogs.api.pyspark.experimental import collect_column_profile_views\n",
"\n",
"column_views_dict = collect_column_profile_views(spark_dataframe)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Yeap. It's done. It is **that** easy.\n",
"\n",
"But what do we get with a `column_views_dict`? "
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'alcohol': <whylogs.core.view.column_profile_view.ColumnProfileView object at 0x11d240e20>, 'chlorides': <whylogs.core.view.column_profile_view.ColumnProfileView object at 0x11d29ec70>, 'citric acid': <whylogs.core.view.column_profile_view.ColumnProfileView object at 0x11d2a2d00>, 'density': <whylogs.core.view.column_profile_view.ColumnProfileView object at 0x11d2a6d90>, 'fixed acidity': <whylogs.core.view.column_profile_view.ColumnProfileView object at 0x11d2a3e20>, 'free sulfur dioxide': <whylogs.core.view.column_profile_view.ColumnProfileView object at 0x11d2aaeb0>, 'pH': <whylogs.core.view.column_profile_view.ColumnProfileView object at 0x11d2adf40>, 'quality': <whylogs.core.view.column_profile_view.ColumnProfileView object at 0x11d2b7100>, 'residual sugar': <whylogs.core.view.column_profile_view.ColumnProfileView object at 0x11d2b7d60>, 'sulphates': <whylogs.core.view.column_profile_view.ColumnProfileView object at 0x11d2bddf0>, 'total sulfur dioxide': <whylogs.core.view.column_profile_view.ColumnProfileView object at 0x11d2c0d90>, 'volatile acidity': <whylogs.core.view.column_profile_view.ColumnProfileView object at 0x11d2c4e20>}\n"
]
}
],
"source": [
"print(column_views_dict)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is a dictionary with one `ColumnProfileView` object per column in your dataset. And we can inspect some of the metrics on each one of them, such as the counts for a given column"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(1599, 1599)"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"column_views_dict[\"density\"].get_metric(\"counts\").n.value, spark_dataframe.count()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Or their `mean` value:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.9967466791744841"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"column_views_dict[\"density\"].get_metric(\"distribution\").mean.value"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And now let's check how accurate whylogs did store that `mean` calculation."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"+------------------+\n",
"| avg(density)|\n",
"+------------------+\n",
"|0.9967466791744831|\n",
"+------------------+\n",
"\n"
]
}
],
"source": [
"from pyspark.sql.functions import mean\n",
"spark_dataframe.select(mean(\"density\")).show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is not the literal exact value, but it gets really close, right? That is because we are not extracting the exact information, but we are also **not sampling** the data. `whylogs` will look at **every data point** and *statistically* decide wether or not that data point is relevant to the final calculation. \n",
"\n",
"Is it just me or this is extremely powerful? Yes, it is.\n",
"\n",
"> \"Cool! But what can I do with a bunch of `ColumnProfileView`'s from my Dataset? I want to see everything together\n",
"\n",
"Well, you've come to the right place, because we will inspect the next method that does just that!"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"from whylogs.api.pyspark.experimental import collect_dataset_profile_view\n",
"\n",
"dataset_profile_view = collect_dataset_profile_view(input_df=spark_dataframe)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Yes, that easy. You now have a `DatasetProfileView`. As you might have seen from other example notebooks in our repo, you can turn this *lightweight* object into a pandas DataFrame, and visualize all the important metrics that we've profiled, like this:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>counts/n</th>\n",
" <th>counts/null</th>\n",
" <th>types/integral</th>\n",
" <th>types/fractional</th>\n",
" <th>types/boolean</th>\n",
" <th>types/string</th>\n",
" <th>types/object</th>\n",
" <th>cardinality/est</th>\n",
" <th>cardinality/upper_1</th>\n",
" <th>cardinality/lower_1</th>\n",
" <th>...</th>\n",
" <th>distribution/min</th>\n",
" <th>distribution/q_10</th>\n",
" <th>distribution/q_25</th>\n",
" <th>distribution/median</th>\n",
" <th>distribution/q_75</th>\n",
" <th>distribution/q_90</th>\n",
" <th>type</th>\n",
" <th>ints/max</th>\n",
" <th>ints/min</th>\n",
" <th>frequent_items/frequent_strings</th>\n",
" </tr>\n",
" <tr>\n",
" <th>column</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>alcohol</th>\n",
" <td>1599</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1599</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>65.000010</td>\n",
" <td>65.003256</td>\n",
" <td>65.000000</td>\n",
" <td>...</td>\n",
" <td>8.40000</td>\n",
" <td>9.30000</td>\n",
" <td>9.5000</td>\n",
" <td>10.20000</td>\n",
" <td>11.10000</td>\n",
" <td>12.00000</td>\n",
" <td>SummaryType.COLUMN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>chlorides</th>\n",
" <td>1599</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1599</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>153.000058</td>\n",
" <td>153.007697</td>\n",
" <td>153.000000</td>\n",
" <td>...</td>\n",
" <td>0.01200</td>\n",
" <td>0.06000</td>\n",
" <td>0.0700</td>\n",
" <td>0.07900</td>\n",
" <td>0.09100</td>\n",
" <td>0.10900</td>\n",
" <td>SummaryType.COLUMN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>citric acid</th>\n",
" <td>1599</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1599</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>80.000016</td>\n",
" <td>80.004010</td>\n",
" <td>80.000000</td>\n",
" <td>...</td>\n",
" <td>0.00000</td>\n",
" <td>0.01000</td>\n",
" <td>0.0900</td>\n",
" <td>0.26000</td>\n",
" <td>0.43000</td>\n",
" <td>0.53000</td>\n",
" <td>SummaryType.COLUMN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>density</th>\n",
" <td>1599</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1599</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>439.557368</td>\n",
" <td>445.310933</td>\n",
" <td>433.943761</td>\n",
" <td>...</td>\n",
" <td>0.99007</td>\n",
" <td>0.99451</td>\n",
" <td>0.9956</td>\n",
" <td>0.99675</td>\n",
" <td>0.99786</td>\n",
" <td>0.99914</td>\n",
" <td>SummaryType.COLUMN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>fixed acidity</th>\n",
" <td>1599</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1599</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>96.000023</td>\n",
" <td>96.004816</td>\n",
" <td>96.000000</td>\n",
" <td>...</td>\n",
" <td>4.60000</td>\n",
" <td>6.60000</td>\n",
" <td>7.1000</td>\n",
" <td>7.90000</td>\n",
" <td>9.20000</td>\n",
" <td>10.70000</td>\n",
" <td>SummaryType.COLUMN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 24 columns</p>\n",
"</div>"
],
"text/plain": [
" counts/n counts/null types/integral types/fractional \\\n",
"column \n",
"alcohol 1599 0 0 1599 \n",
"chlorides 1599 0 0 1599 \n",
"citric acid 1599 0 0 1599 \n",
"density 1599 0 0 1599 \n",
"fixed acidity 1599 0 0 1599 \n",
"\n",
" types/boolean types/string types/object cardinality/est \\\n",
"column \n",
"alcohol 0 0 0 65.000010 \n",
"chlorides 0 0 0 153.000058 \n",
"citric acid 0 0 0 80.000016 \n",
"density 0 0 0 439.557368 \n",
"fixed acidity 0 0 0 96.000023 \n",
"\n",
" cardinality/upper_1 cardinality/lower_1 ... \\\n",
"column ... \n",
"alcohol 65.003256 65.000000 ... \n",
"chlorides 153.007697 153.000000 ... \n",
"citric acid 80.004010 80.000000 ... \n",
"density 445.310933 433.943761 ... \n",
"fixed acidity 96.004816 96.000000 ... \n",
"\n",
" distribution/min distribution/q_10 distribution/q_25 \\\n",
"column \n",
"alcohol 8.40000 9.30000 9.5000 \n",
"chlorides 0.01200 0.06000 0.0700 \n",
"citric acid 0.00000 0.01000 0.0900 \n",
"density 0.99007 0.99451 0.9956 \n",
"fixed acidity 4.60000 6.60000 7.1000 \n",
"\n",
" distribution/median distribution/q_75 distribution/q_90 \\\n",
"column \n",
"alcohol 10.20000 11.10000 12.00000 \n",
"chlorides 0.07900 0.09100 0.10900 \n",
"citric acid 0.26000 0.43000 0.53000 \n",
"density 0.99675 0.99786 0.99914 \n",
"fixed acidity 7.90000 9.20000 10.70000 \n",
"\n",
" type ints/max ints/min \\\n",
"column \n",
"alcohol SummaryType.COLUMN NaN NaN \n",
"chlorides SummaryType.COLUMN NaN NaN \n",
"citric acid SummaryType.COLUMN NaN NaN \n",
"density SummaryType.COLUMN NaN NaN \n",
"fixed acidity SummaryType.COLUMN NaN NaN \n",
"\n",
" frequent_items/frequent_strings \n",
"column \n",
"alcohol NaN \n",
"chlorides NaN \n",
"citric acid NaN \n",
"density NaN \n",
"fixed acidity NaN \n",
"\n",
"[5 rows x 24 columns]"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd \n",
"\n",
"dataset_profile_view.to_pandas().head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Persisting as a file\n",
"\n",
"After collecting profiles, it is a good practice to store them as a file. This will allow you to later on read them back, merge with future profiles and track how is your data behaving along the way."
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"dataset_profile_view.write(path=\"my_super_awesome_profile.bin\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And that's it, you have just written a profile generated with spark to your local environment! If you wish to upload to different locations, such as s3, whylabs or others, please make sure to check out our [other examples](https://github.com/whylabs/whylogs/tree/mainline/python/examples) page.\n",
"\n",
"Hopefully this tutorial will help you get started to profile and observe your data behaviour in your Spark jobs with almost no friction :)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Important note\n",
"\n",
"As you might have seen from the imports, currently this pyspark implementation is the **experimental** phase. We ran some benchmark ourselves with it, and for the sake of example, a `90Gb` dataset with 80M rows could be profiled in under 3 minutes! Cool, right? But we still want more users to try this on their own, see if there are places to be improved and give us feedback before we make it officially **the** spark module here. \n",
"Please, feel free to reach out to our [community Slack](https://communityinviter.com/apps/whylabs-community/rsqrd-ai-community) and interact with us there. We will love to hear from you :)"
]
}
],
"metadata": {
"interpreter": {
"hash": "16a10773934acde374a1cd808bcd53b1085f60e17ec18f4c0c26564dd890a5a0"
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.14"
}
},
"nbformat": 4,
"nbformat_minor": 2
}