python/examples/tutorials/Pyspark_and_Constraints.ipynb
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
">### 🚩 *Create a free WhyLabs account to get more value out of whylogs!*<br> \n",
">*Did you know you can store, visualize, and monitor whylogs profiles with the [WhyLabs Observability Platform](https://whylabs.ai/whylogs-free-signup?utm_source=whylogs-Github&utm_medium=whylogs-example&utm_campaign=Pyspark_Constraints)? Sign up for a [free WhyLabs account](https://whylabs.ai/whylogs-free-signup?utm_source=whylogs-Github&utm_medium=whylogs-example&utm_campaign=Pyspark_Constraints) to leverage the power of whylogs and WhyLabs together!*"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Data Validation for Spark Dataframes with whylogs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[](https://colab.research.google.com/github/whylabs/whylogs/blob/mainline/python/examples/tutorials/Pyspark_and_Constraints.ipynb)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"In this example, we will show how you can perform data validation for profiles that were created from a PySpark dataframe. This example is an advanced scenario that combines three topics. If you want to know more about each of these topics, please refer to the following tutorials:\n",
"\n",
"1. [How to create a whylogs profile from a PySpark dataframe](https://nbviewer.org/github/whylabs/whylogs/blob/mainline/python/examples/integrations/Pyspark_Profiling.ipynb)\n",
"2. [How to create user defined condition count metrics](https://nbviewer.org/github/whylabs/whylogs/blob/mainline/python/examples/advanced/Condition_Count_Metrics.ipynb)\n",
"3. [How to create and visualize constraints](https://nbviewer.org/github/whylabs/whylogs/blob/mainline/python/examples/basic/Constraints_Suite.ipynb)\n",
"\n",
"In this example, we will:\n",
"- Create a PySpark dataframe\n",
"- Create a whylogs profile from a PySpark dataframe\n",
"- Create two condition count metrics to check date format and url addresses\n",
"- Create and visualize a set of constraints based on the condition count and other standard metrics"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## About the Dataset - 🛏️ Airbnb Listings in Rio de Janeiro, Brazil\n",
"\n",
"We will read data made available from Airbnb. It's a listing dataset from the city of Rio de Janeiro, Brazil. We'll access data that was adapted from the following location: \"http://data.insideairbnb.com/brazil/rj/rio-de-janeiro/2021-01-26/data/listings.csv.gz\"\n",
"\n",
"In this example, we want to do some basic data validation. Let's define those:\n",
"\n",
"- Completeness Checks\n",
" - `id` (long): should not contain any missing values\n",
" - `listing_url` (string): should not contain any missing values\n",
" - `last_review` (string): should not contain any missing values\n",
"- Consistency Checks\n",
" - `last_review` (string): date should be in the format YYYY-MM-DD\n",
" - `listing_url` (string): should be an url from airbnb (starting with https://www.airbnb.com/rooms/)\n",
" - `latitude` and `longitude` (double): should be within the range of -24 to -22 and -44 to -43 respectively\n",
" - `room_type` (string): frequent strings should be in the set of expected values\n",
"- Statistics Checks\n",
" - `reviews_per_month` (double): standard deviation should be in expected range\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Installing the extra dependency\n",
"\n",
"As we want to enable users to have exactly what they need to use from whylogs, the `pyspark` integration comes as an extra dependency. In order to have it available, simply uncomment and run the following cell:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# Note: you may need to restart the kernel to use updated packages.\n",
"%pip install 'whylogs[spark]'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Initializing a SparkSession\n",
"\n",
"Here we will initialize a SparkSession. I'm also setting the `pyarrow` execution config, because it makes our methods even more performant. \n",
"\n",
">**IMPORTANT**: Make sure you have Spark 3.0+ available in your environment, as our implementation relies on it for a smoother integration"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pyspark.sql import SparkSession\n",
"\n",
"spark = SparkSession.builder.appName('whylogs-testing').getOrCreate()\n",
"arrow_config_key = \"spark.sql.execution.arrow.pyspark.enabled\"\n",
"spark.conf.set(arrow_config_key, \"true\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Creating the PySpark dataframe\n",
"\n",
"\n",
"This is a relatively small dataset, so we can run this example locally."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
" \r"
]
}
],
"source": [
"from pyspark import SparkFiles\n",
"\n",
"data_url = \"https://whylabs-public.s3.us-west-2.amazonaws.com/whylogs_examples/Listings/airbnb_listings.parquet\"\n",
"spark.sparkContext.addFile(data_url)\n",
"\n",
"spark_dataframe = spark.read.parquet(SparkFiles.get(\"airbnb_listings.parquet\"))"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"-RECORD 0--------------------------------------\n",
" name | Very Nice 2Br in ... \n",
" description | Discounts for lon... \n",
" listing_url | https://www.airbn... \n",
" last_review | 2020-12-26 \n",
" number_of_reviews_ltm | 13 \n",
" number_of_reviews_l30d | 0 \n",
" id | 17878 \n",
" latitude | -22.96592 \n",
" longitude | -43.17896 \n",
" availability_365 | 286 \n",
" bedrooms | 2.0 \n",
" bathrooms | null \n",
" reviews_per_month | 2.01 \n",
" room_type | Entire home/apt \n",
"only showing top 1 row\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" \r"
]
}
],
"source": [
"spark_dataframe.show(n=1, vertical=True)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"root\n",
" |-- name: string (nullable = true)\n",
" |-- description: string (nullable = true)\n",
" |-- listing_url: string (nullable = true)\n",
" |-- last_review: string (nullable = true)\n",
" |-- number_of_reviews_ltm: long (nullable = true)\n",
" |-- number_of_reviews_l30d: long (nullable = true)\n",
" |-- id: long (nullable = true)\n",
" |-- latitude: double (nullable = true)\n",
" |-- longitude: double (nullable = true)\n",
" |-- availability_365: long (nullable = true)\n",
" |-- bedrooms: double (nullable = true)\n",
" |-- bathrooms: double (nullable = true)\n",
" |-- reviews_per_month: double (nullable = true)\n",
" |-- room_type: string (nullable = true)\n",
"\n"
]
}
],
"source": [
"spark_dataframe.printSchema()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Creating the Condition Count Metrics\n",
"\n",
"To create a profile with the standard metrics, we can simply call `collect_dataset_profile_view` from whylog's PySpark extra module. However, if we look at our defined set of constraints, there are two of those that need to checked agains individual values:\n",
"\n",
"- `last_review` (string): date should be in the format YYYY-MM-DD\n",
"- `listing_url` (string): should be an url from airbnb (starting with https://www.airbnb.com/rooms/)\n",
"\n",
"As opposed to the other constraints, that can be checked against aggregate metrics, these two need to be checked against individual values. For that, we will create two condition count metrics. Later on, we will create metric constraints based on these metrics."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"import datetime\n",
"from whylogs.core.relations import Predicate\n",
"from typing import Any\n",
"from whylogs.core.metrics.condition_count_metric import Condition\n",
"from whylogs.core.schema import DeclarativeSchema\n",
"from whylogs.core.resolvers import STANDARD_RESOLVER\n",
"from whylogs.core.specialized_resolvers import ConditionCountMetricSpec\n",
"\n",
"def date_format(x: Any) -> bool:\n",
" date_format = '%Y-%m-%d'\n",
" try:\n",
" datetime.datetime.strptime(x, date_format)\n",
" return True\n",
" except ValueError:\n",
" return False\n",
"\n",
"last_review_conditions = {\"is_date_format\": Condition(Predicate().is_(date_format))}\n",
"listing_url_conditions = {\"url_matches_airbnb_domain\": Condition(Predicate().matches(\"^https:\\/\\/www.airbnb.com\\/rooms\"))}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that we have the our set of conditions for both columns, we can create the condition count metrics. We can do so by creating a Standard Schema and then extending it by adding the condition count metrics with `add_condition_count_metrics`:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"schema = DeclarativeSchema(STANDARD_RESOLVER)\n",
"\n",
"schema.add_resolver_spec(column_name=\"last_review\", metrics=[ConditionCountMetricSpec(last_review_conditions)])\n",
"schema.add_resolver_spec(column_name=\"listing_url\", metrics=[ConditionCountMetricSpec(listing_url_conditions)])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"> To know more about condition count metrics and how to use them, check out the [Metric Constraints with Condition Count Metrics](https://nbviewer.org/github/whylabs/whylogs/blob/mainline/python/examples/advanced/Metric_Constraints_with_Condition_Count_Metrics.ipynb) example."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Profiling the PySpark DataFrame\n",
"\n",
"Now, we can use the schema to pass to our logger through `collect_dataset_profile_view`"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
" \r"
]
}
],
"source": [
"from whylogs.api.pyspark.experimental import collect_dataset_profile_view\n",
"\n",
"dataset_profile_view = collect_dataset_profile_view(input_df=spark_dataframe, schema=schema)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"This will create a profile with the standard metrics, as well as the two condition count metrics that we created. As a sanity check, let's see the metrics for the `last_review` column:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['types',\n",
" 'cardinality',\n",
" 'counts',\n",
" 'distribution',\n",
" 'frequent_items',\n",
" 'condition_count']"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dataset_profile_view.get_column(\"last_review\").get_metric_names()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Creating and Visualizing Metric Constraints\n",
"\n",
"We have all that we need to build our set of constraints. We will use out-of-the-box factory constraints to do that:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[ReportResult(name='last_review meets condition is_date_format', passed=1, failed=0, summary=None),\n",
" ReportResult(name='last_review has no missing values', passed=0, failed=1, summary=None),\n",
" ReportResult(name='listing_url meets condition url_matches_airbnb_domain', passed=1, failed=0, summary=None),\n",
" ReportResult(name='listing_url has no missing values', passed=1, failed=0, summary=None),\n",
" ReportResult(name='latitude is in range [-24,-22]', passed=1, failed=0, summary=None),\n",
" ReportResult(name='longitude is in range [-44,-43]', passed=1, failed=0, summary=None),\n",
" ReportResult(name='id has no missing values', passed=1, failed=0, summary=None),\n",
" ReportResult(name='reviews_per_month standard deviation between 0.8 and 1.1 (inclusive)', passed=1, failed=0, summary=None),\n",
" ReportResult(name=\"room_type values in set {'Shared room', 'Hotel room', 'Private room', 'Entire home/apt'}\", passed=1, failed=0, summary=None)]"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from whylogs.core.constraints.factories import condition_meets\n",
"from whylogs.core.constraints import ConstraintsBuilder\n",
"from whylogs.core.constraints.factories import no_missing_values\n",
"from whylogs.core.constraints.factories import is_in_range\n",
"from whylogs.core.constraints.factories import stddev_between_range\n",
"from whylogs.core.constraints.factories import frequent_strings_in_reference_set\n",
"\n",
"builder = ConstraintsBuilder(dataset_profile_view=dataset_profile_view)\n",
"reference_set = {\"Entire home/apt\", \"Private room\", \"Shared room\", \"Hotel room\"}\n",
"\n",
"builder.add_constraint(condition_meets(column_name=\"last_review\", condition_name=\"is_date_format\"))\n",
"builder.add_constraint(condition_meets(column_name=\"listing_url\", condition_name=\"url_matches_airbnb_domain\"))\n",
"builder.add_constraint(no_missing_values(column_name=\"last_review\"))\n",
"builder.add_constraint(no_missing_values(column_name=\"listing_url\"))\n",
"builder.add_constraint(is_in_range(column_name=\"latitude\",lower=-24,upper=-22))\n",
"builder.add_constraint(is_in_range(column_name=\"longitude\",lower=-44,upper=-43))\n",
"builder.add_constraint(no_missing_values(column_name=\"id\"))\n",
"builder.add_constraint(stddev_between_range(column_name=\"reviews_per_month\", lower=0.8, upper=1.1))\n",
"builder.add_constraint(frequent_strings_in_reference_set(column_name=\"room_type\", reference_set=reference_set))\n",
"\n",
"constraints = builder.build()\n",
"constraints.generate_constraints_report()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"> If you're interested in a more complete list of helper constraints, please check out the [Constraints Suite](https://nbviewer.org/github/whylabs/whylogs/blob/mainline/python/examples/basic/Constraints_Suite.ipynb) example."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, we can visualize the constraints report using the __Notebook Profile Visualizer__:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Connection error. Skip stats collection.\n"
]
},
{
"data": {
"text/html": [
"<div></div><iframe srcdoc=\"<!DOCTYPE html>\n",
"<html lang="en">\n",
" <head>\n",
" <meta charset="UTF-8" />\n",
" <meta http-equiv="X-UA-Compatible" content="IE=edge" />\n",
" <meta name="viewport" content="width=device-width, initial-scale=1.0" />\n",
" <meta name="description" content="" />\n",
" <meta name="author" content="" />\n",
"\n",
" <title>Profile Visualizer | whylogs</title>\n",
"\n",
" <link rel="icon" href="images/whylabs-favicon.png" type="image/png" sizes="16x16" />\n",
" <link rel="preconnect" href="https://fonts.googleapis.com" />\n",
" <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />\n",
" <link href="https://fonts.googleapis.com/css2?family=Asap:wght@400;500;600;700&display=swap" rel="stylesheet" />\n",
" <link rel="preconnect" href="https://fonts.gstatic.com" />\n",
" <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap@5.1.0/dist/css/bootstrap.min.css" />\n",
"\n",
" <script\n",
" src="https://cdnjs.cloudflare.com/ajax/libs/handlebars.js/4.7.7/handlebars.min.js"\n",
" integrity="sha512-RNLkV3d+aLtfcpEyFG8jRbnWHxUqVZozacROI4J2F1sTaDqo1dPQYs01OMi1t1w9Y2FdbSCDSQ2ZVdAC8bzgAg=="\n",
" crossorigin="anonymous"\n",
" referrerpolicy="no-referrer"\n",
" ></script>\n",
"\n",
" <style type="text/css">\n",
"\n",
" /* Screen on smaller screens */\n",
" .no-responsive {\n",
" display: none;\n",
" position: fixed;\n",
" top: 0;\n",
" left: 0;\n",
" z-index: 1031;\n",
" width: 100vw;\n",
" height: 100vh;\n",
" background-color: var(--tealBackground);\n",
" display: flex;\n",
" align-items: center;\n",
" justify-content: center;\n",
" }\n",
"\n",
" @media screen and (min-width: 1000px) {\n",
" .desktop-content {\n",
" display: block;\n",
" }\n",
" .no-responsive {\n",
" display: none;\n",
" }\n",
" }\n",
"\n",
" .no-responsive__content {\n",
" max-width: 600px;\n",
" width: 100%;\n",
" padding: 0 24px;\n",
" }\n",
"\n",
" .no-responsive__title {\n",
" font-size: 96px;\n",
" font-weight: 300;\n",
" color: var(--brandSecondary900);\n",
" line-height: 1.167;\n",
" }\n",
"\n",
" .no-responsive__text {\n",
" margin: 0;\n",
" font-size: 16px;\n",
" font-weight: 400;\n",
" color: var(--brandSecondary900);\n",
" line-height: 1.5;\n",
" }\n",
"\n",
" .header-title {\n",
" font-size: 26px;\n",
" font-weight: 700;\n",
" color: #444444;\n",
" }\n",
"\n",
" .tooltip-full-number {\n",
" position: relative;\n",
" display: inline-block;\n",
" }\n",
"\n",
" .tooltip-full-number .tooltiptext {\n",
" visibility: hidden;\n",
" background: black;\n",
" color: white;\n",
" border: 1px solid black;\n",
" text-align: start;\n",
" padding: 3px;\n",
" position: absolute;\n",
" z-index: 1;\n",
" top: 0;\n",
" left: 100%;\n",
" margin-left: 5px;\n",
" opacity: 0;\n",
" transition: opacity 0.5s;\n",
" font-size: 13px;\n",
" font-weight: normal;\n",
" line-height: 100%;\n",
" }\n",
"\n",
" .tooltip-full-number:hover .tooltiptext {\n",
" visibility: visible;\n",
" opacity: 1;\n",
" }\n",
"\n",
"\n",
" .wl-compare-profile {\n",
" position: relative;\n",
" left: 0;\n",
" padding: 30px;\n",
" margin-bottom: 20px;\n",
" background: var(--white);;\n",
" border-bottom: 1px solid #CED4DA;\n",
" }\n",
"\n",
" .alert-list {\n",
" padding: 30px;\n",
" padding-top: 0;\n",
" }\n",
"\n",
" .drift-detection {\n",
" justify-content: space-between;\n",
" align-items: center;\n",
" }\n",
"\n",
" .drift-detection-info-circle {\n",
" width: 15px;\n",
" height: 15px;\n",
" border-radius: 50px;\n",
" display: inline-block;\n",
" margin-right: 8px;\n",
" }\n",
"\n",
" .drift-detection-info-drifts-item {\n",
" padding-right: 20px;\n",
" }\n",
"\n",
" .drift-detection-info-title {\n",
" font-family: Arial;\n",
" font-weight: bold;\n",
" font-size: 22px;\n",
" line-height: 130%;\n",
" color: #313B3D;\n",
" }\n",
"\n",
" .drift-detection-info-drifts-item-count {\n",
" font-family: Arial;\n",
" font-weight: bold;\n",
" font-size: 14px;\n",
" line-height: 16px;\n",
" color: #000000;\n",
" padding-right: 8px;\n",
" }\n",
"\n",
" .drift-detection-info-drifts-item-name {\n",
" font-family: Arial;\n",
" font-style: normal;\n",
" font-weight: normal;\n",
" font-size: 12px;\n",
" line-height: 14px;\n",
" color: #000000;\n",
" }\n",
"\n",
" .drift-detection-search-input {\n",
" display: flex;\n",
" align-items: center;\n",
" background: rgba(255, 255, 255, 0.7);\n",
" border: 1px solid #DBE5E7;\n",
" box-sizing: border-box;\n",
" border-radius: 4px;\n",
" width: 170px;\n",
" padding-left: 10px;\n",
" }\n",
"\n",
" .drift-detection-search-input img{\n",
" margin-right: 5px;\n",
" }\n",
"\n",
" .drift-detection-search-input input::placeholder {\n",
" font-family: Arial;\n",
" font-weight: normal;\n",
" font-size: 13px;\n",
" line-height: 16px;\n",
" color: #313B3D;\n",
" }\n",
"\n",
" .dropdown-container {\n",
" position: absolute;\n",
" right: 30px;\n",
" top: 80px;\n",
" z-index: 999;\n",
" background: #FFFFFF;\n",
" border: 1px solid #DBE5E7;\n",
" box-sizing: border-box;\n",
" box-shadow: 0px 2px 8px rgba(0, 0, 0, 0.05);\n",
" border-radius: 4px;\n",
" padding: 10px !important;\n",
" border: none !important;\n",
" }\n",
"\n",
" .filter-options-title {\n",
" width: 240px;\n",
" }\n",
"\n",
" .filter-options-title p {\n",
" margin: 0;\n",
" }\n",
"\n",
" .form-check-input:checked {\n",
" background-color: #0E7384;\n",
" border-color: #0E7384;\n",
" }\n",
"\n",
" .form-check-input[type=checkbox] {\n",
" border-radius: 2px;\n",
" }\n",
"\n",
" .search-input{\n",
" padding-top: 0 !important;\n",
" padding-bottom: 0 !important;\n",
" }\n",
"\n",
" .search-input input{\n",
" border: none;\n",
" background: none;\n",
" outline: none;\n",
" height: 40px;\n",
" width: 100%;\n",
" font-size: 14px;\n",
" }\n",
"\n",
" .search-input img{\n",
" height: 19px;\n",
" pointer-events: none;\n",
" }\n",
"\n",
" input::placeholder {\n",
" color: var(--secondaryLight1000);\n",
" }\n",
"\n",
" .statistics {\n",
" width: 100%;\n",
" }\n",
"\n",
" .close-filter-button {\n",
" display: flex;\n",
" justify-content: center;\n",
" align-items: center;\n",
" background: rgba(255, 255, 255, 0.7);\n",
" border: 1px solid #369BAC;\n",
" box-sizing: border-box;\n",
" border-radius: 4px;\n",
" width: 40px;\n",
" height: 40px;\n",
" cursor: pointer;\n",
" margin-left: 10px;\n",
" }\n",
"\n",
" .statistic-number-title {\n",
" font-family: Arial;\n",
" font-weight: normal;\n",
" font-size: 14px;\n",
" line-height: 20px;\n",
" color: #6C757D;\n",
" }\n",
"\n",
" .statistic-number {\n",
" font-family: Arial;\n",
" font-weight: bold;\n",
" font-size: 20px;\n",
" line-height: 140%;\n",
" display: flex;\n",
" align-items: center;\n",
" color: #4F595B;\n",
" }\n",
"\n",
" .full-summary-statistics-wrap {\n",
" padding: 20px;\n",
" }\n",
"\n",
" .statistics {\n",
" width: 100%;\n",
" }\n",
"\n",
" .statistics-list {\n",
" width: 100% ;\n",
" }\n",
"\n",
" mark {\n",
" padding: 5px;\n",
" border-radius: 4px;\n",
" font-family: Arial;\n",
" font-weight: normal;\n",
" font-size: 14px;\n",
" line-height: 140%;\n",
" }\n",
"\n",
" .blue-mark {\n",
" background-color: #369BAC1A;\n",
" color: #369BAC;\n",
" }\n",
"\n",
" .red-mark {\n",
" background-color: #FFEFEE;\n",
" color: #F5473C;\n",
" }\n",
"\n",
" .alert-tag {\n",
" height: 27px;\n",
" padding: 5px;\n",
" border-radius: 4px;\n",
" font-family: Arial;\n",
" font-weight: bold;\n",
" font-size: 12px;\n",
" line-height: 140%;\n",
" color: #FFFFFF;\n",
" }\n",
"\n",
" .turquoise-background-color {\n",
" background-color: #1DBB42;\n",
" }\n",
"\n",
" .bordeaux-background-color {\n",
" background-color: #C6462A;\n",
" }\n",
"\n",
" .border-solid-gray {\n",
" border: 1px solid #CED4DA;\n",
" border-radius: 4px;\n",
" }\n",
"\n",
" .display-flex {\n",
" display: flex;\n",
" }\n",
"\n",
" .justify-content-space-between {\n",
" justify-content: space-between;\n",
" }\n",
"\n",
" .justify-content-center {\n",
" justify-content: center;\n",
" }\n",
"\n",
" .align-items-center {\n",
" align-items: center;\n",
" }\n",
"\n",
" .align-items-flex-start {\n",
" align-items: flex-start;\n",
" }\n",
"\n",
" .padding-right-30 {\n",
" padding-right: 30px;\n",
" }\n",
"\n",
" .notif-circle-container{\n",
" position: absolute;\n",
" top: 25px;\n",
" right: 25px;\n",
" padding: 5.3px;\n",
" border-radius: 50%;\n",
" background-color: white;\n",
" cursor: pointer;\n",
" }\n",
"\n",
" .notif-circle {\n",
" position: absolute;\n",
" top: 2px;\n",
" right: 2px;\n",
" padding: 3.3px;\n",
" border-radius: 50%;\n",
" background-color: #F2994A;\n",
" }\n",
"\n",
" .alert-list-text {\n",
" width: 70%\n",
" }\n",
"\n",
" @media screen and (min-width: 500px) {\n",
" .desktop-content {\n",
" display: block;\n",
" }\n",
" .no-responsive {\n",
" display: none;\n",
" }\n",
" }\n",
" </style>\n",
" </head>\n",
"\n",
" <body id="generated-html"></body>\n",
"\n",
" <script id="entry-template" type="text/x-handlebars-template">\n",
" \n",
" <div class="desktop-content">\n",
" <div class="full-summary-statistics-wrap">\n",
" <div class="full-summary-statistics">\n",
" <div>\n",
" <div class="display-flex justify-content-center">\n",
" <div class="statistics-list border-solid-gray">\n",
" <div>\n",
" <div class="wl-compare-profile" id="compare-profile">\n",
" <div class="drift-detection-wrap">\n",
" <div class="drift-detection display-flex align-items-flex-start">\n",
" <div class="drift-detection-info flex-direction-colum">\n",
" <div class="drift-detection-info-title-wrap display-flex">\n",
" <p class="drift-detection-info-title">\n",
" Constraints Report\n",
" </p>\n",
" </div>\n",
"\n",
" <!-- </div> -->\n",
" </div>\n",
" <div class="drift-detection-search-input-wrap display-flex">\n",
" <div class="drift-detection-search-input search-input">\n",
" <input type="text" id="wl__feature-search" placeholder="Quick search..."/>\n",
" <img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAGdSURBVHgBpZK7TkJBEIZnZoVocdTYyQNALxpNKPQBMEaNJsbCRMKhl8ISWwt6AaksCF5iTHgAGhJD4AHkAaAzGiwUsjvOQnO4SYh/cXbPzO43OxcEj9Zy92EiFSXNIfvPyE1kKFfdoxJMENpP6DrvLC0vJoEwCgwto7DWcxoIIHBYbA3NmKwnDltjAeuZhyul1DaTTlfPB6Nt5Z53DOgky4P875+nlctY2+unjZviLklkJhi5bPUa3y/7qJuQUM7PinMy7CdQc1Gh16vnBxPzrMROmlKQEgKNASAHLQCmSIGpS75O+O5pdQAgVXaIqTkNwDDXHmcnW3VmHZoGMLoTsOt88+NrAMCIZdu+iLTyTwKRa1Md6YKfOgXbzO7K8sWku5u5RxcRV5EpPezrzcHGbXEXWaUkgkweZ/UC9YrK3zqggFw5FBZfm8EUavHj7AjAKpIvBDrGn+pNnlcyhYgqbcC41idr1gvB4SdZkDbzQa21gwv0Vj07aPTtL07XdDOyDXohCDNoHIRmAVRie20f+RKybRDQDvxHkXy/7b/DrayncLbMwQAAAABJRU5ErkJggg=="/>\n",
" </div>\n",
" <div class="wl__dropdown_arrow-icon">\n",
" <div onclick="openFilter()" class="close-filter-button">\n",
" <div class="display-flex close-filter-icon d-none">\n",
" <img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAALCAYAAACprHcmAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAADFSURBVHgBfY+xDcIwEEXvnLQBZQkYAEhDwwKpEEK0CCZgAEjJCEmgjYSAygxAHTZgFRSOsyUjY5mcZFnn/+78PwBXf3+MoKWUPuYjVBPFnTwpr9t/oNJfcTfXsAhRAlDqDhhQIPYgpAqNMDqcUqSAYZT1epr9gAHt6uXshvYme4DYHQJNDKh0dD0m5WXB10Y3Fqjtuh7fROn3oREDWxfeMLyRsMnc0OgDzdduaA0Pi3Plgr7Q2kaAePeBqh6rueSNBVt6fgCwBV1JLF3rlAAAAABJRU5ErkJggg=="/>\n",
" </div>\n",
" <div class="display-flex filter-icon">\n",
" <img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAPCAYAAADtc08vAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAACSSURBVHgBrZLBCYAwDEWTUjw7igcdwI1cxkNXUJzBEVzFAbSVKoKmaVrEB6GHJv/w+AACtRk7P9IONv1QOYUl96k0zv61m2tjARoLtSDI3EFsgIJ4uoXrMLazO72CRG2mzg/8BSdVlEjhpGZJjAWdAZJECpWalEhJSs1pHuUlMad5FFai1Lwg4Ckx1TxKIPFL8w55mEWd8VjPGAAAAABJRU5ErkJggg=="/>\n",
" </div>\n",
" </div>\n",
" <span class="notif-circle-container">\n",
" <span class="notif-circle"></span>\n",
" </span>\n",
" </div>\n",
" </div>\n",
" </div>\n",
" </div>\n",
" <div class="dropdown-container flex-direction-colum mb-2 d-none" id="dropdown-container">\n",
" <div class="filter-options">\n",
" <div class="filter-options-title space-between dropdown">\n",
" <p>Select a view:</p>\n",
" </div>\n",
" <div class="form-check mb-1 mt-2">\n",
" <input\n",
" class="form-check-input wl__feature-filter-input"\n",
" type="checkbox"\n",
" name="checkbox"\n",
" value="Discrete"\n",
" id="inferredDiscrete"\n",
" checked\n",
" />\n",
" <label class="form-check-label" for="inferredDiscrete">\n",
" Failed constraints (<span class="wl__feature-count--discrete"></span>)\n",
" </label>\n",
" </div>\n",
" <div class="form-check mb-1">\n",
" <input\n",
" class="form-check-input wl__feature-filter-input"\n",
" type="checkbox"\n",
" name="checkbox"\n",
" value="Non-discrete"\n",
" id="inferredNonDiscrete"\n",
" checked\n",
" />\n",
" <label class="form-check-label" for="inferredNonDiscrete">\n",
" Passed constraints (<span\n",
" class="wl__feature-count--non-discrete"\n",
" ></span>)\n",
" </label>\n",
" </div>\n",
" <div class="form-check mb-1">\n",
" <input\n",
" class="form-check-input wl__feature-filter-input"\n",
" type="checkbox"\n",
" name="checkbox"\n",
" value="Unknown"\n",
" id="inferredUnknown"\n",
" checked\n",
" />\n",
" <label class="form-check-label" for="inferredUnknown">\n",
" All constraints (<span class="wl__feature-count--unknown"></span>)\n",
" </label>\n",
" </div>\n",
" </div>\n",
" </div>\n",
" </div>\n",
" <div class="alert-list" id="alert-list">\n",
" {{{alertLIst this}}}\n",
" </div>\n",
" </div>\n",
" </div>\n",
" </div>\n",
" </div>\n",
" </div>\n",
" </div>\n",
" </div>\n",
" <div class="no-responsive">\n",
" <div class="no-responsive__content">\n",
" <h1 class="no-responsive__title">Hold on! :)</h1>\n",
" <p class="no-responsive__text">\n",
" It looks like your current screen size or device is not yet supported by the WhyLabs Sandbox. The Sandbox is\n",
" best experienced on a desktop computer. Please try maximizing this window or switching to another device. We\n",
" are working on adding support for a larger variety of devices.\n",
" </p>\n",
" </div>\n",
" </div>\n",
" \n",
" </script>\n",
"\n",
" <script src="https://code.jquery.com/jquery-3.6.0.min.js" integrity="sha256-/xUj+3OJU5yExlq6GSYGSHk7tPXikynS7ogEvDej/m4=" crossorigin="anonymous"></script>\n",
"\n",
" <script src="https://cdnjs.cloudflare.com/ajax/libs/d3/6.7.0/d3.min.js" integrity="sha512-cd6CHE+XWDQ33ElJqsi0MdNte3S+bQY819f7p3NUHgwQQLXSKjE4cPZTeGNI+vaxZynk1wVU3hoHmow3m089wA==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>\n",
"\n",
" <script>\n",
" function registerHandlebarHelperFunctions() {\n",
" //helper fun\n",
" function formatLabelDate(timestamp) {\n",
" const date = new Date(timestamp);\n",
" const format = d3.timeFormat("%Y-%m-%d %I:%M:%S %p %Z");\n",
" return format(date);\n",
" }\n",
"\n",
" function fixNumberTo(number, decimals = 3) {\n",
" return parseFloat(number).toFixed(decimals);\n",
" }\n",
"\n",
" const randomNumbers = (range) => Math.floor(Math.random() * range)\n",
"\n",
" const findFetureWithNumberSummary = (column) => {\n",
" const fetureIndex = Object.values(column.columns)\n",
" .findIndex((feture) => feture.numberSummary)\n",
"\n",
" return Object.keys(column.columns)[fetureIndex]\n",
" }\n",
"\n",
" const alertListItemStatus = (status, passedItem, failedItem) => {\n",
" if (status) {\n",
" return passedItem\n",
" } else {\n",
" return failedItem\n",
" }\n",
" }\n",
"\n",
" const alertListElement = (name, text, status, summary) => {\n",
" if (summary == null){\n",
" return (\n",
" `<div\n",
" data-inferred-type=${alertListItemStatus(status, "passed", "failed")}\n",
" class="alert-list-item display-flex justify-content-space-between align-items-center mb-2"\n",
" >\n",
" <div class="alert-list-text">\n",
" ${\n",
" name &&\n",
" alertListItemStatus(\n",
" status,\n",
" `<mark class="blue-mark">${name}</mark>`,\n",
" `<mark class="red-mark">${name}</mark>`\n",
" )\n",
" }\n",
" ${text}\n",
" </div>\n",
" ${\n",
" alertListItemStatus(\n",
" status,\n",
" `\n",
" <div class="turquoise-background-color alert-tag">Passed</div>\n",
" `\n",
" ,\n",
" `\n",
" <div class="bordeaux-background-color alert-tag">Failed</div>\n",
" `\n",
" )\n",
" }\n",
" </div>`\n",
" )\n",
"\n",
" }\n",
" return (\n",
" `<div\n",
" data-inferred-type=${alertListItemStatus(status, "passed", "failed")}\n",
" class="alert-list-item display-flex justify-content-space-between align-items-center mb-2"\n",
" >\n",
" <div class="alert-list-text">\n",
" ${\n",
" name &&\n",
" alertListItemStatus(\n",
" status,\n",
" `<mark class="blue-mark">${name}</mark>`,\n",
" `<mark class="red-mark">${name}</mark>`\n",
" )\n",
" }\n",
" ${text}\n",
" </div>\n",
" ${\n",
" alertListItemStatus(\n",
" status,\n",
" `\n",
" <div class="tooltip-full-number">\n",
" <div class="turquoise-background-color alert-tag">Passed\n",
" <span class="tooltiptext">\n",
" <pre class="mb-1"> ${JSON.stringify(summary, null, 2)} </pre>\n",
" </span>\n",
" </div>\n",
" </div>`\n",
"\n",
" ,\n",
" `\n",
" <div class="tooltip-full-number">\n",
" <div class="bordeaux-background-color alert-tag">Failed\n",
" <span class="tooltiptext">\n",
" <pre class="mb-1"> ${JSON.stringify(summary, null, 2)} </pre>\n",
" </span>\n",
" </div>\n",
" </div>`\n",
" )\n",
" }\n",
" </div>`\n",
" )\n",
" }\n",
"\n",
" let failedConstraints = 0;\n",
"\n",
" Handlebars.registerHelper("getProfileTimeStamp", function (column) {\n",
" return formatLabelDate(+column.properties.dataTimestamp)\n",
" });\n",
"\n",
" Handlebars.registerHelper("getProfileName", function (column) {\n",
" return column.properties.tags.name\n",
" });\n",
"\n",
"\n",
" Handlebars.registerHelper("alertLIst", function (column) {\n",
" let alertListItem = column.map((value) => {\n",
" if (value[1][0]) {\n",
" let alertListValue = value[1].map((cstr)=>{\n",
" return alertListElement(value[0],cstr[0],cstr[cstr.length - 1] === 0 || (failedConstraints++, false), value[3])\n",
" })\n",
" return alertListValue.join(' ')\n",
" } else {\n",
" return alertListElement('', value[0], value[2] === 0 || (failedConstraints++, false), value[3])\n",
" }\n",
" })\n",
" $(document).ready(() => {\n",
" $(".wl__feature-count--discrete").append(failedConstraints)\n",
" $(".wl__feature-count--non-discrete").append(column.length - failedConstraints)\n",
" $(".wl__feature-count--unknown").append(column.length)\n",
" })\n",
" return alertListItem.join(' ')\n",
" });\n",
"\n",
" }\n",
"\n",
" function openFilter() {\n",
" const $filterOptions = $(".dropdown-container");\n",
" const filterClass = $filterOptions.attr("class");\n",
"\n",
" if (filterClass.indexOf("d-none") > 0) {\n",
" $filterOptions.removeClass("d-none");\n",
" $(".filter-icon").addClass("d-none")\n",
" $(".close-filter-icon").removeClass("d-none")\n",
" } else {\n",
" $filterOptions.addClass("d-none");\n",
" $(".close-filter-icon").addClass("d-none")\n",
" $(".filter-icon").removeClass("d-none")\n",
" }\n",
" }\n",
"\n",
" function initHandlebarsTemplate() {\n",
" // Replace this context with JSON from .py file\n",
" const context = [["last_review meets condition is_date_format", 1, 0, {"metric": "condition_count", "total": 16465, "is_date_format": 16465}], ["last_review has no missing values", 0, 1, {"metric": "counts", "n": 26106, "null": 9641, "nan": 0, "inf": 0}], ["listing_url meets condition url_matches_airbnb_domain", 1, 0, {"metric": "condition_count", "total": 26106, "url_matches_airbnb_domain": 26106}], ["listing_url has no missing values", 1, 0, {"metric": "counts", "n": 26106, "null": 0, "nan": 0, "inf": 0}], ["latitude is in range [-24,-22]", 1, 0, {"metric": "distribution", "mean": -22.965736037217624, "stddev": 0.03505941949491759, "n": 26106, "max": -22.74982, "min": -23.07292, "q_01": -23.02806, "q_05": -23.01204, "q_10": -23.00693, "q_25": -22.98454, "median": -22.97157, "q_75": -22.95076, "q_90": -22.91718, "q_95": -22.90981, "q_99": -22.83753}], ["longitude is in range [-44,-43]", 1, 0, {"metric": "distribution", "mean": -43.24905026415256, "stddev": 0.0965622753402193, "n": 26106, "max": -43.10486, "min": -43.70479, "q_01": -43.52144, "q_05": -43.45984, "q_10": -43.40554, "q_25": -43.30428, "median": -43.19638, "q_75": -43.18631, "q_90": -43.17741, "q_95": -43.17483, "q_99": -43.16903}], ["id has no missing values", 1, 0, {"metric": "counts", "n": 26106, "null": 0, "nan": 0, "inf": 0}], ["reviews_per_month standard deviation between 0.8 and 1.1 (inclusive)", 1, 0, {"metric": "distribution", "mean": 0.6285800182204677, "stddev": 0.8569652523205165, "n": 16465, "max": 25.0, "min": 0.01, "q_01": 0.02, "q_05": 0.03, "q_10": 0.05, "q_25": 0.09, "median": 0.27, "q_75": 0.91, "q_90": 1.7, "q_95": 2.32, "q_99": 3.81}], ["room_type values in set {'Shared room', 'Hotel room', 'Private room', 'Entire home/apt'}", 1, 0, {"metric": "frequent_items", "frequent_strings_top_10": ["Entire home/apt:18876", "Private room:6512", "Shared room:604", "Hotel room:114"]}]];\n",
" // Config handlebars and pass data to HBS template\n",
" const source = document.getElementById("entry-template").innerHTML;\n",
" const template = Handlebars.compile(source);\n",
" const html = template(context);\n",
" const target = document.getElementById("generated-html");\n",
" target.innerHTML = html;\n",
" }\n",
"\n",
" function initWebsiteScripts() {\n",
" const $featureSearch = document.getElementById("wl__feature-search");\n",
" const $alertList = document.getElementById("alert-list");\n",
" const $discrete = document.getElementById("inferredDiscrete");\n",
" const $nonDiscrete = document.getElementById("inferredNonDiscrete");\n",
" const $unknown = document.getElementById("inferredUnknown");\n",
"\n",
" const activeTypes = {\n",
" passed: true,\n",
" failed: true\n",
" };\n",
"\n",
" let searchString = "";\n",
"\n",
" function debounce(func, wait, immediate) {\n",
" let timeout;\n",
"\n",
" return function () {\n",
" const context = this;\n",
" const args = arguments;\n",
" const later = function () {\n",
" timeout = null;\n",
" if (!immediate) func.apply(context, args);\n",
" };\n",
"\n",
" const callNow = immediate && !timeout;\n",
" clearTimeout(timeout);\n",
" timeout = setTimeout(later, wait);\n",
" if (callNow) func.apply(context, args);\n",
" };\n",
" }\n",
"\n",
" function filterNotification() {\n",
" const $notifCircleContainer = $(".notif-circle-container")\n",
" const $boxes = $('.wl_filter-options>.form-check>input[name=checkbox]:checked');\n",
" const item = Object.values($boxes).find(function(value) { return $(value)[0] === undefined});\n",
" if (item === undefined) {\n",
" $notifCircleContainer.removeClass("d-none")\n",
" } else {\n",
" $notifCircleContainer.addClass("d-none")\n",
" }\n",
" }\n",
"\n",
" function handleSearch() {\n",
" const tableBodyChildren = $alertList.children;\n",
"\n",
" for (let i = 0; i < tableBodyChildren.length; i++) {\n",
" const type = tableBodyChildren[i].dataset.inferredType.toLowerCase();\n",
" const name = $(tableBodyChildren[i].children[0]).html().toLowerCase();\n",
" if (activeTypes[type] && name.includes(searchString)) {\n",
" tableBodyChildren[i].style.display = "";\n",
" } else {\n",
" tableBodyChildren[i].style.display = "none";\n",
" }\n",
" }\n",
" }\n",
"\n",
" const checkedBoxes = () => {\n",
" if ($('.form-check-input:checked').length === $(".form-check-input").length - 1) {\n",
" $($(".form-check-input")[$(".form-check-input").length - 1]).prop( "checked", true );\n",
" }\n",
" }\n",
"\n",
" $featureSearch.addEventListener(\n",
" "keyup",\n",
" debounce((event) => {\n",
" searchString = event.target.value.toLowerCase();\n",
" handleSearch();\n",
" }, 100),\n",
" );\n",
"\n",
" $discrete.addEventListener("change", (event) => {\n",
" const currentCheckbox = $(event.currentTarget);\n",
"\n",
" if (event.currentTarget.checked) {\n",
" activeTypes["failed"] = true;\n",
" checkedBoxes();\n",
" } else {\n",
" activeTypes["failed"] = false;\n",
" $($(".form-check-input")[$(".form-check-input").length - 1]).prop( "checked", false );\n",
" }\n",
" handleSearch();\n",
" });\n",
"\n",
" $nonDiscrete.addEventListener("change", (event) => {\n",
" const currentCheckbox = $(event.currentTarget);\n",
"\n",
" if (event.currentTarget.checked) {\n",
" activeTypes["passed"] = true;\n",
" checkedBoxes();\n",
" } else {\n",
" activeTypes["passed"] = false;\n",
" $($(".form-check-input")[$(".form-check-input").length - 1]).prop( "checked", false );\n",
" }\n",
" handleSearch();\n",
" });\n",
"\n",
" $unknown.addEventListener("change", (event) => {\n",
" const currentCheckbox = $(event.currentTarget);\n",
"\n",
" if (event.currentTarget.checked) {\n",
" $(".form-check-input").prop( "checked", true );\n",
" activeTypes["passed"] = true;\n",
" activeTypes["failed"] = true;\n",
" checkedBoxes();\n",
" } else {\n",
" $(".form-check-input").prop( "checked", false );\n",
" activeTypes["passed"] = false;\n",
" activeTypes["failed"] = false;\n",
" }\n",
" handleSearch();\n",
" });\n",
" }\n",
"\n",
" function checkedBoxes() {\n",
" const $boxes = $('input[name=checkbox]:checked');\n",
" const $notifCircleContainer = $(".notif-circle-container")\n",
"\n",
" if ($boxes.length) {\n",
" $notifCircleContainer.removeClass("d-none")\n",
" }\n",
" }\n",
"\n",
" function openFilter() {\n",
" const $filterOptions = $(".dropdown-container");\n",
" const $notifCircleContainer = $(".notif-circle-container")\n",
" const filterClass = $filterOptions.attr("class");\n",
"\n",
" if (filterClass.indexOf("d-none") > 0) {\n",
" $notifCircleContainer.addClass("d-none")\n",
" $filterOptions.removeClass("d-none");\n",
" $(".filter-icon").addClass("d-none")\n",
" $(".close-filter-icon").removeClass("d-none")\n",
" } else {\n",
" $filterOptions.addClass("d-none");\n",
" $(".close-filter-icon").addClass("d-none")\n",
" $(".filter-icon").removeClass("d-none")\n",
" checkedBoxes()\n",
" }\n",
" }\n",
"\n",
" // Invoke functions -- keep in mind invokation order\n",
" registerHandlebarHelperFunctions();\n",
" initHandlebarsTemplate();\n",
" initWebsiteScripts();\n",
" </script>\n",
"</html>\n",
"\" width=100% height=300\n",
" frameBorder=0></iframe>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from whylogs.viz import NotebookProfileVisualizer\n",
"\n",
"visualization = NotebookProfileVisualizer()\n",
"visualization.constraints_report(constraints, cell_height=300)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Looks like we have some missing values for `last_review`. Other than that, the data looks good!"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}