python/examples/advanced/Condition_Count_Metrics.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "XcM_iA58LESk"
},
"source": [
">### 🚩 *Create a free WhyLabs account to get more value out of whylogs!*<br> \n",
">*Did you know you can store, visualize, and monitor whylogs profiles with the [WhyLabs Observability Platform](https://whylabs.ai/whylogs-free-signup?utm_source=whylogs-Github&utm_medium=whylogs-example&utm_campaign=Condition_Count_Metrics)? Sign up for a [free WhyLabs account](https://whylabs.ai/whylogs-free-signup?utm_source=whylogs-Github&utm_medium=whylogs-example&utm_campaign=Condition_Count_Metrics) to leverage the power of whylogs and WhyLabs together!*"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-1c1qLf2LESo"
},
"source": [
"# Condition Count Metrics"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GQtQSWaeLESp"
},
"source": [
"[](https://colab.research.google.com/github/whylabs/whylogs/blob/mainline/python/examples/advanced/Condition_Count_Metrics.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Ud4uQbvZLESp"
},
"source": [
"By default, whylogs tracks several metrics, such as type counts, distribution metrics, cardinality and frequent items. Those are general metrics that are useful for a lot of use cases, but often we need metrics tailored for our application.\n",
"\n",
"__Condition Count Metrics__ gives you the flexibility to define your own customized metrics. It will return the results as counters, which is the number of times the condition was met for a given column. With it, you can define conditions such as regex matching for strings, equalities or inequalities for numerical features, and even define your own function to check for any given condition.\n",
"\n",
"In this example, we will cover:\n",
"\n",
"1. Create metrics for __regex matching__\n",
" - Examples: contains email/credit card number (String features)\n",
"2. Create metrics for __(in)equalities__\n",
" - Examples: equal, less, greater, less than, greater than (Numerical features)\n",
"3. Combining metrics with __logic operators__ (and, or, not)\n",
" - Examples: Between range, outside of range, not equal (Numerical features)\n",
"4. Creating metrics with __custom functions__\n",
" - Examples: is even number, is text preprocessed (Any type)\n",
"5. Going Further: Combining this example with other whylogs' features\n",
"6. (APPENDIX) Complete code snippets - The complete code snippets (to make it easir to copy and paste)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "OXyozsmBLESq"
},
"source": [
"## Installing whylogs and importing modules"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"id": "XM5l89DYLESr"
},
"outputs": [],
"source": [
"# Note: you may need to restart the kernel to use updated packages.\n",
"%pip install whylogs"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GQ8_k2otLESs"
},
"source": [
"Let's import all the dependencies for this example upfront:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "cUNUFdCNLESt"
},
"outputs": [],
"source": [
"import pandas as pd\n",
"from typing import Any\n",
"\n",
"import whylogs as why\n",
"from whylogs.core.resolvers import STANDARD_RESOLVER\n",
"from whylogs.core.specialized_resolvers import ConditionCountMetricSpec\n",
"from whylogs.core.datatypes import Fractional, Integral\n",
"from whylogs.core.metrics.condition_count_metric import Condition\n",
"from whylogs.core.relations import Not, Predicate\n",
"from whylogs.core.schema import DeclarativeSchema"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eg_avmJcLESt"
},
"source": [
"## 1. Regex Matching"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-w4mqELyLESu"
},
"source": [
"Suppose we have textual columns in our data in which we want to make sure certain elements are present / not present.\n",
"\n",
"For example, for privacy and security issues, we might be interested in tracking the number of times a credit card number appears on a given column, or if we have sensitive email information in another column.\n",
"\n",
"With whylogs, we can define metrics that will count the number of occurences a certain regex pattern is met for a given column."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mvsHXXpoLESu"
},
"source": [
"### Creating sample dataframe"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qFtzgMZkLESu"
},
"source": [
"Let's create a simple dataframe.\n",
"\n",
"In this scenario, the `emails` column should have only a valid email, nothing else. As for the `trascriptions` column, we want to make sure existing credit card number was properly masked or removed."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"id": "w-bdwXUzLESv"
},
"outputs": [],
"source": [
"data = {\n",
" \"emails\": [\"my email is my_email_1989@gmail.com\",\"invalidEmail@xyz.toolong\",\"this.is.ok@hotmail.com\",\"not an email\"],\n",
" \"transcriptions\": [\"Bob's credit card number is 4000000000000\", \"Alice's credit card is XXXXXXXXXXXXX\", \"Hi, my name is Bob\", \"Hi, I'm Alice\"],\n",
"}\n",
"df = pd.DataFrame(data=data)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "cQ0BHCubLESw"
},
"source": [
"The conditions are defined through a whylogs' `Condition` object. There are several different ways of assembling a condition. In the following example, we will define two different regex patterns, one for each column. Since we can define multiple conditions for a single column, we'll assemble the conditions into dictionaries, where the key is the condition name. Each dictionary will be later attached to the relevant column."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"id": "uKeefoVbLESx"
},
"outputs": [],
"source": [
"emails_conditions = {\n",
" \"containsEmail\": Condition(Predicate().fullmatch(\"[\\w.]+[\\._]?[a-z0-9]+[@]\\w+[.]\\w{2,3}\")),\n",
"}\n",
"\n",
"transcriptions_conditions = {\n",
" \"containsCreditCard\": Condition(Predicate().matches(\".*4[0-9]{12}(?:[0-9]{3})?\"))\n",
"}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"whylogs must be aware of those conditions while profiling the data. We can do that by creating a Standard Schema, and then simply adding the conditions to the schema with `add_resolver_spec`. That way, we can pass our enhanced schema when calling `why.log()` later."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"schema = DeclarativeSchema(STANDARD_RESOLVER)\n",
"schema.add_resolver_spec(column_name=\"emails\", metrics=[ConditionCountMetricSpec(emails_conditions)])\n",
"schema.add_resolver_spec(column_name=\"transcriptions\", metrics=[ConditionCountMetricSpec(transcriptions_conditions)])\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5HdjR8UpLESx"
},
"source": [
"> Note: The regex expressions are for demonstrational purposes only. These expressions are not general - there will be emails and credit cards whose patterns will not be met by the expression."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2if1p-bELESy"
},
"source": [
"Now, we only need to pass our schema when logging our data. Let's also take a look at the metrics, to make sure everythins was tracked correctly:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"id": "Pf4x-8aBLESy"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>condition_count/containsEmail</th>\n",
" <th>condition_count/containsCreditCard</th>\n",
" <th>condition_count/total</th>\n",
" </tr>\n",
" <tr>\n",
" <th>column</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>emails</th>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>transcriptions</th>\n",
" <td>NaN</td>\n",
" <td>1.0</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" condition_count/containsEmail \\\n",
"column \n",
"emails 1.0 \n",
"transcriptions NaN \n",
"\n",
" condition_count/containsCreditCard condition_count/total \n",
"column \n",
"emails NaN 4 \n",
"transcriptions 1.0 4 "
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"prof_view = why.log(df, schema=schema).profile().view()\n",
"prof_view.to_pandas()[['condition_count/containsEmail', 'condition_count/containsCreditCard', 'condition_count/total']]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WYf6nZazLES0"
},
"source": [
"Let's check the numbers:\n",
"\n",
"For `emails` feature, only one occurence was met for `containsEmail`. That is expected, because the only valid row is the third one (\"this.is.ok@hotmail.com\"). Others either don't contain an email, are invalid emails or have extra text that are not an email (note we're using `fullmatch` as the predicate for the email condition).\n",
"\n",
"For `transcriptions` column, we also have only one match. That is well, since only the first row has a match with the given pattern, and others either don't have a credit card number or are properly \"hidden\". Note that in this case we want to check for the pattern inside a broader text, so we're using `.*` before the pattern, so the text doesn't have to start with the pattern (whylogs' `Predicate.matches` uses python's `re.compile().match()` under the hood.)\n",
"\n",
"The available relations for regex matching are the ones used in this example:\n",
"\n",
"- `matches`\n",
"- `fullmatch`"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zAxreFQ1LES0"
},
"source": [
"## 2. Numerical Equalities and Inequalities"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JD_dyTLCLES0"
},
"source": [
"For this one, let's create integer and floats columns:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"data = {\n",
" \"ints_column\": [1,12,42,4],\n",
" \"floats_column\": [1.2, 12.3, 42.2, 4.8]\n",
"\n",
"}\n",
"df = pd.DataFrame(data=data)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "fmfszjBPLES1"
},
"source": [
"As before, we will create our set of conditions for each column and pass both to our schema:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"id": "WXqqGzEsLES1"
},
"outputs": [],
"source": [
"ints_conditions = {\n",
" \"equals42\": Condition(Predicate().equals(42)),\n",
" \"lessthan5\": Condition(Predicate().less_than(5)),\n",
" \"morethan40\": Condition(Predicate().greater_than(40)),\n",
" \n",
"}\n",
"\n",
"floats_conditions = {\n",
" \"equals42.2\": Condition(Predicate().equals(42.2)),\n",
" \"lessthan5\": Condition(Predicate().less_than(5)),\n",
" \"morethan40\": Condition(Predicate().greater_than(40)),\n",
"}\n",
"\n",
"schema = DeclarativeSchema(STANDARD_RESOLVER)\n",
"schema.add_resolver_spec(column_type=Integral, metrics=[ConditionCountMetricSpec(ints_conditions)])\n",
"schema.add_resolver_spec(column_type=Fractional, metrics=[ConditionCountMetricSpec(floats_conditions)])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "S_pLqQg3LES2"
},
"source": [
"Let's log and check the metrics:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"id": "dcyX0FtwLES2"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>types/fractional</th>\n",
" <th>types/integral</th>\n",
" <th>condition_count/lessthan5</th>\n",
" <th>condition_count/morethan40</th>\n",
" <th>condition_count/equals42</th>\n",
" <th>condition_count/equals42.2</th>\n",
" <th>condition_count/total</th>\n",
" </tr>\n",
" <tr>\n",
" <th>column</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>floats_column</th>\n",
" <td>4</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" <td>1</td>\n",
" <td>NaN</td>\n",
" <td>1.0</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>ints_column</th>\n",
" <td>0</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" <td>1</td>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" types/fractional types/integral condition_count/lessthan5 \\\n",
"column \n",
"floats_column 4 0 2 \n",
"ints_column 0 4 2 \n",
"\n",
" condition_count/morethan40 condition_count/equals42 \\\n",
"column \n",
"floats_column 1 NaN \n",
"ints_column 1 1.0 \n",
"\n",
" condition_count/equals42.2 condition_count/total \n",
"column \n",
"floats_column 1.0 4 \n",
"ints_column NaN 4 "
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"prof_view = why.log(df, schema=schema).profile().view()\n",
"prof_view.to_pandas()[['types/fractional','types/integral','condition_count/lessthan5', 'condition_count/morethan40','condition_count/equals42','condition_count/equals42.2', 'condition_count/total']]\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5aaL0ugwLES3"
},
"source": [
"We can simply check the original data to verify that the metrics are correct.\n",
"We used `equals`, `less_than` and `greater_than` in this example, but here's the complete list of available relations:\n",
"\n",
"- `equals` - equal to\n",
"- `less_than` - less than\n",
"- `less_or_equals` - less than or equal to\n",
"- `greater_than` - greater than\n",
"- `greater_or_equals` - greater than or equal to\n",
"- `not_equal` - not equal to\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Ae6SrQMWLES3"
},
"source": [
"## 3. Combining metrics with logical operators - AND, OR, NOT"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lOuLn__uLES4"
},
"source": [
"You can also combine relations with logical operators such as __AND__, __OR__ and __NOT__.\n",
"\n",
"Let's stick with the numerical features to show how you can combine relations to assemble conditions such as:\n",
"\n",
"- Value is between a certain range\n",
"- Value is outside a certin range\n",
"- Value is NOT a certain number"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 208
},
"id": "PV3ZG1MsLES5",
"outputId": "9cfd388d-875d-45d0-fc49-b64b00fa18e0"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>condition_count/between10and50</th>\n",
" <th>condition_count/outside10and50</th>\n",
" <th>condition_count/not_42</th>\n",
" <th>condition_count/total</th>\n",
" </tr>\n",
" <tr>\n",
" <th>column</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>floats_column</th>\n",
" <td>2</td>\n",
" <td>2</td>\n",
" <td>4</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>ints_column</th>\n",
" <td>2</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" condition_count/between10and50 condition_count/outside10and50 \\\n",
"column \n",
"floats_column 2 2 \n",
"ints_column 2 2 \n",
"\n",
" condition_count/not_42 condition_count/total \n",
"column \n",
"floats_column 4 4 \n",
"ints_column 3 4 "
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"conditions = {\n",
" \"between10and50\": Condition(Predicate().greater_than(10).and_(Predicate().less_than(50))),\n",
" \"outside10and50\": Condition(Predicate().less_than(10).or_(Predicate().greater_than(50))),\n",
" \"not_42\": Condition(Not(Predicate().equals(42))), # could also use X.not_equal(42) or X.not_.equals(42)\n",
"}\n",
"\n",
"schema = DeclarativeSchema(STANDARD_RESOLVER)\n",
"schema.add_resolver_spec(column_name=\"ints_column\", metrics=[ConditionCountMetricSpec(conditions)])\n",
"schema.add_resolver_spec(column_name=\"floats_column\", metrics=[ConditionCountMetricSpec(conditions)])\n",
"\n",
"prof_view = why.log(df, schema=schema).profile().view()\n",
"prof_view.to_pandas()[['condition_count/between10and50', 'condition_count/outside10and50', 'condition_count/not_42', 'condition_count/total']]\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7L4VKkP-LES6"
},
"source": [
"Available logical operators are:\n",
"- `and_`\n",
"- `or_`\n",
"- `not_`\n",
"- `Not`\n",
"\n",
"Note that `and_`, `or_`, and `not_` are methods called on a `Predicate` and passed another `Predicate`, while `Not` is a function that takes a single `Predicate` argument.\n",
"Even though we showed these operators with numerical features, this also works with regex matching conditions shown previously."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TjzelbFXLES6"
},
"source": [
"## 4. Custom Condition with User-defined functions"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "axVo_aiDLES7"
},
"source": [
"If none of the previously conditions are suited to your use case, you are free to define your own custom function to create your own metrics.\n",
"\n",
"Let's see a simple example: suppose we want to check if a certain number is even.\n",
"\n",
"We can define a `even` predicate function, as simple as:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"id": "uPo7Jj0DLES7"
},
"outputs": [],
"source": [
"def even(x: Any) -> bool:\n",
" return x % 2 == 0"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "kCitdHZ9LES7"
},
"source": [
"And then we proceed as usual, defining our condition and adding it to the schema:\n",
"\n",
"We only have to pass the name of the function to `conditions` as a `Condition` object, like below:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 164
},
"id": "yvLsXjXnLES8",
"outputId": "814594ec-a61c-41e8-c7e0-0c8497be1bd4"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>condition_count/isEven</th>\n",
" <th>condition_count/total</th>\n",
" </tr>\n",
" <tr>\n",
" <th>column</th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>floats_column</th>\n",
" <td>0</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>ints_column</th>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" condition_count/isEven condition_count/total\n",
"column \n",
"floats_column 0 4\n",
"ints_column 3 4"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"conditions = {\n",
" \"isEven\": Condition(Predicate().is_(even)),\n",
"}\n",
"\n",
"schema = DeclarativeSchema(STANDARD_RESOLVER)\n",
"schema.add_resolver_spec(column_name=\"ints_column\", metrics=[ConditionCountMetricSpec(conditions)])\n",
"schema.add_resolver_spec(column_name=\"floats_column\", metrics=[ConditionCountMetricSpec(conditions)])\n",
"\n",
"prof_view = why.log(df, schema=schema).profile().view()\n",
"prof_view.to_pandas()[['condition_count/isEven', 'condition_count/total']]\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qt613cdkLES8"
},
"source": [
"### NLP example"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6vCGaYihLES9"
},
"source": [
"For user-defined functions, the sky's the limit for what you can do.\n",
"\n",
"Let's think of another simple cenario for NLP. Suppose our model assumes text to be a certain way. Maybe it was trained and expects:\n",
"\n",
"- lowercased characters\n",
"- no digits\n",
"- no stopwords"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-l6AOeHgLES9"
},
"source": [
"Let's check these conditions for the data below:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"id": "GQdw9WvSLES-"
},
"outputs": [],
"source": [
"data = {\n",
" \"transcriptions\": [\"I AM BOB AND I LIKE TO SCREAM\",\"i am bob\",\"am alice and am xx years old\",\"am bob and am 42 years old\"],\n",
" \"ints\": [0,1,2,3],\n",
"}\n",
"df = pd.DataFrame(data=data)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "E0AmapoKLES-"
},
"source": [
"Once again, let's define our function:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"id": "cH8mavgcLES-"
},
"outputs": [],
"source": [
"def preprocessed(x: Any) -> bool:\n",
" stopwords = [\"i\", \"me\", \"myself\"]\n",
" if not isinstance(x, str):\n",
" return False\n",
" \n",
" # should have only lowercase letters and space (no digits)\n",
" if not all(c.islower() or c.isspace() for c in x):\n",
" return False\n",
" # should not contain any words in our stopwords list\n",
" if any(c in stopwords for c in x.split()):\n",
" return False\n",
" return True"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nmZksJuaLES_"
},
"source": [
"> Since this is an example, our `stopwords` list is only a placeholder for the real thing."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0Fa4qoodLES_"
},
"source": [
"The rest is the same as before:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 164
},
"id": "RZ-upW26LES_",
"outputId": "e5a6fddd-9ecb-48b5-9f07-8d9601efa51e"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>condition_count/isPreprocessed</th>\n",
" <th>condition_count/total</th>\n",
" </tr>\n",
" <tr>\n",
" <th>column</th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>ints</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>transcriptions</th>\n",
" <td>1.0</td>\n",
" <td>4.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" condition_count/isPreprocessed condition_count/total\n",
"column \n",
"ints NaN NaN\n",
"transcriptions 1.0 4.0"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"conditions = {\n",
" \"isPreprocessed\": Condition(Predicate().is_(preprocessed)),\n",
"}\n",
"\n",
"schema = DeclarativeSchema(STANDARD_RESOLVER)\n",
"schema.add_resolver_spec(column_name=\"transcriptions\", metrics=[ConditionCountMetricSpec(conditions)])\n",
"\n",
"prof_view = why.log(df, schema=schema).profile().view()\n",
"prof_view.to_pandas()[['condition_count/isPreprocessed', 'condition_count/total']]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qfIELSXpLETA"
},
"source": [
"For the `transcriptions` feature, we can see that only the second row is properly preprocessed (\"am alice and am xx years old\"). The first one contained uppercase characters, the third contained a stopword and the last one contained digits. For the integers column, isPreprocessed returns 0, since it's not a string value. "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VXXvWKAKLETA"
},
"source": [
"## 5. Going Further"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "pLdUNP_rLETA"
},
"source": [
"You can combine this example with other whylogs' features to cover even more scenarios.\n",
"\n",
"Here are some pointers for some possible use cases:\n",
"\n",
"- I want to track other types of metrics, not only Condition Counts!\n",
" - Check the [Schema Configuration](https://whylogs.readthedocs.io/en/stable/examples/basic/Schema_Configuration.html) example!\n",
"- Ok, I have counters. What now?\n",
" - You can set constraints (such as `containsCreditCardNumber` should always be 0). Check the [Metric Constraints with Condition Count Metrics](https://whylogs.readthedocs.io/en/stable/examples/advanced/Metric_Constraints_with_Condition_Count_Metrics.html) example!\n",
" - You can store it locally or on S3 for future inspection - Check the [Writing Profiles](https://whylogs.readthedocs.io/en/stable/examples/integrations/writers/Writing_Profiles.html) example!\n",
" - You can send your profiles to a monitoring platform, such as WhyLabs - Check the [Writing Profiles to WhyLabs](https://whylogs.readthedocs.io/en/stable/examples/integrations/writers/Writing_to_WhyLabs.html) example!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Fw5DTL9YLETA"
},
"source": [
"## Appendix - Complete Code Snippets"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SY9CWzZRLETB"
},
"source": [
"Here are the complete code snippets - just to make it easier to copy/paste!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "wt2HwhvoLETB"
},
"source": [
"### Regex example"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 208
},
"id": "5wZg7OdHLETB",
"outputId": "4a15d8ff-5dbc-4acb-f6d9-f1ab9e700cad"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>condition_count/containsEmail</th>\n",
" <th>condition_count/containsCreditCard</th>\n",
" <th>condition_count/total</th>\n",
" </tr>\n",
" <tr>\n",
" <th>column</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>emails</th>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>transcriptions</th>\n",
" <td>NaN</td>\n",
" <td>1.0</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" condition_count/containsEmail \\\n",
"column \n",
"emails 1.0 \n",
"transcriptions NaN \n",
"\n",
" condition_count/containsCreditCard condition_count/total \n",
"column \n",
"emails NaN 4 \n",
"transcriptions 1.0 4 "
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"\n",
"import whylogs as why\n",
"from whylogs.core.resolvers import STANDARD_RESOLVER\n",
"from whylogs.core.specialized_resolvers import ConditionCountMetricSpec\n",
"from whylogs.core.metrics.condition_count_metric import Condition\n",
"from whylogs.core.relations import Predicate\n",
"from whylogs.core.schema import DeclarativeSchema\n",
"\n",
"data = {\n",
" \"emails\": [\"my email is my_email_1989@gmail.com\",\"invalidEmail@xyz.toolong\",\"this.is.ok@hotmail.com\",\"not an email\"],\n",
" \"transcriptions\": [\"Bob's credit card number is 4000000000000\", \"Alice's credit card is XXXXXXXXXXXXX\", \"Hi, my name is Bob\", \"Hi, I'm Alice\"],\n",
"}\n",
"df = pd.DataFrame(data=data)\n",
"\n",
"emails_conditions = {\n",
" \"containsEmail\": Condition(Predicate().fullmatch(\"[\\w.]+[\\._]?[a-z0-9]+[@]\\w+[.]\\w{2,3}\")),\n",
"}\n",
"\n",
"transcriptions_conditions = {\n",
" \"containsCreditCard\": Condition(Predicate().matches(\".*4[0-9]{12}(?:[0-9]{3})?\"))\n",
"}\n",
"\n",
"schema = DeclarativeSchema(STANDARD_RESOLVER)\n",
"schema.add_resolver_spec(column_name=\"emails\", metrics=[ConditionCountMetricSpec(emails_conditions)])\n",
"schema.add_resolver_spec(column_name=\"transcriptions\", metrics=[ConditionCountMetricSpec(transcriptions_conditions)])\n",
"\n",
"prof_view = why.log(df, schema=schema).profile().view()\n",
"prof_view.to_pandas()[['condition_count/containsEmail', 'condition_count/containsCreditCard', 'condition_count/total']]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3eRAQaGALETB"
},
"source": [
"### Equalities Example"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 208
},
"id": "R8uNVop2LETC",
"outputId": "4a0f5460-ccf4-4a1a-e904-9051af25f42b"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>types/fractional</th>\n",
" <th>types/integral</th>\n",
" <th>condition_count/lessthan5</th>\n",
" <th>condition_count/morethan40</th>\n",
" <th>condition_count/equals42</th>\n",
" <th>condition_count/equals42.2</th>\n",
" <th>condition_count/total</th>\n",
" </tr>\n",
" <tr>\n",
" <th>column</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>floats_column</th>\n",
" <td>4</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" <td>1</td>\n",
" <td>NaN</td>\n",
" <td>1.0</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>ints_column</th>\n",
" <td>0</td>\n",
" <td>4</td>\n",
" <td>2</td>\n",
" <td>1</td>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" types/fractional types/integral condition_count/lessthan5 \\\n",
"column \n",
"floats_column 4 0 2 \n",
"ints_column 0 4 2 \n",
"\n",
" condition_count/morethan40 condition_count/equals42 \\\n",
"column \n",
"floats_column 1 NaN \n",
"ints_column 1 1.0 \n",
"\n",
" condition_count/equals42.2 condition_count/total \n",
"column \n",
"floats_column 1.0 4 \n",
"ints_column NaN 4 "
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"\n",
"import whylogs as why\n",
"from whylogs.core.resolvers import STANDARD_RESOLVER\n",
"from whylogs.core.specialized_resolvers import ConditionCountMetricSpec\n",
"from whylogs.core.datatypes import Fractional, Integral\n",
"from whylogs.core.metrics.condition_count_metric import Condition\n",
"from whylogs.core.relations import Predicate\n",
"from whylogs.core.schema import DeclarativeSchema\n",
"\n",
"data = {\n",
" \"ints_column\": [1,12,42,4],\n",
" \"floats_column\": [1.2, 12.3, 42.2, 4.8]\n",
"\n",
"}\n",
"df = pd.DataFrame(data=data)\n",
"\n",
"ints_conditions = {\n",
" \"equals42\": Condition(Predicate().equals(42)),\n",
" \"lessthan5\": Condition(Predicate().less_than(5)),\n",
" \"morethan40\": Condition(Predicate().greater_than(40)),\n",
" \n",
"}\n",
"\n",
"floats_conditions = {\n",
" \"equals42.2\": Condition(Predicate().equals(42.2)),\n",
" \"lessthan5\": Condition(Predicate().less_than(5)),\n",
" \"morethan40\": Condition(Predicate().greater_than(40)),\n",
"}\n",
"\n",
"schema = DeclarativeSchema(STANDARD_RESOLVER)\n",
"schema.add_resolver_spec(column_type=Integral, metrics=[ConditionCountMetricSpec(ints_conditions)])\n",
"schema.add_resolver_spec(column_type=Fractional, metrics=[ConditionCountMetricSpec(floats_conditions)])\n",
"\n",
"prof_view = why.log(df, schema=schema).profile().view()\n",
"prof_view.to_pandas()[['types/fractional','types/integral','condition_count/lessthan5', 'condition_count/morethan40','condition_count/equals42','condition_count/equals42.2', 'condition_count/total']]\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Hiwgne8NLETC"
},
"source": [
"### Logical Operators Example"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 208
},
"id": "8Xn-DDLoLETC",
"outputId": "cc5e5a15-fa1c-4de4-f16e-26c361e3794f"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>condition_count/between10and50</th>\n",
" <th>condition_count/outside10and50</th>\n",
" <th>condition_count/not_42</th>\n",
" <th>condition_count/total</th>\n",
" </tr>\n",
" <tr>\n",
" <th>column</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>floats_column</th>\n",
" <td>2</td>\n",
" <td>2</td>\n",
" <td>4</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>ints_column</th>\n",
" <td>2</td>\n",
" <td>2</td>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" condition_count/between10and50 condition_count/outside10and50 \\\n",
"column \n",
"floats_column 2 2 \n",
"ints_column 2 2 \n",
"\n",
" condition_count/not_42 condition_count/total \n",
"column \n",
"floats_column 4 4 \n",
"ints_column 3 4 "
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"\n",
"import whylogs as why\n",
"from whylogs.core.resolvers import STANDARD_RESOLVER\n",
"from whylogs.core.specialized_resolvers import ConditionCountMetricSpec\n",
"from whylogs.core.metrics.condition_count_metric import Condition\n",
"from whylogs.core.relations import Predicate\n",
"from whylogs.core.schema import DeclarativeSchema\n",
"from whylogs.core.relations import Not\n",
"\n",
"data = {\n",
" \"ints_column\": [1,12,42,4],\n",
" \"floats_column\": [1.2, 12.3, 42.2, 4.8]\n",
"\n",
"}\n",
"df = pd.DataFrame(data=data)\n",
"\n",
"conditions = {\n",
" \"between10and50\": Condition(Predicate().greater_than(10).and_(Predicate().less_than(50))),\n",
" \"outside10and50\": Condition(Predicate().less_than(10).or_(Predicate().greater_than(50))),\n",
" \"not_42\": Condition(Not(Predicate().equals(42))), # could also use X.not_equal(42) or X.not_.equals(42)\n",
"}\n",
"\n",
"schema = DeclarativeSchema(STANDARD_RESOLVER)\n",
"schema.add_resolver_spec(column_name=\"ints_column\", metrics=[ConditionCountMetricSpec(conditions)])\n",
"schema.add_resolver_spec(column_name=\"floats_column\", metrics=[ConditionCountMetricSpec(conditions)])\n",
"\n",
"prof_view = why.log(df, schema=schema).profile().view()\n",
"prof_view.to_pandas()[['condition_count/between10and50', 'condition_count/outside10and50', 'condition_count/not_42', 'condition_count/total']]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "w67opSs-LETD"
},
"source": [
"### User-defined function - even"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 240
},
"id": "Q-vZQamsLETD",
"outputId": "7d2c2553-8ac2-43c6-c3c8-6de8f7e03950"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>condition_count/isPreprocessed</th>\n",
" <th>condition_count/isEven</th>\n",
" <th>condition_count/total</th>\n",
" </tr>\n",
" <tr>\n",
" <th>column</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>floats_column</th>\n",
" <td>NaN</td>\n",
" <td>0.0</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>ints_column</th>\n",
" <td>NaN</td>\n",
" <td>3.0</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>transcriptions</th>\n",
" <td>1.0</td>\n",
" <td>NaN</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" condition_count/isPreprocessed condition_count/isEven \\\n",
"column \n",
"floats_column NaN 0.0 \n",
"ints_column NaN 3.0 \n",
"transcriptions 1.0 NaN \n",
"\n",
" condition_count/total \n",
"column \n",
"floats_column 4 \n",
"ints_column 4 \n",
"transcriptions 4 "
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"from typing import Any\n",
"\n",
"import whylogs as why\n",
"from whylogs.core.resolvers import STANDARD_RESOLVER\n",
"from whylogs.core.specialized_resolvers import ConditionCountMetricSpec\n",
"from whylogs.core.metrics.condition_count_metric import Condition\n",
"from whylogs.core.relations import Predicate\n",
"from whylogs.core.schema import DeclarativeSchema\n",
"\n",
"def even(x: Any) -> bool:\n",
" return x % 2 == 0\n",
"\n",
"def preprocessed(x: Any) -> bool:\n",
" stopwords = [\"i\", \"me\", \"myself\"]\n",
" if not isinstance(x, str):\n",
" return False\n",
" \n",
" # should have only lowercase letters and space (no digits)\n",
" if not all(c.islower() or c.isspace() for c in x):\n",
" return False\n",
" # should not contain any words in our stopwords list\n",
" if any(c in stopwords for c in x.split()):\n",
" return False\n",
" return True\n",
"\n",
"data = {\n",
" \"transcriptions\": [\"I AM BOB AND I LIKE TO SCREAM\",\"i am bob\",\"am alice and am xx years old\",\"am bob and am 42 years old\"],\n",
" \"ints_column\": [1,12,42,4],\n",
" \"floats_column\": [1.2, 12.3, 42.2, 4.8]\n",
"}\n",
"df = pd.DataFrame(data=data)\n",
"\n",
"transcriptions_conditions = {\n",
" \"isPreprocessed\": Condition(Predicate().is_(preprocessed)),\n",
"}\n",
"numerical_conditions = {\n",
" \"isEven\": Condition(Predicate().is_(even)),\n",
"}\n",
"\n",
"schema = DeclarativeSchema(STANDARD_RESOLVER)\n",
"schema.add_resolver_spec(column_name=\"ints_column\", metrics=[ConditionCountMetricSpec(numerical_conditions)])\n",
"schema.add_resolver_spec(column_name=\"floats_column\", metrics=[ConditionCountMetricSpec(numerical_conditions)])\n",
"schema.add_resolver_spec(column_name=\"transcriptions\", metrics=[ConditionCountMetricSpec(transcriptions_conditions)])\n",
"\n",
"prof_view = why.log(df, schema=schema).profile().view()\n",
"prof_view.to_pandas()[['condition_count/isPreprocessed','condition_count/isEven', 'condition_count/total']]"
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
},
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "5dd5901cadfd4b29c2aaf95ecd29c0c3b10829ad94dcfe59437dbee391154aea"
}
}
},
"nbformat": 4,
"nbformat_minor": 0
}