python/examples/experimental/performance_estimation.ipynb
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
">### 🚩 *Create a free WhyLabs account to get more value out of whylogs!*<br> \n",
">*Did you know you can store, visualize, and monitor whylogs profiles with the [WhyLabs Observability Platform](https://whylabs.ai/whylogs-free-signup?utm_source=whylogs-Github&utm_medium=whylogs-example&utm_campaign=Performance_Estimation)? Sign up for a [free WhyLabs account](https://whylabs.ai/whylogs-free-signup?utm_source=whylogs-Github&utm_medium=whylogs-example&utm_campaign=Performance_Estimation) to leverage the power of whylogs and WhyLabs together!*"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Performance Estimation - Estimating Accuracy for Binary Classification Problems"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"[](https://colab.research.google.com/github/whylabs/whylogs/blob/mainline/python/examples/experimental/performance_estimation.ipynb)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Once your model is deployed, monitoring its performance plays a crucial role in ensuring the quality of your ML system. To calculate metrics such as accuracy, labels are required. However, in many cases, labels can be unavailable, partially available or come in a delayed fashion.\n",
"\n",
"In this notebook, we will show one possible way of estimating the performance of your model without having access to the labels. We will use the [Ecommerce dataset](https://whylogs.readthedocs.io/en/latest/datasets/ecommerce.html) to demonstrate the process.\n",
"\n",
"We will cover:\n",
"\n",
"- [Importance Weighting for Accuracy Estimation - Rationale](#rationale)\n",
"- [The scenario - Covariate Shift with the Ecommerce Dataset](#scenario)\n",
"- [Using whylogs to estimate accuracy](#whylogs)\n",
"- [Uploading the results to WhyLabs](#whylabs)\n",
"- [Conclusion](#conclusion)\n",
"\n",
"First, let's define the scope of this example:\n",
"\n",
"We are concerned with estimating the __accuracy__ of a __binary classification model__ for an __unlabeled target__ dataset. We will do so by leveraging a __labeled reference__, or baseline, dataset."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Importance Weighting for Accuracy Estimation <a class=\"anchor\" id=\"rationale\"></a>\n",
"\n",
"As previously stated, the challenge is to estimate the accuracy of a model without having access to the labels.\n",
"\n",
"One way to do so is to use a labeled reference dataset. This can be your test set, or a subset of it. We can then identify ways to segment both the reference and the target datasets. For example, we might segment a dataset according to age, profession, or location. We can then calculate the reference dataset's accuracy for each segment. To finally estimate the overall accuracy of the target dataset, we can use the reference dataset's accuracy as a proxy, and weight it according to the proportion of the target dataset's segments.\n",
"\n",
"Let's see how this works in practice.\n",
"\n",
"### Reference Dataset\n",
"\n",
"Assume we have a reference dataset for which we have labels. We then segment this dataset into 4 different categories: A, B, C, and D. Since we have the labels, we can then calculate the accuracy of each segment. If the chosen segments are mutually exclusive and exhaustive, we can also calculate the reference dataset's overall accuracy by simply weighting the accuracy of each segment by its proportion in the reference dataset. So, if we have the following accuracies and proportions:\n",
"\n",
"> Mutually exclusive and exhaustive segments means that our segments don't overlap with each other, and the sum of the segments equal to the complete dataset.\n",
"\n",
"__Reference Dataset__\n",
"\n",
"| Segment | Accuracy | Proportion |\n",
"|:-------:|:--------:|:----------:|\n",
"| A | 0.92 | 32% |\n",
"| B | 0.56 | 27% |\n",
"| C | 0.67 | 16% |\n",
"| D | 0.75 | 25% |\n",
"\n",
"The overall accuracy of the reference dataset is:\n",
"\n",
"$Acc_{ref} = p_{A}*Acc_{A} + p_{B}*Acc_{B} + p_{C}*Acc_{C} + p_{D}*Acc_{D} $\n",
"\n",
"$Acc_{ref} = 0.32*0.92 + 0.27*0.56 + 0.16*0.67 + 0.25*0.75 = 0.74 $\n",
"\n",
"### Target Dataset\n",
"\n",
"Great! So, how do we use this information to estimate the accuracy of our target dataset?\n",
"\n",
"\n",
"Suppose we have the following information about our target dataset:\n",
"\n",
"__Target Dataset__\n",
"\n",
"| Segment | Accuracy | Proportion |\n",
"|:-------:|:--------:|:----------:|\n",
"| A | ? | 16% |\n",
"| B | ? | 67% |\n",
"| C | ? | 6% |\n",
"| D | ? | 11% |\n",
"\n",
"We can see that the proportions for each segment are different from the reference dataset, with a significant increase in the proportion of data belonging to segment B. We don't have the accuracy for each segment, and we want to estimate the overall accuracy of the target dataset. The intuition is that we can use the reference dataset's accuracy for each segment as a proxy, and weight those accuracies by the proportion of each segment in the target dataset. This will give us an estimate of the overall accuracy of the target dataset.\n",
"\n",
"So, if we denote $\\overline{Acc}_{target}$ as the estimated overall accuracy of the target dataset, we can calculate it as follows:\n",
"\n",
"$\\overline{Acc}_{target} = p_{A}*Acc_{ref_A} + p_{B}*Acc_{ref_B} + p_{C}*Acc_{ref_C} + p_{D}*Acc_{ref_D} $\n",
"\n",
"$\\overline{Acc}_{target} = 0.16*0.92 + 0.67*0.56 + 0.06*0.67 + 0.11*0.75 = 0.645$\n",
"\n",
"Where $Acc_{ref_A}$, $Acc_{ref_B}$, $Acc_{ref_C}$, and $Acc_{ref_D}$ are the accuracies of the reference dataset for each segment."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Considerations\n",
"\n",
"In the example above, we're considering that the main reason for the difference in performance between the reference and target datasets is due to a change in the distribution of the input data. This is known as __covariate shift__.\n",
"\n",
"That said, there are other reasons why the performance of a model can change, such as (but not limited to):\n",
"\n",
"- __Concept drift__: The relationship between the input data and the target variable changes. This is known as __concept drift__.\n",
"- __Covariate shift to unknown regions of the feature space__: Suppose our model was trained of a demographic with age 15-70. If during production we receive data from a demographic with age 0-14 or above 70, the model's performance will likely decrease.\n",
"- __Data quality issues__: Missing values, outliers, data schema changes, etc. can affect the performance of a model.\n",
"\n",
"For all of above examples, it is possible that the importance weighting approach presented here will not yield accurate estimates.\n",
"\n",
"Another important consideration is the importance of choosing the proper segments to perform the importance weighting. The segments in the target dataset must be a subset of the reference dataset, as an unseen segment will not have an associated accuracy. The segments also should ideally have high variance in training accuracies: if all segments have the same accuracy, then weighting them would not make much sense. Additionally, as stated previously, in this example we are assuming that the segments are mutually exclusive and exhaustive."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## The Scenario <a class=\"anchor\" id=\"scenario\"></a>\n",
"\n",
"Let's see how this approach works in practice with the [Ecommerce dataset](https://whylogs.readthedocs.io/en/latest/datasets/ecommerce.html).\n",
"\n",
"This dataset contrains transactions made by customers of an online store. The goal is to predict whether a product should be given a discount or not, based on the product's category, market price, product rating and sales history. It is a binary classification task, as the `output_discount` column contains a `1` if the product should be given a discount, and a `0` otherwise.\n",
"\n",
"We will segment the dataset according to the `category` column. This column contains 11 different categories such as `Beverages`, `Vegetables` or `Baby Care`.\n",
"\n",
"We want to simulate a scenario where the distribution of the input data changes, leading to changes in the model's performance. To do so, we will get data for 7 different days, and \"perturb\" the data for each day: we will pick 4 different categories for each day, and randomly subsample each category between a range of 10%-30%: that is, we will end up with 10 to 30% of the original segment size for each subsampled category.\n",
"\n",
"For the reference dataset, we will use the dataset that was originally used to test the model in an experimental (pre-deployment) stage.\n",
"\n",
"In this example, we actually have the labels for the perturbed days. We'll end the example with the plot showing the estimated accuracy vs. the real accuracy for each day."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Accuracy Estimation with whylogs <a class=\"anchor\" id=\"whylogs\"></a>\n",
"\n",
"Let's finally see how we can use whylogs to estimate the accuracy of a model.\n",
"\n",
"first, let's install whylogs and the required extras:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Note: you may need to restart the kernel to use updated packages.\n",
"%pip install 'whylogs[datasets]'"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Getting the Reference and Target Datasets\n",
"\n",
"The first thing we need is to get:\n",
"\n",
"- Reference dataset\n",
"- 7 daily \"perturbed\" target datasets\n",
"\n",
"This dataset is already in the whylogs' `datasets` module, so we'll source it from there. We'll then arrange the data into a more proper format, and then create a function to perturb each day by random subsampling a subset of our categories."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"from whylogs.datasets import Ecommerce\n",
"\n",
"dataset = Ecommerce()\n",
"\n",
"baseline = dataset.get_baseline()"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>product</th>\n",
" <th>sales_last_week</th>\n",
" <th>market_price</th>\n",
" <th>rating</th>\n",
" <th>category</th>\n",
" <th>output_discount</th>\n",
" <th>output_prediction</th>\n",
" </tr>\n",
" <tr>\n",
" <th>date</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2023-02-28 00:00:00+00:00</th>\n",
" <td>Wood - Centre Filled Bar Infused With Dark Mou...</td>\n",
" <td>1</td>\n",
" <td>350.0</td>\n",
" <td>4.500000</td>\n",
" <td>Snacks and Branded Foods</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2023-02-28 00:00:00+00:00</th>\n",
" <td>Toasted Almonds</td>\n",
" <td>1</td>\n",
" <td>399.0</td>\n",
" <td>3.944479</td>\n",
" <td>Gourmet and World Food</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2023-02-28 00:00:00+00:00</th>\n",
" <td>Instant Thai Noodles - Hot & Spicy Tomyum</td>\n",
" <td>1</td>\n",
" <td>95.0</td>\n",
" <td>3.300000</td>\n",
" <td>Gourmet and World Food</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2023-02-28 00:00:00+00:00</th>\n",
" <td>Thokku - Vathakozhambu</td>\n",
" <td>1</td>\n",
" <td>336.0</td>\n",
" <td>4.300000</td>\n",
" <td>Snacks and Branded Foods</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2023-02-28 00:00:00+00:00</th>\n",
" <td>Beetroot Powder</td>\n",
" <td>1</td>\n",
" <td>150.0</td>\n",
" <td>3.944479</td>\n",
" <td>Gourmet and World Food</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" product \\\n",
"date \n",
"2023-02-28 00:00:00+00:00 Wood - Centre Filled Bar Infused With Dark Mou... \n",
"2023-02-28 00:00:00+00:00 Toasted Almonds \n",
"2023-02-28 00:00:00+00:00 Instant Thai Noodles - Hot & Spicy Tomyum \n",
"2023-02-28 00:00:00+00:00 Thokku - Vathakozhambu \n",
"2023-02-28 00:00:00+00:00 Beetroot Powder \n",
"\n",
" sales_last_week market_price rating \\\n",
"date \n",
"2023-02-28 00:00:00+00:00 1 350.0 4.500000 \n",
"2023-02-28 00:00:00+00:00 1 399.0 3.944479 \n",
"2023-02-28 00:00:00+00:00 1 95.0 3.300000 \n",
"2023-02-28 00:00:00+00:00 1 336.0 4.300000 \n",
"2023-02-28 00:00:00+00:00 1 150.0 3.944479 \n",
"\n",
" category output_discount \\\n",
"date \n",
"2023-02-28 00:00:00+00:00 Snacks and Branded Foods 0 \n",
"2023-02-28 00:00:00+00:00 Gourmet and World Food 1 \n",
"2023-02-28 00:00:00+00:00 Gourmet and World Food 0 \n",
"2023-02-28 00:00:00+00:00 Snacks and Branded Foods 0 \n",
"2023-02-28 00:00:00+00:00 Gourmet and World Food 0 \n",
"\n",
" output_prediction \n",
"date \n",
"2023-02-28 00:00:00+00:00 1 \n",
"2023-02-28 00:00:00+00:00 0 \n",
"2023-02-28 00:00:00+00:00 0 \n",
"2023-02-28 00:00:00+00:00 1 \n",
"2023-02-28 00:00:00+00:00 0 "
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"pd.options.mode.chained_assignment = None # default='warn’\n",
"\n",
"def arrange_df(batch):\n",
" df = batch.features\n",
" df['output_discount'] = batch.target['output_discount']\n",
" df['output_prediction'] = batch.prediction['output_prediction']\n",
" return df\n",
"\n",
"reference_df = arrange_df(baseline)\n",
"reference_df.head()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Those are the categories that will be randomly subsampled for each day:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"perturbations_by_day = {\n",
" 0 : ['Kitchen, Garden and Pets','Beauty and Hygiene', 'Fruits and Vegetables','Bakery, Cakes and Dairy'],\n",
" 1 : ['Snacks and Branded Foods','Beverages', 'Baby Care', 'Gourmet and World Food'],\n",
" 2 : ['Beauty and Hygiene','Kitchen, Garden and Pets', 'Bakery, Cakes and Dairy','Fruits and Vegetables'],\n",
" 3 : ['Foodgrains, Oil and Masala','Cleaning and Household','Eggs, Meat and Fish','Bakery, Cakes and Dairy'],\n",
" 4 : ['Cleaning and Household','Gourmet and World Food','Kitchen, Garden and Pets','Beauty and Hygiene'],\n",
" 5 : ['Baby Care','Bakery, Cakes and Dairy','Kitchen, Garden and Pets'],\n",
" 6 : ['Beverages', 'Eggs, Meat and Fish', 'Foodgrains, Oil and Masala'],\n",
"\n",
" }"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"And the following function will be used to perturb the data:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"\n",
"def random_subsample_on_column(df, column, lower_pct=0.1 , upper_pct=0.3, classes = 'all'):\n",
" \"\"\"Subsample each class in a column to a random percentage of the total.\n",
"\n",
" The percentage is sampled uniformly between lower_pct and upper_pct.\n",
" If classes is not 'all', then only subsample the classes in classes.\n",
"\n",
" Args:\n",
" df (pd.DataFrame): The dataframe to subsample.\n",
" column (str): The column to subsample on.\n",
" lower_pct (float): The lower bound of the percentage to subsample.\n",
" upper_pct (float): The upper bound of the percentage to subsample.\n",
" classes (list): The classes to subsample. If 'all', then subsample all classes.\n",
"\n",
" \"\"\"\n",
" if classes == 'all':\n",
" class_names = df[column].unique()\n",
" elif isinstance(classes, list):\n",
" assert all([c in df[column].unique() for c in classes]), \"Classes must be in the column\"\n",
" class_names = classes\n",
" for c in class_names:\n",
" sub_df = df.loc[df[column]==c]\n",
" n = int(len(sub_df) * (lower_pct + (upper_pct - lower_pct) * np.random.random()))\n",
" # remove n rows from the class\n",
" df = df.loc[df[column] != c].append(sub_df.sample(n=n))\n",
" return df"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's get our perturbed dfs: "
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"batches = dataset.get_inference_data(number_batches=7)\n",
"\n",
"perturbed_dfs = []\n",
"for day, batch in enumerate(batches):\n",
" unperturbed_df = arrange_df(batch)\n",
" perturbed_df = random_subsample_on_column(unperturbed_df, 'category', lower_pct=0.1, upper_pct=0.3, classes = perturbations_by_day[day])\n",
" perturbed_dfs.append(perturbed_df)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### whylogs Profiling\n",
"\n",
"We'll start by profiling the reference dataset, and then the target datasets. We'll use the `category` column as the segment column, and we'll use the `output_discount` column as the target column.\n",
"\n",
"\n",
"For each profiling process, we'll end up with 11 different whylogs profiles - one for each category. Those 11 profiles will be encapsulated in a `SegmentedResultSet`. Those sets are what we'll use to perform the accuracy estimation.\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"import whylogs as why\n",
"from whylogs.core.segmentation_partition import segment_on_column\n",
"from whylogs.core.schema import DatasetSchema\n",
"\n",
"def log_dataset(df, labeled=True):\n",
" segment_column = \"category\"\n",
" segmented_schema = DatasetSchema(segments=segment_on_column(segment_column))\n",
" \n",
" # Just to be sure that we're not using actual labels/metrics for the target dataset.\n",
" if labeled:\n",
" results = why.log_classification_metrics(\n",
" df,\n",
" target_column = \"output_discount\",\n",
" prediction_column = \"output_prediction\",\n",
" schema=segmented_schema,\n",
" log_full_data=True\n",
" )\n",
" return results\n",
" else:\n",
" results = why.log(df, schema=segmented_schema)\n",
" return results\n",
"\n",
"reference_results = log_dataset(reference_df, labeled=True)\n",
"perturbed_results_list = [log_dataset(perturbed_df, labeled=False) for perturbed_df in perturbed_dfs]"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that for the reference data we're logging by calling `why.log_classification_metrics`, which will give us access to the performance metrics. As for the target data, since we don't have labels, we're logging by calling `why.log`, which will give us access to the counts for each segment."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Estimating accuracy with the `performance_estimation` module\n",
"\n",
"Once we have the Result Sets available for reference and target datasets, we can use the `AcccuracyEstimator` from the `performance_estimation` module to estimate the accuracy of the target dataset.\n",
"\n",
"Let's do it for the first day to demonstrate:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Day 0 accuracy - estimated: 0.6387135902595148\n"
]
}
],
"source": [
"from whylogs.experimental.performance_estimation.estimators import AccuracyEstimator\n",
"\n",
"estimator = AccuracyEstimator(reference_result_set = reference_results)\n",
"\n",
"first_day_result = perturbed_results_list[0]\n",
"estimation_result = estimator.estimate(first_day_result)\n",
"\n",
"print(f\"Day 0 accuracy - estimated: {estimation_result.accuracy}\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"When initializing the estimator, we pass our reference result set. It will leverage the performance metrics for each segment's profiles to perform the accuracy estimation.\n",
"\n",
"When asked for an estimation for the target dataset, it will weight the reference result accuracies by the proportion of each segment in the target dataset, and return the estimated accuracy."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Is this a good estimate? Let's see how it compares to the real accuracy.\n",
"\n",
"### Plotting the estimated accuracy vs. the real accuracy\n",
"\n",
"\n",
"To plot the accuracies for the 7 days, let's define the different types of accuracies in the plot:\n",
"\n",
"- Real accuracy: The real accuracy for the perturbed data. This would only be available if we had all the labels, which is the problem we're trying to solve.\n",
"- Estimated accuracy: The accuracy estimated by the `AccuracyEstimator`.\n",
"- Pre-deployment accuracy: If we didn't have the labels, nor did any kind of estimation, our best guess would be to assume the model's performance is the same as it was during the pre-deployment stage. This is the accuracy we would use if we didn't have any other information.\n",
"\n",
"Let's first get a list for all 3 different types of accuracies for each day. Then we'll plot them in a line chart.\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Day 0 accuracy - real: 0.644542772861357, estimated: 0.6449857773314478, pre-deploy: 0.6741501885271853\n",
"Day 1 accuracy - real: 0.6866051543111676, estimated: 0.6919219880068938, pre-deploy: 0.6741501885271853\n",
"Day 2 accuracy - real: 0.6291828793774319, estimated: 0.6398225939212645, pre-deploy: 0.6741501885271853\n",
"Day 3 accuracy - real: 0.6890386869871044, estimated: 0.684898937625555, pre-deploy: 0.6741501885271853\n",
"Day 4 accuracy - real: 0.6733477789815818, estimated: 0.6626462398863927, pre-deploy: 0.6741501885271853\n",
"Day 5 accuracy - real: 0.665058189043429, estimated: 0.6551432545163998, pre-deploy: 0.6741501885271853\n",
"Day 6 accuracy - real: 0.6886766712141883, estimated: 0.6867237421366811, pre-deploy: 0.6741501885271853\n"
]
}
],
"source": [
"from whylogs.experimental.performance_estimation.estimators import AccuracyEstimator\n",
"import pandas as pd\n",
"\n",
"pd.options.mode.chained_assignment = None # default='warn’\n",
"\n",
"\n",
"def calculate_real_accuracy(df):\n",
" metrics_df = df[['output_discount','output_prediction']]\n",
" correct = 0\n",
" incorrect = 0\n",
" for index,row in metrics_df.iterrows():\n",
" \n",
" if row['output_discount'] == row['output_prediction']:\n",
" correct += 1\n",
" else:\n",
" incorrect += 1\n",
" acc = correct/(correct+incorrect)\n",
" return acc\n",
"\n",
"\n",
"estimator = AccuracyEstimator(reference_result_set = reference_results)\n",
"\n",
"pre_deployment_accs = []\n",
"real_accs = []\n",
"estimated_accs = []\n",
"for day, perturbed in enumerate(zip(perturbed_results_list, perturbed_dfs)):\n",
" perturbed_results = perturbed[0]\n",
" perturbed_df = perturbed[1]\n",
"\n",
" real_acc = calculate_real_accuracy(perturbed_df)\n",
" pre_deployment_acc = calculate_real_accuracy(reference_df)\n",
" estimation_result = estimator.estimate(perturbed_results)\n",
" estimated_acc = estimation_result.accuracy\n",
"\n",
" pre_deployment_accs.append(pre_deployment_acc)\n",
" real_accs.append(real_acc)\n",
" estimated_accs.append(estimated_acc)\n",
" print(f\"Day {day} accuracy - real: {real_acc}, estimated: {estimated_acc}, pre-deploy: {pre_deployment_acc}\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we can plot the accuracies for each day:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAiwAAAGdCAYAAAAxCSikAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy89olMNAAAACXBIWXMAAA9hAAAPYQGoP6dpAACWYklEQVR4nOzdeXzT9f3A8VeS3je9TygIBcpNgVKQQ0QQj4knUxRlyjaEiTKdogM8wXlN3fzJxIHME4fKnCBMUe77FAEpBUpLT0rpfaRN8vvj26QHPZKSNEffz8ejD2LySfJuhfbd7/v9eX9UBoPBgBBCCCGEA1PbOwAhhBBCiLZIwiKEEEIIhycJixBCCCEcniQsQgghhHB4krAIIYQQwuFJwiKEEEIIhycJixBCCCEcniQsQgghhHB4bvYOwFr0ej3Z2dn4+/ujUqnsHY4QQgghzGAwGCgtLSU6Ohq1uuXrKC6TsGRnZxMXF2fvMIQQQgjRDpmZmcTGxrb4uMskLP7+/oDyCQcEBNg5GiGEEEKYo6SkhLi4ONPP8Za4TMJiLAMFBARIwiKEEEI4mbbaOaTpVgghhBAOTxIWIYQQQjg8SViEEEII4fAkYRFCCCGEw5OERQghhBAOTxIWIYQQQjg8SViEEEII4fAkYRFCCCGEw5OERQghhBAOTxIWIYQQQjg8SViEEEII4fAkYRFCCCGEw5OERVjNmaIzfHziY3R6nb1DEUII4WJc5rRmYX9Pb3+aYxeP4e/hz6+u+pW9wxFCCOFC5AqLsIq88jyOXTwGwPas7XaORgghhKuRhEVYxdasrabbe3L2oDfo7RiNEEIIVyMJi7CKLZlbTLcLqwpJvZRqx2iEEEK4GklYxBWrrK1kd85uAOL84wDYlb3LniEJIYRwMZKwiCu2N2cv1bpqonyjuLvP3YAkLEIIIaxLEhZxxTaf3wzA2NixpESlAHAw/yDVumo7RiWEEMKVtCtheeedd4iPj8fLy4vk5GT27t3b6vqioiLmzJlDVFQUnp6eJCQksH79etPjpaWlPProo3Tr1g1vb29GjRrFvn372hOa6GAGg4GtmUrD7fi48VwVdBXh3uFU66o5mHfQztEJIYRwFRYnLKtXr2b+/PksXryYgwcPMmjQICZPnkx+fn6z67VaLddddx3p6emsWbOGkydPsnz5cmJiYkxrHnroIb777js+/PBDjh49yqRJk5g4cSJZWVnt/8xEhzhReIL8yny83bwZHjkclUrFyOiRAOzKkbKQEEII67A4YXnjjTeYNWsWM2fOJDExkWXLluHj48OKFSuaXb9ixQoKCwtZu3Yto0ePJj4+nnHjxjFo0CAAKisr+eKLL3jllVcYO3YsPXv25Nlnn6Vnz568++67V/bZCZvbcl7ZHZQSlYKnxhOAkVFKwrI7e7fd4hJCCOFaLEpYtFotBw4cYOLEifUvoFYzceJEdu1q/rfpr7/+mpSUFObMmUNERAT9+/dnyZIl6HTK+Pba2lp0Oh1eXl6Nnuft7c327S0PIKuurqakpKTRh+h4xu3M4+PGm+5LiVb6WE4UnqCwqtAeYQkhhHAxFiUsBQUF6HQ6IiIiGt0fERFBbm5us885c+YMa9asQafTsX79ehYuXMjrr7/Oiy++CIC/vz8pKSm88MILZGdno9Pp+Oijj9i1axc5OTktxrJ06VICAwNNH3FxcZZ8KsIKLlRcME23HRM7xnR/qHcovbr0ApQhckIIIcSVsvkuIb1eT3h4OO+99x5JSUlMmzaNZ555hmXLlpnWfPjhhxgMBmJiYvD09OTtt9/m7rvvRq1uObwFCxZQXFxs+sjMzLT1pyKa2HpeabYdEDqAUO/QRo8ZdwvJ9mYhhBDWYFHCEhoaikajIS8vr9H9eXl5REZGNvucqKgoEhIS0Gg0pvv69u1Lbm4uWq0WgKuuuootW7ZQVlZGZmYme/fupaamhh49erQYi6enJwEBAY0+RMcy9q+MjR172WPGstCunF0YDIYOjUsIIYTrsShh8fDwICkpiU2bNpnu0+v1bNq0iZSUlGafM3r0aNLS0tDr68+WSU1NJSoqCg8Pj0ZrfX19iYqK4tKlS2zcuJFbbrnFkvBEB6rWVZum2zbsXzFKikjCXe1Obnku6SXpHRuccD3acntHIISwM4tLQvPnz2f58uWsWrWKEydOMHv2bMrLy5k5cyYAM2bMYMGCBab1s2fPprCwkHnz5pGamsq6detYsmQJc+bMMa3ZuHEjGzZs4OzZs3z33Xdcc8019OnTx/SawvHszdlLZW0lET4R9O7S+7LHvd28GRI+BMCU2AjRLmnfw5IY+O+jIFfrhOi03Cx9wrRp07hw4QKLFi0iNzeXwYMHs2HDBlMjbkZGRqPek7i4ODZu3Mhjjz3GwIEDiYmJYd68eTz55JOmNcXFxSxYsIDz588THBzM7bffzksvvYS7u7sVPkVhC8Zy0LjYcahUqmbXpESnsDd3L7uyd5lG9gthsWNfAQY4sBJ8QuDahfaOSIjO5+w2COoKXbrZLQSVwUUaDEpKSggMDKS4uFj6WWzMYDAw6YtJ5Jbn8s617zTbwwJwrOAYv173a/zc/dj26224qS3Oj4WAvyXBxbT6/77xdRj+kP3iEaKzqSqBvw+DqmK47yvoNsqqL2/uz285S0hYLPVSKrnluXhpvBgROaLFdX2C+xDoGUhZTRk/F/zcgREKl1F2oT5ZGfmw8uf6J+DEN/aLSYjOZusrUJYHAdEQk2S3MCRhERYzloNGRo3Ey82rxXUatYbkyGRAtjeLdsqs638K6wuTl8DQ+8Gghy8ehAzpjRLC5vJ/gd11U+ev/wu4edotFElYhMWM023HxY1rc23D7c1CWMyYlHQdCSoV3PgG9L4Baqvgk2nKN1MhhG0YDPDtn0BfCwlTIGGSXcORhEVYpKCygKMFR4Hm5680ZUxYfrrwE2XaMpvGJlxQRl2i27VubILGDW7/J8SOgKoi+Oh2KMm2W3hCuLTj/4GzW0DjCdcvtXc0krAIy2w7vw0DBhJDEgn3CW9zfYxfDF39u6Iz6NiXu68DIhQuQ1sOOUeU211H1t/v4QP3rIaQXlByHj66AyqL7BKiEC5LWw4bn1FuX/0oBHe3azggCYuwkLF/ZXzseLOfI2Uh0S5ZB5RL0f7RynbKhnyC4d4vwC8C8o/B6nuhtto+cQrhira9ofxCENgVRj9q72gASViEBbQ6LTuzdwIwNq7tcpCRnCsk2qVp/0pTXbrB9DXg4Q/p2+Cr30GDidpCiHa6eBp2vq3cvn6JclXTAUjCIsy2L3cflbWVhHuHkxicaPbzhkcNR61Sk16STm5586d6C3GZpv0rzYkaCL/+GNTuyoC5jU/LNFwhrtSGBaDTwlUToM9N9o7GRBIWYTZjOWhM7JgWp9s2J8AjgP6h/QG5yiLMpKuFzL3K7Yb9K83pMQ5urTv9fc+79b8ZCiEsd3IDnNqo/BIw5RXT1U29wf5XLyVhEWYxGAym7czNHXZYt6jF325HRik/dCRhEWbJPwbaMqXcE9Gv7fUD7oBJLym3v1sER1bbNj4hXFFNFWyoOzYn5WEI7WV66A8//IFntj9Ddpn9duVJwiLMklaURnZ5Np4aT5Kjkptf9N0iWBINF05e9pCxj2V3zm6HyNSFgzP2r8SNALXGvOeMmgspc5Xb/3kYTv9gm9iEcFU7/waX0sE/CsY+Ybo79VIqW89v5Zsz32DAfiVXSViEWYzloOSoZLzdvC9foK2Afe9DTQX8/OVlDw8KG4S3mzeXqi9xsvDyhEaIRszpX2nOdS9A/zuU3UWr74Psw1YPTQiXVJQB215Xbk96ETz9TQ99ePxDACZ2nUiMX4w9ogMkYRFmMk23jW1hum3ad0qyAnBux2UPu2vcGR45HJDtzaINBkPjHUKWUKth6v9B97FKSenjO6HwrPVjFMLVbHwaaiuh22jof7vp7oLKAtadWQfAjH4z7BUdIAmLMENhVSFHLigDvFqcbnvsq/rb5/c1OxNDtjcLsxSdg9IcULu176A1N0+Y9jFEDIDyfGUabnmB9eMUwlWkbYIT/wWVBm54tdEYgdUnV1Ojr2Fg2EAGhQ2yY5CSsAgzbM/ajgEDfYL7EOkbefkCbQWkblRuq92Vc16yD122zDhA7mDeQapqq2wZsnBmxqsrUYPbP//BKwCm/1sZelV4Gj65S5ncKYRorFYL39Y12o74baMm96raKlb/ojSwz0i079UVkIRFmGFz5mbAjHJQUFfoPUW5L337Zct6BPYg3DscrV7LwfyDtglWOD9T/4qF5aCmAqKUabjeXZSpuf+eqWyXFkLU2/MuXDwFvmEw/qlGD31z5hsuVV8i2jeaa7tea6cA60nCIlpVo6sxTbdtMWExloMSb4H4q5Xb53ZetkylUjEyWvkhtDt7t9VjFS7C1L9iYcNtc8IS4O7V4OalzJb4Zp4MlhPCqCQbtryi3J74HHgHmR4yGAymZtvpfafjpnazQ4CNScIiWrU/bz/lNeWEeofSL7SZeRgNy0H9blUatgAy9zT726yxLLQ7RxIW0YyKQrjwi3L7Sq+wGHVNhjtWgkoNhz6CH5dY53WFcHbfLVKa02OHw6C7Gz20I3sHZ4rP4Ovuy229brNTgI1JwiJaZdzOPDZ2LGpVM39dTv2vvhwUPRTCE8ErSPlHYDxptwHjALkThScorCq0ZejCGWXuUf4M6QW+odZ73T43wI1vKLe3vgL7V1jvtYVwRuk74Oi/AZXSaKtu/P39X8f+BcBtvW7Dz8PPDgFeThIW0SKDwWDqX2lxd9DxtcqfiVOVznK1GrqNUu5rZntzqHcoCV0SANiTs8eq8QoX0Ez/yvGLx5nyxRTWpq29stceNhPG1TUXrvsj/LLuyl5PCGelq4X1dYPhkh6A6CGNHk69lMqunF2oVWqm953e8fG1QBIW0aIzxWfIKsvCQ+1h2pLcSKNy0NT6+41loWYSFpDtzaIVxv4VY9ILfHziY86Xneel3S9xvvT8lb3++AUwdAYY9LDmN5AhSbPohPb/Uzn+wrsLXLvosoc/Ov4RYP9BcU1JwiJaZCwHDY8ajo97M9tLm5aDjExXWHaBXnfZ04x9LLtydmGQBkhhVFMJWXW7x+qusNTqa01/D6t0Vby458Ur+zujUsGNf4WE65Xt959OgwupVxq5EM6j7AL8UHfu1oSF4BPc6OGCygK+OfMNYP9BcU1JwiJaZDrsMHZ88wualoOMIgcqh9ZVF0PescueNjRiKO5qd3LLc0kvSbdmyMKZZR8CfQ34RUCX7gAcyj9EcXUxfu5+eKg92JG1gw3pG67sfTRucMcKiBkGlZfgo9ugJMcKn4AQTuD7Z5XvzZEDlXJQE440KK4pSVhEs4qqijh84TDQQv9KS+UgUH4gdK07ILGZ7c3ebt4MDVeuyEhZSJg07F+pS4B/yFAOMJzQdQK/HfhbAF7e+zLF1cVX9l4evnDP5xDSE4oz4eM7oOoKX1MIR5e5Dw4r5R5ufP2yg0UdbVBcU5KwiGZty9qG3qAnoUsC0X7Rly9oqRxkZOpjuXyAHGCaxyLnCgmTJvNXDAYDP2b+CMCEuAn8pv9vuCrwKgqrCvnrgb9e+fv5hiiD5fwiIO9n+Gx6s0dKCOES9DpY/7hye/B05ST0JtadWedQg+KakoRFNGvr+a2AOcPipjYuBxmZEpadzQ7qMvax7MvdR42+5krDFc5Or69vgK3rX0m9lEpWWRaeGk9SolNw17izeNRiAL449QX7c/df+ft2iVdG+Hv4Q/o2+Or3SixCuJqD/4Kcw+AZABOfvexhg8HAv44rW5kdZVBcU5KwiMvU6GvYkaXs8BkX10zCoq1QrrDA5eUgo+gh4OYNFRfhwsnLHu4b3JcgzyDKa8r5ueBnK0UunNaFE0pd3d1XObQQ+CFTKQelRKeYmr6HhA/hzoQ7AXhu13Noddorf++oQTDtQ+UcrGNfwv/+fOWvKYQjqSiETc8pt695GvzCL1viiIPimpKERVzmUN4hSmtKCfYKpn9I/8sXtFUOAnDzqL/k2ExZSK1Skxyl9LlIH4sw9a/EDVd6oIAfM+rLQQ09mvQood6hpJek8/7R963z/lddA1PfVW7vfgd2/s06ryuEI/jhRaXBPDwRhs9qdokjDoprShIWcZnN5zcDMCZmDJomTVlA2+Ugo4ZloWbIPBZh0qR/JacshxOFJ1Cr1Jdd5QvwCOCpEcohbe8ffZ8zRWesE8PAO+G6F5Tb//sz/PRv67yuEPaUfbh+svOUV0y/EDR06tIphxwU15QkLOIypv6VZstB5Q3KQbe2/kLxdQlL+o5W+1iOFhylVFva7niFCzAlLEr/irEcNDhsMMFewZctn9RtEmNjx1Kjr+G5Xc+hN1ip72TUHyB5tnJ77Ww4s9k6ryuEPej1dRNtDdD/dug+ptllxkMOHW1QXFOSsIhGzhaf5VzJOdzUboyKHnX5gkbloCGXP95QTBJoPKAsFwov/y042i+abgHd0Bl07MvdZ6XPQDidokxla7FKo8xGoUE5qOuEZp+iUql4JvkZvN28OZh/kK9OfWWdWFQqmLxEScb1NfDZvZDzk3VeW4iO9tNqOL9X6Q0zXj1souGguPsS7+vI6CwmCYtoxDgsbnjEcHzdfS9fcGyt8mdb5SAAd2/TD6CWxvQbD0OUslAnZjzwMGogePpRXF3M/jxlB1DT/pWGov2imTt4LgCvH3idgsoC68SjVsOt/4D4MaAtVWa0XEq3zmsL0VGqipXTmAHGPQGBzV85aTgobnD44I6Lrx0kYRGNGMegX3E5yMg0pr/1PpbdObstilO4ENPAOOXvwtbzW9EZdPQM6klcQFyrT72n7z0khiRSqi3llb2vWC8mN0/49ccQ3g/K8uCj26H8ovVeXwhb2/wylOcrwxFHPtzskqraKj4/+TngmIPimpKERZgUVxdzKP8Q0ML8FUvKQUYN+1iaMTxqOGqVmvSSdHLKZDx6p9Skf8U0LK6FclBDbmo3FqcsRq1S8236t2w7v816cXkFwr1rIDAOLqYp5w5pK6z3+kLYSt5x2PMP5faUvygJeDPWnVlHYVWhww6Ka0oSFmGyI2uH6TfbWP/YyxcYy0H9bm27HGQUO0LpTSjOgKKMyx4O8Aigf6iydVqm3nZClUX1503FjaRaV832LGUbfGvloIYSQxK5r69Se39x94tU1FgxqQiIVqbhegXB+X2wZiboaq33+kJYm8EA3/4JDDrocxP0nNjCsvpBcff0vcchB8U1JQmLMDFuZ2726oq2vP7soMSp5r+op1/91RjZ3iyaOr8PMEBwD/CPYE/OHiprKwn3CScxJNHsl3l48MNE+0aTXZ7Nu0fetW6MYb3hntXg5gWpG2DdY83uehPCIRz7Upna7OalNJC3wBkGxTUlCYsAoFZfa/rNttn+lVP/g9pKy8pBRqayUPPnChm3N+/J2WO97anCOTTpXzEddhg3AZW5V/EAH3cfnhn5DKBs0Txx8YR14+w6Em7/J6jUyojzzS9b9/WFsIbqMthYN6n56vnQpVuLSxsOivP38O+I6K6YJCwCgEP5hyjVlhLkGcTA0IGXL2hPOciojQFyA8MG4uPmw6XqS/xS+Itlry2cW4P+FZ1eZ+pfuabrNRa/1NjYsVwffz06g47ndj2HTq+zZqTQ9ya44TXl9paX4cAH1n19Ia7UttegNBuCusHoR1pc5iyD4pqShEUA9cPimp1u295ykFHXkYAKCk9Dae5lD7ur3RkeORyQ3UKdSm01ZB1QbndN4WjBUQqrCvF392d4xPB2veSTI57E38OfYxeP8ekvn1ox2DrDH4SxTyi3v3kMTn5r/fcQoj0K0mDn35Xb17+sjJVogXFQ3LVdr3XoQXFNScIiANicuRloqxzUzfJyECi7LSKVA+1amsdiLAtJH0snknMEaqvAJwRCeprKQWNix+CucW/XS4Z6hzI/aT4Abx962zY7z655BobcCwY9/HsmZMrQQ2FnxkZbfQ30vA56T2lxacNBcc6wlbkhSVgE50rOkV6Sjpuqhem2xrOD+k21vBxkFH+18mcL25uNjbcH8w5SVVvVvvcQzqVB/4qB+nH87SkHNXRbr9sYGj6UytpKXtrzEgZrN8iqVHDTm9BrkpLIf3IXFJyy7nsIYYmT6+H0JmWy+JS/tPp92pkGxTUlCYswTbdNiky6vPlKWw6pFg6La04bA+S6B3Yn3CccrV7LwfyD7X8f4Twa9K8Yj4RwV7tzdfTVV/SyapWaxSmLcVO7seX8Fr47950Vgm1C4w53fqAcP1FZCB/e1my5Uwibq6mEDcphoKTMhZCrWlzqbIPimmpXwvLOO+8QHx+Pl5cXycnJ7N27t9X1RUVFzJkzh6ioKDw9PUlISGD9+vWmx3U6HQsXLqR79+54e3tz1VVX8cILL1j/NyPRLNNhhy0NizOWg6IGt/9NutYlLBdONDsxVKVS1U+9zZY+Fpen1zc6odl4dSU5KtkqR9v3COrBQwMeAuDlvS9Toi254te8jIcv3PM5BF+lzBn66A6ossH7CNGaHW8pM64CYmDs460udbZBcU1ZnLCsXr2a+fPns3jxYg4ePMigQYOYPHky+fn5za7XarVcd911pKens2bNGk6ePMny5cuJialv9PnLX/7Cu+++y9///ndOnDjBX/7yF1555RX+9re/tf8zE2Yp1ZZyIE9pfBwfO/7yBdYoBwH4hkBYX+V2W30sMkDO9V08pVyZcPOGyIH125nNmG5rrocGPER8QDwXKi/w1oG3rPa6jfiGKoPlfMMh7yisvhdqtbZ5LyGaupQO2/+q3J70opJEt8AZB8U1ZXHC8sYbbzBr1ixmzpxJYmIiy5Ytw8fHhxUrVjS7fsWKFRQWFrJ27VpGjx5NfHw848aNY9CgQaY1O3fu5JZbbuHGG28kPj6eO+64g0mTJrV55UZcuR1ZO6g11NI9sPvl57ZYqxxkFN/69mbjQYi/FP7CxUo5t8WlGftXYoeRry3iaMFRoIWkuZ08NZ4sSlEOf/s89XPTsRNWF9wdpv8bPPzg7BZYO1u5giSErW14Wmlc7z62ze/RzjgorimLEhatVsuBAweYOLF+1K9arWbixIns2tX8b8Vff/01KSkpzJkzh4iICPr378+SJUvQ6epnJIwaNYpNmzaRmpoKwJEjR9i+fTtTprTc6VxdXU1JSUmjD2E542GHzf6gSN1onXKQkWkeS/MD5EK8Q+jdpTegDJETLqxB/4pxh9rAsIGE+YRZ9W2GRw7n1p7KN/Lndj5Hja7Gqq9vEj0Y7voXqN3g5zXw3ULbvI8QRqe+g5PrlL9zU15t8wq4cSuzMw2Ka8qihKWgoACdTkdERESj+yMiIsjNbb7h7MyZM6xZswadTsf69etZuHAhr7/+Oi+++KJpzVNPPcWvf/1r+vTpg7u7O0OGDOHRRx9l+vSWB9osXbqUwMBA00dcXOunuorL1epr2ZalHBbX7Hbm42uVP9szLK45xoQl92flDJlmSFmokzDtEBrZaLqtLfxx2B8J9grmdPFpVh5baZP3AKDntXDLO8rtXX+HXe/Y7r1E51ZbDd8+qdxO/j2E92l1+alLp9iZvdPpBsU1ZfNdQnq9nvDwcN577z2SkpKYNm0azzzzDMuWLTOt+fzzz/n444/55JNPOHjwIKtWreK1115j1apVLb7uggULKC4uNn1kZmba+lNxOT9d+Ini6mICPAIYFDao8YONykFTrfOG/hHKUecY6n/DbqLhuULSdO2iSnKU2rtKTWl4IntylatpV7qduSWBnoH8afifAPjHkX9wruScTd4HgEG/honPKrc3Pg1H19juvUTntesdZRCnbziMe7LN5c46KK4pixKW0NBQNBoNeXl5je7Py8sjMjKy2edERUWRkJCARlM/PbVv377k5uai1SrNaU888YTpKsuAAQO47777eOyxx1i6dGmLsXh6ehIQENDoQ1jGeNjhmNgxlzdgWbscZGTa3tx84+3QiKF4qD3Iq8jjbMlZ672vcByZdclqRD92XPyJWn0t8QHx9AjsYbO3vKH7DYyOHo1Wr+X5Xc/bNhke/SiM+J1y+6vfw5kttnsv0fkUZ8HWV5Xbk14Ar9Z/9jnzoLimLEpYPDw8SEpKYtOmTab79Ho9mzZtIiUlpdnnjB49mrS0NPQNmtBSU1OJiorCw8MDgIqKCtTqxqFoNJpGzxHWtzWzle3M1i4HGXWrm7HRQsLi5ebFkAhlmq5MvXVRDbcz22B3UHNUKhV/HvlnvDRe7M3dy9env7blm8H1SyHxFmXy6GfTIfeo7d5PdC7/+zPUVEDcSBg4rc3ln5/83GkHxTVlcUlo/vz5LF++nFWrVnHixAlmz55NeXk5M2fOBGDGjBksWLDAtH727NkUFhYyb948UlNTWbduHUuWLGHOnDmmNTfffDMvvfQS69atIz09na+++oo33niDW2+1ws4U0azM0kxOF59Go9IwOmZ04wdtUQ4yMl5hyT4M1aXNLpF5LC6urn+lJnaYqYfqmjjblIMaivWPZfbg2QC8uv9VCqsKbfdmag3c+p6SoGtLlRktl2xYihKdw9mtcOxL5dTwG9putK2qrWL1ydUA3Jd4X0dEaFMWJyzTpk3jtddeY9GiRQwePJjDhw+zYcMGUyNuRkYGOTn153fExcWxceNG9u3bx8CBA3nkkUeYN28eTz31lGnN3/72N+644w4efvhh+vbty+OPP87vfvc7XnjhBSt8iqI5xmFxQyOGEuDR5JKisRzUJd665SCAoDgI6goGHWQ2v23d2Hi7L28fNXob7eoQ9lFdarrasM/bm7KaMkK8QhgY1swJ4TZwX+J99O7Sm+LqYl7b95pt38zdC379MYQnQlkufHQ7VNgwSRKuTVcD65VeLIb9BqLa/jdjHBQX5RvFxK4T21zv6No1OWbu3LnMnTu32cc2b9582X0pKSns3t3yb8v+/v68+eabvPnmm+0JR7SD6bDD1spBiVOtWw4y6jZamcx4boeys6KJPsF9CPIMoqi6iKMXjjI0Yqj1YxD2kblXOTQwqCs/XPwJUJpt1aqOOSXEXe3O4pTFTF8/nf+e+S83XXVT8+dnWYt3EExfA/+8ThmW98k0mPEf8PCx3XsK17R3uTIp3DtYOYCzDQaDwdRsO73vdKccFNeUnCXUCZVpy9iftx9oJmGxZTnIqFvrA+TUKjXJUcmAbG92OXX9K/q4kfyY8SNgu+3MLRkQNoB7+t4DwAu7XqCyttK2bxgYo0zD9QqE83vhiwdBV2vb9xSupTQPNtdtQpm4GHyC23zKzuydnC4+7dSD4pqShKUT2pm907QzIz4wvvGDtiwHGRkn3mYdUA7uakbD7c3ChdT1rxwP705+ZT4+bj6m5LQj/WHIH4jwieB82Xn+ceQftn/D8L5w92rQeCon667/I8i2fWGu7xdDdQlED4Uh5u30MY7hv7XnrU47KK4pSVg6IeN027GxYy9/0Hh2kK3KQQBduoN/FOi0cH5/s0uMfSw/F/xMqbb55lzhZHQ1pv/fP6iqALg65mo8NB4dHoqvuy/PJCuX1VcdW0XqpVTbv2m3FLj9fUAFBz6ALa/Y/j2F88vYDUc+VW7f8Bqo2/6x7SqD4pqShKWT0el1bDuv7MwYHze+8YPacmXcM1jn7KCWqFQNykLNb2+O9oumW0A3dAYde3PlTCmXkPOTcvXOK4gfCo4Att/O3Jprul7DxK4TqTXU8tzO59DpdW0/6Uol/krZ3QGweQkc/Jft31M4L70O1tedwDzkPohNMutpDQfFxfrH2iq6DicJSydztOAol6ov4e/uf/me/EbloEHNPd162hggB/WHIUpZyEXUlYPOxQ3hdPEZ3FRujIkdY9eQnhrxFL7uvvxU8BOfp37eMW86YhaM+aNy+7+PKv/uhGjOgZXKrjqvwPoJym0oqCxg3Zl1gPMPimtKEpZOxlgOujrmatzV7o0f7IhykFF83QC5zH1Qq212ibEstDtH5rG4hLqE5cdApWFwWOSwy7fUd7AI3wgeHfooAG8dfIu88rzWn2AtExbC4OnK9v7P72+xNCo6sfKLsKlutMc1fwbfULOe9vnJz9HqtQwMdf5BcU1JwtLJGLczj41r0r9SXdYx5SCj0ATwCVWu6GQfbHbJiMgRaFQazpWcI7ss2/YxCdsx1J8f9UPtJcC+5aCG7up9FwPDBlJeU87SvS0fB2JVKhXc/Bb0nKj8G/j4TihI65j3Fs7hh+ehqggi+itzV8zQaFBcP+cfFNeUJCydSFZZFmlFaWhUGsbENLkUf6oDy0FQ18fSelnI38Of/qH9AbnK4vQunoaKAgrcvTlcfBromOm25lCr1CxOWYybyo1NGZvYlLGp7SdZg8Yd7lwF0UOgshA+ulXZvipE1kE4UHf47w2vgsa8GSquNiiuKUlYOpEtmUo5aHD4YAI9Axs/eGyt8qe1zw5qjbEslN5yH4uxLCR9LE6urhy0NboXBgwkhiQS6dv8gan2kNAlgQf6PwDAkj1LKNOWdcwbe/rBPf9Wds4VZcDHd0BVSce8t3BMej2sfwIwwIC76n+xa4MrDoprShKWTsQ4jv+yYXENy0GJUzsuIOM/xMw9LQ7SMs5j2ZOzB71BDsN0WsZykI8X0PHD4szxu4G/I84/jvyKfP526G8d98Z+YXDfl0qJNPcn+Py+Fvu6RCdw5BPI2g8efnDd82Y/zTgozsfNx2UGxTUlCUsnUV5TbtoePC6uScLS0eUgo/B+Sve7tgxyjzS7ZEDYAHzdfblUfYlfCn/puNiEdWXsokKlYld1AaBsKXY0Xm5eLEpZBMCnv3zKTxd+6rg3D+4B0/8N7r5wZjP8Z47ym7boXCovwXeLldvjnoSAKLOfahwUd1uv21xmUFxTkrB0Eruzd1OjryHOP47uAd0bP2iPchAoA5C6GvtYmh/T7652Z3jEcEDKQk6rLB8KT7PT2wutoZZYv1h6BfWyd1TNGhk1kpt73IwBA8/teq5jD9+MGQp3/QvUbnD0c2W6qehcflwKFQUQ2htGzjb7aa46KK4pSVg6ic3nNwNKOUjVMCmpLoNTdWcHdWQ5yMg4pr+VPpaR0XXzWORcIedkLAcFK78tTug6ofHfQQfz+PDHCfIMIvVSqqknoMP0mgi/qitH7Xwbdr/bse8v7Cf3Z9i3XLk95S9KU7aZPjrxEeB6g+KakoSlE9Ab9PX9K82Wg6qUpr+OLAcZGftYMnYqUx2bYWy8PZR3iKraqo6KTFhLxm5qgC11E/gdZTtzS4K9gnl8mDJd9N3D75JZmtmxAQy+B65VSlNsWAA/f9mx7y86nsGgNNoa9JB4C1xlfsm0oLKAb05/A7jeoLimJGHpBH4u+JnCqkL83P1ICm8y2tlUDpraseUgo8hBSnNZVTHkH292SfeA7kT4RKDVazmY1/zMFuHAMnZxyMuTEkMtXTy7MDhssL0jatOvrvoVyZHJVOmqeHH3ixg6+qDCq+fD8FmAAb76HZzd2rHvLzrW0TXKL21u3jDpJYue2nBQ3KAwO/zS2YEkYekEjNNtR0WPwr3hZUZ7l4NAmS8QV3dabwtlIZVKVb+9WcpCzkVbDjlH+MHHB1Cu8GnUGjsH1TaVSsXClIV4qD3Ymb2TdWfXdXQASlmg76+UQ0I/m66UDITrqS6F//1ZuT32jxAUZ/5TddWNBsU5cqnVGiRh6QSM81cuO+zQ3uUgo/jWD0KE+u3N0njrZM7vx2DQ8YOfH+CY25lb0i2gG78f9HsAXt33KkVVRR0bgFoDty1XGtOrS5QZLUUdXJ4StrflFSjLVb4Pp/zBoqe6+qC4piRhcXG55bmcvHQStUrN1TFXN37QeHaQvcpBRt3q4jq3U6nlNiM5SrkKc/LSSQoqCzoqMnGlMnbzi4c7ORoVXhovUwO1s3ig3wP0DOpJYVUhrx94veMDcPeCuz+BsL5QmgMf3Q4VhR0fh7CNC6mw+/+U21P+ovz/NpPBYOBfx5StzK46KK4pSVhcnPHqyqCwQXTx6lL/QEefHdSa6CFK7baiAApSm10S4h1Cn+A+gDJETjiJjF38WFcOGhU9Cm83bzsHZBl3jTuLU5TtxWvT1rIvd1/HB+HdBe5dA/7RUHASPv011FR2fBzCugwG+PYJ0NdCwvWQMNmip3eGQXFNScLi4ozbmcfGNjnsMHVDfTkocmDHB9aQmwfEKbNWSN/e4jIpCzkZXS2c38cPPkqS4ui7g1oyOHww03pPA+D5Xc9Travu+CACY+HeL5RBi5l74IuHWtxVJ5zEif8qQwI1HnC95YdudoZBcU1JwuLCKmoq2JujTLcdHzu+8YPH1yp/dvSwuJZ0M/axND9ADpShXqA03nb4rg1hubyfOa+v5KSnB2qV+vKk2YnMGzqPMO8w0kvSWf7TcvsEEZEIv/4UNJ7wyzew/vEWS6jCwWkrYOPTyu3R85RJxxboLIPimpKExYXtztmNVq8lxi+Gq4Kuqn+gUTloql1iu0y3Bo23LXwTHhoxFA+1B/kV+ZwtPtuBwYl2ydjN5rpy0NDwoY1Lkk7G38OfBckLAPjnz//kdNFp+wQSPxpuXw6oYP8K2PqafeIQV2b7G1CcCYFxyhZ2C3WWQXFNScLiwhoedthou5sjlYOMYocpl0ZLc6DwTLNLvNy8GBIxBJDtzU4hY5fTl4Mamth1IuNjx1Orr+W5Xc/Z7zDOxFtgyivK7R9fhEMf2ScO0T4XT8OOt5Tbk5eAh49FT+9Mg+Kacv224itgMBiorHHOOrHeoGdzXcPtyMgxVGjrT0P2+Pkr3ICavrdQ4zCfnzueUUPRnN9N9Znt6Py7NbtqWHgye3L2sP38Tm69aloHxyjMZjBQlbmLAyGeAKREjm30d9BZzR/6JHtz93Io/xCfnvg3t/a83T6BDPkN7kWZuO96G8PXj1DtGYK+53X2iUVYxPPbBWh0WnTdx1N91RSw8N/Fx8c/Q6vX0i9kAL0C+3X4vytvd43d5r2oDC7SDFBSUkJgYCDFxcUEBARY5TUrtLUkLtpoldfqaGqv8/h2/zsGnQdlpxaBQclNfajioOfv8FLVcGP1Eo4Z4u0baAOPu61mrtt/+EI3hj/WNH/wl9orC9/uf1M+r9TFgOMPIeuM4lR5PBHyZ54JC0FfFUH52cfsHZLVuHfZjlfkNxh0XpSfmY+h1jrfbyxn4HX3d7lds50Kgyd3a5/hiKGnnWIR5pigPsgKj9eoMWi4Xvsypw0xlr2Aqgbfni+jdiun8vw91JZ2/BXy489PxsfDutc6zP35LSUhF+XmdwKA2vIEU7ICcK36IF6qGs7qIzhmaP4qhr3s0fcFYITqlxbX6Kui0Nf6otJo0XhndFRowkLDVSdN5aCa0v52jsa6ai6NQlcZi0pThWfEN3aMRMWTNb9lq24APqpqVni8Srwqx47xiNZ4omWRm3KY5j91UyxPVgD3gMOo3crR1wRRW9rP2iE6PLnC0gpnLgnN2HAPJy+dYGHyc9zU41em+z2+uB+3k99QM+pRasYvtGOEzdCW4f16D1QGHZVzjmAIbL6Z7JkdT/J9xv/4Tb9Z/G7gwx0cpDCHft0jTCjaQqVazb8mf0rvuhk6ruJk4S/M/N+96Aw6Xh/7NlfHjLFfMNWleH58C5rcI+iDulE1YwP4hdsvHtEstx2v47FlCXq/CKp+twc8LduKbDAYuHv9HZwtOcMjgx9jel/79K/YoiRk7s9v6WFphUqlsvqlr46QV57HyUsnUKHi2m7j6j+H6jI4/T0A7gNuw93RPjePIIgeDFkH8M7eA2HxzS4bEzua7zP+x4H8vfh4PNKREQozbc7dQ6WPmiiPIAZH9HO5M06GRPbnvsT7+ODYB7x2YClXxybj425Z86TVeNQNlvvndagvpePz71/DA+ss/oEobKgoE3b8FQD1pJfw8bd8x9yOrB2cLTmDj5sPv+57p1P+bLpSUhJyQVuzlN1BA8IGEOIdUv+AcXdQcA/H2R3UlGl7c9sD5H4u+JlSbWlHRCUsUX6RH3TK+Phruk5wuWTFaPag2cT4xZBTnsM7h9+xbzB+4XDvl+ATAjlHYMUUyNht35hEvf89A7WVyrlQA+5o10t8eFwpJ3WmQXFNScLigkyHHTYdFmc8OyhxqmMMi2uOMWFp4eRmgCi/KOID4tEZdOzN3dtBgQlz6TJ2scW4nbnHDXaOxnZ83H3480jllN2PTnzEsYvH7BtQyFUw/d/gFQR5R2HFZPjyt1Caa9+4OrvTP8Lx/4BKDTe82q7vvacunWJH9o5ONyiuKUlYXExlbSW7c5TfrBpNFq0ugzSlHGT3s4Na03UkoILC061+ozVNvZUx/Q7nyOn1FGo0+KvcGBox1N7h2NTVMVczpfsU9AY9z+18jlq9nbduxyTBHw7A0BmACn5aDX9Lgh1vQ63WvrF1RrVa+PZPyu3hsyCyfQ3onXVQXFOSsLiYvTl7qdZVE+UbRUKXhPoHGpWDBtgvwLZ4B9X/oz7X8lWWlGilLGRMzoTj+CH/AADjgvrgrna3czS296fhf8Lfw58ThSf4+MTH9g4HfEPhV3+DWZsgZhhoy+C7hfDuKEjbZO/oOpc9y5QDXX1C4Zqn2/USFysvdtpBcU1JwuJiGh522Kh3wBnKQUbdrlb+bOVcoeGRw9GoNJwrOUd2WXYHBSbaYtBW8IO+BIAJPW6yczQdI9Q7lD8m/RGAdw6/Q1ZZlp0jqhOTBA9+B7e8A75hcPEUfHQbfDYdLqXbOzrXV5IDW/6i3L7uOeWXsXb4/OTnaPVaBoYOZFDYIOvF54QkYXEhBoOBrZlKw+34uPH1D1SXOkc5yKjbKOXPVvpY/D38GRCqXCmSspDjSDv1DZnubngYDIxOmGrvcDrMrb1uJSkiicraSl7a/ZLjHM6pVsOQe5Uy0ciHQaVRDk58Jxl+XAo1lfaO0HV9t0i5uhUzDAbd066XqNZV89nJzwC4L/E+l21gN5ckLC7kROEJ8ivz8XbzZnjk8PoHUjc6RznIyNh4e+EElF9scZmxLCTnCjmOH+suXY/UBOLj4WvnaDqOWqVmUcoi3NXubMvaxsZzDjYh2ysQrl8Ks3dA/Bjl+8GWl+HvI+D413Lqs7Wl74CjnwMqpdFW3b4ftevOrKOwqpAo3ygmdpto3RidkCQsLmTLeWV3UEpUCp4az/oHjOWgfrc6fjkIwDcEwpSpt2S0XBYyJix7cvbY7yA60cgPl5QJyxPCkuwcScfrEdiDWQNmAfDynpcpri62c0TNCO8L9/8X7vwAAmKhOAM+vw8+vBUupNo7Otegq61vtE26H2La13huMBhMW5mn952Om7rzzV1pShIWF2LaztxSOShxaofH1G7GslArfSz9Q/vj6+5LUXURJwpPdFBgoiW5pdkcowqVwcC43nY6FNDOHhzwIN0Du3Ox6iJvHnzT3uE0T6VSfnmZuxfGPgEaTzjzI7ybAhufgaoSe0fo3PavgLyfle3lExa1+2V2Ze8irSgNHzcfbut1m/Xic2KSsLiICxUXTHMgxsQ2GBPubOUgo3jjPJaWB8i5q91NpS/pY7G/zSeUWvsgbS2hXUfbORr78NB4sGik8kNqTeoaDuYdtHNErfDwhQl/hjm7IWEK6Gth19/h78Pg8Kegl6uWFiu7AD++qNy+dqFytbid/nX8X0DnHhTXlCQsLmLr+brptqEDCPUOrX/A2cpBRsY+ltyjUFnU4jLj1Nvd2bK92d5+yFCu5E3wjABN5718PSxyGLf3Uq4wPbfrObQ6B59/EtwD7vkMpq+B4KugLA/W/h5WXg/Zh+0dnXPZ9BxUFSu/HCbNbPfLNBwUd0/f9jXsuiJJWFyEsX+l8bC4Ujj1nXLbmcpBAP6RyjdPDJC5p8Vlxj6Wg/kHqayVHQ/2UqItYV/5eQAmRF9t52js77Gkxwj2CuZM8RlW/LzC3uGYp9d18PAumPgsuPsq/+7eGw//fRQqCu0cnBM4fwAOKT0n3PAaqDXtfqmGg+Li/OOsEZ1LkITFBVTrqk0D1Br1r6RuBF218oPfmcpBRqbtzS2XheID4on0jaRGX+PYl99d3Pbz26nFQA9tDd2ummTvcOwu0DOQp0Y8BcB7P73H2eKzdo7ITG6ecPVj8If90P8OwAAHVsLfhsK+90HvnKfX25xeD+uVWTwMurtuYnf7yKC4lknC4gL25uylsraSCJ8IenfpXf+AqRw01bnKQUbxbQ+QU6lUprKQ9LHYzw9132AnVFRB7DA7R+MYro+/ntExo6nR1/D8rucdZzaLOQKi4Y5/wgPrIaI/VF6CdX+E98bBOfl3dplDH0L2IfAMgInPXdFLGQfFDQgd0OkHxTUlCYsLMJaDxsWOqx8s5MzlICNjH0v2IeUspBaYzhWSeSx2odVp2ZarlO0m+MQqzZwClUrFn5P/jLebN/vz9rM2ba29Q7Jc/Gj47RalxOEVqPSUrbxeDlVsqKIQvn9WuT1+AfhHtPulGg6Km5E4o9MPimuqXQnLO++8Q3x8PF5eXiQnJ7N3b+sn5hYVFTFnzhyioqLw9PQkISGB9evXmx6Pj49HpVJd9jFnzpz2hNepGAyG+oQlblz9A85eDgIIioPArmDQwfmW/44lRyUDkHoplYLKgo6KTtTZm7uXCr2WsNpa+jXcoSaI9Y/l4UEPA/Da/te4WNnyIESHpXGDEbPgDwdh6P00PlTxLTlU8ceXoLJQmR01YtYVvZQMimudxQnL6tWrmT9/PosXL+bgwYMMGjSIyZMnk5+f3+x6rVbLddddR3p6OmvWrOHkyZMsX76cmJgY05p9+/aRk5Nj+vjuO+XKwJ133tnOT6vzSL2USm55Ll4aL0ZEjqh/wNnLQUam7c0tj+kP8Q6hT3AfQA5DtIcfMn4A4JqKStTdUuwcjeO5N/Fe+gT3oURbwqv7X7V3OO3nGwq/ehtm/QCxw+sOVVxUd6ji9/aOzj5yjihzVwBueAU07T/sUwbFtc3ihOWNN95g1qxZzJw5k8TERJYtW4aPjw8rVjTfCb9ixQoKCwtZu3Yto0ePJj4+nnHjxjFoUH1tLiwsjMjISNPHN998w1VXXcW4ceOafU1Rb3PmZkApi3i5eSl3NiwHOcPZQa0xY4AcIH0sdqI36PkxQzkBeEJF5RU1G7oqN7Ubz6Y8i1qlZt2ZdezIajn5dgoxQ+E3/4Nb/q/BoYq3d75DFQ0GWP8EGPTQ7zboPrbt57RCBsW1zaKERavVcuDAASZOrL9UpVarmThxIrt2Nf+D4uuvvyYlJYU5c+YQERFB//79WbJkCTpd893mWq2Wjz76iN/85jet1u+qq6spKSlp9NEZGeevtFgOiuhvp8isxNjHkrW/1YPaRkYrPyh3Z+92ruZGJ3e04CgFVYX46vUM944Gv3B7h+SQ+oX2454+yjyNF3a/4Pxb8NVqGDK97lDFOU0OVVwC2gp7R2h7P61Wtn67+8CkF6/45WRQXNssSlgKCgrQ6XRERDRuKoqIiCA3t/kGrDNnzrBmzRp0Oh3r169n4cKFvP7667z4YvP/g9euXUtRUREPPPBAq7EsXbqUwMBA00dcXOfbq15QWcDRgqNAk/krrlIOAmWolV8k6LRwfn+Ly4aGD8VD7UF+Zb7zbCF1AT9m/AjAmIpKPLqOsnM0jm3ukLlE+kaSVZbFu0fetXc41uEVCNcvgdk7ofu4ukMV/wLvuPihilUl8L+Fyu2xT0BgTOvr25B2KU0GxZnB5ruE9Ho94eHhvPfeeyQlJTFt2jSeeeYZli1b1uz6f/7zn0yZMoXo6OhWX3fBggUUFxebPjIzM20RvkPbdn4bBgwkhiQS7lP3m60rlYNASbiMfSytlIW83LwYGqEcMia7hTrOD5lK/4qUg9rm6+7LM8nPAPCvY//iZOFJO0dkReF9YMZ/4M5VdYcqZtYdqjgVLrjQ52m05S9Qnq9cxU658s0hMijOPBYlLKGhoWg0GvLy8hrdn5eXR2RkZLPPiYqKIiEhAY2mfupf3759yc3NRatt3F1+7tw5vv/+ex566KE2Y/H09CQgIKDRR2dj3B00PnZ8/Z0nN7hOOcjI1MfS8gA5qJ96K30sHeNs8VnOFp/FzWDg6opK6CoNt20ZHzee67pdh86g49mdz6JzpUFsKpVyVXfuPhj7p7pDFTcrTbmudKhi/gnYXXeFbMoryrC9K3Cx8iL/Pf1fAO5LvO9Ko3NpFiUsHh4eJCUlsWnTJtN9er2eTZs2kZLS/Der0aNHk5aWhr7BQVqpqalERUXh4eHRaO3KlSsJDw/nxhtvtCSsTkmr07IzW7niMDauQTno+FrlT2c7O6g13eoGyGXua3ULpbHxdl/uPmr0NR0RWaf2Y6ZSDhpRWYW/dwiEXGXniJzDUyOews/dj58v/myaueFSPHxgwjMwZw/0vqH+UMW/JTn/oYoGA3z7J2XUQp+boNeVbz1uOChucNjgK4/RhVlcEpo/fz7Lly9n1apVnDhxgtmzZ1NeXs7MmcpBTzNmzGDBggWm9bNnz6awsJB58+aRmprKunXrWLJkyWUzVvR6PStXruT+++/HzU22c7VlX+4+KmsrCfcOJzE4UbmzUTloqt1is7qw3uATArWVyhC5FvQO7k2wVzAVtRX8dOGnDgywczJuZzaVg1wlQbaxcJ9wHkt6DIC3D75NbrmLDmAL7g53fwrTv4CQnkoJZe3vYcVk5z1U8fhaOLsV3Lxg8ktX/HIyKM4yFics06ZN47XXXmPRokUMHjyYw4cPs2HDBlMjbkZGBjk5Oab1cXFxbNy4kX379jFw4EAeeeQR5s2bx1NPPdXodb///nsyMjL4zW9+c4WfUudgLAeNiR1T/5fcWA4K6ek65SBQfhCaykItbwlVq9QkRypD5KQsZFsFlQWmpHC8lIMsdkfCHQwOG0xFbQVL9iyxdzi21WsizN6ljKx391WGQL43Hv47D8qdaJBedZlS2gLlvKUu8Vf8kuvPrJdBcRZoV9Pt3LlzOXfuHNXV1ezZs4fk5GTTY5s3b+aDDz5otD4lJYXdu3dTVVXF6dOnefrppxv1tABMmjQJg8FAQkJCe0LqVAwGA1sy6/pXGh52aCwHJU51vd92jWWhVhIWaNDHIo23NrU5czMGDPTX6ojQ6SRhsZBapWZRyiLcVG78mPkjm85tavtJzszNA65+VDlUccCdKIcqfqAcqrh3uXMcqrjtdSjJgqCuMHreFb+cwWAwbWW+p889MijODHKWkBNKK0ojuzwbT42naSw9VSWuWQ4yMl5hydgNutoWlxkTlp8LfqZE6yJNfg7IVA4qKwU3b4gaaOeInE+vLr2Y2V8ppS/Zs4RSbamdI+oAAdFw+/sw81uIGABVRbD+cfjHuDaHQ9pVQRrs/Jty+/qXwd37il+y0aC4BBkUZw5JWJyQsRyUHJWMt1vdPxzjsDhXKwcZRfRTZj5oyyC35f6USN9I4gPi0Rv07MvZ14EBdh7lNeWmIxAmVFQopzNfwUjyzuy3A39LV/+u5Ffm8/bBt+0dTsfpNgp+u7nuUMUgyDsKK6fAFw9BSU5bz+5YBgNseBL0NdBzotJIbAUNB8UFeHS+Xa7tIQmLEzKWg8bFNphu68rlIAC1pr7sIGUhu9qRtYMafQ1dVV70qKmVctAV8HLzYlHKIgBWn1zN4fzD9g2oIzU8VDHpAUAFR/8Nfx8G2990nEMVT36rnJWkdofr/2KV768yKK59JGFxMoVVhRy5cARoMN22UTnIBYbFtaRb2wPkQM4VsjXTsLjKKlQgA+OuUHJUMr+66lcYMPDcruc635Z83xC4+S347Y8QO0K5ivr9Yng3BU7Z+VDFmkrYULdBZNRcCO1plZc1DoqbEDdBBsVZQBIWJ7M9azsGDPQJ7kOkb92wvkbloH72DdCWGiYsrcxyGB45HI1KQ0ZpBlllWR0UXOdQo69ha6ZyftWEwlxQqZWTe8UVeXzY43Tx7EJaURqrjq2ydzj2ET0EfrMRpi4D33C4mAYf3w6f3gOFdjpuY8fbUHQO/KNhzONWecmGg+Jm9JthldfsLCRhcTLG05kblYOMZwe5ajnIKGoQePgpjXr5x1tc5ufhx8AwpQlUrrJY1/7c/ZTWlBLs5svAaq3SL+Ul9fcr1cWrC08MfwKAZUeWkVGSYeeI7ESthsF3K7uJUuaC2g1OrlMOVfzhpY49VPHSOdj+hnJ78ovg6WeVl5VBce0nCYsTqdHVmKbbmhKWqhKlvgquXQ4CpeYdV7crqq0+FikL2YRxuu14ty5oQPpXrOimHjeRHJVMta6a53c/37lPHfcKVAaz/X6Hcqiirhq2vqIcqnhsbcccqrjxaeUwx/gx0M86u3hkUNyVkYTFiezP2095TTkhXiH0C60r/XSWcpCRGQPkoL7xdk/uHtc6r8WODAZD/Xbm4kvKndK/YjUqlYpFIxfhqfFkT84evjnzjb1Dsj/joYp3/QsC45RDFf99P/zrFsj/xXbvm/Y9/PINqDRww6tWu3JtHBQX6Rspg+LaQRIWJ2Lczjw2dixqVd3/OmM5yJXODmpNvHGA3M5Wf8vqH9ofX3dfiquL+aXQht/YOpHjhcfJq8jD282b5Ny6E3glYbGqrgFd+f2g3wPw6r5XuVR1yc4ROQCVChJvgTl7YdyTyqGKZ7fAstGw4WmoKrbu+9VWw7dPKreTfw/hfa3ysg0HxU3vM10GxbWDJCxOwmAw1PevxDVTDkqcao+wOl70EOUcj/ILUJDa4jI3tRvDI5VmUNnebB0/ZijloNGBvfDS6yGomzIITFjV/f3up2dQTy5VX+K1/a/ZOxzH4eED1zwNc/cqBw/qa2H3O/C3YXD4E+sdqrj7/5SGX99wGP+kdV4TGRRnDZKwOIkzxWfIKsvCQ+1h6s8gdUPnKgeBcpS7cVeK9LF0KNN2ZkPdsELpX7EJd7U7z456FhUqvj79NXty9tg7JMfSJR5+/THc2/BQxdmwYlKrh6OapTgLtryq3L7ueaWXxkpkUNyVk4TFSRjLQcOjhuPj7qPceWyt8mdnKQcZGbc3p5vXx3Io/xCVtZW2jsqlZZZmcurSKTQqDWMv1m0Vl3KQzQwKG8S03tMAeH7X81TVVtk5IgfUs+5QxeueV3YPnt8H711zZYcqfrcQasqV5v6B06wWqnFQnAqVDIq7ApKwOAnTYYex45U7OmM5yCi+wTyWVvpY4gPiifSNpEZfw4G8Ax0UnGsyloOSwocQeL7ut1i5wmJT84bOI9w7nIzSDN776T17h+OY3DyUgwjn7ocBd3HZoYqtnDt2mbPb4OcvAJXSaKu23o9H46C4a7teK4PiroAkLE6gqKqIwxcOAw2m25rKQb06TznIKGaYMia7NBsutTxQSqVSSVnISkzloMDeUFsJ3l0gVE5WtyU/Dz8WJC8AYOXPKzl16ZSdI3JgAVFw+3KYuaHxoYrvmXmooq4Gvv2TcnvYb5SZT1Yig+KsRxIWJ7Ataxt6g56ELglE+9U1OZrKQVM7VzkIlOa7mCTldltj+uvKQsbD+oTlLlVd4lC+clXlGm1dY2PcSKv+Biqad23Xa7km7hpqDbU8t+s59AYrNZa6qm4p8LstDQ5V/LnBoYrZLT9v3/vKMErvYJjwZ6uGJIPirEe+4ziBreeVUejNDovrbOUgo3jz+liSo5RBc6mXUimoLLB1VC5py/kt6A16+gT3ITrnmHKn9K90CJVKxdPJT+Pj5sORC0f498l/2zskx6fWNDhUcSamQxX/Ngy2/1XZttxQWT78uES5fe0i8Am2WigNB8Xdl3ifDIq7QpKwOLgafQ07spQfyqbtzJ25HGRkGiC3vdVlwV7B9A1W5ijIVZb2MQ2Li7sGMupKa9K/0mEifSN5ZOgjALx58E3yK/LtHJGT8A2Bm9+E325WDlWsKYfvn4X/a3Ko4vfPQnWJMjJhqHVLNjIozrokYXFwh/IOKWe3eAXTP6S/cqdpWNzUzlcOMopLVqZQFmVAUWarS0dGK1cDpI/FcpW1laav24SAnlBxURncFT3YvoF1Mr/u/WsGhA6grKaMl/e+bO9wnEv04MaHKhaerjtU8W44ugYOf6ysu+E15eqMlTQdFOeudrfaa3dWkrA4uM3nNwMwJmYMGrWmrhy0SXnQ1c8Oao2nf31jXFt9LHWNt7uzd3fu81naYWf2Tqp0VcT4xZBwKUe5MyZJmYcjOoxGrWFxymI0Kg3fnfvONERSmMl0qOKBBocqrocvHlQeH3IvxA6z6lvKoDjrk4TFgRkMBtN25mbLQeGJdozOAZi2N7fexzI0YiieGk/yK/M5U3ymAwJzHcbtzNfEXYMqs26AmfSv2EXv4N6mXSYv7XmJ8ppyO0fkhLwClEMVZ++EHtfU3RcI1z5r9bf61wnl6sqtvW6VQXFWIgmLA0svSSejNAM3tRujout6NqQcVK+beQmLp8aToeFDASkLWaJWX2saWDih6wTpX3EAswfNJsYvhtzyXP5+6O/2Dsd5hfWG+75SSkWzfgS/MKu+fNqlNHZkKYPipvedbtXX7swkYXFgxqsrwyOG4+vu23h3UGcuBxl1HQmolHM/SvNaXWrc3iznCpnvUP4hiqqLCPQMZIh3DBSeAVQQN9zeoXVa3m7eLBy5EIBPfvmEnwt+tnNETkylUr6HhFxl9ZeWQXG2IQmLAzP+dmsqB538FnRaKQcZeXeBiLpG5LbOFapLWPbl7qNGV2PryFzCj5lKOWhc7DjcsvYpd4YnKl93YTejY0ZzQ/cb0Bv0PLfrOWr1FkxzFTYng+JsRxIWB1VcXWwa1mWav3J8rfJnZzs7qDUNx/S3IqFLAsFewVTWVnLkwpEOCMy5GQyGBtuZJ0BG3ZZw6V9xCH8a/icCPAL4pfAXPjr+kb3DEQ18nqoMiusf0l8GxVmZJCwOakfWDnQGHT2DehLrH9ukHDTVrrE5FDP7WNQqtWmInJSF2pZ6KZWssiw8NZ7K1SnpX3EoId4hPD7scQD+78j/cb70vJ0jElA3KO4XZVDcjH4zZFCclUnC4qCM25lNV1eM5aDQBCkHNWQcIJd/vM0TWhtubxatM5aDUqJS8NHrIecn5QG5wuIwpvacyvDI4VTWVvLinhdly74DkEFxtiUJiwOq1deyPUuZ4GrqXzGWgxKnSjmoId9QCOuj3M5o/cqJsY/l54s/U1xdbOvInJqpHNR1AmTtB4MOAmIhSBoIHYVKpWLhyIW4q93ZkbWDDekb7B1SpyaD4mxPEhYHdCj/EKXaUoI8gxgYOhCqiqUc1BrTmP7Wy0KRvpF0D+yO3qBnX+6+DgjMOeWU5XCi8ARqlVpJmKV/xWF1D+zOrIGzAHh578uSiNvRrhxlUJy3m7cMirMRSVgckPGwQ9N025MbpBzUGjP7WKC+LCTzWFpmLAcNDhtMsFdwg/4VSVgc0YP9H6RHYA8Kqwr564G/2jucTst4deW2XrfJoDgbkYTFARnHbks5yEzGhCX3qHI1qhUyj6VtP2Q2KAfpaiGz7mqUNNw6JA+NB4tTFgPwxakv2J+7384RdT4yKK5jSMLiYM6VnCO9JB03Vd1020blIBkW16yAKAjuAQY9ZOxpdenwyOFoVBoySzNlZ0UziquLTT/wrom7BvKOKqfcegbK1T0HNjRiKHck3AHAc7ueQ6vT2jmizkUGxXUMSVgcjHG6bVJkEv4e/k3KQX3tHJ0DM5WFtre6zNfdl4FhAwG5ytKcree3mrbTdw3o2qB/JVk5QE44rEeHPkqIVwjpJem8uPtFSrWl9g6pUyisKjQNirsv8T47R+Pa5DuQgzFNtzVuZzaeHSTloNYZE5Z06WO5Esb+lWvi6g6Gk/4VpxHoGciC5AUAfJX2FVO+nMKqY6uo1lXbOTLXtvrkatOguCHhQ+wdjkuThMWBlGhLOJh3EIDxseOVctDpTcqDUg5qnXHibc5hqC5rdamxj2VPzh50ep2NA3Me1bpq03b6a7teCwZDgyss0r/iDCbHT+bta96mR2APiquLeW3/a9z01U18deor+btuAzIormNJwuJAdmbtpNZQS/fA7sQFxEk5yBJBXSEwDvS1cH5vq0v7h/bHz92PEm0JJwpPdFCAjm9Pzh4qaysJ9wknMSQRLp2FsjzQeED0UHuHJ8x0Tddr+OJXX/D8qOeJ8IkgtzyXRTsXcfvXt/NDxg8yYM6KZFBcx5KExYEYy0HjY8crdxjLQXJ2kHm6mXeukJvajeGRyonDu3Nk6q2RcVjcNXHXKL8pGq+uRA8Bdy87RiYs5aZ249Zet/LNrd/wx6Q/EuARwOni08z7cR4zvp3BgbwD9g7R6TUcFHdPn3tkUFwHkITFQdTqa9mWtQ2AsbFjG5eDEqfaLzBnYhwgZ04fS7T0sTSk0+tM/SsTuk5Q7pT+Fafn5ebFA/0f4Nvbv+WhAQ/hpfHi8IXDPLDhAeZsmsPJwpP2DtFpNRwUd3vC7fYOp1OQhMVB/HThJ4qriwnwCGBw+OAmZwdJOcgs8Vcrf2bth5qqVpcaG28P5R+isrbS1pE5vKMFRymsKsTf3Z/hEcrVJ+lfcR0BHgHMGzqPdbet486EO9GoNGw9v5U7/3snC7YtkC3+7SCD4jqeJCwOwnjY4ZjYMbip3eDYWuUBKQeZL7gH+EUoiV5W68OzugV0I8o3ihp9jVwep35Y3NWxV+OucYfyAihIVR6MS7ZjZMKawn3CWZSyiLW3rGVy/GQMGPjmzDfcvPZmXt77MhcrWz9AVChOF52WQXF2IAmLg9iaqYzjHxc7TspB7aVSmd3HolKppCxUx2AwND7sECCzbgBfWB/wCbZTZMJW4gPjeW3ca3x242eMjBpJrb6Wj098zA1f3sC7h9+lvKbc3iE6tA+Pfwgo/15kUFzHkYTFAWSWZnK6+DQalYbRMaMblIN6SznIUsbtzemtD5CDBvNYOvkAubPFZzlXcg53tTtXR9eV1aR/pVPoF9qP5ZOW895175EYkkhFbQX/d+T/uOHLG/j4xMcyMbcZDQfFzUicYedoOhdJWByA8bDDoRFDlVqoqRw0VcpBljJeYcncC7Wtf7NNjkpGhYpTl05RUFnQAcE5JmM5KDkqGT8PP+VO6V/pVFKiU/j0xk95bdxrdAvoRmFVIS/vfZlfrf0V/z39X/QGvb1DdBgyKM5+2pWwvPPOO8THx+Pl5UVycjJ797Y+96KoqIg5c+YQFRWFp6cnCQkJrF+/vtGarKws7r33XkJCQvD29mbAgAHs3985DvEyHXYo5aArF9YHfEKgtlIZIteKLl5d6BPcB+jcZaEfM5pMt9VWQPZh5bZcYek01Co1k+Mn89UtX7Fw5ELCvMPIKsvi6e1Pc+d/72Tr+a2dfoaLDIqzL4sTltWrVzN//nwWL17MwYMHGTRoEJMnTyY/P7/Z9Vqtluuuu4709HTWrFnDyZMnWb58OTExMaY1ly5dYvTo0bi7u/Ptt99y/PhxXn/9dbp06dL+z8xJlGnL2J+nJGbjYsdJOehKqVT1VwXMKQvV9bF01nks+RX5/FTwE9AgYck+CPoa8I+CoG52jE7Yg7vanbt638U3t37DvKHz8Hf3J/VSKnM2zWHmxpkcuXDE3iHajXFQXIRPhAyKswOLE5Y33niDWbNmMXPmTBITE1m2bBk+Pj6sWLGi2fUrVqygsLCQtWvXMnr0aOLj4xk3bhyDBg0yrfnLX/5CXFwcK1euZMSIEXTv3p1JkyZx1VVXtf8zcxI7s3dSq68lPiCe+MB4KQdZg3F7cxuNt9B4Hktn/O3ReHVvYNhAwnzClDsb9q/I38FOy8fdh4cGPMT629bzQL8H8FB7cCDvAPeuv5d5P8zjTNEZe4fYoRoOipved7oMirMDixIWrVbLgQMHmDixPrNUq9VMnDiRXbuav6T+9ddfk5KSwpw5c4iIiKB///4sWbIEnU7XaM2wYcO48847CQ8PZ8iQISxfvrzVWKqrqykpKWn04YyM020vGxYnZwe1n3GAXMZu0NW2unRI+BA8NZ5cqLzA6aLTHRCcYzH2r5iuroD0r4hGgryC+OOwP7LutnXc2vNW1Co1P2T+wK1f38rCHQvJLc+1d4gdQgbF2Z9FCUtBQQE6nY6IiIhG90dERJCb2/xf2jNnzrBmzRp0Oh3r169n4cKFvP7667z44ouN1rz77rv06tWLjRs3Mnv2bB555BFWrVrVYixLly4lMDDQ9BEX53xby3R6HdvOK9Ntx8eNl3KQtUT0B89A0JZC3tFWl3pqPEmKSAI6326hMm0Ze3KU7cum7cx6ndKwDNK/IhqJ9I3k+dHP89WvvmJC3AT0Bj1r09Zy45c38tq+1yiqKrJ3iDYlg+Lsz+a7hPR6PeHh4bz33nskJSUxbdo0nnnmGZYtW9ZozdChQ1myZAlDhgzht7/9LbNmzWq0pqkFCxZQXFxs+sjMzLT1p2J1RwuOcqn6Ev7u/sp0W9PZQVPtGZbzU2vqf9iaM6Y/qnPOY9metd1UjuwR2EO5M/84VJeAhz+E97NvgMIh9QjqwVsT3uKjGz5iWMQwtHotq46vYsqXU1j+03IqairsHaLVNRoU10cGxdmLRQlLaGgoGo2GvLy8Rvfn5eURGRnZ7HOioqJISEhAo9GY7uvbty+5ublotVrTmsTExEbP69u3LxkZGS3G4unpSUBAQKMPZ2MsB10dczXu2go4rVyel3KQFRjnsZwz/1yh/Xn7qdHV2DIqh2IqB3VtphwUNxw0bnaISjiLQWGDWDF5Bf937f/Ru0tvymrKePvQ29z41Y18fvJzavSu82+p0aC4AOe7mu8qLEpYPDw8SEpKYtOmTab79Ho9mzZtIiWl+Xr36NGjSUtLQ6+v38efmppKVFQUHh4epjUnTzY+hCs1NZVu3Vx7h4Kx4XFs3FgpB1lbtwaNt/rWZ0j06tKLYK9gKmsrOXzhsO1jcwA1uhpTOXJC3IT6B0wNt9K/ItqmUqkYEzuGz2/+nKVjlhLjF0NBZQEv7H6BW/9zKxvSNzj9DBcZFOc4LC4JzZ8/n+XLl7Nq1SpOnDjB7NmzKS8vZ+bMmQDMmDGDBQsWmNbPnj2bwsJC5s2bR2pqKuvWrWPJkiXMmTPHtOaxxx5j9+7dLFmyhLS0ND755BPee++9RmtcTVZZFmlFaahVasbEjGlQDpKrK1YRNRDcfaGqSClztEKtUjMySikhdZay0L7cfZTVlBHiFcLAsIHKnQYDnJMJt8JyapWam3rcxH+n/pcFIxYQ7BXMuZJzPLHlCe5edzc7s9veseeoZFCc47A4YZk2bRqvvfYaixYtYvDgwRw+fJgNGzaYGnEzMjLIyckxrY+Li2Pjxo3s27ePgQMH8sgjjzBv3jyeeuop05rhw4fz1Vdf8emnn9K/f39eeOEF3nzzTaZPd91a4ZZMpRw0OGwwgXpDg3LQVPsF5Uo07tC17tA+C7Y3d5Z5LMZy0Pi48ahVdd8GijOhNBvUbhCTZMfohLNy17hzT997WH/beh4e/DA+bj4cv3ic3333Ox7630McKzhm7xAt0nBQ3H2J98mgODtrV5F67ty5zJ07t9nHNm/efNl9KSkp7N7d+g+Cm266iZtuuqk94Tgl4zh+2R1kQ91GKYngue2Q/NtWlxqvsBy7eIzi6mICPQM7IkK70Bv0/JipTLc17Q6C+v6VqEHg4WuHyISr8HX3Zfag2UzrPY3lPy3ns5OfsSdnD79e92smdZvEH4b8QZk75eAaDoq7Lv46e4fT6clZQnZQXlPO3lxl6+i4uHFwfK3ygJSDrKthH0sbQ+EifSPpHtgdvUFv+n/jqo5fPE5+RT4+bj4kRyXXPyD9K8LKgr2CeXLEk3xz6zfc3ONmVKj437n/MfU/U3l+1/PkVzQ/Id0RyKA4xyMJix3szt5Njb6GOP84urt3kXKQrcQMBTcvKL8ABafaXN5Ztjf/kKH8fbs65mo8NZ71D5gGxkn/irCuGL8YloxZwr9v/jfjYsehM+j4d+q/ufHLG3nzwJuUaB1v8KcMinM8krDYwebzmwHl7CBV6galHBTWR8pB1ubmCbHDldsWbG929YTFWA5qtJ258lJ9c3KcJCzCNnoH9+bv1/6dD67/gEFhg6jSVfHPn//JlC+msPLnlVTVVtk7RBPjVuZbe94qg+IchCQsHUxv0Jv6VxqVg+RkZtswjuk3I2EZHjkcN5Ub58vOk1nqfIMIzZFRkkFaURpuKjdld5qRcbptSE/wC7NPcKLTSIpI4sMpH/L2NW9zVeBVlGhLeOPAG9z01U18eepLavWtH6lha6eLTrM9azsqVNzb9167xiLqScLSwX4u+JnCqkL83P1I8r8K0oxnB021a1wuq1vdALn0HW32sfi6+5q2+LrqbiHj1ZVhkcMaNxZnyHZm0bFUKhXXdL2GL371BS+MfoFI30jyKvJYvHMxt319G5vObbLbgaQyKM4xScLSwYzTbUdFj8I97XvQ10g5yJZih4PaXdmueym9zeUjo117Houxf6XRYYcgBx4Ku9GoNUztOZVvbv2Gx4c9TqBnIGeLz/Lo5ke599t72Ze7r0PjkUFxjksSlg5mnL8yPm68lIM6goeP0nwL5vWx1DXe7snZg06va2O1c7lYeZFD+YeAJtuZa6og64ByWxIWYSeeGk/u73c/3972Lb8d+Fu83bz56cJP/Gbjb/j997/nl8JfOiSOz09+jlavpV9IPxkU52AkYelAOWU5nLx0ErVKzdUhA6Qc1FGMZSEzBsj1D+2Pv7s/JdoSThSesHFgHWvL+S0YMJAYkkikb4Ozv3IOK43fvmEQ3MNu8QkB4O/hzx+G/IH1t61nWu9puKnc2JG1gzv/eydPbn3Spv1l1bpqPv3lU0C5uiKD4hyLJCwdyNhsOyhsEF3Sd0k5qKOY+li2t7nUTe3G8EhlZ5GrlYV+zKjbHXRZOahB/4p8gxYOItQ7lD+P/DP/mfofpsRPAWD92fX8au2vWLJnCQWVBVZ/TxkU59gkYelAxu3MY2PHytlBHalrMqjUUHQOis+3udy0vTnHdRKWipoK03kujcpBIP0rwqF1DejKK+NeYfVNqxkVPYpafS2f/vIpN3x5A+8cfocybZlV3sdgMPDhCaXZVgbFOSZJWDpIRU0Fe3OUraPjw5Lqh8VJ/4rtefor4+bBonOFDuUfoqKmwpaRdZid2TvR6rXE+sXSK6hX/QN6vQyME04hMSSRf1z3D96f9D79Q/pTWVvJsiPLuOHLG/jw+Idoddorev3dObs5demUDIpzYJKwdJDdObvR6rXE+MVwVc7xBuWgPvYOrXOwoCzU1b8r0b7R1OprOZB3wMaBdYyGZwc1qssXnFROtHb3gciB9glOCAskRyXzyY2f8Mb4N4gPiOdS9SVe2fcKN391M1+f/rrdzfLGMfwyKM5xScLSQUzD4mLHoTr+H+VOKQd1nPgG5wq1QaVSuVRZqFZfy+bMzUAr/Suxw5QTroVwAiqViuu6XcdXt3zF4pTFhHuHk12ezTPbn+GO/97BlswtFs1wkUFxzkESlg6gN+hN81fGhQ+TcpA9dB0JqODiKSjNa3O5K81jOZh3kBJtCV08uzA4fHDjB6V/RTgxN7UbdyTcwTe3fcOjQx/F38OftKI05v4wlwc2PMDh/MNmvY4MinMOkrB0gBMXT1BQWYCPmw/DLuXUlYP6SjmoI3l3gYj+yu2Mtq+yjIwciQoVaUVpXKi4YOPgbMtYDhoXNw43tVvjB2XCrXAB3m7ePDjgQb697Vtm9p+Jp8aTg/kHue/b+/jDD38g7VJai88trCrkmzPfAHBf4n0dFbJoB0lYOoBxd9DomNF4nFD+YcjsFTswniuU3vYAuSCvIPqGKNvNnXlMv8FgaHm6bXEWFGUoO6iMh0QK4cQCPQOZnzSfb279htt73Y5GpWFz5mZu+/o2ntn+DNll2Zc95/OTn1Otq6ZfSD+Ghg/t+KCF2SRh6QDG6bZjpRxkX/HmD5CD+qm3zlwWOnnpJNnl2XhpvEx9OSaZdYlY5ABlJ5UQLiLSN5JnRz3Ll7d8yXXdrsOAga9Pf81NX93EK/te4VLVJUAGxTkbSVhsLK88jxOFJ1ChYkx5uZSD7Klr3RWW/GNQUdjmcuMP+N05u+12CNuVMg6LGxU9Cm8378YPSv+KcHE9Anvwxvg3+OSGTxgROYIafQ0fHv+QG768gX8c+QdfpH4hg+KciCQsNrY1S9kdNCBsACGp3yl3SjnIPvzCILS3cjuj7asmQ8KH4KXx4kLlBdKKWq6BO7IfMuvKQV2vufxB6V8RncSAsAG8P+l9lk1cRp/gPpTVlPH3w39n6d6lANzT9x4ZFOcEJGGxMdNhhxHJUg5yBBb0sXhoPEiKSAKcsyyUVZbFL4W/oFapGRc7rvGDVcWQd0y5HScJi3B9KpWK0TGjWX3Tav4y5i/E+sUC4OPmw+29ZFCcM5CExYYqaytNDZtjtXopBzkC0zyWtgfIAYyMqtve7ITzWIzloKHhQ+ni1aXxg+f3gUEPXeIhIKrjgxPCTtQqNTf0uIGvp37Nq+Ne5Z+T/0mgZ6C9wxJmcGt7iWivvTl7qdZVE+UbRcLpuh+QUg6yL+MVltyjylUGr9a/UaVEp8ABOJB3AK1Oi4fGowOCtA7jdubLdgeB9K+ITs9d48718dfbOwxhAbnCYkOmww4jR6I6o9yWcpCdBURDl+7K1YWMPW0u79WlF8FewVTWVnLkwpEOCNA6iqqKTMcKNN+/IucHCSGciyQsNmIwGNiaqTTcjte7SznIkZi2N7fdx6JWqevLQk7Ux7I1ays6g46ELgnE+TeZ3FmrhfP7ldtyhUUI4SQkYbGRE4UnyK/Mx9vNm+EZh5U75ewgx9DN/IQFGm9vdhbG/pUJXSdc/mDuT1BbCd7BEJrQwZEJIUT7SMJiI8azg1IihuF5Rrkt/SsOwpiwZB8CbXmby40D5I5dPEZxdbEtI7OKqtoqdmQryVjz/SsNtjPLoCwhhJOQhMVGjNuZx6n86stBYb3tHJUAIKgrBMSCvhYy97a5PMI3gh6BPdAb9OzNbXu9ve3O2U1lbSWRvpH0De57+QLpXxFCOCFJWGzgQsUFjl1UZlyMzTml3CnlIMehUlk+pj/aecb0G3cHTYibcPmocYOhwRUW6V8RQjgPSVhsYOv5uum2wX0JPbNNuVPKQY7FuL3Z3D4WJzlXSKfXsTlzM9DC7qCLaVBxEdy8IGpQh8YmhBBXQhIWGzD2r4x1D1XKQeGJUg5yNN3qBsid3w81VW0uHxY5DDeVG+fLzpNZmmnj4NrvyIUjFFYV4u/hb5rS24jx6kpMErh5dmxwQghxBSRhsbJqXbVpN8n4grofbDJ7xfGEXAV+EaCrhqwDbS73dfdlYNhAwLGvshjLQeNixzV/Nor0rwghnJQkLFa2N2cvlbWVRHiH0ftM3Q82KQc5HpXK8rKQg29vNhgMbMrYBLSwOwjqe3akf0UI4WQkYbEyYzlonHcsKikHObZ2zmPZk7MHnV5nq6ja7XTRaTJLM/FQezA6ZvTlC0pz4dJZQAWxwzs8PiGEuBKSsFiRwWCoT1iKLih3SjnIcRkTlsy9oKtpc3m/kH74u/tToi3h+MXjNg7Ocj9kKqeBj4weia+77+ULjOWgiH7gHdRxgQkhhBVIwmJFqZdSyS3PxUvjyYj0fcqdUg5yXGF9lGmvNRWQfbjN5W5qN0ZEjQAc8/Rm03TbuGam24L0rwghnJokLFZk3E460icOL52UgxyeWt2gj2W7WU9x1O3NueW5/HzxZ1SoGBc3rvlFMn9FCOHEJGGxIuP8lXFlpcodMizO8RnLQumW9bEcvnCYipoKW0VlMWOyPChsEKHeoZcvqC5VzhACSViEEE5JEhYrKags4GjBUQDGZtb9YJD+FcdnvMKSsRvMaKSN848jxi+GWn0t+/P22zg485mm2zZ32CEo82YMegjsCoExHRiZEEJYhyQsVrLt/DYMGEj0jiS8prquHCQn4Tq8yAHgGQjaBlcgWqFSqRgZpfSAOEpZqERbwt4c5YyjFrczS/+KEMLJScJiJabdQVV1u02kHOQc1Jr6H+IWnivkKPNYtp/fTq2hlh6BPYgPjG9+UcMTmoUQwglJwmIFWp2WndnKD7tx2b8od0o5yHkYy0Jm9rEkRyajQkVaURr5Ffk2DMw8bZaDdDVKSQikf0UI4bQkYbGCfbn7qKytJNzNj8SqSgjvJ+UgZxJfd65Qxk7Q69tcHuQVRGJIImD/qyxanZZtWcoBmy2Wg3KPQk05eAUqW7mFEMIJScJiBcZy0BidBhXI7BVnEzUI3H2h8hJcOGHWUxylj2Vv7l7Ka8oJ8w6jf2j/5hcZ+1fiRipbuYUQwgm167vXO++8Q3x8PF5eXiQnJ7N3795W1xcVFTFnzhyioqLw9PQkISGB9evXmx5/9tlnUalUjT769HGO3wQNBgNbMpWEZXzeGeVOKQc5F407xCkD4drTx2IwGGwVWZuMw+KuibsGtaqFf87SvyKEcAEWJyyrV69m/vz5LF68mIMHDzJo0CAmT55Mfn7ztXytVst1111Heno6a9as4eTJkyxfvpyYmMZbK/v160dOTo7pY/t28wZ52VtaURrZ5dl4qtxIrqiQcpCzMs1jMe/v3ZDwIXhpvCioLCCtKM2GgbVMb9Cb+leu6dpCOchgaLBDSPpXhBDOy83SJ7zxxhvMmjWLmTNnArBs2TLWrVvHihUreOqppy5bv2LFCgoLC9m5cyfu7spx9/Hx8ZcH4uZGZGSkpeHYnbEclGzwxNtgkHKQs4pvcBCiwaCc5twKD40HSRFJ7Mjewa7sXfTq0qsDgmzs54KfuVB5AV93X0ZEjmh+UeEZKM8HjQdED+nYAIUQwoosusKi1Wo5cOAAEydOrH8BtZqJEyeya1fztfyvv/6alJQU5syZQ0REBP3792fJkiXodI2HdJ06dYro6Gh69OjB9OnTycjIaDWW6upqSkpKGn3Yg7EcNO5ilnKHlIOcU/RQ0HhC+QW4aN4VE2NZyF7nCv2QoRx2OCZmDB4aj+YXGa+uRA8Fd68OikwIIazPooSloKAAnU5HREREo/sjIiLIzc1t9jlnzpxhzZo16HQ61q9fz8KFC3n99dd58cUXTWuSk5P54IMP2LBhA++++y5nz55lzJgxlJaWthjL0qVLCQwMNH3ExcVZ8qlYRWFVIUcuHAFgbHm5lIOcmbsXxA5XbptZFjI23h7IO4BWp7VVZC1qczszSP+KEMJl2HzLgF6vJzw8nPfee4+kpCSmTZvGM888w7Jly0xrpkyZwp133snAgQOZPHky69evp6ioiM8//7zF112wYAHFxcWmj8zMTFt/KpcxTrftgyeROp2Ug5ydqSxkXuNtQpcEQrxCqKytNCWuHeVs8VnOFJ/BTe3G1TFXt7xQ+leEEC7CooQlNDQUjUZDXl5eo/vz8vJa7D+JiooiISEBjUZjuq9v377k5uai1Tb/W2lQUBAJCQmkpbV8ad7T05OAgIBGHx3NNN22qEC5Q8pBzs10cnNdH0sbVCoVI6Pts73ZeHVlROQI/D38m19UXgAXTym341rocRFCCCdhUcLi4eFBUlISmzZtMt2n1+vZtGkTKSnN/wY3evRo0tLS0DcYyJWamkpUVBQeHs3X3cvKyjh9+jRRUVGWhNehanQ19dNtpRzkGmJHgNoNSrKg6JxZT0mJqutj6eiEpW4784S41spBdVdXwvqCT3AHRCWEELZjcUlo/vz5LF++nFWrVnHixAlmz55NeXm5adfQjBkzWLBggWn97NmzKSwsZN68eaSmprJu3TqWLFnCnDlzTGsef/xxtmzZQnp6Ojt37uTWW29Fo9Fw9913W+FTtI39efspryknBA39tFo5O8gVePgozalg9ph+Yx/LsYvHKK4utlVkjRRUFphKUOPjxre8UPpXhBAuxOJtzdOmTePChQssWrSI3NxcBg8ezIYNG0yNuBkZGagbTNOMi4tj48aNPPbYYwwcOJCYmBjmzZvHk08+aVpz/vx57r77bi5evEhYWBhXX301u3fvJiwszAqfom0Yy0Fjy0qUrE/6V1xD/Gg4v1cpCw2Z3ubyCN8Irgq8itPFp9mTs4dJ8ZNsHuLmzM0YMNA/pD8RvhEtL5T+FSGEC7E4YQGYO3cuc+fObfaxzZs3X3ZfSkoKu3e3fObKZ5991p4w7MZgMLA5czMA48rrhsWFdvwcDmED3UbD9r8qCYuZUqJTOF18ml05uzokYTFrd5C2AnIOK7flCosQwgXIwSLtcKb4DFllWbijIqWySspBriQuGVRquJQOxVlmPcU0j6UD+ljKa8rZna0k/y0edgiQdQD0teAfDUFdbR6XEELYmiQs7WAsB42orMRHptu6Fq8A5TBEMHt787CIYbip3cgqyyKzxLbb63dk7UCr19LVvytXBV3V8kJTOWhkm1N7hRDCGUjC0g6mww7LKyCiv5SDXI3xXKFz5g2Q83H3YVCYkuTYeuptw3KQqrVExNRwK/0rQgjXIAmLhYqqijh84TAAYysrZfaKK+pm2QA56JjtzTX6GtPVvVbLQXodZNadoC79K0IIFyEJi4W2ZW1Db9CToK0hulam27qkriMBFRSkQlnzp5A3Zexj2ZO7B51e18bq9jmQd4BSbSnBXsGmKzrNyjsG2lLw8IeIfjaJRQghOpokLBbaen4rAOMqpBzksnyC63/Qm3mVpV9IP/w9/CnVlnLs4jGbhGU87HB83Hg0ak3LC439K3EjoLV1QgjhRCRhsUCNvoYdWcp213EVUg5yaQ3H9JtBo9aQHJkM2KYsZDAY6vtXWptuC9K/IoRwSZKwWOBg3kFKa0oJ1unoX62VcpArM/axmDnxFhpsb7ZB4+2JwhPklufi7eZNclRyywsNBplwK4RwSZKwWMDY8DimohKNlINcm/EKS/4xqCg06ynGxtsjF45QUVNh1XCM5aDR0aPxcvNqeWFRBpTmKGcixSRZNQYhhLAnSVjMZDAYTNuZx1VUytUVV+cXDqF1h1lmmHfFJNY/lhi/GGr1tezP22/VcMyabgv1/StRg5WzkYQQwkVIwmKm9JJ0MkozcDMYGFVZBYky3dblWbi9WaVSmQ5DtGYfS2ZpJqmXUtGoNIyNHdv6YikHCSFclCQsZjJeXRleVYVveD8I7WnniITNmfpYzBsgB/V9LLtzWj47y1I/ZihXV5Iikgj0DGx9sRx4KIRwUZKwmMnYvyLloE7E2MeS+xNUlZj1lOTIZFSoSCtKI7/CvBkubTG7HFRRCBdOKLflCosQwsVIwmKG4upiDuUfAozbmaUc1CkExkCXeDDoIXOPWU8J8goiMSQRsM5VlktVlziYfxBoY7ot1E+3DekFvqFX/N5CCOFIJGExw46sHegMOnpqtcSGJEo5qDPpdrXyZzvKQtboY9lyfgt6g54+wX2I9otufbH0rwghXJgkLGbYfH4zYCwH3WLfYETHMg2Qs/xcod05uzEYDFf09sb+lTaHxYH0rwghXJokLG2o1dey/fw2QMpBnVJ8XeNt9kHQlpv1lMHhg/F286agsoBTRafa/daVtZXszFYSpTb7V2qqlBhBrrAIIVySJCxtOJR/iNKaMoJ0OgYGJUg5qLMJ6gYBMaCvhfP7zHqKh8aDoRFDgSsrC+3K3kWVropo32gSuiS0vjj7EOi04BsOwT3a/Z5CCOGoJGFpg/GwwzEVVWhkd1Dno1K1b0x/1JWP6TdOt53QdQIqlar1xQ37V9paK4QQTkgSljZsrvuhMa6iQspBnVW8ZQPkoL7x9kDuAbQ6rcVvWauvNW2lb7McBNK/IoRweZKwtKKipoIIPXjr9Yzyv0rKQZ2V8QrL+X1Kr4gZegX1ItQ7lCpdFYfzD1v8lofzD1NUXUSgZyBDwoe0vlivh0xjwiL9K0II1yQJSyt83H14v9KTLRlZ+Es5qPMK6an0huiq6xtb29BoTH87ykI/ZNZd2Ysdh5varfXFF36BqmJw94XIgRa/lxBCOANJWFpTVQJnt+JtMEg5qDNTqeq3N1vSx9LOeSwGg8HC7cx1rx87DDRtJDdCCOGkJGFpjVcAzDsCU9+VclBnF183QO6c+QPkjFdYjl88TlFVkdnPO1V0ivNl5/HUeJqSnlZJ/4oQohOQhKUtgTEw+B57RyHszXiFJXMv6GrMekq4Tzg9g3piwMCeXPNG+0P97qCUqBR83H3afkKG9K8IIVyfJCxCmCOsL3h3gZoKyD5s9tNMfSwWlIXMPuwQoPg8FGeASqOUhIQQwkVJwiKEOdRq6Goc0295H4u5Y/pzy3M5fvE4KlSMjR3b9hsYr65EDgBPf7PjEkIIZyMJixDmMs1jMT9hGRYxDDe1G1llWWSWZra53lgOGhI+hBDvkLbfQPpXhBCdhCQsQpjLOI8lYzfodWY9xcfdh8FhgwHzykIWlYOMsYD0rwghXJ4kLEKYK3IAeAZAdQnkHjX7aabtzW3MYymuLmZ/7n4Arom7pu0XriqGvJ+V25KwCCFcnCQsQphLralPDCzpY6k7V2hvzl5q9bUtrtuWtY1aQy09g3rSNaBr2y+cuQ8wQJfu4B9pdjxCCOGMJGERwhLG7c0WnCuUGJKIv4c/pTWlHLt4rMV1xv4Vs66uQIMDD6V/RQjh+iRhEcIS3YwD5HYoZ/iYQaPWkByZDMDu7N3NrqnWVbMjS7lqc23Xa82LRfpXhBCdiCQsQlgiejC4+0DlJeUMHzO11ceyJ2cPFbUVhPuEkxiS2PYL1mohS+l3kSssQojOQBIWISyhcYe4EcrtdvSxHLlwhIqaisseb1gOUqlUbb9gzhGorQLvYAjtZXYcQgjhrCRhEcJSDctCZooLiCPGL4ZafS378/Y3ekxv0LM5czNgyXbmBv0r5iQ4Qgjh5CRhEcJSDU9uNmN6rVFLpzf/dOEnLlZdxM/dj+ERw817MelfEUJ0MpKwCGGpmCTQeEJ5Plw8bfbTjGWhpgnLD5lKOWhM7BjcNe5tv5DBIDuEhBCdjiQsQljK3av+oMFz281+WnJUMipUnC4+TV55nun+HzMsnG5bcAoqC8HNC6IGmf3+QgjhzCRhEaI9jGP6083vYwn0DKRfSD9AOQwR4EzxGdJL0nFXu3N19NXmvZDx6krMMHDzMPv9hRDCmUnCIkR7dGtwcnN7+ljqtjcbdweNiBqBn4efeS9i7F/pJuUgIUTnIQmLEO0RNwLUblCSBUXnzH6aMWHZnb0bg8FQf9hhnJnlIGjQvyINt0KIzqNdCcs777xDfHw8Xl5eJCcns3fv3lbXFxUVMWfOHKKiovD09CQhIYH169c3u/bll19GpVLx6KOPtic0ITqGhy9ED1FuWzCmf1DYILzdvLlYdZGd2Tv56cJPgAXj+Etz4dJZUKkhdoSlUQshhNOyOGFZvXo18+fPZ/HixRw8eJBBgwYxefJk8vPzm12v1Wq57rrrSE9PZ82aNZw8eZLly5cTExNz2dp9+/bxj3/8g4EDB1r+mQjR0drRx+Kh8SApIgmAV/e9CsDA0IGE+YSZ9wLGclBEP/AKMPt9hRDC2VmcsLzxxhvMmjWLmTNnkpiYyLJly/Dx8WHFihXNrl+xYgWFhYWsXbuW0aNHEx8fz7hx4xg0qPHuhrKyMqZPn87y5cvp0qVL+z4bITpSvOUD5KB+e/PpYmVL9DVdzby6Ag3mr0j/ihCic7EoYdFqtRw4cICJEyfWv4BazcSJE9m1q/kzUr7++mtSUlKYM2cOERER9O/fnyVLlqDT6RqtmzNnDjfeeGOj1xbCocUlK6WZS2ehJNvspxn7WIzM3s4M0r8ihOi03CxZXFBQgE6nIyIiotH9ERER/PJL8wfBnTlzhh9++IHp06ezfv160tLSePjhh6mpqWHx4sUAfPbZZxw8eJB9+/aZHUt1dTXV1dWm/y4pKbHkUxHiynkFQORAyDms9LEMuMOsp/UM6kmYdxgXKi8QHxBPj8Ae5r1fdSnkKj0vxEnCItpHp9NRU1Nj7zBEJ+Lu7o5Go7ni17EoYWkPvV5PeHg47733HhqNhqSkJLKysnj11VdZvHgxmZmZzJs3j++++w4vLy+zX3fp0qU899xzNoxcCDN0G60kLOnbzU5YVCoVo2NGszZtLdd2vdb89zq/Dwx6COoKgZf3gAnRGoPBQG5uLkVFRfYORXRCQUFBREZGmne4awssSlhCQ0PRaDTk5eU1uj8vL4/IyMhmnxMVFXVZdtW3b19yc3NNJab8/HyGDh1qelyn07F161b+/ve/U11d3WxmtmDBAubPn2/675KSEuLi4iz5dIS4cvGjYfc7FvexPJb0GL279Ob2hNvNf5L0r4grYExWwsPD8fHxuaIfHEKYy2AwUFFRYdqYExUV1e7Xsihh8fDwICkpiU2bNjF16lRAuYKyadMm5s6d2+xzRo8ezSeffIJer0etVlpmUlNTiYqKwsPDg2uvvZajR482es7MmTPp06cPTz75ZIuXkTw9PfH09LQkfCGsz5g8FKRC2QXwM2+3T7BXMPcm3mvZe0n/imgnnU5nSlZCQkLsHY7oZLy9vQHIz88nPDy83eUhi3cJzZ8/n+XLl7Nq1SpOnDjB7NmzKS8vZ+bMmQDMmDGDBQsWmNbPnj2bwsJC5s2bR2pqKuvWrWPJkiXMmTMHAH9/f/r379/ow9fXl5CQEPr379+uT0qIDuMTDOHKuH1Lr7JYRFcD5/crt+UKi7CQsWfFx8fHzpGIzsr4d+9K+qcs7mGZNm0aFy5cYNGiReTm5jJ48GA2bNhgasTNyMgwXUkBiIuLY+PGjTz22GMMHDiQmJgY5s2bx5NPPtnuoIVwKN1GQf4xpfG231TbvEfuT1BTAV5BENrbNu8hXJ6UgYS9WOPvXruabufOndtiCWjz5s2X3ZeSksLu3bvNfv3mXkMIhxU/GvYtt+0VFlP/ykhQy4kaQojOR77zCXGljBNv845BRaFt3kP6V4TocCqVirVr11r1NcePHy9Hz7STJCxCXCm/cAjpBRjqr4RYk8EgO4SEEA7ngw8+ICgoqMPeTxIWIawhvu4qiy3KQoVnoPwCaDzrD1wUQlxGq9XaOwRhQ5KwCGEN3WyYsBjLQTFDwU228ovOY/z48aaeycDAQEJDQ1m4cCEGgwGA+Ph4XnjhBWbMmEFAQAC//e1vAdi+fTtjxozB29ubuLg4HnnkEcrLy1t9r1OnTjF27Fi8vLxITEzku+++u2xNZmYmd911F0FBQQQHB3PLLbeQnp5uevyBBx5g6tSpPPfcc4SFhREQEMDvf//7VhOpS5cuMWPGDLp06YKPjw9Tpkzh1KlTAJSXlxMQEMCaNWsaPWft2rX4+vpSWlpKeno6KpWKzz//3PQ5Dx8+nNTUVPbt28ewYcPw8/NjypQpXLhwodHrvP/++/Tt2xcvLy/69OnD//3f/5keM77ul19+yTXXXIOPjw+DBg0yHcOzefNmZs6cSXFxMSqVCpVKxbPPPtvq1/hKScIihDV0G6X8mXMEqqx8TIT0rwgrMxgMVGhr7fJhTDbMtWrVKtzc3Ni7dy9vvfUWb7zxBu+//77p8ddee41BgwZx6NAhFi5cyOnTp7n++uu5/fbb+emnn1i9ejXbt29vcaMIKPPEbrvtNjw8PNizZw/Lli27bCdrTU0NkydPxt/fn23btrFjxw78/Py4/vrrGyUkmzZt4sSJE2zevJlPP/2UL7/8stWp7A888AD79+/n66+/ZteuXRgMBm644QZqamrw9fXl17/+NStXrmz0nJUrV3LHHXfg7+9vum/x4sX8+c9/5uDBg7i5uXHPPffwpz/9ibfeeott27aRlpbGokWLTOs//vhjFi1axEsvvcSJEydYsmQJCxcuZNWqVY3e65lnnuHxxx/n8OHDJCQkcPfdd1NbW8uoUaN48803CQgIICcnh5ycHB5//PEWP09rsPlofiE6hcBYCOoGRecgcy/0suIhntK/IqysskZH4qKNdnnv489PxsfD/B89cXFx/PWvf0WlUtG7d2+OHj3KX//6V2bNmgXAhAkT+OMf/2ha/9BDDzF9+nRTY2uvXr14++23GTduHO+++26zR8B8//33/PLLL2zcuJHo6GgAlixZwpQpU0xrVq9ejV6v5/333zdt0V25ciVBQUFs3ryZSZMmAcqA1RUrVuDj40O/fv14/vnneeKJJ3jhhRcajfwA5arO119/zY4dOxg1Svml5+OPPyYuLo61a9dy55138tBDDzFq1ChycnKIiooiPz+f9evX8/333zd6rccff5zJkycDMG/ePO6++242bdrE6NHK1d8HH3yQDz74wLR+8eLFvP7669x2220AdO/enePHj/OPf/yD+++/v9Hr3njjjQA899xz9OvXj7S0NPr06UNgYCAqlarFSffWJldYhLCW+KuVP89tt95rll2Ai2nK7bgR1ntdIZzEyJEjG83wSElJ4dSpU+h0OgCGDRvWaP2RI0f44IMP8PPzM31MnjwZvV7P2bNnWbJkSaPHMjIyOHHiBHFxcaZkxfg+TV83LS0Nf39/03ODg4Opqqri9OnTpnWDBg1qNKAvJSWFsrIyMjMzL/vcTpw4gZubG8nJyab7QkJC6N27NydOnABgxIgR9OvXz3Tl46OPPqJbt26MHTu20WsNHDjQdNs4F23AgAGN7jOOxy8vL+f06dM8+OCDjb4WL774YqPPpenrGsfqG1+no8kVFiGspdsoOPyxMkDOWjLrrq6EJ4J3F+u9rujUvN01HH9+st3e25p8fX0b/XdZWRm/+93veOSRRy5b27VrV37/+99z1113me5rmKS0pqysjKSkJD7++OPLHgsLM+9IjvZ66KGHeOedd3jqqadYuXIlM2fOvGwQm7u7u+m28bGm9+n1ekD5XACWL1/eKFkCLhub39zrGl+no0nCIoS1GBtvsw6CtgI8rDAGveHAOCGsRKVSWVSWsac9e/Y0+u/du3fTq1evFs+jGTp0KMePH6dnz57NPh4cHExwcHCj+/r27UtmZqap7GJ8n6avu3r1asLDwwkICGgx3iNHjlBZWWk6P2f37t34+fk1ezhv3759qa2tZc+ePaaS0MWLFzl58iSJiYmmdffeey9/+tOfePvttzl+/Hijkk17REREEB0dzZkzZ5g+fXq7X8fDw8N0pasjSElICGvpEg8BMaCvgfP7rPOapoZb6V8RnVNGRgbz58/n5MmTfPrpp/ztb39j3rx5La5/8skn2blzJ3PnzuXw4cOcOnWK//znP6023U6cOJGEhATuv/9+jhw5wrZt23jmmWcarZk+fTqhoaHccsstbNu2jbNnz7J582YeeeQRzp8/b1qn1Wp58MEHOX78OOvXr2fx4sXMnTv3sv4VUPprbrnlFmbNmsX27ds5cuQI9957LzExMdxyyy2mdV26dOG2227jiSeeYNKkScTGxlryJWzWc889x9KlS3n77bdJTU3l6NGjrFy5kjfeeMPs14iPj6esrIxNmzZRUFBARUXFFcfVGklYhLAWlap+t5A1tjdry5VdRyBXWESnNWPGDCorKxkxYgRz5sxh3rx5pu3LzRk4cCBbtmwhNTWVMWPGMGTIEBYtWtRq6UetVvPVV1+Z3uehhx7ipZdearTGx8eHrVu30rVrV2677Tb69u3Lgw8+SFVVVaMrLtdeey29evVi7NixTJs2jV/96letbvdduXIlSUlJ3HTTTaSkpGAwGFi/fn2jUgwoTbNarZbf/OY3bXzFzPPQQw/x/vvvs3LlSgYMGMC4ceP44IMP6N69u9mvMWrUKH7/+98zbdo0wsLCeOWVV6wSW0tUBkv3mDmokpISAgMDKS4ubvVynRA2tX8lfPModLsaZq67stc6uxVW3axctXnsmJIQCdEOVVVVnD17lu7duze7S8ZRjR8/nsGDB/Pmm2/aOxSzPPDAAxQVFVl9nD/Ahx9+yGOPPUZ2djYeHh5Wf31ba+3voLk/v52jiCmEszD2sZzfB7XVVzborWH/iiQrQnRKFRUV5OTk8PLLL/O73/3OKZMVa5GSkBDWFNoLfMNAVw1ZB67staR/RYhO75VXXqFPnz5ERkayYMECe4djV3KFRQhrMvaxHP+P0sdi7GmxlK5WGUAH0r8iOq3NmzfbOwSLNBzMZi3PPvuszUfeOwu5wiKEtXWrGyCXfgWNt/nHQFsGngHKDBYhhOjkJGERwtqMJzdn7gVdTftew9i/EjcC1NYdtCWEEM5IEhYhrC2srzKVtqbBtmRLyYGHQgjRiCQsQlibWg1d63pX0ttxrpDBIAceCiFEE5KwCGELpgFy7ThXqOgclOaA2h2ih1o3LiGEcFKSsAhhC8Y+loxdoLfwrA3j1ZXowdY5j0gIIVyAJCxC2ELEAPDwh+oSyPvZsudK/4oQNpOeno5KpeLw4cP2DkVYSBIWIWxB41afcFi6vVn6V4QQ4jKSsAhhK+05CLGiEC78otyOS7Z+TEI4Ma1Wa+8QhB1JwiKErcTXDZA7txP0evOek7lH+TM0AXxDbROXEE5i/PjxzJ07l0cffZTQ0FAmT57Mzz//zJQpU/Dz8yMiIoL77ruPgoIC03M2bNjA1VdfTVBQECEhIdx0002cPn3ajp+FsBZJWISwlajB4O4DlYVQcNK850j/iugIBgNoy+3zYTBYFOqqVavw8PBgx44dvPzyy0yYMIEhQ4awf/9+NmzYQF5eHnfddZdpfXl5OfPnz2f//v1s2rQJtVrNrbfeit7cXxqEw5KzhISwFTcPiB0OZ7co81jC+7b9HOlfER2hpgKWRNvnvZ/OBg9fs5f36tWLV155BYAXX3yRIUOGsGTJEtPjK1asIC4ujtTUVBISErj99tsbPX/FihWEhYVx/Phx+vfvb53PQdiFXGERwpZMZSEz+lhqKiHroHJbrrAIAUBSUpLp9pEjR/jxxx/x8/MzffTp0wfAVPY5deoUd999Nz169CAgIID4+HgAMjIyOjx2YV1yhUUIW2o4QM5gUE5zbkn2IdDXgF8EdOneMfGJzsndR7nSYa/3toCvb/3VmLKyMm6++Wb+8pe/XLYuKioKgJtvvplu3bqxfPlyoqOj0ev19O/fXxp2XYAkLELYUsww0HhAWR5cPA2hPVte27B/pbXERogrpVJZVJZxFEOHDuWLL74gPj4eN7fLf3xdvHiRkydPsnz5csaMGQPA9u3tOB5DOCQpCQlhS+5eStICbZeFpH9FiFbNmTOHwsJC7r77bvbt28fp06fZuHEjM2fORKfT0aVLF0JCQnjvvfdIS0vjhx9+YP78+fYOW1iJJCxC2JpxTH9rCYteDxl1W5qlf0WIZkVHR7Njxw50Oh2TJk1iwIABPProowQFBaFWq1Gr1Xz22WccOHCA/v3789hjj/Hqq6/aO2xhJVISEsLWzDkI8cIJqC4Gd19lrL8Qgs2bN192X69evfjyyy9bfM7EiRM5fvx4o/sMDbZSx8fHN/pv4TzkCosQthaXDGo3KM6ES+eaX2PsX4kbroz1F0II0YgkLELYmocvRA9RbrdUFpL+FSGEaJUkLEJ0hLbOFTIlLNK/IoQQzZGERYiO0K1ugFxzJzcXZSrlIpWmfkeREEKIRiRhEaIjdE0GlRounYWSJgO7jAceRg0ET7+Oj00IIZyAJCxCdASvQIis2/3TdLeQaWCc9K8IIURLJGERoqN0a2Eei/SvCCFEmyRhEaKjGBOWhn0slUWQd0y5HScJixBCtEQSFiE6irHkU3ASyi4ot8/vAwwQ3AP8I+wWmhBCODpJWIToKL4hEJ6o3M6o62OR/hUhOtQDDzzA1KlT7R2GRZwxZltoV8LyzjvvEB8fj5eXF8nJyezdu7fV9UVFRcyZM4eoqCg8PT1JSEhg/fr1psffffddBg4cSEBAAAEBAaSkpPDtt9+2JzQhHFvTspD0rwhhE+np6ahUKg4fPtzo/rfeeosPPvjA5u8vSYb1WZywrF69mvnz57N48WIOHjzIoEGDmDx5Mvn5+c2u12q1XHfddaSnp7NmzRrT0d8xMTGmNbGxsbz88sscOHCA/fv3M2HCBG655RaOHTvW/s9MCEfU8Fyh2mrIOqD8t1xhEaJDBAYGEhQUZO8wRDtYnLC88cYbzJo1i5kzZ5KYmMiyZcvw8fFhxYoVza5fsWIFhYWFrF27ltGjRxMfH8+4ceMYNGiQac3NN9/MDTfcQK9evUhISOCll17Cz8+P3bt3t/8zE8IRGa+w5P0MZ7ZAbRX4hEJIT/vGJYSD0uv1LF26lO7du+Pt7c2gQYNYs2YNAJcuXWL69OmEhYXh7e1Nr169WLlyJQDdu3cHYMiQIahUKsaPHw9cfuVj/Pjx/OEPf+DRRx+lS5cuREREsHz5csrLy5k5cyb+/v707Nmz0VV/nU7Hgw8+aIqpd+/evPXWW6bHn332WVatWsV//vMfVCoVKpXKdJBjZmYmd911F0FBQQQHB3PLLbeQnp7e6LXnz59PUFAQISEh/OlPf5LDGutYlLBotVoOHDjAxIkT619ArWbixIns2rWr2ed8/fXXpKSkMGfOHCIiIujfvz9LlixBp9M1u16n0/HZZ59RXl5OSkrLv3VWV1dTUlLS6EMIh+cfUZecGGD7X5X7uo4ElcquYYnOxWAwUFFTYZcPS3/4Ll26lH/9618sW7aMY8eO8dhjj3HvvfeyZcsWFi5cyPHjx/n22285ceIE7777LqGhoQCmVoXvv/+enJycVk94XrVqFaGhoezdu5c//OEPzJ49mzvvvJNRo0Zx8OBBJk2axH333UdFRQWgJFGxsbH8+9//5vjx4yxatIinn36azz//HIDHH3+cu+66i+uvv56cnBxycnIYNWoUNTU1TJ48GX9/f7Zt28aOHTvw8/Pj+uuvR6vVAvD666/zwQcfsGLFCrZv305hYSFfffWVxf+PXZFFx8IWFBSg0+mIiGi8myEiIoJffvml2eecOXOGH374genTp7N+/XrS0tJ4+OGHqampYfHixaZ1R48eJSUlhaqqKvz8/Pjqq69ITExsMZalS5fy3HPPWRK+EI6h22i4mFbfeCvlINHBKmsrSf4k2S7vveeePfi4+5i1trq6miVLlvD999+bfoHt0aMH27dv5x//+AdlZWUMGTKEYcOUIy3i4+NNzw0LCwMgJCSEyMjIVt9n0KBB/PnPfwZgwYIFvPzyy4SGhjJr1iwAFi1axLvvvstPP/3EyJEjcXd3b/Tzp3v37uzatYvPP/+cu+66Cz8/P7y9vamurm703h999BF6vZ73338fVd0vKStXriQoKIjNmzczadIk3nzzTRYsWMBtt90GwLJly9i4caNZXy9XZ/Nz7PV6PeHh4bz33ntoNBqSkpLIysri1VdfbZSw9O7dm8OHD1NcXMyaNWu4//772bJlS4tJy4IFC5g/f77pv0tKSoiLi7P1pyPEles2Gg6uqv9vSViEaFZaWhoVFRVcd911je7XarUMGTKEZ599lttvv910FWTq1KmMGjXK4vcZOHCg6bZGoyEkJIQBAwaY7jP+kt6wV/Odd95hxYoVZGRkUFlZiVarZfDgwa2+z5EjR0hLS8Pf37/R/VVVVZw+fZri4mJycnJITq5PJt3c3Bg2bJiUhbAwYQkNDUWj0ZCXl9fo/ry8vBYz2KioKNzd3dFoNKb7+vbtS25uLlqtFg8PDwA8PDzo2VOp4yclJbFv3z7eeust/vGPfzT7up6ennh6eloSvhCOIX50/W03b+UMISE6kLebN3vu2WO39zZXWVkZAOvWrWu0UQOUnwFxcXGcO3eO9evX891333HttdcyZ84cXnvtNYticnd3b/TfKpWq0X3GqyF6vR6Azz77jMcff5zXX3+dlJQU/P39efXVV9mzp/WvaVlZGUlJSXz88ceXPWa8IiRaZlHC4uHhQVJSEps2bTI1Len1ejZt2sTcuXObfc7o0aP55JNP0Ov1qNVKy0xqaipRUVGmZKU5er2e6upqS8ITwjkExkJQVyjKgNhhoHFv+zlCWJFKpTK7LGNPiYmJeHp6kpGRwbhx45pdExYWxv3338/999/PmDFjeOKJJ3jttddMP19a6pe8Ejt27GDUqFE8/PDDpvtOnz7daI2Hh8dl7z106FBWr15NeHg4AQEBzb52VFQUe/bsYezYsQDU1tZy4MABhg4dauXPwvlYXBKaP38+999/P8OGDWPEiBG8+eabpm5qgBkzZhATE8PSpUsBmD17Nn//+9+ZN28ef/jDHzh16hRLlizhkUceMb3mggULmDJlCl27dqW0tJRPPvmEzZs3S91OuK4e1yhloe5j7R2JEA7L39+fxx9/nMceewy9Xs/VV19NcXExO3bsICAggNOnT5OUlES/fv2orq7mm2++oW/fvgCEh4fj7e3Nhg0biI2NxcvLi8DAQKvE1atXL/71r3+xceNGunfvzocffsi+fftMO5NA6afZuHEjJ0+eJCQkhMDAQKZPn86rr77KLbfcwvPPP09sbCznzp3jyy+/5E9/+hOxsbHMmzePl19+mV69etGnTx/eeOMNioqKrBK3s7M4YZk2bRoXLlxg0aJF5ObmMnjwYDZs2GCq8WVkZJiupADExcWxceNGHnvsMQYOHEhMTAzz5s3jySefNK3Jz89nxowZ5OTkEBgYyMCBA9m4ceNldUshXMbEZ5XTm4fca+9IhHBoL7zwAmFhYSxdupQzZ84QFBTE0KFDefrpp8nMzGTBggWkp6fj7e3NmDFj+OyzzwCl9+Ptt9/m+eefZ9GiRYwZM8a0tfhK/e53v+PQoUNMmzYNlUrF3XffzcMPP9xo6/OsWbPYvHkzw4YNo6ysjB9//JHx48ezdetWnnzySW677TZKS0uJiYnh2muvNV1x+eMf/0hOTg73338/arWa3/zmN9x6660UFxdbJXZnpjK4SCdPSUkJgYGBFBcXt3ipTQghOqOqqirOnj1L9+7d8fLysnc4ohNq7e+guT+/5SwhIYQQQjg8SViEEEII4fAkYRFCCCGEw5OERQghhBAOTxIWIYQQQjg8SViEEKKTME5qFaKjWePvns3PEhJCCGFfHh4eqNVqsrOzCQsLw8PDwzRuXghbMhgMaLVaLly4gFqtbnXCfVskYRFCCBenVqvp3r07OTk5ZGdn2zsc0Qn5+PjQtWvXRoNlLSUJixBCdAIeHh507dqV2tpam5yvI0RLNBoNbm5uV3xVTxIWIYToJIynEDc9nVgIZyBNt0IIIYRweJKwCCGEEMLhScIihBBCCIfnMj0sxkOnS0pK7ByJEEIIIcxl/Llt/DneEpdJWEpLSwGIi4uzcyRCCCGEsFRpaSmBgYEtPq4ytJXSOAm9Xk92djb+/v5WHYhUUlJCXFwcmZmZBAQEWO11XZF8rcwnXyvLyNfLfPK1Mp98rcxny6+VwWCgtLSU6OjoVue0uMwVFrVaTWxsrM1ePyAgQP5Cm0m+VuaTr5Vl5OtlPvlamU++Vuaz1deqtSsrRtJ0K4QQQgiHJwmLEEIIIRyeJCxt8PT0ZPHixXh6eto7FIcnXyvzydfKMvL1Mp98rcwnXyvzOcLXymWaboUQQgjhuuQKixBCCCEcniQsQgjx/+3dX0hTfQDG8WdtHZUapZa2JVtFlpVtmGvDVgQphYTUTUkYjOqmOJImQXS1rpw3QX+Q1QqqG7EI1j8wG2aLIDEnAy2oLKHojyMImbuo2H7dDXzN3pf3wt9PeT5wYJ6rL7uQZztnGxEpj4OFiIiIlMfBQkRERMrjYPkX7e3tWLFiBXJzc+HxeNDf3y87STlPnz5FXV0drFYrDAYD7ty5IztJWYFAAJs3b4bZbEZRURH27t2L169fy85SUjAYhMPhyH5RVVVVFbq6umRnzQptbW0wGAxobm6WnaKkM2fOwGAwTDrKyspkZynr06dPOHjwIAoLC5GXl4eNGzdiYGBgxjs4WP7i5s2baGlpgd/vx+DgIJxOJ3bt2oVEIiE7TSmpVApOpxPt7e2yU5QXjUah6zr6+voQiUTw69cv7Ny5E6lUSnaackpKStDW1oZYLIaBgQHs2LEDe/bswcuXL2WnKe3Fixe4fPkyHA6H7BSlbdiwAV++fMkez549k52kpO/fv8Pr9WL+/Pno6urCq1evcPbsWeTn5898jKBpud1uoet69u90Oi2sVqsIBAISq9QGQITDYdkZs0YikRAARDQalZ0yK+Tn54urV6/KzlBWMpkUpaWlIhKJiO3bt4umpibZSUry+/3C6XTKzpgVTp06JbZu3So7QwghBN9hmcbPnz8Ri8VQU1OTPTdv3jzU1NTg+fPnEstoLhkfHwcAFBQUSC5RWzqdRmdnJ1KpFKqqqmTnKEvXdezevXvS/y36s7dv38JqtWLVqlVoaGjAhw8fZCcp6d69e3C5XNi3bx+KiopQUVGBK1euSGnhYJnGt2/fkE6nUVxcPOl8cXExvn79KqmK5pJMJoPm5mZ4vV6Ul5fLzlHS0NAQFi5ciJycHBw9ehThcBjr16+XnaWkzs5ODA4OIhAIyE5RnsfjwfXr1/Hw4UMEg0GMjo5i27ZtSCaTstOU8/79ewSDQZSWlqK7uxvHjh3D8ePHcePGjRlvmTO/1kw02+i6juHhYV47/4u1a9ciHo9jfHwct2/fhs/nQzQa5Wj5h48fP6KpqQmRSAS5ubmyc5RXW1ubfexwOODxeGC323Hr1i0cOXJEYpl6MpkMXC4XWltbAQAVFRUYHh7GpUuX4PP5ZrSF77BMY8mSJTAajRgbG5t0fmxsDMuWLZNURXNFY2MjHjx4gN7eXpSUlMjOUZamaVi9ejUqKysRCATgdDpx/vx52VnKicViSCQS2LRpE0wmE0wmE6LRKC5cuACTyYR0Oi07UWmLFy/GmjVrMDIyIjtFORaLZcoLhHXr1km5hMbBMg1N01BZWYmenp7suUwmg56eHl5Dp/9NCIHGxkaEw2E8fvwYK1eulJ00q2QyGfz48UN2hnKqq6sxNDSEeDyePVwuFxoaGhCPx2E0GmUnKm1iYgLv3r2DxWKRnaIcr9c75asX3rx5A7vdPuMtvCT0Fy0tLfD5fHC5XHC73Th37hxSqRQOHTokO00pExMTk16ZjI6OIh6Po6CgADabTWKZenRdR0dHB+7evQuz2Zy9H2rRokXIy8uTXKeW06dPo7a2FjabDclkEh0dHXjy5Am6u7tlpynHbDZPuQ9qwYIFKCws5P1Rf3Dy5EnU1dXBbrfj8+fP8Pv9MBqNOHDggOw05Zw4cQJbtmxBa2sr9u/fj/7+foRCIYRCoZmPkf0xJdVdvHhR2Gw2oWmacLvdoq+vT3aScnp7ewWAKYfP55Odppw/PU8AxLVr12SnKefw4cPCbrcLTdPE0qVLRXV1tXj06JHsrFmDH2ueXn19vbBYLELTNLF8+XJRX18vRkZGZGcp6/79+6K8vFzk5OSIsrIyEQqFpHQYhBBi5mcSERER0X/He1iIiIhIeRwsREREpDwOFiIiIlIeBwsREREpj4OFiIiIlMfBQkRERMrjYCEiIiLlcbAQERGR8jhYiIiISHkcLERERKQ8DhYiIiJSHgcLERERKe83PWNQBvEKR5MAAAAASUVORK5CYII=",
"text/plain": [
"<Figure size 640x480 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Estimated MSE: 5.36355459317329e-05, Pre-deployment MSE: 0.0005099683612281597\n"
]
}
],
"source": [
"import matplotlib.pyplot as plt\n",
"# plot pre-deployment, real and estimated accuracy\n",
"plt.plot(pre_deployment_accs, label='pre-deployment')\n",
"plt.plot(real_accs, label='real')\n",
"plt.plot(estimated_accs, label='estimated')\n",
"plt.legend()\n",
"plt.show()\n",
"\n",
"mse = np.mean((np.array(real_accs) - np.array(estimated_accs))**2)\n",
"baseline_mse = np.mean((np.array(real_accs) - np.array(pre_deployment_accs))**2) \n",
"\n",
"print(f\"Estimated MSE: {mse}, Pre-deployment MSE: {baseline_mse}\")\t"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"The __Estimated MSE__ is the mean squared error between Estimated and Real accuracies, while the __Pre-deployment MSE__ is the mean squared error between Pre-deployment and Real accuracies.\n",
"\n",
"As we can see, even though the estimates are not perfect, it is very closer to the real ones than if we had used the pre-deployment accuracy. The seeds are not fixed in this example, so you can rerun to see different results. I find that the errors for the estimated accuracies are usually roughly one order of magnitude smaller than the errors for the pre-deployment accuracies. This is a sign that the accuracy estimation is working well."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Uploading the results to WhyLabs <a class=\"anchor\" id=\"whylabs\"></a>\n",
"\n",
"Let's see how to have the estimated accuracy available in your WhyLabs dashboard.\n",
"\n",
"First, let's set our WhyLabs environment variables:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import getpass\n",
"import os\n",
"\n",
"# set your org-id here - should be something like \"org-xxxx\"\n",
"print(\"Enter your WhyLabs Org ID\") \n",
"os.environ[\"WHYLABS_DEFAULT_ORG_ID\"] = input()\n",
"\n",
"# set your datased_id (or model_id) here - should be something like \"model-xxxx\"\n",
"print(\"Enter your WhyLabs Dataset ID\")\n",
"os.environ[\"WHYLABS_DEFAULT_DATASET_ID\"] = input()\n",
"\n",
"\n",
"# set your API key here\n",
"print(\"Enter your WhyLabs API key\")\n",
"os.environ[\"WHYLABS_API_KEY\"] = getpass.getpass()\n",
"print(\"Using API Key ID: \", os.environ[\"WHYLABS_API_KEY\"][0:10])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, let's iterate through the days again. The difference this time is we'll write both the result set and estimated result to WhyLabs.\n",
"\n",
"Just be sure to set the result set's timestamp before performing the estimation, so that the estimated result will have the same timestamp as the result set.\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"from datetime import datetime, timedelta, timezone\n",
"from whylogs.experimental.performance_estimation.estimators import AccuracyEstimator\n",
"from whylogs.api.writer.whylabs import WhyLabsWriter\n",
"\n",
"writer = WhyLabsWriter()\n",
"estimator = AccuracyEstimator(reference_result_set = reference_results)\n",
"\n",
"for day, perturbed in enumerate(zip(perturbed_results_list, perturbed_dfs)):\n",
" dataset_timestamp = datetime.now() - timedelta(days=day)\n",
" dataset_timestamp = dataset_timestamp.replace(tzinfo = timezone.utc)\n",
" perturbed_results = perturbed[0]\n",
"\n",
" perturbed_results.set_dataset_timestamp(dataset_timestamp)\n",
"\n",
" #logging the data\n",
" perturbed_results.writer(\"whylabs\").write()\n",
" estimator.estimate(perturbed_results).writer(\"whylabs\").write()\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"You will find the estimated accuracy in the `Outputs` tab of your WhyLabs dashboard. Since we're logging a single row for each day, the `estimated median` graph will show the estimated accuracy for each day:\n",
"\n",
"\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"> The PerformanceEstimator is still an experimental feature, and the WhyLabs support for it is still under development. It is likely that both whylogs and WhyLabs will evolve to better support the performace estimation use case."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Conclusion <a class=\"anchor\" id=\"conclusion\"></a>\n",
"\n",
"In this example, we showed how we can use whylogs to estimate the accuracy of a model when the distribution of the input data changes, and when we don't have available labels to calculate the real accuracy. We used the `AccuracyEstimator` from the `performance_estimation` module to estimate the accuracy of a model, and we compared the estimated accuracy to the real accuracy. However, even when we do have the labels, or a subset of them, it can still be useful to draw estimates to help further debug and analyse the root cause of eventual changes in performance.\n",
"\n",
"It's also important to note the approach's limitations and assumptions, with regards to the root cause of changes in the performance and the importance of properly choosing the segments to perform the importance weighting.\n",
"\n",
"The estimator is still in beta and we are always working on improving it. We'd love to hear your feedback!"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## References\n",
"\n",
"- [Shankar, Shreya, and Aditya Parameswaran. \"Towards Observability for Production Machine Learning Pipelines.\"](https://arxiv.org/pdf/2108.13557.pdf)\n",
"- [Oreilly event - Monitor Real-Time Machine Learning Performance](https://learning.oreilly.com/live-events/monitor-real-time-machine-learning-performance/0636920075104/0636920075102/)\n",
"- [whylogs - Ecommerce dataset](https://whylogs.readthedocs.io/en/latest/datasets/ecommerce.html)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
},
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "5dd5901cadfd4b29c2aaf95ecd29c0c3b10829ad94dcfe59437dbee391154aea"
}
}
},
"nbformat": 4,
"nbformat_minor": 2
}