examples/docs/09_lazy.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
},
"tags": []
},
"source": [
"# Lazy File Loading\n",
"\n",
"With ZnTrack > 0.3.5 a lazy loading feature was introduced. This is essential for graphs with many dependencies and large Files.\n",
"Lazy file loading allows us to only load data when it is accessed.\n",
"This tutorial will show the benefits but also the difficulties that come with it.\n",
"\n",
"By default `config.lazy == True` which globally enables lazy file loading. See the Note section when this can cause problems. You can disable it by changing the `zntrack.config.lazy = False`"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"from zntrack import config\n",
"\n",
"# When using ZnTrack we can write our code inside a Jupyter notebook.\n",
"# We can make use of this functionality by setting the `nb_name` config as follows:\n",
"config.nb_name = \"09_lazy.ipynb\"\n",
"config.lazy = False"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"nbsphinx": "hidden",
"tags": []
},
"outputs": [],
"source": [
"from zntrack.utils import cwd_temp_dir\n",
"\n",
"temp_dir = cwd_temp_dir()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"nbsphinx": "hidden",
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Initialized empty Git repository in /tmp/tmp45orcu_7/.git/\n",
"Initialized DVC repository.\n",
"\n",
"You can now commit the changes to git.\n",
"\n",
"+---------------------------------------------------------------------+\n",
"| |\n",
"| DVC has enabled anonymous aggregate usage analytics. |\n",
"| Read the analytics documentation (and how to opt-out) here: |\n",
"| <https://dvc.org/doc/user-guide/analytics> |\n",
"| |\n",
"+---------------------------------------------------------------------+\n",
"\n",
"What's next?\n",
"------------\n",
"- Check out the documentation: <https://dvc.org/doc>\n",
"- Get help and share ideas: <https://dvc.org/chat>\n",
"- Star us on GitHub: <https://github.com/iterative/dvc>\n"
]
}
],
"source": [
"!git init\n",
"\n",
"!dvc init"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's start by creating some Example Nodes"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"import zntrack\n",
"import random"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"We will now create a PrintOption that is identical to `zn.outs` but prints a message every time the data is read from files."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"from zntrack.fields.zn.options import Output\n",
"\n",
"\n",
"class PrintOption(Output):\n",
" def __init__(self):\n",
" super().__init__(dvc_option=\"outs\", use_repr=False)\n",
" # zntrack will try dvc --PrintOption outs.json\n",
" # we must tell it to use dvc --outs outs.json instead\n",
"\n",
" def _get_value_from_file(self, instance) -> any:\n",
" print(f\"Loading data from files for {instance.name}\")\n",
" return super()._get_value_from_file(instance)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/tmp/ipykernel_3870159/2524714966.py:2: DeprecationWarning: Use 'zntrack.params' instead.\n",
" start = zntrack.zn.params()\n",
"/tmp/ipykernel_3870159/2524714966.py:3: DeprecationWarning: Use 'zntrack.params' instead.\n",
" stop = zntrack.zn.params()\n"
]
}
],
"source": [
"class RandomNumber(zntrack.Node):\n",
" start = zntrack.zn.params()\n",
" stop = zntrack.zn.params()\n",
" number = PrintOption() # = zn.outs() + print\n",
"\n",
" def run(self):\n",
" self.number = random.randrange(self.start, self.stop)"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"In this first Example we will not use lazy loading."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Running DVC command: 'stage add --name RandomNumber --force ...'\n",
"Jupyter support is an experimental feature! Please save your notebook before running this command!\n",
"Submit issues to https://github.com/zincware/ZnTrack.\n",
"\u0000"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u0000"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Running DVC command: 'repro'\n",
"\u0000"
]
}
],
"source": [
"with zntrack.Project() as project:\n",
" random_number = RandomNumber(start=1, stop=1000)\n",
"project.run()"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u0000"
]
}
],
"source": [
"random_number.load(lazy=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As we can see, the RandomNumber is already loaded into memory"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"data": {
"text/plain": [
"598"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"random_number.number"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"Now let us do the same thing with `lazy=True`"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'zntrack.utils.LazyOption'>\n"
]
}
],
"source": [
"lazy_random_number = RandomNumber.from_rev(lazy=True)\n",
"print(lazy_random_number.__dict__[\"number\"])"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"We can see, that the random number is not yet available but as soon as we access the attribute it will be loaded for us (and stored in memory)."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"data": {
"text/plain": [
"598"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"lazy_random_number.number"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"Let's build some dependencies to show where lazy loading is especially useful."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/tmp/ipykernel_3870159/790841409.py:2: DeprecationWarning: Use 'zntrack.deps' instead.\n",
" deps = zntrack.zn.deps()\n"
]
}
],
"source": [
"class AddOne(zntrack.Node):\n",
" deps = zntrack.zn.deps()\n",
" number = PrintOption()\n",
"\n",
" def run(self):\n",
" self.number = self.deps.number + 1"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Running DVC command: 'stage add --name RandomNumber --force ...'\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u0000"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u0000Running DVC command: 'stage add --name AddOne_0 --force ...'\n",
"\u0000"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u0000"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Running DVC command: 'stage add --name AddOne_1 --force ...'\n",
"\u0000"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u0000"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Running DVC command: 'stage add --name AddOne_2 --force ...'\n",
"\u0000"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u0000"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Running DVC command: 'stage add --name AddOne_3 --force ...'\n",
"\u0000"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u0000"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Running DVC command: 'stage add --name AddOne_4 --force ...'\n",
"\u0000"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u0000"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Running DVC command: 'stage add --name AddOne_5 --force ...'\n",
"\u0000"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u0000"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Running DVC command: 'stage add --name AddOne_6 --force ...'\n",
"\u0000"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u0000"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Running DVC command: 'stage add --name AddOne_7 --force ...'\n",
"\u0000"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u0000"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Running DVC command: 'stage add --name AddOne_8 --force ...'\n",
"\u0000"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u0000"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Running DVC command: 'stage add --name AddOne_9 --force ...'\n",
"\u0000"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u0000"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Running DVC command: 'stage add --name AddOne_10 --force ...'\n",
"\u0000"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u0000"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Running DVC command: 'repro'\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u0000"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u0000"
]
}
],
"source": [
"with zntrack.Project() as project:\n",
" random_number = RandomNumber(start=1, stop=100)\n",
"\n",
" add_one = AddOne(deps=random_number, name=\"AddOne_0\")\n",
" for index in range(10):\n",
" add_one = AddOne(deps=add_one, name=f\"AddOne_{index+1}\")\n",
"\n",
"project.run()"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"+--------------+ \n",
"| RandomNumber | \n",
"+--------------+ \n",
" * \n",
" * \n",
" * \n",
" +----------+ \n",
" | AddOne_0 | \n",
" +----------+ \n",
" * \n",
" * \n",
" * \n",
" +----------+ \n",
" | AddOne_1 | \n",
" +----------+ \n",
" * \n",
" * \n",
" * \n",
" +----------+ \n",
" | AddOne_2 | \n",
" +----------+ \n",
" * \n",
" * \n",
" * \n",
" +----------+ \n",
" | AddOne_3 | \n",
" +----------+ \n",
" * \n",
" * \n",
" * \n",
" +----------+ \n",
" | AddOne_4 | \n",
" +----------+ \n",
" * \n",
" * \n",
" * \n",
" +----------+ \n",
" | AddOne_5 | \n",
" +----------+ \n",
" * \n",
" * \n",
" * \n",
" +----------+ \n",
" | AddOne_6 | \n",
" +----------+ \n",
" * \n",
" * \n",
" * \n",
" +----------+ \n",
" | AddOne_7 | \n",
" +----------+ \n",
" * \n",
" * \n",
" * \n",
" +----------+ \n",
" | AddOne_8 | \n",
" +----------+ \n",
" * \n",
" * \n",
" * \n",
" +----------+ \n",
" | AddOne_9 | \n",
" +----------+ \n",
" * \n",
" * \n",
" * \n",
" +-----------+ \n",
" | AddOne_10 | \n",
" +-----------+ \n"
]
}
],
"source": [
"!dvc dag"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"If we now load the latest `AddOne` we will see that it loads up everything into memory, although we might only be interested in the most recent number."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"add_one = AddOne.from_rev(name=\"AddOne_10\", lazy=False)"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"It is rather unlikely that we need all these data to be stored in memory. So we can use `lazy=True` to avoid that."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"add_one_lazy = AddOne.from_rev(name=\"AddOne_10\", lazy=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"We can check with an arbitrary depth of dependencies that both instances yield the same value."
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"data": {
"text/plain": [
"71"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"add_one_lazy.deps.deps.deps.deps.deps.deps.deps.number"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"data": {
"text/plain": [
"71"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"add_one.deps.deps.deps.deps.deps.deps.deps.number"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"## Notes\n",
"When using ZnTrack to compare data of different versions it is important to either not use `lazy=True` or load the data manually before loading another version of the data.\n",
"In the following example we store the result of `dvc repro` for three different experiments with and without `lazy=True` and compare the results."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Running DVC command: 'stage add --name RandomNumber --force ...'\n",
"\u0000"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u0000"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Running DVC command: 'repro'\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u0000"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u0000Running DVC command: 'stage add --name RandomNumber --force ...'\n",
"\u0000"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u0000"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Running DVC command: 'repro'\n",
"\u0000Running DVC command: 'stage add --name RandomNumber --force ...'\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u0000\u0000"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u0000Running DVC command: 'repro'\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u0000"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u0000"
]
}
],
"source": [
"with zntrack.Project() as project:\n",
" node = RandomNumber(start=0, stop=5000)\n",
"project.run()\n",
"\n",
"random_number_lazy_1 = RandomNumber.from_rev(lazy=True)\n",
"random_number_1 = RandomNumber.from_rev(lazy=False)\n",
"\n",
"\n",
"node.stop = 5001\n",
"project.run()\n",
"\n",
"random_number_lazy_2 = RandomNumber.from_rev(lazy=True)\n",
"random_number_2 = RandomNumber.from_rev(lazy=False)\n",
"\n",
"node.stop = 5002\n",
"project.run()\n",
"\n",
"random_number_lazy_3 = RandomNumber.from_rev(lazy=True)\n",
"random_number_3 = RandomNumber.from_rev(lazy=False)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"3008 == 3008 == 3008\n"
]
}
],
"source": [
"# with lazy we get the same number for every run which is not what we expect.\n",
"print(\n",
" f\"{random_number_lazy_1.number} == {random_number_lazy_2.number} ==\"\n",
" f\" {random_number_lazy_3.number}\"\n",
")\n",
"assert random_number_lazy_1.number == random_number_lazy_2.number\n",
"assert random_number_lazy_1.number == random_number_lazy_3.number"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1647 != 849 != 3008\n"
]
}
],
"source": [
"# With lazy=False we get the results we expect.\n",
"# (Except for some random scenarios, where two random numbers are the same.)\n",
"print(f\"{random_number_1.number} != {random_number_2.number} != {random_number_3.number}\")\n",
"assert random_number_1.number != random_number_2.number\n",
"assert random_number_1.number != random_number_3.number"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can \"lock\" one value into place (loading it into memory) by accessing it e.g. through `_ = add_one_lazy_1.number`. This way you are able to only load certain values and still having the benefit of `lazy=True` if you only want to compare certain values."
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"temp_dir.cleanup()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.9"
}
},
"nbformat": 4,
"nbformat_minor": 4
}