docs/ipynb/multi.ipynb
{
"cells": [
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"!pip install autokeras"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"import autokeras as ak"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"In this tutorial we are making use of the\n",
"[AutoModel](/auto_model/#automodel-class)\n",
" API to show how to handle multi-modal data and multi-task.\n",
"\n",
"## What is multi-modal?\n",
"\n",
"Multi-modal data means each data instance has multiple forms of information.\n",
"For example, a photo can be saved as a image. Besides the image, it may also\n",
"have when and where it was taken as its attributes, which can be represented as\n",
"numerical data.\n",
"\n",
"## What is multi-task?\n",
"\n",
"Multi-task here we refer to we want to predict multiple targets with the same\n",
"input features. For example, we not only want to classify an image according to\n",
"its content, but we also want to regress its quality as a float number between\n",
"0 and 1.\n",
"\n",
"The following diagram shows an example of multi-modal and multi-task neural\n",
"network model.\n",
"\n",
"<div class=\"mermaid\">\n",
"graph TD\n",
" id1(ImageInput) --> id3(Some Neural Network Model)\n",
" id2(Input) --> id3\n",
" id3 --> id4(ClassificationHead)\n",
" id3 --> id5(RegressionHead)\n",
"</div>\n",
"\n",
"It has two inputs the images and the numerical input data. Each image is\n",
"associated with a set of attributes in the numerical input data. From these\n",
"data, we are trying to predict the classification label and the regression value\n",
"at the same time.\n",
"\n",
"## Data Preparation\n",
"\n",
"To illustrate our idea, we generate some random image and numerical data as\n",
"the multi-modal data.\n"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"num_instances = 10\n",
"# Generate image data.\n",
"image_data = np.random.rand(num_instances, 32, 32, 3).astype(np.float32)\n",
"# Generate numerical data.\n",
"numerical_data = np.random.rand(num_instances, 20).astype(np.float32)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"We also generate some multi-task targets for classification and regression.\n"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"# Generate regression targets.\n",
"regression_target = np.random.rand(num_instances, 1).astype(np.float32)\n",
"# Generate classification labels of five classes.\n",
"classification_target = np.random.randint(5, size=num_instances)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Build and Train the Model\n",
"Then we initialize the multi-modal and multi-task model with\n",
"[AutoModel](/auto_model/#automodel-class).\n",
"Since this is just a demo, we use small amount of `max_trials` and `epochs`.\n"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"# Initialize the multi with multiple inputs and outputs.\n",
"model = ak.AutoModel(\n",
" inputs=[ak.ImageInput(), ak.Input()],\n",
" outputs=[\n",
" ak.RegressionHead(metrics=[\"mae\"]),\n",
" ak.ClassificationHead(\n",
" loss=\"categorical_crossentropy\", metrics=[\"accuracy\"]\n",
" ),\n",
" ],\n",
" overwrite=True,\n",
" max_trials=2,\n",
")\n",
"# Fit the model with prepared data.\n",
"model.fit(\n",
" [image_data, numerical_data],\n",
" [regression_target, classification_target],\n",
" epochs=1,\n",
" batch_size=3,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Validation Data\n",
"By default, AutoKeras use the last 20% of training data as validation data.\n",
"As shown in the example below, you can use `validation_split` to specify the\n",
"percentage.\n"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"model.fit(\n",
" [image_data, numerical_data],\n",
" [regression_target, classification_target],\n",
" # Split the training data and use the last 15% as validation data.\n",
" validation_split=0.15,\n",
" epochs=1,\n",
" batch_size=3,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"You can also use your own validation set\n",
"instead of splitting it from the training data with `validation_data`.\n"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"split = 5\n",
"\n",
"image_val = image_data[split:]\n",
"numerical_val = numerical_data[split:]\n",
"regression_val = regression_target[split:]\n",
"classification_val = classification_target[split:]\n",
"\n",
"image_data = image_data[:split]\n",
"numerical_data = numerical_data[:split]\n",
"regression_target = regression_target[:split]\n",
"classification_target = classification_target[:split]\n",
"\n",
"model.fit(\n",
" [image_data, numerical_data],\n",
" [regression_target, classification_target],\n",
" # Use your own validation set.\n",
" validation_data=(\n",
" [image_val, numerical_val],\n",
" [regression_val, classification_val],\n",
" ),\n",
" epochs=1,\n",
" batch_size=3,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Customized Search Space\n",
"You can customize your search space.\n",
"The following figure shows the search space we want to define.\n",
"\n",
"<div class=\"mermaid\">\n",
"graph LR\n",
" id1(ImageInput) --> id2(Normalization)\n",
" id2 --> id3(Image Augmentation)\n",
" id3 --> id4(Convolutional)\n",
" id3 --> id5(ResNet V2)\n",
" id4 --> id6(Merge)\n",
" id5 --> id6\n",
" id7(Input) --> id9(DenseBlock)\n",
" id6 --> id10(Merge)\n",
" id9 --> id10\n",
" id10 --> id11(Classification Head)\n",
" id10 --> id12(Regression Head)\n",
"</div>\n"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"input_node1 = ak.ImageInput()\n",
"output_node = ak.Normalization()(input_node1)\n",
"output_node = ak.ImageAugmentation()(output_node)\n",
"output_node1 = ak.ConvBlock()(output_node)\n",
"output_node2 = ak.ResNetBlock(version=\"v2\")(output_node)\n",
"output_node1 = ak.Merge()([output_node1, output_node2])\n",
"\n",
"input_node2 = ak.Input()\n",
"output_node2 = ak.DenseBlock()(input_node2)\n",
"\n",
"output_node = ak.Merge()([output_node1, output_node2])\n",
"output_node1 = ak.ClassificationHead()(output_node)\n",
"output_node2 = ak.RegressionHead()(output_node)\n",
"\n",
"auto_model = ak.AutoModel(\n",
" inputs=[input_node1, input_node2],\n",
" outputs=[output_node1, output_node2],\n",
" overwrite=True,\n",
" max_trials=2,\n",
")\n",
"\n",
"image_data = np.random.rand(num_instances, 32, 32, 3).astype(np.float32)\n",
"numerical_data = np.random.rand(num_instances, 20).astype(np.float32)\n",
"regression_target = np.random.rand(num_instances, 1).astype(np.float32)\n",
"classification_target = np.random.randint(5, size=num_instances)\n",
"\n",
"auto_model.fit(\n",
" [image_data, numerical_data],\n",
" [classification_target, regression_target],\n",
" batch_size=3,\n",
" epochs=1,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Data Format\n",
"You can refer to the documentation of\n",
"[ImageInput](/node/#imageinput-class),\n",
"[Input](/node/#input-class),\n",
"[TextInput](/node/#textinput-class),\n",
"[RegressionHead](/block/#regressionhead-class),\n",
"[ClassificationHead](/block/#classificationhead-class),\n",
"for the format of different types of data.\n",
"You can also refer to the Data Format section of the tutorials of\n",
"[Image Classification](/tutorial/image_classification/#data-format),\n",
"[Text Classification](/tutorial/text_classification/#data-format),\n",
"\n",
"## Reference\n",
"[AutoModel](/auto_model/#automodel-class),\n",
"[ImageInput](/node/#imageinput-class),\n",
"[Input](/node/#input-class),\n",
"[DenseBlock](/block/#denseblock-class),\n",
"[RegressionHead](/block/#regressionhead-class),\n",
"[ClassificationHead](/block/#classificationhead-class),\n",
"[CategoricalToNumerical](/block/#categoricaltonumerical-class).\n"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "multi",
"private_outputs": false,
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}