02 - Image Classification.ipynb

{"cells":[{"cell_type":"markdown","metadata":{},"source":["# Image Classification\n","\n","The *Computer Vision* cognitive service provides useful pre-built models for working with images, but you'll often need to train your own model for computer vision. For example, suppose the Northwind Traders retail company wants to create an automated checkout system that identifies the grocery items customers want to buy based on an image taken by a camera at the checkout. To do this, you'll need to train a classification model that can classify the images to identify the item being purchased.\n","\n","![A robot holding a clipboard, classifying pictures of an apple, a banana, and an orange](./images/image-classification.jpg)\n","\n","In Azure, you can use the ***Custom Vision*** cognitive service to train an image classification model based on existing images. There are two elements to creating an image classification solution. First, you must train a model to recognize different classes using existing images. Then, when the model is trained you must publish it as a service that can be consumed by applications.\n","\n","## Create a Custom Vision resource\n","\n","To use the Custom Vision service, you need an Azure resource that you can use to *train* a model, and a resource with which you can *publish* it for applications to use. The resource for either (or both) tasks can be a general **Cognitive Services** resource, or a specific **Custom Vision** resource. You can use the same Cognitive Services resource for each of these tasks, or you can use different resources (in the same region) for each task to manage costs separately.\n","\n","Use the following instructions to create a new **Custom Vision** resource.\n","\n","1. In a new browser tab, open the Azure portal at [https://portal.azure.com](https://portal.azure.com), and sign in using the Microsoft account associated with your Azure subscription.\n","2. Select the **&#65291;Create a resource** button, search for *custom vision*, and create a **Custom Vision** resource with the following settings:\n","    - **Create options**: Both\n","    - **Subscription**: *Your Azure subscription*\n","    - **Resource group**: *Select or create a resource group with a unique name*\n","    - **Name**: *Enter a unique name*\n","    - **Training location**: *Choose any available region*\n","    - **Training pricing tier**: F0\n","    - **Prediction location**: *The same region as the training resource*\n","    - **Prediction pricing tier**: F0\n","\n","    > **Note**: If you already have an F0 custom vision service in your subscription, select **S0** for this one.\n","\n","3. Wait for the resources to be created, and note that two Custom Vision resources are provisioned; one for training, and another for prediction. You can view these by navigating to the resource group where you created them.\n","\n","## Create a Custom Vision project\n","\n","To train an object detection model, you need to create a Custom Vision project based on your training resource. To do this, you'll use the Custom Vision portal.\n","\n","1. Download and extract the training images from https://aka.ms/fruit-images. **Note:** as a temporary workaround, if you are not able to access the training images, please go to https://www.github.com, then go to https://aka.ms/fruit-images.  \n","2. In another browser tab, open the Custom Vision portal at [https://customvision.ai](https://customvision.ai). If prompted, sign in using the Microsoft account associated with your Azure subscription and agree to the terms of service.\n","3. In the Custom Vision portal, create a new project with the following settings:\n","    - **Name**: Grocery Checkout\n","    - **Description**: Image classification for groceries\n","    - **Resource**: *The Custom Vision resource you created previously*\n","    - **Project Types**: Classification\n","    - **Classification Types**: Multiclass (single tag per image)\n","    - **Domains**: Food\n","4. Click **\\[+\\] Add images**, and select all of the files in the **apple** folder you extracted previously. Then upload the image files, specifying the tag *apple*, like this:\n","\n","![Upload apple with apple tag](./images/upload_apples.jpg)\n","   \n","5. Repeat the previous step to upload the images in the **banana** folder with the tag *banana*, and the images in the **orange** folder with the tag *orange*.\n","6. Explore the images you have uploaded in the Custom Vision project - there should be 15 images of each class, like this:\n","\n","![Tagged images of fruit - 15 apples, 15 bananas, and 15 oranges](./images/fruit.jpg)\n","    \n","7. In the Custom Vision project, above the images, click **Train** to train a classification model using the tagged images. Select the **Quick Training** option, and then wait for the training iteration to complete (this may take a minute or so).\n","8. When the model iteration has been trained, review the *Precision*, *Recall*, and *AP* performance metrics - these measure the prediction accuracy of the classification model, and should all be high.\n","\n","## Test the model\n","\n","Before publishing this iteration of the model for applications to use, you should test it.\n","\n","1. Above the performance metrics, click **Quick Test**.\n","2. In the **Image URL** box, type `https://aka.ms/apple-image` and click &#10132;\n","3. View the predictions returned by your model - the probability score for *apple* should be the highest, like this:\n","\n","![An image with a class prediction of apple](./images/test-apple.jpg)\n","\n","4. Close the **Quick Test** window.\n","\n","## Publish and consume the image classification model\n","\n","Now you're ready to publish your trained model and use it from a client application.\n","\n","9. Click **&#128504; Publish** to publish the trained model with the following settings:\n","    - **Model name**: groceries\n","    - **Prediction Resource**: *The prediction resource you created previously*.\n","\n","### (!) Check In \n","Did you use the same model name: **groceries**?   \n","\n","10. After publishing, click the *settings* (&#9881;) icon at the top right of the **Performance** page to view the project settings. Then, under **General** (on the left), copy the **Project Id**. Scroll down and paste it into the code cell below step 13 replacing **YOUR_PROJECT_ID**.\n","\n","![Project ID in project settings](./images/cv_project_settings.jpg)\n","\n","> _**Note**: If you used a **Cognitive Services** resource instead of creating a **Custom Vision** resource at the beginning of this exercise, you can copy its key and endpoint from the right side of the project settings, paste it into the code cell below, and run it to see the results. Otherwise, continue completing the steps below to get the key and endpoint for your Custom Vision prediction resource._\n","\n","11. At the top left of the **Project Settings** page, click the *Projects Gallery* (&#128065;) icon to return to the Custom Vision portal home page, where your project is now listed.\n","\n","12. On the Custom Vision portal home page, at the top right, click the *settings* (&#9881;) icon to view the settings for your Custom Vision service. Then, under **Resources**, expand your **prediction** resource (<u>not</u> the training resource) and copy its **Key** and **Endpoint** values to the code cell below step 13, replacing **YOUR_KEY** and **YOUR_ENDPOINT**.\n","\n","### (!) Check In \n","If you are using a **Custom Vision** resource, did you use the **prediction** resource (<u>not</u> the training resource)?\n","\n","![Prediction resource key and endpoint in custom vision settings](./images/cv_settings.jpg)\n","\n","13. Run the code cell below by clicking the **Run cell** (&#9655;) button (to the left of the cell) to set the variables to your project ID, key, and endpoint values."]},{"cell_type":"code","execution_count":null,"metadata":{"gather":{"logged":1599691949340}},"outputs":[],"source":["project_id = 'YOUR_PROJECT_ID'\n","cv_key = 'YOUR_KEY'\n","cv_endpoint = 'YOUR_ENDPOINT'\n","\n","model_name = 'groceries' # this must match the model name you set when publishing your model iteration (it's case-sensitive)!\n","print('Ready to predict using model {} in project {}'.format(model_name, project_id))"]},{"cell_type":"markdown","metadata":{},"source":["Now you can use your key and endpoint with a Custom Vision client to connect to your custom vision classification model.\n","\n","Run the following code cell to classifiy a selection of test images using your published model.\n","\n","> **Note**: Don't worry too much about the details of the code. It uses the Computer Vision SDK for Python to get a class prediction for each image in the /data/image-classification/test-fruit folder"]},{"cell_type":"code","execution_count":null,"metadata":{"gather":{"logged":1599692327514}},"outputs":[],"source":["from azure.cognitiveservices.vision.customvision.prediction import CustomVisionPredictionClient\n","from msrest.authentication import ApiKeyCredentials\n","import matplotlib.pyplot as plt\n","from PIL import Image\n","import os\n","%matplotlib inline\n","\n","# Get the test images from the data/vision/test folder\n","test_folder = os.path.join('data', 'image-classification', 'test-fruit')\n","test_images = os.listdir(test_folder)\n","\n","# Create an instance of the prediction service\n","credentials = ApiKeyCredentials(in_headers={\"Prediction-key\": cv_key})\n","custom_vision_client = CustomVisionPredictionClient(endpoint=cv_endpoint, credentials=credentials)\n","\n","# Create a figure to display the results\n","fig = plt.figure(figsize=(16, 8))\n","\n","# Get the images and show the predicted classes for each one\n","print('Classifying images in {} ...'.format(test_folder))\n","for i in range(len(test_images)):\n","    # Open the image, and use the custom vision model to classify it\n","    image_contents = open(os.path.join(test_folder, test_images[i]), \"rb\")\n","    classification = custom_vision_client.classify_image(project_id, model_name, image_contents.read())\n","    # The results include a prediction for each tag, in descending order of probability - get the first one\n","    prediction = classification.predictions[0].tag_name\n","    # Display the image with its predicted class\n","    img = Image.open(os.path.join(test_folder, test_images[i]))\n","    a=fig.add_subplot(int(len(test_images)/3), 3,i+1)\n","    a.axis('off')\n","    imgplot = plt.imshow(img)\n","    a.set_title(prediction)\n","plt.show()"]},{"cell_type":"markdown","metadata":{},"source":["Hopefully, your image classification model has correctly identified the groceries in the images.\n","\n","## Learn more\n","\n","The Custom Vision service offers more capabilities than we've explored in this exercise. For example, you can also use the Custom Vision service to create *object detection* models; which not only classify objects in images, but also identify *bounding boxes* that show the location of the object in the image.\n","\n","To learn more about the Custom Vision cognitive service, view the [Custom Vision documentation](https://docs.microsoft.com/azure/cognitive-services/custom-vision-service/home)"]}],"metadata":{"kernel_info":{"name":"python3-azureml"},"kernelspec":{"display_name":"Python 3.6 - AzureML","language":"python","name":"python3-azureml"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.9"},"nteract":{"version":"nteract-front-end@1.0.0"}},"nbformat":4,"nbformat_minor":2}