Image Segmentation

Learn to perform image segmentation using the Hugging Face Inference API.

We'll cover the following

Image segmentation is used to partition an image into different segments (sets of pixels) corresponding to a distinct object. It's usually used for detailed information about the image segments rather than object detection. It finds its use in several applications, especially in biomedical imaging. It can be divided into the following three types:

  • Semantic: This refers to pixel-level segmentation.

  • Instance: This is similar to semantic segmentation and is also done on the pixel-level. Here, however, each instance is treated as a distinct segment. For example, instance segmentation of an image of three cats results in all three cats being treated as separate instances.

  • Panoptic: This is a new type of segmentation and is a hybrid of both semantic and instance segmentation.

Hugging Face provides us with models for all of three aforementioned types.

Image segmentation using the API

The facebook/detr-resnet-50-panoptic model is recommended for image segmentation task. However, there are many models available. Some panoptic segmentation models are below:

Models for Panoptic Segmentation

Model

Description

facebook/detr-resnet-50-panoptic


Based on resnet-50. Trained on the COCO 2017 panoptic dataset of object detection that has around 118 thousand labeled images and calls an object query for each object in the image, which is 100 for the COCO dataset.

facebook/detr-resnet-101-panoptic


Based on resnet-101. Trained on the same dataset as facebook/detr-resnet-50-panoptic.

We can call the following endpoint via the POST request method for the image segmentation by replacing the path parameter {model} with any model mentioned above:

https://api-inference.huggingface.co/models/{model}

Request parameters

This endpoint only takes binary representation of an image file.

The code below does the segmentation of a cat image. We can take input directly from the URL.

Press + to interact
// Endpoint URL
const endpointUrl = "https://api-inference.huggingface.co/models/facebook/detr-resnet-50-panoptic";
const headerParameters = {
  "Authorization": "Bearer {{ACCESS_TOKEN}}"
};
async function classifyImage(imgUrl) {
  try {
    // Reading image from URL
    const imgResponse = await fetch(imgUrl);
    const buffer = await imgResponse.buffer();
    const options = {
      method: "POST",
      headers: headerParameters,
      body: buffer
    };
    // Calling endpoint URL 
    const response = await fetch(endpointUrl, options);
    printResponse(response);
  } catch (error) {
    printError(error);
  }
}
// URL of the image
let imgUrl = "https://images.unsplash.com/photo-1604675223954-b1aabd668078";
classifyImage(imgUrl);

We specify the endpoint URL with the facebook/detr-resnet-50-panoptic model at line 2.

Response fields

This API call returns a dictionary object containing possible labels ordered by the likelihood scores and masks of the objects in the image.

Parameter

Type

Description

score

Float

Specifies the likelihood score of the label

label

String

Specifies the predicted label of the image

mask

String

Specifies the base64 string code of the object

We can convert the base64 string to an image, as shown below:

Press + to interact
// base64 str of educative logo
let mask = "iVBORw0KGgoAAAANSUhEUgAAADIAAAAyCAYAAAAeP4ixAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAhGVYSWZNTQAqAAAACAAFARIAAwAAAAEAAQAAARoABQAAAAEAAABKARsABQAAAAEAAABSASgAAwAAAAEAAgAAh2kABAAAAAEAAABaAAAAAAAAAEgAAAABAAAASAAAAAEAA6ABAAMAAAABAAEAAKACAAQAAAABAAAAMqADAAQAAAABAAAAMgAAAACG8cKoAAAACXBIWXMAAAsTAAALEwEAmpwYAAACymlUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczp0aWZmPSJodHRwOi8vbnMuYWRvYmUuY29tL3RpZmYvMS4wLyIKICAgICAgICAgICAgeG1sbnM6ZXhpZj0iaHR0cDovL25zLmFkb2JlLmNvbS9leGlmLzEuMC8iPgogICAgICAgICA8dGlmZjpZUmVzb2x1dGlvbj43MjwvdGlmZjpZUmVzb2x1dGlvbj4KICAgICAgICAgPHRpZmY6UmVzb2x1dGlvblVuaXQ+MjwvdGlmZjpSZXNvbHV0aW9uVW5pdD4KICAgICAgICAgPHRpZmY6WFJlc29sdXRpb24+NzI8L3RpZmY6WFJlc29sdXRpb24+CiAgICAgICAgIDx0aWZmOk9yaWVudGF0aW9uPjE8L3RpZmY6T3JpZW50YXRpb24+CiAgICAgICAgIDxleGlmOlBpeGVsWERpbWVuc2lvbj4yMDA8L2V4aWY6UGl4ZWxYRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpDb2xvclNwYWNlPjE8L2V4aWY6Q29sb3JTcGFjZT4KICAgICAgICAgPGV4aWY6UGl4ZWxZRGltZW5zaW9uPjIwMDwvZXhpZjpQaXhlbFlEaW1lbnNpb24+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgqse+HeAAAGIUlEQVRoBc2aT6hVVRTGX381JIhMgjcwFLLCgVGBDeIR2MRxg5omQmmRROIgEh5YAwsaJAgO4kENrEEUSH8m+QxFKyIdFUkIRmqhZr2nZJra99tnf6ftuefczr333HvPgu/sdfZee+9v7bX2Pufc9yYmMrlBxY1RX6pyWvhGmBOutQR/isdXwlZhUkDgDPcguaK7l4S/hLaQr+IxL44bBQT+wQdHYocq3PGS9H+Eq7FEHzfMBW7muV06Yh9CJGjE6IqADnF3aENpR+ACx8uR33MqQ0jYEz8KCwWM8Q7DmwTksPCHkKYg9aMSiN8prIoTmpu5so/vpW1awNghcyT2qm6l0BZ5UEQOCHA1R3PeAsmvk0Yb4ITlZim3CrdEEDGjWxsRTftx7369lmk2HNQ4dsZ896kuP2IJFQbAkVggvUrSwYs2VW1V9cX+ZfcsCvKIAEf4ej//Kj0nb+++o1LCaiMvCnuED4W3BK+m1ImnhU+ED4QZ4XbB8rgU+u0WaF8qIPQfROj/vYAz5sw+6XDEaeUVgAydwC8C9T4I3o71bp/UveUFKa6nZCUR983uers6oofUjTF9cs2VrY6NMUTYUJa/rcSSgSwXrcSS1UrF46V1/ermmJdOn24DHlfjSQEnjgrkpoX63wReH84LqaNndH9K4OhmngsC0qRD2YjxysDA+TYb6x0tStIJFB1nReq0ceI1IY4A71wp5/kiMSazMW2svju4rdjHKUS/YhuniqXY5vpeSuYgnc0x71s2uPeEy9y4RYr3JoscJHXEXt6nltcFUiY3DNbjv8CRDFgWqTj9Q4hSsuh2KNq2tki5nk8jAmOcwACv2+oQ/OBtroFn0RG1BxnkoeUxhll2LHLREYerw3CYrJoYO3XE7/m8az0r3CakD78m5ht0DBaYE+sdgRdb+IXsSR1RXZBzun4b9bYWvEkgZFCQMkdcx9O4LCKsgJ8xVTbZ6M1fiQgHkTnmW8AV6ZT2klQrOoI9TvDpOSV8LHCWF+1UNRQxcXPMJ8kfKHlNtUIkWI1JYVb4SHhKwImm3qU0VP+CdwCSlJBEUietP6Z63mqxI2KUjwrIKJxxRDpeGk0wo1J99QDLZbJYIL1wAuHDiyhxmozt+VPXER/N74rstOD3MKJ4l/CZgBPY1R1Tps0KKwu6pRYzEhWT3C2dPkTG0WHPINg5gqGiwYvH7Ugt5qjrCLZ2BN2D4QhpxTjbBaTsNMxaBrs25gg0vA+WSD8hOJqO6HqMJMNwplFHUpIP6WZewBmfYuhrBMROZ3eDXysdSVOll2lYfd7FeC/bGTvyPPGX23vSiQjOeXKpw5N+w88zg7+jPCFsivRYFJN+RTrOUjeqp35Pm1288gffKun8llXcIy9jJEmjjd4PwkDJxQvlg8b7kvTuyRFH8B71Ox37kk4+tV6TjjS9N7JR/4t4hyMmZsNuJeRYAR6AX8bSK0Kq7RJeFcrSabnqFwqkmVdValch0scEjvdaQgenB+Vs7AUhi3We6IcF7JgAoPMDt8VEXS5WA79GYscvkY5gtxI77KcExBH2mH1HhEGRzwX+4MJEOEck9gpPCgh1rHoqRJ0TDmEh6ojHWFDHGBuvdF17HEFwABwR1goIq2YCoSJeWARHjiiQji7RjWKE1BSOb8r/lbI94tVPO1NHWN8QfhZ41zourBEgyTgQKhMWi/RCcL4XqYqgOeZlmSN8/SE2yu6ye+zfF84KOPS70M0JNYdf6neoxBmcdZ5LrRTmZtyfooW5UBL5O2L9dWPxIY8BaUEJeEYgZSuSOu9NmFl3Xq+bqLO5Vo3H8H7hQ858/Vp0ipEOxQbnKkb7aYjCQEXgDE4U68vuSS3vqV5L+jKmBZ1/LYFjyvcLDLbGBm9IDDA8IDwsMNi4BQ6rhdQJOJrzZjycFI4KiwTSi06EzGnDHx5JP+rpPEqBH5zYEw/Eic3NXM+pfkVsm3heij3EEJ3IpPuGunECLs4WOHJcw2edEISVRt4UTJSQ0YkOlGXnPvWjgOc2F6cTXLcJSMgewgeQDcKcYIfaWnLsOxKBvx2wM4TvbuEZgSf2/QJ7pw3CX41/ED4VZgR+XyMSIf3/BeNi1QRIVJB5AAAAAElFTkSuQmCC";
let img = "data:image/png;base64," + mask;
console.log(`<img src=${img} width="50px" height="50px" />`);

Examples

In the previous example, the cat was in a portrait. That's no fun. Let's confuse the model a bit with these images:

//Example#1
let imgUrl = "https://images.unsplash.com/photo-1599889959407-598566c6e1f1?ixlib=rb-4.0.3&ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&auto=format&fit=crop&w=500&q=40"
//Example#2
let imgUrl = "https://images.unsplash.com/photo-1643251935745-4209d215f221?ixlib=rb-4.0.3&ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&auto=format&fit=crop&w=1200&q=40"

Try out the images above by replacing the imgUrl at line 29 in the widget below, and observe the results.

Press + to interact
// Endpoint URL
const endpointUrl = "https://api-inference.huggingface.co/models/facebook/detr-resnet-50-panoptic";
const headerParameters = {
  "Authorization": "Bearer {{ACCESS_TOKEN}}"
};
async function classifyImage(imgUrl) {
  try {
    // Reading image from URL
    const imgResponse = await fetch(imgUrl);
    const buffer = await imgResponse.buffer();
    const options = {
      method: "POST",
      headers: headerParameters,
      body: buffer
    };
    // Calling endpoint URL 
    const response = await fetch(endpointUrl, options);
    printResponse(response);
  } catch (error) {
    printError(error);
  }
}
// URL of the image
let imgUrl = "https://images.unsplash.com/photo-1599889959407-598566c6e1f1?ixlib=rb-4.0.3&ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&auto=format&fit=crop&w=500&q=40";
// Rendering html to show image
console.log(`Original image:<br><img src=${imgUrl} width="400px" height="500px">`);
classifyImage(imgUrl);