Identify Object Attributes and Moderating Content

Learn to detect objects, brands, faces, and moderate content in images.

We'll cover the following

Detect common objects in images

Object detection in the Image Analysis API works similarly to the object tagging feature with a few differences. When tagging images, the API only returns a list of the detected objects with a confidence measurement. On the contrary, the object detection API also returns the pixel coordinates of a box encapsulating the detected object. We can use this information to derive the relationships between the objects and check whether there are multiple instances of a single object in the image.

Let’s see how object detection works with the following image:

To detect objects, we will call the detect_objects method. Let’s see how it works.

Note: The image URL is provided at line 5. Feel free to change it and test the API on other images.

Press + to interact
# Authenticate the client
computervision_client = ComputerVisionClient("{{ENDPOINT}}", CognitiveServicesCredentials("{{SUBSCRIPTION_KEY}}"))
# Get URL image with different objects
remote_image_url_objects = "https://images.pexels.com/photos/9497630/pexels-photo-9497630.jpeg?auto=compress&cs=tinysrgb&dpr=2&h=750&w=1260"
# Call API with URL
detect_objects_results_remote = computervision_client.detect_objects(remote_image_url_objects)
# Print detected objects results with bounding boxes
print("Detecting objects in remote image:")
if len(detect_objects_results_remote.objects) == 0:
    print("No objects detected.")
else:
    for object in detect_objects_results_remote.objects:
        print("object at location {}, {}, {}, {}".format( \
        object.rectangle.x, object.rectangle.x + object.rectangle.w, \
        object.rectangle.y, object.rectangle.y + object.rectangle.h))

Detect popular brand logos in images

The Image Analysis API can also detect and recognize popular brand logos within images by checking them against Microsoft’s built-in database. Like the object detection tool, the API returns the name of the detected brand logo, a confidence score, and pixel coordinates of a box surrounding the logo.

Note: Since the logos are matched against a predefined database, there may be cases where the API fails to recognize certain logos. In cases like this, you can train custom models using Microsoft’s Custom Vision service.

We’ll use the following image to show it works:

The image above has a Starbucks coffee cup in it. To detect the brand, we’ll use the analyze_image method and specify the feature as "brands". This will return the detected brand name, confidence, and location as a result.

Note: The image URL is provided at line 5. Feel free to change it and test the API on other images.

Press + to interact
# Authenticate the client
computervision_client = ComputerVisionClient("{{ENDPOINT}}", CognitiveServicesCredentials("{{SUBSCRIPTION_KEY}}"))
# Get a URL with a brand logo
remote_image_url = "https://images.pexels.com/photos/1437318/pexels-photo-1437318.jpeg?auto=compress&cs=tinysrgb&dpr=2&w=500"
# Select the visual feature(s) you want
remote_image_features = ["brands"]
# Call API with URL and features
detect_brands_results_remote = computervision_client.analyze_image(remote_image_url, remote_image_features)
print("Detecting brands in remote image: ")
if len(detect_brands_results_remote.brands) == 0:
    print("No brands detected.")
else:
    for brand in detect_brands_results_remote.brands:
        print("'{}' brand detected with confidence {:.1f}% at location {}, {}, {}, {}".format( \
        brand.name, brand.confidence * 100, brand.rectangle.x, brand.rectangle.x + brand.rectangle.w, \
        brand.rectangle.y, brand.rectangle.y + brand.rectangle.h))

Detect faces

Just like detecting objects, we can detect faces and the coordinates of their bounding box. Let’s see how that works with the following image:

There are two people in this image. To detect faces in the above image, we’ll use the analyze_image method and specify the feature as "faces". Let’s run the code below to see how it works.

Note: The image URL is provided at line 5. Feel free to change it and test the API on other images.

Press + to interact
# Authenticate the client
computervision_client = ComputerVisionClient("{{ENDPOINT}}", CognitiveServicesCredentials("{{SUBSCRIPTION_KEY}}"))
# Get an image with faces
remote_image_url_faces = "https://images.pexels.com/photos/1405833/pexels-photo-1405833.jpeg?auto=compress&cs=tinysrgb&dpr=2&h=750&w=1260"
# Select the visual feature(s) you want.
remote_image_features = ["faces"]
# Call the API with remote URL and features
detect_faces_results_remote = computervision_client.analyze_image(remote_image_url_faces, remote_image_features)
# Print the results with gender, age, and bounding box
print("Faces in the remote image: ")
if (len(detect_faces_results_remote.faces) == 0):
    print("No faces detected.")
else:
    for face in detect_faces_results_remote.faces:
        print("'{}' of age {} at location {}, {}, {}, {}".format(face.gender, face.age, \
        face.face_rectangle.left, face.face_rectangle.top, \
        face.face_rectangle.left + face.face_rectangle.width, \
        face.face_rectangle.top + face.face_rectangle.height))

As we can see in the results above, the gender and the age match the image shown above.

Moderate the content in images

We can also use the Image Analysis API to detect inappropriate content in images and tag the content according to a set of pre-defined tags:

  • Adult: These images contain content that depicts nudity and sexual acts.
  • Racy: These images contain sexually suggestive content, often less explicit than images tagged as adult.
  • Gory: These images include depictions of blood, violence, and gore.

Let’s see how that works with the following image:

Note: The above image is designed by Freepik.com

This image contains blood so it should have high confidence for gory content and low for both adult and racy content. To analyze the image for adult and racy content, we’ll use the analyze_image method and specify the feature as "adult".

Note: The image URL is provided at line 5. Feel free to change it and test the API on other images.

Press + to interact
# Authenticate the client
computervision_client = ComputerVisionClient("{{ENDPOINT}}", CognitiveServicesCredentials("{{SUBSCRIPTION_KEY}}"))
# Provide image URL
remote_image_url = "https://image.freepik.com/free-photo/gory-terrified-zombie_1194-51.jpg?1"
# Select the visual feature(s) you want
remote_image_features = ["adult"]
# Call API with URL and features
detect_adult_results_remote = computervision_client.analyze_image(remote_image_url, remote_image_features)
# Print results with adult/racy score
print("Analyzing remote image for adult or racy content ... ")
print("Is adult content: {} with confidence {:.2f}%".format(detect_adult_results_remote.adult.is_adult_content, detect_adult_results_remote.adult.adult_score * 100))
print("Has racy content: {} with confidence {:.2f}%".format(detect_adult_results_remote.adult.is_racy_content, detect_adult_results_remote.adult.racy_score * 100))
print("Has gory content: {} with confidence {:.2f}%".format(detect_adult_results_remote.adult.is_gory_content, detect_adult_results_remote.adult.gore_score * 100))

As we can see from the results, the image’s gory content confidence score is high.

Limitations

The object detection tool comes with its limitations. Let’s look at some of these limitations so we can understand when the API may produce some incorrect results:

  • Objects that are too small or take up less than 5% of the image may not be detected.
  • Objects that are cluttered together may not be recognized.
  • Products from different brands will not be classified based on brands. If we want to retrieve brand information, we can use the brand detection feature.