Cloudinary AI Content Analysis

Cloudinary is a cloud-based service that provides solutions for image and video management. These include server or client-side upload, on-the-fly image and video transformations, fast CDN delivery, and a variety of asset management options.

The Cloudinary AI Content Analysis add-on uses AI-based object detection and content-aware algorithms to provide the following functionality:

• Object-aware cropping: Ensures that your image crops keep the specific objects that matter to you, even when you significantly modify the aspect ratio.

  Original Crop to the sink
  Important

  By default, delivery URLs that use this add-on either need to be signed or eagerly generated. You can optionally remove this requirement by selecting this add-on in the Allow unsigned add-on transformations section of the Security page in the Console Settings.

  (For simplicity, most of the examples on this page show eagerly generated URLs without signatures.)

• Automatic image tagging: Adds tags to your images based on objects or abstract concepts detected by the content-aware detection models specified on upload, or when invoked on images already stored in your product environment.

• AI-based image captioning: Analyzes an image and suggests a caption to use appropriate to the image's contents.

Getting started

Before you can use the Cloudinary AI Content Analysis add-on:

• You must have a Cloudinary account. If you don't already have one, you can sign up for a free account.

• Register for the add-on: make sure you're logged in to your account and then go to the Add-ons page. For more information about add-on registrations, see Registering for add-ons.

• Keep in mind that many of the examples on this page use our SDKs. For SDK installation and configuration details, see the relevant SDK guide.

• If you're new to Cloudinary, you may want to take a look at the Developer Kickstart for a hands-on, step-by-step introduction to Programmable Media features.

Supported content-aware detection models

The Cloudinary AI Content Analysis add-on supports a number of built-in content-aware detection models, each supporting a specific set of categories and objects. You can specify which version of each model to invoke for each use of the add-on.

Cloudinary currently supports the following models:

Model	Description
coco	The Common Objects in Context model contains just 80 common objects.
cld-fashion	Cloudinary's fashion model is specifically dedicated to items of clothing. Used with automatic image tagging, the response includes attributes of the clothing identified, for example whether the garment contains pockets, its material and the fastenings used.
lvis	The Large Vocabulary Instance Segmentation model contains thousands of general objects.
unidet	The UniDet model is a unified model, combining a number of object models, including Objects365, which focuses on diverse objects in the wild.
human-anatomy	Cloudinary's human anatomy model identifies parts of the human body in an image. It works best when the majority of a human body is detected in the image.
cld-text	Cloudinary's text model tells you if your image includes text, and where it's located. Used with automatic image tagging, you can then search for images that contain blocks of text. Used with object-aware cropping, you can choose to keep only the text part, or specify a crop that avoids the text.
shop-classifier	Cloudinary's shop classifier model detects if the image is a product image taken in a studio, or if it's a natural image.
image-type	Cloudinary's image type model detects generic properties about a photographic image, for example, photographic style, setting and time of the photo.
captioning	Cloudinary's captioning model is used to describe the contents of an image. See AI-based image captioning.
watermark-detection	Cloudinary's watermark detection model identifies if the image contains different types of watermark. See Watermark detection.
iqa	The Image Quality Analysis (IQA) model can predict the quality of a given image on a scale from 0 to 1 and provides a general quality estimation, categorized as 'low', 'medium', or 'high'. See Image quality analysis.

Model capabilities

This table shows the capabilities of each supported version of each model:

• Default version is the version of the model that is invoked if left unspecified.
• Version indicates support for a particular version of the model - different versions have different accuracies.
• Default confidence shows the confidence level used when auto_tagging is set to default.
• Tag indicates support for returning tags. This is a required capability for automatic image tagging.
• Confidence indicates support for returning confidence levels.
• Bounding Box indicates support for returning bounding boxes. This is a required capability for object-aware cropping.
• Attributes indicates support for returning attributes for each tag in a (key,value) list.

Supported objects and categories

Start typing the name of an object or category to see if it's supported by one of the built-in models.

---

---

• For object-aware cropping:

  • The Full URL Syntax column shows the syntax to use to detect a specific object or category in a particular version of a model (e.g. coco_v2_tie). You can also omit the version (e.g. coco_tie), or both the model and version (e.g. tie).

• For automatic image tagging:

  • You can specify the model and version (e.g. coco_v2), or only the model (e.g. coco).

• For dynamic video overlays:

  • Specify the object from the cld-fashion model (e.g. g_track_person:obj_hat)

Private models

If you have your own content-aware detection models that you would like to use, these can be integrated as private models that work only on your product environment. This service is provided for customers on Enterprise plans through Professional Services. Contact our Enterprise support and sales team or your CSM to find out more.

Object detection demo

This demo lets you choose one of the content-aware detection models, and shows up to twenty objects that are detected by that model in an image of your choice.

Automatic image tagging is requested on upload, and the response provides the necessary information to overlay bounding boxes around the detected objects, together with the confidence level.

Object-aware cropping

When object-aware cropping is invoked, Cloudinary applies advanced AI-based object detection algorithms on the fly during the crop process. You can either use it in conjunction with auto-gravity to give higher priority to the objects you care about, or directly specify that the crop should be exactly based on the detected coordinates of the specified objects.

Watch this demo to see how the same image is cropped according to the parameters specified in the URL:

Applying object-aware cropping

After registering for the Cloudinary AI Content Analysis add-on, you can apply it in one of two ways:

• Automatic gravity with a high weighting towards a specified object
  This variant of auto-gravity cropping enables you to indicate specific objects or object categories that should be given priority when parts of a photo are cropped out. This is done by specifying an object or an object category as the focal_gravity attribute for the auto gravity parameter (for example, g_auto:cat in URLs) together with a cropping option. If the specified content is not found in the image, the gravity is determined by the standard auto-gravity algorithm.

• Object-specific gravity
  By specifying an object or object category as the gravity parameter (for example, g_cat in URLs) together with a cropping option, you can accurately crop around objects without needing to specify dimensions or aspect ratio. If the specified content is not found in the image, the gravity remains at the center of the image.

When specifying an object or category, you can optionally include a specific model (that supports bounding boxes) and version. For example, you can specify:

• Only the object/category, e.g.: g_auto:cat or g_cat
• The model with the object/category, e.g.: g_auto:coco_cat or g_coco_cat
• The model and version with object/category, e.g.: g_auto:coco_v2_cat or g_coco_v2_cat

If you choose not to specify a model, each model that supports bounding boxes is invoked in turn until the specified content is detected. The order in which they are invoked is: coco > cld-fashion > lvis > unidet > human-anatomy > cld-text.

Consider the original image of a kitchen below:

Using auto-gravity, you can deliver a square thumbnail crop that prioritizes the detected coordinates of the sink, microwave, or refrigerator. To do this, specify the relevant object option for the g_auto gravity definition in conjunction with the thumb or auto cropping option:

Using object-specific gravity, you can choose not to give dimensions or aspect ratio, and deliver an image that is tightly cropped to the object. To do this, specify the relevant object option for the gravity definition in conjunction with the crop cropping option:

You can also specify an aspect ratio together with the crop cropping option, without including specific dimensions. This keeps the object but may show more of the image to fit the aspect ratio.

In addition to the crop, thumb and auto cropping modes, object aware cropping can also be used with the fill and lfill (limit fill) cropping modes. The fill_pad and auto_pad cropping modes work with the auto-gravity variant of object aware cropping, but not object-specific gravity.

Notes on specifying categories and objects

When applying object-aware cropping, you can specify either individual objects or more general object categories.

• When you specify a category, the algorithm gives priority to any objects that are detected from that category.
• The regular auto-gravity behavior also impacts the cropping decision. But if requested objects are detected, they get significantly higher priority than the subjects or salient areas that the regular auto-gravity algorithm selects.
• If you specify the generic object category with auto-gravity (g_auto:object), then any detected objects from any category get priority.
• If there are multiple objects of the same type in the image, object-specific gravity selects the most prominent of the objects, and bases its crop around only that object, whereas auto-gravity may choose to keep more than one of the objects in the crop.
• The categories and objects also work in their plural forms when using object-specific gravity. So, for example, c_crop,g_birds keeps all birds in the crop, whereas c_crop,g_bird keeps only the most prominent bird.

Combining focal gravity options using auto-gravity

When using auto gravity to determine the area to keep in a crop, you can specify multiple focal_gravity options.

This means that in a single auto-gravity parameter, you can optionally specify:

• One or multiple objects (from the same or different categories and/or models)
• Built-in focal gravity options such as face/faces or custom_no_override
• Other add-on based focal gravity options, such as the adv_face, adv_eyes options from the Advanced Facial Attributes Detection add-on
• Only the classic or only the subject auto-gravity algorithm, which in some cases may have some impact on the exact coordinates of the crop, even if other specified objects or focal gravity options are detected. Note that the default algorithm, which combines both of these algorithms, is recommended in the majority of cases.

For example, your auto-gravity URL parameter might be: g_auto:cat:sofa:faces:adv_eyes

This would instruct the cropping mechanism to give top priority to any cats, sofas, faces, or eyes detected in the photo.

For a complete list of all focal_gravity options, see the g_<special_position> section of the Transformation URL API Reference.

Combining focal gravity options using object-specific gravity

When using object-specific gravity to determine the area to keep in a crop, you can specify multiple focal_gravity options, but unlike auto-gravity, the order in which they are specified has an impact on the delivered image.

For example, consider this photo of a cat and dog:

By setting the gravity parameter to cat:dog the cat gets precedence:

Whereas, if you switch the order to dog:cat the dog gets precedence:

You can also combine the auto option to invoke the auto-gravity algorithm if none of the specified objects are found. For example:

• g_dog:cat:auto - auto-gravity is invoked only if no dogs and cats are detected.
• g_dog:auto:cat - auto-gravity weighted by cat (g_auto:cat) is invoked if no dogs are detected.

For example, consider this photo of a cat and three birds:

As there is no dog in the photo, auto-gravity weighted by bird is invoked when using dog:auto:bird. In this case, two birds are kept in the crop:

Notice that if auto-gravity is not specified, the object-specific algorithm chooses the most prominent bird out of the three and only keeps this bird in the crop:

Specifying objects to avoid using auto-gravity

In addition to specifying objects to keep in an image, you can specify objects that you would rather not see. To minimize the likelihood of including a particular object in the cropped image, use auto-gravity with the avoid option for the relevant object or category.

For example, in photos like the one below, you may prefer not to include people because the purpose of the photo is to show an interesting storefront, and the people are a distraction.

Using g_auto by itself makes the people the focal point, but if we use g_auto:person_avoid, the other side of the photo is shown, without the people.

Choosing the cropping mode

When you specify an object, either specifically or in your auto-gravity parameter, the Object-Aware Cropping AI algorithm detects the coordinates of the object and those coordinates are used by the cropping mode.

• When using thumb cropping (c_thumb), the image is cropped as closely as possible to the detected coordinates of the object given the requested aspect ratio, and then scaled to the requested pixel size. Note that if the requested pixel size is greater than the crop, the image is not scaled up, but filled with further pixels from the image.

• When using crop mode (c_crop), the detected coordinates are prioritized as the area to keep when determining how much to cut from each edge of the photo in order to achieve the requested pixel size. If using auto-gravity and the requested pixel size is larger than the coordinates of the detected object, other elements of the image that receive priority from g_auto may impact what else is included in the photo and where in your resulting image the detected object may be located, meaning that the detected object will not necessarily be the center of the photo.

• When using any of the fill-based modes (c_fill, c_lfill, c_fill_pad), the coordinates of the detected object should be retained if any cropping is required after scaling. If using auto-gravity, other elements of the image that receive priority from g_auto may impact what else is included in the photo and where in your resulting image the detected object may be located, meaning that the detected object will not necessarily be the center of the photo.

• When using the auto cropping mode (c_auto), the crop is focused on the object, but also takes into account more of the whole picture, so gives a more 'zoomed out' result than thumb, and crop but more 'zoomed in' than fill. If the requested dimensions are smaller than the best crop, the result is downscaled. If the requested dimensions are larger than the original image, the result is upscaled.

The following examples show how different your cropping results may be for the same requested object in the gravity, but with different cropping modes. In this case, we take the original photo below and apply g_auto:camera and g_camera with fill, crop, thumb and auto cropping modes. In all cases, the same width and aspect ratio are requested (ar_1,w_200).

g_auto:camera

g_camera

Using object-aware cropping for responsive delivery

You can take advantage of object-aware cropping with various cropping modes to assist in responsive art direction. This means that when you deliver different sized images to different devices, you don't just scale the same image, but rather crop images differently for different sizes, so that the important objects are always highly visible.

For example, you may:

• deliver a full-size image to large HD screens
• use g_auto:[your_important_object], or g_[your_important_object] with fill cropping for medium sized screens
• use g_auto:[your_important_object], or g_[your_important_object] with thumb or auto cropping for very small screens.

For more details on delivering responsive images, see the Responsive images guide.

Using objects with the zoompan effect

In addition to cropping, the Cloudinary AI Content Analysis add-on allows you to use objects for start and end points of a zoompan transformation.

The zoompan effect lets you create a video or animated GIF from an image by zooming and panning from one area of the image to another. Use the from and/or to options with objects as gravity and specify a video or animated image format.

The example below is a seven second MP4 video (.mp4) of a model wearing fashionable items, starting zoomed into the hat (from_(g_hat;zoom_4.5)), then zooming out and panning to the pants (to_(g_pants;zoom_1.6)).

Signed URLs

Cloudinary's dynamic image transformation URLs are powerful tools. However, due to the potential costs of your customers experimenting with dynamic URLs that apply the object-aware cropping algorithm, image transformation add-on URLs are required (by default) to be signed using Cloudinary's authenticated API. Alternatively, you can eagerly generate the requested derived images using Cloudinary's authenticated API.

To create a signed delivery URL using SDKs, set the sign_url parameter to true when building a URL or creating an image tag.

The following code example applies object-aware cropping to the skater image, including a signed Cloudinary URL:

The generated Cloudinary URL shown below includes a signature component (/s--acvfjq2y--/). Only URLs with a valid signature that matches the requested image transformation will be approved for on-the-fly image transformation and delivery.

For more details on signed URLs, see Signed delivery URLs.

Automatic image tagging

The automatic image tagging behavior of the Cloudinary AI Content Analysis add-on can be invoked on uploading an image, or by updating an image that's already stored in your product environment. Using the specified model, it analyzes the image, identifies categories and objects, and suggests tags that could be applied to the image.

Object and category detection

Take a look at the following photo of a woman dressed fashionably for winter:

By setting the detection parameter to the name of the model (and optionally the version, e.g. cld-fashion_v3) you want to invoke when calling Cloudinary's upload or update methods, the add-on automatically analyzes the content of the uploaded or specified existing image. For example, invoking the cld-fashion detection model while uploading winter_fashion.jpg:

The upload API response includes the categories and objects automatically identified by the model you requested. As can be seen in the response snippet below, a hat and a specific type of outerwear are automatically detected in the uploaded photo. Depending on the capabilities of each model, different information is returned. In the example below, a confidence score, bounding box and in some cases, attributes, are returned for each detected object. The confidence score is a numerical value representing the certainty of a correct detection, where 1.0 means 100% confidence. The bounding-box parameter shows the location of the object in the image, as an array: [x-coordinate of top left corner, y-coordinate of top left corner, width of box, height of box]. Bounding-box information is used in the object detection demo.

Adding tags to images

By providing the auto_tagging parameter to an upload or update request, images are automatically assigned tags based on the detected content. The value of the auto_tagging parameter is the minimum confidence score of a detected category or object that should be automatically used as an assigned tag. You can also set auto_tagging to default, which uses the model's default confidence.

The following code example automatically tags an uploaded image with all detected categories that have a confidence score higher than 0.6.

The response to the upload request returns the detected categories as well as the assigned tags for categories meeting the minimum confidence score of 0.6:

You can also use the update method to apply auto tagging to images already stored in your product environment.

The following example uses Cloudinary's update method on the puppy image in the product environment, to detect objects and categories in the LVIS model. Tags are automatically assigned based on the objects and categories detected with over a 90% confidence level.

You can use the Admin API's resource_by_tag method to return all resources with a certain tag, for example hat:

You can also use the search method or the Media Explorer advanced search to find images with certain tags.

Asynchronous handling

As automatic image tagging may not be immediate, it is good practice to use asynchronous handling for these calls.

To make the call asynchronous, set the async parameter of the upload method to true. To be notified when the processing is complete, you can either set the notification_url parameter of the upload method (as in the example below) or the global webhook Notification URL in the Upload page of your Cloudinary Console Settings.

The response to an asynchronous upload call looks similar to this:

When the processing is finished, the complete upload response is sent to the notification URL that you specified.

Image quality analysis

You can analyze the quality of an image using the Image Quality Analysis (IQA) model by setting the detection parameter to iqa when calling Cloudinary's upload or update methods.

A quality score from 0 to 1 is returned in the score attribute of the response, and a general quality estimation of low, medium, or high is returned in the quality attribute.

For example, invoking the iqa model while uploading winter_fashion.jpg:

The response includes the iqa-analysis field:

Watermark detection

You can detect watermarks in images by setting the detection parameter to watermark-detection when calling Cloudinary's upload or update methods. The response can be one of the following: banner (see banners), watermark (see watermarks), or, if neither of these are detected, clean.

Banners

If the image contains an opaque text/logo layer with a semi-transparent background it is likely that the image will be flagged as containing a banner.

For example, uploading the following image, requesting watermark-detection, the response shows 99% confidence that it contains a banner:

Upload request:

The response includes:

Watermarks

If the image contains a semi-transparent layer, it is likely that the image will be flagged as containing a watermark.

For example, uploading the following image, requesting watermark-detection, the response shows 99% confidence that it contains a watermark:

Upload request:

The response includes:

AI-based image captioning

The Cloudinary AI Content Analysis add-on can be used to analyze an image and suggest a caption based on the image's contents.

Some example captions suggested by the AI:

1. A brown dog standing on top of a street next to a sidewalk with a building in the background
2. A group of young children playing soccer on a soccer field with a goal post in the foreground and a goal post in the background
3. A hand reaching for a donut with chocolate and sprinkles on it on a dark surface

By setting the detection parameter to captioning when calling Cloudinary's upload or update methods, the add-on automatically analyzes the content of the image. For example, invoking the captioning detection model while uploading toy_room.jpg:

The upload API response includes the captioning information:

Asynchronous handling

As the response may not be immediate, it is good practice to use asynchronous handling for these calls.

To make the call asynchronous, set the async parameter of the upload method to true. To be notified when the processing is complete, you can either set the notification_url parameter of the upload method (as in the example below) or the global webhook Notification URL in the Upload page of your Cloudinary Console Settings.

The response to an asynchronous upload call looks similar to this:

When the processing is finished, the complete upload response is sent to the notification URL that you specified.