Migration guide

If you already have many media assets uploaded and stored somewhere, you have a few options to migrate your existing assets to your Cloudinary product environment. A successful Cloudinary migration requires careful planning, evaluation, and a clear understanding of your objectives.

This guide serves as a valuable starting point, and a high-level overview of your migration planning process, and covers the considerations you should make before and after migration, as well as details of Cloudinary's lazy migration with auto-upload or executing a direct migration.

Migration overview

There are various ways to migrate your resources to your Cloudinary account. The following table summarizes the main options to migrate assets and some considerations to take into account for each of them:

Migration considerations

You should be aware of the following considerations before planning a migration of your assets to Cloudinary.

Pre migration

• Product environment name: If you recently signed up for a new account, a random product environment name, which appears as part of your asset delivery URLs, is generated automatically. If you still have less than 1000 assets in your environment, you can change it to a more meaningful name.
• Automatic backup: Consider activating automatic backup for all uploaded assets, which can be very useful in case assets are deleted by accident, by a rogue script, etc. You can also override this setting on individual uploads by including the backup boolean parameter.
• Custom metadata: If you plan to manage your assets on Cloudinary, you should plan on using tags, contextual metadata, and structured metadata - a powerful way to organize assets and make them discoverable. If you have existing metadata, it is recommended to add the metadata to the assets when migrating the assets. For more details, see the custom metadata comparison table.
• Add-ons: Cloudinary offers a variety of add-ons for moderation, detection, translation, auto-tagging, analyzing, and malware detection. Take note of the various quota and prices when registering for an add-on, and keep in mind that you will probably need more quota for the migrating phase. Most add-ons also have a free tier with a small quota so you can try them out first.
• Storage types: Cloudinary offers a way to restrict access to your assets based on the asset's storage type. Cloudinary supports three different storage types (type in the API):
  • upload - The asset is publicly available. This is the default storage type when uploading files.
  • private - Original assets are only accessible by a signed URL.
  • authenticated - Original assets and all their asset derivations are only accessible through signed URLs.

Migration customization

The following details should be taken into consideration regardless of whether you are planning a lazy migration with auto-upload or a direct migration.

• Upload options: There are extensive parameters available when uploading your assets. The full details can be found in the Upload API reference where you should take special note of the parameters that can aid your migration, such as:
  • Adding custom metadata with metadata, tags, and context. Alternatively, you can use the Media Library UI to import metadata from a CSV file after migration. Structured metadata limitations include a maximum key length of 256 chars and a maximum value length of 1024 chars.
  • Parameters that run add-ons for moderation, detection, translation, auto-tagging, analyzing, and malware detection.
• Public IDs: Every asset uploaded to Cloudinary is assigned a unique identifier in the form of a public ID, which is a URL-safe string that is used to reference the uploaded resource as well as for building delivery URLs. If you don't supply a public_id in the upload API call, you will receive a randomly assigned public ID in the response from the upload API call.
  • Public IDs can be up to 255 characters, including non-English characters, periods (.), forward slashes (/), underscores (_), and hyphens (-).
  • Public ID values cannot begin or end with a space or forward slash (/). Additionally, they cannot include the following characters: ? & # \ % < > +
• Incoming transformations: You can modify the original asset with incoming transformations before saving them to your account.
• Upload presets: Use an upload preset to ensure consistency in your uploaded assets and to centrally define a set of upload options. It is highly recommended to define a default upload preset, as well as a dedicated upload preset for each non-default use case.
• Analysis on upload: Cloudinary offers various analysis options when uploading assets, including image quality and accessibility analysis, extracting semantic data from assets, requesting moderation for the uploaded assets, and even evaluating and modifying any upload parameters.
• Eager transformations: You can pre-generate the different versions of an asset you will want to deliver with eager transformations. This is especially useful for videos and large images that may take time to generate for the first time.
• Notifications: - You can tell Cloudinary to notify your application as soon as the upload completes by adding the notification_url parameter and setting it to any valid URL. You also can set notifications globally for all uploads in the Cloudinary Console.

Post migration

After you migrate your assets, you should look into using the following features:

• Asset delivery:
  • Take advantage of the extensive transformations available when delivering your assets.
  • Apply different transformations to different assets with conditional transformations.
  • Named transformations are the best way to ensure consistency in media delivery. Each art direction, layout, or styling can have a dedicated named transformation.
  • Automatic format should be specified with f_auto so the best media encoding/format is delivered depending on the requesting browser. f_auto should be added even when using a named transformation and cannot be included in the named transformation itself.
  • Automatic quality should be specified with q_auto to deliver the smallest file size without reducing visual quality.
• Web performance: How your site performs is a major factor in your users’ experience and SEO. Cloudinary delivers optimized media via global CDNs, so you should expect performance improvements. You can measure your website performance and get actionable recommendations with Cloudinary's Web Speed Test. See Media optimization best practices for more details.
• Monitor your account errors.
• Monitor delivery reports.
• Update subscriptions: Get automatic notifications from Cloudinary on:
  • New features.
  • Cloudinary Service status.
• Support: You can open support tickets or ask questions in our Community forum.

Lazy migration with auto-upload

The Cloudinary auto-upload feature offers a powerful way to migrate media files within your own existing web application to your Cloudinary product environment, by dynamically uploading them only when first accessed rather than pre-uploading them. This "lazy migration" is especially useful if your site has lots of media assets, some of which may be very rarely (or never) accessed.

Auto-uploaded URLs share the exact same URL convention as Cloudinary's regular uploaded asset URLs, which means that for each asset accessed, Cloudinary first checks whether an asset with the given public ID exists in your Cloudinary product environment, and if it doesn't (and it matches a given auto-upload folder mapping), it is retrieved on the fly from the remote host.

The auto-upload feature enables on-the-fly transformation of existing remote media assets and optimized delivery via a CDN, while automatically uploading the file to your Cloudinary product environment for further management and transformation.

Auto-upload overview

The auto-upload feature is implemented by mapping a base remote URL to a specified folder in your Cloudinary Media Library. Then, whenever accessing a Cloudinary delivery URL containing the folder prefix, the media assets are automatically retrieved from the mapped URL if they are not already uploaded to the folder, and are available for further transformation, management, and CDN delivery like any other asset uploaded to Cloudinary.

Mapping a remote URL prefix also allows your Cloudinary delivery URLs to be shorter and more SEO friendly, gives you fine-grained control on which remote domains are allowed for automatic uploads, enables lazy migration of all your existing assets to the cloud as well as other powerful origin pulling options. After an asset has been auto-uploaded from the remote location, all requests for the asset will reference the asset stored in your Cloudinary product environment.

The standard auto-upload URL takes the following structure:

For example: Creating a folder called remote_media and then mapping it to the URL prefix https://upload.wikimedia.org/wikipedia/ allows you to generate a Cloudinary delivery URL that substitutes the remote_media folder prefix for the URL prefix. When the Cloudinary delivery URL is accessed for the first time, Cloudinary automatically retrieves the remote asset from https://upload.wikimedia.org/wikipedia/ and stores it in your Cloudinary product environment.

To retrieve the following Olympic flag image from Wikimedia Commons, and automatically upload it to your Cloudinary product environment:

Access the following Cloudinary URL:

The image is dynamically retrieved the first time this URL is accessed and stored in your Cloudinary product environment with a public ID of remote_media/commons/a/ae/Olympic_flag.

Configuring auto-upload URL mapping

The auto-upload feature is configured in the auto-upload mapping section on the Upload page of the Console Settings:

1. Fill in the Folder name or path for the auto-upload mapping e.g. remote_media/remote_assets. The folder path can be up to three folders deep.

2. Fill in a URL prefix that will be mapped to the specified folder e.g. https://upload.wikimedia.org/wikipedia/ or s3://my-bucket/my-path/

   

   
   To use a private storage (Amazon S3 or Google Storage) bucket as the remote URL prefix, you must make sure your bucket is whitelisted for your Cloudinary product environment.

3. Click the Save button at the bottom of the page.

Once saved, any media stored under the directory you set as the URL prefix and requested by a Cloudinary URL that contains the remote_media folder in the public ID, will automatically be retrieved and stored in your Cloudinary product environment. This asset is treated just like any other asset that was uploaded to your Cloudinary product environment.

Auto-uploading with upload presets

When you use Cloudinary's upload API you can leverage many powerful features (eagerly created derived/transformed assets, notifications on upload, adding tags, contextual and structured metadata, requesting moderation, and more). To be able to benefit from the same features together with auto-upload, you can use upload presets for defining the automatic upload actions. Upload presets include one or more upload parameters, and any of Cloudinary's upload parameters can be defined and included in a preset.

To instruct a particular upload mapping to use a particular upload preset, set the name of the upload preset to be the same name as the auto-upload mapped folder.

For example, naming an auto-upload mapped folder as remote_media and also naming an upload preset as remote_media will automatically apply all the actions specified in the upload preset when auto-uploading images to the remote_media folder.

Auto-uploading with on-the-fly transformations

You can also use the auto-upload feature to apply any of Cloudinary's image or video transformations on the fly to the delivered media asset. Simply add the transformation parameters to the URL directly before the mapped folder name.

For example, the following Cloudinary delivery URL retrieves an image from Wikimedia and stores it in Cloudinary's demo product environment, then crops a thumbnail of the original image to a width of 200 pixels and a height of 200 pixels, centers the image on the detected face, makes it circular, applies a sharpen effect - and then delivers that modified image optimized and cached via CDN:

Creating auto-upload URLs via code

Auto-uploading from within your code can be easily accomplished using Cloudinary's SDKs for all popular development frameworks. The principle is exactly the same as with any URL/tag building of an uploaded asset, except in this case the public ID does not have to match an existing asset, but can match a mapped folder name of a remote existing asset that will be implicitly fetched on first access. For example, the following code snippet retrieves the same transformed image as shown above:

Restricting access to auto-uploaded assets by delivery type

The auto-upload examples given so far allow public access to the original asset as well as all its derived versions. As an alternative, you can control access to auto-uploaded remote assets by defining them as delivery type private or authenticated, and including the sign_url parameter set to true (required for authenticated assets).

For example, to auto-upload an image as authenticated, set the delivery type (type parameter) as authenticated and include a signature in the delivery URL. (In SDKs, set the sign_url parameter as true.) This setting prevents access to the image without an authentication signature:

Restricting access to the auto-upload feature

Besides restricting an asset based on its delivery type, you can also globally control the behavior of the auto-upload feature so that auto-uploaded dynamic URLs require a signature in all cases, even for public assets of type upload:

In the Cloudinary Console, access the Security page of the Console Settings and then check the Uploaded option under the Restricted image types section.

For example, if the auto-upload feature is set as restricted, then the following code snippet allows auto-uploading by signing the URL (In SDKs, set the sign_url parameter to true):

Direct migration

Direct migration is a one-time migration to Cloudinary for migrating all your existing resources in one phase. This option fits one of the following use cases:

• All of your images and videos are actively used.
• You intend to shut down your existing media storage.

Cloudinary CLI

The Cloudinary CLI (Command Line Interface) enables you to interact with Cloudinary through the command line, allowing you to perform Upload API operations by typing commands into a terminal without having to spend time setting up a formal coding environment. Additional helper commands are provided to help you migrate your files over to Cloudinary with minimal effort. You can also combine CLI commands in a batch file to automate the task.

Especially useful are the following commands:

• migrate - migrates a list of external assets to Cloudinary. The URLs of the files to migrate are listed in a separate file and must all have the same prefix.
• sync - synchronizes between a local folder and a Cloudinary folder, maintaining the folder structure.
• upload_dir - uploads a folder of assets, maintaining the folder structure.

Cloudinary Media Library

You can drag and drop selected media files or folders into the Media Library from your file system. The files or folders you drag are added to the currently active folder. When you select folders, the entire folder structure, including any sub-folders, is created within the folder you dragged to.

When using this option, upload options are applied according to your product environment's Media Library upload preset defaults.

Cloudinary Upload Widget

Cloudinary’s Upload Widget includes a complete graphical interface. The widget supports a drag-and-drop functionality, interactive cropping, upload progress indication, and thumbnail previews. The widget also monitors and handles uploading errors. The widget can be easily integrated into your JavaScript code.

Upload from your local file system or from remote sources such as a Web URL, Google Search, DropBox, Shutterstock, and more. The widget allows you to control the public ID, add custom tags for the files you select to upload, or select a different upload preset. If you set a public ID when uploading multiple files, each uploaded file gets the specified public ID plus a counter suffix.

Alternatively, you can use the built-in Upload widget available in the Media Library.

Migration script

Cloudinary provides a secure and comprehensive API for easily uploading media files from server-side code, directly from the browser or from a mobile application, based on the upload method of the Upload API.

Using Cloudinary's upload capabilities, you can upload media assets in bulk with a variety of options for customizing how they will be uploaded, including naming, whether to apply manually specified or automatically generated tags and metadata, whether to apply incoming transformations or other AI-based analysis of the uploaded assets, and much more. You can upload local files, or even specify a remote URL or bucket as the source for your upload calls, so there's no need to download files locally before uploading to Cloudinary.

Implement this migration by creating a script that runs on your media assets and for each file it calls the upload API call.

Script considerations

• Test your script and run scripts on a test product environment first.
• Decide on a Backup and Overwrite policy. Set it explicitly in your migration script.
• Store the responses of all API calls in a database, useful for troubleshooting.
• Take note of public ID naming limitations, especially if also including the use_filename parameter, and validate for duplicate public IDs.