Storage mode transitions

When using LIBSAFE Go to manage and preserve your content, you may have some content that you want to be immediately accesible for your users, while for another content you may want to benefit from a lower storage cost if you do not need to access it frequently. This is usually referred to as hot or cold storage. Check Storage for a description of the storage architecture in the platform.

Storage types and storage classes

Every object preserved in the platform is assigned to a storage that has a type and a class. Storage type refers to the provider and geographic location of the underlying storage (AWS in Europe, for instance), while the storage class defines the mode in which the file is, from the range offered by the provider in the region (S3 standard, S3 cold).

Main concepts are:

Ingestion storage class: storage class used to upload content to the platform. The platform will keep files in this class for a pre-defined period of time (to allow their processing) and then, initiates a transfer process to the Default Storage Class.
Default storage class: storage class that the user wants for the new uploaded content to go.
Current storage class: storage class in which a file is at a given point in time.

Off-line and on-line storage

Storage clases can be on-line or off-line.

On-line (usually hot storage) means that your files are immediately accesible. You can open, download or share them.
Off-line (usually cold storage) means that your files are NOT immediately accessible. You cannot access (open, download, share, etc.) them until you transfer them to an on-line storage class. Transferring them may take some time or have some additional costs.

Data container default storage

For every container, it is possible to define its storage type and the default class that will be assigned to the content when it is uploaded to the platform:

Every storage type has an ingestion storage class that is used to upload content to. After a period of time defined in the policy, the platform will change the class for every new file to the indicated default storage class for the selected container.

For example: You want for your data to be in Amazon AWS, in Central Europe and using AWS Deep Archive to get low storage costs.

In this case, your container's storage type is going to be AWS (eu-central-1) and your container is going to have a default storage class set to AWS S3 Deep Archive.

When you upload a file, the platform is not going to immediately move it to AWS S3 Deep Archive (even if it is your default storage class), as reading the file once it is in Deep Archive is expensive and slow and the platform needs to read it in order to generate its hashes, characterize it, etc. The platform is going to keep it in the ingestion storage class (which is S3 standard), process it and then migrate it to the default storage class.

Change the storage class for your content using the Management Interface

Locate the data container you want to upload content to using the Containers menu section or by searching.
Select Check-in in case you are not checked in the container, and you have the check-in/out enabled for the data container.
In the data container page, choose Explore content:

Select the files or folders you would like to move to another storage class. If you select a folder, everything that is inside will also be transferred.

Select the Change storage class function in the Functions side bar:

Confirm that the files the platform will show you are the files you want to transfer and select Next:

Select the desired storage class from the ones you have available for the content and select next:

Select "Run action" to initiate the process.

If you have selected a large amount of files to transfer, you must remain in the same page until the process is completed. If you leave the page, your transfer will be only partially completed.

If the storage type for your container supports file versioning, every previous version of a file is removed in advance to the storage transition, leaving only the last/most recent one. If you want to preserve any of them, you need to manually retrieve them in advance to any migration.

E.g.: If you have an on-line stored file with three versions (the last one plus two previous modifications) and you transition it to a cold storage, only the most recent version is moved to the cold storage and every previous version is discarded, not being possible to recover them.

Change the storage class for your content using the API

API examples here are just illustrative. Check the API documentation for additional information and all available methods.

Sign in to the platform's Management Interface
Obtain your API key by selecting your name and then Access Methods:

and then:

Get the available storage types and classes

To get the storage TYPES available, use:

curl --request GET \    
     --url "$your_platform_url/api/storages" \    
     --header "Content-Type: application/json" \    
     --header "authorization: Bearer $your_platform_api_key" \    
     --data ''

and then, as every storage type has storage CLASSES, you can get them this way:

curl --request GET \  
     --url  "$your_platform_url/api/storage/{your storage type id}/classes" \  
     --header "Content-Type: application/json" \  
     --header "authorization: Bearer $your_platform_api_key" \  
     --data ''

Get the storage class in which a file is located

curl --request GET  
     --url "$your_platform_url/api/file/{your_file_id}" \     
     --header "Content-Type: application/json" \     
     --header "authorization: Bearer $your_platform_api_key" \     
     --data '{}'

Where,

{your file id} is the id of the file that you are querying for.

And them, look for the storage_class section:

Get the scheduled or running storage class transfer processes

curl --request GET  
     --url "$your_platform_url/api/container/{your_container_id}/storage_class/migrations" \     
     --header "Content-Type: application/json" \     
     --header "authorization: Bearer $your_platform_api_key" \     
     --data '{"conditions" : [{"status" : "{your_status}"}]}'

Where,

{your container id} is the id of the container in which your files are.
{your_status} is the status you are interested in, and it must be:
- PENDING: Scheduled to be executed or waiting for available resources
- RUNNING: Transfer process has been initiated
- SUCCESS: Transfer completed successfully
- ERROR: Transfer completed unsuccessfully.

Request a transfer from one storage class to another for a file

Execute the following method:

curl --request POST  
     --url "$your_platform_url/api/container/{your_container_id}/files/storage_class" \     
     --header "Content-Type: application/json" \     
     --header "authorization: Bearer $your_platform_api_key" \     
     --data '{
              "files_id": [{id_of_the_file/s_to_transfer}],
              "storage_class_id": {destination_class_id}     
             }'

Where,

{your container id} is the id of the container in which your files are located.
{id of the file to transfer} is the id of the file to apply the transfer process to. You can include up to 1,000 file ids in the same request, separated by commas.
- All files must belong to the same container indicated in the previous parameter.
- If you request to change the class for a folder, the migration for every file inside it is also requested automatically by the platform.
{destination class id} is the id of the destination storage class.

If the storage type for your container supports file versioning, every previous version of a file is removed in advance to the storage transition, leaving only the last/most recent one. If you want to preserve any of them, you need to manually retrieve them in advance to any migration.

E.g. if you have an on-line stored file with three versions (the last one plus two previous modifications) and you transition it to a cold storage, only the most recent version is moved to the cold storage and every previous version is discarded, not being possible to recover them.

PreviousFunctions NextJupyter Notebooks

Last updated 3 years ago