Functions
Last updated
Last updated
LIBSAFE Go Functions let you run code in the platform, in response to certain events or triggers, without needing any client-side script. They are similar to Amazon AWS' Lambda functions.
Functions are useful when you want the platform to behave in a specific way, in response to external events or when you want to add your own code to be executed on demand by you.
With the LIBSAFE Go Functions, you just upload your code and add the triggers you would like to make it execute.
Users are able to define data container-level functions (LIBSAFE Go Functions) that are executed on certain events:
CRUD Functions for files and metadata: When files are created, read, updated or deleted.
E.g.: Every time you upload one of your photographs:
Extract the date from the file name to the metadata catalogue, so it is searchable using the web interface or the API,
Calculate each file integrity and
Tag images that contain faces
Periodic functions: Every minute, hour and day.
E.g. Webhooks to other systems
Executed-by-demand functions: When the user selects files using the GUI, or launched using the API
E.g.: Apply a massive rename for the selected items.
This guide focuses on the executed-by-demand functions (the ones users manually trigger using the Management Interface or the API). See Create functions to learn how to create your own functions.
A python library to make using the API easier and more convenient is available: library and its documentation. This library can be used in Functions, Jupyter Notebooks and in your own scripts.
Let's say you have the following use case: You would like to perform an upload of your bagits. Then, you would like to perform an integrity verification to maintain the custody/integrity chain (to detect uploading errors) and finally you would like to assign metadata to them.
There are multiple ways to achieve this behavior in LIBSAFE Go, but this use case is a perfect example of combining Workflows and Functions. LIBSAFE Go Functions can be triggered by container's workflow changes, so it would be relatively easy to implement a 6-step workflow:
Upload content: You would upload content to the container. Then, when your upload has finished, you would make a single API call (or use the Management Interface) to advance the container to the next status of the workflow. To do that, you can use the following API call:
You upload your metadata as a file, as part of your package, or as an Excel spreadsheet, for instance.
Waiting for ingestion to finish: A function will be triggered by the platform itself, that will wait for all files to be ingested. When this is completed, the function will move the container to the next status.
To know if a container is still ingesting content that you have uploaded, look for the files_pending_ingestion property for the container. If true, LIBSAFE Go is still processing content.
Integrity verification: A function will be triggered by the platform itself, that will launch a bagit verification process. If the process is positive (your bags are fine), it will move the container to the “Assign metadata” step. If not, to the “Validation errors detected”.
Validation errors detected: A function will send you an email, telling you the bags are not fine.
Assign metadata: A function will be triggered by the platform itself, that will assign all your metadata to your bags. Or, alternatively, it would wait for you to update it.
Archive data: A function will be triggered by the platform itself, that will launch the process to move all data to a cold storage.
You could achieve the same results using API calls client-side and managing this process in your code, but Functions deliver a more integrated approach, and allow other users to simply upload data without needing to perform scripting. As an additional benefit, you have integrated logging for the whole process, and your code performs better (as it is executed server-side).
With this approach, you would be uploading the content and making a single API call at the end of the upload.
Locate the data container you would like to add metadata to using the Containers menu section or by searching. This guide assumes metadata is properly configured for the data container. See Configuration\Metadata for more details or Working with data containers to see how to create them.
Select Check-in in case you are not checked in the container, and you have the check-in/out enabled for the data container.
In the data container page, choose Explore content:
Select the file you would like to execute the function over and select the function you would like to launch in the sidebar:
It is possible to select multiple items using your mouse to click-and-drag a box around the files or folders that you want to select. You can also use Ctrl and Shift and use the Select all, Select none or Invert selection in the file browser top bar.
You can also filter the files in the folder you are looking at by file name, for instance, for selecting all XML or JPG files:
and then, launch your function from the side bar.
Some functions are going to execute immediately, while others may require several hours. To track the function progress, you can use the link provided in the confirmation window that LIBSAFE Go shows when you launch the function:
You can also go to the Container and select Functions, to see the ones in execution and their outcome:
Some functions will process and change your content, while others may create new files in the container. When a function produces a new report or file, it is shown as an Asset when you open the function execution details page:
API examples here are just illustrative. Check the LIBSAFE Go API documentation for additional information and all available methods.
Sign in to the LIBSAFE Go Management Interface
Obtain your API key by selecting your name and then Access Methods.
To execute a Function you need to know the Function ID and the container or file IDs you want to execute it over (or apply it to). You can get all Functions that are loaded in the LIBSAFE Go platform using the following method:
Some Functions receive parameters. To call them, use the following method:
For instance, this Function requests the file IDs and the path as parameters:
For small Functions that are immediately completed, LIBSAFE Go will answer with a Success/error code, but for more complex Functions, LIBSAFE Go will create a job to execute them, so you can track the execution progress.
When the Function is launched, LIBSAFE Go will provide the job id in the response:
Then, you can do three things:
Some Functions may take hours to complete. You can use the /job API endpoint with the job_id returned in the previous method to monitor its progress:
For each job, you can get its log using the /job/{job id}/messages:
And finally, some Functions could create new assets (files). For example, your Function could produce a report in a PDF file, or if the Function is to compress data, it will create a ZIP file.
You can list the assets a job has created with the /job/{job id}/assets method:
And download the asset like any other file with the file_id, using the API or any other available download method (remember to include the "-L" in your call):
We have created a Python library that simplifies many actions and makes your programming easier when creating a function.
A python library to make using the API easier and more convenient is available: library and its documentation. This library can be used in Functions, Jupyter Notebooks and in your own scripts.
For example, lets say you would like to create a function that hashes your files with a new algorithm you would like to use.
First, you should initialize your function, loading the LIBNOVA LIBSAFE Go libraries:
If your code is going to be called from a LIBSAFE Go Function, you will receive some parameters from LIBSAFE Go every time your function is called. This variable is initialized in the following way:
Depending on the function type, the structure you receive can change, but usually you can find the following:
Every function executes in relation to an (Execution) Job, that is really useful for logging the execution progress. You should initialize it with:
And you can log to it using:
The JobMessage.JobMessageType defines the type of message. You can see a list of the available types in the method documentation.
And then, you would usually have your payload. In this example:
And finally, we must let LIBSAFE Go know that our function has finished, with the result status:
The full code sample:
