Organize your content

Once you gather your content, it can easily and quickly become disorganized.‌

To be more efficient and prevent errors later on, organizations should decide how they will name and structure their content. This includes metadata, that will provide context to the content, so future users can also understand it.‌

Flexible Intake offers a great degree of flexibility on how information is organized when in the platform, and it is able to adapt to any kind of data structure. In Flexible Intake, organizations and users are in charge of defining their own data models for their content.

As Flexible Intake also supports multiple metadata schemas and policies for the content, it is also possible to accommodate content coming from multiple sources and disciplines, and to make it usable by organizations that may be complex and large, or simple and small.

A good place to start is to develop a logical data structure that includes organising by type of processing and by data type. When doing so, you need to pay attention to the following topics:

• Permissions: The lowest level for permissions is at container level. It is not possible to define that, for two folders in a container, one should be accessed by a given user while the other should not. If you need to have different access permissions for two groups of content, you cannot have them in the same container.

• Storage policies: Everything you place in a container will have the same INITIAL default policy for storage. You can change it at file or folder level later, but the platform will assign one initially.

• Reports: You may want to have a report that groups only certain information, or you may want to use reporting to chargeback other platform tenants/departments or users. You may need to arrange your content in a particular way to be able to fulfill this need.

• Search: You can restrict a search down to a single container. Perhaps your search needs could be limited to a fraction of the information you have. It may be simpler and easier to just split your content in two containers than creating more complex searches.

• Metadata: All items in a container share the same metadata schema. It is not possible to have some items with one schema and other items with another schema in the same container. You may need to combine your schemas in one, or to have content with different schemas in two containers.

• Functions: The platform can execute functions (like macros or scripts) to organize, transform or analyse your content. Having (or not having) some functions available may impact your decision on how to arrange your information.

• Preference: Sometimes you are used to organize the information in a way, and it may be convenient to continue using it.

The tools you have in Flexible Intake to organize your information are:‌

• Archival node structure and the user/group permissions, and includes

  • Permissions

  • Policies

  • Reports and

  • Search

• Data containers share several commonalities with the archival nodes, including

  • Permissions

  • Policies

  • Reports

  • Search and

  • Metadata

• Folder and file names

• Items' metadata (including descriptive fields and tags)‌.

Archival nodes

‌Archival nodes contain other nodes or containers, and allow organizations to create a first level of organization for their data.

There are four factors to consider when deciding how to use them:

Permissions

Access permissions can be adjusted at node level. You can group your information in the archival nodes based on who should be able to access it.

Policies

Flexible Intake allows you to define container templates, that can be used to define how the content is going to be managed in the containers created associated to them, including:

• The content source for the container: When creating a new container, Flexible Intake can copy the content from another data container into it. This is useful if you want to define a template or a pre-defined folder structure for it.

• Container metadata schema: The available set of fields to describe a container.

• Items metadata schema: The available set of fields to describe the items inside a container (files and folders).

• Workflow: The workflow to use to handle the container.

• Storage: The type and class of storage to use for the content that is placed in the container.

• Check-in/out policy: That defines if you want to allow multiple users to work simultaneously over a data container or not.

• Quota: The storage space you would like to assign to the container.

At the archival node level, you can define the templates that are going to be available to the users when creating new data containers in a given archival node. This way, if you want to enforce a certain policy for a certain group of experiments, you can create a node and associate a certain policy to it that enforces the use of certain metadata fields, for instance.

To define a container template, go to Configuration and then Data container templates. When the template has been created, you can associate it to an archival node in the Configuration menu under the Archival Structure option.

‌

Reports

‌Certain reports take into consideration the archival node the content is in, rather the individual data container. For instance, if you create a first level classification for each organization department, you can then launch a report that will tell you how much storage is every department using.

‌

Search

Search results can be limited to a certain node or set of nodes. This may be useful to you as a way to easily filter the content you are looking for.

‌

Data containers

Data containers are created inside an archival node, and are the next level of granularity you can use. You can use the same elements available for the archival nodes described in the preceding section.

You can create containers for each collection, type of object, year, etc., or you could have everything in the same container. This decision depends on how you plan to organize your data, but keep in mind that every item in the container is sharing:

• The same permissions: You cannot adjust permissions by folder or by file; only by archival node or container.

• The same metadata schema: Every file and folder inside a data container shares the same metadata schema.

‌When you are creating a container in a given archival node, Flexible Intake will show you the possibility to choose from a list of templates (outlined in the previous section) if they are configured for the node into which you are creating the container, or offer you a full range of settings to choose from.

‌

Folder and file names

‌Inside a Flexible Intake data container, you can create a hierarchical folder/subfolder structure like in traditional filesystems. This makes it possible to:

• Use folder/subfolder: Group files within folders so information on a particular topic is located in one place. Start with a limited number of folders for the broader topics, and then create more specific folders within those.

• Use a template: Flexible Intake allows you to start from a template when creating a new container. If you plan to have multiple projects with the same folder/file structure, create a template and use it as the starting point.

• Use existing naming conventions: If you are already using a certain approach in your organization, you can always use it.

• Name folders in a meaningful way: name folders after the areas of work to which they relate, and do not create folders for individuals where each one will have its own way to organize the information. This makes the file system easier to navigate for new people joining the workspace, and makes locating the information easier for them.

• Keep naming consistency: When developing a naming scheme for your folders it is important that, once you have decided on a method, you stick to it.

• Structure folders hierarchically: Create a limited number of folders for the broader areas or concepts, and then create more specific folders inside them.

• Separate ongoing and final versioned work: As you start to create lots of folders and files, it is a good idea to start thinking about separating your older documents from those you are currently working on.

• Review your content: Assess your team naming and organization schema, and call their attention to the content that is not following the naming schema. It is a good opportunity to train your team and to revisit your naming schema for needed changes.

When using Flexible Intake, you are capable of searching by files, folders, etc., using any part of the file/file name. Check the Advanced API File Search to know how.

Item metadata

Files and folders can be associated with metadata in Flexible Intake. Object metadata can be used for searching and grouping content, independent of the container/folder the files are in.

For instance, you could create a metadata schema with three fields for your datasets:

• Title (string)

• Date (date/time)

• Author (string).

‌And then, you can easily query Flexible Intake for the objects that include a certain word in the title, or that were created by a given author.

‌If you are using the API or the management interface, take a look at the Search page. Remember that you can always use Jupyter Notebooks to extend Flexible Intake capabilities and to create your advanced search interfaces and custom reports.