Create a manifest before transferring files
Last updated
Last updated
While the file transfer processes when getting content ingested in the platform usually include some degree of integrity verifications, it is considered a good practice (and we always recommend it here at LIBNOVA) to generate a manifest of the content to transfer, transfer it to the platform and then verify, platform-side, that the content in the platform is exactly the same as the content in the source of the transfer.
This process decreases the corruption and data loss risk, with the tradeoff of the additional time and effort to generate the manifest in advance to the file transfer.
Make sure that the tool you chose is safe to use, safe to be used in your environment, and it is approved that you use it.
LIBNOVA has no relation with the creators/maintainers and we can't ensure that the tool will work as expected, will fit its purpose, will cause no damage or will not contain malware.
LIBNOVA can't accept any consequences on you using 3rd party software, and using them is a Customer responsibility.
There are multiple tools to create an integrity manifest in Windows, and any of them will work as long as it generates the manifest in the MD5SUM format, but there are two open source tools that are easy to use and generate manifests fast: HashCheck and OpenHashTab.
Both are shell extension that allows users to create manifests, instead of stand alone applications, but our preferred one is HashCheck (even if it is older than OpenHashTab), so the process is explained in detail for it:
1. Go to https://github.com/gurnec/HashCheck, download and install the tool.
2. Open the folder that contains the content you want to transfer. Right click and select the "Create Checksum File..."
3. The tool will request for you to chose the name of the manifest to generate. Save it outside the folder you are creating the manifest for, and make sure you select the MD5 type of file:
4. The tool will create the manifest file (this may take a long time, depending on your volume of content, as every file needs to be analyzed in full in order to produce the manifest).
5. When the process is complete, open your text file and make sure the relative path for every file matches the relative path to the file you plan for it to have in Go, so relative paths match. In other words, the path to each file when the file is uploaded in the platform should match the actual relative path for the file.
6. Upload the file to the platform, select it and launch the "Validate MD5SUM manifest" function.
If the function does not show up in the Functions panel, please make a support request for the function to be enabled in your platform. Not all available functions are deployed to all instances.
To generate the MD5SUM manifest in Linux, you can use the md5sum command, that is available for many distributions.
Instructions on how to use it are available here.
The command processes your content sequentially, which can be slow, but you can achieve better results parallelizing your workload with a little command line Kung Fu, like in the following example:
1. Launch the command
2. The tool will create the manifest file (this may take a long time, depending on your volume of content, as every file needs to be analyzed in full in order to produce the manifest).
3. When the process is complete, open your text file and make sure the relative path for every file matches the relative path to the file you plan for it to have in Go, so relative paths match. In other words, the path to each file when the file is uploaded in the platform should match the actual relative path for the file.
4. Upload the file to the platform, select it and launch the "Validate MD5SUM manifest" function. If the function does not show up in the Functions panel, please make a support request for the function to be enabled in your platform.
If the function does not show up in the Functions panel, please make a support request for the function to be enabled in your platform. Not all available functions are deployed to all instances.
