CatDV2XenData User Manual

CatDV-XenData User Manual


Introduction

This document is considered in draft format for review only as of 5/27/2016

Welcome to the North Shore Automation CatDV2XenData user and administration guide. This document outlines the functions, usage, and basic technical information of the application and its related systems. While we advise that all users read the entire manual, system operators may only need to view the first few chapters. A system administrator or installer will want to review the document in its entirety.

In a nutshell, the CatDV2XenData application enables CatDV DAM systems to connect to XenData archive systems. By using the CatDV2XenData solution, CatDV can communicate with an XenData system to archive and restore assets, then record archive-related metadata in the CatDV database.

This document will cover the following topics:

  • CatDV2XenData user operation from the CatDV client application

  • CatDV2XenData metadata fields

  • Archive system and CatDV2XenData application overview

  • Basic troubleshooting

Document Conventions and Assumptions

This document assumes knowledge of the CatDV and XenData software. If you need more detailed information on CatDV Pro or XenData, please refer to their respective documentation.

In some cases (typically the overview sections), the term "archive action" may be used generically to refer to any archive related operation such as archive to tape, restore, or asset deletion. When a specific type of operation is referred to, such as a restore, it will be referred to by name.

These systems are highly customized in most cases, so all references to topology and system configuration are examples. Consider undertaking a system design with your CatDV and XenData integrator to determine the correct configuration for your organization.

Chapter 1 - User Operations

This section will demonstrate how to initiate archive operations from the CatDV Pro interface.

All actions here are demonstrated on a single asset. However users may select as many assets as they choose and send all of them to a workflow at once. The system will queue the jobs and perform them sequentially. (Note that exceptionally large jobs may take many hours or days to complete, so be aware of this in large multi-user operations to ensure that multiple operators are not simultaneously submitting jobs and creating very large queues.)

If you require additional information about using CatDV Pro, please refer to CatDV's user documentation or the Squarebox website.

This chapter is included first in the manual for ease of access for users who want a refresher on how to perform basic archive actions and as a basic “quick start” guide. If you require deeper information or a more general overview of the system, you will find that data beginning with Chapter 2.

In most installations, the CatDV2XenData Archive App is accessed through the CatDV "XenData" details panel (as shown below). The specific fields used to manage and track archive jobs are defined and described in Chapter 2 "CatDV Metadata." Your system may be configured differently, but the field contents and functions will be comparable to what is documented in this chapter.

Note: XenData is abbreviated "XD" in all displays to keep label size down.

Archiving An Asset

Select an asset in the Main window of CatDV Pro.

Then, in the XenData Details Panel, select the "Send to Archive" value in the XD Archive Action pulldown.

Once you have selected the job type you want from the list, publish the selection to the CatDV Server ("Command-S" or "Control-S").

Note: You can include multiple records in an archive, restore, or delete job. Select multiple records in CatDV's "Main Window", then select the desired action in the "Details Panel".

Note: The XD Archive Action Field has four options you can choose, corresponding to the four XenData job types:

  • "Send to Archive" initiates an archive job for a specific record

  • "Update Metadata" initiates a metadata update job for a specific record

  • "Restore from Archive" initiates a restore job for a specific record

  • "Delete from Archive" initiates a delete job for a specific record

Viewing Archive Status

After submitting an asset/s for archive, if you refresh your CatDV server connection (in Server Menu or f5 key) you will see status updates on the selected record/s. Typically job submission times may vary depending on how your system is configured, but normally jobs are sent to XenData in 5 minutes or less.

While XenData processes an archive job, the CatDV Worker Node will update and reset the "Archive Action" field to blank.

Once a job is successfully submitted to XenData, the XD Job ID field is updated with a job ID.

Troubleshooting Tip: If  no Job ID is present after more than 5 minutes (and the user has confirmed a save and/or refresh in CatDV), this might indicate a problem with the archive operation.If the Archive Action field does not reset to blank (it still says "Send-to-Archive"), that may indicate that the Worker Node has failed to submit the job. If the Archive Action and the XD Job ID are both blank, it would indicate that Worker Node submitted the job, but XenData did not receive the data or there is a problem with the XenData system.


Indication Send to Archive Completed Successfully

Once XenData successfully completes an archive job, the app will update the CatDV record(s) with:

  • A unique XenData reference number in the "XenData Job ID" field

  • A summary of the archive job in the "XenData Summary" field

  • "Archived" in the "XenData Status" field

  • A detailed log for the job in the "XenData Log" field

Note: To see more information on the job, please refer to the XenData Administration app on the XenData server.

If there are errors in the operation, the XD Monitor Status will display "ERROR" and a brief section of the log will be written to the XD Log. Depending on the error displayed you may report the problem to your administrator or begin troubleshooting the appropriate system or workflow step.

Selecting a File Group for Send-to-XenData (optional)

As an optional step, you may want specific records to be archived to a specific XenData file group. Multiple files groups may be configured on a XenData system, each with different archive policies. For example one file group may be configured on the XenData system to store content to a non-replicated set of LTO cartridges and a second file group may be configured to write to a duplicated set of LTO cartridges. You can specify which file group to use before initiating an archive job by selecting a specific file group in the "XenData File Group" field.

Once you have selected the job type you want from the list, publish the selection to the CatDV Server ("Command-S" or "Control-S").

Note: This functionality requires your administrator to set up a list of XenData file groups in CatDV and a corresponding set of file group rules on the XenData system during the installation.

This image shows an archive job ready for submission to a File Group named "CatDV".

Restore From XenData

To initiate a restore, select Restore-From-XenData from the Archive Action menu and publish your changes. Please confirm was previously archived and that the asset is not online in the original path on storage. If the asset is online, and you still need to restore a copy, please move or temporarily rename the online copy or use an alternate restore path for this operation.

Once you have selected the job type you want from the list, publish the selection to the CatDV Server ("Command-S" or "Control-S").

Once again, using the Server Menu or f5 key will allow you to refresh your CatDV window and view restore status changes.

Note: CatDV records available for restore jobs will display the relevant XenData archive information in the XenData Job ID, XenData Job State, and XenData Log fields. If these fields are blank, it's likely that the asset has not been archived to the XenData system.

Because the CatDV2XenData application uses the XenData Workflow API as opposed to a drag and drop or copy workflow assets not archived using the CatDV2XenData application cannot be restored using CatDV Pro. To restore these assets, please use the XenData cache

Optional: Select alternate restore location

When initiating a restore you have two options

  1. You may leave the location blank and the asset will be restored to its original location/file path.

  2. If your system administrator has set up specific alternate restore locations, you  may choose a predefined location from the ”XD Alt Restore Path” field, the asset will be restored to the designated location. You can specify which location to use before initiating a restore job in the "Restore Location" field, shown below. This will restore the file into the specified location preserving the original file path from root.

Note: If no location is specified in this field, the app directs XenData to restore to the location in the "Media Path" field for the record(s). This should be the original location of the asset on the storage. If that location does not exist (for example if that volume is not accessible to the XenData Server), the restore job will fail and an ERROR message will be displayed for the asset.

Restore in Progress

Indication that Restore has Completed

Upon completion of the restore operation, the XD Archive Action field will be reset to blank, the XD Log will display date and time of the restore, and the XD Monitor Status will be set to "Complete."

Delete From XenData

While assets cannot be truly “deleted” from archive media, due to their linear nature, without rewriting the entire tape or optical cartridge, XenData will support “hiding” assets, when a delete operation is requested. This is most commonly desired when an existing asset is modified or a new version of an asset is made to fix a problem with the media. In these cases the name of the file and its path will be the same as the old version and this will prevent it from being archived to the XenData system. By first deleting the file from the archive, you are able to archive the new version.

Note: This behavior is different than the traditional XenData behavior and follows the architecture of CatDV where an asset is identified by path, including file name and the CatDV Clip ID. For more information on the differences between these workflows, please contact XenData.

Before initiating a deletion job, the user will have to check the "Confirm Delete" box. While XenData processes a delete job, the app will update the CatDV record/s with "Restore-from-XenData" in the "Archive Action" field.

Note: CatDV records for delete jobs will display the relevant XenData archive information in the XenData Job ID, XenData History, and XenData Log fields.

Indication of Delete-from-XenData completed

Once XenData successfully completes an archive job, the app will update the CatDV record/s with:

  • A summary of the delete job in the "XD Log" field, along with previous archive jobs

  • "Deleted" in the "XenData Status" field

  • "Delete Job Sent" in the "XenData Log" field

Note: Any information about previous archive jobs in the "XenData Log" field will be overwritten by a delete job as this field only displays the log data for the last action. To see more information on all jobs, please refer to the XenData Administration app on the XenData server. However, the "XenData Job ID" field will show the job ID for the last completed archive prior to the deletion.


#IMAGE MISSING


Chapter 2 - CatDV Metadata

This section lists all the metadata fields in CatDV that are used and also gives a description of them.

The XenData metadata fields are created in the CatDV metadata during the system deployment and can be mapped to a details panel in CatDV. We recommend collecting these fields into a dedicated CatDV details pane for clarity.

In this example, the user has created a details panel called "XenData" in which the XenData and CatDV2XenData metadata are displayed.

Note: In the "Main View" of the CatDV window, you can set up a column display to view the XenData Status for every clip.

#IMAGE MISSING

This view is very useful in avoiding one of the most common sources of user errors and support tickets. Sending an asset to an archive more than once. If a previously archived asset is sent to archive a second time, it will be refused by the archive system. Additionally, this view will allow you to avoid trying to restore assets that have not been previously archived. These are very common sources of user errors and support tickets.  

The metadata fields for the CatDV2XenData workflow are named in a way that aids in tracking status and troubleshooting the archive workflow.

The data in fields labeled "XD" are typically coming from the XenData server via API. All other fields are either updated by user input or by a Worker Node action (e.g., Archive Action, XD File Group. Restore Location).

Note: There are two exceptions to the “XD” naming convention. “XD File Group” and “XD Delete Confirm” are user input fields that are set by the user, as needed.

CatDV2XenData Metadata Fields

File Name - Built-in CatDV field that lists the selected file's name

Media Path - Built-in CatDV field that lists the selected file's path. This will also be the asset’s original file path, stored in the archive, and used for restore operations.

Clip ID - Built-in CatDV field that lists the selected file's Clip ID (GUID)

XD Job ID - Returned XenData Job ID

XD Tape IDs - Comma separated list of tape IDs this asset exists on.

Note: The XD Tape ID for an asset may not be immediately available in the CatDV interface. For more information on this please refer to the introduction to Chapter 3. Retrieving the XD Tape ID is discussed in detail in this section.

XD Archive Action - CatDV User field watched by the Worker Node in order to start the job enqueue/archive process.  After the Worker Node action runs and the archive action has been submitted to XenData, this value will be set to blank (a null value).

Listed here are the possible values for the XD Archive Action field:

  • Send to Archive

  • Restore from Archive

  • Update Metadata

  • Delete from Archive

XD File Group - Archive destination in the XenData system. For more information on these groups or how they are configured, please refer to the XenData manual.

XD Confirm Delete - Required additional safety check to make sure ‘Delete from Archive’ actions only delete files you want them to delete.

XD Job State - The status of a job as reported directly by XenData

Listed here are the possible values for the XD Job State field:

  • In Progress

  • Waiting

  • Complete

  • Fault

  • etc

XD Monitor State - State as determined by monitoring code, which may include special states such as “Swap Cartridge Needed”.

Listed here are the possible values for the XD Monitor State field:

  • Complete

  • Fault

  • In Progress

  • etc


XD Alt Restore Path - Allows restoring to an alternative path while keeping the archive directory structure by adding this prefix.

Example: If your administrator has designated an alternate restore path, such as:

\\unc-server\sharepoint\new\restore\location

And an asset has an original file path of:

/Volumes/SAN/old/file/location/file.mov

Then the asset will be restored to:

\\unc-server\sharepoint\new\restore\location\Volumes\SAN\old\file\location\file.mov

XD Bytes Processed - The number of bytes XenData has processed.

XD Bytes Total - The number of total bytes XenData will process.

XD Files Processed - The number of files XenData has processed.

XD Total Files - The total number of files XenData will process. Note: This will be 1 for normal clips and potentially multiple for metaclips.

XD Percent Processed - A report of percent complete of a XenData job.

XD Monitor Flag - A flag to trigger the CatDV2XenData monitoring process to monitor this asset’s status because of a job in progress. Note: This field is hidden from users and is only for use by background processes.

XD Log - A log of actions on this asset by the archive, restore, and update operations of CatDV2XenData. Failures will be documented in more detail here.


CatDV2XenData Metadata Fields


Archive Action - showing different states of the files archive process.



XenData Queue Details Panel



This completes the description of the CatDV metadata fields used by the CatDV2XenData and XenData applications. If you have any questions, please contact your systems administrator or North Shore Automations.


Chapter 3 - CatDV2XenData App


This section provides a functional overview of CatDV2XenData and describes the application and basic system topology. The chapter includes a block diagram and step-by-step workflow overview describing not only the steps visible to the user but the back-end processes that are at work in archive operations.

The CatDV2XenData app is middleware, managing the information exchange between CatDV and XenData. The app delivers archive jobs, as metadata and commands, to XenData and delivers the responses from XenData back to the CatDV Server. CatDV2XenData does not directly move or manipulate media or files. All file archive actions are performed by XenData, and all metadata is stored by CatDV.

One important thing to note is that using XenData with CatDV2XenData presents a significant change in the archive workflow. Traditionally assets are archived to a XenData system by copying them to the XenData server’s cache. Restores are performed by accessing the stub files on the cache and then dragging them to their original location (or an alternate location) on the user’s storage.

When using the CatDV2XenData workflow to send to archive, the asset’s file path is delivered to the XenData Workflow API from CatDV and the XenData system then typically pulls the file into its archive cache and then moves it to a group of data tape or optical cartridges . Restores operate similarly, where the file is pushed back to its original file path (or an alternate path) by the XenData Workflow API

Another important, and frequently asked question, is how to access the Tape ID for an asset via CatDV. When archiving an asset the XenData system caches the file to disk and then notifies the CatDV2XenData app once this caching operation has completed. It then writes the cached data to tape or optical cartridges and stores the Tape ID in its database. However, this Tape ID is not reported back to CatDV2XenData by default. Currently an “Update Status” operation must be run from CatDV in order to collect the Tape ID from XenData.

Application Topology

This drawing shows a sample topology with an CatDV Pro Client, a CatDV Server, with a CatDV Worker Node system running both Worker Node and the CatDV2XenData software and a XenData system consisting of a server connected to an LTO Library.

Contact your integrator or system administrator or refer to your system deployment documentation for info on your specific configuration.

Workflow Overview

The following section describes the data interchange in the system. Only the archive action is described for brevity, but the other workflows (restore etc.) employ a similar methodology.

The order of operations in the workflow is as follows. (See the drawing below for a visual representation of these steps.)


Archive from CatDV to XenData

  1. A user flags assets in CatDV to send to XenData archive by setting a metadata value in a field to “send-to-archive” (or similar)

  1. May also be triggered from Worker action based on metadata received from another workflow (e.g., an ingest operation may auto-set the archive metadata as part of the workflow)

  2. If XenData has multiple File Groups, the user can select a specific File Group from a pull-down menu

    1. If the File Group is not set by the user prior to submitting the Archive request, the default File Group specified in the system configuration will be used

    2. Each file can only be sent to a single File Group

  1. The CatDV Worker Node is triggered by a server query to send the asset to the CatDV2XenData app via a Worker Node command

  2. Assets are “queued” for archiving by the CatDV2XenData app

  3. Every five minutes queued assets are submitted to the XenData archive system as individual jobs by CatDV2XenData

  4. The XenData Server is queried for asset status

  1. If asset(s) are already in XenData, then:

    1. This info will be added to the XD Log Field in CatDV

    2. The XD Monitor State field will be updated to show “archived” (or similar)

    3. The asset(s) will not be sent to XenData

Note: In a XenData system using the Workflow API, if a job is submitted for an asset that exists in the system, XenData will fail the job and report an error. CatDV2XenData is designed to check and catch duplicate archive jobs to prevent this from happening in the case of a user submitting an asset to an archive more than one time.

  1. If asset(s) are not in XenData, then the asset(s) are sent to XenData

  2. A unique Job ID is created for the asset by XenData

  3. The Job ID is written to the XD Job ID field in CatDV

    1. Example: 503BBCDE-56F1EB0B-00-00000012

Note: This ID may refer to a single media file or to the elements of a metaclip. Metaclips in CatDV have a single XD Job ID.

  1. The archive status is updated in CatDV for each asset as its job progresses

  1. The XD Monitor Status field in CatDV lists the status of each submitted asset

    1. Example: “Complete”, “In-Progress” (or similar)

  1. If the archive action is successful, the asset’s XD Log field is updated with job information, and the operation is complete.

Log Example: 2016-03-22T17:28:39.803021 - Archive - Archive of asset 5177FE04 complete

2016-03-22T17:45:54.185589 - Restore - Asset 5177FE04 - Restore mode - Fault (Code:80): The file exists

2016-03-22T19:07:56.136539 - Restore - Restore of asset 5177FE04 complete

  1. If an archive job fails, the XD monitor State field will display a “Fault” message. The CatDV XD Log Field will list more detailed error information.

  1. Operation complete


Chapter 4 - Worker Node

The CatDV Worker Node application is used to initiate archive actions. When the system is integrated, three Worker Node actions are set up:

  • Queue Archive to XenData

  • Queue Restore from XenData

  • Queue Delete from XenData

Worker Node Actions

See the Appendix for example screenshots of these actions.

Note that the Worker Node does not actually perform the archive operations, nor communicate directly with the XenData server. The Worker Node merely serves as a bridge from the CatDV Server to the CatDV2XenData application to queue an action. Once queued, all further actions are performed by CatDV2XenData or the XenData application. This leaves the Worker Node free to perform other automations while archive operations are running.

The CatDV2XenData application uses XenData's API and CatDV Server's REST API to pass information between the systems. The Worker Node CLI is not used in this workflow.

See the Worker Node Read Me doc for more information on managing and troubleshooting Worker Node.

This concludes the summary of the CatDV Worker Node and its roles in CatDV and XenData archiving operations.

For more detailed information on your CatDV system, please contact Square Box Systems or use the online support resources located at:

http://www.squarebox.com/support/


.



Chapter 5 - XenData


Xendata System Configuration Application

While a full discussion of XenData administration is beyond the scope of this document, we have included a quick overview of the main screens that a user might access while troubleshooting a system.

The XenData System Configuration application provides a way to configure the XenData system and view the system status. If you are seeing XenData errors reported in your asset records, access the XenData server either physically, or via remote access to check the system status, storage mounts, and error logs.

Note: Be careful to not open the Windows Explorer window for the XenData cache while the archiving operations are in progress. This may cause the archive operations to fail.

To access the XenData System Configuration application, launch it via the shortcut on the desktop or from the Start Menu as shown below.

Xendata System Configuration Interface

This is the default screen for the Xendata System Configuration Application. If you do not see the full list in the sidebar, click the folder icon to expand the list.

File Group Configuration

This screen shows the available XenData File Groups. This example shows a File Group named "CATDV" highlighted in the sidebar that is located on the E: drive. The example is set to use the CATDV Volume Set.

Volume Set Configuration

This screen shows the available XenData Volume Sets/Groups. The CATDV Volume Set is highlighted in the sidebar for viewing or editing.

LTO Tape Info

This screen shows the information for a specific LTO tape in the CatDV Volume Set.

XenData System Log

This screen shows the available XenData System Log. Note that it is accessed by clicking the Diagnostics>System icons in the sidebar.

Click the "System" icon to monitor the XenData processes in a log window. This will show the XenData system logs for the LTO hardware and data transfers.

XenData Drive Configuration

This screen shows the XenData Library configuration. This information is accessed by clicking the Diagnostics>Library icons in the sidebar.

Note: Your system configuration may vary.

Drive 0 Info

This screen shows the XenData Drive information. This  is accessed by clicking the Diagnostics>Library>Drive icons in the sidebar.

Note: Your system configuration may vary. This system has two LTO drives installed in a tape library.

Drive 1 Info

This screen shows the XenData Drive information. This  is accessed by clicking the Diagnostics>Library>Drive icons in the sidebar.

Note: Your system configuration may vary. This system has two LTO drives installed in a tape library.


This concludes our description of the Xendata System Configuration and the screens that may be most useful in troubleshooting CatDV and XenData archiving operations.

For more detailed information on your XenData system, please contact XenData directly or use the online support resources located at:

http://www.xendata.com/support/index.html


Appendix

Appendix A - CatDV Worker Node Configuration Examples


The following screenshots are included both for reference and educational purposes. Note that your system configuration may vary. If you have any questions about your system configuration contact your system administrator or North Shore Automation support.

They are in order, tab by tab, left to right (in the Worker Node interface) so that you may reference them as you confirm your Worker Node settings.


Trigger Tab

Server Query Tab

Processing Steps Tab

Processing Step: enqueue.exe

This should be set and correct, but you may hit the "Choose" button to view the path.

Processing Step: enqueue.exe

Post Processing Tab


End


Appendix B - System Requirements

  • CatDV Server

    • Confirm the latest version of CatDV Server 6.10.x or 7.1.x is installed on your CatDV Server

    • Confirm a CatDV Web Client license is available on the CatDV Server

    • Confirm and Supply CatDV Administrative login credentials.

    • Confirm and supply test media is available on the Media Storage and catalogued and clearly marked/labeled in CatDV Server

  • XenData Server

    • XenData Server Software version 6.20 build 2366 or later is installed

    • Confirm Full or Restricted CatDV Worker Node version 6.1.x or higher is installed on the XenData Server and licenses are applied or available.

    • Confirm that storage access to the test material that is catalogued in CatDV is available from the XenData Server

      • Confirm the volume is mounted on XenData Server

    • Confirm UNC access to the storage device/media catalogued in CatDV from the XenData Server

      • This must include UNC Login Credentials
  • Network Access

    • Confirm network access to the CatDV Server Web Client from the XenData Server for API communication is available