NSA QSM-Connect CatDV Quantum Archive App

This document is considered in draft format for review only as of 05/27/2022

Introduction

Welcome to the North Shore Automation QSM-Connect user and administration guide. This document outlines the functions, usage, and basic technical information of the application and its related systems. While we advise that all users read the entire manual, system operators may only need to view the first few chapters. A system administrator or installer will want to review the document in its entirety.

The QSM-Connect application enables CatDV DAM systems to connect to Quantum Storage Manager archive systems. By using the QSM-Connect solution, CatDV can communicate with a Quantum Storage Manager system to archive and restore assets, then record archive-related metadata in the CatDV database.

This document will cover the following topics:

Document Conventions and Assumptions

This document assumes knowledge of CatDV and Quantum Storage Manager software. If you need more detailed information on CatDV Pro or Quantum Storage Manager, please refer to their respective documentation.

In some cases (typically the overview sections), the term "archive action" may be used generically to refer to any archive related operation such as archive to tape, restore, or asset deletion. When a specific type of operation is referred to, such as a restore, it will be referred to by name.

These systems are highly customized in most cases, so all references to topology and system configuration are examples. Consider undertaking a system design with your CatDV and Quantum integrator to determine the correct configuration for your organization.

Chapter 1 - Workflow & User Operations

Key Concepts

The following is a simplified description of the workflow steps and applications performing them. For more detailed information see Chapter 3 for a full description of archive and restore operations. The application performing the operation for each step is in bold text.

For the purposes of this discussion we have defined the two main file locations as follows:

Archive Workflows

The QSM Connect application has two archive configuration options "Copy/Move to Archive" and "Archive in Place". The QSM Connect application can only run in one of the two configurations and the configuration should be determined prior to policy configuration on new StorNext deployments. If this is an existing StorNext deployment with existing archive policies, consulting will be required.

For current QSM Connect users/administrators, an easy way to determine which configuration is installed, check the "QSM Archive Action" field in CatDV.

If the values in the field consists of "Copy to Archive", "Move to Archive", "Restore from Archive", etc. then the "Copy/Move to Archive" configuration is installed.

If the values in the field consists of "Archive", "Restore", "Truncate", etc. then the "Archive in Place" configuration is installed.

The first option is referred to as the "Copy/Move to Archive" QSM Connect configuration. This configuration will move or copy a file from the active storage location to the archive policy. The archive policy folder or volume is commonly hidden from normal users to keep the archived stub/reference files separate from active storage.

Move To Archive - In this workflow the active file is moved to a relative path in the Archive file location in a StorNext policy. This removes the file from active storage, and allows archive and truncation increasing available space. See diagram below.

Note: If a move operation is performed from one volume to another volume (where the Archive Policy is located) this in effect is still a “copy” operation, though the active file may be deleted after the copy is complete.

Copy to archive - In this workflow the active file is copied to a relative path in the Archive file location in a StorNext policy. This does not remove the file from active storage and allows archive and truncation while keeping the original file in the Active file location where it is available for use. See diagram below, noting that the only visible difference between a move and copy operation is the removal of the original file.

Move To Archive Workflow Diagram:

Move To Archive (Dual Volume) Workflow Diagram:

The second option is referred to as "Archive in Place" QSM Connect configuration. In this configuration the active storage location is also the managed archive policy location and no file copies or moves are made. StorNext stub/reference files are always present to this location and requires additional software on client systems to show online and offline files in the policy.

Archive in place - In this workflow, the Active file location is actually the Archive file location. The policy is set on the original file folder. In this case QSM archives the file in its current location without a move or copy. Upon archive and truncation the representative “stub” file is created and is managed by StorNext and QSM for restore and other operations.

Archive In Place Workflow Diagram:

Chapter 2 - CatDV Metadata

QSM-Connect Metadata CatDV Fields & Panels

QSM Archive Panel

Name - Built-in CatDV field that lists the selected file's name (NM1 clip.name).

Media Path - Built-in CatDV field that lists the selected file's path. This will also be the asset’s original file path, stored in the archive, and used for restore operations. (media.file.Path).

Clip Ref - Built-in CatDV field that lists the selected file's Clip ID (GUID) - (CREF clip.clipref).

QSM Archive Action - CatDV User field watched by the Worker Node in order to start the archive process.  After the Worker Node action stages the file in the policy and the QSM Connect Worker plugin archive action submits to the Quantum Storage Manager for archive. This value will be updated by the QSM Connect app during the archive and then finally reset to blank (a null value) when the archive command is submitted to Storage Manager.

Copy/Move to Archive values for the QSM Archive Action field:

Archive values for the QSM Archive Action field:

QSM Archive Status - Simple status from NSA app denoting whether or not an asset is in the Quantum archive or if it is simply on disk and has not yet transitioned to the archive (cloud or tape.)

Possible values for the QSM Archive Action field:

QSM Location - More detailed Quantum archive status denoting what storage types the asset exists upon. e.g.  is this on tape, disk, or both? This is reported by Quantum archive system

Possible values for the QSM Archive Action field:

QSM Tape IDs - List of tape IDs this asset exists on.

New line for each copy / location

QSM Disk Location - (storedPathFileName) - Location of file on the StorNext system when it was archived from a policy folder This contrasts with the “Media Path” from CatDV which lists the path of the asset in its original “tier 1” location before being moved to the policy folder by Worker Node for archive.

QSM Archive History - List of operations performed by NSA application

Ex.

QSM Admin Panel

Name - Built-in CatDV field that lists the selected file's name (NM1 clip.name).

Media Path - Built-in CatDV field that lists the selected file's path. This will also be the asset’s original file path, stored in the archive, and used for restore operations. (media.file.Path).

Clip Ref - Built-in CatDV field that lists the selected file's Clip ID - (CREF clip.clipref).

QSM Policy ID - Storage Manager policy name which the file was archived to.

QSM Copy Count - Number of copies of an asset stored in Quantum Archive. The Quantum Policy defines the number of copies.

QSM Last Job ID - Storage Manager Job ID for the last job performed.

QSM Last Job Type - Displays the last job type that was performed by NSA app and Storage Manager. This will not display “Update” because it’s not technically a “job” it’s just a metadata request.

Possible values for the QSM Last Job Status field:

QSM Last Job Status - Status of the last job performed

Possible values for the QSM Last Job Status field:

QSM Archive Source Path - CatDV Media Path at time of Archive request. Set by the Worker Node at time of copy or move to policy.

e.g. The original Media Path as it exists in CatDV when the archive action runs. This is stored as a safety, in case a user updates or edits the Media Path at any time. This is what related the archived asset to the original tier 1 file system location and should never be altered.

Note: In “Archive In Place” workflow, this will be the same as the CatDV Media Path.

QSM Archive Path - Relative path to Stornext policy location as mounted to the Worker Node. Used in cases where the Worker OS path differs from the StorNext OS path (Linux.)

Source path = Z:\Media\Footage\File.mov

Local Worker Node policy path = Z:\.archive_policy\Media\Footage\File.mov

StorNext = /stornext/volume/Media/Footage/File.mov

QSM Last Job Log - Log snippet from Quantum archive of the last job performed.

  

        Note: Not a full list of all jobs. Overwritten by each operation.

If your CatDV Panels do not look like the above, please note that your system may have been customized by your CatDV admin. All panels and views may be edited without affecting function of the QSM-Connect app. However we recommend that the above fields be visible to admins for monitoring and troubleshooting.

Chapter 3 - QSM-Connect Application

This section provides a functional overview of the North Shore CatDV QSM-Connect application.

The QSM-Connect application consists of three software elements. An initiation application installed on the Worker as a JS plugin, A Java CatDV Server plugin, installed on the CatDV Server, and the main application (deployed as a virtual machine appliance.) The NSA VM/App communicate metadata between the CatDV Server, Worker, and StorNext MDC. All file movement and archive actions are handled by the CatDV Worker Node or the StorNext Storage Manager.

The QSM-Connect VM does not require a dedicated network interface. When using Hyper-V or a Type 2 Hypervisor, bridged networking is preferred rather than using NAT to route traffic to the VM ports on the host.

The VM is configured by default with 2 vCPUs, 4GB RAM, and a 40GB root volume.

Block Diagram

(click to expand)

Troubleshooting

Storage Manager wsar_agent issues

The log below points to the Storage Manager "wsar_agent" service having crashed. 2021-05-17 08:57:18,315 ERROR com.quantum.rest.swx.FileResource - Unable to connect to the WSAR database com.quantum.rest.swx.WebServiceException: Connection refused (Connection refused) If the wsar_agent is down the API calls to Quantum go unanswered. It needs to be restarted via SSH on the StorNext server. Here are instructions below on how to do that.

Steps to restart wsar agent

Instructions for StorNext 6

1. Log into sudo rootsh 

2. Type: ps -ef | grep wsar_agent 

(see if wsar agent is running. Results will looks like below.) 

a. root 7797 1 1 Aug21 ? 01:06:15 wsar_agent 

b. root 10923 41432 0 20:09 pts/1 00:00:00 grep wsar_agent 

3. If wsar agent it down and there is not a "wsar_agent" process 

a. type: cd ~ 

4. Type: wsar_agent 

a. This will start wsar agent 

5. Type: ls -l /proc/ `ps --no-headers -o "%p" -C wsar_agent`/fd 

a. This will clean up sockets.

Instructions for StorNext 5

Go on the active MDC (in cvadmin, look for the * next to the file system names to indicate which one is the active server), and do 

a: # ps -ef | grep wsar_agent 

If it’s not running, I would suggest restarting it by running: 

# /usr/adic/bin/wsar_agent 

If it’s running, it may have run out of sockets. You can do: 

ls -l /proc/`ps --no-headers -o "%p" -C wsar_agent`/fd