CatDV-FlashNet User Manual

NSA CatDV-FlashNet User Manual

Introduction

This document is considered in draft format for review only. Updated 02/05/2018

Welcome to the North Shore Automation CatDV2FlashNet user and administration guide. This document outlines the functions, usage, and basic technical information of the application. While we advise that all users read the entire manual, system operators may only need to view the first few chapters. A system administrator or installer will want to review the document in its entirety.

In a nutshell, the CatDV2FlashNet application enables CatDV DAM systems to connect to SGL FlashNet archive systems. By using the CatDV2FlashNet solution, CatDV can communicate with a SGL FlashNet system to initiate and monitor archive and restore operations and have archive-related metadata recorded in CatDV Server.

This document will cover the following topics:

Document Conventions and Assumptions

This document assumes knowledge of the CatDV and FlashNet software. If you need more detailed information on CatDV Pro or FlashNet, please refer to their respective documentations.

In some cases in this document the term "archive action" may be used generically to refer to any archive related operation, such as archive to tape, restoring from tape, or asset deletion. When a specific type of operation is referred to, such as a restore, it will be referred to by name.

These systems are highly customized in most cases, so all references to topology and system configuration are considered examples. North Shore, Square Box and SGL assume that any CatDV2FlashNet integration will include a system design discussion with your CatDV and SGL integrator to determine the correct configuration for your organization.

Chapter 1: CatDV User Operation

This section is a step-by-step user guide on how to initiate archive operations using CatDV Pro Interface. The functions for archiving, restoration, and deletion are covered. Importantly, all actions here are demonstrated on a single asset; however, users may select as many assets as they choose and send all of them to a workflow at one time.

If you require additional information about using CatDV Pro, please refer to CatDV's user documentation or the Squarebox website.

The CatDV2FlashNet software can be installed on either an OS X or Windows Worker Node.

In most installations, the CatDV-to-FlashNet Archive App is accessed through the CatDV “SGL” details panel (as shown). The specific fields used to manage and track archive jobs are defined and described in Chapter ##2 "CatDV Metadata." Your system may be configured somewhat differently, but the metadata fields and functions will be the comparable to what is documented in this chapter.

Note: FlashNet may be abbreviated "FN" in field displays to keep label size down.

Archiving An Asset

To send an asset to the FlashNet archive system, perform the following actions.

Select an asset in the Main window of CatDV Pro.

Fig. 1.1- CatDV Overview

In the SGL FlashNet Details Panel, select the "Send-to-FlashNet" value in the Archive Action pulldown. Confirm that the asset/s are online and storage accessible to the FlashNet Server and that they have not been previously archived.

Once you have selected the job type you want from the list, publish the selection to CatDV ("⌘-S" or "Control-S").

Note: Multiple records can be included in an archive, restore, or delete job. Select multiple records in CatDV's "Main Window", then select the desired action in the "Details Panel".

Archive Action Field options

Note: The Archive Action Field has three options corresponding to the three FlashNet job types:

Checking operation status

If, after selecting an action to initiate, you refresh your CatDV server connection (in Server Menu or f5 key) you will see status updates on the selected record/s. Typically job submission times may vary depending on how your system is configured, but normally jobs are sent to FlashNet in 5 minutes or less.

While FlashNet processes an archive job, the CatDV Worker Node will update and reset the "Archive Action" field to blank.

Once a job is successfully submitted to FlashNet, the FN Job ID field is updated with a Job ID.

Troubleshooting Tip: If no Job ID is present after more than 5 minutes (and the user has confirmed a save and/or refresh in CatDV), this might indicate a problem with the archive operation. If the Archive Action field does not reset to blank (i.e., it still says "Send-to-Archive"), that may indicate that the Worker Node has failed to submit the job and you, or your administrator, should check the Worker Node. If the Archive Action and the FN Job ID are both blank, that might indicate that Worker Node submitted the job, but FlashNet did not receive the data or there is a problem with the FlashNet system.

Fig. 1.3 - Send-to-FlashNet

Operation completion notice 

Once FlashNet successfully completes an archive job, the app will update the CatDV record/s with:

A unique FlashNet reference number in the "FlashNet Job ID" field

A summary of the archive job in the "FlashNet Summary" field

A status of "Archived" in the "FlashNet Status" field

A detailed log for the job in the "FlashNet Log" field

Note: To see more detailed information on the FlashNet job, please refer to the FlashNet Administration app on the FlashNet server.

Fig. 1.4 - Operation complete

If there are errors in the operation, the FN Status will display "ERROR" and a brief section of the log will be written to the FN Log. Depending on the error displayed you may report the problem to your administrator or begin troubleshooting the appropriate system or workflow step.

Selecting a Volume Group for an archive (optional)

As an optional step, you may want specific records archived to a specific FlashNet volume/tape group. You can specify which volume group to use before initiating an archive job by selecting a specific volume group in the "FlashNet Volume" field.

Note: This functionality requires your administrator to set up a list of FlashNet volume groups in CatDV.

Fig. 1.5 - FlashNet Volume Group Selection

This image shows an archive job ready for submission to a "CATDV" Volume Group. (In this example the "%%" is a wildcard and will be replaced with a Volume Group ID Number. The Volume Groups are named CATDV_1 - CATDV_10 and this will select the next available ID. Your system may be configured with a single Volume ID and so the value might be something more like “2016_tapes.”)

Fig. 1.6 Send-to-FlashNet with Volume Group selected

Restore From FlashNet

In order to initiate a restore, select Restore-from-FlashNet from the Archive Action menu and publish your changes. Please confirm the asset was previously archived and is not online in the original path on storage. If the asset is online, and you still need to restore a copy, please move or temporarily rename the online copy or use an alternate restore path for this operation.

Note: CatDV records available for restore jobs will display the relevant FlashNet archive information in the FlashNet Job ID, FlashNet History, and FlashNet Log fields. If these fields are blank, it is likely the asset has not been archived to the FlashNet system.

Fig. 1.7 - Restore from FlashNet

Optional: Select alternate restore location

When initiating a restore you have two options:

Note: If no location is specified in the “Restore Location” field, the app CatDV2FlashNet app directs FlashNet to restore to the location listed in the "Media Path" field for the record/s. This will normally be the original location of the asset in storage. If that location does not exist (for example, if that volume is not accessible to the FlashNet Server), the restore job will fail and an ERROR message will be displayed for the asset.

Fig. 1.8 - Restore with Restore Location Set

Fig. 1.9 - Indication of Restore Completed

Upon completion of the restore operation the Archive Action field will be reset to blank, the FN Job History will display date and time of the restore, and the FN Status will be set to "RESTORED."

                                                    Fig. 1.10 Restore from FlashNet Completed Successfully

Delete-from-FlashNet #needs caption content

Delete From FlashNet

While assets cannot be truly “deleted” from an LTO tape due to their linear nature, FlashNet can set an asset to be “forgotten” on the tape so that it may be archived. This is most commonly desired when an asset is re-created or a new version is made to fix a problem with the media. In these cases the name of the file and its path will be the same as the old version, and this could prevent it from being archived to the FlashNet system.

Before initiating a delete job, the user will have to check the "Confirm Delete" box. While FlashNet processes a delete job, the app will update the CatDV record/s with "Restore-from-FlashNet" in the "Archive Action" field.

Note: CatDV records for delete jobs will display the relevant FlashNet archive information in the FlashNet Job ID, FlashNet History, and FlashNet Log fields.

Fig. 1.11 -Delete-from-FlashNet #needs caption content

Indication of Delete-from-FlashNet completed

Once FlashNet successfully completes an archive job, the app will update the CatDV record/s with:

# Note: Any information about previous archive jobs in the "FlashNet Log" field will be overwritten by a delete job as this field only displays the log data for the last action. To see more information on all jobs, please refer to the FlashNet Administration app on the FlashNet server. However, the "FlashNet Job ID" field will show the job ID for the last completed archive prior to the deletion.

Fig. 1.12 -Delete-from-FlashNet #needs caption content

##Section Addition - Add pre-flight rules for queue submission process

Chapter 2: CatDV Metadata

This section lists all the metadata fields in CatDV that are used and also gives a brief description of them.

The FlashNet metadata fields are created in the CatDV field set during the system deployment and can be mapped to a details panel in CatDV. We recommend collecting these fields into a dedicated CatDV details pane for clarity.

In this example, the user has created a details panel called "SGL FlashNet" in which the FlashNet and CatDV2FlashNet metadata are displayed.

The metadata fields for this workflow are named to aid in tracking status and troubleshooting the archive workflow.

The data in fields labeled "FN" is supplied from the SGL FlashNet server via API. All other fields are either updated by user input or by a Worker Node action (ex. Archive Action, FN Volume Group, Restore Location).

Note: There are two exceptions to the “FN” naming convention. FN Volume Group and FN Confirm Delete are user input fields that are set by the user, as needed.

CatDV2FlashNet Metadata Fields

Fig. 2.1 - FlashNet Metadata Fields as shown in CatDV

File Name - Built-in CatDV field that lists the selected file's name

Media Path - Built-in CatDV field that lists the selected file's path.

Clip ID - Built-in CatDV field that lists the selected file's Clip ID (GUID).

FN Job ID - CatDV User field, updated by FlashNet, that lists the FlashNet Job ID.

Archive Action - CatDV User field watched by the Worker Node in order to start the job enqueue process.  After the Worker Node action runs and the archive action has been submitted to FlashNet, this value will be set to blank (a null value).

Listed here are the possible values for the Archive Action field:

FN Volume Group - CatDV User field that indicates the FlashNet Volume Group on which the asset is stored.

FN Confirm Delete - CatDV User field, configured as a checkbox, which must be set in order to send a deletion request to the archive system. This helps prevent accidental deletions.

FN Job History - CatDV User field, updated by FlashNet, that lists the following information: Timestamp, Completion Status, and CatDV2FlashNet Job ID. This data is updated at the completion of an archive operation. New operations are appended to this field.

FN Status - CatDV User field, updated by FlashNet, that lists the assets archive status. This field will be updated dynamically during archive and restore operations.

When a Job is being monitored, if the FlashNet job status is ‘PASSED’, the FN Status will be ARCHIVED, RESTORED, or DELETED depending on the job type sent.

In the case of restoring, if the media is offline, the status will be OFFLINE and details will be provided in the log field.

Jobs that are in-progress will have a status of ‘Archive Job Sent’, ‘Restore Job Sent’, until the status updates with the current status.

A failed deletion status is ‘Failed Deletion Request’.

All other flashnet job statuses will be reported as:

<Job Type> - <Flashnet Status Passthrough>

For example:

ARCHIVE - Awaiting Cache Fill

Here are the possible values for the FlashNet Status Passthrough:

Restore Location - CatDV User field that lists an alternate location for an asset restore operation. If this field is left blank, the asset will be restored to its original file path.

Note: Please confirm that the original file location still exists before initiating a restore operation in order to prevent restore errors.

FN Log - CatDV User field, updated by FlashNet, that lists additional FlashNet job information. This is the first 10,000 characters of the FlashNet log for the archive operation by default. This can be changed by the administrator in the settings for the application.

## add mention and screenshot of “View” panel for archive

This completes a description of all of the metadata fields in CatDV that are used. If you have any questions, please contact your systems administrator or North Shore Automations.  

Chapter 3: CatDV2FlashNet App

This section provides a functional overview of CatDV2FlashNet and describes the application and basic system topology. The chapter includes a block diagram and step-by-step workflow overview describing the steps visible to the user and the back-end processes at work in archive operations.

The CatDV2FlashNet app is middleware, managing the information exchange between CatDV and FlashNet. In simple terms CatDV2FlashNet initiates archive jobs, via metadata and API calls, to a FlashNet Server and delivers the responses from FlashNet back to the CatDV Server. CatDV2FlashNet does not directly move or manipulate actual media or files. All file archive actions are performed by FlashNet, and all metadata is stored by CatDV.

Application Topology

This drawing shows a block diagram of the applications and systems involved in a typical CatDV2FlashNet system.

Actual system topologies vary widely, but a typical system deployment might consist of the following:  An edit system running CatDV Pro Client; a CatDV Server computer; a CatDV Worker Node system running Worker Node and the CatDV2FlashNet software; and a SGL FlashNet system consisting of a server connected to an LTO Library. All or most of these systems would typically be mounting a shared storage system, typically SAN or NAS.

Contact your integrator or system administrator or refer to your system deployment documentation for information on your specific configuration.

Fig. 3.1 - CatDV2FalshNet Data Flow Topology

Workflow Overview

The following section describes the manner in which data is interchanged through the system. Only the archive action is described for the sake of brevity, but the other workflows (restore, etc.) employ a similar methodology.

The order of operations in the workflow is as follows: (See the drawing above for a visual representation of these steps and data transfers.)

Archive an asset/s

Chapter 4: Worker Node

#In Progress

Chapter 5: SGL FlashNet

#In Progress

Chapter 6: Troubleshooting

Error Reporting - Send-to-FlashNet failure notification

When an archive job fails, the app will update the CatDV record/s with "ARCHIVE - FAILED" in the "Archive Action" field. It will also provide the job ID in the "FlashNet Job ID" field and a detailed log for the job in the "FlashNet Log" field.

Note: The FlashNet log field may contain additional information that can be viewed by using the scroll bar on the right of the field or increasing the size of the "Details Panel".

Failure Log Notification

Failure Log Details

North Shore Utility Scripts

North Shore provides a set of utility scripts to assist in troubleshooting and checking the status of the CatDV2FlashNet middleware. These are typically stored in a folder on the desktop of the Worker Node system running the CatDV2FlashNet middleware.

Fig. ## List of North Shore utilities

CatDV2FlashNet Utility Scripts

FlashNet Flush Queue - Clears all jobs from NSA app queue. Resets the job queue. Used in cases where FlashNet did not respond or a user sent duplicate jobs or jobs for which the media is offline. Once this has been run, user must go into CatDV and resubmit any pending jobs.

FlashNet Print Queue - Displays all queued assets. Used for reviewing jobs submitted to FlashNet. Helpful for comparing jobs shown as “sent” in CatDV to what was received by CatDV2FlashNet middleware when troubleshooting. Also useful for checking for duplicate jobs since users in CatDV can sometimes submit the same asset multiple times without noticing.

FlashNet Remove Lock - Used to clear the “lock” state during troubleshooting operations.

The purpose catdv2flasnet_enqueue pid file locking mechanism is to avoid duplicating flashnet commands for any given CatDV asset.

The issue is that if two copies of catdv2flastnet_enqueue run at the same time, the same asset may get processed twice. To avoid this, a pid file is written, which prevents a second instance catdv2flasnet_enqueue from running at the same time... If it is run at the same time, the second instance will detect the existing pid file and exit without running. When it runs successfully, this pid file is removed. When there is an unexpect error, this pid file is remove. In some circumstances, such as catdv2flasnet_enqueue process being killed, the pid file won't be cleaned up and will need to be removed manually.

Run FlashNet - Sends the currently queued assets to FlashNet immediately. This overrides the polling time for running archive jobs. Allows an administrator to immediately trigger the sending of a job to FlashNet instead of waiting for the thresholds to be met.

Reading & Using FlashNet Logs

You may view the logs for the CatDV2FlashNet application at

Catdv2flashnet_enqueue.log - Logs the asset queuing process for the job - lookup, check FlashNet, add to queue

catdv2flashnet.log - Logs the job construction and submission process to FlashNet

Log Use Tips:

Note: Search “CRITICAL” for fast search of error messages

Note: The CatDV CREF is the GUID that is passed to FlashNet!!

Troubleshooting Scenarios:

Issue: Filesystem - Mounted Storage and Paths

By far the most common source of issues in any networked system is pathing. Whether by systems not having the proper storage mounted or from some mis-configuration of storage mounts or paths, inconsistencies or errors in the file path will disrupt archive operations.

##Discuss mixed OS environments - #ADD archive drawing Mac to PC

Confirm that all storage is mounted to the SGL FlashNet Server as a UNC path. (If you are not sure how to confirm this, contact your system administrator.)

#DC Note: UNC paths do not show up in "My Computer". - #BJ Note, how do users test this?

##troubleshooting??  Note: SGLSVC acct - Confirm that a user named "sglsvc" (set up when your FlashNet software was installed) has access to all storage locations.

Also, if you are running CatDV or your Worker Node primarily on OS X systems, confirm that your path mapping is set up correctly in both your CatDV Settings as well as the configuration of your CatDV2FlashNet application. (#Add section to manual about config check for app?)

If you are unclear on how to confirm this, contact your system administrator.

Issue: Queue is frozen / jobs are not running

If assets are in the CatDV2FlashNet queue but jobs are not running, check to see if the queue is locked.

Launch the “Run FlashNet.command”. If it says the Queue is in use but there are no jobs running then you need to unlock the Queue.

The following is an example of the output of the “Run FlashNet.command” showing the “Queue locked” state:

2016-11-09 14:12:11,200 - northshorelib.dam.session.catdv_session - INFO - No catdv_session_manager.cfg [SessionManager] python_exe_path provided, assuming #!/usr/bin/env python will work.

2016-11-09 14:12:11,211 - northshorelib.dam.session.catdv_session - INFO - Session Manager Running: True

2016-11-09 14:12:11,211 - northshorelib.dam.session.catdv_session - INFO - Requesting current session from session manager

2016-11-09 14:12:11,232 - c2fcore.queuemaster - INFO - Queue is currently locked, cannot acquire lock

2016-11-09 14:12:11,233 - catdv2flashnet - INFO - Queue in use, exiting.

If you see that the Queue is currently locked (as noted in the bold text in the example above), and no jobs are running, you may unlock the Queue using the “FlashNet Remove Lock.command”.

To run the “FlashNet Remove Lock.command” simply double click the script (normally located on the desktop of the Worker Node system.)

Running the “FlashNet Remove Lock.command” will forcibly remove the lock and allow jobs to be resent. After removing the lock, re-run “Run FlashNet.command”, or wait for the timeout for the CatDV2FlashNet app to resume function

CatDV2FlashNet Error Messages

Here are a few of the most common errors you may encounter:

Error: ConnectionError: ('Connection aborted.', gaierror(8, 'nodename nor servname provided, or not known'))

Definition: CatDV Server unreachable. Please check that the CatDV Server is accessible on the network to the CatDV2FlashNet computer. This may be caused by disruption of network traffic, a routing or topology error, or the CatDV Server address in the CatDV2FlashNet application config being set incorrectly.

Error: CatDVRestApiError: Invalid user name or password

Definition: The CatDV Server credentials in the CatDV2FlashNet config are incorrect. Please confirm your login credentials and settings.

Error: error: [Errno 51] Network is unreachable

Definition: There are issues communicating with the FlashNet server. This may be caused by network connection disruption or network/computer latency.

###on the FlashNetserver or CatDV2FN server?

Error: Process Already Running - Process already running (based on pid file: <pid_file_path>)

Definition: There are two possible causes for this error.

Error: CatDVRestApiError: Expected 1 clip from query of CREF=2AB8EF36, received 0

Definition: You have attempted to archive an asset that is no longer in the CatDV database. This may happen in situations where a large archive operation is running and users delete assets from CatDV while the operation is in progress. If you see this error, search CatDV for the Clip ID listed in the error log.  

Note: The CREF, for example, 2AB8EF36 in the example above.

FlashNet Errors

##FlashNet errors are reported in the FlashNet log and FlashNet status fields in CatDV.

Conclusion

This completes this document. If you have questions about the configuration or operation of this system, please contact North Shore support at: