NSA CatDV-FlashNet User Manual

Introduction

This document is considered in draft format for review only. Updated 02/05/2018

Welcome to the North Shore Automation CatDV2FlashNet user and administration guide. This document outlines the functions, usage, and basic technical information of the application. While we advise that all users read the entire manual, system operators may only need to view the first few chapters. A system administrator or installer will want to review the document in its entirety.

In a nutshell, the CatDV2FlashNet application enables CatDV DAM systems to connect to SGL FlashNet archive systems. By using the CatDV2FlashNet solution, CatDV can communicate with a SGL FlashNet system to initiate and monitor archive and restore operations and have archive-related metadata recorded in CatDV Server.

This document will cover the following topics:

• • CatDV2FlashNet user operation from the CatDV client application

  • CatDV2FlashNet application overview

  • CatDV2FlashNet configuration

  • CatDV2FlashNet system troubleshooting

Document Conventions and Assumptions

This document assumes knowledge of the CatDV and FlashNet software. If you need more detailed information on CatDV Pro or FlashNet, please refer to their respective documentations.

In some cases in this document the term "archive action" may be used generically to refer to any archive related operation, such as archive to tape, restoring from tape, or asset deletion. When a specific type of operation is referred to, such as a restore, it will be referred to by name.

These systems are highly customized in most cases, so all references to topology and system configuration are considered examples. North Shore, Square Box and SGL assume that any CatDV2FlashNet integration will include a system design discussion with your CatDV and SGL integrator to determine the correct configuration for your organization.

Chapter 1: CatDV User Operation

This section is a step-by-step user guide on how to initiate archive operations using CatDV Pro Interface. The functions for archiving, restoration, and deletion are covered. Importantly, all actions here are demonstrated on a single asset; however, users may select as many assets as they choose and send all of them to a workflow at one time.

If you require additional information about using CatDV Pro, please refer to CatDV's user documentation or the Squarebox website.

The CatDV2FlashNet software can be installed on either an OS X or Windows Worker Node.

In most installations, the CatDV-to-FlashNet Archive App is accessed through the CatDV “SGL” details panel (as shown). The specific fields used to manage and track archive jobs are defined and described in Chapter ##2 "CatDV Metadata." Your system may be configured somewhat differently, but the metadata fields and functions will be the comparable to what is documented in this chapter.

Note: FlashNet may be abbreviated "FN" in field displays to keep label size down.

Archiving An Asset

To send an asset to the FlashNet archive system, perform the following actions.

Select an asset in the Main window of CatDV Pro.

Archive Action Field options

Note: The Archive Action Field has three options corresponding to the three FlashNet job types:

• • "Send-to-FlashNet" initiates an archive job for a specific record

  • "Restore-from-FlashNet" initiates a restore job for a specific record

  • "Delete-from-FlashNet" initiates a delete job for a specific record

Checking operation status

If, after selecting an action to initiate, you refresh your CatDV server connection (in Server Menu or f5 key) you will see status updates on the selected record/s. Typically job submission times may vary depending on how your system is configured, but normally jobs are sent to FlashNet in 5 minutes or less.

While FlashNet processes an archive job, the CatDV Worker Node will update and reset the "Archive Action" field to blank.

Once a job is successfully submitted to FlashNet, the FN Job ID field is updated with a Job ID.

Troubleshooting Tip: If no Job ID is present after more than 5 minutes (and the user has confirmed a save and/or refresh in CatDV), this might indicate a problem with the archive operation. If the Archive Action field does not reset to blank (i.e., it still says "Send-to-Archive"), that may indicate that the Worker Node has failed to submit the job and you, or your administrator, should check the Worker Node. If the Archive Action and the FN Job ID are both blank, that might indicate that Worker Node submitted the job, but FlashNet did not receive the data or there is a problem with the FlashNet system.

Fig. 1.4 - Operation complete

If there are errors in the operation, the FN Status will display "ERROR" and a brief section of the log will be written to the FN Log. Depending on the error displayed you may report the problem to your administrator or begin troubleshooting the appropriate system or workflow step.

Selecting a Volume Group for an archive (optional)

As an optional step, you may want specific records archived to a specific FlashNet volume/tape group. You can specify which volume group to use before initiating an archive job by selecting a specific volume group in the "FlashNet Volume" field.

Note: This functionality requires your administrator to set up a list of FlashNet volume groups in CatDV.

Fig. 1.5 - FlashNet Volume Group Selection

This image shows an archive job ready for submission to a "CATDV" Volume Group. (In this example the "%%" is a wildcard and will be replaced with a Volume Group ID Number. The Volume Groups are named CATDV_1 - CATDV_10 and this will select the next available ID. Your system may be configured with a single Volume ID and so the value might be something more like “2016_tapes.”)

Fig. 1.7 - Restore from FlashNet

Optional: Select alternate restore location

When initiating a restore you have two options:

• 1. You may leave the location blank and the asset will be restored to its original location/file path.

  2. If your system administrator has set up specific alternate restore locations, you  may choose a predefined location from the “Restore Location” menu. In such a case, the asset will be restored to the designated location. You can specify which location to use before initiating a restore job in the "Restore Location" field, shown below. This will restore the file into the specified location preserving the original file path from root.

Note: If no location is specified in the “Restore Location” field, the app CatDV2FlashNet app directs FlashNet to restore to the location listed in the "Media Path" field for the record/s. This will normally be the original location of the asset in storage. If that location does not exist (for example, if that volume is not accessible to the FlashNet Server), the restore job will fail and an ERROR message will be displayed for the asset.

Fig. 1.8 - Restore with Restore Location Set

Fig. 1.9 - Indication of Restore Completed

Upon completion of the restore operation the Archive Action field will be reset to blank, the FN Job History will display date and time of the restore, and the FN Status will be set to "RESTORED."

                                                    Fig. 1.10 Restore from FlashNet Completed Successfully

Delete-from-FlashNet #needs caption content

Delete From FlashNet

While assets cannot be truly “deleted” from an LTO tape due to their linear nature, FlashNet can set an asset to be “forgotten” on the tape so that it may be archived. This is most commonly desired when an asset is re-created or a new version is made to fix a problem with the media. In these cases the name of the file and its path will be the same as the old version, and this could prevent it from being archived to the FlashNet system.

Before initiating a delete job, the user will have to check the "Confirm Delete" box. While FlashNet processes a delete job, the app will update the CatDV record/s with "Restore-from-FlashNet" in the "Archive Action" field.

Note: CatDV records for delete jobs will display the relevant FlashNet archive information in the FlashNet Job ID, FlashNet History, and FlashNet Log fields.

Fig. 1.11 -Delete-from-FlashNet #needs caption content

Indication of Delete-from-FlashNet completed

Once FlashNet successfully completes an archive job, the app will update the CatDV record/s with:

• • A summary of the delete job in the "FN Job History" field, along with previous archive jobs

  • "Deleted" in the "FlashNet Status" field

  • "Delete Job Sent" in the "FlashNet Log" field

# Note: Any information about previous archive jobs in the "FlashNet Log" field will be overwritten by a delete job as this field only displays the log data for the last action. To see more information on all jobs, please refer to the FlashNet Administration app on the FlashNet server. However, the "FlashNet Job ID" field will show the job ID for the last completed archive prior to the deletion.

Fig. 1.12 -Delete-from-FlashNet #needs caption content

##Section Addition - Add pre-flight rules for queue submission process

• • Duplicate asset consequences

  • Viewing/checking asset archive status before submitting

    • Using Archive View in CatDV Pro

  • Simple threshold discussion

    • View detailed discussion in Chapt. 3

Chapter 2: CatDV Metadata

This section lists all the metadata fields in CatDV that are used and also gives a brief description of them.

The FlashNet metadata fields are created in the CatDV field set during the system deployment and can be mapped to a details panel in CatDV. We recommend collecting these fields into a dedicated CatDV details pane for clarity.

In this example, the user has created a details panel called "SGL FlashNet" in which the FlashNet and CatDV2FlashNet metadata are displayed.

The metadata fields for this workflow are named to aid in tracking status and troubleshooting the archive workflow.

The data in fields labeled "FN" is supplied from the SGL FlashNet server via API. All other fields are either updated by user input or by a Worker Node action (ex. Archive Action, FN Volume Group, Restore Location).

Note: There are two exceptions to the “FN” naming convention. FN Volume Group and FN Confirm Delete are user input fields that are set by the user, as needed.

CatDV2FlashNet Metadata Fields

Fig. 2.1 - FlashNet Metadata Fields as shown in CatDV

File Name - Built-in CatDV field that lists the selected file's name

Media Path - Built-in CatDV field that lists the selected file's path.

Clip ID - Built-in CatDV field that lists the selected file's Clip ID (GUID).

FN Job ID - CatDV User field, updated by FlashNet, that lists the FlashNet Job ID.

Archive Action - CatDV User field watched by the Worker Node in order to start the job enqueue process.  After the Worker Node action runs and the archive action has been submitted to FlashNet, this value will be set to blank (a null value).

Listed here are the possible values for the Archive Action field:

• • Send-to-FlashNet

  • Restore-from-FlashNet

  • Delete-from-FlashNet

FN Volume Group - CatDV User field that indicates the FlashNet Volume Group on which the asset is stored.

FN Confirm Delete - CatDV User field, configured as a checkbox, which must be set in order to send a deletion request to the archive system. This helps prevent accidental deletions.

FN Job History - CatDV User field, updated by FlashNet, that lists the following information: Timestamp, Completion Status, and CatDV2FlashNet Job ID. This data is updated at the completion of an archive operation. New operations are appended to this field.

FN Status - CatDV User field, updated by FlashNet, that lists the assets archive status. This field will be updated dynamically during archive and restore operations.

When a Job is being monitored, if the FlashNet job status is ‘PASSED’, the FN Status will be ARCHIVED, RESTORED, or DELETED depending on the job type sent.

In the case of restoring, if the media is offline, the status will be OFFLINE and details will be provided in the log field.

Jobs that are in-progress will have a status of ‘Archive Job Sent’, ‘Restore Job Sent’, until the status updates with the current status.

A failed deletion status is ‘Failed Deletion Request’.

All other flashnet job statuses will be reported as:

<Job Type> - <Flashnet Status Passthrough>

For example:

ARCHIVE - Awaiting Cache Fill

Here are the possible values for the FlashNet Status Passthrough:

• Cache Fill Failed

• Candidate Only

• Changer Error

• Dtool Error

• Queue Error

• Stopped

• Finished

• Running

• Sent to Launcher

• Ready to send to Launcher

• Master Job

• Not Processed

• Not Checked

• Unknown Volume/Group

• Drives Busy

• Media Busy

• Client Busy

• Group Empty

• No Device Drivers

• Changer Offline

• Rule Blocked

• Overwrite Disallowed

• No I/O Slaves

• No Changer Drives Defined

• Changer Drives Offline

• Specified Drive Busy

• Out Of Changer

• No Available Cache

• Filling Cache

• Awaiting Cache Fill

Restore Location - CatDV User field that lists an alternate location for an asset restore operation. If this field is left blank, the asset will be restored to its original file path.

Note: Please confirm that the original file location still exists before initiating a restore operation in order to prevent restore errors.

FN Log - CatDV User field, updated by FlashNet, that lists additional FlashNet job information. This is the first 10,000 characters of the FlashNet log for the archive operation by default. This can be changed by the administrator in the settings for the application.

## add mention and screenshot of “View” panel for archive

This completes a description of all of the metadata fields in CatDV that are used. If you have any questions, please contact your systems administrator or North Shore Automations.  

Chapter 3: CatDV2FlashNet App

This section provides a functional overview of CatDV2FlashNet and describes the application and basic system topology. The chapter includes a block diagram and step-by-step workflow overview describing the steps visible to the user and the back-end processes at work in archive operations.

The CatDV2FlashNet app is middleware, managing the information exchange between CatDV and FlashNet. In simple terms CatDV2FlashNet initiates archive jobs, via metadata and API calls, to a FlashNet Server and delivers the responses from FlashNet back to the CatDV Server. CatDV2FlashNet does not directly move or manipulate actual media or files. All file archive actions are performed by FlashNet, and all metadata is stored by CatDV.

Application Topology

This drawing shows a block diagram of the applications and systems involved in a typical CatDV2FlashNet system.

Actual system topologies vary widely, but a typical system deployment might consist of the following:  An edit system running CatDV Pro Client; a CatDV Server computer; a CatDV Worker Node system running Worker Node and the CatDV2FlashNet software; and a SGL FlashNet system consisting of a server connected to an LTO Library. All or most of these systems would typically be mounting a shared storage system, typically SAN or NAS.

Contact your integrator or system administrator or refer to your system deployment documentation for information on your specific configuration.

Fig. 3.1 - CatDV2FalshNet Data Flow Topology

Workflow Overview

The following section describes the manner in which data is interchanged through the system. Only the archive action is described for the sake of brevity, but the other workflows (restore, etc.) employ a similar methodology.

The order of operations in the workflow is as follows: (See the drawing above for a visual representation of these steps and data transfers.)

Archive an asset/s

• 1. A user flags asset/s in CatDV to send to FlashNet archive by setting a metadata value in a field to “send-to-archive” (or similar value)

     1. Archive actions may also be triggered from Worker action based on metadata received from another workflow (ex. an ingest operation may auto-set the archive metadata as part of the workflow)

  2. If FlashNet has multiple Volume Groups, the user can select a specific Volume Group from a pull-down menu

• • 1. If the Volume Group is not set by the user prior to submitting the Archive request, the default Volume Group specified in the system configuration will be used

    2. Each job may only be sent to a single Volume Group

• 3. The CatDV Worker Node is triggered by a server query to execute the catdv2flashnet_enqueue.py python application and send the asset/s Clip ID (CREF) to the CatDV2FlashNet app.

  4. Using the Clip ID, the CatDV2FlashNet application retrieves the asset/s metadata (including file path) from the CatDV Server via REST API.

  5. Each asset record is “queued” for archiving by the CatDV2FlashNet app

  6. Asset records are held in the CatDV2FlashNet Queue (a SQLite db) until either a size or time threshold is reached

• • 1. These thresholds are specified in the config file and can be modified by the system administrator

    2. The default size threshold is 5GB

    3. The default time threshold is 480 minutes

    4. If any one asset is 5GB or greater, it is sent as a job immediately, skipping the queue

    5. The time is measured from the first asset that reaches the queue for each job

• 7. When either threshold is reached, all queued assets are sent as a single job by the Job Submission engine in CatDV2FlashNet

  8. The FlashNet Server is queried for each asset’s status

  9. If the asset is already in FlashNet, then:

• • 1. This info will be added to the FN Log Field in CatDV

    2. The FN Status field will be updated to show “archived” (or similar)

    3. The asset will not be sent to FlashNet as part of the current job.

    4. Note: In FlashNet if a job is submitted that contains an asset that exists in the system, FlashNet will fail the entire job and report an error. CatDV2FlashNet is designed to check and modify archive jobs to prevent this from happening in case a user submits an asset multiple times

• 10. If the asset is not in FlashNet, then an archive job is created from the queue

  11. The archive job/s are submitted to FlashNet

  12. A unique Job ID is created for the archive job

  13. The Job ID is written to the “FN Job ID” field in CatDV for each asset in the job

• • 1. example: catdv2fn jobid: 295-HISXPA

    2. Users may search that field in CatDV to list files queued in a single FlashNet job

• 14. If a job only has one file or metaclip, the ID number refers to the asset

• • 1. The asset extension is appended to the FN Job ID

    2. example: d01a555d-69dc-4b8e-acae-872e94d405b4-IMG_2188.mp4

    3. NOTE: This Job ID is different than the ClipID ##ADD

• 15. The FN Status is updated in CatDV for each asset in the job as the job progresses

  16. The FN Status field in CatDV lists the status of each submitted asset

• • 1. e.g. “20150925T160735 - Archive Queued”, “ARCHIVED”, “RESTORED” (or similar)

• 17. Note: As noted above, multiple CatDV assets may share the same Job ID and a CatDV search on that ID will return the job’s entire contents

  18. If the archive action is successful, the asset’s FN Job History field is updated with Date/Time, archive status, and Job ID. The operation is now complete.

  19. If an archive job fails, the FlashNet Status field displays a FAILED message

• • 1. ARCHIVE - FAILED

    2. The log file is filtered for the line/s that show the failure

    3. The CatDV FN Log Field for each of the assets is updated with the results of this operation

    4. The operation is now complete.

Chapter 4: Worker Node

#In Progress

Chapter 5: SGL FlashNet

#In Progress

Chapter 6: Troubleshooting

Error Reporting - Send-to-FlashNet failure notification

When an archive job fails, the app will update the CatDV record/s with "ARCHIVE - FAILED" in the "Archive Action" field. It will also provide the job ID in the "FlashNet Job ID" field and a detailed log for the job in the "FlashNet Log" field.

Note: The FlashNet log field may contain additional information that can be viewed by using the scroll bar on the right of the field or increasing the size of the "Details Panel".

Failure Log Notification

Failure Log Details

North Shore Utility Scripts

North Shore provides a set of utility scripts to assist in troubleshooting and checking the status of the CatDV2FlashNet middleware. These are typically stored in a folder on the desktop of the Worker Node system running the CatDV2FlashNet middleware.

Fig. ## List of North Shore utilities

CatDV2FlashNet Utility Scripts

FlashNet Flush Queue - Clears all jobs from NSA app queue. Resets the job queue. Used in cases where FlashNet did not respond or a user sent duplicate jobs or jobs for which the media is offline. Once this has been run, user must go into CatDV and resubmit any pending jobs.

FlashNet Print Queue - Displays all queued assets. Used for reviewing jobs submitted to FlashNet. Helpful for comparing jobs shown as “sent” in CatDV to what was received by CatDV2FlashNet middleware when troubleshooting. Also useful for checking for duplicate jobs since users in CatDV can sometimes submit the same asset multiple times without noticing.

FlashNet Remove Lock - Used to clear the “lock” state during troubleshooting operations.

The purpose catdv2flasnet_enqueue pid file locking mechanism is to avoid duplicating flashnet commands for any given CatDV asset.

The issue is that if two copies of catdv2flastnet_enqueue run at the same time, the same asset may get processed twice. To avoid this, a pid file is written, which prevents a second instance catdv2flasnet_enqueue from running at the same time... If it is run at the same time, the second instance will detect the existing pid file and exit without running. When it runs successfully, this pid file is removed. When there is an unexpect error, this pid file is remove. In some circumstances, such as catdv2flasnet_enqueue process being killed, the pid file won't be cleaned up and will need to be removed manually.

Run FlashNet - Sends the currently queued assets to FlashNet immediately. This overrides the polling time for running archive jobs. Allows an administrator to immediately trigger the sending of a job to FlashNet instead of waiting for the thresholds to be met.

Reading & Using FlashNet Logs

You may view the logs for the CatDV2FlashNet application at

• • Mac

    • /Library/NSA/logs/

  • Windows

    • C:\Users\<USERNAME>\northshore\logs\

Catdv2flashnet_enqueue.log - Logs the asset queuing process for the job - lookup, check FlashNet, add to queue

catdv2flashnet.log - Logs the job construction and submission process to FlashNet

Log Use Tips:

Note: Search “CRITICAL” for fast search of error messages

Note: The CatDV CREF is the GUID that is passed to FlashNet!!

Troubleshooting Scenarios:

Issue: Filesystem - Mounted Storage and Paths

By far the most common source of issues in any networked system is pathing. Whether by systems not having the proper storage mounted or from some mis-configuration of storage mounts or paths, inconsistencies or errors in the file path will disrupt archive operations.

##Discuss mixed OS environments - #ADD archive drawing Mac to PC

Confirm that all storage is mounted to the SGL FlashNet Server as a UNC path. (If you are not sure how to confirm this, contact your system administrator.)

#DC Note: UNC paths do not show up in "My Computer". - #BJ Note, how do users test this?

##troubleshooting??  Note: SGLSVC acct - Confirm that a user named "sglsvc" (set up when your FlashNet software was installed) has access to all storage locations.

Also, if you are running CatDV or your Worker Node primarily on OS X systems, confirm that your path mapping is set up correctly in both your CatDV Settings as well as the configuration of your CatDV2FlashNet application. (#Add section to manual about config check for app?)

If you are unclear on how to confirm this, contact your system administrator.

Issue: Queue is frozen / jobs are not running

If assets are in the CatDV2FlashNet queue but jobs are not running, check to see if the queue is locked.

Launch the “Run FlashNet.command”. If it says the Queue is in use but there are no jobs running then you need to unlock the Queue.

The following is an example of the output of the “Run FlashNet.command” showing the “Queue locked” state:

2016-11-09 14:12:11,200 - northshorelib.dam.session.catdv_session - INFO - No catdv_session_manager.cfg [SessionManager] python_exe_path provided, assuming #!/usr/bin/env python will work.

2016-11-09 14:12:11,211 - northshorelib.dam.session.catdv_session - INFO - Session Manager Running: True

2016-11-09 14:12:11,211 - northshorelib.dam.session.catdv_session - INFO - Requesting current session from session manager

2016-11-09 14:12:11,232 - c2fcore.queuemaster - INFO - Queue is currently locked, cannot acquire lock

2016-11-09 14:12:11,233 - catdv2flashnet - INFO - Queue in use, exiting.

If you see that the Queue is currently locked (as noted in the bold text in the example above), and no jobs are running, you may unlock the Queue using the “FlashNet Remove Lock.command”.

To run the “FlashNet Remove Lock.command” simply double click the script (normally located on the desktop of the Worker Node system.)

Running the “FlashNet Remove Lock.command” will forcibly remove the lock and allow jobs to be resent. After removing the lock, re-run “Run FlashNet.command”, or wait for the timeout for the CatDV2FlashNet app to resume function

CatDV2FlashNet Error Messages

Here are a few of the most common errors you may encounter:

Error: ConnectionError: ('Connection aborted.', gaierror(8, 'nodename nor servname provided, or not known'))

Definition: CatDV Server unreachable. Please check that the CatDV Server is accessible on the network to the CatDV2FlashNet computer. This may be caused by disruption of network traffic, a routing or topology error, or the CatDV Server address in the CatDV2FlashNet application config being set incorrectly.

Error: CatDVRestApiError: Invalid user name or password

Definition: The CatDV Server credentials in the CatDV2FlashNet config are incorrect. Please confirm your login credentials and settings.

Error: error: [Errno 51] Network is unreachable

Definition: There are issues communicating with the FlashNet server. This may be caused by network connection disruption or network/computer latency.

###on the FlashNetserver or CatDV2FN server?

Error: Process Already Running - Process already running (based on pid file: <pid_file_path>)

Definition: There are two possible causes for this error.

• 1. Process running: A process such as #define# is already running. This is an expected behavior to prevent multiple copies of the process from running at the same time.

  2. No process is running: A stale pid file exists.

  3. Fix: Delete the pid file based on path provided in the error message

  4. Note: Contact NSA support to confirm either state.

Error: CatDVRestApiError: Expected 1 clip from query of CREF=2AB8EF36, received 0

Definition: You have attempted to archive an asset that is no longer in the CatDV database. This may happen in situations where a large archive operation is running and users delete assets from CatDV while the operation is in progress. If you see this error, search CatDV for the Clip ID listed in the error log.  

Note: The CREF, for example, 2AB8EF36 in the example above.

FlashNet Errors

##FlashNet errors are reported in the FlashNet log and FlashNet status fields in CatDV.

Conclusion

This completes this document. If you have questions about the configuration or operation of this system, please contact North Shore support at:

• • support@northshoreautomation.com

  • http://support.northshoreautomation.com

  • 818-450-3625