NSA CatDV-Vantage User Manual

Introduction

This document is provided in incomplete draft format, for review only, as of 03/07/2017.

Welcome to the North Shore Automation CatDV-Vantage user and administration guide. This document outlines the functions, usage, and basic technical information of the application and its related systems. While we advise that all users read the entire manual, system operators may only need to view the first few chapters. A system administrator or installer will want to review the document in its entirety.

The CatDV-Vantage application enables CatDV DAM systems to connect to Vantage systems. By using the CatDV-Vantage solution, CatDV can communicate with a Vantage system to send assets to transcoding and other types of workflows, then record job-related metadata in the CatDV database.

This document will cover the following topics:

## Topics to be updated as draft is edited

Document Conventions and Assumptions

This document assumes knowledge of the CatDV and Vantage software. If you need more detailed information on CatDV Pro or Vantage, please refer to their respective documentation.

In some cases (typically the overview sections), the term "Vantage action" may be used generically to refer to any workflow related operation such as transcode/flip, analyze, or asset copy/move. When a specific type of operation is referred to, such as a flip, it will be referred to by name.These systems are highly customized in most cases, so references to topology and system configuration are examples. Consider undertaking a system design with your CatDV and Vantage integrator to determine the correct configuration for your organization.

Chapter 1 – User Operations

This section...

Chapter 2 - CatDV Metadata

This section lists all metadata fields in CatDV that are used for Vantage operations and a description of their data.

The Vantage metadata fields are created in the CatDV metadata during the system deployment and can be mapped to a details panel in CatDV or distributed amongst other panels, as needed. We recommend collecting these fields into a dedicated CatDV details pane for clarity.

In this example, the user has created a details panel called "Vantage" in which the Vantage and #CatDV-Vantage metadata are displayed.

Note: In the "Main View" of the CatDV window, you can set up a column display to view the Vantage Status for every clip.

The metadata fields for the CatDV-Vantage workflow are named in a way that aids in tracking status and troubleshooting the workflow.

The data in fields beginning with "Vantage" are typically coming from the Vantage server via API. All other fields are either updated by user input or by a Worker Node action (e.g., Vantage Workflow State, Vantage Log, etc.)

Note: There is one exception to this convention. “Vantage Action” is a user input field that is set by the user to initiate the Worker Node action.

CatDV-Vantage Metadata Fields

• • Vantage Workorder –  Pick list metadata field to send a specific Vantage workorder CSV to a watch folder

  • Vantage Job ID –  Job ID from Vantage

  • Vantage % –  Percentage of job progress in Vantage; note that the percentage given by Vantage is imprecise and may not indicate overall completion

  • Vantage Log –  Log of statuses for each clip that is submitted to Vantage

  • Vantage Workflow Status – Job status reported directly from Vantage

    • Sample Values

      • Complete

      • Failed

      • Waiting

      • StoppedByUser

      • Active

  • Vantage Monitor Status – Job status interpreted by North Shore Automation’s CatDV-Vantage app

    • NSA interpretation of the workflow state

    • Sample values:

      • Job Submitted

      • In Progress

      • Submit Failed

      • Job Failed

      • Complete

      • Job ID Missing

      • Workflow Initiating

      • Unknown State

  • (Optional) Vantage Output Path –  This is provided in case an output path is needed for the workflow to be initiated. An example of this might be in a CatDV proxy creation workflow, where the proxy location must be derived in the Worker Node, using variables, and then provided to Vantage so that the proxy is placed in the correct path to be detected by CatDV Pro.

    • Note: This field is optional and may not be used in all workflows

Chapter 3 - CatDV-Vantage App

This section provides a functional overview of the North Shore CatDV-Vantage application.

The CatDV-Vantage application consists of two parts, an initiation application that is run via a Worker Node command line operation and the main application that communicates with both the CatDV Server and Vantage Server as well as creating the Vantage work orders to hand off CatDV metadata to Vantage.

The work order job submission method was chosen at the suggestion of Telestream for its increased flexibility and ability to add metadata to workflows without requiring substantial programming skills or undue complexity of the middleware application. However, work orders alone, while simple and effective for job submission, don’t provide job status feedback to the CatDV Server.

North Shore combines the simplicity of workorder workflows with the reliability and live feedback of a REST API-based solution.

CatDV-Vantage Block Diagram

CatDV-Vantage Sample Topology

Note: CatDV Worker Node is typically not hosted on Vantage Server, but in some instances this may be necessary or desired. Worker Node and the CatDV-Vantage app may run on OS X, Windows, or Linux Worker systems. CatDV-Vantage handles all cross-platform path translations in its config file.

Workflow Overview

• 1. A user flags assets in CatDV to send to Vantage by setting a metadata value in a field to “Create Proxy,”or similar. (These are usually named for the workflow that they will initiate).

  2. The CatDV Worker Node is triggered by a server query to send the asset Clip ID and workflow name/ID to the CatDV-Vantage app via a North Shore application initiated by a Worker Node command

     1. Vantage Action field is reset to blank once the Worker Node action runs.

     2. The CatDV-Vantage app creates the Vantage workorder

Note: CatDV asset metadata is inserted into the workorder during the workflow and workorder set up - All workorders must include source media path and CREF at a minimum in order to enable monitoring.

• 1. CatDV-Vantage app puts the workorder in the designated watch folder for the workflow (set in Vantage and the NSA config). This triggers the Vantage workflow

     1. A unique Job ID is created for the asset by Vantage

  2. North Shore CatDV-Vantage app is monitoring workflows specified in the North Shore config file for new jobs. When the app detects a new job it retrieves the asset’s ClipID from Vantage, uses that to perform a lookup in CatDV and then updates the following metadata:

• • 1. Workflow State

    2. Vantage Job ID

    3. Percent complete

    4. Vantage Log info

• 3. CatDV-Vantage polls the Vantage server every 15 seconds throughout the job (configurable setting). The job status is updated in CatDV for each asset as its job progresses.

• • 1. If there is no change in the job in Vantage, the app does not send an update to CatDV. Only Vantage is polled.

• 4. Outcomes

• • 1. Success

       1. Workflow State is updated to “Complete”

       2. Vantage % updates throughout and shows 100% upon completion

       3. Vantage Log is written

          1. Log Example - 2016-06-22T09:58:27.714565 - proxy - Vantage Workorder Submitted to drop folder successfully.

          2. 2016-06-22T10:01:37.395756 - MISUNDERSTOOD_16x9HD_BSD009172H_120_PRHQ_SLATED_master.mov - CatDV Asset D8BD35EF vantage job: Complete

    2. Error - If a job fails, the status fields will display a “Fault” message. The Vantage Log field will list more detailed error information.

• 5. Vantage % displays percentage of completion at time of failure

• • 1. Note: This resets the next time a Vantage job is run on the asset

• 6. Vantage Log is written

• • 1. Log Example – 2015-11-06T15:55:38.040691 - F59B02BA-catdv-proxy-be35ff5ceecc42869bc1378b5160fbbf - Source: Flip error: RenderSession:StartSession(): A problem occurred when processing the input version, error =The Transcode service on host: VANTAGE-PC is unable to access the source file at location: \\NSA-NAS\SourceMedia\Demo Media\GoPro-Dive-1.mp4.  Please verify path references and permission settings.

1. Operation complete

Workflow Notes

IMPORTANT: North Shore defines a Vantage workflow as an operation initiated by a single workorder, having a single Vantage Job ID. Vantage operations submitted from CatDV should be either submitted serially, in order, one after another, or if a chain of operations must be performed by a single CatDV Worker Node action, then those operations should be grouped in Vantage. This is due to the fact that CatDV metadata should only be received from one Vantage workflow at a time to avoid conflicts and/or race conditions and to keep the complexity of the CatDV metadata to a minimum.

Chapter 4 - NSA CatDV-Vantage Deployment

Config File Overview

This section explains the settings contained in the North Shore CatDV-Vantage config file.

You may open it for editing in any text editor.

It is located at:

/etc/northshore/config/vantage_workorder.cfg - on OS X

%UserProfile%\northshore\config\vantage_workorder.cfg - on Windows

In the sections below we outline the specific settings and options for each section.

Config File - System

This is the top section of the config file, showing settings mostly related to system-wide functions.

Note: In the config file lines that start with "#" are ignored and so if you want to add notes for yourself, you may do so by adding  hash (#) to the beginning of the

License - This is your software license key. It will be supplied by NSA and is locked to the machine's hardware.

Vantage - This contains the network information for the Vantage server. "Host" should be set to the hostname or IP address of the Vantage server. "Port" should be set to 8676 by default, unless you have modified your Vantage server API configuration.

NOTE: Please enter the IP address or a valid hostname in the configuration. "localhost" is not a valid option, even if the software is running on the Vantage server.

Monitor - This section contains settings related to how the app monitors Vantage and reports updates to CatDV Server. The "sleep_amount" is the polling time for checking the Vantage job queue. the "CREF" is the CatDV field ID that is used to identify running jobs that were submitted from CatDV.

Filesystem - This section covers where logs are stored and where the PID (##define) is stored. The .pid file location is needed for certain troubleshooting operations.

Fields - This sections lists the fields that are mapped from CatDV Pro in order to initiate and complete Vantage jobs.

Note: Spaces in field names are converted to periods by CatDV Server upon creation.

Note: When configuring a system with CatDV Server v7+, yhe CatDV-Vantage app uses Server 7 slugs instead of USER field ID's. Contact your CatDV integrator or administrator for more information.

Config File - Workflows

This is the top section of the config file, showing settings mostly related to workflows.

Workflows To Monitor - This section contains a list of the workflows monitored by the app. Add additional workflows to the list by typing a descriptive name (of your choosing, i.e an alias) an "equals" sign and then the workflow ID from Vantage. (##add where to get this from Vantage)

Pattern: <alias> = <Vantage ID>

Path Mappings - If your system contains Mac OS X clients or Worker Nodes you will need to set path equivalencies so that the system can translate OS X paths to Windows paths for the Vantage application.

Pattern: <incoming path root>;<translated path root>

See the examples in the image below and contact your system administrator for more info.

Note: Vantage prefers UNC paths.

Workflows - This section contains info about the workflows managed by the CatDV-Vantage app. Each workflow is comprised of:

• • A workflow name, in brackets (i.e. [proxy]). This is simply for the clarity, it has no bearing on the operation of the system, however we recommend that you use the workflow alias from the "Workflows to Monitor" section.

  • The Work Order watch folder. This is the directory where the CSV file will be dropped to initiate the job.

  • The fields, in order, included in the workflow schema.

You may add as many workflows as you like to the system. Simply append them to the bottom of the config file and include them in the "Workflows to Monitor" section above.

Note: The doc convention varies slightly here in that the workflow names are enclosed in brackets "[ ]" as opposed to the section label.

Appendix A – Troubleshooting

Asterisk (*) in timecode values from CatDV

Vantage will not take 60 frame TC with an “*” in the TC display

https://documentation.apple.com/en/finalcutpro/usermanual/index.html#chapter=51%26section=5%26tasks=true

Apostrophes in timecode values from CatDV

Vantage will not accept time code values with apostrophes in them. CatDV uses apostrophes as a visual reminder that timecode is 23.98/24 fps. This is a preference that can be turned off in the “General” preference pane in CatDV Pro/Pegasus.

Vantage Errors

Vantage Error: RenderSession:StartSession(): A problem occurred when processing the output version, error =The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.

If you see this error, your path and/or filenames are too long. This is a Windows limitation, unrelated to North Shore, CatDV, or Telestream. For more information on this error, see the following article from Microsoft for best practices.

https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath

Appendix D – Vantage Work Order Fields

Special Fields

source_path –  Path mapping modified media path

source_path_orig –  Unmodified media path

CatDV System Fields

Simple

* ID

* bigNotes

* bin

* catalogID

* catalogName

* clipref

* duration2

* format

* fps

* good

* groupID

* groupName

* hidden

* importNotes

* in

* in2

* isEditable

* marked

* markers

* mediaEndMs

* mediaStartMs

* members

* metadata

* modifiedDate

* modifyDate

* name

* notes

* orientation

* out

* out2

* posterID

* rating

* recordedDate

* recordingDate

* seqItems

* seqNo

* sourceMedia

* sourceMediaID

* status

* tape

* tapeInfo

* tapeInfoRef

* thumbnailIDs

* type

* underlyingType

* userID

* userName

Special Case: Media Variables:

Special non-standard naming:

* media_ID

* media_metadata <-- Note: not very useful formatting

* media_modifiedDate

* media_tape

Standard naming:

* altPaths

* archiveStatus

* aspectRatio

* audioFormat

* dataRate

* filePath / file_path / source_path / source_path_orig

* fileSize

* importer

* isMediaAvailable

* omfRef

* proxyPath

* qttc

* still

* tcFmt

* tracks

* videoFormat

Special Variables –  These multiple potential formatting (only txt format available currently):

* end

* start

Special Case: Complex variables:

* history <-- Note: not very useful formatting

* importSource <-- Note: not very useful formatting

Special Case: Multi-formatting (only txt format available currently)

* duration

* in

* mediaEnd

* mediaStart

* out

User Fields

In the [Fields] section of vantage_workorder.cfg, you can add a mapping such as:

   vantage_job_id = vantage.job.id (when using CatDVServer v7)

   or

   vantage_job_id = U54 (when using CatDVServer v6)

This will allow you to use 'vantage_job_id' in the csv field definition accessing the field, user field 54 (CatDVServer 6) or vantage.job.id (CatDVServer 7.)

Appendix E – Workflow Notes

Handling CatDV Metaclips in Vantage

Metaclips are a CatDV method for collecting certain multi-part camera media assets into a single playable clip. Examples of this are Panasonic P2, and Sony XDCAM HD media types.

CatDV Pro and Pegasus interpret these collections of assets when the media is loaded, however, since there is no one media asset, the path for a metaclip is expressed as:

/path/to/media/file.***

Where instead of a file extension there are 3 asterisks (***). Because this convention exists only in the CatDV eco-system, Telestream’s Vantage cannot use these paths to parse a metaclip for processing.

The work-around for this is to insert a pre-flight Worker action that creates a QT reference movie, which of course has a valid path, for the metaclip media.

NOTE: This requires a Worker Node system with Quicktime installed as well as the necessary codecs (http://calibratedsoftware.com) for playback. This is due to a strange QuickTime behavior whereby reference movies created without the correct codecs installed can result in incorrect codec information being sent to Vantage, affecting the resulting transcoded material.

Appendix F – CatDV Server API notes

CatDV Server & Vantage API connections

The Vantage API communication port may be set in the config file as noted in Chapter 4 above. Typically, this is set to port 8676 by default.

CatDV’s REST API uses the web port set in your CatDV Control Panel. Typically, this is set to port 8080 by default. See the CatDV Server documentation for more info.