CatDV-Vantage User Manual
NSA CatDV-Vantage User Manual
Introduction
This document is provided in incomplete draft format, for review only, as of 03/07/2017.
Welcome to the North Shore Automation CatDV-Vantage user and administration guide. This document outlines the functions, usage, and basic technical information of the application and its related systems. While we advise that all users read the entire manual, system operators may only need to view the first few chapters. A system administrator or installer will want to review the document in its entirety.
The CatDV-Vantage application enables CatDV DAM systems to connect to Vantage systems. By using the CatDV-Vantage solution, CatDV can communicate with a Vantage system to send assets to transcoding and other types of workflows, then record job-related metadata in the CatDV database.
This document will cover the following topics:
## Topics to be updated as draft is edited
Document Conventions and Assumptions
This document assumes knowledge of the CatDV and Vantage software. If you need more detailed information on CatDV Pro or Vantage, please refer to their respective documentation.
In some cases (typically the overview sections), the term "Vantage action" may be used generically to refer to any workflow related operation such as transcode/flip, analyze, or asset copy/move. When a specific type of operation is referred to, such as a flip, it will be referred to by name.These systems are highly customized in most cases, so references to topology and system configuration are examples. Consider undertaking a system design with your CatDV and Vantage integrator to determine the correct configuration for your organization.
Chapter 1 – User Operations
This section...
Chapter 2 - CatDV Metadata
This section lists all metadata fields in CatDV that are used for Vantage operations and a description of their data.
The Vantage metadata fields are created in the CatDV metadata during the system deployment and can be mapped to a details panel in CatDV or distributed amongst other panels, as needed. We recommend collecting these fields into a dedicated CatDV details pane for clarity.
In this example, the user has created a details panel called "Vantage" in which the Vantage and #CatDV-Vantage metadata are displayed.
Note: In the "Main View" of the CatDV window, you can set up a column display to view the Vantage Status for every clip.
The metadata fields for the CatDV-Vantage workflow are named in a way that aids in tracking status and troubleshooting the workflow.
The data in fields beginning with "Vantage" are typically coming from the Vantage server via API. All other fields are either updated by user input or by a Worker Node action (e.g., Vantage Workflow State, Vantage Log, etc.)
Note: There is one exception to this convention. “Vantage Action” is a user input field that is set by the user to initiate the Worker Node action.
CatDV-Vantage Metadata Fields
Vantage Workorder – Pick list metadata field to send a specific Vantage workorder CSV to a watch folder
Vantage Job ID – Job ID from Vantage
Vantage % – Percentage of job progress in Vantage; note that the percentage given by Vantage is imprecise and may not indicate overall completion
Vantage Log – Log of statuses for each clip that is submitted to Vantage
Vantage Workflow Status – Job status reported directly from Vantage
Sample Values
Complete
Failed
Waiting
StoppedByUser
Active
Vantage Monitor Status – Job status interpreted by North Shore Automation’s CatDV-Vantage app
NSA interpretation of the workflow state
Sample values:
Job Submitted
In Progress
Submit Failed
Job Failed
Complete
Job ID Missing
Workflow Initiating
Unknown State
(Optional) Vantage Output Path – This is provided in case an output path is needed for the workflow to be initiated. An example of this might be in a CatDV proxy creation workflow, where the proxy location must be derived in the Worker Node, using variables, and then provided to Vantage so that the proxy is placed in the correct path to be detected by CatDV Pro.
Note: This field is optional and may not be used in all workflows
Chapter 3 - CatDV-Vantage App
This section provides a functional overview of the North Shore CatDV-Vantage application.
The CatDV-Vantage application consists of two parts, an initiation application that is run via a Worker Node command line operation and the main application that communicates with both the CatDV Server and Vantage Server as well as creating the Vantage work orders to hand off CatDV metadata to Vantage.
The work order job submission method was chosen at the suggestion of Telestream for its increased flexibility and ability to add metadata to workflows without requiring substantial programming skills or undue complexity of the middleware application. However, work orders alone, while simple and effective for job submission, don’t provide job status feedback to the CatDV Server.
North Shore combines the simplicity of workorder workflows with the reliability and live feedback of a REST API-based solution.
CatDV-Vantage Block Diagram
CatDV-Vantage Sample Topology
Note: CatDV Worker Node is typically not hosted on Vantage Server, but in some instances this may be necessary or desired. Worker Node and the CatDV-Vantage app may run on OS X, Windows, or Linux Worker systems. CatDV-Vantage handles all cross-platform path translations in its config file.
Workflow Overview
A user flags assets in CatDV to send to Vantage by setting a metadata value in a field to “Create Proxy,”or similar. (These are usually named for the workflow that they will initiate).
The CatDV Worker Node is triggered by a server query to send the asset Clip ID and workflow name/ID to the CatDV-Vantage app via a North Shore application initiated by a Worker Node command
Vantage Action field is reset to blank once the Worker Node action runs.
The CatDV-Vantage app creates the Vantage workorder
Note: CatDV asset metadata is inserted into the workorder during the workflow and workorder set up - All workorders must include source media path and CREF at a minimum in order to enable monitoring.
CatDV-Vantage app puts the workorder in the designated watch folder for the workflow (set in Vantage and the NSA config). This triggers the Vantage workflow
A unique Job ID is created for the asset by Vantage
North Shore CatDV-Vantage app is monitoring workflows specified in the North Shore config file for new jobs. When the app detects a new job it retrieves the asset’s ClipID from Vantage, uses that to perform a lookup in CatDV and then updates the following metadata:
Workflow State
Vantage Job ID
Percent complete
Vantage Log info
CatDV-Vantage polls the Vantage server every 15 seconds throughout the job (configurable setting). The job status is updated in CatDV for each asset as its job progresses.
If there is no change in the job in Vantage, the app does not send an update to CatDV. Only Vantage is polled.
Outcomes
Success
Workflow State is updated to “Complete”
Vantage % updates throughout and shows 100% upon completion
Vantage Log is written
Log Example - 2016-06-22T09:58:27.714565 - proxy - Vantage Workorder Submitted to drop folder successfully.
2016-06-22T10:01:37.395756 - MISUNDERSTOOD_16x9HD_BSD009172H_120_PRHQ_SLATED_master.mov - CatDV Asset D8BD35EF vantage job: Complete
Error - If a job fails, the status fields will display a “Fault” message. The Vantage Log field will list more detailed error information.
Vantage % displays percentage of completion at time of failure
Note: This resets the next time a Vantage job is run on the asset
Vantage Log is written
Log Example – 2015-11-06T15:55:38.040691 - F59B02BA-catdv-proxy-be35ff5ceecc42869bc1378b5160fbbf - Source: Flip error: RenderSession:StartSession(): A problem occurred when processing the input version, error =The Transcode service on host: VANTAGE-PC is unable to access the source file at location: \\NSA-NAS\SourceMedia\Demo Media\GoPro-Dive-1.mp4. Please verify path references and permission settings.
Operation complete
Workflow Notes
IMPORTANT: North Shore defines a Vantage workflow as an operation initiated by a single workorder, having a single Vantage Job ID. Vantage operations submitted from CatDV should be either submitted serially, in order, one after another, or if a chain of operations must be performed by a single CatDV Worker Node action, then those operations should be grouped in Vantage. This is due to the fact that CatDV metadata should only be received from one Vantage workflow at a time to avoid conflicts and/or race conditions and to keep the complexity of the CatDV metadata to a minimum.
Chapter 4 - NSA CatDV-Vantage Deployment
Config File Overview
This section explains the settings contained in the North Shore CatDV-Vantage config file.
You may open it for editing in any text editor.
It is located at:
/etc/northshore/config/vantage_workorder.cfg - on OS X
%UserProfile%\northshore\config\vantage_workorder.cfg - on Windows
In the sections below we outline the specific settings and options for each section.
Config File - System
This is the top section of the config file, showing settings mostly related to system-wide functions.
Note: In the config file lines that start with "#" are ignored and so if you want to add notes for yourself, you may do so by adding hash (#) to the beginning of the
License - This is your software license key. It will be supplied by NSA and is locked to the machine's hardware.
Vantage - This contains the network information for the Vantage server. "Host" should be set to the hostname or IP address of the Vantage server. "Port" should be set to 8676 by default, unless you have modified your Vantage server API configuration.
NOTE: Please enter the IP address or a valid hostname in the configuration. "localhost" is not a valid option, even if the software is running on the Vantage server.
Monitor - This section contains settings related to how the app monitors Vantage and reports updates to CatDV Server. The "sleep_amount" is the polling time for checking the Vantage job queue. the "CREF" is the CatDV field ID that is used to identify running jobs that were submitted from CatDV.
Filesystem - This section covers where logs are stored and where the PID (##define) is stored. The .pid file location is needed for certain troubleshooting operations.
Fields - This sections lists the fields that are mapped from CatDV Pro in order to initiate and complete Vantage jobs.
Note: Spaces in field names are converted to periods by CatDV Server upon creation.
Note: When configuring a system with CatDV Server v7+, yhe CatDV-Vantage app uses Server 7 slugs instead of USER field ID's. Contact your CatDV integrator or administrator for more information.
Config File - Workflows
This is the top section of the config file, showing settings mostly related to workflows.
Workflows To Monitor - This section contains a list of the workflows monitored by the app. Add additional workflows to the list by typing a descriptive name (of your choosing, i.e an alias) an "equals" sign and then the workflow ID from Vantage. (##add where to get this from Vantage)
Pattern: <alias> = <Vantage ID>
Path Mappings - If your system contains Mac OS X clients or Worker Nodes you will need to set path equivalencies so that the system can translate OS X paths to Windows paths for the Vantage application.
Pattern: <incoming path root>;<translated path root>
See the examples in the image below and contact your system administrator for more info.
Note: Vantage prefers UNC paths.
Workflows - This section contains info about the workflows managed by the CatDV-Vantage app. Each workflow is comprised of:
A workflow name, in brackets (i.e. [proxy]). This is simply for the clarity, it has no bearing on the operation of the system, however we recommend that you use the workflow alias from the "Workflows to Monitor" section.
The Work Order watch folder. This is the directory where the CSV file will be dropped to initiate the job.
The fields, in order, included in the workflow schema.
You may add as many workflows as you like to the system. Simply append them to the bottom of the config file and include them in the "Workflows to Monitor" section above.
Note: The doc convention varies slightly here in that the workflow names are enclosed in brackets "[ ]" as opposed to the section label.
Appendix A – Troubleshooting
Asterisk (*) in timecode values from CatDV
Vantage will not take 60 frame TC with an “*” in the TC display
Apostrophes in timecode values from CatDV
Vantage will not accept time code values with apostrophes in them. CatDV uses apostrophes as a visual reminder that timecode is 23.98/24 fps. This is a preference that can be turned off in the “General” preference pane in CatDV Pro/Pegasus.
Vantage Errors
Vantage Error: RenderSession:StartSession(): A problem occurred when processing the output version, error =The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.
If you see this error, your path and/or filenames are too long. This is a Windows limitation, unrelated to North Shore, CatDV, or Telestream. For more information on this error, see the following article from Microsoft for best practices.
https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath
Appendix D – Vantage Work Order Fields
Special Fields
source_path – Path mapping modified media path
source_path_orig – Unmodified media path
CatDV System Fields
Simple
* ID
* bigNotes
* bin
* catalogID
* catalogName
* clipref
* duration2
* format
* fps
* good
* groupID
* groupName
* hidden
* importNotes
* in
* in2
* isEditable
* marked
* markers
* mediaEndMs
* mediaStartMs
* members
* metadata
* modifiedDate
* modifyDate
* name
* notes
* orientation
* out
* out2
* posterID
* rating
* recordedDate
* recordingDate
* seqItems
* seqNo
* sourceMedia
* sourceMediaID
* status
* tape
* tapeInfo
* tapeInfoRef
* thumbnailIDs
* type
* underlyingType
* userID
* userName
Special Case: Media Variables:
Special non-standard naming:
* media_ID
* media_metadata <-- Note: not very useful formatting
* media_modifiedDate
* media_tape
Standard naming:
* altPaths
* archiveStatus
* aspectRatio
* audioFormat
* dataRate
* filePath / file_path / source_path / source_path_orig
* fileSize
* importer
* isMediaAvailable
* omfRef
* proxyPath
* qttc
* still
* tcFmt
* tracks
* videoFormat
Special Variables – These multiple potential formatting (only txt format available currently):
* end
* start
Special Case: Complex variables:
* history <-- Note: not very useful formatting
* importSource <-- Note: not very useful formatting
Special Case: Multi-formatting (only txt format available currently)
* duration
* in
* mediaEnd
* mediaStart
* out
User Fields
In the [Fields] section of vantage_workorder.cfg, you can add a mapping such as:
vantage_job_id = vantage.job.id (when using CatDVServer v7)
or
vantage_job_id = U54 (when using CatDVServer v6)
This will allow you to use 'vantage_job_id' in the csv field definition accessing the field, user field 54 (CatDVServer 6) or vantage.job.id (CatDVServer 7.)
Appendix E – Workflow Notes
Handling CatDV Metaclips in Vantage
Metaclips are a CatDV method for collecting certain multi-part camera media assets into a single playable clip. Examples of this are Panasonic P2, and Sony XDCAM HD media types.
CatDV Pro and Pegasus interpret these collections of assets when the media is loaded, however, since there is no one media asset, the path for a metaclip is expressed as:
/path/to/media/file.***
Where instead of a file extension there are 3 asterisks (***). Because this convention exists only in the CatDV eco-system, Telestream’s Vantage cannot use these paths to parse a metaclip for processing.
The work-around for this is to insert a pre-flight Worker action that creates a QT reference movie, which of course has a valid path, for the metaclip media.
NOTE: This requires a Worker Node system with Quicktime installed as well as the necessary codecs (http://calibratedsoftware.com) for playback. This is due to a strange QuickTime behavior whereby reference movies created without the correct codecs installed can result in incorrect codec information being sent to Vantage, affecting the resulting transcoded material.
Appendix F – CatDV Server API notes
CatDV Server & Vantage API connections
The Vantage API communication port may be set in the config file as noted in Chapter 4 above. Typically, this is set to port 8676 by default.
CatDV’s REST API uses the web port set in your CatDV Control Panel. Typically, this is set to port 8080 by default. See the CatDV Server documentation for more info.