NSA CatDV Content Agent User Manual

Introduction

This document is provided in incomplete draft format, for review only, as of 02/16/2022.

Welcome to the North Shore Automation CatDV-Content Agent user and administration guide. This document outlines the functions, usage, and basic technical information of the application and its related systems. While we advise that all users read the entire manual, system operators may only need to view the first few chapters. A system administrator or installer will want to review the document in its entirety.

The CatDV-Content Agent application enables CatDV DAM systems to connect to Content Agent systems. By using the CatDV-Content Agent solution, CatDV can communicate with a Content Agent system to send assets to transcoding and other types of workflows, then record job-related metadata in the CatDV database.

This document will cover the following topics:

## Topics to be updated as draft is edited

Document Conventions and Assumptions

This document assumes knowledge of the CatDV and Content Agent software. If you need more detailed information on CatDV applications or Content Agent , please refer to their respective documentation.

In some cases (typically the overview sections), the term "Content Agent Action" may be used generically to refer to any workflow related operation such as transcode, analyze, or asset copy/move. When a specific type of operation is referred to, such as a transcode, it will be referred to by name.These systems are highly customized in most cases, so references to topology and system configuration are examples. Consider undertaking a system design with your CatDV and Content Agent integrator to determine the correct configuration for your organization.

Chapter 1 – User Operations

This section...

Chapter 2 - CatDV Metadata

This section lists all metadata fields in CatDV that are used for Content Agent operations and a description of their data.

The Content Agent metadata fields are created in the CatDV metadata during the system deployment and can be mapped to a details panel in CatDV or distributed amongst other panels, as needed. We recommend collecting these fields into a dedicated CatDV details panel for clarity.

In this example, the user has created a details panel called "Content Agent " in which the Content Agent and CatDV-Content Agent metadata are displayed.

Note: In the "Main View" of the CatDV window, you can set up a column display to view the Content Agent Status for every clip.

The metadata fields for the CatDV-Content Agent workflow are named in a way that aids in tracking status and troubleshooting the workflow.

The data in fields beginning with "Content Agent" are typically coming from the Content Agent server via API. All other fields are either updated by user input or by a Worker Node action (e.g., Content Agent Workflow State, Content Agent Log, etc.)

Note: There is one exception to this convention. “Content Agent Action” is a user input field that is set by the user to initiate the Worker Node action.

CatDV-Content Agent Metadata Fields

• • Content Agent Action–  Pick list metadata field to send a CatDV clip/file specific Content Agent workflow

  • Content Agent Job ID –  Job ID from Content Agent

  • Content Agent Percent –  Percentage of job progress in Content Agent; note that the percentage given by Content Agent is imprecise and may not indicate overall completion

  • Content Agent Log –  Log of statuses for each clip that is submitted to Content Agent

  • Content Agent Workflow Status – Job status reported directly from Content Agent

    • Sample Values

      • Complete

      • Failed

      • Waiting

      • StoppedByUser

      • Active

  • Content Agent Monitor Status – Job status interpreted by North Shore Automation’s CatDV-Content Agent app

    • NSA interpretation of the workflow state

    • Sample values:

      • Job Submitted

      • In Progress

      • Submit Failed

      • Job Failed

      • Complete

      • Job ID Missing

      • Workflow Initiating

      • Unknown State

  • (Optional) Content Agent Output Path –  This is provided in case an output path is needed for the workflow to be initiated. An example of this might be in a CatDV proxy creation workflow, where the proxy location must be derived in the Worker Node, using variables, and then provided to Content Agent so that the proxy is placed in the correct path to be detected by CatDV Pro.

    • Note: This field is optional and may not be used in all workflows

Chapter 3 - CatDV-Content Agent App

This section provides a functional overview of the North Shore CatDV-Content Agent application.

The CatDV-Content Agent application consists of two parts, an initiation application that is run via a Worker Node command line operation and the main application that communicates with both the CatDV Server and Content Agent Server as well as creating the Content Agent API call to submit the CatDV metadata to Content Agent .

CatDV-Content Agent Block Diagram

CatDV-Content Agent Sample Topology

Note: CatDV Worker Node is typically not hosted on Content Agent Server, but in some instances this may be necessary or desired. Worker Node and the CatDV-Content Agent app may run on macOS, Windows, or Linux Worker systems. CatDV-Content Agent handles all cross-platform path translations in its config file.

Workflow Overview

• 1. A user flags assets in CatDV to send to Content Agent by setting a metadata value in a field to “Create Proxy,”or similar. (These are usually named for the workflow that they will initiate).

  2. The CatDV Worker Node is triggered by a server query to send the asset Clip ID and workflow name/ID to the CatDV-Content Agent app via a North Shore application initiated by a Worker Node command

     1. Content Agent Action field is reset to blank once the Worker Node action runs.

     2. The CatDV-Content Agent sends a Content Agent API request

Note: CatDV asset metadata is inserted into the API request during the API send - All requests include source media path and CREF at a minimum in order to enable monitoring.

• 1. CatDV-Content Agent app sends the API request for the workflow which triggers the Content Agent workflow

     1. A unique Job ID is created for the asset by Content Agent

  2. North Shore CatDV-Content Agent app is monitoring workflows specified in the North Shore config file for new jobs. When the app detects a new job it retrieves the asset’s ClipRef from Content Agent, uses that to perform a lookup in CatDV and then updates the following metadata:

• • 1. Content Agent Workflow State

    2. Content Agent Job ID

    3. Content Agent Percent complete

    4. Content Agent Log

• 3. CatDV-Content Agent polls the Content Agent server every 15 seconds throughout the job (configurable setting). The job status is updated in CatDV for each asset as its job progresses.

• • 1. If there is no change in the job in Content Agent, the app does not send an update to CatDV. Only Content Agent is polled.

• 4. Outcomes

• • 1. Success

       1. Content Agent Workflow State is updated to “Complete”

       2. Content Agent percent updates throughout and shows 100% upon completion

       3. Content Agent Log is written

          1. 2022-01-26T23:17:21.749951 - af0c70ca-2046-408d-98e1-afe6620c927f - CatDV asset C341821A submitted to Content Agent Workflow (af0c70ca-2046-408d-98e1-afe6620c927f) with job_id = 76cecd34-ec64-4353-bd77-6cd64a851573

          2. 2022-01-26T23:27:25.611845 - af0c70ca-2046-408d-98e1-afe6620c927f - CatDV Asset C341821A Content Agent job: Complete

    2. Error - If a job fails, the status fields will display a “Fault” message. The Content Agent Log field will list more detailed error information.

• 5. Content Agent Percent displays percentage of completion at time of failure

• • 1. Note: This resets the next time a Content Agent job is run on the asset

• 6. Content Agent Log is written

• • 1. 2022-01-26T23:17:21.749951 - af0c70ca-2046-408d-98e1-afe6620c927f - CatDV asset C341821A submitted to Content Agent Workflow (af0c70ca-2046-408d-98e1-afe6620c927f) with job_id = 76cecd34-ec64-4353-bd77-6cd64a851573

    2. 2022-01-26T23:27:25.611845 - af0c70ca-2046-408d-98e1-afe6620c927f - CatDV Asset C341821A Content Agent job: Complete

    3. 2022-01-31T12:30:34.328167 - 4a858549-25e8-4ac3-b3b4-83e553eeb106 - CatDV asset C341821A submitted to Content Agent Workflow (4a858549-25e8-4ac3-b3b4-83e553eeb106) with job_id = 51506415-d8bb-434f-a27a-36c1b86d7096

    4. 2022-01-31T16:37:12.036988 - 4a858549-25e8-4ac3-b3b4-83e553eeb106 - CatDV Asset C341821A Content Agent job: Complete

1. Operation complete

Workflow Notes

IMPORTANT: North Shore defines a Content Agent workflow as an operation initiated by a single workorder, having a single Content Agent Job ID. Content Agent operations submitted from CatDV should be either submitted serially, in order, one after another, or if a chain of operations must be performed by a single CatDV Worker Node action, then those operations should be grouped in Content Agent. This is due to the fact that CatDV metadata should only be received from one Content Agent workflow at a time to avoid conflicts and/or race conditions and to keep the complexity of the CatDV metadata to a minimum.

Chapter 4 - NSA CatDV-Content Agent Deployment

Config File Overview

This section explains the settings contained in the North Shore CatDV-Content Agent config file.

You may open it for editing in any text editor.

It is located at:

macOS or Linux

/etc/northshore/config/content_agent_workflow.cfg

Windows

%UserProfile%\northshore\config\content_agent_workflow.cfg

C:\northshore\config\content_agent_workflow.cfg

In the sections below we outline the specific settings and options for each configuration file section.

Config File - System

This is the top section of the config file, showing settings mostly related to system-wide functions.

Note: In the config file lines that start with "#" are ignored and so if you want to add notes for yourself, you may do so by adding  hash (#) to the beginning of the

License - This is your software license key. It will be supplied by NSA and is locked to the machine's hardware.

Content Agent - This contains the network information for the Content Agent server. "Host" should be set to the hostname or IP address of the Content Agent server. "Port" should be set to 8731 by default, unless you have modified your Content Agent server API configuration.

NOTE: Please enter the IP address or a valid hostname in the configuration. "localhost" is not a valid option, even if the software is running on the Content Agent server.

Monitor - This section contains settings related to how the app monitors Content Agent and reports updates to CatDV Server. The "sleep_amount" is the polling time for checking the Content Agent job queue.

Workflows - This section contains a list of the workflows monitored by the app. Add additional workflows to the list by typing a descriptive name (of your choosing, i.e an alias) an "equals" sign and the Content Agent workflow which must match the name in Content Agent which is case sensitive.

Using the matching workflow name from Content Agent can be used when sending single CatDV clips and files like .mov or .mp4 wrapped media to Content Agent

Pattern: <alias> = <Content Agent workflow name>

If sending CatDV Metaclips which refer to multiple pieces of media that need to be "joined" together to a single output file like a mp4, mov or mxf wrapped media then the Content Agent Job ID needs to be pulled from the database using a Soup API client. This workflow ID is a string of letters and numbers and is unqiue to each Content Agent workflow.

Pattern: <alias> = <Content Agent workflow ID>

Filesystem - This section covers where logs are stored and where the PID (Process ID file) is stored. The .pid file location is needed for certain troubleshooting operations.

Fields - This sections lists the fields that are mapped from CatDV Pro in order to initiate and complete Content Agent jobs.

Fields

Note: Spaces in field names are converted to periods by CatDV Server upon creation.

Note: When configuring a system with CatDV Server v7+, the CatDV-Content Agent app uses CatDV database identifiers instead of USER field ID's. Contact your CatDV integrator or administrator for more information.

Config File - Workflows

This is the top section of the config file, showing settings mostly related to workflows.

Path Mappings - If your system contains macOS clients or Worker Nodes you will need to set path equivalencies so that the system can translate macOS paths to Windows paths for the Content Agent application.

Pattern: <incoming path root>;<translated path root>

See the examples in the image below and contact your system administrator for more info.

Note: Content Agent prefers UNC paths.

Appendix A – Troubleshooting

Asterisk (*) in timecode values from CatDV

Content Agent will not take 60 frame TC with an “*” in the TC display

https://documentation.apple.com/en/finalcutpro/usermanual/index.html#chapter=51%26section=5%26tasks=true

Apostrophes in timecode values from CatDV

Content Agent will not accept time code values with apostrophes in them. CatDV uses apostrophes as a visual reminder that timecode is 23.98/24 fps. This is a preference that can be turned off in the “General” preference pane in CatDV Pro/Pegasus.

Content Agent Errors

Content Agent Error: RenderSession:StartSession(): A problem occurred when processing the output version, error =The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.

If you see this error, your path and/or filenames are too long. This is a Windows limitation, unrelated to North Shore, CatDV, or Telestream. For more information on this error, see the following article from Microsoft for best practices.

https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath

Appendix D – Built in CatDV Fields for mappings

Special Fields

source_path –  Path mapping modified media path

source_path_orig –  Unmodified media path

CatDV System Fields

Simple

* ID

* bigNotes

* bin

* catalogID

* catalogName

* clipref

* duration2

* format

* fps

* good

* groupID

* groupName

* hidden

* importNotes

* in

* in2

* isEditable

* marked

* markers

* mediaEndMs

* mediaStartMs

* members

* metadata

* modifiedDate

* modifyDate

* name

* notes

* orientation

* out

* out2

* posterID

* rating

* recordedDate

* recordingDate

* seqItems

* seqNo

* sourceMedia

* sourceMediaID

* status

* tape

* tapeInfo

* tapeInfoRef

* thumbnailIDs

* type

* underlyingType

* userID

* userName

Special Case: Media Variables:

Special non-standard naming:

* media_ID

* media_metadata <-- Note: not very useful formatting

* media_modifiedDate

* media_tape

Standard naming:

* altPaths

* archiveStatus

* aspectRatio

* audioFormat

* dataRate

* filePath / file_path / source_path / source_path_orig

* fileSize

* importer

* isMediaAvailable

* omfRef

* proxyPath

* qttc

* still

* tcFmt

* tracks

* videoFormat

Special Variables –  These multiple potential formatting (only txt format available currently):

* end

* start

Special Case: Complex variables:

* history <-- Note: not very useful formatting

* importSource <-- Note: not very useful formatting

Special Case: Multi-formatting (only txt format available currently)

* duration

* in

* mediaEnd

* mediaStart

* out

User Fields

In the [Fields] section of content_agent_workflow.cfg, you can add a mapping such as:

   content_agent_job_id = content.agent.job.id (when using CatDVServer v7)

   or

   content_agent_job_id = U54 (when using CatDVServer v6)

This will allow you to use 'content_agent_job_id' in the csv field definition accessing the field, user field 54 (CatDVServer 6) or content.agent.job.id (CatDVServer 7.)

Appendix E – CatDV Server API notes

CatDV Server & Content Agent API connections

The Content Agent API communication port may be set in the config file as noted in Chapter 4 above. Typically, this is set to port 8731 by default.

CatDV’s REST API uses the web port set in your CatDV Control Panel. Typically, this is set to port 8080 by default. See the CatDV Server documentation for more info.