Middleware FAQ

FAQ

This FAQ is organized by product. If you do not see your product listed here, please contact support for the PDF version of your documentation.

NSA Middleware General

What are the CatDV-related requirements for North Shore middleware?

This varies by product, so please contact NSA engineering for a pre-sale review. However, in general most middleware products require at least one Worker Node and access to the CatDV Web Client (CatDV's REST API is accessed through the Web Client.)

If you do not have any Web Clients already, you may add them easily, and there is a fixed Web Client that will meet your needs nicely for a reduced cost or an API License now that provides unlimited connections for middleware apps.

What are the CatDV proficiency requirements to be trained to operate North Shore middleware?

Due to the wide variances in CatDV user skills, North Shore has drafted a short list of the skills a user should possess in order to successfully operate our middleware apps. If any (or all!) of these are unfamiliar to you or your staff, we offer that you re-engage with your service provider to get additional training. Or, contact your reseller to purchase North Shore's CatDV Training Package before undertaking middleware training.

Users should be comfortable with the following functions and features in CatDV before booking training on the North Shore middleware.

• • CatDV System Overview

    • Are you clear about the function of each CatDV app and it’s location in their infrastructure?

      • e.g How to log in and monitor the Worker Node and 3rd party systems? (Archive, Transcoding, etc.)

  • CatDV Pro Client

    • Basic Functions

      • Ingest to CatDV

        • Using 2 methods

          • Drag and Drop

          • Import Directory / Quick scan

      • Create & Organize Catalogs

        • Do you understand why NSA mandates that a clip should never be copied between Catalogs, only moved?

      • Tag Metadata

      • Proxy Creation

        • With Desktop Client

        • With Worker Automation

      • Search

        • Are you comfortable with the Advanced Remote Query functions of CatDV Pro? (Search dialog box.)

        • e.g. In an archive system can you correctly search for assets which are on a specific tape?

        • Can you search for only assets imported on a specific day?

      • Filter

        • Can you filter catalogs or search results?

        • e.g. Are you able to filter a catalog or search results to show only assets of a specific file type, frame rate, or import date?

    • CatDV Web Client

      • Search (as noted above)

      • Tag/update metadata on a record or multiple records?

  •  CatDV Worker Node

    • Interface

      • Can you log into the Worker Node machine that runs the middleware?

      • Can you start and stop the Worker process?

      • Error reporting

        • Do you know how to find a specific job in the Worker queue to report or confirm an error to SBS/NSA?

    • Topology

      • Are you clear on what storage is mounted to the Worker and 3rd party systems? (Archive, Transcoding, etc.)

    • Logs

      • Do you know how to choose and export logs from the Worker Node if required by a support representative?

My reseller says that I need a Worker Node dedicated to my archive (or other) system. Do I need a Worker Node for each workflow? This seems excessive.

This comes up a lot, and like a lot of things in DAM... it varies.

The Worker Node requirement is not set in stone. However, depending on the archive and other workflow pace, you might need to a dedicated Worker Node for a high volume workflow (such as archiving.) To ward off issues later, we try to lay this out early in the planning (and budgeting) process.

For instance, archive systems often are sending thousands of files (read that, "actions") a day. If there are other automations to handle at the same time, then something will be delayed, either the archive jobs, or the other jobs.

Somewhere, someone got the idea that "a Worker Node" (meaning one) is the right starting number for a system. This is seldom the case in a system that's well designed and performing valuable automations. We typically would offer that a Worker Node would be needed for any workflow that is utilized heavily. Note that we at North Shore make no more or less money if the client buys more or less Worker Nodes.

If the client has no other automations in mind, than say archiving, then one Worker Node for system will suffice.

However (BIG one here) if the client has, for instance, a need to transcode a significant amount of proxy, then their Worker Node will be busy doing that. (A single Worker Node can typically create 12 to 48 hours of proxy per day depending on the media format, and the Worker Node hardware. We use 24 hours of footage per day as a planning guideline as of 2016.) So if a client often gets 20 hours of footage and needs to both make proxy and archive that footage in the same day, they will likely be disappointed with one Worker. It's all about, "How much do you want to do, in how much time?"

In most shops, 2 to 3 Worker Nodes are a good estimate and you can keep someone pretty happy with 2 in most cases. But I'd have a serious discussion about your automation needs (proxy and archive first) before spec-ing a number of Worker Nodes.

We have seen customers very happy with one Worker Node and we have seen customers with 2 dozen running to keep up with their automation needs.

Short answer, we recommend a separate Worker Node for all high volume and critical workflows but if you only have one, it will work, but performance may suffer.

What characters can I use in a file name to avoid problems?

The short answer for how to name a file is:

"Use standard alpha numeric characters (A-Z, 0-9) underscores and dashes."

View our article, "Invalid characters in file names", for more info.

Can I update my CatDV Pro system without affecting my NSA middleware or custom scripting?

As with many things, "it depends." All NSA middleware is sold "as deployed" on the version of CatDV and host OS installed at the time of integration. Updates and support for those updates are billable and can be covered by purchasing NSA professional services.

Many CatDV application updates are harmless and you can use the general guidelines here for planning.

• If the middleware is only triggered by a Worker Action, meaning there is no connection or plugin to the CatDV Pro or CatDV Server applications, updates to CatDV Pro and CatDV Server generally do not affect the middleware. All CatDV Worker Node updates should be confirmed with North Shore Automation.

  • Examples of this are the Storage DNA scripting, Compressor & Episode Worker Node connectors.

• If the middleware uses the CatDV Server API or the CatDV Server REST API, then updates to CatDV Server application should be confirmed with NSA.

  • • Examples of this are the YouTube uploader, Amazon S3 & Glacier Archiving app.

  • If the application or script uses Python or another programming language or runtime, make sure to consult NSA before updating your host OS as the installed versions may vary from OS to OS.

Fortunately most server-based systems are fairly stable and are typically not subject to major OS changes/updates, only security and bug-fix updates, but if you have a need to perform a major system update, please contact NSA support to determine whether or not your system needs re-deployment or modification.

When in doubt, email support@northshoreautomation.com

I have to backup/restore my system. Where is my middleware installed?

We recommend you speak to your reseller or NSA support before making any changes to your system but in the event of the need for a 911 backup, if you backup the following locations, you should be fine.

User home folder - (NSA middleware is typically installed in ~/Scripts directory but we recommend a full backup of the home folder for many other reasons, when possible.)

/LIbrary/NSA

/etc/northshore

If you have these folders, in addition to any Akomi config and data and CatDV settings/data, you will be able to restore the system to functionality.

I notice that NSA sells several scripting items that are listed for free on the Squarebox site. Why should I buy something that's "free"?

There are two situations in this answer.

For simple scripts such as our Compressor and Episode scripts, these are a simple bit of scripting that anyone with Shell Scripting experience can do. But if you, or your reseller, aren't familiar with the shell you can purchase these as services from NSA through your CatDV reseller. North Shore is offering a guaranteed outcome.

Even these simple scripts have subtle but critical differences from the posted solutions. Often, as in the case of Episode, NSA has identified inconsistencies in the product's command line syntax and has corrected them. Again, we are certain that a competent scripter/programmer, given time, will find and correct these. If you have the time and skills, we recommend you set these up for yourself. However, if you find yourself changing your workflow to accommodate these actions rather than coding around the problem, you might want to seek assistance.

Other workflows offered for free by SBS, such as the youTube upload are offered as limited capability "scripts" as examples of what can be done with Worker Node. In cases where there might be a need for more functionality, North Shore has created complete apps around these workflows. 

In the case of the YouTube app, the North Shore solution offers error logging, email notifications and a direct 2-way web API connection to/from YouTube, as opposed to using a less reliable and less flexible command line tools. Again these offer a guaranteed, supported outcome with training and a bit of workflow assistance.

CatDV2Vantage App

What versions of Vantage are required?

Telestream Vantage 6.3 SP3 or newer is required to use the CatDV2Vantage application.

Why isn't the monitoring app picking up my jobs?

Make sure that the Worker Node machine and the Vantage server's clocks are in sync.

This is because the NSA app uses an optimization to only check on automation jobs that have changed since the last check. Otherwise the system would hit Vantage for every job ever, every time.

Example: Monitor script sees that it’s currently 13:00 and checks for new jobs. None.

Vantage with a clock set to 12:58 gets a new job.

Monitor at 13:05 checks for all jobs happening after 13:00. New job at 12:58 is skipped.

NOTE: Also confirm that both machines are set to the same time zone.

CatDV2Episode Scripts

We upgraded our Episode from 6 to 7 and our Worker Node scripts no longer work.

Telestream Episode 7 changed the way their command line interface works. Please contact North Shore or your reseller to purchase services to update this script.

NOTE: Unless you have purchased a yearly support contract, simple scripts such as the Episode and Compressor scripts are delivered "as is" and require professional services to update. This is not necessarily the case with our middleware applications as many of them are sold with yearly support contracts that cover updates, etc. Check with your reseller or North Shore for support status.

CatDV Isilon Archive App

What do the error codes I'm seeing in my logs mean?

Here are the most common errors you will see.

Error Status 3 - Typically this means that it the operation can't continue because one of the files it needs to find doesn't exist or isn't readable. This is most commonly seen where an asset is imported into CatDV and then the original file is deleted or moved. Additionally, if the file is visible to the user, it may mean that the archiving application (typically installed on a Worker Node) does not have permission to read the file.

Status 4 - The path is not correct in CatDV. This is usually caused by a user or workflow error involving path mapping. For example this error can happen when a Windows path is sent to an operation expecting an OS X path or vice versa. Confirm that your path in CatDV is the correct format.

Sales Engineer - System Designer Notes

Connectors - Can North Shore build a connector from CatDV to another system? (BrightCove, Kaltura, Kontiki, Akamai, Ooyala, etc.)

In general, when exploring connectors for distribution workflows, you can use these guidelines as an SE/designer to gauge the scope/scale/likelihood of success:

• General Distribution Connector info

  • • The simplest deliveries involve sending a file with an accompanying XML file (often called an XML Sidecar.) These connectors use CatDV and/or NSA apps to style XML files and the Worker Node or a 3rd party transcoder to create the media file and then transmit the files using FTP or some other file transfer method. - 99% do-able. (See notes below for more info.)

• • If the system requires/provides an upload using a REST (or similar wev services) API, these are typically also very do-able, however the time and expense is greater. North Shore prefers this method in the long run due to increased stability and interactivity, but often will offer a phase one solution using a video and sidecar to start the process. - 99% doable

  • If the system accepts mRSS (Yahoo, MSN, Daily Motion, Verizon) spec the North Shore mRSS feeder for CatDV. 99.9% doable

• The following are guidelines specific to XML Sidecar upload workflows

• • If the system requires a single video asset, with a sidecar XML, these are typically 99% do-able.

  • If the system requires a single video asset, with a thumbnail or other related asset, with a sidecar XML, these are typically 75% do-able and require a bit of manual assist.

  • If the system requires multiple video assets, with a sidecar XML, these are typically 75% do-able.

  • If the system requires multiple video assets, and thumbnails, with a sidecar XML, these are typically 50% do-able and require a bit more manual fiddling.

How do I order a custom XML translation?

NSA offers two main types of XML translations. 

Simple Translations - Which are 1-to-1 XSLT files, in which all fields retain their exact values and are simply reformatted into the new xml schema. (Ex.1) These are installed into and run directly in a Worker Node action.

Advanced Translations - Which require data transformation such as date reformatting etc. (Ex.2) These utilize NSA's XML translation applications which work in concert with the Worker Node to complete more complex transformations.

Once you've determined what type of translation you need NSA project managers will collect examples of your output format with you and deliver a XML creation form where your CatDV database info will be collected and the XML output will be planned.

Advanced Translation Types:

These are the types of translations that are possible in the NSA advanced translation application.

1. Mapped from CatDV metadata with no data reformatting. These items have a corresponding field in CatDV such as USER47 and are written as is to the new XML.

2. Fixed. These items have a fixed value that is auto entered in each xml exported. This data never changes from XML to XML. A good example of this might be the company name in a delivery form. 

Ex. <submitter>North Shore Automation</submitter>.

3. Element. These items are fixed parts of the XML document and can denote elements or simply be a document component.

Ex. <clip>

4. Date Calculation/Re-format. These items are calculated as part of the export script. Either data read from a source such as a date from the machine's clock or a translation such as reformatting a date from CatDV's format to an ISO standard. (6/15/12 to 2012-06-16 for example.)

5. Concatenation. Pulling values from more than one CatDV field into a single item, such as USER47 - USER21 - USER33 which might look like "film - 03 - action". Concatenations should have the delimiters and format explicitly specified as in, "USER47 - USER21; USER33. Where we see value, space, hyphen, space, value, semicolon, space, value.

Ex. The values from CatDV USER 23 - USER17_USER5 are concatenated together into:

USER 23 - ProjectName

USER 17 - Damien

USER 23 - RoughCut

<info>ProjectName - Damien_RoughCut</info>

XML Translation Samples

EX.1 - Simple Translation

EX.2 - Advanced Translation

[insert xml creation template image]

What are the guidelines for custom db migration?

Here are some general guidelines to help gauge the likelihood of success of migrating data from an outside database into CatDV Server.

Can you deliver a standard text output file (CSV, TAB, XML, JSON?)

We try to avoid directly extracting from a db as this is most often time consuming and therefore cost prohibitive. It's best to start from a clean export of the data in one of the approved formats above.

Are you willing to work with us on or simply deliver the key pair mapping schema?

We need to know how each field maps into CatDV fields.

Does the data map directly to assets?

This means that the data has a 1 to 1 relationship between db records and assets. If not, then we can extract the data but you will need to map or copy the data to the records.

Example 1 - perfect mapping w/path - The client has a db or spreadsheet that contains a file path to an asset in each record. /Volumes/Media_Drive/Folder/other folder/file.mov

Example 2 - perfect mapping w/o path but GUID - The client has a db or spreadsheet that contains a reference to an asset in each record but no path. There is, however a clear way to associate the assets with the db records such as in the DB there is a barcode number (ABC123) and in each asset file name there is a clearly extractable example of that GUID (ABC123_file-name.mov)

Example 3 - perfect mapping w/o path or GUID - The client has a db or spreadsheet that contains a reference to an asset in each record but no path or GUID. Example record has info (Project ID:Project A Reference Number: ABC123) but no extractable correlation in the asset name (some-project-with-bad-name.mov) In this case, the client will be responsible for manually linking the assets/files to the records.

Example 4 - Non 1 to 1 mapping - The client has data that are records that contain more info that relates to more than one video clip so that for each record in the db, there are multiple clips. For example a db of video tapes where each barcode number refers to a tape that may have contained more than one video clip. (Barcode:ABC123 - assets named: file-1-ABC123.mov, file-2-ABC123.mov, file-3-ABC123.mov, etc.) In this case, the user would be responsible for linking the files manually unless the reference ID is easily extractable from the asset file name.)

Example 5 - Non 1 to 1 mapping, no path, no GUID - Give up and type. (Not an official response, but perhaps the best advice we can offer.)