MineMeld is an “extensible Threat Intelligence processing framework and the ‘multi-tool’ of threat indicator feeds. Based on an extremely flexible engine, MineMeld can be used to collect, aggregate and filter indicators from a variety of sources and make them available for consumption to peers or to the Palo Alto Networks security platforms.”

• Minemeld Edl Input Definition
• Minemeld Edl Inputs
• Minemeld Edl Input Tools

Mindmeld is licensed under the GNU General Public License.See the LICENSE file in the distribution for more information. Although Mindmeld's core search technology, the self-organizing neural mapper (or SONM) is patent pending, ars Cognita has chosen to make the patent available to the Open Source community.The technology contained in the Mindmeld source code may be used freely. MineMeld use cases. MineMeld supports a variety of use cases, with more being added each day by the community, including: Aggregation and correlation of threat intelligence feeds; Enforcement of new prevention controls, including IP blacklists. Evaluate the value of a specific threat intelligence feed for your environment. An an open-source tool, MineMeld was built to be extensible, allowing organizations to tailor the input, processing, and output of information for their environments. We have made the source code available on GitHub, as well as pre-built virtual machines (VMs) for easy deployment.

It was recently open-sourced by Palo Alto and can be found on Github.

In the past I used stackstorm as a source for External Dynmaic List (EDL) but in fact could use Minemeld (mm). Stackstorm (st2) is better documented and has bigger community. I thought that st2 is. MineMeld automates many of these manual processes so you can use indicators to dynamically enforce policy with your firewall or to investigate threats with AutoFocus. Three types of MineMeld. Make it possible to automate the flow of indicators from source to destination.

Essentially it can be used to grab IP/URL/Domain feeds from anywhere on the internet (a miner), aggregate and process the feed or feeds using regex if necessary (a processor) and output them in a format suitable to use in an External Dynamic List object on a Palo Alto firewall.

Technically the outputs can be used for anything you want but they work with dynamic lists on the Palo Alto’s out of the box.

I’ve only used MineMeld for a few weeks but I have a few feeds configured – I’ll go through the configuration of one of them now. It’s pretty straight forward but hopefully it’ll come in handy.

Blocking Tor Exit Nodes

In this example we’ll do the following:

• Configure the tor exit node (miner)
• Configure an aggregator (processor)
• Configure the output in a format suitable for your PAN firewall (output)
• Configure a new External Dynamic List (EDL) object on your Palo to look for the output you created in MineMeld
• Create a new security policy on the firewall to block outbound access to the Tor exit nodes.
• Confirm the EDL object on the firewall is being populated
• Confirm that traffic to Tor exit addresses are indeed being blocked

Let’s get started… if you don’t have MineMeld set-up already then you should probably do that first before continuing! You can download the .ova so you can use it in VMware (I have it set up on VMware workstation at the moment) or install it manually on Ubuntu (installing it manually is probably best for a production environment)

1. First let’s configure the Tor miner. This essentially sets up a process in MineMeld to go and grab the list of Tor exit nodes. Tor makes this information available publicly. As you can see, the format is not suitable for import just yet.Click Config in MineMeld. You’ll see a bunch of default miners, processors and outputs. I deleted all of them as they weren’t useful for me.
   Click the Add button and give the miner a useful name. From the prototype dropdown select tor.exit_addresses. There are no inputs. Click Save.
2. Now we want to set up a feed aggregator/processor. Click the Add button again and this time choose the processor stdlib.aggregatorIPv4Generic
3. Lastly we want to create an output. This is essentially a clean, formatted version of the raw IP addresses we saw in step 1. Click the Add button and give the output an appropriate name and select stdlib.feedHCGreen from the dropdown. Make sure you select the processor/aggregator as the input.
4. Commit the changes by clicking on the Commit button on the top left of the Config screen. Within a few minutes your Nodes page should look like the below. Don’t forget that if you are blocking the app-id tor on your Palo, MineMeld won’t be able to get the IP address list from the tor web server!If you click the tor-exit-nodes-output, you’ll see a feed base url field with a direct link to the feed which is now hosted on your MineMeld server. This is what we’ll use in the Palo next.
5. Now let’s create an External Dynamic List object on the firewall. Click Objects then External Dynamic List. Click Add and fill in the details – the most important is the feed url which is the one we looked at just above. Click Test Source URL which should report back a success message. If it doesn’t then ensure your Palo can access your MineMeld server).
6. Now we’ll create a security policy that will block all outbound access to this dynamic list; aka Tor exit node IP’s. Create a security policy as you normally would but this time put the new external dynamic list as a destination address. For example:
7. Now we want to make sure the EDL is being populated correctly on the firewall. Log-in to the CLI and run the following command:

   request system external-list show type ip name minemeld-tor-exit-nodes

   You should see something like this if the firewall is successfully pulling the information down from your MineMeld server.

8. Finally… time to test the block list to make sure we’re actually blocking requests to the Tor exit nodes. I attempted to initiate a few requests to a Tor exit node via http, https and ssh. As expected, they were all blocked by the firewall:

Related posts:

Brooks Harris
Second Edition - June 1995

CONTENTS - Guide to EDL Management

Contents (top)

INTRODUCTION

Edit Decision Lists (EDLs) are the primary method of transferring information about an edit project between editing systems. This is especially important in transferring Off-line decisions to the On-line assembly.

There are many subtleties about EDLs which can foil a successful exchange of information and spoil your day. This paper covers many important aspects of managing EDLs, concentrating first on aspects directly related to EDL exchange - EDL formats, reel names, and disk types. Sections on cleaning and tracing review these important tools and their use. Also included are a review of timecode, EDL format cross-reference tables, use of the AVID EDL Tool, and an 'EDL Check-list' - a brief synopsis of EDL 'gotchas'.

EDL Compatibilities

Edit Decision Lists provide the means of transferring edit information from one system to another. Since an EDL from a non-linear or off-line edit is destined for use in other systems (the ON-line) it is important to understand the capabilities and limitations of those systems so you may create appropriate information for them. This knowledge is critical both during the edit (especially assigning reel names during editing or digitization) and when creating and saving EDLs at the conclusion of the job.

The most important steps in the process: Before you begin a job Determine the EDL format you will need for ON-line and review its capabilities. As you edit Use appropriate reel names during editing or digitizing! The importance of this cannot be over-emphasized. When the edit is finalized Generate a EDL in the appropriate format with appropriate options. Save the EDL to the appropriate floppy disk type: A) Be sure the disk is properly formatted and initialized. B) Use a legal file name when saving. Keep these ideas in mind:

EDL formats (CMX, GVG, SONY, etc) are similar, but differ in many small (but important) ways, making them incompatible, but convertible.

EDL formats differ in their capabilities where REEL NAMES are concerned. This has important implications on how you should name source reels when editing or loading non-linear systems.

To make a usable EDL for a particular On-line session you must save the list on the correct type of floppy, in the correct EDL format, and with a legal file name for the disk type.

Edit Decision Lists and Formats

General EDL Description

An Edit Decision List (EDL) is a special kind of database. It is a list of 'events' that include the source to be recorded, and where to record it. In addition, it includes information about transitions (Cuts, Dissolves, Wipes), transition durations, etc. An EDL is saved on disc with a file name that must include the '.EDL' extension (MYEDL.EDL). This is how the ON-line controller identifies a file as an EDL.

Most EDLs are simple ASCII files, that is, files containing alphanumeric characters, or 'text-only'. This is true of the EDL formats of most manufacturers (CMX, GVG), regardless of the disk type they are saved on (CMX, GVG, DOS). ASCII EDLs can be viewed and edited with a word processor if the disk type can be accessed (most word processors can't read a CMX or GVG disk).

Sony saves the EDL as a 'JIS' file, which is a modified ASCII file. You can't see these lists with a word processor. However, the data contained therein can be converted to ASCII the EDL can be converted to be similar to the other manufacturers (with some exceptions).

The EDL formats (CMX, GVG, SONY, etc) vary slightly in the exact location of the 'fields' of information, and in the exact 'encoding' of these fields. These variations make each format incompatible - you can't load a Grass Valley format list into a CMX editor. They are similar enough, however, so the basic edit information can be converted from one to another.

As an example, here is a simple video edit in both CMX and Grass Valley:

CMX:

GVG:

Note that the spacing between fields is different. This illustrates one of the many subtle differences between list types. Sony is somewhat different still. This causes incompatibility - you need to generate an EDL of the correct format for whichever machine (editor) or program you are going to use. We will return to this format issue in following sections.

These format differences are not so different that we can't examine one format to understand the general nature of an EDL. The following examples are CMX format.

A simple cut is represented by a single edit line and an 'effect' (dissolve, wipe, key) is represented by two lines. In each line, the edit number comes first, followed the source reel name, the channel(s) to record, the transition (Cut, dissolve, etc), transition duration (if applicable), Source IN time code, Source OUT, Record IN, Record OUT. You may see a '*' ('R' in GVG) at the end of a line denoting 'recorded'.

An edit can have notes (any number) and other information for GPIs (General Purpose Interface), motion events, switcher event-memories, master/slave machines, etc.

Here are several examples (CMX format):

Cut to 'BLACK' on video and A1 & A2:

75 frame dissolve from reel 123 to reel 123B on video and A1:

30 frame wipe (pattern 001) from reel 002 to reel 003 on video only:

'Split' edit to reel 021, first to video-only, then video and A1, with two notes:

KEY- Background-reel 131, Foreground-reel 051, with freeze on reel 131, with two notes:

Reel names in EDLs

Reel names present one of the most error-prone aspects of EDLs. An edit with an improperly labeled source reel name can create mix-ups in EDL preparation and costly confusion in the ON-line session. Selecting proper reel names is a bit of an art (as labeling anything can be), but there are some rules that must be followed for the EDL to be trouble free.

The reel name we are concerned with is the one first assigned when digitizing in AVID (or other non-linear system) or loading a reel in conventional OFF-line. If production has already labeled the reels and these labels are inappropriate to EDL management, they should be re-labeled.

Reel Name Lengths and Character Sets

ON-line systems and their EDL formats accept different reel name lengths and character sets:

CMX 340:
3 characters, 0-9, that evaluate to numbers between 1 and 253, and this same set with a 'B' (4 chars).
'001', '01', and '1' are all the same reel.
'001B', '01B', and '1B' are all the same reel, but different from the set above.
'253' is legal. 'BBB' is illegal.

CMX 3400:
3 characters, 0-9, that evaluate to numbers between 1 and 999, and this same set with a 'B' (4 chars).
'001', '01', and '1' are all the same reel.
'001B', '01B', and '1B' are all the same reel, but different from the set above.
'999' is legal. 'BBB' is illegal.

CMX3400A:
CMX3600 :
8 character alphanumeric A-Z and 0-9 (no other chars)

GVG:
6 character alphanumeric A-Z and 0-9 (no other chars)

Sony 9100:
Minimum of 4 characters, maximum of 6. 4 char names that evaluate to a number from 1 to 9999, alphabetic names shorter than 4 characters are padded with trailing underscores '_' to 4 char, names longer than 4 characters are handled alphanumerically.

The use of the old CMX340 format is limited today, so this doesn't usually create problems, but the fact that modern CMX formats take 8 characters and the GVG format only 6 can create problems. If you have an 8 character reel name and output to GVG, the last two characters will be truncated! This is obviously dangerous.
Consider what happens when two reels named 'CAMERA10' and 'CAMERA20' are output as GVG - you get 'CAMERA' and 'CAMERA'. Bad enough that they're now named the same, but what if CAMERA10 is NON-DROP FRAME and CAMERA20 is DROP FRAME? - big mess.

Creation of 'B-ROLLS'

There is another extremely important point that impacts reel name selection - creation of 'B' rolls. AVID (and other non-linear systems) can dissolve from a tape to itself. This is, of course, a great feature of non-linear systems, but exactly how is this represented in the list?

The AVID (and others) take the very workable approach of labeling the duplicate reel as a 'B-ROLL', that is, changing the duplicate reel name to have a 'B' on the end. This resurrects the old CMX340 idea of 'B' rolls originally used to label scenes duplicated to another tape for dissolve purposes - the CMX340 ability to use 'B' in reel names, as indicated above.

(NOTE: Always select Dupes: 'Use Dupe Reel for each Source' in EDL TOOL. Don't use 'One Dupe Reel' except for very rare circumstances. See: Using AVID EDL TOOL)

This works great if the reel name is short- 'TAPE1' becomes 'TAPE1B' and 'TAPE2' becomes 'TAPE2B'. Everything is fine in both CMX and GVG. But what if the original reel name is 8 characters long and the system creates a B-ROLL of it? If outputting in CMX, the last character is replaced with a 'B'. 'CAMERA11' becomes 'CAMERA1B' and so does 'CAMERA12'. Again, big mess. If you were outputting this as GVG you'd get: 'CAMERB' and 'CAMERB'. Ugh.

So, what can be done? The lesson is:

• If you are planning to output CMX, use reel names no longer than 7 characters. This way, when a 'B-ROLL' is created, the final result is 8 characters long.
• If your output is to be GVG, use 5 characters or less.
• If you're not sure, use 5 characters or less.
• Never use anything but A-Z and 0-9.

WARNING: The AVID allows long, complicated reel name assignments. It is tempting to name a reel 'SCENES 10-20', but if you do, you're asking for trouble with the EDLs! It might be argued the AVID should not allow this, but since it does, it's up to the editor to use reel names that will work in the real world of EDL management.

Reel Names Rules

• A) Use alphanumeric: A-Z and 0-9 only (no other characters).
• B) For CMX use 7 characters or less.
• C) For GVG use 5 characters or less.
• D) If not sure, use 5 characters or less.

TIP: If you inherit a job with long reel names, you can use the option in EDL TOOL: 'Convert to numbers'. This will re-label all the reels in the list to conform to the old CMX340 specifications. A report of the reel re-assignments is included as notes at the bottom of the EDL. This assures no information is lost, but you will have to cross-reference the labels in the EDL to the actual tape names.

WARNING: It is especially tempting to use reel names like 'JOB 1' because media files belonging to a particular job can then be identified from their file name on media disks. But the space is illegal in most ON-line systems! If you need or want to maintain this sort of referencing, use a very short job code ('Amercan Airlines' becomes 'AA') and no spaces between the job code and the reel. Remember, use 7 characters for CMX and 5 characters for GVG.

JOB: 'Amercan Airlines'
Example Names

• AA001
• AA002
• AAPBX
• AACM1

Saving EDLs - Floppy Disk Types and File Names

In addition to using different EDL formats, ON-LINE systems use different disk types to read and write EDLs. The three most common are CMX and GVG (both are versions of a format called 'RT-11'), and DOS.

CMX and Grass Valley Floppies

CMX and Grass Valley use floppy disk formats based on the old RT-11 operating system, from which both systems evolved. These disks are actually DOS 720K 3.5 floppies with a special 'directory' structure. A digression into a bit of techno-babble may help clarify what's going on here.

All types of disk drives have two important types of information on them:

• A) the many 'tracks' where the actual file information is recorded (sort of like a Hi-Fi record) and
• B) a 'directory' structure which tells the computer operating system exactly where (which tracks) the files are recorded.

Historically, CMX and GVG used 8 inch and 5 1/4 inch RT-11 type floppies. When 3.5 inch floppies became popular in the PC/MAC world, both decided to use them. But to do so, they needed to make them compatible with the RT-11 operating system the machines were based on. To do this, they retained the DOS style 'tracks' and replaced the DOS directory structure with an RT-11 directory structure. Unfortunately, both manufacturers did this in slightly different ways, resulting in incompatibility between the two (and DOS, of course).

To the user, these types are different and incompatible. To complicate matters further, these disks are based on 720K DOS 3.5 floppies, NOT higher density such as 1.44K which is more prevalent in the modern PC world. You MUST use LOW-DENSITY 3.5 disks. These are usually referred to as 2DD (not 2HD (high density)).

Don't confuse a PC high-density 3.5 inch floppy with a MAC high-density 3.5 inch floppy. DOS and MAC disks are physically identical but the DOS tracks and directory are entirely different from the MAC tracks and directory.

By the way, an ASCII file of any kind (like an EDL) can be recorded on any of these disk types if you have the right hardware/software combination. It is quite possible to put a CMX EDL on a MAC disk, or a GVG EDL on a CMX disk, but, of course, the CMX or GVG will not read them. We have to avoid doing this!

Making Correct CMX and GVG Floppies

There are three steps to making a good CMX or GVG floppy:

• 1) Be certain you are using low-density 2DD floppies.
• 2) Format the 2DD floppy as a true 720K DOS disk.
• 3) Initialize the floppy as CMX or GVG.

Formatting

Formatting refers to recording the empty 'tracks' on the blank floppy. This 'cuts the grooves in the record', so to speak. If you are formatting with a DOS PC of some type, use the following DOS command:

C:>format a: /t:80 /n:9

(replace the a: with b: if your 3.5 inch drive is 'B:')

This command line instructs DOS to format with a very specific 720K track structure and will reliably make a good 'base' for RT-11 style disks (CMX or GVG).

Some 2DD disks are pre-formatted when you purchase them. If you do this, be certain they are formatted as 720K. If you're not sure, re-fromat them with this command.

Initializing

RT-11 (GVG and CMX) Initializing refers to writing the directory structure on the floppy. There are several ways to accomplish this for RT-11 (GVG and CMX):

• A) Use the ON-line machine to initialize the disk.
• B) On PC:
  • a) Use 'RT11A' Utilities (from Software Systems/Sierra Madre, 'freeware', available on Compuserve Broadcast Professional Forum, Library 2. (go BPF, LIB 2, RT11A.zip).
  • b) Use 'EDLMAX' (from Brooks Harris Film & Tape,Inc.)
• C) On AVID: Media Composer will ask how to initial a floppy when you are in EDL Tool and insert a floppy. You can use a correctly formated 720K DOS (as above) or a blank (no formating at all) 2DD disk. Note, do not use a DOS disk with a 'boot record' (formatted by DOS with a '/s' (system) switch!

Sony 9100 Use a standard formatted high density (2HD) DOS disk.

EDL File Names

There is one more aspect of saving an EDL properly - file names. If an application allows an illegal file name length (number of characters) or an illegal character into the file name the ON-line machine (or an other application, may not read it. This point is especially pertinent to RT-11 (CMX and GVG). Here are the rules.

RT-11 (CMX and GVG):

File names must be 6 characters or less (plus the '.EDL') and contain characters that are alphanumeric ONLY. This means A-Z and 0-9 - no spaces, no '-' no ',' etc. This is an important point - CMX or GVG disks with file names including any illegal characters will probably not load.

DOS (Sony):

File name length: 6 characters or less (plus the '.EDL'). The two last DOS charaters are reserved by Sony for file management. Legal Characters: A-Z and 0-9 and ~`!@#$%^()_-& Spaces and other illegal characters will be saved as '_'.

MAC (Not usually used for EDLs):

Macintosh 32 character file names can be almost anything. But remember - other systems can't do it.

ADVICE: Use only 6 characters and A-Z and 0-9. This is the lowest common denominator - use it for safety. Remember all EDL files have the '.EDL' extension.

WARNING: AVID will allow illegal characters into a CMX or GVG file name. Further, if the sequence from which the EDL was made was named something like 'CUT 1', the AVID will use a file name default of 'CUT_1.EDL'. The underscore is illegal and will cause problems for other systems trying to read it. BE SURE TO USE LEGAL NAMES!

Cleaning EDLs - a Review

Cleaning edit decision lists is an important part of EDL management. It is usually performed by a computer program external to the editor. It assures that the events in the EDL represent the material recorded on the edit master precisely and are organized in a meaningful way. While the need for cleaning is more obvious in conventional video editing it still has relevance to non-linear systems.

What is EDL 'cleaning'?

In conventional video editing, edits are recorded to tape and 'events' representing them are recorded in the EDL. As editing progresses it may become necessary to re-record a section of the master. This changes the contents of the tape and adds an event to the EDL, but it does not, generally, update the previously recorded event (or events) at that location of the master. This leaves a 'dirty' event (or events) in the list.

Over-records

The first most obvious example of a 'dirty' event is the situation where the tail of a previous event is over-recorded:

A previously recorded event:

Now the current event 'over-records' the tail of it:

Note that edit 002 goes in at 01:00:14:00, a second earlier than the out point of edit 001. Pictorially:

An EDL cleaning program would change these edits to:

Note that the source OUT and record OUT of edit 001 have been trimmed 1:00 earlier. The list (only two edits) is now 'clean' - the extra frames on the tail of edit 001 are removed and edit 002 follows it exactly.

Inserts

Another slightly more complex example is the situation where an event is 'inserted' into a previous event. First, a previous event exists in the list:

Now the current event is 'inserted' in the middle of it:

To clean this situation we need to make a new event. An EDL cleaning program would change these edits to:

The first edit has been broken in two by the second edit - edit 001s OUT points have been trimmed to match the IN of edit 002, and edit 100 (the third 'created' edit) is the remainder of edit 001 (the tail of it).

These two examples illustrate the essence of cleaning EDLs. Needless to say, it becomes rather complex as the subtleties of Video and 4 Audio channels, dissolves, wipes, keys, split edits, etc. are considered.

Why is cleaning important?

A clean list is important for several reasons:

• A) A clean list is very much easier for humans to read.
• B) A clean list is essential for ON-line assembly.
• C) A list must be absolutely clean if the events are sorted.

A 'raw' or 'dirty' EDL is virtually impossible to read. For any event you may be scrutinizing it is very difficult to see whether another edit somewhere in the list may have altered it. When you are looking for a shot, or using the list for logging purposes, etc. a clean list is a must.

A clean list is critical for ON-line assembly. If the assembly proceeded exactly the way the 'raw' list was created, it would work, but it is obviously inefficient to execute all the edits of the Off-line. A strictly 'sequential' (A-mode) assembly can be very in-efficient, especially when creative and technical requirements of ON-line are considered. Effects, audio mixing/levels, TBC 'painting' all effect the order in which events are selected for execution. With a clean list, the show will fit together no matter the order it is done. If the list is dirty, you have a nightmare.

Sorting EDLs is a useful tool for may purposes, especially optimizing EDLs for assembly. A list must be clean if it is to be sorted. If the list is dirty and the order of the events is changed, the EDL may be wrong. Consider what would happen if the two edits of the 'over-record' example, above, were to be reversed in order (edit 002 preceding edit 001). The assembly of those two edits would be entirely incorrect!

Tracing EDLs - a Review

What is Tracing?

The trace process refers to a program that can relate several tapes (via their EDLs) to one another. It can 'trace' or 'track back' into previously edited tapes which have been used as source tapes in another generation of edit master, producing an EDL which represents the final show as edits to the original reels and timecode numbers. This 'traced' EDL is used in ON-line assembly.

The underlying organization of a trace is created during the edit itself. As the edit proceeds, sub-master tapes are created from original source tapes. These tapes, in turn, are used as source tapes in successive generations of sub-masters until a 'final' cut is reached.

Along the way, an EDL is kept of each tape, and each sub-master is given a name which is used to refer to it as a source reel in each following EDL.

This on-going organization process in the edit room generates the raw data that the trace process will act on - EDLs and the tape names they contain.

Edit Organizations for Tracing

To illustrate the nature of tracing, consider two edit project examples, loosely called 'horizontal' and 'vertical'1.

Horizontal

In a 'horizontal' trace, sections of the show are edited as modules (say, the opening, part1, part2, and close) and these are strung together to make the completed show:

Vertical

A contrasting organization might be called 'vertical'- where each succeeding generation contains rearranged sections of the previous sub-masters:

Mixed Horizontal & Vertical

These two extremes can be combined any way required. In fact, most real-world edits will be more complex:

There is no restriction to how the edit can be organized. In general, the nature of the show, availability of source material, and politics all contribute to the way the edit organization, and hence, the trace organization progress.

Corresponding EDLs and the Tracing Process

As the edit proceeds, a set of EDLs corresponding to each edit master tape is maintained. These EDLs contain edits to original source tapes and edits to previous sub-masters, which may themselves contain edits to other sub-masters and source reels, and so on.

When the tracing program is applied to these EDLs, a 'traced' EDL emerges - it contains edits corresponding to the edits in the 'FINAL' EDL with the source tapes and timecode of the original source tapes:

The traced EDL is the desired result of the tracing process. It contains the same edits as the 'final' EDL, but the source information (the reels and timecode) now comes from the original source reels, rather than the sub-master they were actually recorded from. This is the correct EDL from which to ON-line the show.

Advantages of Tracing

Armed with the knowledge that tracing can do this, the editor can approach the project in a 'non-linear' manner, creating sub-masters of important aspects of the show, and combining them as needed as the editing progresses.

The use of tracing vastly extends the usability of conventional off-line editing and also provides an important tool to 'reach back' to information outside the edit room, like previously created masters or post-sync audio EDLs.

The trace process is a powerful tool. Sub-mastering elements of a show helps winnow out the chaff, work out ideas, present alternate cuts. Be creative:

• Create selects reels (SUB1 = horses, SUB-2 = NASA, SUB-3 = Charlies interview clips)
• Create modules (SUB-1 = hospital scene, SUB-2 = gunfight)
• Cut the opening after the body is complete
• Create alternate cuts

Let the show, the creative flow, and editing logistics drive your organization.

STRONG RECOMMENDATION: DON'T GO BACKWARD! Do not edit some part of CUT-3 to the actual tape (and EDL) of CUT-1. While the trace may find it, is very confusing to humans. If parts of CUT-1 and CUT-3 need to be combined, do it by creating CUT-4, then use that into CUT-5.

Advanced Tracing Examples

Tracing to Post-Sync Sound EDLs

Sometimes there is information from outside the edit room, that is, EDLs from other places. An important example is a 'post-sync' session of video dailies. Here, the lab transfers the picture to cassette and then syncs the sound by making edits from the Nagra to the cassettes, keeping an EDL of this session.

The dailies (cassettes) have picture and sound located at some timecode. Go ahead and edit, using the dailies as you will. When you're finished, trace to the post-sync EDL to create an audio-only EDL for audio assembly.

To do this, remember about reel labeling. Typically, the EDLs the lab has created with have reel names like SR1, SR2, etc. referring to the Nagra sound rolls. These are the reel names you want to get at.

There may be one or more EDLs, and they may or may not refer to a single dailies tape. In fact, EDLs from labs often have all the sync edits to all the dailies tapes combined. This is not as much of a problem as it might first appear, because typically dailies do not have repeating timecode. Usually each camera roll has a unique TC hour. (remember to request this in your dailies specs to the lab.)

A TRICK ! What we're going to do is the following:

• A. Trace out your 'FINAL' to produce a finished EDL. Save this for picture purposes.
• B. Merge the lab post-sync EDLs together.
• C. Change ALL the reel names in the finished edl to relate to the merged sound EDL,
• D. Trace to the sound EDL, audio-only, producing a sound-only EDL with the original Nagra timecode.

Minemeld Edl Input Definition

If there are more than one post-sync EDL, examine these to be sure they have no over-lapping record side timecode. If not, merge them to a single EDL using EDL-MAX in 'clean' mode (load each, appending them together) and save this EDL to disk. This will become a sub-master for the trace. We'll call it 'SOUND'. (If record timecode does overlap, you'll make up two or more sound sub-reel EDLs and treat them independently.)

Your 'FINAL' EDL has no reference to this new EDL. But we'll make it so! First, trace it out to the dailies, and save it. Next, reload this 'finished' EDL as 'final' to EDL-MAX/TRACE. Select audio-only (Include -> Video OFF). Change ALL the reel names in it to 'SOUND'. EDL-MAX has a function for this in 'EDIT': Change reel names - ALL.

Now all the edits in 'FINAL' refer to a sub-master called 'SOUND'. Load the merged post-sync EDL as a sub-master and designate it 'SOUND'. Trace it. All the sound edits of the finished EDL are replaced with the original Nagra sound roll names and timecode.

Et Voila, a first-generation sound EDL for audio assembly!

Coping With Very Large Traces

While EDL-MAX has a very large capacity (more than 6500 edit lines with 640K of memory, as a rule) there are may still be a project bigger than this. If the whole project cannot be loaded at once there are two methods to try.

• 1. Trace one or two levels at a time.
• 2. If that doesn't work, cut the 'FINAL' EDL in half, trace it in two parts, and recombine the two traced EDLs.

This is accomplished by careful selection of the EDLs, doing the trace and saving this to disk, then tracing again into this EDL. You'll have to think this sort of problem trough carefully as far as the sub-master designations and the best path to trace through. You probably won't run into this with EDL-MAX. 6500 edits is a really big show.

Doing a Trace - an In-depth Example

In using a tracing program, the operator provides it with input EDLs (EDL files) and the tape names (sub-master designations) applied to each of these EDLs during the edit. The tracing program takes several EDLs, each representing a particular edit-master, as input. One of these is the 'final' edit, the one we are interested in tracing. The others are sub-masters, edits used as playback sources included in the final master. The program then 'traces' each edit in the 'final' EDL, searching the list of loaded reels (sub-reel designations) for matches to reel names in the edits of the final EDL, and replacing the source information with the earliest reel and timecode found amongst the sub-masters. This results in a 'traced' EDL representing the final EDL as edits from the original (or earliest) source reels.

The assumption is, of course, that the submasters and the final edit are, in fact, related to one another. This is where the edit organization with respect to tape names is critical. The trace relies on the tape names in the EDL to find a sub-master (sub-master designation). It then searches the contents of the sub-master EDL for the edit at that timecode location, and searches again. In this way it 'digs' for the earliest edit in the chain. With this in mind, the editor can plan successive sub-masters and their labels. When a cut is approved ('final' becomes known), the trace can be performed.

A TRACING EXAMPLE

Let's examine this in more detail by example. To illustrate the concept behind TRACE in more detail, we'll follow an example through:

• 1) the process of recording successive cuts (making the sub-masters and EDLs themselves)
• 2) loading these EDLs to the TRACE program, and
• 3) walking through the process the program uses to produce the traced EDL.

This example is 'hand made' to be easy to read and to follow through the process. It is a legitimate, though unlikely, set of EDLs. Note that all edits are 10 secs long, all original reels have the same timecode hour as its name (reel 001 starts at 01:00:00:00, reel 002 at 02:00:00:00, etc), all edit masters start at 01:00:00:00, and all edits are video-only.

The explanation will walk through the tracing of only one edit, though there are five edits in the example lists.

Edit Organization and Tracing Path

Input EDLs

The multiple cuts (edits on separate tapes and their EDLs) are created in the following manner:

The first cut contains edits from original source reels 001, 002, 003, 004, & 005. It is labeled 'CUT-1' in the title, and will be referred to in following 'generations' as 'CUT1'. This will be its 'reel label'. It is saved to disk as 'CUT1A.EDL'

The second cut contains edits from 'CUT1' (the CUT 1 master played back as source) and original reels 003, 004, & 005. It is labeled 'CUT-2' in the title, and will be referred to in following generations as 'CUT2'. Again, this is its reel label. It is saved to disk as 'CUT2A.EDL'.

The third cut contains edits from 'CUT1' and 'CUT2' (both the CUT-1 and CUT-2 masters are used as playback sources), and, again, original reels 003, 004 & 005. It is labeled 'CUT-3' in the title and would be referred to in following generations as 'CUT3'. As it happens here, this cut, 'CUT-3' is our last cut- it is the one we want to 'trace'. It is saved to disk as 'CUT3A.EDL'.

Why the naming of reels and files is done this way will become more clear as we examine the trace process itself. The important part is to be aware that the reel label applied to a sub-master (like 'CUT1' & 'CUT2') is the way the trace process finds an edit.

Loading the Submaster EDLs and 'Final' EDL to EDL-MAX1

To load EDL-MAX (in TRACE EDL mode), select FILE > LOAD > SUB-MAST and navigate to the file. For our example:

Load the file that contains 'CUT-1' (CUT1A.EDL) and give it the sub-reel designation of 'CUT1'. Note this is the same name it is referred to as in the CUT-2 and CUT-3 EDLs, NOT the file name on disk. The disk file name is only important in terms of organizing your project, not to the trace. The trace needs reel labels - in the EDLs and as designations.

Load the other sub-master, 'CUT-2' (CUT2A.EDL), and designate it 'CUT2'. Again, the name it is referred to as in the 'CUT-3' EDL.

Now load the 'FINAL' EDL. In this case it is 'CUT-3' (CUT3A.EDL). There is no reel designation for 'FINAL' because this is where the process starts and it is automatically understood to be 'FINAL'.

We are ready to trace. In our example, 'audit' was turned ON to produce the audit notes seen in the traced EDL. In EDL-MAX, select PROCESS > audit > 0010PROCESS/DISPLAY0011. This cleans all the input EDLs and produces a traced EDL with audit notes.

The Tracing Process

Lets walk through the process the trace uses to do this for one edit, the first edit of 'FINAL'. This edit will be traced through two generations (CUT-1, and CUT-2) to find the original reel label 'REEL4' and its edit points. Refer to the following page.

• A) The trace process starts at the first edit of FINAL. This is the edit is going to trace. It first looks at the reel name of the edit and searches the list of 'sub-master designations' (as input by the operator) to find a match. If it does, it goes to that EDL. In this case it finds 'CUT2'.
• B) The trace process next searches the record timecode of the CUT2 EDL for a an edit that includes the source timecode of the edit it is tracing. Here, it finds edit 002 in CUT2.
• C) As in the first step ('A'), the program now tries to find a match with this edit's reel name ('CUT1') to the list of sub-masters. It finds 'CUT1'.
• D) Again, it searches the record timecode of CUT1 to find an edit including the source timecode of this last edit (edit 002 of CUT2). It finds edit 004 of CUT1.
• E) It tries again to find a match to this reel name (REEL4) in the designated sub-masters list. It fails because 'REEL4' is not a designated sub-master. So, this is the earliest edit. The 'trace' is done, and the information for the first edit of the TRACED EDL is known.
• F) The information from the source side of this last edit (edit 004 of CUT1) is copied to makeup the source side of the traced edit. This includes the reel name (REEL4) and the timecode.
• G) The record IN & OUT points of the edit being traced (edit 001 of 'FINAL', where the trace began) are copied to form the record side of the TRACED edit. This is the first edit of the Traced EDL and is so numbered 001.
• H) The trace process goes to the next edit of 'FINAL' and begins again..

Of course this example is simplified. Most edits from sub-masters will not point, as they are here. The program must keep track of all the off-sets t durations are correct. It must also handle the details of dissolves and tw all the audio channels correctly.

In the example, we have followed the process of tracing edit 001 of FINAL through two generations.The other edits in 'FINAL' show other trace paths.

* Edit 002 of 'FINAL' comes from 'CUT1'. In this case the trace will go directly to 'CUT1' to find Edit 003 to produce Edit 002 of TRACED EDL.

* The other edits in 'FINAL' (003, 004, & 005) are edits directly to original source reels. They simply reproduce themselves in 'TRACED EDL'- they need not be traced.

These illustrate that trace can be used in very flexible ways. There is no restriction as the order in which sub-masters and original reels are mixed and matched as long as the flow continues forward onto new tapes (and their EDLs). Note that sub-masters and final may have any number of edits. As long as the tape names in the EDLs and the reel designations are correct, trace will 'dig' for the earliest edit information.

APPENDIX

Minemeld Edl Inputs

Timecode - A Review

Timecode is an electronic signal which labels video frames on video tape. The ideas for the format were invented by NASA for telemetry tapes tracking space missions. The concept was adopted by EECO in 1967 for video editing, and, after many folks had built similar (but in-compatible) systems, SMPTE created the standard in 1969. The European Broadcasting Union adopted the standard and it is now called SMPTE/EBU (referred to simply as 'timecode').

Timecode may be recorded on tape in two ways:

• A) Longitudinal Time Code, or LTC, is recorded on an audio channel or a dedicated 'address' channel.
• B) Vertical Interval Time Code or VITC, is recorded in the vertical interval of the video signal itself.

These two formats are similar as far as the edit information they contain.

Longitudinal Time Code (LTC) is an electronic signal that switches from one voltage to another, forming a string of pulses, or bits. Each one-second long chunk of this signal is 'sliced' into 2400 parts (for NTSC) or 2000 parts (for PAL/SECAM). This makes 80 time code bits for each video frame:

NTSC 2400 bits/sec divided by 30 frames/sec = 80 bits/frame PAL/SECAM 2000 bits/sec divided by 25 frames/sec = 80 bits/frame

These 80 bits are given certain assignments according to the standard. There are these important groups of information:

• A) The hours/minutes/seconds/frames
• B) Drop-frame flag - Whether this timecode is Drop-frame
• C) Color-frame Flag - Whether color-framing is intended
• D) User bits - 'leftover' bits available for user assignment.
• E) 'Sync word' - Tells an electronic reader where the frame information begins & ends and which direction the tape is moving.
• F) 'Sync bits' - Help verify (together with the sync word) the position of the data as the tape moves.

VITC (Vertical Interval Time Code) format adds 10 extra bits to each frame label. These add the following to the LTC list, above:

• A) Additional 'sync' bits between data bit groups.
• B) Field bit (allows video-field indexing accuracy).
• C) Cyclical Redundancy Check Code (CRC), used for error detection.

The hours/minutes/seconds/frames is the primary information we are interested in. This data is Drop-frame, Non-drop frame or EBU (25 frames/sec, PAL/SECAM) as indicated by the 'drop-frame' flag.

The 'sync word' and 'sync bits' are the heart of how timecode works. As the tape moves, these bits instruct the electronic reader which direction the tape is moving, where the hours/minutes/seconds/frames, drop-frame bit, etc. are located and (if properly aligned on the tape) where the video frames themselves are located with respect to the timecode stream.

All of this is pretty clever, when you think about it. The tape is moving at an unknown speed in either direction as the machine is used, yet the timecode can be read at all times (given a good recording and properly functioning equipment). Note that, at microscopic levels, the tape is jumping and jittering across the reader heads, stopping, reversing direction, going into fast-forward, etc. The sync words and the design of the readers makes it all work! Cool.

NTSC timecode comes in two flavors - Non-drop frame and Drop-frame, and there is only one PAL/SECAM, sometimes called 'EBU'. These are usually displayed as:

Most sytems:
Non-dropframe: 1:00:00:00 - colon in last position
Dropframe: 1:00:00;00 - semicolon in last position
PAL/SECAM: 1:00:00:00 - colon in last position

SONY:
Non-dropframe: 1:00:00.00 - period in last position
Dropframe: 1:00:00,00 - comma in last position
PAL/SECAM: 1:00:00.00 - period in last position

NTSC - DROP Frame versus NON-DROP, and PAL (EBU)

Non-drop Frame

NTSC video actually runs at 29.97 frames per second, not 30 fps, as is usually implied. If you label all the NTSC frames incrementally, as with NON-DROP FRAME, a 01:00:00:00 label (a colon in the last position indicates non-drop frame) will represent 108,000 video frames (from zero). 108,000 NTSC video frames actually takes 01:00:03:18 to play back. In other words, 01:00:00:00 in NON-DROP frame is not truly 1 hour, but 1 hour and 3 seconds, 18 frames.

This leads to the inconvenient fact that timecode calculations do not represent real-time. 2:00:00:00 minus 1:00:00:00 equals 1:00:00:00 in non-drop frame, BUT the running time of this duration is really 1:00:03:18.

Drop-frame

Many people are very concerned with the actual running time. (Imagine a network making a 3 second & 18 frame error when broadcasting the Superbowl with advertisers paying $33,333 per second! (1995)) The drop-frame labeling scheme was designed to over-come the inaccuracy of non-drop frame labeling, and it is gospel in broadcasting.

The drop-frame format represents the NTSC running time accurately by 'dropping out' some of the frame labels - specifically, 2 frame labels at the end of each minute except each 10th minute. This way, a display of drop-frame 01:00:00;00 (semicolon indicates drop-frame) represents 107,892 video frames, or exactly 1 hour of real time from zero.

No actual video frames are 'dropped' (where could they go?) - only the timecode labels of the frames. At any point through the labeling sequence the timecode label is very close to the actual elasped time (from zero). Since the labels represent an accurate time-line, adding or subtracting drop-frame timecode results in an accurate calculation - 2:00:00;00 minus 1:00:00;00 equals 1:00:00;00 - one true hour of running time.

PAL/SECAM - EBU

Minemeld Edl Input Tools

PAL and SECAM video run at precisely 25 frames per second so the DROP frame/NON-DROP frame issue is meaningless. 01:00:00:00 in EBU represents 90,000 PAL video frames- precisely 1 hour of running time.

EDL Formats and Floppy Types

EDL Audio Channel Cross-Reference

GVG 6.0 CMX340 CMX3400 & 3600 SONY 9100
------------------ ------------------

IN EDIT CHAN 3&4: NOTE IN EDIT IN EDIT
V A1 A2 A3 A4 LINE FOLLOWING EDIT LINE EXTENSION
* A1 A A A1
* A2 A2 A2 A2
* A3 NONE AUD 3 A A3
* A4 NONE AUD 4 A A4
* * A12 AA AA A1A2
* * A23 A2 AUD 3 A2 A3
* * A34 NONE AUD 3 4 A A3A4
* * A13 A AUD 3 A A3
* * A24 A2 AUD 4 A2 A4
* * A14 A AUD 4 A1 A4
* * * A123 AA AUD 3 A1A2 A3
* * * A124 AA AUD 4 A1A2 A4
* * * A134 A AUD 3 4 A1 A3A4
* * * A234 A2 AUD 3 4 A2 A3A4
* * * * A1234 AA AUD 3 4 A1A2 A3A4
* V V V V
* * A1V B B VA1
* * A2V A2/V A2/V VA2
* * A3V V AUD 3 A A3
* * A4V V AUD 4 V A4
* * * A12V AA/V AA/V VA1A2
* * * A23V A2/V AUD 3 VA2 A3
* * * A34V V AUD 3 4 V A3A4
* * * A13V B AUD 3 VA1 A3
* * * A24V A2/V AUD 4 VA2 A4
* * * A14V B AUD 4 VA1 A4
* * * * A123V AA/V AUD 3 VA1A2 A3
* * * * A124V AA/V AUD 4 VA1A2 A4
* * * * A134V B AUD 3 4 VA1 A3A4
* * * * A234V A2/V AUD 3 4 VA2 A3A4
* * * * * A1234V AA/V AUD 3 4 VA1A2 A3A4

USING AVID EDL TOOL

Note - This applies to the Avid's 'EDL Tool' (now mostly obsolete), not the more recent 'EDL Manager'.

The AVID EDL Tool (from the Output pulldown) generates EDLs from the sequence currently in the
'record' monitor, applying the options selected in the EDL Tool and EDL Options window
(double-click on 'options' or select options from the EDL Tool hamburger).

Format: This is the EDL format to be generated. Be sure you choose the right one! (see
EDL formats).

Sort Mode: - The order in which the edits are output
A- Record IN
B- Source
C- Reel ID D- Source, Fx at end D- Reel ID, Fx at end

Show Lists: - Which list to generate
* Main List - Almost always!
* Dupe Reel List - A secondary list of the 'B-Rolls' required

Record Start Time: - The record-IN of first edit in list

* Sequence * Other: 01:00:00:00 Dupes: - How to treat 'dissolves to same reel'
* Dupe Reel for each Source - Almost Always!
* One Dupe Reel. - Special circumstances only

Show Info: - Info included as notes in EDL Comments Clip Names - Very useful, no reason
not to use it
Patch Info Audio Info Source Table - Be sure to use this if using
'Convert to numbers'
Reel IDs: - Which reel name to use in EDL Tape - Almost
always Soundroll - Use alternate name in 'soundroll' column of source BIN
Camroll - Use alternate name in 'camroll' column of source BIN Aux Source
1,2,3,4 - Use alternate name in 'aux source x' column of source BIN

Convert to numbers - converts all reel names to CMX340 style (001, 002, etc). A very useful tool if the reel names used during digitizing are inappropriate for the output EDL format (too long, illegal characters). Be sure to use Show Info: Source Table. This includes a cross reference of the new reel names as notes at the bottom of the EDL.

Other Options:

Optimize - Edits in the sequence that would normally be joined will not be. Use this if there are edits in the middle of a sequence with some important secondary command, such as motion effects.

Show Dissolves as Cuts - Useful for film-to-tape pull lists and some audio assembly lists

Show Wipes as Cuts - same as dissolves Include Black Edits - Almost always!

Start Events at: - The event number of the first edit. This is useful if you are outputting several lists and want to distinguish them by their event numbers (List1: events 1-35, List2 events 100-47).

Switcher: - outputs wipe codes in appropriate format. Important only if wipes are used extensively.

Timecode: - Which timecode in the source BIN is used.
Address (LTC) - Almost always!
VITC Sound TC
Film TC
Aux TC 1,2,3,4

Recommendations

* Ask your ON-LINE editor what he wants (and needs)! If you're unsure
about anything, send a test EDL to him and have him check it for obvious problems (wrong
disk or format type). This effort can save big headaches at the ON-LINE session.

* Always 'clear' the EDL and 'make list' again after changing selections. This
assures all parameters are updated. (If you don't, sometimes it doesn't all happen)

* Always use 'Dupe Reel for each Source'. ('One Dupe Reel' is only
useful for very special situations). Remember that dupe reel names have a 'B'
appended to them, and if your reel name fills the expected length (CMX - 8 chars, GVG - 6
chars) the 'B' will over-write the last letter of the reel name, confusing the
list. Again, when assigning reel name during digitization, use 5 characters if going to
GVG, 7 characters if going to CMX. If not sure, use 5 character reel names! (See:
'reel names')

* Don't use the 'Record Start Time: Other 01:00:00:00' option under normal circumstances. Change the start-time of the sequence, instead (select
the record monitor, then 'Show Clip Info' from the FILE menu). This way your
sequence matches the EDL and the ON-line. This can reduce confusion if revisions become
necessary later.

* Always Use a 6 character (or less) file name when saving to floppy.
Alphanumeric ONLY - NO spaces, '.', ',' or other strange characters!
Be careful not to allow the AVID to use its default file name, as this is not always a
legal file name.

* If you anticipate using some external EDL processor on the EDL (like
EDLMAX), output a V1234 list (all channels) in 'A' sort-mode (Record-IN). This
is the most comprehensive list. Do the channel selection, sorting and processing with the
external processor.

Contents (top)

EDL Check List

Nov 6, 1995