a discussion of NATO+3d modular
Jeremy Bernstein
August, 2000,
revised January 2001
revised again July 2001, and again August 2001

Here is the chapterized article, better for online reading

This article is written for the benefit of non- NATO.0+55+3d modular users, but might offer some insights for experienced users as well. I am going to assume a basic knowledge of video and computer graphics terminology. Although prior knowledge of Max will assist in understanding some of the examples, it is not assumed. If you are an old hand at Max, please excuse the efforts made on behalf of those who are not.

The continuing changes to this overview involve several developments. First, NATO is in a constant state of evolution, and since the initial article, there have been some upgrades and new objects. Second, my work with NATO has changed. When writing the original piece, I was focused primarily on (non-live) standalone work that incorporates image playback and manipulation. My use of NATO has shifted a bit since then, and I spend a lot of time working in live performance situations. Third, I've been exploring NATO from a development perspective, authoring numerous extensions to the system, mostly for my own use. In doing so, I've gained an enormous amount of understanding (some of it admittedly guesswork, but educated guesswork) about how NATO is operating behind the scenes. So, my perspective has changed a bit. Hopefully, the additions will prove useful.

A Little Background - Max/MSP

An understanding of NATO.0+55+3d modular is impossible without a basic understanding of NATO's underlying platform, Cycling '74's Max programming environment. Max was initially developed at IRCAM (http://www.ircam.fr/) by Miller Puckette, about 10 years ago, and has subsequently been taken over by David Zicarelli (first, through Opcode Systems (http://www.opcode.com/) and, more recently, through David's own company, Cycling '74 (http://www.cycling74.com/)).

Max was conceived as a toolkit or `kitchen` for electronic music (which, given the technology of the day, meant MIDI). It utilizes a graphical paradigm in which various `objects` (representing programming routines) are connected using virtual patchcords (indicating data flow). It's sort of like building a flowchart, or working with an analog synthesizer. Max comes with all sorts of basic objects to get MIDI information in and out, move numerical data around, transform the data in various ways and control some elements of the user interface. (If you don't know what MIDI is, it's basically control information for a wide variety of electronic devices, like sound synthesizers and lighting boards).

A couple of years ago, Cycling '74 introduced MSP, which extends Max by adding _audio capabilities into the same basic architecture. With MSP, Max became a fully configurable software synthesizer, sampler and sound laboratory. As with Max, functions are provided to get audio in and out (with support for numerous 3rd party sound cards), move audio data around and alter it with filters, numerical functions, whatever. Max and MSP can interact, of course, which means that the control structures built in Max can interface with the new audio objects -- the audio can be controlled and affected by MIDI data.

Max is extensible in 2 ways: if you create a `patch` -- your own configuration of objects that performs a particular task -- you can save it, and call up your patch as if it were a built-in function of Max; and if you know C, you can program your own `externals` -- optimized Max objects which perform whatever functions you could dream of. Which is where NATO.0+55+3d modular comes in...

What is NATO?

NATO.0+55+3d modular, authored by 0f0003.MASCHIN3NKUNST comprises a set of QuickTime externals for Max. Which is to say, NATO.0+55+3d modular (NATO from here forward) permits you to deal with any sort of QuickTime media (films, images, sound, QuickTime VR, QuickDraw 3D, Flash movies, etc.) from within Max in the same fashion as you can deal with MIDI or audio data using the built-in functions or MSP. NATO interfaces with Max in the same manner that MSP does -- seamlessly. MIDI and numerical data can be used to control any NATO function. Because it's Max, you can build complex structures around your QuickTime data that permit control at whatever level you prefer.

A little history: NATO began its life as a single Max external which accepted about 300 commands. NATO classic (still available) could load and playback QuickTime media, edit it, display it, generate QuickDraw and capture video (from an external source, or as generated by NATO) to disk. NATO _modular revised the paradigm, added hundreds of new functions and over a hundred objects.

[Sidebar: there were two big changes. The most obvious revision was the number of objects -- NATO modular is broken down into sets of related objects, each with their own specific function. You only use the ones you need to accomplish the task at hand (hence, modular). The bigger and less obvious change was the introduction of offscreen graphic worlds. NATO classic did all of its processing `onscreen` -- basically, the display space was the workspace. NATO modular does all of its work offscreen -- image data chugs invisibly through NATO objects until you `project` it onto a display object (sensibly called a screen, or `ekran`)].

As I write this, there are over 120 objects in the standard distribution of NATO modular (more on the extended distribution later on). These range in function from image generation, to media import/playback/editing, to digitizing (audio & video), to display and recording, to image analysis, to object tracking, to image processing (special effects), compositing, 3D and inter-object communication. There are also some more whimsical or utilitarian objects which perform tasks like shutting down your computer or changing the color of the Desktop. In terms of breadth, it's an exciting package.

What's the Big Deal?

As I said before, Max was envisioned as a toolkit. Which is to say, you don't pull Max out of the box, and immediately start playing with an interactive performance environment -- you have to build it first. So too with NATO. What you lose in `ease of use` is more than made up for in the flexibility of the system. What can you build with it? Here are a few examples of practical functions that are fairly straightforward to implement using NATO:

- VJ control environment: If you're using NATO as a platform for live video performance, you will want to develop some sort of control system. using Max and NATO, it's fairly simple to bring in external data (from a graphics tablet, or a MIDI controller, or a plain old mouse and keyboard) and process it as instructions to play video, trigger effects, adjust composition, etc.

- Sound to visual processor: If you've got MSP, too, you can analyze incoming audio data, based on frequency, amplitude or whathaveyou, convert it into numbers useful to NATO and then instruct NATO to trigger or alter images (or generate images on the fly) in response to the sound.

- Live video processor: Incoming video data (from a camera or a VCR or your cable TV feed, for instance) can be manipulated and then displayed on any connected monitor or saved to disk. With some of the extended distribution objects, you can route image data through your FireWire port, or send it out over the internet (please read the extended distribution section, below, for important information about these objects and their limitations).

- Autonomous collage generator: Specify a source folder, specify a destination folder, specify the number of collages you want. Any still images in the source folder are randomly opened in groups of two (or more), combined into a single image (using the whole original images or a (randomly) selected portion of them), and saved to the destination folder. In fact, you don't even need to write it yourself -- here's the (simple and sort of lame) one I wrote. Please enjoy.

- `Multimedia`: There's no reason why you can't use Max and NATO to build interactive or narrative work. Obviously, for some types of work, Director or Flash might be more suitable, but if you want to take advantage of the flexibility of Max and NATO, or if you want to have elements of the piece generated in real time (e.g. audio may be generated by MSP, video may be selected at random, or generated/manipulated stochastically), Max plus NATO is perfectly capable of just about anything (honestly, things can get a little messy, but if you don't mind a challenge, you can come up with some lovely work).

- Standalone works: Max, MSP and NATO permit the construction of standalone applications. Anything you build with NATO can be rolled into an application and distributed to the public -- the recipients don't need to own Max, MSP or NATO.

One of the ways in which I've worked with NATO involves the construction of applications that process incoming email (ok, to be totally up-front, I'm using AppleScript, too -- NATO won't check my mail for me). Basically, they import text data from the email and convert it into NATO instructions, which are used to generate QuickDraw images. The resulting pictures are then mailed back to the sender. A variation on this idea uses a webcam, which is directed to take a picture when incoming email is detected. NATO then creates a collage using the image, combined with text data from the email.

Let's Get Wet

I'm going to demonstrate NATO modular by doing a run-down of the major object classes, and by showing some quick and dirty usage examples. If you're not familiar with Max programming keep in mind, when reading the screenshots, that the events move from right to left and from bottom to top. Data flows through objects from the top -- the `inlets` -- to the bottom -- the `outlets`. I'll try to make the examples as clear as possible, but don't get too caught up in trying to decipher the screenshots.

Most of the NATO objects take NATO image data and commands in through one or more inlets, process/analyze the data, and output it from one or more outlets. Exceptions to this rule are image generators, which don't take any NATO image data in, and display/recording objects, which generally don't send image data out. NATO image data flows along the same patch cords that regular Max data does. SIDE NOTE: MSP has a special patch cord color which identifies audio data flow, to distinguish it from the regular Max data. It would be lovely if NATO did the same, but my understanding is that this is an accommodation which Cycling '74 would have to make -- 0f0003.MACHIN3NKUNST can't incorporate this feature on its own.

Display Objects

Since seeing your QuickTime media is the most basic application of NATO, we're going to start with the display objects. The display objects in NATO are referred to as `ekran`s. NATO's object-naming convention places a `242.` before the name of each object. Currently, there are six display objects -- `242.ekran`, `242.ekran02`, `242.ekran04`, `242.ekran07`, `242.ekran08` and `242.ekran09`. The screen shot below shows three of them.

The image on the bottom left is being displayed in a `242.ekran` window. The object itself is contained within the Max patch (on the left). When image information is sent to this object, and the object is `opened`, a window is created containing the image data. `242.ekran` has numerous features, including masking and matting, a variety of transfer mode settings (images may interact with the foreground, background, or opcolor of the window), clear mode settings, and several scaling modes (so that image data is properly sized -- or not -- to fit the window). It can be resized via commands (note the `dim.set` message attached to the object), or manually, by command-click-dragging on the lower right corner. It will report its size, fill the screen, hide the cursor or menubar, and will register mouse activity above it.

The big image in the top left is being displayed by `242.ekran02`. Although it appears to be just another `242.ekran` window, it's significantly different. Like `242.ekran`, `242.ekran02` is an object in the Max patch (on the right). However, `242.ekran02` permits images to be displayed directly on the Macintosh Desktop. If I put Max in the background, the image is still playing on my Desktop, covering up everything else. It also has a huge variety of commands available to control placement, size and most of the other options available to the basic `242.ekran` object. `242.ekran02` will also output a specified portion of the Macintosh Desktop (through its outlet), which can then be recorded to a file (using the `242.rekord` object).

Finally, within the Max patch itself, you can see `242.ekran04`. In this case, the in-patch image is both the object and the screen (Max calls this a user interface object). `242.ekran04` is useful for previewing images onscreen, or monitoring image output (if, for example, you're sending your video out to an external display, you can keep your eyes on your computer screen, and still know what's going on).

Although not shown, `242.ekran07` is just a simplified and optimized version of `242.ekran`. If you don't need masking and matting and can live without a few other features, you'll benefit from a faster display window. `242.ekran08` is an expanded version of `242.ekran04` which supports masks. `242.ekran09` adds transfer mode support to the `242.ekran04` family.

Display objects can be placed at any screen coordinates, which means that if you've got multiple displays attached to your computer, you can place your `242.ekran` window on the second or third display (which might be a projector, for instance) and tell it to fill the screen. There's even a NATO object (`242.monitor`) which will report the number of attached displays and their screen coordinates. Nice.

For the remainder of this article, I'll be using `242.ekran04` exclusively in the screenshots.

Media Import/Playback Objects

The `242.film` object is (arguably) the soul of NATO. It is NATO's QuickTime import, playback and editing object. With the exception of QuickDraw 3D films (for which `242.3d` is a better choice), `242.film` is responsible for pretty much every other file type -- films, stills, Flash movies, QuickTime VR, DV, MP3, etc. Click here for a complete list of supported file types.

In addition to basic import and playback, `242.film` accepts a huge number of commands for controlling playback and editing features, too many to cover here. But as a taste, you may adjust playback rate and direction and looping, and you've got total control over the film transport. See below for an example of a basic playback console.

Many of the commands are probably self-evident: read, dispose, start, pause and stop all do what you'd imagine. The loop command takes 3 possible arguments (the $1 indicates a variable -- numbers arriving in the inlet of the message box replace $1) -- 0, 1 or 2 -- which correspond to `no loop`, `forward loop` or `palindrome loop`, respectively. The `<<` and `>>` are rewind/fast forward commands -- the range of their effect is determined by the stepsize command, seen to their left. The rate command determines the playback speed. I've included it twice to demonstrate the two rate modes: the top mode uses a single integer to determine playback speed (x1, x2, x3, etc. or negative numbers for reverse-direction); the bottom mode uses two numbers -- the rate is determined by dividing the first and second numbers (so that you can get speeds between 0 and 1, for instance).

The `242.film` object accepts a `dounload` message, which allows it to read in files from internet URLs. It supports unlimited tracks per film, which can be played and edited independently. It is also capable of importing additional data into a loaded film at any specified point, selecting portions, deleting segments or frames, changing timescales, framerates or codecs. You can even load multiple films into a single film object and switch between them (at random if you like) as they play. Any changes you make to a film can be saved to the original or a new file.

By the way, all of those outlets at the bottom of `242.film` are used to report information -- film name, size, file path, loop-hit-end notification and much more. This permits you to set up complex processes (constructed with basic Max objects) that utilize this information -- probabilistic loopers, smart video shredders, whatever.

The example below shows another media import/playback object at work, `242.3d`.

Although NATO is NOT a modeler, it is capable of rendering and adjusting many of the characteristics of 3D object presentation. You've got control over the object position and scale, and the camera and lights. I'm personally fairly inexperienced with QuickDraw 3D, so I'm only going to mention this object, and show the texture mapping feature.

As you can see, above, I've set up two `242.3d` objects. They share a `read` command (which reads in 3dmf format files). The object displayed on the left ekran is the original, as read in by NATO. The one on the right has the film (shown above) projected onto it as a texture.

Image Generation Objects

Importing media is only one way to get started with NATO. Several objects exist which generate images directly from within Max. These include `242.fraktal` and `242.lissajou`, which produce fractal and lissajous patterns, respectively. `242.plasma` and `242.plasma2` make abstract, um, plasma patterns -- they look like this. Most notable, though, is `242.qd`, which is a full-featured QuickDraw synthesizer, like Max's built-in `lcd` object, but for NATO. The example below shows three of these objects at work.

The routine on the right is used to generate the displayed QuickDraw pattern with a single mouseclick. I include it to demonstrate a rudimentary sampling of commands understood by `242.qd`. With `242.qd`, you may create any QuickDraw shape (framed or filled), and display text in any font installed in your system, at any size, in any color or style.

Digitizing and Recording Objects

Getting live video into NATO is fairly straightforward. Hook up a camera or VCR to your computer (via USB, S-Video in, PC card, FireWire, etc.) and set up a `242.vdig` object. I've gotten video in to work on everything I've tried, from a 7100/80AV with the built-in AV card, to a Powerbook with FireWire or a CapSure card. USB and serial webcams work fine, too. If you have multiple input sources set up, `242.vdig` will tell you about them, and allow you to choose between them. This means that with multiple `242.vdig` objects, you can work with multiple video input sources simultaneously. If you've got a video card with multiple input modes, you may also switch between them.

As with most video input software, you can control the audio and video input parameters using the standard QuickTime configuration dialogs. These will permit you to set up codecs, color balance, gamma, etc. on input, if desired. You may also specify a portion of your digitizer's `window size` for use in NATO, so you can crop on the way in.

Once you've got your media coming in, you can record it to disk, using `242.rekord`. `242.rekord` is, in fact, a bit more than a basic recording object. You have control over basic parameters such as film dimensions and framerate, but you can also specify options such as samplerate (which is sort of like framerate plus, permitting you to create time-lapse recording routines) and timescale.

Once you've specified a folder to record into, using a command or through a dialog box, this object has the option of saving all subsequent record operations to that same directory, which is handy for building automated routines. `242.rekord` will also accept multiple image data streams and create either multitrack movies (each stream is a track), or record multiple movies simultaneously (each stream is a movie).

`242.rekord` only records QuickTime movies, which means that if you set it up to record only one frame at a time (stills), you get single-frame movies (.mov), rather than PICT or JPEG files. You can then re-load these .mov files into NATO and export them from the `242.film` object as PICTs, if you like (or use the QuickTime Player), and then convert them to JPEGs or PNGs in your utility of choice. But you unfortunately can't generate standard still image file formats directly from `242.rekord`.

Here's a patch I wrote which will capture a single frame to a movie, re-open it and save it as a PICT file.

Image Analysis Objects

NATO incorporates several objects for analyzing and generating numerical information from your films and images. Check out the two examples below.

The first example demonstrates the `242.histo` object, which generates histograms of pixel intensities for the red, green and blue channels. `242.histo` outputs 256 possible intensities for each channel (x-axis), and graphs the number of occurrences of that intensity (e.g. the number of pixels in the image with that intensity) along the y-axis. In this case, I'm sending the (x,y) data to a user interface object to graph it as the movie plays, but obviously, you could use the information however you like -- for instance, you might adjust audio amplitude or filter envelopes based on the histogram data.

In the second example, you can see the `242.argb` object, as well as some of the data output properties of the `ekran` objects. `242.ekran04` (in fact, all of the `ekran`s) can be told to report cursor coordinates and rgb color values (in a range of 0-65535) for a particular pixel. You can see the numbers displayed below the `242.ekran04` object. I am sending the cursor coordinates to the `242.argb` object, which reports pixel intensities (in a range of 0-255) for alpha, red, green and blue channels at a particular (x,y) location. These numbers can be used for anything: controlling sound mix based on relative channel intensities, adjusting framerate and gamma based on (x,y) location of the cursor within an `ekran`, etc.

In the next example, you can see NATO's object tracking capabilities. In actuality, NATO tracks colors, not objects, with `242.pupille`. It sends the coordinates of the tracking box (visible in the example, although it can be turned off) through its outlet. A side note: STEIM's BigEye also tracks colors, rather than objects, in case you were wondering. With simple images, like the one shown, NATO performs competently -- check out this output movie, which shows NATO tracking the moon. More complex images, in which multiple areas contain similar color information, confuse NATO. However, you can subdivide the original image using the `242.decoupage` object and process regions separately to deal with this.

In this example, I'm using the rgb color tracking shown above (in the `242.ekran04` example with the fish picture) to specify the color parameters for `242.pupille`. The `scale` object is being used to convert the relatively large range of the `242.ekran04` rgb value output to the more modest range (0-255) used by `242.pupille`. Note that I specify the red, green and blue values independently. The `fuzzi` command is used to specify a +/- range for the object, so that close colors will be considered part of the color being tracked. With the `fuzzi` parameter set to low numbers, NATO is extremely discerning, and can distinguish between very similar colors. The `fuzzi` parameter can be set independently for red, green and blue channels.

Two other products should be noted here, both of which were developed for object tracking within Max. Eric Singer's Cyclops, and David Rokeby's SoftVNS 2 are both advanced implementations of tracking technology. At the time of this writing, both packages are in late beta stages, but are available for testing. If your primary interest in Max-based visual technologies lies in computer vision, you'd be well-advised to investigate them.

Image Processing Objects

So, this is the fun stuff. The image processing objects available in NATO are very likely the centerpiece of any live video programming you're going to be doing. As with everything in NATO, they work in real time, and process incoming image data (playing from disk, or digitized on the fly) fairly efficiently (depending on the object, of course).

Given the number of processing objects, it would take far too long to go through each of them (more than half of NATO's objects are for processing images), so I'm going to limit this section to a brief discussion.

The processing objects range from the practical to the whimsical. They encompass everything from band-pass and -reject filters, to gain controls, to blur and sharpen objects, to image swirlers and noise synthesizers. Many of the effects bundled with NATO resemble basic Photoshop filters, and are only useful to a point, unless you really like that sort of thing. I've ended up writing my own effects (in C) and using them in preference to the built-in processing, for the most part.

As far as speed goes, NATO's image processors range from very efficient to very slow. Basic point/color processors are fairly quick (bandpass, bandreject, invert, etc.). More complex processors (blur, edge detect, displace) are quite a bit slower, and some processors are really unusable in real-time situations, at least on current hardware.

Each object takes multiple commands, only one or two of which are shown in the example above. Using many processing objects will eat up processor cycles and can drop your framerate down to a crawl. When building effect chains, you'll want to arrange it so that you can easily bypass objects you're not using. Nearly every NATO object has a `bypass` command which causes signals to pass through without change, so that's one method. Certain objects, though, such as the compositing objects (the `242.collage` series of objects), appear to slow processing down, even when bypassed using the `bypass` command. After a lot of trial and error, I've ended up building a network of gates that completely avoids unused objects and routines. This seems to work pretty well. Using only one or two processing objects at a time is completely practical, even with a lot of other Max activity going on, such as digitizing and recording.

[Technical side note: When image data is received by NATO objects, an offscreen graphics world is created in RAM. The incoming frame is copied to the gworld, and then read, analyzed, operated upon and written to an output buffer, before it's sent out of the outlet. This happens for each frame. When an object is bypassed, the processing function generally returns without any action being taken. However, variables are still declared, and the incoming frame is fed into a temporary buffer. Although the overhead may be minimal, with several objects in series, it can add up, even with all objects bypassed.]

Another usage of the image processing objects is demonstrated below. This example uses a couple of the channel operations objects supplied with NATO. In this case, the film is split into its rgb color channels using the `242.channel` object bank on the top left. These channels are reintegrated on the right using the `242.channel3` object. The `inlet $1 $2` message is turning the channel inlets off and on in response to random numbers. When the inlet turns off, the channel image `freezes` , causing the video channels to `slide` against each other when they reintegrate, since the channel information is being updated at different times. You can see the output of this patch here.

Compositing Objects

The `242.collage` series of objects round out the NATO distribution. With them, you can create composite images and films that would be impossible otherwise. Generally, when you send two images to a single object, like an `ekran`, you generate irregular flickering between the two competing image data streams, as they battle for the graphics real estate. The collage objects permit a smooth mix to occur by blending or substituting color information between multiple sources.

The example above shows the `242.collage2` object in action. All four of the collage objects (`242.collage`, `242.collage2`, `242.collage3` and `242.collage4`) operate on the same principle, but each permits a slightly different set of commands for specific usage needs. All of the collage objects (except for `242.collage4`) also accept commands from `242.matrix` objects which perform resizing, skewing, translation and rotation operations to the data on an `inlet-specific` basis. `242.collage` in combination with `242.matrix`, `242.decoupage` (for cropping) and judicious use of feedback can yield some beautiful effects unattainable through the standard NATO processing objects. Check out this short movie for a little example.

The key to the collage objects is the `copymode` command seen on the far right of the example. QuickTime (and NATO) features several possible transfer modes that dictate the way materials interact with each other and with the background color of the offscreen graphics world.

How well does it work? Functionally, it works fine. Practically, NATO's compositing is a little slow. The tradeoff is that NATO permits much more flexibility in terms of layers and materials permitted, as compared to existing packages, like Image/ine. With a fast computer, you can use the collage objects without too much trouble for realtime work, although your frame rate may slow down noticeably.

`242.collage4` was released to address some speed concerns with NATO's collage series. It's quite a bit faster, but doesn't feature the matrix operation support. If you require that, I'd recommend one of my objects, nato.matrix, which will provide the missing functionality.

Extended Distribution

If the built-in NATO objects aren't enough for you, you may purchase several `extended` objects for NATO. At the time of this writing, they include:

- 242.wto: video streaming objects
- 242.parazit: a plug-in host for Image/ine plug-ins
- 242.fireuire: several FireWire objects, which permit NATO to send various data (image & effects data, DV data, MSP audio data) through your FireWire port
- 242.o204: an internet data transfer (download/upload) object
- 242.axial: a directory/search engine utility
- 242.of02: a realtime, recursive multi-layer photoshop file to qt movie converter
- 242.obl!ke: a media preview object
- 242.gl: an OpenGL renderer
- 242.qtfx: a QuickTime effects host
- 242.nr+: some non-realtime objects
- 242.rna: NATO to MSP sound manager driver and associated software.

There are a few others, and I expect the list will continue to grow. I'd have to describe these extras (the ones I've used) as a mixed bag. Please read on for specifics.

242.wto package

I had the good fortune to use `242.wto` at a NATO `summit` in Rotterdam. These objects use freely available Max network objects (the Open Sound Control/UDP objects) to communicate over a network (local or remote) via UDP. The `242.wto` objects encode the NATO image data into a packet format compatible with the UDP objects and then decode them back (upon receipt) into the format that NATO works with. They integrate right into your existing patches without any hassle.

The first time we got `242.wto` to work, I was amazed. Admittedly, the person transmitting was sitting right next to me, so it wasn't as dramatic as it could have been, but nevertheless, video data was travelling over the network from his machine to mine. We were using a 10baseT hub, and framerates were still pretty good. Before too long, we'd built a 3-way sending/receiving/compositing program. As we increased the complexity of our programming, though, some of the limitations of `242.wto` became very clear.

The amount of data being generated by NATO is enormous. When that much data is packeted and sent into the ethernet port of a computer, it seems to completely overwhelm the machine. It's probably similar to a `denial of service` attack -- a massive flood of information. If all you are doing with `242.wto` is receiving data from a remote location and playing it, it seems to work fine. If you try to apply even modest image processing, compositing, complex signal routing or whatever, your machine will slow down to a crawl, hang for 30 seconds at a time or crash (at least this is how it happened for us). After adding the `242.wto` objects into a performance patch I was using, the patch became too slow to use. As soon as I pulled the ethernet cable from the back of my machine, speed and responsiveness returned.

The `242.wto` objects do allow you to set packet size and buffer size, and adjusting these will improve performance to some degree. We were never able to find reliable settings, though. My guess is that as faster networking becomes more common, the technique used by `242.wto` will work better, but I'm not a networking expert, and I don't know where the bottleneck is occurring.

Another downside of the `242.wto` objects is that they only work for point-to-point communication. You have to specify an IP address and port to send the data to. You may set up multiple sender objects (and transmit to multiple known receivers), or open multiple ports on a single machine (and receive multiple streams). And, of course, both the sender and the receiver need to have `242.wto` (a $74.47 value, at last glance). While `242.wto` does, as is claimed, perform `video streaming`, it does so in a highly limited fashion. In order to get NATO image data into a conventional multi-streamed web broadcast, you'd need to send the image data using conventional means, like an S-Video cable, to a machine capable of broadcasting the images over the internet (running a RealServer or Sorenson Broadcaster, for instance). `242.wto` wouldn't be necessary at all.

242.fireuire package

At the same time I played with `242.wto`, I had a chance to use the `242.fireuire` package. There are several packages available for purchase -- an `ultralux` version that contains all of the available FireWire objects, and various single-object packages. Two of the objects (`242.filmdv` and `242.fireuire`) are NATO-specific, one object (`242.fireuire~`) is for MSP, and the other two objects are for Image/ine (I won't be discussing those).

The NATO FireWire externals are for image _output. Image input via FireWire is supported in the basic package. All of the FireWire objects use a special shared library supplied with the package. This library permits multi-client / multi-application operation, which means that other applications can use the FireWire port at the same time NATO does (simultaneous operation causes flickering, though). Each instance of a NATO FireWire object counts as a client. Generally, if you're using `242.fireuire` (the object, not the package), it's the last object in the chain, and so there will be no flickering (any compositing you're doing will happen earlier in the signal network). Running multiple `242.filmdv` objects will cause flickering, since they cannot be composited.

- 242.filmdv: This object substitutes for the 242.film object, but is primarily intended to work with DV media. It incorporates the features of 242.film, but adds the functionality necessary to connect to the FireWire port as a client. With DV films, `242.filmdv` works well -- at normal rates, playing forward, it's fast and performs as advertised. If you begin to change playback rates or direction, things seem to fall apart a little -- the video begins to `choke` and stutter. This seems to be a limitation of DV media, rather than NATO, though (QuickTime Player has the same problem with DV media when looped back and forth, and it's not even sending image to the FireWire port).

If you like to misuse your tools, `242.filmdv` also functions with non-DV films, but has unpredictable results -- generally the color space of the film is completely wrong and the image has been multiplied and tiled by the time it hits your camera. Of course, this effect might be desirable... Results depend on the dimensions of the film and the codec used.

- 242.fireuire: This is the most exciting, and also the most disappointing object in the NATO extended family. `242.fireuire` is intended to send NATO image data (any image data played or generated by NATO plus effects, etc.) out to your FireWire device. Like `242.rekord` or `242.ekran`, it is placed at the end of data chain. If you want to record your work digitally via FireWire, `242.fireuire` seems like the perfect solution. The problem is that it doesn't work well.

Here's the issue: to send image data to your FireWire camera, NATO has to encode the data as DV. The encoding is extremely processor-intensive, and, on my 500MHz Powerbook, cuts my framerate down to between 1/2 and 1/3 of normal. You can simulate the effect by running this patch. Here's a side-by-side comparison -- this movie was recorded via the s-video port on my computer (into the analog in port on my camera) and this movie was recorded via `242.fireuire` (using FireWire input to the camera). I expect that as QuickTime's DV encoding becomes more efficient and as computers become faster, this process will be less taxing, and `242.fireuire` may become usable. Today, though, it's not, really. If you absolutely need FireWire image transfer, save your money and put it toward a Sony DVMC-DA2 media converter.

- 242.fireuire~: This external is used to send MSP audio out to your FireWire device. It works great, actually, within the stated limitations (sample rate must be 44.1KHz, I/O and signal vectors must be at 2048). Since the release of `242.fireuire~`, MSP version 2 has been released, permitting signal vectors of size 2, so, hopefully `242.fireuire~` will be updated to accommodate the change.

242.gl package

The 242.gl package is really quite nice. It includes an extensive offering of objects: around 20 externals which cover shape generation (cylinders, disks, cubes, lines, rectangles, spheres and the `242.glmesoderm` object which produces the default OpenGL shapes like, for instance, teapots), fog, lights, texturing (like `242.3d`, you can project any NATO image data onto the `surface` of an OpenGL shape) and positioning (offsets, rotations, POV).

I've used `242.gl` for a number things, including sprite animation (since `242.gl` objects are persistent, you can move them around and manipulate them without needing to redefine the entire scene, as you do with `242.qd`), and virtual space descriptions (creating OpenGL scenes, and projecting films into the `empty` space).

The above patch generates two OpenGL rectangle objects (`242.glrekt`), each being positioned and rotated by a `242.glmatrix` object. Also demonstrated is `242.gllight`, which is specifying the position of lights and reflectiveness of the scene and its objects, `242.glfog`, which is producing a reddish tint as the image recedes, and `242.gltexture`, which is using NATO image data as a texture map.

Overall, the `242.gl` package works well and seems fairly quick. I haven't used it to generate more than 4 or 5 objects at a time and add texturing, but it does that splendidly.

242.parazit package

242.nr+ package

One of NATO's initial shortcomings was its lack of non-realtime capabilities. There are now specialized versions of `242.rekord` and `242.buffer` (called `242.rekord.nrt` and `242.buffer.nrt`) which fill this gap. Essentially, these externals offer an extra mode, placing the object into non-realtime operation. In this mode, the object will record 1 frame at a time, and send out a message after each frame is done. You can then use that message to advance your process to the next frame. It works fine and, in fact, is so simple, it's a wonder that this functionality is not included in the basic objects.

You might also want to check out these gratis patches by xampl@hotmail.com. They provide a less elegant, but working, solution to the problem of frame-by-frame capture in NATO, and are free.

The Downside

Although I'm largely enthusiastic about it, there are a few areas where NATO could be improved.

- Documentation: While NATO comes with full documentation (over 500 pages of SimpleText instructions), it doesn't adhere to the Max standard of example-based help with tutorials, which gives it a steeper-than-necessary learning curve. That said, it's actually pretty simple to get a basic project off the ground. But a more comprehensive help system would be valuable, and assist new users immeasurably.

A couple of people have done some work in this area. xampl@hotmail.com has built a pretty comprehensive set of Max-style help files for NATO, which you can download here -- they make a great tutorial, if you want to explore the objects. Timothy Place, also, has begun an ambitious set of help files. And if you don't mind the text files, you can download modular.ref which permits you to open them up from within Max, instead of having to switch to SimpleText to check a command.

- Overdrive: Max has two modes, Overdrive ON and OFF. With Overdrive OFF, onscreen events and processing are given equal time, which can result in slightly inaccurate timing for music. With Overdrive ON, calculation and time-critical processing is done at the interrupt level, at the expense of onscreen events. Currently, NATO does not work properly with Overdrive ON, which is a real shame, because it limits the integration possibilities between NATO and MSP, which operates best with Overdrive ON. At present, if you want to use the two packages together, you'll have to live with slight irregularities in MSP-based musical timing, so that your video displays properly.

[Jeffrey Burns has written a patch which helps to `correct` these inaccuracies by substituting the Macintosh system clock as the timing basis for Max (instead of Max's internal clock)].

- Reliance on QuickTime: This is actually both an upside and a downside. NATO is cleverly built on top of QuickTime, which means that as QuickTime grows, improves and supports more image formats and codecs, so does NATO. That's the upside. The downside is that, right now, there are areas of QuickTime that truly disappoint -- speed of compositing (the copymodes) and DV media handling are the two main areas that bother me. I'd love it if these problem areas were addressed by overriding the default QuickTime routines with more efficient code.

- Speed: Most of NATO's basic signal processing routines are independent of QuickTime -- point processing is just math, handled by your computer's microprocessor. Many of the supplied objects are a little slow, and would benefit in both elegance and usefulness with some basic optimization. Such basic functions as compositing and matrix transformations are much slower than they ought to be. Cropping (`242.decoupage`), too, takes a lot longer that it should. I suspect that, in these cases, NATO's development team has chosen to rely on QuickTime toolbox routines instead of doing the not-so-hard work of writing their own, more efficient code. Since NATO is advertised as a `real-time` image manipulation environment, it would be sensible for the developers to pay some extra attention to runtime speed.

- Price: NATO is not cheap -- the basic NATO modular distribution costs $549.54. To put this into context, the bundle of Max and MSP together is $495. Simply put, NATO is priced far beyond other retail Max extensions, or even Max itself. Historically, the Max user community has been fairly small, and largely academic or artistic. Most 3rd party objects for Max have been free. I'll grant, though, that compared to many other video packages for the Macintosh, NATO is at the low end of the pricing spectrum.

Given the price of the core package, one might hope that the extended distribution externals were less expensive. They range from $74.47 to $274.47. Some of them (like `242.nr+`) would have been nice and logical additions to the main package, and others (like `242.fireuire` -- the object, not the package) don't work as well as their price tag might suggest. Still others, (like `242.gl`) are fabulous, and probably worth the money. In the end, though, functionality is functionality, and if you require it, it's available for the price.

I think that NATO's cost is a downside because it might be disuasive or prohibitive to a large group of potential users, most notably artists. The pricing slows its adoption and use by the very people who would most benefit a product like NATO (if you accept that good work made with a good product is ultimately good for the product). 0f0003.MACHIN3NKUNST does offer discounts on site licenses, institutional licenses and academic licenses.

Software Development Kit & Third Party Development

For me, the defining quality of NATO is that it's an open system. It's based around a shared library, which contains the imaging functions into which each object hooks. The NATO distribution includes a rudimentary software development kit, whereby any interested C programmers can extend NATO's functionality. Essentially, the SDK is little more than lightly commented source code for a couple of NATO objects, and a header file. With diligence, the committed programmer can divine what the largely undocumented code does. It's not terribly difficult, really, but the documentation could be improved.

The beauty of an extensible toolkit is that whatever functions the core system does not offer can be fabricated. With a little math (much of it freely available online) and a little effort, the expressive range of NATO can be exploded from broad to nearly endless. That's not hyperbole. At this point, I've written about 60 objects to further my work with the system -- some are personalized replacements for built-in functions, but most are idiosyncratic, completely personalized code, which I've been able to use to push my visual language further.

Several third parties have generously donated their efforts and, in some cases, their source code, to the NATO community. These include:
Trond Lossius (externals, patchers and source code)
Kurt Ralske (externals and patchers)
Luke Dubois (externals with source code)
Jeremy Bernstein (externals and patchers)
Timothy Place (patchers)
André Sier (externals)

More Personal

I trained as a composer. I've been using Max since 1992, off and on, and only in the last couple of years did I begin to approach a level of comfort which I'd characterize as fluency, although I still keep the manuals by my side most of the time. Before NATO came along, I mostly used Max and MSP to create tools, rather than pieces. If I needed an audio interface for live performance, I'd write a patch that used a MIDI fader board to trigger and control audio playback from my hard drive. If I had an idea for a particular sound treatment, I'd build a patch to realize it. But generally, the `work` -- making the pieces -- was done out of Max, using a sequencer.

One of my principal frustrations as an electronics-based artist has been the lack of `venue` for recorded music (apart from releasing CDs). I don't generally perform my work live. Until recently, I've mostly approached the problem by writing semi-flexible scores that accompany live performances of dance or theater, so that my `canned` music could be heard in an active space. With the introduction of NATO, the landscape shifted for me.

What NATO offers me is a new approach to venue. The idea of creating self-contained, distributable and DYNAMIC work that combines visual and sonic elements is irresistible to me (I'm also a photographer, and have been searching for ways of uniting the two art forms in my work). I've avoided Director because I've never seen results that had any more than just basic dynamism, by which I mean `performance deviations`. I want a framework in which ambiguous, shifting and poetic work can happen, under total control. Max is an ideal platform for this purpose, because it permits both serial and parallel processes, and is time-oriented -- the basic unit of Max is the millisecond (and, soon, the sample). Combined with NATO, Max finally permits me to invent my own venues -- intimate, digital performance spaces -- where my work can be more fully realized.

Where the combined platform of Max/MSP/NATO really excels is in the ability to create a synthesis of many different audio and video techniques. I can easily and efficiently combine on-the-fly sound generation with audio buffer playback, all layered on top of an existing audio track being read from the hard drive, while I cut in and out of video clips, perform realtime manipulations to a different set of films or stills and layer text over the whole thing, all under precise timing control. I tend to build a fair amount of semi-randomness into my work -- different audio, different video, different timing relationships. All of these possibilities are easily managed with Max (and therefore NATO). And NATO permits total control over the visual space. It's a mixed-media dream.

In the last several months, my use of NATO has undergone an important development, since I've been concentrating more on performance in preference to standalone, self-contained pieces. For this purpose, this platform is the only one I could imagine working in -- it's completely open and far more customizable than I need it to be. As my technical and aesthetic needs have grown, I've had no trouble accommodating them using Max/NATO. My next move will involve folding the self-contained work into a performance context. With Max and NATO, I expect I'll have no trouble.

For me, the greatest significance of NATO is that it has transformed Max from wonderful toolkit software, into a viable authoring environment, virtually overnight. Through NATO, I've rediscovered Max in a very meaningful way, and I'm using it to accomplish tasks I'd never considered before -- not just with video, but with audio and text work as well. It's caused me to reconsider my views about computer-based performance. Before NATO, Max was useful. After NATO, Max is indispensable.

Is it easy? No. It isn't (although it's easier than it sounds). Is the challenge worthwhile? Absolutely. With Max under my belt, NATO was pretty straightforward to learn (although I keep the manual by my side at all times), but I wouldn't suggest that it's easy. As with any deep software, it will take me years to fully grasp the possibilities, which is a feature I cherish.

Conclusion

NATO is unquestionably a flexible and extremely useful package of uncommon depth and breadth. From that perspective, it may be the most exciting software to arrive on the new media scene in years. It's not perfect, but In numerous ways, NATO has changed the ways I live with my work.

I've run across people who treat NATO as a religion. I think that's silly, but, in my experience, it may have become something of a way of life.

(Jeremy Bernstein, NYC, August 2000)
Revised January 2001, again July 2001, and once again in August 2001.

portions of this article were made possible by generous donations of time and indulgence on the part of several NATO operators, who graciously permitted me to mess around with their extended distribution objects. thank you con mucho amor.

feedback? Please feel free to email me at jeremy@bootsquad.com if you have any questions or comments about this article.

Links

memegarden (MAX & visual technologies mailing list)
55 (unofficial, but excellent, NATO.0+55 operators mailing list)

all materials on this site (text, images, etc.) © 2000-2001 Jeremy Bernstein

legal information