### Work in progress

This is a draft document which tries to give you an overview how new technologies will be integrated in aRts. Namely, it does cover the following:

- how interfaces work
- codecs - decoding of mp3 or wav streams in a form that they can be used as data
- video
- threading
- synchronization
- dynamic expansion/masquerading
- dynamic composition
- gui
- midi

This is work in progress. However, it should be the base if you want to see new technology in aRts. It should give you a general idea how these problems will be adressed.

However, feel free to correct anything you see here.

Things that will be use aRts technology (so please: coordinate your effords):

- KPhone (voice over IP)
- Noatun (video / audio player)
- aRtscontrol (sound server control program, for scopes)
- Brahms (music sequencer)
- kaiman (KDE2 media player - kmedia2 compliant)
- mpglib/kmpg (mpg audio and video playing technology)
- SDL (direct media layer for games not yet started but maybe nice)
- electric ears (author contacted me - status unknown)

#### how interfaces work

MCOP interfaces are the base of the aRts concept. They are the network transparent equivalent to C++ classes. Whenever possible you should orient your design towards interfaces. Interfaces consist of four parts:

- synchronous streams
- asynchronous streams
- methods
- attributes

These can be mixed in any way you like. New technologies should be defined in terms of interfaces. Read the sections about asynchronous streams and synchronous streams, as well as the KMedia2 interfaces, which are a good example how such things work

Interfaces are specified in .idl code and run through the mcopidl compiler. You derive the Interfacename_impl class to implement them, and use REGISTER_IMPLEMENTATION(Interfacename_impl) to insert your object implementations into the MCOP object system.

#### codecs - data-decoding

The kmedia2 interfaces allow you to ignore that wav files, mp3s and whatever consist of data streams. Instead, you only implement methods to play them.

Thus, you can write a wave loading routine in a way that you can play wave files (as PlayObject), but nobody else can use your code.

Asynchronous streams would be the alternative. You define an interface which allows you to pass data blocks in, and get data blocks out. This looks like that in MCOP:

    interface Codec {
      in async byte stream indata;
      out async byte stream outdata;
    };

Of course codecs could also provide attributes to emit additional data, such as format information.

    interface ByteAudioCodec {
      in async byte stream indata;
      out async byte stream outdata;
      readonly attribute samplingRate, bits, channels;
    };

This ByteAudioCodec for instance could be connected to a ByteStreamToAudio object, to make real float audio.

Of course, other Codec types could involve directly emitting video data, such as

    interface VideoCodec {
      in async byte stream indata;
      out video stream outdata;      /* note: video streams do not exist yet */
    };

Most likely, a codec concept should be employed rather than the "you know how to play and I don't" way for instance WavPlayObject currently uses. However, somebody needs to sit down and do some experiments before an API can be finalized.

#### video

My idea is to provide video as asynchronous streams of some native MCOP data type which contains images. This data type is to be created yet. Doing so, plugins which deal with video images could be connected the same way audio plugins can be connected.

There are a few things that are important not to leave out, namely:

- there are RGB and YUV colorspaces
- the format should be somehow tagged to the stream
- synchronization is important

My idea is to leave it possible to reimplement the VideoFrame class so that it can store stuff in a shared memory segment. Doing so, even video streaming between different processes would be possible without too much pain.

However, the standard situation for video is that things are in the same process, from the decoding to the rendering.

I have done a prototypic video streaming implementation, which you can download here . This would need to be integrated into MCOP after some experiments.

A rendering component should be provided that supports XMITSHM (with RGB and YUV), Martin Vogt told me he is working on such a thing.

#### threading

Currently, MCOP is all single threaded. Maybe for video we will no longer be able to get around threading. Ok. There are a few things that should be treated carefully:

- SmartWrappers - they are not threadsafe due to non-safe reference counting and similar
- Dispatcher / I/O - also not threadsafe

However, what I could imagine is to make selected modules threadsafe, for both, synchronous and asynchronous streaming. That way - with a thread aware flow system, you could schedule the signal flow over two or more processors.

This would also help audio a lot on multiprocessor things.

How it would work:

- The Flow System decides which modules should calculate what - that is
  - video frames (with process_indata method)
  - synchronous audio streams (calculateBlock)
  - other asynchronous streams, mainly byte streams
- Modules can calculate these things in own threads. For audio, it makes sense to reuse threads (e.g. render on four threads for four processors, no matter if 100 modules are running). For video and byte decompression, it may be more confortable to have a blocking implementation in an own thread, which is synchronized against the rest of MCOP by the flow system.
- Modules may not use MCOP functionality (such as remote invocations) during threaded operation

#### synchronization

Video and midi (and audio) may require synchonization. Basically, that is timestamping. The idea I have is to attach timestamps to asynchronous streams, by adding one timestamp to each packet. If you send two video frames, simply make it two packets (they are large anyway), so that you can have two different time stamps.

Audio should implicitely have time stamps, as it is synchronous.

#### dynamic composition

It should be possible to say: An effect FX is composed out of these simpler modules. FX should look like a normal MCOP module (see masquerading), but in fact consist of other modules.

This is required for aRtsbuilder.

#### gui

All GUI components will be MCOP modules. They should have attributes like size, label, color, ... . A RAD builder (aRtsbuilder) should be able to compose them visually.

The GUI should be saveable by saving the attributes.

#### midi

The MIDI stuff will be implemented as asynchronous streams. There are two options, one is using normal MCOP structures to define the types and the other is to introduce yet another custom types.

I think normal structures may be enough, that is something like

    struct MidiEvent {
      byte b1,b2,b3;
      sequence<byte> sysex;
    }

Asynchronous streams should support custom stream types

back to index