[LAD] need help mixing on the fly

[David Griffith]

For a couple years I've been trying, without success, to reliably mix two
audio streams on the fly in C. I'm using libao, libmodplug,
libsamplerate, libsndfile, and libvorbis. The program is Frotz
(https://github.com/DavidGriffith/frotz), a Z-machine emulator for playing
old Infocom games as well as newer ones. Could I please get some help
with this?

[Fons Adriaensen]

Post by David Griffith
For a couple years I've been trying, without success, to reliably
mix two audio streams on the fly in C. I'm using libao, libmodplug,
libsamplerate, libsndfile, and libvorbis. The program is Frotz
(https://github.com/DavidGriffith/frotz), a Z-machine emulator for
playing old Infocom games as well as newer ones. Could I please get
some help with this?

Mixing audio from two audio streams just requires adding the streams
sample by sample. I'm pretty sure that the real problem here is not
mixing but something else. But with the limited info you provide we
can only guess.

Where are the streams coming from ? What are the formats ? Where is
the result supposed to go ? etc. etc. etc.


Ciao,

[David Griffith]

Post by Fons Adriaensen

Post by David Griffith
For a couple years I've been trying, without success, to reliably
mix two audio streams on the fly in C. I'm using libao, libmodplug,
libsamplerate, libsndfile, and libvorbis. The program is Frotz
(https://github.com/DavidGriffith/frotz), a Z-machine emulator for
playing old Infocom games as well as newer ones. Could I please get
some help with this?

Mixing audio from two audio streams just requires adding the streams
sample by sample. I'm pretty sure that the real problem here is not
mixing but something else. But with the limited info you provide we
can only guess.
Where are the streams coming from ? What are the formats ? Where is
the result supposed to go ? etc. etc. etc.

The audio comes from AIFF, OGGV, and MOD files which are embedded in an
IFF container file which also contains game data. Using libmodplug,
libsamplerate, libsndfile, and libvorbis; these audio chunks are turned
into stereo streams of floats. There are only two streams. If two
streams are active, mixing is done. The resulting single stream of floats
is converted to pcm16 and fed into libao for output.

The two streams are either "music" or "bleep". The music stream can be
fed with audio data from an OGGV chunk or MOD chunk, but not both. The
bleep stream is fed from an AIFF chunk. If a new stream of the same type
is started, the new one immediately takes over the old one. This works.
If both types are played at once, they're supposed to be mixed. That
process doesn't go right and I don't know where or why, though I suspect
trouble with threads, mutexes, and/or semaphores. The problem manifests
in distorted sound and usually a segfault.

More detail...

A mixer thread is spawned when Frotz starts up. It waits for one or both
float buffers to fill whereupon it mixes their contents, converts the
result to pcm16 and calls libao to play it. playaiff() or playmusic() are
spawned by the main thread as separate threads to read audio data from the
container IFF and fill up the float buffers for the mixer to read.

The defective code is in
https://github.com/DavidGriffith/frotz/blob/ao-curses/src/curses/ux_audio.c

I have created a test game to be loaded by Frotz for testing audio
development. It's at http://661.org/soundtest2.blb. The game is a text
adventure that simulates a single room which contains a Commodore 64, a
Commodore Amiga, and a small box with buttons on it. Turning on the C64
causes an OGGV file to play. The Amiga plays a MOD file. The
buttons on the small box cause AIFFs to play. To tickle the bug, turn on
one of the computers and then press a button on the box.

I'm not interested in using SDL or Pulseaudio for this project.

[David Griffith]

Post by David Griffith
I have created a test game to be loaded by Frotz for testing audio
development.

Oops. I forgot to add an URL to the test game:
http://661.org/soundtest2.blb

[Fons Adriaensen]

Post by David Griffith
A mixer thread is spawned when Frotz starts up. It waits for one or
both float buffers to fill whereupon it mixes their contents,
converts the result to pcm16 and calls libao to play it. playaiff()
or playmusic() are spawned by the main thread as separate threads to
read audio data from the container IFF and fill up the float buffers
for the mixer to read.

/me looks at code...

This is all so horribly wrong that I don't know where to be
begin. Honestly, I'm surprised it works at all (assuming it
does, with one source).

But the main error seems to be the way the audio_full and
audio_empty semas are used. If there are two producers,
audio_full is posted twice. And then the two producers
are waiting on the same sema, audio_empty. This can't
work ever. It's not some small bug -- the entire logic
and structure of this code is just wrong.

Apart from that, if this code is meant to be Linux only,
consider using ALSA or Jack. If not, consider something
like Portaudio. OSS and everything built on it have been
deprecated for at least 12 years now.

Ciao,

[David Griffith]

Post by Fons Adriaensen

Post by David Griffith
A mixer thread is spawned when Frotz starts up. It waits for one or
both float buffers to fill whereupon it mixes their contents,
converts the result to pcm16 and calls libao to play it. playaiff()
or playmusic() are spawned by the main thread as separate threads to
read audio data from the container IFF and fill up the float buffers
for the mixer to read.

/me looks at code...
This is all so horribly wrong that I don't know where to be
begin. Honestly, I'm surprised it works at all (assuming it
does, with one source).
But the main error seems to be the way the audio_full and
audio_empty semas are used. If there are two producers,
audio_full is posted twice. And then the two producers
are waiting on the same sema, audio_empty. This can't
work ever. It's not some small bug -- the entire logic
and structure of this code is just wrong.
Apart from that, if this code is meant to be Linux only,
consider using ALSA or Jack. If not, consider something
like Portaudio. OSS and everything built on it have been
deprecated for at least 12 years now.

I admit that I don't have a good handle on the way the producer/consumer
setup works. I got it with one producer and one consumer. With two
producers, I got lost.

The code is meant to be portable across at least Linux and BSD. That's
why I picked libao for output.

[Julius Smith]

Hi David,

Attached is the output from

faust -a sndfile.cpp tmix2.dsp > tmix2.cpp

for the Faust program (tmix2.dsp):

// two-channel audio mixer:
gain1 = hslider("Gain1",0.5,0,1,0.001);
gain2 = hslider("Gain2",0.5,0,1,0.001);
process = _, _ : *(gain1), *(gain2) :> _;

(See tmix2.cpp attached)

Cheers,
- Julius

Post by David Griffith
For a couple years I've been trying, without success, to reliably mix two
audio streams on the fly in C. I'm using libao, libmodplug, libsamplerate,
libsndfile, and libvorbis. The program is Frotz (
https://github.com/DavidGriffith/frotz), a Z-machine emulator for playing
old Infocom games as well as newer ones. Could I please get some help with
this?
--
David Griffith
_______________________________________________
Linux-audio-dev mailing list
http://lists.linuxaudio.org/listinfo/linux-audio-dev