How to send system audio via WebRTC?

[nodminger]

Is it possible to send system audio using SendAudio Source Code?

[nodminger]

I have found that by using Stereo Mix in windows or by installing a virtual audio driver we can share system audio. Is there any alternative way to share system sound, I am asking this because in some system stereo mix will not be available. Is it possible to capture sound using C# program and send it via WebRTC?

[DavidMartynWood]

One approach to this would be using WasapiLoopbackCapture from the NAudio library.

https://github.com/naudio/NAudio/blob/master/Docs/WasapiLoopbackCapture.md

[nodminger]

@DavidMartynWood, thank you for your response. I've reviewed the NAudio documentation, but I'm having difficulty figuring out where to begin with the implementation. Specifically, I'm looking for guidance on how to start or which aspects I should prioritize when integrating NAudio into the InitialiseSource() method from the AudioDecoder.cs file in the SipsorceryMedia.FFmpeg library.

public unsafe Boolean InitialiseSource(int clockRate)
{
    if (!_isInitialised)
    {
        _isInitialised = true;

        _fmtCtx = ffmpeg.avformat_alloc_context();
        _fmtCtx->flags = ffmpeg.AVFMT_FLAG_NONBLOCK;

        var pFormatContext = _fmtCtx;
        if (ffmpeg.avformat_open_input(&pFormatContext, _sourceUrl, _inputFormat, null) < 0)
        {
            ffmpeg.avformat_free_context(pFormatContext);
            _fmtCtx = null;

            RaiseError("Cannot open source");
            return false;
        }

        if (ffmpeg.avformat_find_stream_info(_fmtCtx, null) < 0)
        {
            RaiseError("Cannot get info from stream");
            return false;
        }

        ffmpeg.av_dump_format(_fmtCtx, 0, _sourceUrl, 0);

        // Set up audio decoder.
        AVCodec* audCodec = null;
        _audioStreamIndex = ffmpeg.av_find_best_stream(_fmtCtx, AVMediaType.AVMEDIA_TYPE_AUDIO, -1, -1, &audCodec, 0);
        if (_audioStreamIndex < 0)
        {
            RaiseError("Cannot get audio stream using specified codec");
            return false;
        }


        logger.LogDebug($"FFmpeg file source decoder {ffmpeg.avcodec_get_name(audCodec->id)} audio codec for stream {_audioStreamIndex}.");
        _audDecCtx = ffmpeg.avcodec_alloc_context3(audCodec);
        if (_audDecCtx == null)
        {
            RaiseError("Cannot create audio context");
            return false;
        }

        if ( ffmpeg.avcodec_parameters_to_context(_audDecCtx, _fmtCtx->streams[_audioStreamIndex]->codecpar) < 0)
        {
            var pCodecContext = _audDecCtx;
            ffmpeg.avcodec_free_context(&pCodecContext);
            _audDecCtx = null;

            RaiseError("Cannot set parameters in this context");
            return false;
        }

        if (ffmpeg.avcodec_open2(_audDecCtx, audCodec, null) < 0)
        {
            var pCodecContext = _audDecCtx;
            ffmpeg.avcodec_free_context(&pCodecContext);
            _audDecCtx = null;

            RaiseError("Cannot open Codec context");
            return false;
        }

        // Set up an audio conversion context so that the decoded samples can always be delivered as signed 16 bit mono PCM.

        _swrContext = ffmpeg.swr_alloc();
        ffmpeg.av_opt_set_sample_fmt(_swrContext, "in_sample_fmt", _audDecCtx->sample_fmt, 0);
        ffmpeg.av_opt_set_sample_fmt(_swrContext, "out_sample_fmt", AVSampleFormat.AV_SAMPLE_FMT_S16, 0);

        ffmpeg.av_opt_set_int(_swrContext, "in_sample_rate", _audDecCtx->sample_rate, 0);
        ffmpeg.av_opt_set_int(_swrContext, "out_sample_rate", clockRate, 0);

        //FIX:Some Codec's Context Information is missing
        if (_audDecCtx->channel_layout == 0)
        {
            long in_channel_layout = ffmpeg.av_get_default_channel_layout(_audDecCtx->channels);
            ffmpeg.av_opt_set_channel_layout(_swrContext, "in_channel_layout", in_channel_layout, 0);
        }
        else
            ffmpeg.av_opt_set_channel_layout(_swrContext, "in_channel_layout", (long)_audDecCtx->channel_layout, 0);
        ffmpeg.av_opt_set_channel_layout(_swrContext, "out_channel_layout", ffmpeg.AV_CH_LAYOUT_MONO, 0);

        if ( ffmpeg.swr_init(_swrContext) < 0 )
        {
            RaiseError("Cannot init context with specifiec parameters");
            return false;
        }

        _audioTimebase = ffmpeg.av_q2d(_fmtCtx->streams[_audioStreamIndex]->time_base);
        _audioAvgFrameRate = ffmpeg.av_q2d(_fmtCtx->streams[_audioStreamIndex]->avg_frame_rate);
        _maxAudioFrameSpace = (int)(_audioAvgFrameRate > 0 ? 1000 / _audioAvgFrameRate : 10000 * clockRate);
    }
    return true;
}

[DavidMartynWood]

If you're using windows then I think it would be simpler to stay away from ffmpeg. Take a look at how the softphone example uses WaveIn, and adapt that to use Wasapi Loopback.

[nodminger]

@DavidMartynWood
I think I have found a way but still have some issues. Instead of using FFmpeg I used SiporceryMedia.Windows to send system audio. It uses PCM format with rate:8000 and bit:16 and channel:1. I modified the AudioExtraSources.cs from Sipsorcery and added AudioSourcesEnum.SystemSoundCapture condition with NAudio. Now system audio is sharing but it lags for every 1 second or for every frame.

AudioExtraSources.cs

public void SetSource(AudioSourceOptions sourceOptions)
{
    if (!_isStarted)
    {
        // Audio source is being adjusted prior to starting.
        _audioOpts = sourceOptions;
    }
    else
    {
        // If required start the audio source.
        if (sourceOptions != null)
        {
            _sendSampleTimer?.Dispose();
            _musicStreamReader?.Close();
            StopSendFromAudioStream();

            _audioOpts = sourceOptions;

            if (_audioOpts.AudioSource == AudioSourcesEnum.SystemSoundCapture)
            {
                _isStarted = true;
                _loopbackCapture.WaveFormat = new WaveFormat(8000, 16, 1);
                _loopbackCapture.DataAvailable += (sender, e) =>
                {
                    byte[] audioData = e.Buffer;
                    short[] shortAudioData = new short[audioData.Length / 2]; // Two bytes make up one short.

                    //Buffer.BlockCopy(audioData, 0, shortAudioData, 0, audioData.Length);
                    for (int i = 0; i < shortAudioData.Length; i++)
                    {
                        shortAudioData[i] = BitConverter.ToInt16(audioData, i * 2);
                    }

                    EncodeAndSend(shortAudioData, _audioFormatManager.SelectedFormat.ClockRate);
                };
                //_sendSampleTimer = new Timer(LoopbackCapture_DataAvailableWrapper, null, 0, 1);
                _loopbackCapture.StartRecording();
            }
            else if (_audioOpts.AudioSource == AudioSourcesEnum.None)
            {
..... .... ....

[DavidMartynWood]

I'm sorry but audio lag is really hard to diagnose without more context. I've spent days looking through my own code in the past trying to figure out things like that.

A few things to consider:

• when you create WasapiLoopbackCapture how much audio are you asking for? 20 milliseconds would be what I would expect to be sent in an Rtp packet but I'm not sure what your webrtc session is expecting.
• I assume at some point this is being encoded to alaw/ulaw? I can't follow your code to be sure but make sure you've not skipped that somehow.
• I don't understand why you needed to add the new AudioSourceEnum. But I'm not all that familiar with audio handling in this library.

If you could try to be more specific about the audio symptoms it might help, but that might be wishful thinking on my part.

[nodminger]

Thanks for the explanation. In my code AudioSourceEnum is just used to tell the program that it want to get System Audio. It is just condition. It has nothing to do with codec and other settings. This is the settings for Sample Audio

public const int AUDIO_SAMPLE_PERIOD_MILLISECONDS_DEFAULT = 20;
public const int AUDIO_SAMPLE_PERIOD_MILLISECONDS_MIN = 20;
public const int AUDIO_SAMPLE_PERIOD_MILLISECONDS_MAX = 500;

with this I get breaking audio with some noise in the background. I set all the numbers to 90 and I got continuous audio but there is some background noise.

This is my loop back capturer code

if (_audioOpts.AudioSource == AudioSourcesEnum.SystemSoundCapture)
{
    _isStarted = true;
    _loopbackCapture.WaveFormat = new WaveFormat(_audioFormatManager.SelectedFormat.ClockRate, 16, 1);
    _loopbackCapture.DataAvailable += (sender, e) =>
    {
        byte[] audioData = e.Buffer;
        short[] shortAudioData = new short[audioData.Length / 2]; // Two bytes make up one short.

        Buffer.BlockCopy(audioData, 0, shortAudioData, 0, audioData.Length);

        EncodeAndSend(shortAudioData, _audioFormatManager.SelectedFormat.ClockRate);
    };
    _loopbackCapture.StartRecording();
}

private void EncodeAndSend(short[] pcm, int pcmSampleRate)
{
    if (pcm.Length > 0)
    {
        if (pcmSampleRate != _audioFormatManager.SelectedFormat.ClockRate)
        {
            pcm = _audioEncoder.Resample(pcm, pcmSampleRate, _audioFormatManager.SelectedFormat.ClockRate);
        }

        byte[] encodedSample = _audioEncoder.EncodeAudio(pcm, _audioFormatManager.SelectedFormat);

        uint rtpUnits = (uint)(_audioFormatManager.SelectedFormat.RtpClockRate / 1000 * _audioSamplePeriodMilliseconds);

        OnAudioSourceEncodedSample?.Invoke(rtpUnits, encodedSample);
    }
}

in this code _audioFormatManager.SelectedFormat.ClockRate is equal to 8000. I also checked alaw/ulaw codecs. Since they are codec, I did not touch those files.

[nodminger]

I have implemented WasapiLoopbackCapture via SipsorceryMedia.Windows.