Playing streamed custom sound data?

[Pranav-Badrinathan]

The title isn't really descriptive, so I'll try to explain as succinctly as possible.

I am streaming sound data encoded with Opus from a native app to the discord bot application, and was wondering the correct way to play back this audio stream using songbird and serenity. Currently my approach is to just take the data and make an Input for it, and send it over to the play_source function. It kinda works, in the sense that I can hear something, except it is mainly just weird crackling and popping sounds and nothing like the audio I am sending through.

I was wondering if there is a certain size the buffer sent to play_source has to be in order for songbird to play the audio properly, or something else I am missing here as the way I am currently doing does not feel like the best way to go about it...
After hours of scouring through the docs and code, I have decided to ask this question. Sorry if I missed something obvious!

Relevant code:

pub async fn play_music(sb: std::sync::Arc<Songbird>, mut rcv: mpsc::Receiver<AudioSet>) {
	use songbird::input::{
		Reader, 
		Codec, 
		codec::OpusDecoderState, 
		Container, 
		Input,
	};
	use songbird::tracks::PlayMode;

	loop {
		// Receive audio data and guild id as a "Set";
		let set = match rcv.recv().await {
			Some(aset) => aset,
			None => continue
		};

		if let Some(h)= sb.get(set.guild_id) {
			let mut handler = h.lock().await;

			let audio: Input = Input::new(
				true, 
				Reader::from_memory(set.audio_data), 
				Codec::Opus(OpusDecoderState::new().unwrap()), 
				Container::Raw,
				None
			);
			
			let track_handle = handler.play_source(audio);

			//This is simply here to keep the audio from overlapping. Without this, the program plays through
			//minutes worth of audio in seconds.
			while track_handle.get_info().await.unwrap().playing != PlayMode::End {}
		}
	}
}

Any alternatives for the very temporary and bad while loop at the bottom would also be appreciated 😅. Thanks.

EDIT: Forgot to mention, the data that is being played is very small, usually less half a second. Hence why I was asking about the buffer size.

[Pranav-Badrinathan]

A quick update: Changing the buffer size to include audio for more than a second before sending the data over to be played does not work either. Also each packet that is supposed to take slightly more than a second takes wayyyy longer (more than a minute) before the track's status changes to END. This could very easily be a fault in my method here, but that does seem very weird.

[FelixMcFelix]

The problem is that 'Raw' and 'Opus' is technically an illegal combination: the frames are not easily self-delimiting. current only supports Opus passthrough with the weird DCA format -- our implementation was primarily to un-break the impl that had been shoved into serenity.

The reason you are getting weird audio popping etc is likely because the Raw container is a) chewing through data at the rate of an uncompressed stream and b) incorrectly decoding since it doesn't know where frame boundaries are.

If you want to handle Opus [Ogg-container or WEBM] natively, I'd suggest moving to the next branch. I appreciate this means moving to an in-dev version of serenity which is not nice. The alternative is tagging your frames with lengths as required by DCA and using that.

Re: that while loop, please look into the event system. You can register callbacks for track state changes. The source code for TrackQueue is probably educational here.

[FelixMcFelix]

...Ah. The root-cause then is probably that you're effectively spinning up a new decoder for every 20ms of audio. If this is all from the same stream, you need to make sure you use the same decoder or your audio will sound very strange.

[Pranav-Badrinathan]

Ah true I didn't consider that. In that case it would be wise to decode it into PCM myself for the control over how the audio is decoded.

I was under the impression that songbird just took the Opus frames and forwarded them to Discord. Doesn't Discord take Opus encoded audio in the first place? But would that mean that each time I play_source the PCM audio, it would create a new encoder to send audio encoded in Opus to Discord? I was unaware there needed to be a step to decode from Opus.

[FelixMcFelix]

My bad, it should just be passing through without issue (defaults to passthrough support). I assumed I had set that to false, it's been a while since I looked into the details. In which case I think all should function, if a little jittery.

[Pranav-Badrinathan]

Hmm. This panics here with a range end index 10497 out of range for slice of length 1432. Some quick looking into the frame variable reveals that the size is set to be std::mem::sizeof::<f32>() for Opus and FloatPCM. The reader takes in bytes, and I don't think it converts them. What is happening here?

EDIT: Looks like it is trying to read a single f32 from the buf, but the range error does not make sense. If it is doing what it should be, the frame_len should be 4, not 10497.

[Pranav-Badrinathan]

I figured it out!

For anyone with the similar issues in the future
The problem was that to insert the frame header, I was doing buf.write_i16(buf.len().to_le() as i16). The problem is that write_i16 internally does to_ne_bytes() and adds the bytes as the header. To remedy this, I simply did buf.write(&buf.len().to_le_bytes()); to get control over the kind of bytes to be written. Thanks to @FelixMcFelix for all the patience and all the help, it is greatly appreciated.