-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Master effects #6359
Master effects #6359
Conversation
As in, it doesn't honor the micro-fades preference? |
@vsverchinsky when playing a mono clip, only the left channel is playing audio: |
7c13e9c
to
5eb496e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Haven't finished yet the review, but stumbled on a potential crash.
@@ -68,6 +72,14 @@ void RealtimeEffectManager::Initialize( | |||
|
|||
// Leave suspended state | |||
SetSuspended(false); | |||
|
|||
//Register the master channel group |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know if this comment is relevant only to the following two out-commented lines or to what follows, too. Why keep the dead code?
@@ -50,6 +50,8 @@ class REALTIME_EFFECTS_API RealtimeEffectManager final : | |||
public Observer::Publisher<RealtimeEffectManagerMessage> | |||
{ | |||
public: | |||
static const ChannelGroup* MasterGroup; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like even the pointer should be const
, i.e., static const ChannelGroup* const MasterGroup;
Also, I'd prefer the initialization to take place here as well for clarity, rather than in the source file.
@@ -151,9 +153,10 @@ class REALTIME_EFFECTS_API RealtimeEffectManager final : | |||
}; | |||
|
|||
void ProcessStart(bool suspended); | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
trailing spaces
@@ -202,7 +200,7 @@ class REALTIME_EFFECTS_API RealtimeEffectManager final : | |||
} | |||
|
|||
AudacityProject &mProject; | |||
Latency mLatency{ 0 }; | |||
//Latency mLatency{ 0 }; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not just removing it?
//mRates.insert({MasterGroup, sampleRate}); | ||
|
||
VisitGroup(MasterGroup, [&](RealtimeEffectState& state, bool) { | ||
scope.mInstances.push_back(state.AddGroup(nullptr, numPlaybackChannels, sampleRate)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nullptr
aka MasterGroup
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If so I'd prefer MasterGroup
over nullptr
, in case one looks for all places where it's used.
libraries/lib-audio-io/AudioIO.cpp
Outdated
@@ -148,6 +148,7 @@ struct AudioIoCallback::TransportState { | |||
assert(false); | |||
continue; | |||
} | |||
//TODO: initialize with sequence own channels count and sampleRate |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why ? Isn't this used by AllocateChannelsToProcessors
to "plug" an effect's output channels to the number of playback channels. Why should this change with the master channel?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The comment isn't specifically about master channel.
When we allocate two mono effects to one stereo track there is at least one problem with UI : you can't see the output of the second channel.
I think configuring effects to process samples in native track format and rate could have benefit of reducing CPU load in case if playback sample rate is higher compared to playback rate. But that may be conceptually wrong.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Regarding the CPU, I don't know, it looks like there is no negotiation at the moment: the framework dictates the sampling rate to the effect instances, which I guess is much simpler. Also, I think resampling is inevitable, so I'm not sure we want to try and minimize it.
My point is that this TODO should probably be tempered with some doubt, e.g., "consider initializing (...) because (...)"
src/RealtimeEffectPanel.h
Outdated
@@ -81,5 +86,9 @@ class RealtimeEffectPanel : public wxPanel | |||
void SetFocus() override; | |||
|
|||
private: | |||
|
|||
void MakeTrackEffectsPane(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think correct English would be MakeTrackEffectPane
(even if there's more than one effect).
} | ||
} | ||
|
||
auto samplesAvailable = std::min_element( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This crashes when playing a MIDI file on an empty project, because mPlaybackSequences
and, consequently, mProcessingBuffers
, are then empty.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe a solution: the size of mMasterBuffers
equals the number of playback channels, maybe it could be used instead.
Also, AFAICS these buffers always have the same size, so a simple mMasterBuffers[0].size()
would be more expressive and sufficient.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Or mPlaybackBuffers
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Finished reviewing the first commit (the audio part) ; nothing major besides that crash, but hopefully this is an easy fix.
Please remove the dead code ; I don't see why it should be kept, and it's noisy and sometimes confusing.
libraries/lib-audio-io/AudioIO.cpp
Outdated
mOldChannelGains.resize(mPlaybackSequences.size()); | ||
size_t iBuffer = 0; | ||
// Bug 1763 - We must fade in from zero to avoid a click on starting. | ||
//mOldPlaybackGain = 0.0f; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Noise
libraries/lib-audio-io/AudioIO.cpp
Outdated
mPlaybackBuffers[iBuffer++] = std::make_unique<RingBuffer>( | ||
floatSample, playbackBufferSize); | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
trailing spaces
libraries/lib-audio-io/AudioIO.cpp
Outdated
// Avoiding std::vector | ||
const auto pointers = stackAllocate(float*, mNumPlaybackChannels); | ||
// Do any realtime effect processing, after all the little | ||
// slices have been written. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't see any side effect if pScope == nullptr
, so shouldn't the if (pScope)
be on this line?
5eb496e
to
91823e4
Compare
@@ -166,26 +171,21 @@ class REALTIME_EFFECTS_API RealtimeEffectManager final : | |||
// using StateVisitor = | |||
// std::function<void(RealtimeEffectState &state, bool listIsActive)> ; | |||
|
|||
//! Visit the per-project states first, then states for group | |||
//! Visit states for group |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
" or for the master when group is null"
libraries/lib-audio-io/AudioIO.cpp
Outdated
@@ -2015,104 +2023,208 @@ bool AudioIO::ProcessPlaybackSlices( | |||
// warping | |||
if (frames > 0) { | |||
size_t produced = 0; | |||
// Check for asynchronous user changes in mute, solo status |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Comment "mPlaybackBuffers correspond many-to-one with mPlaybackSequences" above this line is no longer correct
libraries/lib-audio-io/AudioIO.cpp
Outdated
auto& buffer = mProcessingBuffers[iBuffer++]; | ||
const auto offset = buffer.size(); | ||
//there could be leftovers from the previous pass, don't discard them | ||
buffer.resize(buffer.size() + frames, 0); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would add a comment that this resize
is expected not to allocate memory in the worker thread, because sufficient size should have been reserved in AllocateBuffers
. Perhaps, test capacity before and after and assert
no change.
libraries/lib-audio-io/AudioIO.cpp
Outdated
.0f); | ||
} | ||
else | ||
//TODO: mixer->Process() may be wasting CPU cycles |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think the mixer can be properly eliminated in the case that it does resampling of a track whose rate does not match the device output rate, or there is a time track. There will be some state in the resampler dependent on previous input.
However, note that the mixer stands as a producer in relation to this function which is its consumer. Perhaps, if it really matters, another thread could mix in advance.
But all that effort at overlapping computation may be unnecessary because this thread really spends most of its time idle anyway, going faster than the ultimate real-time consumption of its output.
Or then again maybe it might matter in future because we want to have ever more capacity for complicated calculations like more effects, more stretching, more pitch shifting.
Or again, maybe you were thinking of power consumption, not performance.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I though that mixer could be "advanced" without doing any actual work
libraries/lib-audio-io/AudioIO.cpp
Outdated
else | ||
//TODO: mixer->Process() may be wasting CPU cycles | ||
//TODO: fade out smoothly | ||
std::fill_n(buffer.data() + offset, frames, 0); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I notice that as part of this PR you have moved the logic to silence tracks (which might be driven by unpredictable presses of the mute buttons by the user) from the low-latency thread to this high-throughput thread.
Was that for simplification of the former, so that it does less work? Or some other reason?
Does this comment mean you have not yet restored the microfading feature? (I have not yet read enough to answer that for myself)
I think I like this change because as I recollect, the microfading had a dependency on the buffer size given to audacityAudioCallback
which can depend on operating system or larency settings. That should not be: the duration of the fade in or out should be independent of that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Was that for simplification of the former, so that it does less work? Or some other reason?
The reason is that we can't do master effects processing on low-latency thread and that processing should happen after all the tracks have been mixed into playback channels.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this comment mean you have not yet restored the microfading feature? (I have not yet read enough to answer that for myself)
Micro-fading isn't completely restored, it doesn't happen when you move gain, pan sliders or press solo/mute buttons.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not requesting changes because I'm not certain of my analysis, but I think there is another bug, already present in the first commit but that I missed before.
libraries/lib-audio-io/AudioIO.cpp
Outdated
// Choose a common size to take from all ring buffers | ||
const auto toGet = | ||
std::min<size_t>(framesPerBuffer, GetCommonlyReadyPlayback()); | ||
|
||
// Poke: If there are no playback sequences, then the earlier check |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know what earlier check is meant here. Could it be that this comment is also outdated?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The comment refers to what now happens inside CallbackCheckCompletion
. Amend the comment to make that more explicit. And it's now "below" not "above".
@@ -2682,19 +2719,21 @@ bool AudioIoCallback::FillOutputBuffers( | |||
return true; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A little bit higher, line 2712 on this commit, is the line float *outputFloats = outputBuffer;
, which only seems to be an alias to the same thing, maybe a remnant of older code. Would you remove it if you get the chance?
for(auto& buffer : mMasterBuffers) | ||
std::fill_n(buffer.begin(), samplesAvailable, 0); | ||
//introduced latency may be too high... | ||
//but we don't discard what already was written |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There might be a mini bug here: if adding a real-time effect during playback increased latency such that samplesAvailable
becomes 0, what would be played for that frame would be the samples of the previous mPlayoutBuffers
, possibly leading to a repeated playout of the last few ms.
Solution might be to return false
, such that the AudioThread
doesn't read the content of mPlayoutBuffers
?
It would be worse if a real-time effect had variable latency, but I don't think there's a use case for that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That would apply to the next if(samplesAvailable == 0)
too.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the answer to your question is in FillPlayBuffers
. If for some reason samplesAvailable
will be zeroed, then there will be more processing passes until GetNeeded() == 0
. Otherwise if I return false
as you suggest then extra buffer exchange step will be initiated for same frame, which I think is less preferrable.
May be we should test how it works now with latency values that are much larger than our regular buffer sizes
libraries/lib-audio-io/AudioIO.cpp
Outdated
//Prepare master buffers. | ||
//These buffers may also contain leftovers from the previous pass | ||
//Remember initial offset for each of them | ||
const auto masterOffsets = stackAllocate(int, mMasterBuffers.size()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have doubts about the correctness of the handling of latency for the master channel.
The discardable
value returned by a real-time effect is the number of leading samples that can be discarded.
For the track effects, this is indeed what is done:
for(int i = 0; i < seq->NChannels(); ++i)
{
auto& buffer = mProcessingBuffers[bufferIndex + i];
buffer.erase(buffer.begin(), buffer.begin() + discardable);
}
For the master track, however, no leading samples are discarded, and then
unsigned bufferIndex = 0;
for(auto& buffer : mMasterBuffers)
{
mPlaybackBuffers[bufferIndex++]->Put(
reinterpret_cast<constSamplePtr>(buffer.data()),
floatSample,
samplesAvailable,
0
);
}
which means trailing samples are discarded instead. It was already that way in the previous commit but I missed it.
If I'm correct, the consequence of this mistake would be that, if the master channel has an effect that adds delay, an artefact due to zeroed audio would be heard at the beginning of playback.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I also don't see the necessity of masterOffsets
. I don't think there can be left-overs from one ProcessPlaybackSlices
to the next, as it would mean this method would produce more audio samples than asked for, and I don't see it to be the case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The problem does exist indeed. In the attached video, you can hear an audible crackle at the beginning of the playback with a non-zero latency effect (TDR Limiter) added to the stack:
Screen.Recording.2024-05-10.at.16.15.45.video-converter.com.mp4
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @dozzzzer .
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doing latencies really right is a known bit of technical debt. I have had ideas. I didn't find time to elaborate and share them yet.
Removing change request because the particular bug was fixed.
libraries/lib-audio-io/AudioIO.h
Outdated
// Temporary buffers, each as large as the playback buffers | ||
std::vector<SampleBuffer> mScratchBuffers; | ||
std::vector<float *> mScratchPointers; //!< pointing into mScratchBuffers | ||
|
||
std::vector<std::unique_ptr<Mixer>> mPlaybackMixers; | ||
std::unique_ptr<Mixer> mMasterMixer; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unused
libraries/lib-audio-io/AudioIO.cpp
Outdated
mLastPlaybackTimeMillis = ::wxGetUTCTimeMillis(); | ||
return false; | ||
} | ||
|
||
// The drop and dropQuickly booleans are so named for historical reasons. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Comment refers to variables that no longer exist
libraries/lib-audio-io/AudioIO.cpp
Outdated
} | ||
} | ||
|
||
//samples at the beginning of could have been discarded |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
delete "of"
//introduced latency may be too high... | ||
//but we don't discard what already was written | ||
if(samplesAvailable == 0) | ||
return progress; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am reminded that the handling of latency is still not satisfactory. I had ideas for fixing it that I have not found time to implement.
unsigned bufferIndex = 0; | ||
for(const auto& seq : mPlaybackSequences) | ||
{ | ||
//TODO: apply micro-fades |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is that really TODO? I think you have conserved the micro-fade logic on the consumer side in FillOutputBuffers
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IIUC that one handles the changes in the volume slider, which are global. This is for the track-specific mute/unmute.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have completed a review of changes in AudioIO.cpp. If my contribution of per track meters is also wanted, there is a lot of needed work to make these compatible. I may volunteer my own very different rewrite of AudioIO.cpp to support a master channel in the right way. I understand that:
The old version, without a master stack, differed in these ways:
Questions might arise about subtle differences of results, moving the muting before the effect processing. (Is it better with your change? Have you tried muting a track with a reverb, during play -- which before, would just mute it at once, but after, might instead produce the decaying tail of what was unmuted?) Also, whether pan should apply before a per-track effect or after: It might make a difference for certain effects like reverbs that are not simple LTIs but might work in terms of mid and side channels. (@saintmatthieu, opinion?) More later... |
More: The existing ring buffers should not be removed. Ring buffers for the master mix should be added. The consumer thread should take equal samples out of all of them. Only the samples from the master mix are stored in the output buffer of the audio io callback. This allows future development to send the buffers of pre-mixed per-track data inter-thread, back to the master thread for drawing. See My thinking is this: the consumer thread needs to have access to intermediate states of calculation of the mix samples, corresponding in time to the ultimate output. Those intermediates should be sent to other code that performs graphical display, keeping as closely synchronized as possible with the real-time playback. That synchronization will not be perfect, given the reality of frame refresh rates. Still, the latency of the RingBuffer itself should be accounted for in that drawing update. Sending the per-track information to the main thread more directly from the producer thread would make less perfect synchronization of the display. Therefore, more data should be travelling through ring buffers from the high-throughput but variable-latency producer thread, to the more regularly updating consumer thread. |
libraries/lib-audio-io/AudioIO.cpp
Outdated
@@ -2036,8 +2037,15 @@ bool AudioIO::ProcessPlaybackSlices( | |||
|
|||
auto& buffer = mProcessingBuffers[iBuffer++]; | |||
const auto offset = buffer.size(); | |||
const auto toConsume = std::min( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What are these new lines fixing ? It looks like we are now ensuring that no new memory is allocated in that function. This comes at the risk, though, that we provide less than frames
samples, if for some reason the capacity of the buffer is insufficient. Wouldn't that be worse ?
I wonder if this change is motivated by the initial playback click that we hear in @dozzzzer 's video above. I suspect this not to be due to buffer allocation, but simply because he is playing a sine wave without any sort of smoothing envelope, and it's then completely normal to hear this click.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like we are now ensuring that no new memory is allocated in that function.
Yes
This comes at the risk, though, that we provide less than frames samples
Memory allocations are not better. The disadvantage of this code is that it's not capable to handle very large latencies (usually > 250ms).
I wonder if this change is motivated by the initial playback click that we hear in @dozzzzer 's video above
No, I did that in response to Paul's comment above
@Paul-Licameli good point about reverb. |
About panning before or after, yes, if the effect is non LTI, this would make a difference. If there is one place for panning and gain, I think it should be where it is a the moment, after the effect stack. I guess other DAWs do it that way, too (or may have also a "pre-gain"), and if a non LTI effect such as a compressor comes in, it will come with its own pre-processing gain parameter. |
Will approve now, pending Vitaly's rework on the mute behaviour.
I disapprove. please wait to see my alternative that will be compatible with oerctrack meters. expext it today. |
Approving for now. My purpose is not to override @Paul-Licameli's disapproval, because maybe his alternative makes this very complex logic easier ? I'll leave up to @vsverchinsky to decide but I don't want to block QA anymore. |
Let it merge then: but my PR will begin with reversion of all the changes in AudioIO.cpp (not of other UI changes) and then a more careful stepwise rewrite. QA will likely need to re-test everything involving sound quality, but not the UI changes. |
Found issues:
|
Yes! |
My changes are more complicated than expected. However, I might be fixing long standing problems with effect latencies. Search for issues with “latency” |
Looking into the export/render problem. |
9640a33
to
cb589a0
Compare
@vsverchinsky . Serious accessibility issue, logged here: #6456 |
@vsverchinsky . Another accessibility issue, logged here: #6457 |
6c76721
to
a4856a2
Compare
libraries/lib-audio-io/AudioIO.cpp
Outdated
@@ -1243,7 +1243,7 @@ bool AudioIO::AllocateBuffers( | |||
)); | |||
for(auto& buffer : mProcessingBuffers) | |||
buffer.reserve(playbackBufferSize); | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@saintmatthieu Please avoid unnecessary formatting changes if you don't change the line itself - it complicates the search through commit history and review process
@@ -2241,7 +2241,10 @@ bool AudioIO::ProcessPlaybackSlices( | |||
); | |||
} | |||
} | |||
|
|||
|
|||
for(auto& buffer : mMasterBuffers) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is a finally
block already that does cleanup job
1f29456
to
d37c90f
Compare
Completed review of @saintmatthieu commits 👍 |
7925014
to
86a297a
Compare
c2848dc
to
c7b4779
Compare
Resolves: #6346
Known issues:
UI: Master effects section does not fit automatically to the contents
No micro fades applied to solo/gain change
Latency increased for track controls in real-time (delay between the moment you press TC button and the moment you hear changes is audiable)
I signed CLA
The title of the pull request describes an issue it addresses
If changes are extensive, then there is a sequence of easily reviewable commits
Each commit's message describes its purpose and effects
There are no behavior changes unnecessary for the stated purpose of the PR
Recommended:
QA: