Could you support lyrics editting on notes?

[SkyAndCloud]

Thank you for the development of this great tool! I really like it and use it almost everyday.
For many songwriters, they need to align melody and lyrics to know the singing effects of the whole song. However, I felt inconvenient when I was writing lyrics for my melody because I need to align the notes and the lyrics words. The current UI did not support edit lyrics on each note so that I could only align them in my brain, which is very inefficient.
Hope for this feature! :)

[poohcom1]

I'm interested in working on this feature since I've implemented something very similar on a fork. However, it required some kind of hacks to make it work properly. The main issue is that MIDI lyrics events are just regular events that aren't tied to any note. Realistically, we would want a lyric to be connected to a note, like how you would edit a lyric in Musescore.

The hack I implemented was giving each lyrics event a note_id, and the controller would pair each note to a lyric event. While this works, it adds a lot of potential decoupling issues, such as when a note is deleted, where we would need to constantly check for orphaned lyrics to delete.

On the other hand, we could just implement the lyric like any other controller independent of the note, but this wouldn't really solve OP's issue.

@ryohey what do you think? The current implementation I have in mind is to have an opt-in option to automatically pair lyrics and notes. To do so, I would use the PianoRollStore to compute each lyrics event and correctly update them to the note position.

[SkyAndCloud]

Thanks for your attention! I think that MIDI's support for lyrics is relatively poor. Maybe we can avoid extending MIDI events and take an easier way that just editing lyrics inside the note box and export a .txt lyrics file with the MIDI file. Because the lyrics and the melody are already aligned, we can reconstruct the alignments by loading those files, too.

[poohcom1]

I think that could work pretty well! I think we should still support the native midi events since the web app is centered around midi editing, so I'll try and see if I can implement something in the middle (or like a toggle to switch between using midi events or custom data).

The main hurdle we'll need to figure out is having a modular system that fits with the history (undo/redo) of the web app. As currently all data are stored as midi events, the history is entirely based on that. If we're going to add a new data type (i.e. lyrics) for each track, we might need to think about how to properly implement it in a way thats extendable. (i.e. implement a "plugin" API to allow non-midi data to be tracked in the history)

[ryohey]

@poohcom1 I checked your code. Very nice!
To prevent decoupling as you say, I think it would be better to move the lyrics event if there are notes with the same timing, instead of adding a note_id.
Moving the lyrics event might be accomplished by adding a LyricsControl and code to move the lyrics event with the same note event tick.
https://github.com/ryohey/signal/blob/main/src/main/actions/selection.ts#L175

[poohcom1]

@ryohey Thanks for the advice about using the selection actions! Just updating lyrics events based on selection actions made the code a lot cleaner

@SkyAndCloud I've made a basic implementation just using native midi events (poohcom1@3213132), but there are still some details that I want to discuss.

1. I'm currently using the control pane to display/edit the lyrics, since it's the most extendable component. Displaying the lyrics on the notes themselves might be more intuitive, but I'm concerned if it will interfere with more common shortcuts as most people won't be using lyrics.
2. Stacked notes that have the same ticks will result in two lyrics on the same spot. I actually did thought of a solution to this by ordering the lyrics by note value and displaying them accordingly, but I was wondering if you guys have other ideas for dealing with this.
   (EDIT: This solution doesn't work because there's no way to discern which lyrics belong to which note. The order of which lyrics match which note needs to be stored somewhere)
3. The current system assumes that all lyrics have a corresponding note. In the rare case that someone uploads a midi file where the lyrics does NOT line up with a note, the lyrics would be orphaned and unmovable until a note is written on it. Maybe I can implement a system where lyrics themselves and be moved.

Preview: https://signal-poohcom1-htte86klv-poohcom1.vercel.app/edit

P.S. I made some minor changes to track actions by changing createEvent and creating updateEvent, since I think it could be useful for features in the future. If you think it's unnecessary I can move all the code to lyrics actions instead.