Switch to mv3 (Firefox) and make any necessary further changes (Chromium)

[SebastianZimmeck]

As Firefox is switching to manifest version 3 (mv3) in a couple of months, we should keep an eye on if functionality is breaking, what we need to change, etc. I tend to think that the impact will be a bit more limited than Chrome's switch to mv3 as it is my understanding that Firefox intends to keep some of the deprecated APIs around. However, at the beginning of next year, when Chrome is making the switch, we should address this.

Currently, the timeline of Firefox's switch to mv3 and which APIs will be deprecated is unclear but will become more clear over the next couple of months. Here are some official resources:

• Manifest v3 in Firefox: Recap & Next Steps
• Manifest V3 Firefox Developer Preview — how to get involved

Please everyone keep this on the radar. We do not want to wake up one day and everything is broken (and we can't even update anymore on the Firefox Store).

[SebastianZimmeck]

We already moved to MV3 in Chrome (protection mode only). So, there is nothing more to be done there. But the question is to which extent is MV3 still relevant for our Firefox version. Ideally, if we still have protection mode in Firefox, then the protection mode extension (for Chrome and Firefox) should have the same code. @sophieeng and @OliverWang13 will look into this.

@Jocelyn0830 and @katehausladen, if you could look into how the MV3 changes affect the analysis mode crawler (Firefox only). @Jocelyn0830 opens an issue in the crawler repo to discuss MV3 over there.

Here is the last from Firefox.

[sophieeng]

In preparation of the new release, @OliverWang13 and i have been looking to the Firefox MV3 status. Because analysis mode has been removed from the OptMeowt extension, we are left with two protection modes: one for Chrome and one for Firefox. The main difference between the two protection modes is which Manifest version they're built for. The Chrome version has been changed to MV3, which involves new service workers and different APIs.

Because we are unsure if Firefox MV3 will have the same changes that Chrome had when migrating between versions, we don't want to make Firefox and Chrome the same. We prefer leaving Firefox in MV2 until a list of changes for FF MV3 comes out and we can compare FF MV3 and Chrome MV3. We will reevaluate from there. In the future, it would be nice to have one protection file that works for Chrome and Firefox, but we don't know if that's a possibility.

[SebastianZimmeck]

@sophieeng will update on the current Firefox MV3 status.

[sophieeng]

Firefox has not had any timeline changes to the switch from MV2 to MV3. Currently, both versions are supported on Firefox but developers cannot create an extension that supports both, so they must be able to distribute both types. Firefox began supporting MV3 on November 21, 2022 and has stated they will look for an appropriate timeline to depreciate MV2 near the end of 2023. While we keep monitoring the situation, it would be especially good to check then.

Also, I do not see it mentioned but Google has put their own MV3 timeline under review. This means that switching to MV3 has been postponed.

In accordance with earlier posts in this issue, we will continue to wait for more clarity on what Firefox's MV3 will require before we make changes to our extension.

[SebastianZimmeck]

@sophieeng, can you check and report on the status of MV3?

• Manifest v3 signing available November 21 on Firefox Nightly
• Resuming the transition to Manifest V3

The questions are (for both Firefox and Chrome):

• Do we need to do anything now?
• Should we prepare anything for later?

[sophieeng]

Chromium:
Chrome has been making changes to MV3 to close gaps in functionality. We do not need to do anything as we were able to fully migrate to MV3 without losing functionality because it was unsupported in MV3. MV2 will be phased out in June of 2024 at the minimum. We do not need to prepare anything else for Chrome as we are fully migrated.

Firefox:
Firefox is supporting MV3 and allowing MV3 version extensions starting the Mozilla store on November 21st. Here is their MV3 documentation. As of now, MV2 is still supported for the foreseeable future. At the end of 2023, Firefox will decide the timeline to deprecate MV2.
To do now: Start reading over MV3 documentation, and identify aspects that will be critical for our extension.
Prepare for later: Keep an eye on how the MV3 documentation as it may be modified over time. Come back to this issue at the end of the year to document the depreciation timeline

[SebastianZimmeck]

Thanks, @sophieeng!

To do now: ... Prepare for later

It would be great if you could take the lead and prepare a plan (with the help of @OliverWang13, as necessary) on the concrete changes that we need to make. Especially, we need to find out:

• Do we need to make big architectural changes or just switch out APIs?
• Which changes exactly on these different levels?
• By when? In other words, when is a good time for making these changes? We do not want to do it when the MV3 changes are still in flux, but we also should not do it too late.

@katehausladen, is the crawler also affected by the change to MV3?

[sophieeng]

Plan going forward: I'll start comparing the Firefox and Chrome MV3 documentation and make note of where the two browsers may need different code. In general we want to have as similar codebases between the two as we can.

I'll look for the Firefox MV2 depreciation timeline at the beginning of 2024 / the Spring semester.

[SebastianZimmeck]

Excellent!

[Mattm27]

@OliverWang13 and I met today and implemented a popup in the options page upon installation prompting the user to enable host permissions using a button that then sets these permissions. After the button is clicked, the header signal sends correctly however we are still working on implementing these same strategies for sending the DOM Signal.

[Mattm27]

We have implemented DOM Signal functionality into our MV3 migration and have removed a large majority of the browser checks from the codebase. In the reference server, the DOM signal is now present when OptMeowt is enabled however we are still working through bugs relating to the website response dropdown in the popup and the OptMeowt extension icon not changing colors when Do Not Sell is enabled. @OliverWang13 and I will continue to work through these bugs this upcoming week, but it seems we are nearing an end to this migration process.

[SebastianZimmeck]

There are a number of details that go with the switch to MV3. For the time being, I am not opening separate issues. Please add as things come to mind:

• Re-organize compilation process and structure via Webpack as necessary. E.g., currently, we have two distinct steps; one for Chromium, one for Firefox
• Update unit test suite and continuous integration as necessary, e.g., remove no longer necessary tests and add new tests
• Update readme, e.g., how to install, test, permission use, directory structure; update Wiki
• Create new release for both Chromium and Firefox and deploy

[Mattm27]

@OliverWang13 and I have met and solved the problem resulting in bugs with the Website Response and extension icon. These bugs occurred because of the argument sender.origin was marked as undefined not allowing us to access the necessary URL. To work around this we implemented a new message argument message.origin.url to access this URL instead. We are putting together a checklist for final bug checks that we will continue to work through.

We also stumbled upon another slight structural issue surrounding our 3rd party domain storage that should have been fixed for Chrome MV3 but must have slipped through the cracks which we will look into more this week.

[Mattm27]

Here is a checklist @OliverWang13 and I put together to go through after implementing MV3 changes:

• - 3rd Party Domains button responding correctly
• - Website Response button in popup reflects GPC Signal information
• - Domain List button works and the list builds correctly when visiting new sites
• - Domains can be toggled on/off
• - Remaining two buttons on the settings page (Enable and Disable) work correctly
• - Export and Import Domain List buttons are fully functional
• - Disable/Enable button works visually
• - Disable/Enable button works functionally
• - Domain List button Popup button works
• - Darkmode Toggle functional

We are still working through errors with 3rd Party Domain and Website Response dropdown functionality as we mentioned in this weeks meeting.

[Mattm27]

For @OliverWang13:

• - 3rd Party Domains button responding correctly
• - Website Response button in popup reflects GPC Signal information
• - Domain List button works and the list builds correctly when visiting new sites
• - Domains can be toggled on/off
• - Remaining two buttons on the settings page (Enable and Disable) work correctly
• - Export and Import Domain List buttons are fully functional
• - Disable/Enable button works visually
• - Disable/Enable button works functionally
• - Domain List button Popup button works
• - Darkmode Toggle functional

[Mattm27]

@OliverWang13 and I have implemented a new storage system using IndexDB to store 3rd Party Domain data to work around the deprecation of persistent background pages in MV3. This has proven to be a good approach as the 3rd party data is no longer wiped when the background page is terminated, allowing the 3rd Party Domains dropdown to function properly in the popup.

We began attempting to use the same approach to tackle problems with the Website Response button dropdown, storing wellknown data fetched in the contentScripts.js using IndexDB. However we have run into problems accessing this data in other background scripts like protection.js, so we are planning on moving forward by instead just fetching and storing this wellknown data within protection.js.

[Mattm27]

@OliverWang13 and I have separately implemented changes to tackle issues with the Website Response button dropdown as well as changing the color of the extension icon when a site accepts the GPC signal. Although both functions are working correctly, they are a bit buggy at times only working on some websites once the site is reloaded.

This issue seems to be occurring because the wellknown data is stored in the database only after the webpage is reloaded, which was a similar bug we ran into with 3rd party domains.

We will continue working through this bug as the week goes on.

[SebastianZimmeck]

Good progress, @Mattm27!

It would be good if you can demo today.

[Mattm27]

@OliverWang13 and I have fixed a bug that was occurring with 3rd party domains, in which some 3rd party domains on one site would accumulate within the 3rd party domains drop-down of a separate site although they were not present on the current site. We resolved this issue by storing 3rd party domains based on the tab ID they are associated with using details.tabId in the updateDomainlist function in protection.js.

At this point, some refactoring is left to do before we are set for a new release but the migration to MV3 in terms of main functionality now seems to be complete.

[SebastianZimmeck]

Excellent! Well done!

[Mattm27]

Refactoring has been finished, we are now ready to create a new release.

[SebastianZimmeck]

Excellent! Great work, @Mattm27 and @OliverWang13!

@Mattm27, if all tests are passing and the UI tests are good, feel free to create a release in the repo.

Also, it would be good if you can create a new release on the Firefox Add On Store. You should have all rights to do so. They might approve it right away but there may also be some back-and-forth.

For Chrome, can you provide me a distribution version so that I can create a release on the Chrome Web Store?

Before creating the releases be sure to update the version number, permissions, ..., anything else necessary in the manifest files.

[SebastianZimmeck]

Note to update readme/Wiki and tutorials accordingly based on new permission requests after MV3 migration.

Were you able to make these updates, @Mattm27?

[Mattm27]

I have not, but will read through and update today.

[SebastianZimmeck]

The green ring issue (#460) popped up again (for Firefox).

[SebastianZimmeck]

Both the Firefox Add-on Store and the Chrome Web Store now have v5.0.1 with mv3.

@Mattm27, if there is nothing more to do from your end, feel free to close this issue!

Great work!

[SebastianZimmeck]

I have not, but will read through and update today.

Especially, if you can confirm, that this is done.