Next steps for JS Toolkit and target platforms

[izaera]

This issue gathers the following steps we need to follow to replace the old Liferay JS Toolkit Yeoman generator by our new
@liferay/cli tool so that we can progress in our migration from the old JS Toolkit repo to the new one, then evolve the liferay-npm-bundler from an AMD architecture to -hopefully- a browser modules one.

Current status

✅ We have already merged the necessary target platforms to cover the previous functionality provided by liferay-npm-build-support.

✅ We have also merged a functional @liferay/cli that can completely replace the old Yeoman generator.

✅ Our documentation still points to the old artifacts.

✅ We still need to release some artifacts to the public npm registry.

Following steps

Sorted by the order we should follow when making them happen:

1. ✔️ Publish, at least, portal-agnostic and all portal-adapt-* platforms so that old projects can continue working.
2. ✔️ Compile a list of desired target platforms based on the current releases of Liferay Portal available.
3. ✔️ Generate and publish the target platforms listed in step 2.
4. ✔️ Update @liferay/cli with the published platforms. (feat(liferay-cli): add new published target platforms #756)
5. ✔️ Update our docs (copying them, as needed, to the v3 repo). (docs(js-toolkit): migrate docs from toolkit v2 to v3 #757)
6. ✔️ : Investigate how to migrate old projects, based on liferay-npm-build-support to target-platforms
   1. regular projects: ✔️ feat(liferay-cli): implement liferay update-project to update old yeoman-based projects #763
   2. adapted projects: ✔️ feat(liferay-cli): upgrade adapted projects #765
7. ✔️ Document project migration command: c0d8003
8. ✔️ Publish @liferay/cli https://github.com/liferay/liferay-frontend-projects/releases/tag/liferay-cli%2Fv1.0.0
9. ✔️ Add a message in the old Yeoman generator telling users to switch to @liferay/cli feat(generator-liferay-js): encourage users to use @liferay/cli #771
10. ✔️ Publish the "last" Yeoman generator version.
11. Add a message in liferay-npm-build-support to encourage users to migrate to target-platforms.
12. ✔️ Publish the last liferay-npm-build-support version. Deprecate liferay-npm-build-support in npmjs.com
13. ✔️ Freeze/remove liferay-npm-build-support and generator-liferay-js source code. This will be a formal freeze for now, there's no real need to remove anything from GitHub yet.
14. ✔️ Continue investigating browser modules PoC so that we can arrive to the specs we need to fulfill for liferay-npm-bundler v3.

Legend:
⚙️ in review process
⏳ approved, waiting to be merged
✔️ merged, finished

[izaera]

Point 2

EDIT: It turns out 7.0.x has no support for JS Toolkit, so just ignore it

The list of available target platforms spans to the first 7.0 version. Prior to that we had 6.2 which has never had support for npm modules.

The list of tags in github follows:

• 7.4.3.4-ga4

• 7.4.2-ga3

• 7.4.1-ga2

• 7.4.0-ga1

• 7.3.7-ga8

• 7.3.6-ga7

• 7.3.5-ga6

• 7.3.4-ga5

• 7.3.3-ga4

• 7.3.2-ga3

• 7.3.1-ga2

• 7.3.0-ga1

• 7.2.1-ga2

• 7.2.0-ga1

• 7.2.0-rc3

• 7.2.0-rc2

• 7.2.0-b3

• 7.2.0-b2

• 7.2.0-b1

• 7.2.0-a1

• 7.2.0-m2

• 7.1.3-ga4

• 7.1.2-ga3

• 7.1.1-ga2

• 7.1.0-ga1

• 7.1.0-rc1

• 7.1.0-b3

• 7.1.0-b2

• 7.1.0-b1

• 7.1.0-a2

• 7.1.0-a1

• 7.1.0-m2

• 7.1.0-m1

• 7.0.6-ga7

• 7.0.5-ga6

• 7.0.4-ga5

• 7.0.3-ga4

• 7.0.2-ga3

• 7.0.1-ga2

• 7.0.0-ga1

• 7.0.0-b7

• 7.0.0-b6

• 7.0.0-b5

• 7.0.0-b4

• 7.0.0-b3

• 7.0.0-b2

• 7.0.0-b1

• 7.0.0-a5

• 7.0.0-a4

• 7.0.0-a3

• 7.0.0-a2

• 7.0.0-a1

• 7.0.0-m7

• 7.0.0-m6

• 7.0.0-m5

• 7.0.0-m4

• 7.0.0-m3

• 7.0.0-m2

• 7.0.0-m1

In addition to this we would have to add DXP versions and branches. So, obviously, we don't want to publish so many packages to npmjs.com unless absolutely needed.

Most probably, we may do with one package per major version, then we release new versions whenever:

1. Something needs to be fixed
2. A new GA or similar is released

So, for example, a possible series of versions for package portal-7.0 could be:

• 7.0.0: the target platform for 7.0.0-ga1 (assuming we don't release for milestones and alphas)
• 7.0.1: some internal fix to the target platform itself (still for 7.0.0-ga1)
• 7.0.2: when 7.0.1-ga2 is released (and only if it changes anything)
• 7.0.3: when 7.0.2-ga3 is released (and only if it changes anything)
• ...

As long as we don't branch version releases, this would suffice, I think.

Ideally, we would also compile a table (in the wiki of this project, for example) to track npm version numbers and portal versions.

I don't know what is the release cycle of DXP, but we may do something similar using a different package name (for example: dxp-7.0 instead of portal-7.0).

If anyone can think of any scheme, I'm very open to hear it. Thinks that come to mind are, for example, using npm dist-tags to alias version numbers (in the above example, we may define a dist-tag 7.0.1-ga2 pointing to 7.0.2, for instance).

[bryceosterhaus]

If anyone can think of any scheme, I'm very open to hear it.

I typically like to go for simple and small first, so I like your idea of starting with one version per major release. For future versions and updates I think we can possibly utilize the version of the npm package as well but maybe we wait and see what the updates for the target platform would actually look like. Overall though, this plan moving forward looks good to me.

[izaera]

Mmm, I'm now seeing I got it wrong...

Where it says:

So, for example, a possible series of versions for package portal-7.0 could be:

- 7.0.0: the target platform for 7.0.0-ga1 (assuming we don't release for milestones and alphas)
- 7.0.1: some internal fix to the target platform itself (still for 7.0.0-ga1)
- 7.0.2: when 7.0.1-ga2 is released (and only if it changes anything)
- 7.0.3: when 7.0.2-ga3 is released (and only if it changes anything)
- ... 

It should be:

So, for example, a possible series of versions for package portal-7.0 could be:

- 1.0.0: the target platform for 7.0.0-ga1 (assuming we don't release for milestones and alphas)
- 1.0.1: some internal fix to the target platform itself (still for 7.0.0-ga1)
- 1.0.2: when 7.0.1-ga2 is released (and only if it changes anything)
- 1.0.3: when 7.0.2-ga3 is released (and only if it changes anything)
...

Also, I've realized that this schema will never be able to honor semantic versioning as required by npm because we may need to release a fix for 7.0.0-ga1 after 7.0.2-ga3 has been released, so we would need another extra number.

This can be fixed by using the major/minor version to point to different Liferay 7.0 releases, like this:

- 1.0.0: the target platform for 7.0.0-ga1 (assuming we don't release for milestones and alphas)
- 1.0.1: some internal fix to the target platform itself (still for 7.0.0-ga1)
- 2.0.0: when 7.0.1-ga2 is released (and only if it changes anything)
- 3.0.0: when 7.0.2-ga3 is released (and only if it changes anything)

I still have doubts if we should map the major version number to GAs, or the minor one. Also, I'm not sure if this type of issue may also appear in more scenarios (I'm thinking about hot fixes, or any deviation from the usual versioning branches in Liferay Portal) 🤔 .

[bryceosterhaus]

I don't know if we can follow semver exactly if we are publishing a platform for every major dxp release, but I think we can get close. My "Alt NPM" one I think is closest to your last example and I think it makes sense to me. I'm sure we might run into issues in some weird edge case, but its likely because DXP breaks some versioning rule.

DXP Version	NPM Package	NPM Version (dxp 7.4.x -> npm 1.x.0)	Alt NPM (dxp 7.4.0-gaX -> npm x.0.0)
7.4.2-ga3	portal-7.4	1.2.0	3.0.0
7.4.1-ga2	portal-7.4	1.1.0	2.0.0
7.4.0-ga1	portal-7.4	1.0.0	1.0.0
7.3.7-ga8	portal-7.3	1.7.0	8.0.0
7.3.6-ga7	portal-7.3	1.6.0	7.0.0
7.3.5-ga6	portal-7.3	1.5.0	6.0.0

Internal fixes:

1. Use patch version for internal fixes
2. Use minor & patch versions for internal fixes

[izaera]

Yeah, that will leave more room for internal management (two numbers instead of one). I think this is the best to start with 👍 .

[izaera]

I'll begin analyzing point 3 now: how to create the platforms. I already have a script for the latest versions but it needs a local copy of liferay-portal pointing to the desired tag. I doubt it will be future-proof as it needs to inspect npmscripts.config.js and that will change between versions, most probably, but I need to decide if keeping it will be less work than doing everything by hand.

I will begin by tweaking it for older versions (7.3, 7.2, and so on, and see what happens 🤞 ).

[izaera]

Point 3

• 7.4 series: npmscripts.config.js contains public npm-exports
• 7.3 series: npmscripts.config.js exists but doesn't contain public npm-exports (I guess we can find them looking at .npmbundlerrc files or something like that; I need to check)
• 7.2 series: no npmscripts.config.js (same as 7.3 ?)
• 7.1 series: no npmscripts.config.js (same as 7.3 ?)
• 7.0 series: no support for npm-bundler in this branch (I was wrong in the previous comment)

[izaera]

As show in the create-platform script the only thing we need to grab from any liferay-portal version to create its target platform is the list of npm packages it bundler-exports.

That information is inside the global npmscripts.config.js file for the latest versions (7.4+), but for previous versions it must be derived from the .npmbundlerrc files or built by hand.

I'm gonna check if there's any way to derive the list of bundler-exports from .npmbundlerrc files in previous versions. If not, we will need to define it by hand and/or send PRs for older versions to use a global npmscripts.config.js file (however I'm not 100% sure if that would be possible).

[izaera]

So, I have created target platforms for all 7.4-ga* versions. Now we have to decide how to publish them in npmjs.com.

I would say we don't need a formal release process like we do with other packages, just a npm publish with the repo pointing to the relevant commit (and maybe create a git tag, at most). This is because:

1. We will need to branch heavily (once for every released version) if we need to fix/change anything in the future.
2. Usually there will be a 1:1 relation between commits and npmjs versions
3. The contents of target platforms are so small in any case, that I think it will be almost impossible that we need to check/follow the git history in the future for anything.

Related to 1, should we need branching, we could create proper git branches in the repo, or simply stack commits unsorted. I would say we don't need branches because they would pollute this repo a lot for very little benefit. And, for the reasons stated above, a stack of unsorted commits will probably be enough.

Graphically, we could have something like this:

$ git log
feat(portal-7.4): add 7.4.3.4-ga4 version    # npm 4.0.0
feat(portal-7.4): add 7.4.2-ga3 version    # npm 3.0.0
feat(portal-7.4): fix 7.4.1-ga2 version    # npm 2.0.1
feat(portal-7.4): fix 7.4.0-ga1 version    # npm 1.0.1
feat(portal-7.4): add 7.4.1-ga2 version    # npm 2.0.0
feat(portal-7.4): add 7.4.0-ga1 version    # npm 1.0.0

So, every time we need to fix anything (will happen rarely) we will pile a stack of commit for all released versions to the date with the fixes on top of the history.

This will leave the HEAD always pointing to the very highest released version, which is something good..

[bryceosterhaus]

I definitely prefer a stack of commits instead of multiple branches. I think we can just try this simplest approach first and see how it goes. If we need to pivot at a later date and move to branches or something else, we can always do so.

[bryceosterhaus]

One thought I just had. Do we want to set up some sort of automation for releases since we now have this 2-week release cycle? I imagine it being a headache to manually do every 2 weeks.

[izaera]

Yes, I'm waiting to see what we really need, but it's definitely something that needs automation because otherwise it will be a nightmare 😱 . Maybe we can create a script and associate it to a github action, so that it can be run from GH's UI by anyone with permissions...

[bryceosterhaus]

@izaera I like that idea of creating a script that we can execute with a click from github. I don't know if we can jump to full automation yet since the 2 week release is still pretty new, but if we can reduce the amount of manual steps, that would be great.

[izaera]

Point 8

We should provide some way to migrate old projects based on liferay-npm-build-support so that we don't have to maintain it, in addition to target-platforms. However, given the nature of development projects, that could be very difficult for some customers (or even impossible), if we want to do it automatically.

Also, we have not ported the translate command from liferay-npm-build-support to target-platforms because it has not been evolved since long ago and nobody has contacted us regarding it, so I suspect nobody uses it...

Same happens for the npm run start functionality. We created the start eject feature to slowly deprecate/freeze it, but not everybody has opted out (we know that from issues and conversations) so people willing to move from liferay-npm-build-support to target-platforms will lose that functionality.

Possible solutions to this:

1. Migrate the functionalities to target-platforms. We may not want to do this as both functionalities are not being actively maintained and this is a great moment to "get rid" of them instead of keeping them in the baggage.
2. Provide some "extra" package that people may use with target-platforms to keep the old functionality working at the same level it was before and declare it frozen/deprecated.
3. Simply drop the functionalities and declare them no longer supported.

I would like to know what you think about this @nhpatt @dsanz @bryceosterhaus ...

[izaera]

Regarding the automatic upgrade of projects based on liferay-npm-build-support we may provide some command in liferay-cli to do some automatic steps, then leave the rest up to the user, possibly asking us for help in case they have doubts (i.e: inform them about that possibility).

I'm going to implement this automatic -limited- process with the things I know that need be migrated, then, when we decide what to do with npm run start and npm run translate, I may continue with that.

I'm starting to convince myself that 2 (above) is the most reasonable option. It doesn't abandon current users and, at the same time, doesn't make us pay a toll later for having included those features in target platforms again.

[bryceosterhaus]

Do we have any metrics on people who use npm run start or eject? We may consider just dropping them as features until we hear of an explicit need for them. If there is a need, I think we could aim at #2 and either utilize dev-server or something new that would give them that functionality. But I think we could possibly just not worry about them yet until it gets brought up.

This thought also makes me wonder about metrics for our tooling... I seems like we often ask "does anyone use this?" and we might want to start considering to add some analytics in our tooling so that we can see just how many people use our tooling and what features they use. I'm sure react-scripts has something like this that we could learn from, and since everyone is rightly concerned about privacy, we should provide an opt-out. We can open a new issue for this idea of analytics and metrics.

[izaera]

Do we have any metrics on people who use npm run start or eject?

Nope. We don't have metrics for anything :-(, just the communications I make with some users.

We may consider just dropping them as features until we hear of an explicit need for them

OK, let's do this: in the upgrade process I will drop them and tell the user about it so that, if it is really needed, they can contact us and we may provide 2. Providing 2 should be more or less straightforward as it is just a matter of packaging what we have in liferay-npm-build-support now in a new standalone package.

In case the user cannot drop the feature for any reason, they can always choose not to upgrade, contact us, and wait for the implementation, then upgrade. I mean, they don't need to upgrade if they don't want to. It's only that we encourage people to follow the new path, but don't make it mandatory for now.

This thought also makes me wonder about metrics for our tooling... I seems like we often ask "does anyone use this?" and we might want to start considering to add some analytics in our tooling so that we can see just how many people use our tooling and what features they use.

The tool had metrics and we even have a user in google analytics to access them. However, to exploit and maintain those metrics we need more resources (people) because it's a time consuming task. As I was almost alone and nobody took advantage of the metrics I deactivated them long ago.

But we can resurrect them if we see it fits. The code is there and it worked.

[bryceosterhaus]

But we can resurrect them if we see it fits.

I'd be curious to just get an idea of what tasks people use and how many people are using them. Not super important, but it is a nice to have.

OK, let's do this: in the upgrade process I will drop them and tell the user about it so that, if it is really needed, they can contact us and we may provide 2. Providing 2 should be more or less straightforward as it is just a matter of packaging what we have in liferay-npm-build-support now in a new standalone package.

Sounds good to me!

[izaera]

This is done, apparently 🎉