Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: migrate to pnpm #2560

Draft
wants to merge 16 commits into
base: main
Choose a base branch
from
Draft

chore: migrate to pnpm #2560

wants to merge 16 commits into from

Conversation

kark
Copy link
Contributor

@kark kark commented Jul 11, 2023

Summary

Description

@kark kark added the 🚧 Status: WIP Work in progress label Jul 11, 2023
@changeset-bot
Copy link

changeset-bot bot commented Jul 11, 2023

🦋 Changeset detected

Latest commit: 4fd2bd0

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 95 packages
Name Type
@commercetools-uikit/localized-multiline-text-field Minor
@commercetools-uikit/localized-multiline-text-input Minor
@commercetools-uikit/async-creatable-select-field Minor
@commercetools-uikit/async-creatable-select-input Minor
@commercetools-uikit/localized-rich-text-input Minor
@commercetools-uikit/selectable-search-input Minor
@commercetools-uikit/spacings-inset-squish Minor
@commercetools-uikit/secondary-icon-button Minor
@commercetools-uikit/creatable-select-field Minor
@commercetools-uikit/creatable-select-input Minor
@commercetools-uikit/localized-money-input Minor
@commercetools-uikit/localized-text-field Minor
@commercetools-uikit/multiline-text-field Minor
@commercetools-uikit/localized-text-input Minor
@commercetools-uikit/multiline-text-input Minor
@commercetools-uikit/search-select-field Minor
@commercetools-uikit/search-select-input Minor
@commercetools-uikit/accessible-button Minor
@commercetools-uikit/async-select-field Minor
@commercetools-uikit/async-select-input Minor
@commercetools-uikit/secondary-button Minor
@commercetools-uikit/search-text-input Minor
@commercetools-uikit/spacings-inline Minor
@commercetools-uikit/date-range-field Minor
@commercetools-uikit/date-range-input Minor
@commercetools-uikit/primary-action-dropdown Minor
@commercetools-uikit/spacings-inset Minor
@commercetools-uikit/spacings-stack Minor
@commercetools-uikit/primary-button Minor
@commercetools-uikit/date-time-field Minor
@commercetools-uikit/date-time-input Minor
@commercetools-uikit/rich-text-input Minor
@commercetools-uikit/rich-text-utils Minor
@commercetools-uikit/password-field Minor
@commercetools-uikit/checkbox-input Minor
@commercetools-uikit/password-input Minor
@commercetools-uikit/flat-button Minor
@commercetools-uikit/icon-button Minor
@commercetools-uikit/link-button Minor
@commercetools-uikit/number-field Minor
@commercetools-uikit/select-field Minor
@commercetools-uikit/number-input Minor
@commercetools-uikit/select-input Minor
@commercetools-uikit/select-utils Minor
@commercetools-uikit/toggle-input Minor
@commercetools-uikit/collapsible-motion Minor
@commercetools-uikit/data-table-manager Minor
@commercetools-uikit/money-field Minor
@commercetools-uikit/radio-field Minor
@commercetools-uikit/input-utils Minor
@commercetools-uikit/money-input Minor
@commercetools-uikit/radio-input Minor
@commercetools-uikit/accessible-hidden Minor
@commercetools-uikit/collapsible-panel Minor
@commercetools-uikit/date-field Minor
@commercetools-uikit/text-field Minor
@commercetools-uikit/time-field Minor
@commercetools-uikit/date-input Minor
@commercetools-uikit/text-input Minor
@commercetools-uikit/time-input Minor
@commercetools-uikit/loading-spinner Minor
@commercetools-uikit/notifications Minor
@commercetools-uikit/view-switcher Minor
@commercetools-uikit/field-errors Minor
@commercetools-uikit/collapsible Minor
@commercetools-uikit/constraints Minor
@commercetools-uikit/field-label Minor
@commercetools-uikit/data-table Minor
@commercetools-uikit/pagination Minor
@commercetools-uikit/messages Minor
@commercetools-uikit/tooltip Minor
@commercetools-uikit/avatar Minor
@commercetools-uikit/icons Minor
@commercetools-uikit/label Minor
@commercetools-uikit/stamp Minor
@commercetools-uikit/card Minor
@commercetools-uikit/grid Minor
@commercetools-uikit/link Minor
@commercetools-uikit/text Minor
@commercetools-uikit/localized-utils Minor
@commercetools-local/generator-package-json Minor
@commercetools-uikit/calendar-utils Minor
@commercetools-uikit/tag Minor
visual-testing-app Minor
@commercetools-local/generator-readme Minor
@commercetools-uikit/hooks Minor
@commercetools-uikit/utils Minor
@commercetools-frontend/ui-kit Minor
@commercetools-uikit/design-system Minor
@commercetools-uikit/fields Minor
@commercetools-uikit/inputs Minor
@commercetools-uikit/spacings Minor
@commercetools-uikit/buttons Minor
@commercetools-uikit/calendar-time-utils Minor
@commercetools-uikit/i18n Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel
Copy link

vercel bot commented Jul 11, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
ui-kit ❌ Failed (Inspect) Jul 31, 2023 8:33am

Comment on lines -6 to +7
import TextInput from '@commercetools-uikit/text-input';
// eslint-disable-next-line import/no-unresolved
import TextInput from '../../../packages/components/inputs/text-input';
Copy link
Contributor Author

@kark kark Jul 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the moment of writing this comment storybook builds are not consistent.
But in order to mitigate the effect of yarn berry "ghost-dependencies" yet not adding any circular dependencies to the monorepo, I have refactored most of the stories to load the packages based on the relative path.

This was a convention in some of the stories created earlier, e.g. here or here that I think we should go back to (instead of falling back on the hoisted packages).

Additionally, in this particular case the eslint-disable comment was added because this whole package is copied to presets and then the path is wrong. (As a side note, the story is loaded from here, not from presets)

Comment on lines -20 to -21
echo "Patching packages"
yarn patch-package
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is now done by pnpm itself during install (https://pnpm.io/cli/patch)

Comment on lines 37 to +40
"devDependencies": {
"react": "17.0.2"
"@types/react": "17.0.59",
"react": "17.0.2",
"react-select": "5.7.3"
Copy link
Contributor Author

@kark kark Jul 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

react-select added to devDependencies only due to some missing types. Same thing was done in other packages

"@typescript-eslint/eslint-plugin": "5.62.0",
"@typescript-eslint/parser": "5.62.0",
"core-js-compat": "^3.23.4",
"nwsapi": "2.2.5"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kark
Copy link
Contributor Author

kark commented Jul 21, 2023

For the time being the migration is blocked by failing storybook builds.

The issue (as I see it) is the overall amount of modules (installed with pnpm) to process with storybook’s internal instance of webpack (v4).

I managed to get it started locally only a handful of times. The builds typically end up with FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory error.

Increasing max-old-space-size, doesn’t help even when set to over 8GB.

I have tried:

  • alternative dependency hoisting configs that pnpm provides (hoisting and shameful-hoisting)
  • adding docs to monorepo workspaces
  • building docs in a linux-based container for local dev (same thing happens in linux env)
  • lots of other solutions from storybook & pnpm gh issues (not listing them here)

In a fork of uikit with minimal amount of packages and storybook works fine there in combination with pnpm. Even in the uikit monorepo on kk-pnpm branch reducing number of packages to load by deleting e.g. packages/components/inputs & fields enables webpack to bundle.

@@ -0,0 +1 @@
node-linker=hoisted
Copy link
Contributor Author

@kark kark Jul 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes storybook v5 work with pnpm (obviously at a cost of flat node_modules structure and hoisting of packages).

I figured it out the hard way and then I found this comment of pnpm's creator storybookjs/storybook#13428 (comment)

I assume this can only be considered as a workaround and perhaps migration to storybook v7 will unblock all benefits of pnpm for us.

Comment on lines +10 to +15
optimizeDeps: {
include: ['@emotion/react'],
},
resolve: {
dedupe: ['@emotion/react'],
},
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For some reason adding this enables proper compilation with@emotion/babel-plugin when creating a production build of visual testing app with vite.
Based on emotion-js/emotion#2795 (comment)

},
"dependencies": {
"@babel/core": "7.22.8",
"@emotion/react": "^11.4.0",
"@emotion/styled": "^11.3.0",
"@commercetools-frontend/babel-preset-mc-app": "21.25.2",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This had to be changed from a devDependency to a regular dependency. Otherwise babel compilation with @emotion/babel-plugin didn't work properly.

@kark
Copy link
Contributor Author

kark commented Jul 27, 2023

Hey @commercetools/shield-team-fe,
could you please give it a shot locally in a spare moment?
I mean to run storybook on kk-pnpm branch.
Many thanks 🙏

@ddouglasz
Copy link
Contributor

Hey @commercetools/shield-team-fe, could you please give it a shot locally in a spare moment? I mean to run storybook on kk-pnpm branch. Many thanks 🙏

Weldon @kark , really great work 💯
Some of the things I observed from trying this out. Seems like node is restricted to >=16 <17. I don't know if that was intentional for now.
I get this error:
Unsupported engine: wanted: {"node":">=16 <17"} (current: {"node":"v18.4.0","pnpm":"8.6.9"})

After switching to node v16, I cannot seem to start locally after pnpm install which seemed to run fine, although I had a bunch of warnings. It does not start because of the following error: Command failed with ENOENT: --cwd docs start spawn --cwd ENOENT

However, navigating to the directory(docs) manually, I can start the application with pnpm start and so far, seems to work fine from my end.
Tests seem to run fine too 🦾

@kark
Copy link
Contributor Author

kark commented Jul 28, 2023

Thank you @ddouglasz 🙏

I don't know if that was intentional for now. I get this error: Unsupported engine: wanted: {"node":">=16 <17"} (current: {"node":"v18.4.0","pnpm":"8.6.9"})

This is expected, current version of storybook doesn't work in node >17.

After switching to node v16, I cannot seem to start locally after pnpm install which seemed to run fine, although I had a bunch of warnings. It does not start because of the following error: Command failed with ENOENT: --cwd docs start spawn --cwd ENOENT

Oops, many thanks for noticing.

@CarlosCortizasCT
Copy link
Contributor

I also tested this locally and both build and test scripts works fine.

Regarding Storybook, it took a while to start but it did at the end (aprox. 3 minutes).
Something I noticed is that there's a huge file (275MB) being loaded in the browser that does not exist in the main branch.

This branch files loaded:
Screenshot 2023-07-28 at 22 18 59

Main branch:
Screenshot 2023-07-28 at 22 19 17

@kark
Copy link
Contributor Author

kark commented Jul 31, 2023

I also tested this locally and both build and test scripts works fine.

Thank you Carlos 🙇‍♂️

Regarding Storybook, it took a while to start but it did at the end (aprox. 3 minutes). Something I noticed is that there's a huge file (275MB) being loaded in the browser that does not exist in the main branch.

This branch files loaded: Screenshot 2023-07-28 at 22 18 59

Main branch: Screenshot 2023-07-28 at 22 19 17

That's sadly true 🙁 Things even up in production builds,

yarn (main) pnpm
Screenshot 2023-07-31 at 09 35 44 Screenshot 2023-07-31 at 09 34 30

but in webpack's development mode the difference in the main bundle size is huge.
I'm not sure what to do about it, I added some missing dependencies but they there's no chance they are the reason behind the difference.

@kark
Copy link
Contributor Author

kark commented Jul 31, 2023

Hey @commercetools/shield-team-fe,
It seems to me that some issues with the migration have already been addressed, there are still some workarounds used (e.g. node-linker=hoisted). Many thanks for the comments up until now.

I'd appreciate your feedback and further suggestions 🌟💪
Update 01.08 - please disregard until we have some conclusions from the spike 👇

@kark kark requested review from a team and removed request for a team July 31, 2023 08:55
@kark
Copy link
Contributor Author

kark commented Aug 1, 2023

During today's meeting we discussed the current state of migration. The key takeaway was that we decided to work on a spike where we'll attempt to migrate Storybook to version 7. The migration is intended to allow us to use Vite as the bundler and determine if this will enable migration to pnpm without any workarounds.

@kark
Copy link
Contributor Author

kark commented Aug 1, 2023

In the following branch kk-pnpm-storybook-v7 storybook v7 is introduced (all credit for the migration to storybook v7 goes to @CarlosCortizasCT 🙇‍♂️).

  • storybook dev build:
    pnpm install, pnpm build, pnpm storybook

  • storybook production build:
    pnpm install, pnpm build, pnpm build-storybook, npx http-server storybook-static

This allows to make a conclusion that the setup with storybook v7 and pnpm works fine (in the scope of the spike).
The additional benefit is possibility of migration to node v18 (for now we need to use v16 in docs).

NOTES

  • only a limited number of stories was added in the spike
  • for now storybook is colocated in the monorepo's root folder

@CarlosCortizasCT
Copy link
Contributor

In the following branch kk-pnpm-storybook-v7 storybook v7 is introduced (all credit for the migration to storybook v7 goes to @CarlosCortizasCT 🙇‍♂️).

  • storybook dev build:
    pnpm install, pnpm build, pnpm storybook
  • storybook production build:
    pnpm install, pnpm build, pnpm build-storybook, npx http-server storybook-static

This allows to make a conclusion that the setup with storybook v7 and pnpm works fine (in the scope of the spike). The additional benefit is possibility of migration to node v18 (for now we need to use v16 in docs).

NOTES

  • only a limited number of stories was added in the spike
  • for now storybook is colocated in the monorepo's root folder

Wow! That's great news! 🎉

One thing we might do to have some comparison data is to configure current Storybook (v5) to use only the same stories as the ones migrated in Storybook v7 and then run a build to compare the timings.

@kark
Copy link
Contributor Author

kark commented Aug 2, 2023

One thing we might do to have some comparison data is to configure current Storybook (v5) to use only the same stories as the ones migrated in Storybook v7 and then run a build to compare the timings.

Thanks Carlos, I did what you suggested.
I think the two are not really comparable. Storybook v7 uses lazy loading, Storybook v5 pre-loads all modules.

Anyways, for the 5 stories (avatar, primary button, select input, rich text input, text input) here are the results:

Storybook v5 Storybook v7
dev mode Screenshot 2023-08-02 at 09 32 44 Screenshot 2023-08-02 at 09 37 32
prod mode Screenshot 2023-08-02 at 09 42 13 Screenshot 2023-08-02 at 09 45 08

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🚧 Status: WIP Work in progress
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants