Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade simdjson lib to 3.7.0 (from 3.2.0) #9041

Closed
wants to merge 1 commit into from

Conversation

PHILO-HE
Copy link
Contributor

@PHILO-HE PHILO-HE commented Mar 12, 2024

Recently, we are proposing a patch to support a Spark JSON function based on simdjson lib. It requires to use json path API which is only available since 3.7.0. So we are trying to upgrade the lib.

Background:

  1. See the discussion with @mbasmanova:
    Optimize get_json_object Spark function using simdjson #5179 (comment) (3.7.0 is enough)
  2. I also note Masha's discussion with simdjson community to request supporting json path:
    Add full support for JSONPath simdjson/simdjson#2070

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Mar 12, 2024
Copy link

netlify bot commented Mar 12, 2024

Deploy Preview for meta-velox canceled.

Name Link
🔨 Latest commit 2b1875a
🔍 Latest deploy log https://app.netlify.com/sites/meta-velox/deploys/6613dbb5be88b900088a5850

@@ -31,7 +31,7 @@ by Velox. See details on bundling below.
| xsimd | 10.0.0 | Yes |
| re2 | 2021-04-01 | Yes |
| fmt | 10.1.1 | Yes |
| simdjson | 3.2.0 | Yes |
| simdjson | 3.7.0 | Yes |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we actually REQUIRE this version (i see the version set in cml.txt but I mean in the C++? Or is it just a nice to have so we bump the version in resolve_dependency?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @assignUser, thanks for your review!
I posted some background message in PR description. As explained, we would like to use a simdjson API which is available since 3.7.0.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, then this change makes sense. Just wanted to make sure we are not raising the minimum version needlessly. 👍

@kgpai
Copy link
Contributor

kgpai commented Mar 12, 2024

@PHILO-HE We have some dependencies on simdjson internally, I need to make sure that these changes wont affect them. Please give me some time to look into this.

@kgpai
Copy link
Contributor

kgpai commented Mar 12, 2024

@PHILO-HE So internally Meta still uses 3.1.8 (which is close to 3.2.0 but not exactly it). We will also need to upgrade our internal dependencies to 3.7.0 and then validate that this doesnt break anything or has any performance regressions. I will require some time for this.

@PHILO-HE
Copy link
Contributor Author

PHILO-HE commented Mar 13, 2024

@PHILO-HE So internally Meta still uses 3.1.8 (which is close to 3.2.0 but not exactly it). We will also need to upgrade our internal dependencies to 3.7.0 and then validate that this doesnt break anything or has any performance regressions. I will require some time for this.

@kgpai, thanks so much for your time!

@PHILO-HE
Copy link
Contributor Author

PHILO-HE commented Apr 8, 2024

@PHILO-HE So internally Meta still uses 3.1.8 (which is close to 3.2.0 but not exactly it). We will also need to upgrade our internal dependencies to 3.7.0 and then validate that this doesnt break anything or has any performance regressions. I will require some time for this.

Hi @kgpai, do you have any update? Thanks!

@mbasmanova
Copy link
Contributor

@PHILO-HE I'm upgrading simdjson internally to 3.8.0. Would it be possible to update to the same version here?

@mbasmanova mbasmanova changed the title Upgrade simdjson lib to 3.7.0 Upgrade simdjson lib to 3.7.0 (from 3.2.0) Apr 11, 2024
@assignUser
Copy link
Collaborator

Superseeded by #9456

@assignUser assignUser closed this Apr 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants