Skip to content

Release v0.27.0

Latest
Latest
Compare
Choose a tag to compare
@lidongze0629 lidongze0629 released this 29 Mar 13:46
· 51 commits to main since this release
v0.27.0
b2db3bd
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.

We are delighted to bring a number of improvements to GraphScope, alongside the GraphScope 0.27.0 release. This release contains many important features and improvements to GraphScope FLEX、gsctl command-line utility、Graph Interactive Engine (GIE) and Groot persistent storage.

We highlight the following improvements included in this release:

1. gsctl command-line utility

gsctl is a command-line utility designed to simplify the usage of GraphScope. With gsctl, users can easily interact with GraphScope Flex product, such as Interactive(for high QPS queries) and Insight(for online BI analysis), build binaries and docker images, and perform various utility tasks related to GraphScope. See more detailed informaton from doc.

Key Features of gsctl

  • Utility Scripts: gsctl provides a collection of utility scripts that simplify common tasks such as installing dependencies, building binaries and docker images, and running test suites.
  • Client/Server Mode: With the client/server mode, users can connect to a launched coordinator and interact with specific GraphScope products behind it. This mode enables managing sessions, resources, and other product-specific functionalities.
  • Scope Management: gsctl allows users to switch between global and local scopes, enabling them to work with graphs and resources at different levels. This flexibility enhances productivity and enables efficient management of graph computation tasks.
  • Configuration Management: gsctl includes configuration options that can be customized to suit specific requirements. Users can specify a custom coordinator endpoint and manage other configuration settings as needed.

2. GraphScope FLEX

  • Support build docker images through gsctl flexbuild command.
$ git clone https://github.com/alibaba/GraphScope.git && cd GraphScope
$ python3 gsctl.py flexbuild insight/interactive --app docker
  • Support deploy Interactive and Insight product through docker image.
$ docker pull registry.cn-hongkong.aliyuncs.com/graphscope/graphscope-store:0.27.0-flex
# launch engine and coordinator service: 12312 gremlin endpoint, 55556 grpc endpoint, 8080 coordinator endpoint
$ docker run -p 12312:12312 -p 55556:55556 -p 8080:8080 registry.cn-hongkong.aliyuncs.com/graphscope/graphscope-store:0.27.0-flex /usr/local/groot/bin/start_local_cluster.sh
  • Connect to a launched coordinator and interact with the products
graphscope@host:/work/python$ gsctl connect --coordinator-endpoint http://127.0.0.1:8081
[SUCCESS] Connected to http://127.0.0.1:8081, coordinator is serving with GRAPHSCOPE_INSIGHT mode.

Try 'gsctl --help' for help.

graphscope@host:/work/python$ gsctl ls
schema
├── vertex types
│   ├── person
│   │   ├── Property(name: id, type: LONG, is_primary_key: True)
│   │   ├── Property(name: name, type: STRING, is_primary_key: False)
│   │   └── Property(name: age, type: LONG, is_primary_key: False)
│   └── software
│       ├── Property(name: id, type: LONG, is_primary_key: True)
│       ├── Property(name: name, type: STRING, is_primary_key: False)
│       └── Property(name: lang, type: STRING, is_primary_key: False)
└── edge types
    ├── (person) -[created]-> (software)
    │   ├── Property(name: edge_id, type: LONG, is_primary_key: False)
    │   └── Property(name: weight, type: DOUBLE, is_primary_key: False)
    └── (person) -[knows]-> (person)
        ├── Property(name: edge_id, type: LONG, is_primary_key: False)
        └── Property(name: weight, type: DOUBLE, is_primary_key: False)

3. Enhancements for Groot

  • Integrate a prototype of OpenTelemetry into Groot, users could switch on option trace.enabled=true, which will have a jaeger container that collect traces. And it will serve a website on POD_IP:16686.

Docker Images

# Coordinator Image
docker pull registry.cn-hongkong.aliyuncs.com/graphscope/coordinator:0.27.0

# Graph Analytical Engine(GAE) Image
docker pull registry.cn-hongkong.aliyuncs.com/graphscope/analytical:0.27.0

# Frontend component of Graph Interactive Engine(GIE) image
docker pull registry.cn-hongkong.aliyuncs.com/graphscope/interactive-frontend:0.27.0
# Executor component of Graph Interactive Engine(GIE) image 
docker pull registry.cn-hongkong.aliyuncs.com/graphscope/interactive-executor:0.27.0

# Graph Learning Engine(GLE) Image
docker pull registry.cn-hongkong.aliyuncs.com/graphscope/learning:0.27.0

# GraphScope persistent storage, user can only perform GIE query on it.
docker pull registry.cn-hongkong.aliyuncs.com/graphscope/graphscope-store:0.27.0

# Develop Image
docker pull registry.cn-hongkong.aliyuncs.com/graphscope/graphscope-dev:latest

What's Changed

  • feat(flex): refactor memory management of mmap_array to support Hugepage. by @luoxiaojian in #3505
  • fix(interactive): Fix install_dependency.sh by @zhanglei1949 in #3500
  • fix(flex): Fixed the usage of storage strategy. by @luoxiaojian in #3506
  • fix(interactive): Fix building flex with -DBUILD_HQPS=OFF by @zhanglei1949 in #3497
  • fix(interactive): 离线全量导入时, 新增wait参数, 在download全部完成后, wait指定时间(单位ms), 再进行提交 by @bufapiqi in #3509
  • ci: Fix ci failure and pre-release v0.27.0 by @lidongze0629 in #3512
  • feat(flex): Configure the memory strategy with memory-level. by @luoxiaojian in #3511
  • refactor: coordinator and gsctl under FLEX architecture by @lidongze0629 in #3481
  • chore(deps): bump werkzeug from 0.16.1 to 2.3.8 in /flex/coordinator by @dependabot in #3518
  • chore(deps): bump flask from 2.1.1 to 2.2.5 in /flex/coordinator by @dependabot in #3519
  • docs(interactive): Update doc by @zhanglei1949 in #3514
  • ci: fixes a syntax error in CI by @sighingnow in #3521
  • fix(interactive): Fix argument passing in hqps_app by @zhanglei1949 in #3522
  • chore: change manylinux2014-ci to manylinux2014 by @siyuan0322 in #3529
  • feat(flex): Implemented Immutable Csrs. by @luoxiaojian in #3527
  • fix(interactive): Fix interactive building Dockerfile by @zhanglei1949 in #3528
  • fix(interactive): Fix error when loading edge with string properties by @zhanglei1949 in #3523
  • feat(interactive): Update graph graph_algo by @zhanglei1949 in #3486
  • fix(interactive): support non-blocking data loading interface in coordinator and fix failure during flexbuild process by @lidongze0629 in #3530
  • feat(interactive): Decouple ingestor and store, remote direct RPC connections. by @siyuan0322 in #3525
  • Update README.md for SIGMOD 2024 paper by @wenyuanyu in #3534
  • fix(interactive): Integrate Gremlin Into New Compilation Stack by @shirly121 in #3332
  • fix(k8s): Bump up vineyard to v0.20.3 by @dashanji in #3532
  • fix(interactive): Fix sinking ElementID for TwoLabelVertexSet by @zhanglei1949 in #3535
  • fix(interactive): Fix PathExpand related bugs by @zhanglei1949 in #3538
  • fix(interactive): fix install_dependencies by @liulx20 in #3539
  • fix(flex): Fix warning when compiling with AOCC compilers by @zhanglei1949 in #3510
  • fix(interactive): Fix test_gremlin_timeout in Python Unit Tests by @shirly121 in #3536
  • fix(interactive): Fix Bugs of Group Returning Results from Compiler by @shirly121 in #3533
  • ci: fixes some dependencies issues by @sighingnow in #3553
  • chore: Spelling comments by @jsoref in #3549
  • docs: fixes typos in documentation by @jsoref in #3554
  • fix(analytical): Fix Louvain algorithm's move on const message by @songqing in #3551
  • docs: fixes typos in YAML configurations by @jsoref in #3555
  • feat: Impl the file upload interface for FLEX dataloading by @lidongze0629 in #3557
  • fix(interactive): support src and dst of an edge with different type by @liulx20 in #3552
  • feat(interactive): Support PTHash indexer by @liulx20 in #3558
  • fix(interactive): Fix secondary catchup error by reopening secondary by @siyuan0322 in #3567
  • fix(interactive): Fix a bug in pkscan with expected values given in Within by @BingqingLyu in #3540
  • fix(analytical): fix bug that flash app can not run on string oid graph by @acezen in #3570
  • fix: Spelling python graphscope tests by @jsoref in #3568
  • docs: fixes typos in flex coordinator by @jsoref in #3556
  • fix(interactive): Fix typo in cypher query examples by @jsoref in #3559
  • fix(analytical): trim the physical memory usage when unloading graphs by @sighingnow in #3548
  • fix(interactive): snapshot id may lag behind in some scenarios by @siyuan0322 in #3576
  • feat(interactive): Add profile to build compiler fat-jar by @lnfjpt in #3574
  • ci: fixes typos in flex coordinator by @jsoref in #3572
  • fix(interactive): Drop unused classes and modules in groot by @siyuan0322 in #3577
  • fix: fixes typos/spelling in flex engines by @jsoref in #3582
  • fix: fixes typos/spelling in flex codegen by @jsoref in #3581
  • fix(interactive): Migrating Physical Layer into New Compilation Stack by @BingqingLyu in #3560
  • fix: fixes the compiler error under rust >= 1.76.0 by @sighingnow in #3584
  • fix(interactive): 离线全量导入支持vipserver发现endpoint, 支持replay, 支持compact, 支持reopen by @bufapiqi in #3585
  • fix(python): Fixes graph.unload(): makes the warning loudly, and use the new DelData API by @sighingnow in #3593
  • fix: fixes spelling errors in analytical engine java by @jsoref in #3588
  • fix(interactive): Use a larger version upper bound by @siyuan0322 in #3595
  • fix(interactive): Rename modern_graph.yaml to graph.yaml by @zhanglei1949 in #3603
  • feat(interactive): Add entrypoint.sh for Interactive by @zhanglei1949 in #3604
  • refactor(interactive): Wrap schema parsing errors by @zhanglei1949 in #3607
  • fix(analytical): remove scripts/install_deps.sh and update the related document by @acezen in #3608
  • fix(interactive): Fix Bug When Global Id Parameters Having Long Type by @shirly121 in #3597
  • feat(interactive): Add flag files to track bulk loading progress by @zhanglei1949 in #3605
  • fix(interactive): Fix Bug in Operator Precedence of not by @shirly121 in #3598
  • fix(interactive): Migrating Physical Layer (Repartition) into New Compilation Stack by @BingqingLyu in #3586
  • fix(interactive): Fix interactive-runtime.Dockerfile by @zhanglei1949 in #3614
  • fix: fixes spelling in analytical engine core by @jsoref in #3587
  • fix(analytical): Fix graph template parsing in Java SDK by @zhanglei1949 in #3617
  • fix(analytical): Fix documentation for GRAPE-JDK by @zhanglei1949 in #3618
  • fix(interactive): use pk hash to generate eid if has pk by @siyuan0322 in #3621
  • fix(interactive): Support Gremlin union Step in Calcite-Based IR Layer by @shirly121 in #3594
  • refactor(interactive): Add type and queryStr for StoredProcedureMeta by @zhanglei1949 in #3622
  • feat(interactive): Add OpenTelemetry instrument for flex interactive by @siyuan0322 in #3628
  • fix(interactive): fix a bug in PathExpand with 'PATH_OPT' as 'SIMPLE' by @BingqingLyu in #3623
  • fix(interactive): Add Cypher Queries used in GOpt Benchmark by @BingqingLyu in #3625
  • fix(python): Enable the BUILD_VINEYARD_GRAPH_WITH_GAR compile option in gsctl by @acezen in #3627
  • fix(interactive): Fix CMakeLists.txt when otel not found in flex by @zhanglei1949 in #3633
  • ci: Building vineyard-dev image by @lidongze0629 in #3635
  • fix(learning): fix gltorch wheel building by @LiSu in #3629
  • refactor(interactive): Fix bug and use ServiceConfig to gather all configurations by @zhanglei1949 in #3634
  • fix(interactive): Fix CI for HQPS engine by @zhanglei1949 in #3638
  • fix(interactive): Fix argument parsing for enable-admin-service by @zhanglei1949 in #3640
  • feat(interactive): Support gremlin query. by @zhanglei1949 in #3575
  • feat: Integrate groot into flex architecture by @lidongze0629 in #3566
  • fix(interactive): Fix Timeout Bug & Process Results From Pegasus Synchronously by @shirly121 in #3630
  • feat(interactive): Integrate otel into groot, and add charts for one-pod by @siyuan0322 in #3641
  • fix(interactive): Fix CI Bug Caused By PR #3630 by @shirly121 in #3647
  • fix(interactive): Minor typo fix by @zhanglei1949 in #3643
  • refactor(interactive): Support Gremlin Expression in Calcite-Based IR Layer by @shirly121 in #3637
  • fix(interactive): Fix the incorrect casting from int into string when converting from YAML to JSON by @zhanglei1949 in #3649
  • refactor: Make gsctl to be more like a database design by @lidongze0629 in #3648
  • refactor: Make gsctl to be more like a database design for Groot storage by @lidongze0629 in #3651
  • fix(interactive): Fix failing to install Arrow by @liulx20 in #3652
  • fix(interactive): Introduce the high-order statistics provider GLogue into the new Compilation Stack by @BingqingLyu in #3609
  • feat(interactive): Enable starting compiler as a subprocess of interactive server by @zhanglei1949 in #3650
  • fix(interactive): Prevent persist invalid meta during abnormal shutdown. by @siyuan0322 in #3661
  • ci: Fixes CI failure on macOS by @lidongze0629 in #3662
  • fix(interactive): add store-state service back by @siyuan0322 in #3664
  • fix(interactive): Fix the occasionally incorrect pegasus server routing configuration by @siyuan0322 in #3673
  • fix(learning): Fix torch version to be compatible with gcc8 and python3.7 by @LiSu in #3677
  • fix(interactive): Fix unexpected alias when fusion Expand + GetV by @BingqingLyu in #3676
  • fix(learning): specify proper torch version for macos by @LiSu in #3679
  • fix(interactive): fix the bug in missing metadata in PathExpand by @BingqingLyu in #3667

New Contributors

Full Changelog: v0.26.0...v0.27.0

Contributors

LiSu, jsoref, and 15 other contributors
Assets 2