Pixiu Control Plane Support Service Metadata #484

AlbumenJ · 2022-09-08T09:42:52Z

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

AlbumenJ · 2022-09-08T10:53:03Z

需要定义一个 proto 用于上报应用级服务发现的元数据信息和接口的服务定义信息，前者是应用级服务发现的时候消费端通过注册中心拿到 revision 之后获取全量服务信息的接口，后者是客户端上报服务元数据的信息用于记录接口的各种参数，满足运维的需要（如服务测试等）技术上本质就是一个存储的功能，提供 set 和 get 的接口就可以了，目前阶段还不需要对数据进行加工。这个 proto 可以直接绑定到 xDS 的 grpc channel 上就行。

元数据的主要涉及的接口有
set(app string, revision string, metadataInfo string)
get(app string, revision string)

服务定义的有
set(identifier string, definition string)
get(identifier string)

https://dubbo.apache.org/zh/blog/2021/06/02/dubbo3-%E5%BA%94%E7%94%A8%E7%BA%A7%E6%9C%8D%E5%8A%A1%E5%8F%91%E7%8E%B0/

https://dubbo.apache.org/zh/overview/tasks/mesh/proxyless/

MasterKenway · 2022-09-08T13:44:01Z

working on it

MasterKenway · 2022-09-08T16:10:11Z

针对服务上线对应元数据的删除问题较为复杂，在第一版中暂时不实现
revision 区分于 etcd 的 MVCC，可以视作一个根据元数据内容计算出来的 MD5 的 key （哈希算法需要考虑碰撞的情况，考虑使用比较不容易碰撞的算法或设计一个检查机制）
注册中心使用的服务元数据”，用于服务发现的， “服务运维使用的服务元数据”，描述了接口出入参等，用于运维的。前者可以理解成是为了构建url，后者则是让运维的人知道现在发布的接口具体有哪些参数（参考 Java 版 Dubbo 的实现）

MasterKenway · 2022-09-10T07:43:18Z

元数据对于 Endpoint 维度来说是静态的，在服务启动之后不会进行改动，因此对于服务端来说可以只考虑上报的逻辑
原本消费端通过 Listener 监听 Provider 的元数据变动，现在则是将监听逻辑做到控制面中，通过 grpc 主动推送给消费端

chickenlj · 2022-09-15T07:12:09Z

这两类 Metadata，我觉得有必要区分开来定义，对于每一部分目前需要确认的主要内容如下：

描述资源的 CRD 定义，主要内容是 revision、metadata
用于通信的协议定义，基于 proto（service、message）

chickenlj · 2022-09-15T07:13:08Z

2. 可以视作一个根据元数据内容计算出来的 MD5 的 key （哈希算法需要考虑碰撞的情况，考虑使用比较不容易碰撞的算法或设计一个检查机制）

这个 md5 计算过程应该是由数据面完成的，控制面可以只做存储不用关心计算方式

chickenlj · 2022-09-15T07:14:51Z

5. 原本消费端通过 Listener 监听 Provider 的元数据变动，现在则是将监听逻辑做到控制面中，通过 grpc 主动推送给消费端

消费端每次都会主动通过revision查询metadata，不需要推送过程。

MasterKenway · 2022-09-25T13:08:46Z

https://dubbogoproxy.yuque.com/dubbogoproxy/vld1hq/hm6vdz

MasterKenway · 2022-09-26T01:31:09Z

服务发现元数据
CRD 定义

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # name must match the spec fields below, and be in the form: <plural>.<group>
  name: metadata.networking.dubbo.io
spec:
  # group name to use for REST API: /apis/<group>/<version>
  group: metadata.networking.dubbo.io
  # list of versions supported by this CustomResourceDefinition
  versions:
    - name: v1
      # Each version can be enabled/disabled by Served flag.
      served: true
      # One and only one version must be marked as the storage version.
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                applicationName:
                  type: string
                revision:
                  type: string
                metadataInfo:
                  type: string
                
  # either Namespaced or Cluster
  scope: Namespaced
  names:
    # plural name to be used in the URL: /apis/<group>/<version>/<plural>
    plural: metadata
    # singular name to be used as an alias on the CLI and for display
    singular: metadata
    # kind is normally the CamelCased singular type. Your resource manifests use this.
    kind: Metadata
    # shortNames allow shorter string to match your resource on the CLI
    shortNames:
    - md

Proto

syntax = "proto3";

package pixiu.pkg.metadata;

option go_package = "pixiu/pkg/metadata/protos";

service MetadataService {
    rpc Publish(PublishMetadataRequest ) returns (PublishMetadataResponse );
    rpc Get(GetMetadataRequest) returns (GetMetadataResponse);
}

message PublishMetadataRequest {
    string application_name = 1;
    string revision = 2;
    string metadata_info = 3;
}

message PublishMetadataResponse {
}

message GetMetadataRequest {
    string application_name = 1;
    string revision = 2;
}

message GetMetadataResponse {
    string metadata_info = 1;
}

流程设计

服务启动时上报元数据到控制面，通过控制面注册到 k8s CRD 中
监听服务下线，对于 application 维度下的所有服务均下线时，移除对应的元数据
● ~~在服务发现中增加监听器监听对应的上下线事件~~
● 为元数据设置一个时间戳，控制面以一天的频率去监听这部分数据的更新状态，客户端以更高的频率上报数据，若在一天之内的数据没有更新则视作服务下线，将对应的元数据删除。
存疑
由于控制面不关心 revision 参数的生成，需要考虑出现重复的情况
● ~~在数据面更换其他的哈希算法~~
● 直接更新对应 revision 的数据
服务下线时的监听
服务定义
CRD

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # name must match the spec fields below, and be in the form: <plural>.<group>
  name: service-definition.networking.dubbo.io
spec:
  # group name to use for REST API: /apis/<group>/<version>
  group: service-definition.networking.dubbo.io
  # list of versions supported by this CustomResourceDefinition
  versions:
    - name: v1
      # Each version can be enabled/disabled by Served flag.
      served: true
      # One and only one version must be marked as the storage version.
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                identifier:
                  type: string
                serviceDefinition:
                  type: string
                	
  # either Namespaced or Cluster
  scope: Namespaced
  names:
    # plural name to be used in the URL: /apis/<group>/<version>/<plural>
    plural: serviceDefinitions
    # singular name to be used as an alias on the CLI and for display
    singular: serviceDefinition
    # kind is normally the CamelCased singular type. Your resource manifests use this.
    kind: ServiceDefinition
    # shortNames allow shorter string to match your resource on the CLI
    shortNames:
    - sd

Proto

syntax = "proto3";

package pixiu.pkg.metadata;

option go_package = "pixiu/pkg/metadata/protos"

service ServiceDefinitionService {
    rpc Publish(PublishServiceDefinitionRequest c) returns (PublishServiceDefinitionResponse );
    rpc Get(GetServiceDefinitionRequest) returns (GetServiceDefinitionResponse);
}

message PublishServiceDefinitionRequest {
    string identitfier = 1;
  	string serivce_definition = 2;
}

message PublishServiceDefinitionResponse {
}

message GetServiceDefinitionRequest {
   string identitfier = 1;
}

message GetServiceDefinitionResponse {
    string serivce_definition = 1;
}

注

proto 中的错误信息处理均通过 grpc error handling 进行处理
元数据中心以及服务定义中的数据保存时只保存原始字符串
参考连接
K8s 的核心是 API 而非容器：从理论到 CRD 实践（2022） http://arthurchiao.art/blog/k8s-is-about-apis-zh/#33-api-%E6%98%AF-sql

MasterKenway · 2022-10-30T13:11:44Z

目前进度遇到使用 istio build-tools 生成 CRD 相关代码时遇到 import proto 文件查找失败的问题
repo: https://github.com/dubbo-go-pixiu/operator-api
日志如下：

kenway@DESKTOP-9LOOPCM:/mnt/d/Workspace/GolandProjects/api$ make gen
./clean.sh
./gen.sh
Syncing ./networking/v1beta1/destination_rule.proto from networking/v1alpha3/destination_rule.proto
Syncing ./networking/v1beta1/gateway.proto from networking/v1alpha3/gateway.proto
Syncing ./networking/v1beta1/service_entry.proto from networking/v1alpha3/service_entry.proto
Syncing ./networking/v1beta1/sidecar.proto from networking/v1alpha3/sidecar.proto
Syncing ./networking/v1beta1/virtual_service.proto from networking/v1alpha3/virtual_service.proto
Syncing ./networking/v1beta1/workload_entry.proto from networking/v1alpha3/workload_entry.proto
Syncing ./networking/v1beta1/workload_group.proto from networking/v1alpha3/workload_group.proto
authentication/v1alpha1/policy.proto:23:8:google/api/field_behavior.proto: does not exist
networking/v1alpha3/destination_rule.proto:16:8:google/api/field_behavior.proto: does not exist
networking/v1alpha3/envoy_filter.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1alpha3/gateway.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1alpha3/service_entry.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1alpha3/sidecar.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1alpha3/virtual_service.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1alpha3/workload_entry.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1alpha3/workload_group.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1beta1/destination_rule.proto:16:8:google/api/field_behavior.proto: does not exist
networking/v1beta1/gateway.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1beta1/service_entry.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1beta1/sidecar.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1beta1/virtual_service.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1beta1/workload_entry.proto:17:8:google/api/field_behavior.proto: does not exist
networking/v1beta1/workload_group.proto:17:8:google/api/field_behavior.proto: does not exist
security/v1beta1/authorization_policy.proto:16:8:google/api/field_behavior.proto: does not exist
security/v1beta1/jwt.proto:16:8:google/api/field_behavior.proto: does not exist
type/v1beta1/selector.proto:16:8:google/api/field_behavior.proto: does not exist
make[1]: *** [Makefile.core.mk:36: gen-proto] Error 100
make: *** [Makefile:44: gen] Error 2

AlexStocks · 2023-02-12T13:20:52Z

0215日周会，上面问题都已经解决掉。蔡同学还在 review 中。

AlbumenJ assigned MasterKenway Sep 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pixiu Control Plane Support Service Metadata #484

Pixiu Control Plane Support Service Metadata #484

AlbumenJ commented Sep 8, 2022

AlbumenJ commented Sep 8, 2022

MasterKenway commented Sep 8, 2022

MasterKenway commented Sep 8, 2022

MasterKenway commented Sep 10, 2022

chickenlj commented Sep 15, 2022 •

edited

chickenlj commented Sep 15, 2022

chickenlj commented Sep 15, 2022

MasterKenway commented Sep 25, 2022

MasterKenway commented Sep 26, 2022 •

edited

MasterKenway commented Oct 30, 2022

AlexStocks commented Feb 12, 2023

Pixiu Control Plane Support Service Metadata #484

Pixiu Control Plane Support Service Metadata #484

Comments

AlbumenJ commented Sep 8, 2022

AlbumenJ commented Sep 8, 2022

MasterKenway commented Sep 8, 2022

MasterKenway commented Sep 8, 2022

MasterKenway commented Sep 10, 2022

chickenlj commented Sep 15, 2022 • edited

chickenlj commented Sep 15, 2022

chickenlj commented Sep 15, 2022

MasterKenway commented Sep 25, 2022

MasterKenway commented Sep 26, 2022 • edited

MasterKenway commented Oct 30, 2022

AlexStocks commented Feb 12, 2023

chickenlj commented Sep 15, 2022 •

edited

MasterKenway commented Sep 26, 2022 •

edited