sdk support push to graphite (#1586)

juicedata · Mar 18, 2022 · 1daa6c3 · 1daa6c3
1 parent 685daf4
commit 1daa6c3
Show file tree

Hide file tree

Showing 9 changed files with 954 additions and 75 deletions.
diff --git a/docs/en/administration/monitoring.md b/docs/en/administration/monitoring.md
@@ -143,9 +143,9 @@ For more information about Prometheus Operator, please check [official document]
 
 ### Hadoop
 
-The [JuiceFS Hadoop Java SDK](../deployment/hadoop_java_sdk.md) supports reporting monitoring metrics to [Pushgateway](https://github.com/prometheus/pushgateway) and then letting Prometheus scrape the metrics from Pushgateway.
+The [JuiceFS Hadoop Java SDK](../deployment/hadoop_java_sdk.md) supports reporting monitoring metrics to [Pushgateway](https://github.com/prometheus/pushgateway) and [Graphite](http://graphiteapp.org/)
 
-Please enable metrics reporting with the following configuration:
+Report metrics to Pushgateway:
 
 ```xml
 <property>
@@ -154,7 +154,7 @@ Please enable metrics reporting with the following configuration:
 </property>
 ```
 
-At the same time, the frequency of reporting metrics can be modified through the `juicefs.push-interval` configuration. The default is to report once every 10 seconds. For all configurations supported by JuiceFS Hadoop Java SDK, please refer to [documentation](../deployment/hadoop_java_sdk.md#client-configurations).
+At the same time, the frequency of reporting metrics can be modified through the `juicefs.push-interval` configuration. The default is to report once every 10 seconds.
 
 :::info
 According to the suggestion of [Pushgateway official document](https://github.com/prometheus/pushgateway/blob/master/README.md#configure-the-pushgateway-as-a-target-to-scrape), Prometheus's [scrape configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) needs to set `honor_labels: true`.
@@ -176,6 +176,19 @@ $ curl -X PUT http://host:9091/api/v1/admin/wipe
 
 For more information about Pushgateway, please check [official document](https://github.com/prometheus/pushgateway/blob/master/README.md).
 
+Report metrics to Graphite:
+
+```xml
+<property>
+  <name>juicefs.push-graphite</name>
+  <value>host:port</value>
+</property>
+```
+
+At the same time, the frequency of reporting metrics can be modified through the `juicefs.push-interval` configuration. The default is to report once every 10 seconds.
+
+For all configurations supported by JuiceFS Hadoop Java SDK, please refer to [documentation](../deployment/hadoop_java_sdk.md#client-configurations).
+
 ### Use Consul as registration center
 
 :::note

diff --git a/docs/en/deployment/hadoop_java_sdk.md b/docs/en/deployment/hadoop_java_sdk.md
@@ -158,20 +158,21 @@ Please refer to the following table to set the relevant parameters of the JuiceF
 
 #### Other Configurations
 
-| Configuration             | Default Value | Description                                                  |
-| ------------------------- | ------------- | ------------------------------------------------------------ |
-| `juicefs.bucket`          |               | Specify a different endpoint for object storage              |
-| `juicefs.debug`           | `false`       | Whether enable debug log                                     |
-| `juicefs.access-log`      |               | Access log path. Ensure Hadoop application has write permission, e.g. `/tmp/juicefs.access.log`. The log file will rotate  automatically to keep at most 7 files. |
-| `juicefs.superuser`       | `hdfs`        | The super user                                               |
-| `juicefs.users`           | `null`        | The path of username and UID list file, e.g. `jfs://name/etc/users`. The file format is `<username>:<UID>`, one user per line. |
+| Configuration             | Default Value | Description                                                                                                                                                                 |
+|---------------------------| ------------- |-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `juicefs.bucket`          |               | Specify a different endpoint for object storage                                                                                                                             |
+| `juicefs.debug`           | `false`       | Whether enable debug log                                                                                                                                                    |
+| `juicefs.access-log`      |               | Access log path. Ensure Hadoop application has write permission, e.g. `/tmp/juicefs.access.log`. The log file will rotate  automatically to keep at most 7 files.           |
+| `juicefs.superuser`       | `hdfs`        | The super user                                                                                                                                                              |
+| `juicefs.users`           | `null`        | The path of username and UID list file, e.g. `jfs://name/etc/users`. The file format is `<username>:<UID>`, one user per line.                                              |
 | `juicefs.groups`          | `null`        | The path of group name, GID and group members list file, e.g. `jfs://name/etc/groups`. The file format is `<group-name>:<GID>:<username1>,<username2>`, one group per line. |
-| `juicefs.umask`           | `null`        | The umask used when creating files and directories (e.g. `0022`), default value is `fs.permissions.umask-mode`. |
-| `juicefs.push-gateway`    |               | [Prometheus Pushgateway](https://github.com/prometheus/pushgateway) address, format is `<host>:<port>`. |
-| `juicefs.push-interval`   | 10            | Prometheus push interval in seconds                          |
-| `juicefs.push-auth`       |               | [Prometheus basic auth](https://prometheus.io/docs/guides/basic-auth) information, format is `<username>:<password>`. |
-| `juicefs.fast-resolve`    | `true`        | Whether enable faster metadata lookup using Redis Lua script |
-| `juicefs.no-usage-report` | `false`       | Whether disable usage reporting. JuiceFS only collects anonymous usage data (e.g. version number), no user or any sensitive data will be collected. |
+| `juicefs.umask`           | `null`        | The umask used when creating files and directories (e.g. `0022`), default value is `fs.permissions.umask-mode`.                                                             |
+| `juicefs.push-gateway`    |               | [Prometheus Pushgateway](https://github.com/prometheus/pushgateway) address, format is `<host>:<port>`.                                                                     |
+| `juicefs.push-auth`       |               | [Prometheus basic auth](https://prometheus.io/docs/guides/basic-auth) information, format is `<username>:<password>`.                                                       |
+| `juicefs.push-graphite`    |               | [Graphite](http://graphiteapp.org/) address, format is `<host>:<port>`.                                                                                                     |
+| `juicefs.push-interval`   | 10            | Metric push interval (in seconds)                                                                                                   |
+| `juicefs.fast-resolve`    | `true`        | Whether enable faster metadata lookup using Redis Lua script                                                                                                                |
+| `juicefs.no-usage-report` | `false`       | Whether disable usage reporting. JuiceFS only collects anonymous usage data (e.g. version number), no user or any sensitive data will be collected.                         |
 
 #### Multiple file systems configuration
 

diff --git a/docs/zh_cn/administration/monitoring.md b/docs/zh_cn/administration/monitoring.md
@@ -143,9 +143,9 @@ spec:
 
 ### Hadoop
 
-[JuiceFS Hadoop Java SDK](../deployment/hadoop_java_sdk.md) 支持把监控指标上报到 [Pushgateway](https://github.com/prometheus/pushgateway)，然后让 Prometheus 从 Pushgateway 抓取指标。
+[JuiceFS Hadoop Java SDK](../deployment/hadoop_java_sdk.md) 支持把监控指标上报到 [Pushgateway](https://github.com/prometheus/pushgateway) 或者 [Graphite](http://graphiteapp.org/)
 
-请用如下配置启用指标上报：
+启用指标上报到 Pushgateway ：
 
 ```xml
 <property>
@@ -154,7 +154,7 @@ spec:
 </property>
 ```
 
-同时可以通过 `juicefs.push-interval` 配置修改上报指标的频率，默认为 10 秒上报一次。JuiceFS Hadoop Java SDK 支持的所有配置参数请参考[文档](../deployment/hadoop_java_sdk.md#客户端配置参数)。
+同时可以通过 `juicefs.push-interval` 配置修改上报指标的频率，默认为 10 秒上报一次。
 
 :::info 说明
 根据 [Pushgateway 官方文档](https://github.com/prometheus/pushgateway/blob/master/README.md#configure-the-pushgateway-as-a-target-to-scrape)的建议，Prometheus 的[抓取配置](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config)中需要设置 `honor_labels: true`。
@@ -176,6 +176,19 @@ $ curl -X PUT http://host:9091/api/v1/admin/wipe
 
 有关 Pushgateway 的更多信息，请查看[官方文档](https://github.com/prometheus/pushgateway/blob/master/README.md)。
 
+启用指标上报到 Graphite ：
+
+```xml
+<property>
+  <name>juicefs.push-graphite</name>
+  <value>host:port</value>
+</property>
+```
+
+同时可以通过 `juicefs.push-interval` 配置修改上报指标的频率，默认为 10 秒上报一次。
+
+JuiceFS Hadoop Java SDK 支持的所有配置参数请参考[文档](../deployment/hadoop_java_sdk.md#客户端配置参数)。
+
 ### 使用 Consul 作为注册中心
 
 :::note 注意

diff --git a/docs/zh_cn/deployment/hadoop_java_sdk.md b/docs/zh_cn/deployment/hadoop_java_sdk.md
@@ -158,20 +158,21 @@ $ make win
 
 #### 其他配置
 
-| 配置项                    | 默认值  | 描述                                                         |
-| ------------------------- | ------- | ------------------------------------------------------------ |
-| `juicefs.bucket`          |         | 为对象存储指定跟格式化时不同的访问地址                             |
-| `juicefs.debug`           | `false` | 是否开启 debug 日志                                          |
-| `juicefs.access-log`      |         | 访问日志的路径。需要所有应用都有写权限，可以配置为 `/tmp/juicefs.access.log`。该文件会自动轮转，保留最近 7 个文件。 |
-| `juicefs.superuser`       | `hdfs`  | 超级用户                                                     |
-| `juicefs.users`           | `null`  | 用户名以及 UID 列表文件的地址，比如 `jfs://name/etc/users`。文件格式为 `<username>:<UID>`，一行一个用户。 |
-| `juicefs.groups`          | `null`  | 用户组、GID 以及组成员列表文件的地址，比如 `jfs://name/etc/groups`。文件格式为 `<group-name>:<GID>:<username1>,<username2>`，一行一个用户组。 |
-| `juicefs.umask`           | `null`  | 创建文件和目录的 umask 值（如 `0022`），如果没有此配置，默认值是 `fs.permissions.umask-mode`。 |
-| `juicefs.push-gateway`    |         | [Prometheus Pushgateway](https://github.com/prometheus/pushgateway) 地址，格式为 `<host>:<port>`。 |
-| `juicefs.push-interval`   | 10      | 推送数据到 Prometheus 的时间间隔，单位为秒。                 |
-| `juicefs.push-auth`       |         | [Prometheus 基本认证](https://prometheus.io/docs/guides/basic-auth)信息，格式为 `<username>:<password>`。 |
-| `juicefs.fast-resolve`    | `true`  | 是否开启快速元数据查找（通过 Redis Lua 脚本实现）            |
-| `juicefs.no-usage-report` | `false` | 是否上报数据。仅上版本号等使用量数据，不包含任何用户信息。   |
+| 配置项                         | 默认值  | 描述                                                                                                          |
+|-----------------------------| ------- |-------------------------------------------------------------------------------------------------------------|
+| `juicefs.bucket`            |         | 为对象存储指定跟格式化时不同的访问地址                                                                                         |
+| `juicefs.debug`             | `false` | 是否开启 debug 日志                                                                                               |
+| `juicefs.access-log`        |         | 访问日志的路径。需要所有应用都有写权限，可以配置为 `/tmp/juicefs.access.log`。该文件会自动轮转，保留最近 7 个文件。                                    |
+| `juicefs.superuser`         | `hdfs`  | 超级用户                                                                                                        |
+| `juicefs.users`             | `null`  | 用户名以及 UID 列表文件的地址，比如 `jfs://name/etc/users`。文件格式为 `<username>:<UID>`，一行一个用户。                                |
+| `juicefs.groups`            | `null`  | 用户组、GID 以及组成员列表文件的地址，比如 `jfs://name/etc/groups`。文件格式为 `<group-name>:<GID>:<username1>,<username2>`，一行一个用户组。 |
+| `juicefs.umask`             | `null`  | 创建文件和目录的 umask 值（如 `0022`），如果没有此配置，默认值是 `fs.permissions.umask-mode`。                                        |
+| `juicefs.push-gateway`      |         | [Prometheus Pushgateway](https://github.com/prometheus/pushgateway) 地址，格式为 `<host>:<port>`。                 |
+| `juicefs.push-auth`         |         | [Prometheus 基本认证](https://prometheus.io/docs/guides/basic-auth)信息，格式为 `<username>:<password>`。              |
+| `juicefs.push-graphite`         |         | [Graphite](http://graphiteapp.org/) 地址，格式为 `<host>:<port>`。                                                 |
+| `juicefs.push-interval`     | 10      | 指标推送的时间间隔，单位为秒。                                                                                             |
+| `juicefs.fast-resolve`      | `true`  | 是否开启快速元数据查找（通过 Redis Lua 脚本实现）                                                                              |
+| `juicefs.no-usage-report`   | `false` | 是否上报数据。仅上版本号等使用量数据，不包含任何用户信息。                                                                               |
 
 #### 多文件系统配置
 

diff --git a/go.mod b/go.mod
@@ -22,6 +22,7 @@ require (
 	github.com/go-redis/redis/v8 v8.4.0
 	github.com/go-sql-driver/mysql v1.6.0
 	github.com/gofrs/flock v0.8.1
+	github.com/golang/protobuf v1.4.3
 	github.com/google/btree v1.0.1
 	github.com/google/gops v0.3.13
 	github.com/google/uuid v1.1.2
@@ -47,6 +48,7 @@ require (
 	github.com/pkg/xattr v0.4.4
 	github.com/prometheus/client_golang v1.9.0
 	github.com/prometheus/client_model v0.2.0
+	github.com/prometheus/common v0.15.0
 	github.com/qingstor/qingstor-sdk-go/v4 v4.4.0
 	github.com/qiniu/api.v7/v7 v7.8.0
 	github.com/satori/go.uuid v1.2.0
@@ -120,7 +122,6 @@ require (
 	github.com/golang-jwt/jwt v3.2.2+incompatible // indirect
 	github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b // indirect
 	github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e // indirect
-	github.com/golang/protobuf v1.4.3 // indirect
 	github.com/golang/snappy v0.0.3 // indirect
 	github.com/gomodule/redigo v1.8.3 // indirect
 	github.com/google/flatbuffers v1.12.1 // indirect
@@ -193,7 +194,6 @@ require (
 	github.com/pingcap/failpoint v0.0.0-20210316064728-7acb0f0a3dfd // indirect
 	github.com/pingcap/kvproto v0.0.0-20211122024046-03abd340988f // indirect
 	github.com/pquerna/ffjson v0.0.0-20190930134022-aa0246cd15f7 // indirect
-	github.com/prometheus/common v0.15.0 // indirect
 	github.com/prometheus/procfs v0.2.0 // indirect
 	github.com/remyoudompheng/bigfft v0.0.0-20200410134404-eec4a21b6bb0 // indirect
 	github.com/rivo/uniseg v0.2.0 // indirect