Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sdk support push to graphite #1586

Merged
merged 5 commits into from
Mar 18, 2022
Merged

sdk support push to graphite #1586

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
de38a2d
support push to graphite
zhijian-pro Mar 15, 2022
747cd04
Merge branch 'main' into sdk_support_graphite
zhijian-pro Mar 17, 2022
0afebe1
go mod tidy
zhijian-pro Mar 17, 2022
5105f98
refactor
zhijian-pro Mar 17, 2022
b304099
refactor
zhijian-pro Mar 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
19 changes: 16 additions & 3 deletions docs/en/administration/monitoring.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,9 +143,9 @@ For more information about Prometheus Operator, please check [official document]

### Hadoop

The [JuiceFS Hadoop Java SDK](../deployment/hadoop_java_sdk.md) supports reporting monitoring metrics to [Pushgateway](https://github.com/prometheus/pushgateway) and then letting Prometheus scrape the metrics from Pushgateway.
The [JuiceFS Hadoop Java SDK](../deployment/hadoop_java_sdk.md) supports reporting monitoring metrics to [Pushgateway](https://github.com/prometheus/pushgateway) and [Graphite](http://graphiteapp.org/)

Please enable metrics reporting with the following configuration:
Report metrics to Pushgateway:

```xml
<property>
Expand All @@ -154,7 +154,7 @@ Please enable metrics reporting with the following configuration:
</property>
```

At the same time, the frequency of reporting metrics can be modified through the `juicefs.push-interval` configuration. The default is to report once every 10 seconds. For all configurations supported by JuiceFS Hadoop Java SDK, please refer to [documentation](../deployment/hadoop_java_sdk.md#client-configurations).
At the same time, the frequency of reporting metrics can be modified through the `juicefs.push-interval` configuration. The default is to report once every 10 seconds.

:::info
According to the suggestion of [Pushgateway official document](https://github.com/prometheus/pushgateway/blob/master/README.md#configure-the-pushgateway-as-a-target-to-scrape), Prometheus's [scrape configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) needs to set `honor_labels: true`.
Expand All @@ -176,6 +176,19 @@ $ curl -X PUT http://host:9091/api/v1/admin/wipe

For more information about Pushgateway, please check [official document](https://github.com/prometheus/pushgateway/blob/master/README.md).

Report metrics to Graphite:

```xml
<property>
<name>juicefs.push-graphite</name>
<value>host:port</value>
</property>
```

At the same time, the frequency of reporting metrics can be modified through the `juicefs.push-interval` configuration. The default is to report once every 10 seconds.

For all configurations supported by JuiceFS Hadoop Java SDK, please refer to [documentation](../deployment/hadoop_java_sdk.md#client-configurations).

### Use Consul as registration center

:::note
Expand Down
27 changes: 14 additions & 13 deletions docs/en/deployment/hadoop_java_sdk.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,20 +158,21 @@ Please refer to the following table to set the relevant parameters of the JuiceF

#### Other Configurations

| Configuration | Default Value | Description |
| ------------------------- | ------------- | ------------------------------------------------------------ |
| `juicefs.bucket` | | Specify a different endpoint for object storage |
| `juicefs.debug` | `false` | Whether enable debug log |
| `juicefs.access-log` | | Access log path. Ensure Hadoop application has write permission, e.g. `/tmp/juicefs.access.log`. The log file will rotate automatically to keep at most 7 files. |
| `juicefs.superuser` | `hdfs` | The super user |
| `juicefs.users` | `null` | The path of username and UID list file, e.g. `jfs://name/etc/users`. The file format is `<username>:<UID>`, one user per line. |
| Configuration | Default Value | Description |
|---------------------------| ------------- |-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `juicefs.bucket` | | Specify a different endpoint for object storage |
| `juicefs.debug` | `false` | Whether enable debug log |
| `juicefs.access-log` | | Access log path. Ensure Hadoop application has write permission, e.g. `/tmp/juicefs.access.log`. The log file will rotate automatically to keep at most 7 files. |
| `juicefs.superuser` | `hdfs` | The super user |
| `juicefs.users` | `null` | The path of username and UID list file, e.g. `jfs://name/etc/users`. The file format is `<username>:<UID>`, one user per line. |
| `juicefs.groups` | `null` | The path of group name, GID and group members list file, e.g. `jfs://name/etc/groups`. The file format is `<group-name>:<GID>:<username1>,<username2>`, one group per line. |
| `juicefs.umask` | `null` | The umask used when creating files and directories (e.g. `0022`), default value is `fs.permissions.umask-mode`. |
| `juicefs.push-gateway` | | [Prometheus Pushgateway](https://github.com/prometheus/pushgateway) address, format is `<host>:<port>`. |
| `juicefs.push-interval` | 10 | Prometheus push interval in seconds |
| `juicefs.push-auth` | | [Prometheus basic auth](https://prometheus.io/docs/guides/basic-auth) information, format is `<username>:<password>`. |
| `juicefs.fast-resolve` | `true` | Whether enable faster metadata lookup using Redis Lua script |
| `juicefs.no-usage-report` | `false` | Whether disable usage reporting. JuiceFS only collects anonymous usage data (e.g. version number), no user or any sensitive data will be collected. |
| `juicefs.umask` | `null` | The umask used when creating files and directories (e.g. `0022`), default value is `fs.permissions.umask-mode`. |
| `juicefs.push-gateway` | | [Prometheus Pushgateway](https://github.com/prometheus/pushgateway) address, format is `<host>:<port>`. |
| `juicefs.push-auth` | | [Prometheus basic auth](https://prometheus.io/docs/guides/basic-auth) information, format is `<username>:<password>`. |
| `juicefs.push-graphite` | | [Graphite](http://graphiteapp.org/) address, format is `<host>:<port>`. |
| `juicefs.push-interval` | 10 | Metric push interval (in seconds) |
| `juicefs.fast-resolve` | `true` | Whether enable faster metadata lookup using Redis Lua script |
| `juicefs.no-usage-report` | `false` | Whether disable usage reporting. JuiceFS only collects anonymous usage data (e.g. version number), no user or any sensitive data will be collected. |

#### Multiple file systems configuration

Expand Down
19 changes: 16 additions & 3 deletions docs/zh_cn/administration/monitoring.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,9 +143,9 @@ spec:

### Hadoop

[JuiceFS Hadoop Java SDK](../deployment/hadoop_java_sdk.md) 支持把监控指标上报到 [Pushgateway](https://github.com/prometheus/pushgateway)，然后让 Prometheus 从 Pushgateway 抓取指标。
[JuiceFS Hadoop Java SDK](../deployment/hadoop_java_sdk.md) 支持把监控指标上报到 [Pushgateway](https://github.com/prometheus/pushgateway) 或者 [Graphite](http://graphiteapp.org/)

请用如下配置启用指标上报：
启用指标上报到 Pushgateway ：

```xml
<property>
Expand All @@ -154,7 +154,7 @@ spec:
</property>
```

同时可以通过 `juicefs.push-interval` 配置修改上报指标的频率，默认为 10 秒上报一次。JuiceFS Hadoop Java SDK 支持的所有配置参数请参考[文档](../deployment/hadoop_java_sdk.md#客户端配置参数)。
同时可以通过 `juicefs.push-interval` 配置修改上报指标的频率，默认为 10 秒上报一次。

:::info 说明
根据 [Pushgateway 官方文档](https://github.com/prometheus/pushgateway/blob/master/README.md#configure-the-pushgateway-as-a-target-to-scrape)的建议，Prometheus 的[抓取配置](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config)中需要设置 `honor_labels: true`。
Expand All @@ -176,6 +176,19 @@ $ curl -X PUT http://host:9091/api/v1/admin/wipe

有关 Pushgateway 的更多信息，请查看[官方文档](https://github.com/prometheus/pushgateway/blob/master/README.md)。

启用指标上报到 Graphite ：

```xml
<property>
<name>juicefs.push-graphite</name>
<value>host:port</value>
</property>
```

同时可以通过 `juicefs.push-interval` 配置修改上报指标的频率，默认为 10 秒上报一次。

JuiceFS Hadoop Java SDK 支持的所有配置参数请参考[文档](../deployment/hadoop_java_sdk.md#客户端配置参数)。

### 使用 Consul 作为注册中心

:::note 注意
Expand Down
29 changes: 15 additions & 14 deletions docs/zh_cn/deployment/hadoop_java_sdk.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,20 +158,21 @@ $ make win

#### 其他配置

| 配置项 | 默认值 | 描述 |
| ------------------------- | ------- | ------------------------------------------------------------ |
| `juicefs.bucket` | | 为对象存储指定跟格式化时不同的访问地址 |
| `juicefs.debug` | `false` | 是否开启 debug 日志 |
| `juicefs.access-log` | | 访问日志的路径。需要所有应用都有写权限，可以配置为 `/tmp/juicefs.access.log`。该文件会自动轮转，保留最近 7 个文件。 |
| `juicefs.superuser` | `hdfs` | 超级用户 |
| `juicefs.users` | `null` | 用户名以及 UID 列表文件的地址，比如 `jfs://name/etc/users`。文件格式为 `<username>:<UID>`，一行一个用户。 |
| `juicefs.groups` | `null` | 用户组、GID 以及组成员列表文件的地址，比如 `jfs://name/etc/groups`。文件格式为 `<group-name>:<GID>:<username1>,<username2>`，一行一个用户组。 |
| `juicefs.umask` | `null` | 创建文件和目录的 umask 值（如 `0022`），如果没有此配置，默认值是 `fs.permissions.umask-mode`。 |
| `juicefs.push-gateway` | | [Prometheus Pushgateway](https://github.com/prometheus/pushgateway) 地址，格式为 `<host>:<port>`。 |
| `juicefs.push-interval` | 10 | 推送数据到 Prometheus 的时间间隔，单位为秒。 |
| `juicefs.push-auth` | | [Prometheus 基本认证](https://prometheus.io/docs/guides/basic-auth)信息，格式为 `<username>:<password>`。 |
| `juicefs.fast-resolve` | `true` | 是否开启快速元数据查找（通过 Redis Lua 脚本实现） |
| `juicefs.no-usage-report` | `false` | 是否上报数据。仅上版本号等使用量数据，不包含任何用户信息。 |
| 配置项 | 默认值 | 描述 |
|-----------------------------| ------- |-------------------------------------------------------------------------------------------------------------|
| `juicefs.bucket` | | 为对象存储指定跟格式化时不同的访问地址 |
| `juicefs.debug` | `false` | 是否开启 debug 日志 |
| `juicefs.access-log` | | 访问日志的路径。需要所有应用都有写权限，可以配置为 `/tmp/juicefs.access.log`。该文件会自动轮转，保留最近 7 个文件。 |
| `juicefs.superuser` | `hdfs` | 超级用户 |
| `juicefs.users` | `null` | 用户名以及 UID 列表文件的地址，比如 `jfs://name/etc/users`。文件格式为 `<username>:<UID>`，一行一个用户。 |
| `juicefs.groups` | `null` | 用户组、GID 以及组成员列表文件的地址，比如 `jfs://name/etc/groups`。文件格式为 `<group-name>:<GID>:<username1>,<username2>`，一行一个用户组。 |
| `juicefs.umask` | `null` | 创建文件和目录的 umask 值（如 `0022`），如果没有此配置，默认值是 `fs.permissions.umask-mode`。 |
| `juicefs.push-gateway` | | [Prometheus Pushgateway](https://github.com/prometheus/pushgateway) 地址，格式为 `<host>:<port>`。 |
| `juicefs.push-auth` | | [Prometheus 基本认证](https://prometheus.io/docs/guides/basic-auth)信息，格式为 `<username>:<password>`。 |
| `juicefs.push-graphite` | | [Graphite](http://graphiteapp.org/) 地址，格式为 `<host>:<port>`。 |
| `juicefs.push-interval` | 10 | 指标推送的时间间隔，单位为秒。 |
| `juicefs.fast-resolve` | `true` | 是否开启快速元数据查找（通过 Redis Lua 脚本实现） |
| `juicefs.no-usage-report` | `false` | 是否上报数据。仅上版本号等使用量数据，不包含任何用户信息。 |

#### 多文件系统配置

Expand Down
4 changes: 2 additions & 2 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ require (
github.com/go-redis/redis/v8 v8.4.0
github.com/go-sql-driver/mysql v1.6.0
github.com/gofrs/flock v0.8.1
github.com/golang/protobuf v1.4.3
github.com/google/btree v1.0.1
github.com/google/gops v0.3.13
github.com/google/uuid v1.1.2
Expand All @@ -47,6 +48,7 @@ require (
github.com/pkg/xattr v0.4.4
github.com/prometheus/client_golang v1.9.0
github.com/prometheus/client_model v0.2.0
github.com/prometheus/common v0.15.0
github.com/qingstor/qingstor-sdk-go/v4 v4.4.0
github.com/qiniu/api.v7/v7 v7.8.0
github.com/satori/go.uuid v1.2.0
Expand Down Expand Up @@ -120,7 +122,6 @@ require (
github.com/golang-jwt/jwt v3.2.2+incompatible // indirect
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b // indirect
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e // indirect
github.com/golang/protobuf v1.4.3 // indirect
github.com/golang/snappy v0.0.3 // indirect
github.com/gomodule/redigo v1.8.3 // indirect
github.com/google/flatbuffers v1.12.1 // indirect
Expand Down Expand Up @@ -193,7 +194,6 @@ require (
github.com/pingcap/failpoint v0.0.0-20210316064728-7acb0f0a3dfd // indirect
github.com/pingcap/kvproto v0.0.0-20211122024046-03abd340988f // indirect
github.com/pquerna/ffjson v0.0.0-20190930134022-aa0246cd15f7 // indirect
github.com/prometheus/common v0.15.0 // indirect
github.com/prometheus/procfs v0.2.0 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20200410134404-eec4a21b6bb0 // indirect
github.com/rivo/uniseg v0.2.0 // indirect
Expand Down