Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pyspark] Add tracker_on_driver to decide where the tracker will be launched #10281

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

wbo4958
Copy link
Contributor

@wbo4958 wbo4958 commented May 13, 2024

df_train = spark.createDataFrame(
    [
        (Vectors.dense(1.0, 2.0, 3.0), 0, False, 1.0),
        (Vectors.sparse(3, {1: 1.0, 2: 5.5}), 1, False, 2.0),
        (Vectors.dense(4.0, 5.0, 6.0), 0, True, 1.0),
        (Vectors.sparse(3, {1: 6.0, 2: 7.5}), 1, True, 2.0),
    ]
    * 100,
    ["features", "label", "isVal", "weight"],
)

from xgboost.spark import SparkXGBRegressor

callbacks = EvaluationMonitor()
xgb_regressor = SparkXGBRegressor(
    num_workers=5,
    callbacks=[callbacks],
    tracker_on_driver=True,
    validation_indicator_col="isVal",
)
xgb_reg_model = xgb_regressor.fit(df_train)

With the above test code, The below log will be printed on the driver. Or else, they will be printed on the executor side.

[0]	training-rmse:0.35149	validation-rmse:0.35149
[0]	training-rmse:0.35149	validation-rmse:0.35149
[1]	training-rmse:0.24708	validation-rmse:0.24708
[1]	training-rmse:0.24708	validation-rmse:0.24708
[2]	training-rmse:0.17369	validation-rmse:0.17369
[2]	training-rmse:0.17369	validation-rmse:0.17369
[3]	training-rmse:0.12210	validation-rmse:0.12210
[3]	training-rmse:0.12210	validation-rmse:0.12210
[4]	training-rmse:0.08583	validation-rmse:0.08583
[4]	training-rmse:0.08583	validation-rmse:0.08583
[5]	training-rmse:0.06034	validation-rmse:0.06034
[5]	training-rmse:0.06034	validation-rmse:0.06034
[6]	training-rmse:0.04242	validation-rmse:0.04242
...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant