Pin the NumPy version with v1.23.5 in some images #2070

tenzen-y · 2022-12-23T18:02:27Z

Signed-off-by: Yuki Iwai yuki.iwai.tz@gmail.com

What this PR does / why we need it:
I pinned the NumPy version with v1.23.5 in the mxnet-mnist image and the skopt suggestion image to avoid the following errors:

mxnet-mnist

I1223 17:32:59.235460      19 main.go:396] Trial Name: grid-ssmbrpvl
I1223 17:33:07.242688      19 main.go:139] /usr/local/lib/python3.9/site-packages/mxnet/numpy/utils.py:37: FutureWarning: In the future `np.bool` will be defined as the corresponding NumPy scalar.  (This may have returned Python scalars in past versions.
I1223 17:33:07.242711      19 main.go:139]   bool = onp.bool
I1223 17:33:07.244035      19 main.go:139] Traceback (most recent call last):
I1223 17:33:07.244094      19 main.go:139]   File "/opt/mxnet-mnist/mnist.py", line 24, in <module>
I1223 17:33:07.245554      19 main.go:139]     import mxnet as mx
I1223 17:33:07.245565      19 main.go:139]   File "/usr/local/lib/python3.9/site-packages/mxnet/__init__.py", line 33, in <module>
I1223 17:33:07.245949      19 main.go:139]     from . import contrib
I1223 17:33:07.245959      19 main.go:139]   File "/usr/local/lib/python3.9/site-packages/mxnet/contrib/__init__.py", line 30, in <module>
I1223 17:33:07.246245      19 main.go:139]     from . import text
I1223 17:33:07.246299      19 main.go:139]   File "/usr/local/lib/python3.9/site-packages/mxnet/contrib/text/__init__.py", line 23, in <module>
I1223 17:33:07.246526      19 main.go:139]     from . import embedding
I1223 17:33:07.246539      19 main.go:139]   File "/usr/local/lib/python3.9/site-packages/mxnet/contrib/text/embedding.py", line 36, in <module>
I1223 17:33:07.246899      19 main.go:139]     from ... import numpy as _mx_np
I1223 17:33:07.246914      19 main.go:139]   File "/usr/local/lib/python3.9/site-packages/mxnet/numpy/__init__.py", line 23, in <module>
I1223 17:33:07.247114      19 main.go:139]     from .multiarray import *  # pylint: disable=wildcard-import
I1223 17:33:07.247122      19 main.go:139]   File "/usr/local/lib/python3.9/site-packages/mxnet/numpy/multiarray.py", line 47, in <module>
I1223 17:33:07.247384      19 main.go:139]     from .utils import _get_np_op
I1223 17:33:07.247435      19 main.go:139]   File "/usr/local/lib/python3.9/site-packages/mxnet/numpy/utils.py", line 37, in <module>
I1223 17:33:07.247715      19 main.go:139]     bool = onp.bool
I1223 17:33:07.247731      19 main.go:139]   File "/usr/local/lib/python3.9/site-packages/numpy/__init__.py", line 284, in __getattr__
I1223 17:33:07.248506      19 main.go:139]     raise AttributeError("module {!r} has no attribute "
I1223 17:33:07.248685      19 main.go:139] AttributeError: module 'numpy' has no attribute 'bool'

skopt suggestion service

ERROR:grpc._server:Exception calling application: module 'numpy' has no attribute 'int'
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/grpc/_server.py", line 443, in _call_behavior
    response_or_iterator = behavior(argument, context)
  File "/opt/katib/pkg/suggestion/v1beta1/skopt/service.py", line 55, in GetSuggestions
    new_trials = self.base_service.getSuggestions(trials, request.current_request_number)
  File "/opt/katib/pkg/suggestion/v1beta1/skopt/base_service.py", line 121, in getSuggestions
    skopt_suggested = self.skopt_optimizer.ask(n_points=current_request_number)
  File "/usr/local/lib/python3.9/site-packages/skopt/optimizer/optimizer.py", line 395, in ask
    x = opt.ask()
  File "/usr/local/lib/python3.9/site-packages/skopt/optimizer/optimizer.py", line 367, in ask
    return self._ask()
  File "/usr/local/lib/python3.9/site-packages/skopt/optimizer/optimizer.py", line 434, in _ask
    return self.space.rvs(random_state=self.rng)[0]
  File "/usr/local/lib/python3.9/site-packages/skopt/space/space.py", line 900, in rvs
    columns.append(dim.rvs(n_samples=n_samples, random_state=rng))
  File "/usr/local/lib/python3.9/site-packages/skopt/space/space.py", line 158, in rvs
    return self.inverse_transform(samples)
  File "/usr/local/lib/python3.9/site-packages/skopt/space/space.py", line 528, in inverse_transform
    inv_transform = super(Integer, self).inverse_transform(Xt)
  File "/usr/local/lib/python3.9/site-packages/skopt/space/space.py", line 168, in inverse_transform
    return self.transformer.inverse_transform(Xt)
  File "/usr/local/lib/python3.9/site-packages/skopt/space/transformers.py", line 309, in inverse_transform
    X = transformer.inverse_transform(X)
  File "/usr/local/lib/python3.9/site-packages/skopt/space/transformers.py", line 275, in inverse_transform
    return np.round(X_orig).astype(np.int)
  File "/usr/local/lib/python3.9/site-packages/numpy/__init__.py", line 284, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'int'

/assign @andreyvelich @johnugeorge @gaocegege

Blocking: #2039 #2060 #2067

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Checklist:

Docs included if any changes are user facing

google-oss-prow · 2022-12-23T18:02:31Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

johnugeorge · 2022-12-23T18:31:12Z

Thanks @tenzen-y for quick fix.

/lgtm

johnugeorge · 2022-12-23T18:32:30Z

We need to provide a clean fix in future when available

tenzen-y · 2022-12-23T18:35:20Z

We need to provide a clean fix in future when available

Yes, I agree. We might be able to use the latest NumPy version once mxnet v2.0 or v1.9.2 is released.

tenzen-y · 2022-12-23T18:37:18Z

Since the errors are caused by mxnet codes, not by our training codes.

andreyvelich · 2022-12-23T18:45:44Z

Thank you for the prompt fix @tenzen-y !
/lgtm

tenzen-y · 2022-12-23T19:52:17Z

It seems that E2E for bayesian-optimazation only failed...

2022/12/23 19:13:55 Deleting Experiment bayesian-optimization
2022/12/23 19:13:55 Wait Experiment finish failed: Experiment run timed out
NAME READY STATUS RESTARTS AGE
bayesian-optimization-bayesianoptimization-58d76b556b-pzzl5 1/1 Terminating 0 50m
katib-cert-generator-mx24r 0/1 Completed 0 53m
katib-controller-5c98d47df-vggdd 1/1 Running 0 53m
katib-db-manager-6d8679bd8c-d44hr 1/1 Running 0 53m
katib-mysql-6975d6c6c4-9bf6m 1/1 Running 0 53m

https://github.com/kubeflow/katib/actions/runs/3767418520/jobs/6404930467#step:4:12390

I could not reproduce it on my local :(

tenzen-y · 2022-12-23T20:58:18Z

It looks like CI failure is caused by the skopt suggestion service image.

{"level":"error","ts":1671828340.3335319,"msg":"Reconciler error","controller":"suggestion-controller","object":{"name":"bayesian-optimization","namespace":"kubeflow"},"namespace":"kubeflow","name":"bayesian-optimization","reconcileID":"57312598-0447-4fba-95af-cb5d56b22f31","error":"rpc error: code = Unknown desc = Exception calling application: module 'numpy' has no attribute 'int'","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234"}

https://github.com/tenzen-y/katib/actions/runs/3768079424/jobs/6406216537#step:4:10644

tenzen-y · 2022-12-23T21:11:33Z

ERROR:grpc._server:Exception calling application: module 'numpy' has no attribute 'int'
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/grpc/_server.py", line 443, in _call_behavior
    response_or_iterator = behavior(argument, context)
  File "/opt/katib/pkg/suggestion/v1beta1/skopt/service.py", line 55, in GetSuggestions
    new_trials = self.base_service.getSuggestions(trials, request.current_request_number)
  File "/opt/katib/pkg/suggestion/v1beta1/skopt/base_service.py", line 121, in getSuggestions
    skopt_suggested = self.skopt_optimizer.ask(n_points=current_request_number)
  File "/usr/local/lib/python3.9/site-packages/skopt/optimizer/optimizer.py", line 395, in ask
    x = opt.ask()
  File "/usr/local/lib/python3.9/site-packages/skopt/optimizer/optimizer.py", line 367, in ask
    return self._ask()
  File "/usr/local/lib/python3.9/site-packages/skopt/optimizer/optimizer.py", line 434, in _ask
    return self.space.rvs(random_state=self.rng)[0]
  File "/usr/local/lib/python3.9/site-packages/skopt/space/space.py", line 900, in rvs
    columns.append(dim.rvs(n_samples=n_samples, random_state=rng))
  File "/usr/local/lib/python3.9/site-packages/skopt/space/space.py", line 158, in rvs
    return self.inverse_transform(samples)
  File "/usr/local/lib/python3.9/site-packages/skopt/space/space.py", line 528, in inverse_transform
    inv_transform = super(Integer, self).inverse_transform(Xt)
  File "/usr/local/lib/python3.9/site-packages/skopt/space/space.py", line 168, in inverse_transform
    return self.transformer.inverse_transform(Xt)
  File "/usr/local/lib/python3.9/site-packages/skopt/space/transformers.py", line 309, in inverse_transform
    X = transformer.inverse_transform(X)
  File "/usr/local/lib/python3.9/site-packages/skopt/space/transformers.py", line 275, in inverse_transform
    return np.round(X_orig).astype(np.int)
  File "/usr/local/lib/python3.9/site-packages/numpy/__init__.py", line 284, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'int'

tenzen-y · 2022-12-23T21:24:00Z

This skopt error is reported upstream, but the PR has not been merged...

scikit-optimize/scikit-optimize#1123

…uggestion service image Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

johnugeorge · 2022-12-24T03:15:03Z

/lgtm

google-oss-prow bot assigned andreyvelich, gaocegege and johnugeorge Dec 23, 2022

google-oss-prow bot requested review from andreyvelich, anencore94 and johnugeorge December 23, 2022 18:02

google-oss-prow bot added approved size/XS labels Dec 23, 2022

tenzen-y changed the title ~~Pin the numpy version with v1.23.5 in the mxnet-mnist image~~ Pin the NumPy version with v1.23.5 in the mxnet-mnist image Dec 23, 2022

This was referenced Dec 23, 2022

Backend for getting logs of a trial #2039

Merged

kwa(front): Sort conditions table by timestamp #2067

Closed

Support for grid search algorithm in Optuna Suggestion Service #2060

Merged

google-oss-prow bot added the lgtm label Dec 23, 2022

tenzen-y force-pushed the fix-mxnet-mnist-image branch from 4b364ad to 0a9812e Compare December 23, 2022 21:17

google-oss-prow bot added size/S and removed lgtm size/XS labels Dec 23, 2022

tenzen-y changed the title ~~Pin the NumPy version with v1.23.5 in the mxnet-mnist image~~ Pin the NumPy version with v1.23.5 in the some images Dec 23, 2022

tenzen-y changed the title ~~Pin the NumPy version with v1.23.5 in the some images~~ Pin the NumPy version with v1.23.5 in some images Dec 23, 2022

tenzen-y force-pushed the fix-mxnet-mnist-image branch from 0a9812e to 3c696f8 Compare December 23, 2022 21:32

Pin the NumPy version with v1.23.5 in the mxnet-mnist and the skopt s…

789bfc2

…uggestion service image Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

tenzen-y force-pushed the fix-mxnet-mnist-image branch from 3c696f8 to 789bfc2 Compare December 23, 2022 21:32

google-oss-prow bot added the lgtm label Dec 24, 2022

google-oss-prow bot merged commit 1dd7251 into kubeflow:master Dec 24, 2022

tenzen-y deleted the fix-mxnet-mnist-image branch December 24, 2022 04:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pin the NumPy version with v1.23.5 in some images #2070

Pin the NumPy version with v1.23.5 in some images #2070

tenzen-y commented Dec 23, 2022 •

edited

google-oss-prow bot commented Dec 23, 2022

johnugeorge commented Dec 23, 2022

johnugeorge commented Dec 23, 2022

tenzen-y commented Dec 23, 2022

tenzen-y commented Dec 23, 2022

andreyvelich commented Dec 23, 2022

tenzen-y commented Dec 23, 2022

tenzen-y commented Dec 23, 2022

tenzen-y commented Dec 23, 2022

tenzen-y commented Dec 23, 2022

johnugeorge commented Dec 24, 2022

Pin the NumPy version with v1.23.5 in some images #2070

Pin the NumPy version with v1.23.5 in some images #2070

Conversation

tenzen-y commented Dec 23, 2022 • edited

google-oss-prow bot commented Dec 23, 2022

johnugeorge commented Dec 23, 2022

johnugeorge commented Dec 23, 2022

tenzen-y commented Dec 23, 2022

tenzen-y commented Dec 23, 2022

andreyvelich commented Dec 23, 2022

tenzen-y commented Dec 23, 2022

tenzen-y commented Dec 23, 2022

tenzen-y commented Dec 23, 2022

tenzen-y commented Dec 23, 2022

johnugeorge commented Dec 24, 2022

tenzen-y commented Dec 23, 2022 •

edited