SUPP: Add support to Python 3.8 #2066

xmnlab · 2020-02-01T01:23:49Z

Resolves #2036

this PR:

adds support to Python 3.8 without pyspark and pymapd*
adds a udf pytest mark for bigquery.udf tests
improves ci/azure/linux variables
skip a doctest check on ibis.bigquery.udf for Python 3.8

* package conflicts:

it needs pyspark v3 [SPARK-29536][PYTHON] Upgrade cloudpickle to 1.1.1 to support Python 3.8 apache/spark#26194 (comment)
it needs omniscidb/pymapd with pyarrow=0.15 support

ci/azure/linux.yml

jreback · 2020-02-02T19:16:06Z

ci/requirements-3.8-dev.yml

+  - lz4
+  - multipledispatch>=0.6.0
+  - mypy
+  - numpy>=1.11


you should bump numpy / pandas to the min for 3.8, I think 1.15 and 0.25.3; they don't really matter as that's what conda will install anyhow, but better to reflect in this file. also pyarrow >=0.15

thanks @jreback for the review. I will work on these changes.

I pinned pyarrow>=0.13 because pymapd has some restrictions pyarrow >=0.12,<0.14

jreback · 2020-02-11T00:26:30Z

@xmnlab can you merge master and we see if we can get this in

xmnlab · 2020-02-11T19:59:21Z

I need to move back to the previous pre-commit isort hook .. because it was conflicting with black.

xmnlab · 2020-02-18T20:35:57Z

it seems pyspark 2.* doesn't work properly with python3.8 ... it should work when pyspark 3 is released (apache/spark#26194 (comment))

Traceback (most recent call last):
  File "/ibis/ci/impalamgr.py", line 14, in <module>
    import ibis
  File "/ibis/ibis/__init__.py", line 60, in <module>
    import ibis.spark.api as spark  # noqa: F401
  File "/ibis/ibis/spark/api.py", line 2, in <module>
    from ibis.spark.client import SparkClient
  File "/ibis/ibis/spark/client.py", line 2, in <module>
    import pyspark as ps
  File "/opt/conda/envs/ibis-env/lib/python3.8/site-packages/pyspark/__init__.py", line 51, in <module>
    from pyspark.context import SparkContext
  File "/opt/conda/envs/ibis-env/lib/python3.8/site-packages/pyspark/context.py", line 31, in <module>
    from pyspark import accumulators
  File "/opt/conda/envs/ibis-env/lib/python3.8/site-packages/pyspark/accumulators.py", line 97, in <module>
    from pyspark.serializers import read_int, PickleSerializer
  File "/opt/conda/envs/ibis-env/lib/python3.8/site-packages/pyspark/serializers.py", line 71, in <module>
    from pyspark import cloudpickle
  File "/opt/conda/envs/ibis-env/lib/python3.8/site-packages/pyspark/cloudpickle.py", line 145, in <module>
    _cell_set_template_code = _make_cell_set_template_code()
  File "/opt/conda/envs/ibis-env/lib/python3.8/site-packages/pyspark/cloudpickle.py", line 126, in _make_cell_set_template_code
    return types.CodeType(
TypeError: an integer is required (got type bytes)
Makefile:101: recipe for target 'load' failed
make: *** [load] Error 1

xmnlab · 2020-02-21T01:00:31Z

for some reason a doctest in ibis.bigquery.udf.api is failing. for now I am just skipping doctest for python3.8

============================================= FAILURES =============================================
_______________________________ [doctest] ibis.bigquery.udf.api.udf ________________________________
070     `the BigQuery documentation
071     <https://cloud.google.com/bigquery/docs/reference/standard-sql/user-defined-functions#sql-type-encodings-in-javascript>`_.
072 
073     Examples
074     --------
075     >>> from ibis.bigquery import udf
076     >>> import ibis.expr.datatypes as dt
077     >>> @udf(input_type=[dt.double], output_type=dt.double)
UNEXPECTED EXCEPTION: NotImplementedError("'visit_Constant' nodes not yet implemented")
Traceback (most recent call last):

  File "/home/xmn/miniconda3/envs/ibis-py38/lib/python3.8/doctest.py", line 1329, in __run
    exec(compile(example.source, filename, "single",

  File "<doctest ibis.bigquery.udf.api.udf[3]>", line 2, in <module>

  File "/home/xmn/dev/quansight/ibis-project/ibis/ibis/bigquery/udf/api.py", line 215, in wrapper
    source = PythonToJavaScriptTranslator(f).compile()

  File "/home/xmn/dev/quansight/ibis-project/ibis/ibis/bigquery/udf/core.py", line 133, in compile
    return self.visit(self.ast)

  File "/home/xmn/dev/quansight/ibis-project/ibis/ibis/bigquery/udf/core.py", line 146, in visit
    result = method(node)

  File "/home/xmn/dev/quansight/ibis-project/ibis/ibis/bigquery/udf/core.py", line 413, in visit_Module
    return '\n\n'.join(map(self.visit, node.body))

  File "/home/xmn/dev/quansight/ibis-project/ibis/ibis/bigquery/udf/core.py", line 146, in visit
    result = method(node)

  File "/home/xmn/dev/quansight/ibis-project/ibis/ibis/bigquery/udf/core.py", line 211, in visit_FunctionDef
    body = indent(map(self.visit, node.body))

  File "/home/xmn/dev/quansight/ibis-project/ibis/ibis/bigquery/udf/core.py", line 49, in indent
    text = '\n'.join(lines)

  File "/home/xmn/dev/quansight/ibis-project/ibis/ibis/bigquery/udf/core.py", line 146, in visit
    result = method(node)

  File "/home/xmn/dev/quansight/ibis-project/ibis/ibis/bigquery/udf/core.py", line 63, in wrapper
    return f(*args, **kwargs) + ';'

  File "/home/xmn/dev/quansight/ibis-project/ibis/ibis/bigquery/udf/core.py", line 232, in visit_Return
    return 'return {}'.format(self.visit(node.value))

  File "/home/xmn/dev/quansight/ibis-project/ibis/ibis/bigquery/udf/core.py", line 146, in visit
    result = method(node)

  File "/home/xmn/dev/quansight/ibis-project/ibis/ibis/bigquery/udf/core.py", line 273, in visit_BinOp
    self.visit(left), self.visit(op), self.visit(right)

  File "/home/xmn/dev/quansight/ibis-project/ibis/ibis/bigquery/udf/core.py", line 141, in visit
    raise NotImplementedError(

NotImplementedError: 'visit_Constant' nodes not yet implemented

jreback · 2020-02-21T01:09:30Z

@xmnlab pls open an issue for the skip

xmnlab · 2020-02-21T01:20:53Z

thanks @jreback! issue created: https://github.com/ibis-project/ibis/issues/2085

xmnlab · 2020-02-21T01:48:38Z

it is done for review! thanks!

jreback

ok passing is good, some style items. ping when green.

ci/azure/linux.yml

jreback · 2020-02-23T14:40:01Z

ibis/bigquery/udf/api.py

@@ -12,6 +13,13 @@

 __all__ = ('udf',)

+# NOTE: used for skip doctest for Python 3.8


do we have an issue for this?

define these constants in ibis/compat.py instead

sounds good, thanks!

jreback · 2020-02-23T14:40:18Z

ibis/bigquery/udf/api.py

@@ -64,6 +72,8 @@ def udf(input_type, output_type, strict=True, libraries=None):

    Examples
    --------
+    >>> if PY38:
+    ...     import pytest; pytest.skip("")


put the issue number here, use pytest.xfail instead

sounds good. thanks!

jreback · 2020-02-23T14:40:48Z

ibis/bigquery/udf/tests/test_core.py

@@ -6,7 +6,7 @@

 from ibis.bigquery.udf.core import PythonToJavaScriptTranslator, SymbolTable

-pytestmark = pytest.mark.bigquery
+pytestmark = [pytest.mark.bigquery, pytest.mark.udf]


can you put a TODO and issue ref's for why skipping these tests

sounds good. I will work on that.

jreback · 2020-02-23T14:40:55Z

ibis/bigquery/udf/tests/test_find.py

@@ -5,7 +5,7 @@
 from ibis.bigquery.udf.find import find_names
 from ibis.util import is_iterable

-pytestmark = pytest.mark.bigquery
+pytestmark = [pytest.mark.bigquery, pytest.mark.udf]


jreback · 2020-02-23T14:41:00Z

ibis/bigquery/udf/tests/test_udf_execute.py

@@ -11,7 +11,7 @@

 pytest.importorskip('google.cloud.bigquery')

-pytestmark = pytest.mark.bigquery
+pytestmark = [pytest.mark.bigquery, pytest.mark.udf]


ibis/expr/datatypes.py

jreback · 2020-02-23T14:41:30Z

pyproject.toml

@@ -1,5 +1,5 @@
 [tool.black]
 line-length = 79
 skip-string-normalization = true
-target-version = ["py35", "py36", "py37"]
+target-version = ["py36", "py37", "py38"]


do we need a change in setup.py?

setup.py already has python_requires='>=3.6'. do you have another property in mind that should be changed?

classifiers=[ 'Development Status :: 4 - Beta', 'Operating System :: OS Independent', 'Intended Audience :: Science/Research', 'Programming Language :: Python', 'Programming Language :: Python :: 3', 'Topic :: Scientific/Engineering',

we usually list explicity versions here. but can be a followup (as i guess wasn't before)

xmnlab

@jreback thanks for the review! I am working on that.

ci/azure/linux.yml

xmnlab · 2020-02-24T01:52:22Z

ibis/bigquery/udf/api.py

@@ -12,6 +13,13 @@

 __all__ = ('udf',)

+# NOTE: used for skip doctest for Python 3.8


sounds good, thanks!

xmnlab · 2020-02-24T01:55:51Z

ibis/bigquery/udf/api.py

@@ -64,6 +72,8 @@ def udf(input_type, output_type, strict=True, libraries=None):

    Examples
    --------
+    >>> if PY38:
+    ...     import pytest; pytest.skip("")


sounds good. thanks!

ibis/expr/datatypes.py

xmnlab · 2020-02-24T02:13:20Z

pyproject.toml

@@ -1,5 +1,5 @@
 [tool.black]
 line-length = 79
 skip-string-normalization = true
-target-version = ["py35", "py36", "py37"]
+target-version = ["py36", "py37", "py38"]


setup.py already has python_requires='>=3.6'. do you have another property in mind that should be changed?

xmnlab · 2020-02-24T02:13:45Z

ibis/bigquery/udf/tests/test_core.py

@@ -6,7 +6,7 @@

 from ibis.bigquery.udf.core import PythonToJavaScriptTranslator, SymbolTable

-pytestmark = pytest.mark.bigquery
+pytestmark = [pytest.mark.bigquery, pytest.mark.udf]


sounds good. I will work on that.

jreback · 2020-02-26T12:14:38Z

@xmnlab this is blocking PR for the release @xmnlab ping on green.

xmnlab · 2020-02-26T16:04:58Z

@jreback I added the changes you requested. it seems CI is very slow today.

xmnlab · 2020-02-26T19:54:57Z

it seems pytest.xfail() will break the doctest: pytest-dev/pytest#310

xmnlab · 2020-02-27T00:16:46Z

green :)

jreback · 2020-02-27T12:45:32Z

ibis/bigquery/udf/tests/test_find.py

 from ibis.util import is_iterable

-pytestmark = pytest.mark.bigquery
+if PY38:


so as a followup, I'd like to change this to use the class based features rather than skips, e.g. e.g. in ibis/tests/backends, UDF can be modeled like for example RoundConvention (we explicitly add these tests to the backends that work with it).

please create an issue for this. thanks.

cc @icexelloss

jreback · 2020-02-27T12:45:56Z

thanks @xmnlab

xmnlab force-pushed the add-support-to-python38 branch 2 times, most recently from c9669d7 to b7d7d57 Compare February 1, 2020 01:40

jreback suggested changes Feb 2, 2020

View reviewed changes

jreback added the packaging label Feb 2, 2020

jreback added this to the Next Feature Release milestone Feb 2, 2020

xmnlab force-pushed the add-support-to-python38 branch from b7d7d57 to 5f7e4fa Compare February 11, 2020 01:58

xmnlab force-pushed the add-support-to-python38 branch from fb6088f to de40f53 Compare February 18, 2020 17:16

xmnlab force-pushed the add-support-to-python38 branch from 7d2c0ae to eb9b1dc Compare February 21, 2020 00:57

xmnlab marked this pull request as ready for review February 21, 2020 01:47

jreback suggested changes Feb 23, 2020

View reviewed changes

xmnlab mentioned this pull request Feb 24, 2020

CI: pymapd conda package is not compatible with Ibis on Python3.8 #2090

Closed

xmnlab commented Feb 24, 2020

View reviewed changes

SUPP: Add support to Python 3.8

93ab244

xmnlab force-pushed the add-support-to-python38 branch from eb9b1dc to 93ab244 Compare February 26, 2020 14:53

Add note for biquery.udf skipt tests

50b9bfb

Replace pytest.xfail by pytest.skip on doctest.

fc8a9e7

jreback approved these changes Feb 27, 2020

View reviewed changes

jreback reviewed Feb 27, 2020

View reviewed changes

jreback merged commit 2d6dfa4 into ibis-project:master Feb 27, 2020

xmnlab mentioned this pull request Mar 7, 2020

[IBIS] Add support to Python 3.8 Quansight/omnisci#90

Closed

xmnlab deleted the add-support-to-python38 branch December 21, 2020 18:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SUPP: Add support to Python 3.8 #2066

SUPP: Add support to Python 3.8 #2066

xmnlab commented Feb 1, 2020 •

edited

jreback Feb 2, 2020

xmnlab Feb 3, 2020

xmnlab Feb 11, 2020

jreback commented Feb 11, 2020

xmnlab commented Feb 11, 2020

xmnlab commented Feb 18, 2020

xmnlab commented Feb 21, 2020

jreback commented Feb 21, 2020

xmnlab commented Feb 21, 2020

xmnlab commented Feb 21, 2020

jreback left a comment

jreback Feb 23, 2020

xmnlab Feb 24, 2020

jreback Feb 23, 2020

xmnlab Feb 24, 2020

jreback Feb 23, 2020

xmnlab Feb 24, 2020

jreback Feb 23, 2020

jreback Feb 23, 2020

jreback Feb 23, 2020

xmnlab Feb 24, 2020

jreback Feb 27, 2020

xmnlab left a comment

xmnlab Feb 24, 2020

xmnlab Feb 24, 2020

xmnlab Feb 24, 2020

xmnlab Feb 24, 2020

jreback commented Feb 26, 2020

xmnlab commented Feb 26, 2020 •

edited

xmnlab commented Feb 26, 2020

xmnlab commented Feb 27, 2020

jreback Feb 27, 2020

jreback commented Feb 27, 2020

		@@ -12,6 +13,13 @@

		__all__ = ('udf',)

		# NOTE: used for skip doctest for Python 3.8

SUPP: Add support to Python 3.8 #2066

SUPP: Add support to Python 3.8 #2066

Conversation

xmnlab commented Feb 1, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Feb 11, 2020

xmnlab commented Feb 11, 2020

xmnlab commented Feb 18, 2020

xmnlab commented Feb 21, 2020

jreback commented Feb 21, 2020

xmnlab commented Feb 21, 2020

xmnlab commented Feb 21, 2020

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xmnlab left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Feb 26, 2020

xmnlab commented Feb 26, 2020 • edited

xmnlab commented Feb 26, 2020

xmnlab commented Feb 27, 2020

Choose a reason for hiding this comment

jreback commented Feb 27, 2020

xmnlab commented Feb 1, 2020 •

edited

xmnlab commented Feb 26, 2020 •

edited