Error on Win10 if wrapping py function with default int value that overflows with int32

[fultoncoe]

Keeping the top part so I don't forget.... I will post utils::sessionInfo() and reticulate::py_config() in the next post.

##' Please use version 3.9 or 3.10 of python
##' important line of code to test if is python has crashed or not
print("hello")

# Import library
from tableauhyperapi import HyperProcess

##' This is the failing line.
from tableauhyperapi import SqlType

print("test finished")

By crash I mean that once the error is "triggered" any input or python code after the error produces the same error. To fix the issue I have to completely restart R, use rm() to remove any persistant local variables then use gc() to free back the memory in unallocated state. This seems to be an issue with the reticulate version 1.31 and later as back in 08/2022 did not have this bug.

The bug is that the syntax user expects to use is for python not R. So warnings and Error should be thrown for things that are not compatible between the two languages. Errors should also not themselves crash the application. There should be error handling to double check syntax. The bug is the error right now is fatal. Less fatal errors in reticulate would be nice.

Ok let get into things.... with some pictures of running different lines of code

Edit: Added a comma to reproducible example so the correct error code produces

[fultoncoe]

First the session info.

Here is is as code:

    -- Attaching core tidyverse packages ------------------------------------------------- tidyverse 2.0.0 --
    v dplyr     1.1.2     v readr     2.1.4
    v forcats   1.0.0     v stringr   1.5.0
    v ggplot2   3.4.3     v tibble    3.2.1
    v lubridate 1.9.2     v tidyr     1.3.0
    v purrr     1.0.2     
    -- Conflicts ------------------------------------------------------------------- tidyverse_conflicts() --
    x dplyr::filter() masks stats::filter()
    x dplyr::lag()    masks stats::lag()
    i Use the conflicted package to force all conflicts to become errors
    Error in eval(ei, envir) : 
      Code stop earlly. Python packages are ready setup for use reproducing error
> utils::sessionInfo()
      R version 4.3.1 (2023-06-16 ucrt)
      Platform: x86_64-w64-mingw32/x64 (64-bit)
      Running under: Windows Server 2019 x64 (build 17763)
      
      Matrix products: default
      
      
      locale:
      [1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
      [3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
      [5] LC_TIME=English_United States.1252    
      
      time zone: America/New_York
      tzcode source: internal
      
      attached base packages:
      [1] stats     graphics  grDevices utils     datasets  methods   base     
      
      other attached packages:
       [1] reticulate_1.31 lubridate_1.9.2 forcats_1.0.0   stringr_1.5.0   dplyr_1.1.2     purrr_1.0.2    
       [7] readr_2.1.4     tidyr_1.3.0     tibble_3.2.1    ggplot2_3.4.3   tidyverse_2.0.0
      
      loaded via a namespace (and not attached):
       [1] Matrix_1.6-1      gtable_0.3.4      jsonlite_1.8.7    compiler_4.3.1    tidyselect_1.2.0 
       [6] Rcpp_1.0.11       png_0.1-8         scales_1.2.1      here_1.0.1        lattice_0.21-8   
      [11] R6_2.5.1          generics_0.1.3    RODBC_1.3-21      rprojroot_2.0.3   munsell_0.5.0    
      [16] pillar_1.9.0      tzdb_0.4.0        rlang_1.1.1       utf8_1.2.3        stringi_1.7.12   
      [21] timechange_0.2.0  cli_3.6.1         withr_2.5.0       magrittr_2.0.3    grid_4.3.1       
      [26] rstudioapi_0.15.0 hms_1.1.3         lifecycle_1.0.3   vctrs_0.6.3       glue_1.6.2       
      [31] fansi_1.0.4       colorspace_2.1-0  tools_4.3.1       pkgconfig_2.0.3  

And python version

      reticulate::py_config()
        python:         E:/Scripts/PycharmProjects/Selenium/venv/Scripts/python.exe
        libpython:      E:/Program Files/Python/python310.dll
        pythonhome:     E:/Scripts/PycharmProjects/Selenium/venv
        virtualenv:     E:/Scripts/PycharmProjects/Selenium/venv/Scripts/activate_this.py
        version:        3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
        Architecture:   64bit
        numpy:          E:/Program Files/Python/Lib/site-packages/numpy
        numpy_version:  1.23.3

NOTE: Python version was forced by use_python() function

[fultoncoe]

So after digging some more digging with reproduciable examples, I found its the Sqltype python script (and function) in tableau hyper api that is triggering the crash not the connection script. I share more in the morning.

[t-kalinowski]

Hi, thanks for reporting. Would you be able to provide instructions I can follow to reproduce the error locally?

Error in eval(ei, envir) : 
Code stop earlly. Python packages are ready setup for use reproducing error

Where is this output from? Do you have something in .Rprofile perhaps?

reticulate_1.31

Can you please install the development version of reticulate, just to rule out that the issue hasn't already been fixed?

pak::pak("rstudio/reticulate")

[fultoncoe]

Error in eval(ei, envir) : 
Code stop earlly. Python packages are ready setup for use reproducing error

Where is this output from? Do you have something in .Rprofile perhaps?

So I thought I upload the pictures of the error last week and the reproducible example, but I did not. The error is in the reply below.
I am a bit embarrassed I forgot to crop out that "stop()" error line. On the primary machine its at work, so to not damage anyone else's work, I always work in development virtual environments. I often use stop() to perfectly reproduce the virtual environment as .Rprofile has not be reliable in the past. Here the full line of code in case your curious.

P.S. Fixing this is not urgent as I can just roll back the version of R.

[fultoncoe]

@t-kalinowski Sorry for the delay. So the error is this...

hello
test finished
Error in py_get_formals(callable) : 
  SystemError: _PyEval_EvalFrameDefault returned a result with an exception set
Run `reticulate::py_last_error()` for details.

Directions to Reproduce

1. Install reticulate version that is newer that December of last year. Setup an virtual environment with python 3.9, or 3.10. Then use use_virtualenv() in your R script to protect the rest of your python environment*
2. Pip install just the most basic tableau api in that virtual environment by running pip install tableauhyperapi. More documentation found here: https://tableau.github.io/hyper-db/docs/installation
3. Copy to a python script the latest reprex code above at the very top . (Mine is called PythonErrorExample_version2.py)
4. Have a simple R script to run that python code as (given the fatal nature of this error) you may be restarting your R session and terminating it alot. For convenience I have have included both below.
   R file:

library(reticulate)
use_virtualenv("path to the venv and python.exe file in your virtual environment")
utils::sessionInfo()
reticulate::py_config()

# Run test 1
source_python("PythonErrorExample_version2.py")

python file:

print("hello")
from tableauhyperapi import HyperProcess
##' This is the failing line.
from tableauhyperapi import SqlType
print("test finished")

Now your ready to run the examples and tests.

[fultoncoe]

Now for convenience I will document all the tests below showing how the python file changes between the tests.

Reproducible Tests

Test 1: Does the reticulate installation work?

Comment out the failing python code (as shown below). This way you can test that you have installed python and Tableau hyper api (or any python code) correctly.

print("hello")
from tableauhyperapi import HyperProcess
##' This is the failing line.
# from tableauhyperapi import SqlType
print("test finished")

Picture of output:

Test 2 Reproduce the error

Have no code commented out and just verbatim copy/paste the code at the very top of this thread:

print("hello")
from tableauhyperapi import HyperProcess
##' This is the failing line.
from tableauhyperapi import SqlType
print("test finished")

Picture of output:

Test 3 Test if the Error from Test 2 is fatal

So an error in my opinion becomes a fatal error for an application (R/Rstudio) in one of 3 ways--1) Crashes OS; 2) Crashes or freezes the application; 3) Persists even when the line of code that caused the error was not run. This means that every line of code now triggers the error.

Comment out everything except the "hello" line and run the script below again:

print("hello")
# from tableauhyperapi import HyperProcess
##' This is the failing line.
# from tableauhyperapi import SqlType
# print("test finished")

Picture of output:

[fultoncoe]

Can you please install the development version of reticulate, just to rule out that the issue hasn't already been fixed?

pak::pak("rstudio/reticulate")

Its not fixed t-kalinowski, in the development version reticulate_1.32.0.9002. To make a long story short, I been able to isolate it down to an error with one function in the tableauhyperapi module I called SqlType and the reticulate function called source_python. I had to install this on test machine so here is the test code followed by the error and new sessioninformation.

Test 4 test developer version of reticulate on a new machine

The new error code that is produced because the new machine uses a miniconda and slightly different version of python

The full error trace is as follows:

> source_python("PythonErrorExample_version2.py")
Error in py_get_formals(callable) : 
  SystemError: _PyEval_EvalFrameDefault returned a result with an error set
Run `reticulate::py_last_error()` for details.
> reticulate::py_last_error()

── Python Exception Message ──────────────────────────────────────────────────────
OverflowError: Python int too large to convert to C long

The above exception was the direct cause of the following exception:

SystemError: _PyEval_EvalFrameDefault returned a result with an error set

── R Traceback ───────────────────────────────────────────────────────────────────
     ▆
  1. ├─reticulate::source_python("PythonErrorExample_version2.py")
  2. │ ├─main_dict[[name]]
  3. │ └─reticulate:::`[[.python.builtin.dict`(main_dict, name)
  4. │   └─reticulate:::py_maybe_convert(item, py_has_convert(x))
  5. │     ├─reticulate::py_to_r(x)
  6. │     └─reticulate:::py_to_r.default(x)
  7. │       └─reticulate:::py_ref_to_r(x)
  8. └─reticulate (local) `<fn>`(`<python.builtin.type>`, TRUE)
  9.   ├─base::as.function.default(...)
 10.   └─reticulate:::py_get_formals(callable)

The R code was as follows

library(reticulate)
library(tidyverse)
utils::sessionInfo()
reticulate::py_config()
# Run test 4
source_python("PythonErrorExample_version2.py")

# Run test 5
# discover running the line of code in interactive mode produces no error
foo <- py_run_string("from tableauhyperapi import SqlType")
bar <- py_run_file("PythonErrorExample_version2.py")

The python file PythonErrorExample_version2.py was as follows with everything except the line with the error running:

##' Please use version 3.9 or 3.10 of python
##' important line of code to test if is python has crashed or not
# print("hello")

# Import library
# from tableauhyperapi import HyperProcess

##' This is the failing line.
from tableauhyperapi import SqlType

#print("test finished")

Session information and python config():

> utils::sessionInfo()
R version 4.3.1 (2023-06-16 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19045)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.utf8  LC_CTYPE=English_United States.utf8   
[3] LC_MONETARY=English_United States.utf8 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.utf8    

time zone: America/New_York
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] lubridate_1.9.3        forcats_1.0.0          stringr_1.5.0         
 [4] dplyr_1.1.3            purrr_1.0.2            readr_2.1.4           
 [7] tidyr_1.3.0            tibble_3.2.1           ggplot2_3.4.3         
[10] tidyverse_2.0.0        reticulate_1.32.0.9002

reticulate::py_config()
python:         C:/Users/Computer/AppData/Local/r-miniconda/envs/r-reticulate/python.exe
libpython:      C:/Users/Computer/AppData/Local/r-miniconda/envs/r-reticulate/python39.dll
pythonhome:     C:/Users/Computer/AppData/Local/r-miniconda/envs/r-reticulate
version:        3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:28:38) [MSC v.1929 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:/Users/Computer/AppData/Local/r-miniconda/envs/r-reticulate/Lib/site-packages/numpy
numpy_version:  1.25.1

[fultoncoe]

Now what is was odd is that that only source() errors out. For example, running the following code for test 5 produces no errors:

Test 5 trying to see if other reticulate functions have the same error

# Run test 5
# discover running the line of code in interactive mode produces no error
foo <- py_run_string("from tableauhyperapi import SqlType")
bar <- py_run_file("PythonErrorExample_version2.py")

Proof:

[fultoncoe]

So here is test 6... I think I finally was able to get to a 2 line reprex.

Test 6

The final simple example:

# Run test 6
# What if we to use without converting? Then 
foo <- reticulate::py_run_string("from tableauhyperapi import SqlType",
                          convert = FALSE)
foo$SqlType

The error result:

> foo$SqlType
Error in py_get_formals(callable) : 
  SystemError: _PyEval_EvalFrameDefault returned a result with an error set
Run `reticulate::py_last_error()` for details.
> reticulate::py_last_error()

── Python Exception Message ─────────────────────────────────────────────────────
OverflowError: Python int too large to convert to C long

The above exception was the direct cause of the following exception:

SystemError: _PyEval_EvalFrameDefault returned a result with an error set

── R Traceback ──────────────────────────────────────────────────────────────────
     ▆
  1. ├─foo$SqlType
  2. ├─reticulate:::`$.python.builtin.dict`(foo, SqlType)
  3. │ └─reticulate:::`[.python.builtin.dict`(x, name)
  4. │   └─reticulate:::py_maybe_convert(item, py_has_convert(x))
  5. │     ├─reticulate::py_to_r(x)
  6. │     └─reticulate:::py_to_r.default(x)
  7. │       └─reticulate:::py_ref_to_r(x)
  8. └─reticulate (local) `<fn>`(`<python.builtin.type>`, FALSE)
  9.   ├─base::as.function.default(...)
 10.   └─reticulate:::py_get_formals(callable)

[t-kalinowski]

Thank you for the detailed investigation! It was very helpful.

I can reproduce the error without Tableau, on Windows, but not on macOS. The error occurs if one of the default arguments of a python function is an integer greater than a 32 bit int can represent.

sprintf("lambda x = %s + 2: x", .Machine$integer.max) |>
  py_eval() |> 
  formals() |> _$x

On macOS I see a silent overflow of x to -2147483647. On Windows we see the Python Exception.

[t-kalinowski]

An even smaller MRE:

> py_eval(paste(.Machine$integer.max, "+ 19"))
Error in py_eval_impl(code, convert) : 
  OverflowError: Python int too large to convert to C long
Run `reticulate::py_last_error()` for details.

[t-kalinowski]

@fultoncoe Until this is fixed, you can wrap the SqlType function to avoid the error:

reticulate::py_run_string(r"---(
from tableauhyperapi import SqlType as _SqlType
tableauhyperapi.SqlType = lambda *args, **kwargs: _SqlType(*args, **kwargs)
)---", local = TRUE)