Incorrect URL for Vega data files #7017

malramsay64 · 2019-08-15T00:45:06Z

Description

When creating a figure with altair, using the data_transformer functionality, which stores the data in a local file and gets Vega load the file, results in a URL which is improperly formed, so the data never loads.

The URL has the _xsrf information in front of the name of the file which is being loaded.

403 GET /files/?_xsrf=2|572196ba|42fbfea4f325901e3948cbc75bd747c8|1565583318/altair-data-6572fe4ca88650e09d67c52dd6a94b8a.json (172.17.0.1): is not a file

Issue first raised vega/altair#1651 and directed here.

Reproduce

Install jupyterlab

conda create -n jupyter-test jupyterlab==1.0.5 altair==3.2.0 pandas==0.25.0

In a new notebook run

import altair as alt
import pandas as pd
alt.data_transformers.enable("json")
df = pd.DataFrame({"x": [1,2,3], "y": [1,2,3]})
alt.Chart(df).mark_line().encode(x="x", y="y")

In the browser tools there is the error

WARN Loading failed altair-data-6572fe4ca88650e09d67c52dd6a94b8a.json Error: "403Forbidden"

while the console has the error message

403 GET /files/?_xsrf=2|572196ba|42fbfea4f325901e3948cbc75bd747c8|1565583318/altair-data-6572fe4ca88650e09d67c52dd6a94b8a.json (172.17.0.1): is not a file

The problem seems to be in the GET request, which has the _xsrf information before the file information. Using the firefox dev tools to edit and resend the request, both the removal of the _xsrf parameter from the url, or moving it after the filename result in a successful request.

Expected behavior

It is expected that the chart is filled with a straight line,
instead there is no data in the chart.

Context

Operating System and version: Fedora 30
Browser and version: Firefox 68
JupyterLab version: 1.0.5

Troubleshoot Output

$PATH: /opt/conda/bin /usr/local/sbin /usr/local/bin /usr/sbin /usr/bin /sbin /bin

sys.path:
/opt/conda/envs/jupyter/bin
/opt/conda/envs/jupyter/lib/python37.zip
/opt/conda/envs/jupyter/lib/python3.7
/opt/conda/envs/jupyter/lib/python3.7/lib-dynload
/opt/conda/envs/jupyter/lib/python3.7/site-packages

sys.executable:
/opt/conda/envs/jupyter/bin/python3.7

sys.version:
3.7.4 (default, Aug 9 2019, 18:51:30)
[GCC 7.3.0]

platform.platform():
Linux-5.2.7-200.fc30.x86_64-x86_64-with-debian-9.8

pip list:
Package Version
----------------- ---------
altair 3.2.0
attrs 19.1.0
backcall 0.1.0
bleach 3.1.0
certifi 2019.6.16
decorator 4.4.0
defusedxml 0.6.0
entrypoints 0.3
ipykernel 5.1.1
ipython 7.7.0
ipython-genutils 0.2.0
jedi 0.13.3
Jinja2 2.10.1
json5 0.8.5
jsonschema 3.0.1
jupyter-client 5.3.1
jupyter-core 4.5.0
jupyterlab 1.0.5
jupyterlab-server 1.0.0
MarkupSafe 1.1.1
mistune 0.8.4
mkl-fft 1.0.12
mkl-random 1.0.2
nbconvert 5.5.0
nbformat 4.4.0
notebook 6.0.0
numpy 1.16.4
pandas 0.25.0
pandocfilters 1.4.2
parso 0.5.0
pexpect 4.7.0
pickleshare 0.7.5
pip 19.1.1
prometheus-client 0.7.1
prompt-toolkit 2.0.9
ptyprocess 0.6.0
Pygments 2.4.2
pyrsistent 0.14.11
python-dateutil 2.8.0
pytz 2019.1
pyzmq 18.0.0
Send2Trash 1.5.0
setuptools 41.0.1
six 1.12.0
terminado 0.8.2
testpath 0.4.2
toolz 0.10.0
tornado 6.0.3
traitlets 4.3.2
wcwidth 0.1.7
webencodings 0.5.1
wheel 0.33.4

conda list:
# packages in environment at /opt/conda:
#
# Name Version Build Channel
asn1crypto 0.24.0 py37_0
ca-certificates 2019.1.23 0
certifi 2019.3.9 py37_0
cffi 1.12.2 py37h2e261b9_1
chardet 3.0.4 py37_1
conda 4.6.14 py37_0
cryptography 2.6.1 py37h1ba5d50_0
idna 2.8 py37_0
libedit 3.1.20181209 hc058e9b_0
libffi 3.2.1 hd88cf55_4
libgcc-ng 8.2.0 hdf63c60_1
libstdcxx-ng 8.2.0 hdf63c60_1
ncurses 6.1 he6710b0_1
openssl 1.1.1b h7b6447c_1
pip 19.0.3 py37_0
pycosat 0.6.3 py37h14c3975_0
pycparser 2.19 py37_0
pyopenssl 19.0.0 py37_0
pysocks 1.6.8 py37_0
python 3.7.3 h0371630_0
readline 7.0 h7b6447c_5
requests 2.21.0 py37_0
ruamel_yaml 0.15.46 py37h14c3975_0
setuptools 41.0.0 py37_0
six 1.12.0 py37_0
sqlite 3.27.2 h7b6447c_0
tk 8.6.8 hbc83047_0
urllib3 1.24.1 py37_0
wheel 0.33.1 py37_0
xz 5.2.4 h14c3975_4
yaml 0.1.7 had09818_2
zlib 1.2.11 h7b6447c_3

Command Line Output

[W 00:20:36.600 LabApp] 403 GET /files/?_xsrf=2|572196ba|42fbfea4f325901e3948cbc75bd747c8|1565583318/altair-data-6572fe4ca88650e09d67c52dd6a94b8a.json (172.17.0.1):  is not a file
[W 00:20:36.601 LabApp] 403 GET /files/?_xsrf=2|572196ba|42fbfea4f325901e3948cbc75bd747c8|1565583318/altair-data-6572fe4ca88650e09d67c52dd6a94b8a.json (172.17.0.1) 1.67ms referer=http://127.0.0.1:8889/lab

Browser Output

WARN Loading failed altair-data-6572fe4ca88650e09d67c52dd6a94b8a.json Error: "403Forbidden"

The text was updated successfully, but these errors were encountered:

jasongrout · 2019-08-15T00:58:36Z

Looks like Vega extension should not assume that getdownloadurl returns just a path it can append to:

jupyterlab/packages/vega5-extension/src/index.ts

Line 80 in 84539f1

const baseURL = await this._resolver.getDownloadUrl(path);

jasongrout · 2019-08-15T12:37:47Z

@domoritz - is there an easy way to override https://github.com/vega/vega/blob/dd006488d508e0f1e5154cb20eeea3ac7a3f4cd2/packages/vega-loader/src/loader.js#L85-L91 in the vega 4/5 jlab plugin to call a function in JLab that, given a relative path, will return a URL to use to download it?

We could try to work under the assumption that we can get a base url (which is where we have an error in the vega4/5 extensions now), but really it would be more correct to let the system give us a download URL given a path.

domoritz · 2019-08-15T12:51:26Z

We can provide a custom loader to Vega and replace all the logic we want. We are currently only customizing the base URL in

jupyterlab/packages/vega5-extension/src/index.ts

Line 97 in 84539f1

loader: {

and HTTP header. The easiest way would probably be to just override the sanitize function. A cleaner way would probably be to request a file:// url and write a custom file handler that we set to https://github.com/vega/vega/blob/dd006488d508e0f1e5154cb20eeea3ac7a3f4cd2/packages/vega-loader/src/loader.js#L33.

Does this answer your question?

jasongrout · 2019-08-15T13:06:04Z

It gives hope :).

What do you mean by "request a file:// url" - ask the python side in altair to explicitly give a file:// url instead of just a relative path to load?

To override the default loader, we just pass in a {loader: {load, sanitize, http, file}} object at

jupyterlab/packages/vega5-extension/src/index.ts

Lines 97 to 100 in 84539f1

    
           loader: { 
        
             baseURL, 
        
             http: { credentials: 'same-origin' } 
        
           }

instead of the loader options we are currently passing in?

jasongrout · 2019-08-15T13:20:22Z

What do you mean by "request a file:// url" - ask the python side in altair to explicitly give a file:// url instead of just a relative path to load?

I read https://tools.ietf.org/html/rfc8089 for a bit, and it seems that there is no correct way to give a file: uri that is relative to some external root - it seems that file: uris are always absolute. That won't work for our case, where the python code does not know the right absolute path to give.

jasongrout · 2019-08-15T18:07:38Z

@malramsay64 - did this ever work in JupyterLab? If so, what version?

Copying reproducing instructions from vega/altair#1651:

conda create -n jupyter-test jupyterlab==1.0.5 altair==3.2.0 pandas==0.25.0

import altair as alt
import pandas as pd
from pprint import pprint
alt.data_transformers.enable("json")
df = pd.DataFrame({"x": [1,2,3], "y": [1,2,3]})
alt.Chart(df).mark_line().encode(x="x", y="y")

jasongrout · 2019-08-15T18:11:25Z

@malramsay64 - did this ever work in JupyterLab? If so, what version?

It's working for me in jlab 1.0.2:

conda create -n jupyter-test jupyterlab==1.0.2 altair==3.2.0 pandas==0.25.0

…rver. Perhaps we should also use the services package to actually send the request to the server. That would complicate these plugins beyond simple rendermime plugins, though. Fixes jupyterlab#7017

jasongrout · 2019-08-15T19:20:18Z

This should be fixed in #7022

malramsay64 · 2019-08-16T00:03:17Z

The fix works for me.

Thank you @jasongrout for such a quick response and fix for this issue!!

jasongrout mentioned this issue Aug 15, 2019

data_transformers not working in Jupyterlab vega/altair#1651

Closed

jasongrout modified the milestones: 1.0.x, 1.1 Aug 15, 2019

jasongrout mentioned this issue Aug 15, 2019

Fix Vega downloads and download URLs in general #7022

Merged

blink1073 closed this as completed in #7022 Aug 16, 2019

This was referenced Aug 19, 2019

Vega extension not loading in subdirectory #7047

Closed

Fix baseURL in vega extensions #7047 #7048

Closed

lock bot added the status:resolved-locked Closed issues are locked after 30 days inactivity. Please open a new issue for related discussion. label Sep 15, 2019

lock bot locked as resolved and limited conversation to collaborators Sep 15, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incorrect URL for Vega data files #7017

Incorrect URL for Vega data files #7017

malramsay64 commented Aug 15, 2019

jasongrout commented Aug 15, 2019

jasongrout commented Aug 15, 2019 •

edited

domoritz commented Aug 15, 2019

jasongrout commented Aug 15, 2019

jasongrout commented Aug 15, 2019 •

edited

jasongrout commented Aug 15, 2019

jasongrout commented Aug 15, 2019 •

edited

jasongrout commented Aug 15, 2019

malramsay64 commented Aug 16, 2019

Incorrect URL for Vega data files #7017

Incorrect URL for Vega data files #7017

Comments

malramsay64 commented Aug 15, 2019

Description

Reproduce

Expected behavior

Context

jasongrout commented Aug 15, 2019

jasongrout commented Aug 15, 2019 • edited

domoritz commented Aug 15, 2019

jasongrout commented Aug 15, 2019

jasongrout commented Aug 15, 2019 • edited

jasongrout commented Aug 15, 2019

jasongrout commented Aug 15, 2019 • edited

jasongrout commented Aug 15, 2019

malramsay64 commented Aug 16, 2019

jasongrout commented Aug 15, 2019 •

edited

jasongrout commented Aug 15, 2019 •

edited

jasongrout commented Aug 15, 2019 •

edited