Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Default email template fails with latest scrapinghub client version #429

Open
curita opened this issue Jan 15, 2024 · 0 comments
Open

Default email template fails with latest scrapinghub client version #429

curita opened this issue Jan 15, 2024 · 0 comments
Assignees

Comments

@curita
Copy link
Member

curita commented Jan 15, 2024

Issue

data.job.metadata (an instance of scrapinghub.client.jobs:JobMeta) cannot be subscribed. There are multiple places in Spidermon's templates where job meta keys are accessed via data.job.metadata[x]. Those cases fail because of it.

It's unclear when job.metadata stopped being subscribable or if this is a change in the latest Jinja versions (see: https://jinja.palletsprojects.com/en/3.0.x/templates/#variables), but it doesn't seem to work now.

Locally can be partially reproduced via:

In[1]: from scrapinghub import ScrapinghubClient

In[2]: job = ScrapinghubClient(SHUB_APIKEY).get_job(JOB_ID)

In[3]: job.metadata["spider"]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[3], line 1
----> 1 job.metadata["spider"]

TypeError: 'JobMeta' object is not subscriptable

In[4]: from jinja2 import Template

In[5]: Template("{% set is_script = job.metadata['spider'].startswith('py:') %}{{ is_script }}").render(job=job)
---------------------------------------------------------------------------
UndefinedError                            Traceback (most recent call last)
Cell In[5], line 1
----> 1 Template("{% set is_script = job.metadata['spider'].startswith('py:') %}{{ is_script }}").render(job=job)

File ~/src/project/.venv/lib/python3.10/site-packages/jinja2/environment.py:1301, in Template.render(self, *args, **kwargs)
   1299     return self.environment.concat(self.root_render_func(ctx))  # type: ignore
   1300 except Exception:
-> 1301     self.environment.handle_exception()

File ~/src/project/.venv/lib/python3.10/site-packages/jinja2/environment.py:936, in Environment.handle_exception(self, source)
    931 """Exception handling helper.  This is used internally to either raise
    932 rewritten exceptions or return a rendered traceback for the template.
    933 """
    934 from .debug import rewrite_traceback_stack
--> 936 raise rewrite_traceback_stack(source=source)

File <template>:1, in top-level template code()

File ~/src/project/.venv/lib/python3.10/site-packages/jinja2/environment.py:485, in Environment.getattr(self, obj, attribute)
    481 """Get an item or attribute of an object but prefer the attribute.
    482 Unlike :meth:`getitem` the attribute *must* be a string.
    483 """
    484 try:
--> 485     return getattr(obj, attribute)
    486 except AttributeError:
    487     pass

UndefinedError: 'scrapinghub.client.jobs.JobMeta object' has no attribute 'spider'

In [6]: Template("{% set is_script = job.metadata.get('spider').startswith('py:') %}{{ is_script }}").render(job=job)
Out[6]: 'False'

Proposal

Replace all instances of data.job.metadata[x] in the templates with data.job.metadata.get(x).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants