[Bug]: RuntimeError: exception in Python subject: KeyError: 'data' #13

vikassinghvi2007 · 2024-03-15T13:51:52Z

Steps to reproduce

Getting an error when trying to run the Airbyte showcase example from here: https://pathway.com/developers/showcases/etl-python-airbyte

Relevant log output

Traceback (most recent call last):
  File "/Users/vikassinghvi/Documents/GitHub/veloraapp/services/github_issues.py", line 53, in <module>
    pw.run()
  File "/Users/vikassinghvi/Documents/GitHub/veloraapp/services/venv/lib/python3.11/site-packages/pathway/internals/runtime_type_check.py", line 19, in with_type_validation
    return beartype.beartype(f)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<@beartype(pathway.internals.run.run) at 0x1478f8360>", line 129, in run
  File "/Users/vikassinghvi/Documents/GitHub/veloraapp/services/venv/lib/python3.11/site-packages/pathway/internals/run.py", line 47, in run
    ).run_outputs()
      ^^^^^^^^^^^^^
  File "/Users/vikassinghvi/Documents/GitHub/veloraapp/services/venv/lib/python3.11/site-packages/pathway/internals/graph_runner/__init__.py", line 103, in run_outputs
    self.run_nodes(self._graph.global_scope.output_nodes, after_build=after_build)
  File "/Users/vikassinghvi/Documents/GitHub/veloraapp/services/venv/lib/python3.11/site-packages/pathway/internals/graph_runner/__init__.py", line 79, in run_nodes
    self._run(all_nodes, after_build=after_build)
  File "/Users/vikassinghvi/Documents/GitHub/veloraapp/services/venv/lib/python3.11/site-packages/pathway/internals/graph_runner/__init__.py", line 179, in _run
    return api.run_with_new_graph(
           ^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: exception in Python subject: KeyError: 'data'

What did you expect to happen?

Expected to pull the commits into a jsonlines file, as demonstrated.

Version

0.8.3

Docker Versions (if used)

No response

OS

MacOS

On which CPU architecture did you run Pathway?

None

The text was updated successfully, but these errors were encountered:

zxqfd555-pw · 2024-03-15T16:29:08Z

Hi Vikas!

Thank you for reporting this issue!

I wasn't able to reproduce it with Linux, however, it indeed reproduces when running on MacOS. We're researching right now why it happens on MacOS, and how to make this showcase work on both platforms. We will keep you updated and will be back a bit later when we understand why there's a platform-specific difference.

Right now, please consider running it on Linux, if you have such a possibility. Another option is to run it in Docker, but please keep in mind that you may probably need to deal with the DinD issue because the airbyte-serverless connector uses Docker to access Airbyte connectors.

vikassinghvi2007 · 2024-03-15T17:45:12Z

Thank you. I tried it with GitHub connector but is this issue across all AirByte connectors? If yes, it will stall my POC efforts - would appreciate if it can be expedited. Thanks in advance.

…

On Fri, 15 Mar 2024 at 9:59 PM, Sergey Kulik ***@***.***> wrote: Hi Vikas! Thank you for reporting this issue! I wasn't able to reproduce it with Linux, however, it indeed reproduces when running on MacOS. We're researching right now why it happens on MacOS, and how to make this showcase work on both platforms. We will keep you updated and will be back a bit later when we understand why there's a platform-specific difference. Right now, please consider running it on Linux, if you have such a possibility. Another option is to run it in Docker, but please keep in mind that you may probably need to deal with the DinD issue because the airbyte-serverless connector uses Docker to access Airbyte connectors. — Reply to this email directly, view it on GitHub <#13 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUNJK34UKQIEVQHRAICEGVDYYMOWTAVCNFSM6AAAAABEYBMS3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBQGAZDGMBQGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

zxqfd555-pw · 2024-03-15T18:59:40Z

After further investigation, I see that the main reason is the updated version of the protocol used in the GitHub connector. Could you please try pinning the connector version in the ./connections/github.yaml config to airbyte/source-github:1.6.0 and check if it helped? Of course, it is a half-measure only to make it possible to run it right now.

I'll shortly commit a fix with the compatible state processing and we will release the new version of Pathway which correctly works with the modern protocols. This release will happen at the beginning of the next week, most likely on Monday.

vikassinghvi2007 · 2024-03-16T06:29:31Z

Thanks Sergey. The workaround did work. Wait wait for the fix commit, to remove the 1.6.0 version dependency.

…

On Sat, Mar 16, 2024 at 12:30 AM Sergey Kulik ***@***.***> wrote: After further investigation, I see that the main reason is the updated version of the protocol used in the GitHub connector. Could you please try pinning the connector version in the ./connections/github.yaml config to airbyte/source-github:1.6.0 and check if it helped? Of course, it is a half-measure only to make it possible to run it right now. I'll shortly commit a fix with the compatible state processing and we will release the new version of Pathway which correctly works with the modern protocols. This release will happen at the beginning of the next week, most likely on Monday. — Reply to this email directly, view it on GitHub <#13 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUNJK36NUMP564I3K3VBMDLYYNALFAVCNFSM6AAAAABEYBMS3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBQGI3TKNBXGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

vikassinghvi2007 · 2024-03-16T08:50:59Z

@zxqfd555-pw :

Although the workaround works, the pipeline breaks after some time, with the following error:

RuntimeError: exception in Python subject: AirbyteSourceException: {"message": "Something went wrong in the connector. See the logs for more details.", "internal_message": "Could not read json file /mnt/temp/catalog.json: Expecting ':' delimiter: line 1 column 8192 (char 8191). Please ensure that it is a valid JSON.", "stack_trace": "Traceback (most recent call last):\n File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/connector.py", line 65, in _read_json_file\n return json.loads(contents)\n File "/usr/local/lib/python3.9/json/init.py", line 346, in loads\n return _default_decoder.decode(s)\n File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n File "/usr/local/lib/python3.9/json/decoder.py", line 353, in raw_decode\n obj, end = self.scan_once(s, idx)\njson.decoder.JSONDecodeError: Expecting ':' delimiter: line 1 column 8192 (char 8191)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/airbyte/integration_code/main.py", line 8, in \n run()\n File "/airbyte/integration_code/source_github/run.py", line 17, in run\n launch(source, sys.argv[1:])\n File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 214, in launch\n for message in source_entrypoint.run(parsed_args):\n File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 118, in run\n config_catalog = self.source.read_catalog(parsed_args.catalog)\n File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/source.py", line 91, in read_catalog\n return ConfiguredAirbyteCatalog.parse_obj(cls._read_json_file(catalog_path))\n File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/connector.py", line 67, in _read_json_file\n raise ValueError(f"Could not read json file {file_path}: {error}. Please ensure that it is a valid JSON.")\nValueError: Could not read json file /mnt/temp/catalog.json: Expecting ':' delimiter: line 1 column 8192 (char 8191). Please ensure that it is a valid JSON.\n", "failure_type": "system_error"}

zxqfd555-pw · 2024-03-16T13:30:12Z

Hi Vikas!

Thank you for another piece of valuable feedback!

It looks like those failures are spurious and the reason is connected to one of the libraries we used to implement the connector. I did some research and created an issue in the related repo - you can see it linked.

Besides, we can handle it gracefully on our side by implementing retries for these cases. I've done a PR for that internally and they will also be in the release I announced yesterday.

vikassinghvi2007 · 2024-03-16T13:31:59Z

Thanks Sergey.. I love the super fast and prompt responsiveness from your side 😊

…

On Sat, 16 Mar 2024 at 7:00 PM, Sergey Kulik ***@***.***> wrote: Hi Vikas! Thank you for another piece of valuable feedback! It looks like those failures are spurious and the reason is in one of the libraries we used to implement the connector. I did some research and created an issue in the related repo - you can see it linked. Besides, we can handle it gracefully on our side by implementing retries for these cases. I've done a PR for that internally and they will also be in the release I announced yesterday. — Reply to this email directly, view it on GitHub <#13 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUNJK3ZRSI52NFI45BGBE6LYYRCPTAVCNFSM6AAAAABEYBMS3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBRHE4DONRYGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

zxqfd555-pw · 2024-03-18T23:47:23Z

Hi Vikas!

We've released the version with an update! It contains a compatibility fix and retries for the spurious errors I've mentioned above.

Feel free to test it!

vikassinghvi2007 · 2024-03-19T09:39:25Z

Thanks, I'll check it out.

…

On Tue, 19 Mar 2024 at 5:17 AM, Sergey Kulik ***@***.***> wrote: Hi Vikas! We've released the version with an update! It contains a compatibility fix and retries for the spurious errors I've mentioned above. Feel free to test it! — Reply to this email directly, view it on GitHub <#13 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUNJK36VHFLYREYXEQZK3Q3YY54KDAVCNFSM6AAAAABEYBMS3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBVGMZDSMJWGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

vikassinghvi2007 · 2024-03-19T14:07:33Z

Hi @zxqfd555-pw , I am now getting error on 1.6.0 version and latest:

Traceback (most recent call last):
File "/Users/vikassinghvi/Documents/GitHub/veloraapp/services/github_issues.py", line 79, in
issues_table = pw.io.airbyte.read(
^^^^^^^^^^^^^^^^^^^
File "/Users/vikassinghvi/Documents/GitHub/veloraapp/services/venv/lib/python3.11/site-packages/pathway/io/airbyte/init.py", line 205, in read
for stream in source.configured_catalog["streams"]:
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/vikassinghvi/Documents/GitHub/veloraapp/services/venv/lib/python3.11/site-packages/airbyte_serverless/sources.py", line 169, in getattr
return getattr(self.source, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/vikassinghvi/Documents/GitHub/veloraapp/services/venv/lib/python3.11/site-packages/airbyte_serverless/sources.py", line 102, in configured_catalog
configured_catalog = self.catalog
^^^^^^^^^^^^
File "/Users/vikassinghvi/Documents/GitHub/veloraapp/services/venv/lib/python3.11/site-packages/airbyte_serverless/sources.py", line 97, in catalog
message = self._run_and_return_first_message('discover')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/vikassinghvi/Documents/GitHub/veloraapp/services/venv/lib/python3.11/site-packages/airbyte_serverless/sources.py", line 73, in _run_and_return_first_message
message = next(
^^^^^
File "/Users/vikassinghvi/Documents/GitHub/veloraapp/services/venv/lib/python3.11/site-packages/airbyte_serverless/sources.py", line 74, in
(message for message in messages if message['type'] not in ['LOG', 'TRACE']),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/vikassinghvi/Documents/GitHub/veloraapp/services/venv/lib/python3.11/site-packages/airbyte_serverless/sources.py", line 68, in _run
raise AirbyteSourceException(json.dumps(message['trace']['error']))
airbyte_serverless.sources.AirbyteSourceException: {"message": "Config validation error: None is not of type 'string'", "internal_message": "None is not of type 'string'", "stack_trace": "Traceback (most recent call last):\n File "/airbyte/integration_code/main.py", line 8, in \n run()\n File "/airbyte/integration_code/source_github/run.py", line 17, in run\n launch(source, sys.argv[1:])\n File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 214, in launch\n for message in source_entrypoint.run(parsed_args):\n File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 116, in run\n yield from map(AirbyteEntrypoint.airbyte_message_to_string, self.discover(source_spec, config))\n File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 150, in discover\n self.validate_connection(source_spec, config)\n File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 171, in validate_connection\n check_config_against_spec_or_exit(connector_config, source_spec)\n File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/utils/schema_helpers.py", line 175, in check_config_against_spec_or_exit\n raise AirbyteTracedException(\nairbyte_cdk.utils.traced_exception.AirbyteTracedException: None is not of type 'string'\n", "failure_type": "config_error"}

zxqfd555-pw · 2024-03-19T14:21:48Z

Hi Vikas!

I suspect that you have a not-filled optional field in the config or an unspecified required field. At least, this part of an error makes me think that:

None is not of type 'string'

Could you please ensure that your config doesn't contain them? If there are optional fields you don't use, please just delete them. If there are required fields that are not filled, please fill them in. You can also refer to the config example that we give in the tutorial.

Also, for the reference, here is a config I used to run it:

source:
  docker_image: "airbyte/source-github:latest"
  config:
    credentials:
      option_title: "PAT Credentials"
      personal_access_token: <github token here>
    repositories:
      - pathwaycom/pathway
    api_url: "https://api.github.com/"

It's stored in connections/github.yaml and I specify this path in the path parameter of the connector.

vikassinghvi2007 · 2024-03-20T06:30:30Z

Thanks, I tried after correcting my config file, with the latest airbyte connector version. I still get this:

Traceback (most recent call last):
File "/Users/vikassinghvi/Documents/GitHub/veloraapp/services/github_issues.py", line 89, in
pw.run()
File "/Users/vikassinghvi/Documents/GitHub/veloraapp/services/venv/lib/python3.11/site-packages/pathway/internals/runtime_type_check.py", line 19, in with_type_validation
return beartype.beartype(f)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<@beartype(pathway.internals.run.run) at 0x13fafc5e0>", line 129, in run
File "/Users/vikassinghvi/Documents/GitHub/veloraapp/services/venv/lib/python3.11/site-packages/pathway/internals/run.py", line 47, in run
).run_outputs()
^^^^^^^^^^^^^
File "/Users/vikassinghvi/Documents/GitHub/veloraapp/services/venv/lib/python3.11/site-packages/pathway/internals/graph_runner/init.py", line 103, in run_outputs
self.run_nodes(self._graph.global_scope.output_nodes, after_build=after_build)
File "/Users/vikassinghvi/Documents/GitHub/veloraapp/services/venv/lib/python3.11/site-packages/pathway/internals/graph_runner/init.py", line 79, in run_nodes
self._run(all_nodes, after_build=after_build)
File "/Users/vikassinghvi/Documents/GitHub/veloraapp/services/venv/lib/python3.11/site-packages/pathway/internals/graph_runner/init.py", line 179, in _run
return api.run_with_new_graph(
^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: exception in Python subject: KeyError: 'data'

It still does process extraction on older airbyte connector version 1.6.0.

zxqfd555-pw · 2024-03-20T15:01:32Z

Please make sure you have the latest pathway version. If you use pip, can do it by calling pip show pathway. The required version is 0.8.4.

If the version is below 0.8.4, you would need to upgrade it. It can be done, for example, by reinstalling pathway with pip uninstall pathway && pip install pathway.

umarbasha007 · 2024-03-21T14:13:56Z

Hi,

Thanks, I tried below suggestions,

Suggestion 1:

source:
  docker_image: "airbyte/source-github:latest"
  config:
    credentials:
      option_title: "PAT Credentials"
      personal_access_token: <github token here>
    repositories:
      - pathwaycom/pathway
    api_url: "https://api.github.com/"

in connections/github.yaml

Suggestion 2:

pip uninstall pathway && pip install pathway

to update pathway version to 0.8.4 and verified it using pip show pathway.

However, the run is breaking after 1-2 mins, and below is the error, I'm getting. Please help !!

Traceback (most recent call last):
  File "/Users/techmonk/Documents/Hustle/Velora_ai/veloraapp-main/services/github_issues.py", line 99, in <module>
    pw.run()
  File "/Users/techmonk/Documents/Hustle/Velora_ai/veloraapp-main/services/venv/lib/python3.11/site-packages/pathway/internals/runtime_type_check.py", line 19, in with_type_validation
    return beartype.beartype(f)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<@beartype(pathway.internals.run.run) at 0x16df04ae0>", line 129, in run
  File "/Users/techmonk/Documents/Hustle/Velora_ai/veloraapp-main/services/venv/lib/python3.11/site-packages/pathway/internals/run.py", line 47, in run
    ).run_outputs()
      ^^^^^^^^^^^^^
  File "/Users/techmonk/Documents/Hustle/Velora_ai/veloraapp-main/services/venv/lib/python3.11/site-packages/pathway/internals/graph_runner/__init__.py", line 107, in run_outputs
    self.run_nodes(self._graph.global_scope.output_nodes, after_build=after_build)
  File "/Users/techmonk/Documents/Hustle/Velora_ai/veloraapp-main/services/venv/lib/python3.11/site-packages/pathway/internals/graph_runner/__init__.py", line 83, in run_nodes
    self._run(all_nodes, after_build=after_build)
  File "/Users/techmonk/Documents/Hustle/Velora_ai/veloraapp-main/services/venv/lib/python3.11/site-packages/pathway/internals/graph_runner/__init__.py", line 203, in _run
    raise error from None
  File "/Users/techmonk/Documents/Hustle/Velora_ai/veloraapp-main/services/github_issues.py", line 85, in process_github_issue
    return extract_info_from_github_issues(issue)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/techmonk/Documents/Hustle/Velora_ai/veloraapp-main/services/github_issues.py", line 63, in extract_info_from_github_issues
    category, feature, summary = categorize_and_summarize_github_issue(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/techmonk/Documents/Hustle/Velora_ai/veloraapp-main/services/github_issues.py", line 33, in categorize_and_summarize_github_issue
    if keyword in issue_title or keyword in issue_body:
                                 ^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable
Occurred here:
    Line: issues_info = issues_table.select(data=pw.apply(process_github_issue, pw.this.data))
    File: /Users/techmonk/Documents/Hustle/Velora_ai/veloraapp-main/services/github_issues.py:95

zxqfd555-pw · 2024-03-22T10:23:07Z

Hi @umarbasha007

I am not sure it's related to the Pathway<->Airbyte connector. But, as I can see, you have a variable issue_body in your code that was set to None. Then, when you do a condition: keyword in issue_body, it fails, because NoneType is indeed non-iterable. So, code-wise, I would recommend checking that issue_body is None. You can replace it with an empty string if set to None. Another piece of advice here is to log such incorrect entries and see how they look in the Github interface.

The main point to consider is that we don't control the data that comes from any of the airbyte connectors, so it can be a good idea to refer to the connector docs and the format, for example, here. You can navigate by the link for "Issues" and check the schema for the fields you're interested at. That is, for body fields, the part of the schema looks as follows:

      "body": {
        "description": "Contents of the issue",
        "type": [
          "string",
          "null"
        ],

So, null is indeed a possible variant.

dxtrous · 2024-03-29T01:52:29Z

Hi @umarbasha007, have you been able to verify/resolve the issue on your side? Do any further problems persist, or should we close the issue?

dxtrous · 2024-05-09T15:13:01Z

@umarbasha007 we will be closing this issue as resolved on May 13 unless we hear from you by then.

vikassinghvi2007 added the bug Something isn't working label Mar 15, 2024

dxtrous assigned zxqfd555-pw Mar 15, 2024

dxtrous added confirmed Problem or bug confirmed priority:high Triaged: high priority issue labels Mar 15, 2024

dxtrous mentioned this issue Mar 15, 2024

Document how to run Airbyte connectors (via Airbyte Serverless) with Pathway inside Docker #14

Open

zxqfd555-pw mentioned this issue Mar 16, 2024

The catalog file is not written in full by the time the connector starts unytics/airbyte_serverless#5

Closed

dxtrous closed this as completed May 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: RuntimeError: exception in Python subject: KeyError: 'data' #13

[Bug]: RuntimeError: exception in Python subject: KeyError: 'data' #13

vikassinghvi2007 commented Mar 15, 2024

zxqfd555-pw commented Mar 15, 2024

vikassinghvi2007 commented Mar 15, 2024 via email

zxqfd555-pw commented Mar 15, 2024

vikassinghvi2007 commented Mar 16, 2024 via email

vikassinghvi2007 commented Mar 16, 2024

zxqfd555-pw commented Mar 16, 2024 •

edited

vikassinghvi2007 commented Mar 16, 2024 via email

zxqfd555-pw commented Mar 18, 2024

vikassinghvi2007 commented Mar 19, 2024 via email

vikassinghvi2007 commented Mar 19, 2024

zxqfd555-pw commented Mar 19, 2024

vikassinghvi2007 commented Mar 20, 2024

zxqfd555-pw commented Mar 20, 2024

umarbasha007 commented Mar 21, 2024

zxqfd555-pw commented Mar 22, 2024

dxtrous commented Mar 29, 2024

dxtrous commented May 9, 2024

[Bug]: RuntimeError: exception in Python subject: KeyError: 'data' #13

[Bug]: RuntimeError: exception in Python subject: KeyError: 'data' #13

Comments

vikassinghvi2007 commented Mar 15, 2024

Steps to reproduce

Relevant log output

What did you expect to happen?

Version

Docker Versions (if used)

OS

On which CPU architecture did you run Pathway?

zxqfd555-pw commented Mar 15, 2024

vikassinghvi2007 commented Mar 15, 2024 via email

zxqfd555-pw commented Mar 15, 2024

vikassinghvi2007 commented Mar 16, 2024 via email

vikassinghvi2007 commented Mar 16, 2024

zxqfd555-pw commented Mar 16, 2024 • edited

vikassinghvi2007 commented Mar 16, 2024 via email

zxqfd555-pw commented Mar 18, 2024

vikassinghvi2007 commented Mar 19, 2024 via email

vikassinghvi2007 commented Mar 19, 2024

zxqfd555-pw commented Mar 19, 2024

vikassinghvi2007 commented Mar 20, 2024

zxqfd555-pw commented Mar 20, 2024

umarbasha007 commented Mar 21, 2024

zxqfd555-pw commented Mar 22, 2024

dxtrous commented Mar 29, 2024

dxtrous commented May 9, 2024

zxqfd555-pw commented Mar 16, 2024 •

edited