Skip to content
This repository has been archived by the owner on Jun 11, 2018. It is now read-only.

Boto instrumentation incorrectly assumes structure of URLs #198

Open
brett-lempereur opened this issue Feb 22, 2018 · 1 comment · May be fixed by #199
Open

Boto instrumentation incorrectly assumes structure of URLs #198

brett-lempereur opened this issue Feb 22, 2018 · 1 comment · May be fixed by #199

Comments

@brett-lempereur
Copy link

The boto OpBeat integration incorrectly makes some assumptions about the structure of URLs and fails when those assumptions don't hold. To replicate:

  1. Enable storages support for the Boto3 backend.
  2. Change the setting AWS_S3_ENDPOINT_URL to a hostname that does not match the service-region convention, more specifically one that does not contain a '.' character.
  3. Attempt to perform an action (e.g, deleting an object from an S3 bucket) on the bucket.

This will cause an error when unpacking the hostname string, shown in the traceback below.

The error was discovered while running unit tests in a docker-compose environment that simulates our production deployments, with a local instance of the minio S3 server.

======================================================================
ERROR: test_delete (--REDACTED--)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "--REDACTED--", line 297, in test_delete
    response = self.client.delete(url)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/test.py", line 323, in delete
    path, data=data, format=format, content_type=content_type, **extra)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/test.py", line 224, in delete
    return self.generic('DELETE', path, data, content_type, **extra)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/test.py", line 237, in generic
    method, path, data, content_type, secure, **extra)
  File "/usr/local/lib/python3.6/site-packages/django/test/client.py", line 404, in generic
    return self.request(**r)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/test.py", line 288, in request
    return super(APIClient, self).request(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/test.py", line 240, in request
    request = super(APIRequestFactory, self).request(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/test/client.py", line 485, in request
    raise exc_value
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 128, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.6/contextlib.py", line 52, in inner
    return func(*args, **kwds)
  File "/usr/local/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/viewsets.py", line 95, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/views.py", line 494, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/views.py", line 454, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/views.py", line 491, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/mixins.py", line 93, in destroy
    self.perform_destroy(instance)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/mixins.py", line 97, in perform_destroy
    instance.delete()
  File "/usr/local/lib/python3.6/site-packages/django/db/models/base.py", line 891, in delete
    return collector.delete()
  File "/usr/local/lib/python3.6/site-packages/django/db/models/deletion.py", line 307, in delete
    sender=model, instance=obj, using=self.using
  File "/usr/local/lib/python3.6/site-packages/django/dispatch/dispatcher.py", line 178, in send
    for receiver in self._live_receivers(sender)
  File "/usr/local/lib/python3.6/site-packages/django/dispatch/dispatcher.py", line 178, in <listcomp>
    for receiver in self._live_receivers(sender)
  File "/app/kubota/assets/storage.py", line 64, in delete_file
    instance.file.delete(save=False)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/fields/files.py", line 105, in delete
    self.storage.delete(self.name)
  File "/usr/local/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 471, in delete
    self.bucket.Object(self._encode_name(name)).delete()
  File "/usr/local/lib/python3.6/site-packages/boto3/resources/factory.py", line 520, in do_action
    response = action(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(**params)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 324, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.6/site-packages/opbeat/instrumentation/packages/base.py", line 63, in __call__
    args, kwargs)
  File "/usr/local/lib/python3.6/site-packages/opbeat/instrumentation/packages/base.py", line 222, in call_if_sampling
    return self.call(module, method, wrapped, instance, args, kwargs)
  File "/usr/local/lib/python3.6/site-packages/opbeat/instrumentation/packages/botocore.py", line 21, in call
    service, region, _ = parsed_url.hostname.split('.', 2)
ValueError: not enough values to unpack (expected 3, got 1)
beniwohli added a commit that referenced this issue Feb 22, 2018
@beniwohli
fix issue with handling nonstandard s3 endpoint URLs
ce1e02c 
closes #198
@beniwohli beniwohli linked a pull request Feb 22, 2018 that will close this issue
Open
@beniwohli
Copy link
Member

Thanks for the report @brett-lempereur! I don't have the setup to properly test this, but if I understand you correctly, #199 should fix this. Would you mind trying out that branch in your test suite?

beniwohli added a commit that referenced this issue Feb 22, 2018
@beniwohli
fix issue with handling nonstandard s3 endpoint URLs
c4bc1d3 
closes #198
beniwohli added a commit that referenced this issue Feb 22, 2018
@beniwohli
fix issue with handling nonstandard s3 endpoint URLs
6d1bd50 
closes #198
beniwohli added a commit that referenced this issue Feb 22, 2018
@beniwohli
fix issue with handling nonstandard s3 endpoint URLs
13c89e3 
closes #198
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix issue with handling nonstandard s3 endpoint URLs opbeat/opbeat_python
2 participants
@beniwohli @brett-lempereur