Skip to content
This repository has been archived by the owner on May 17, 2019. It is now read-only.

Mirroring treasuredata fails #235

Open
jmellor opened this issue Jul 19, 2016 · 9 comments
Open

Mirroring treasuredata fails #235

jmellor opened this issue Jul 19, 2016 · 9 comments

Comments

@jmellor
Copy link

jmellor commented Jul 19, 2016

Help!
I am unsuccessfully attempting to construct an internal mirror of the Ubuntu treasuredata repository using debmirror on a trusty machine that mirrors many other repositories, and suspecting that there is something wrong with the debian repository site.

The call to debmirror looks like:

debmirror --no-source --no-check-gpg --rsync-extra=none
-a amd64 -s contrib -h td-agent-package-browser.herokuapp.com
-d trusty -r /2/ubuntu/trusty
--debug --progress --nocleanup --method=https
/home/treasuredata-mirror

which results in the following error log:

$ sudo ./update-treasuredata-mirror
[sudo] password for esadmin:
Warning: --rsync-extra is not configured to mirror the trace files.
This configuration is not recommended.
Mirroring to /home/treasuredata-mirror from https://anonymous@td-agent-package-browser.herokuapp.com/2/ubuntu/trusty/
Arches: amd64
Dists: trusty
Sections: contrib
Pdiff mode: use
Not checking Release gpg signatures.
Will NOT clean up.
Attempting to get lock ...
Getting meta files ...
https://td-agent-package-browser.herokuapp.com/2/ubuntu/trusty/dists/trusty/Release => [ 0%] Getting: dists/trusty/Release... #* GET https://td-agent-package-browser.herokuapp.com/2/ubuntu/trusty/dists/trusty/Release ==> 200 OK
200 OK
ok
https://td-agent-package-browser.herokuapp.com/2/ubuntu/trusty/dists/trusty/Release.gpg => [ 0%] Getting: dists/trusty/Release.gpg... #
* GET https://td-agent-package-browser.herokuapp.com/2/ubuntu/trusty/dists/trusty/Release.gpg ==> 200 OK
200 OK
ok
Release file contains neither Codename nor Suite; using trusty.
/usr/bin/debmirror: Created directory: dists/trusty/contrib/binary-amd64/
/usr/bin/debmirror: Created directory: .temp/dists/trusty/contrib/binary-amd64/
dists/trusty/contrib/binary-amd64/Packages.gz does not exist locally or in Release, skipping.
Parsing Packages and Sources files ...
No packages after parsing Packages and Sources files!
WARNING: releasing 1 pending lock...

I've narrowed the problem down to:

$ curl https://td-agent-package-browser.herokuapp.com/2/ubuntu/trusty/dists/trusty/contrib/binary-amd64/Release
Unable to load bucket: packages.treasuredata.com

This looks like it might be some kind of proxy error of the server-side. Is it possible to identify what the problem is, and how to get around it? Should I be using some other mirror tool that gets around this?

@repeatedly
Copy link
Member

We don't have a experience with mirroring tool so we don't have a good answer for this issue...
Our repository is hosted on S3 and we use reprepro for creating repository.
Does S3 or reprepro have a problem with mirroring tool?

@jmellor
Copy link
Author

jmellor commented Jul 21, 2016

I think that the repository layout is incorrect. If I look at the distribution website, I can see:

packages.treasuredata.com/2/Ubuntu/trusty/dists/contrib

which is not correct. This layout means that there is prefix of 2/Ubuntu/trusty and a distribution called contrib and no component layer, so this is not a valid repository layout. The dists directory is currently at the wrong layer in the tree.

It should instead be:

packages.treasuredata.com/2/Ubuntu/dists/trusty/contrib

so that the distribution component would then be trusty and the component would be contrib.

Is it possible to fix this?

From: Masahiro Nakagawa [mailto:notifications@github.com]
Sent: July-20-16 03:48
To: fluent/fluentd-docs
Cc: John Mellor; Author
Subject: Re: [fluent/fluentd-docs] Mirroring treasuredata fails (#235)

We don't have a experience with mirroring tool so we don't have a good answer for this issue...
Our repository is hosted on S3 and we use reprepro for creating repository.
Does S3 or reprepro have a problem with mirroring tool?


You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/235#issuecomment-233870528, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKkBZaBRTO4Vm9_vw-O2aPoxNLsoqr9Iks5qXdKwgaJpZM4JPskY.

@jmellor
Copy link
Author

jmellor commented Jul 21, 2016

I believe that there is an error either in the S3 bucket configuration, or in the web service that it is running.

I have attached a screenshot showing a firefox query for the Release file, along with the error response that appears to show that the Amazon site is misconfigured.

Is the repository site not public, and therefore only works for yourself or something simple like that?

From: Masahiro Nakagawa [mailto:notifications@github.com]
Sent: July-20-16 03:48
To: fluent/fluentd-docs
Cc: John Mellor; Author
Subject: Re: [fluent/fluentd-docs] Mirroring treasuredata fails (#235)

We don't have a experience with mirroring tool so we don't have a good answer for this issue...
Our repository is hosted on S3 and we use reprepro for creating repository.
Does S3 or reprepro have a problem with mirroring tool?


You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/235#issuecomment-233870528, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKkBZaBRTO4Vm9_vw-O2aPoxNLsoqr9Iks5qXdKwgaJpZM4JPskY.

@jmellor
Copy link
Author

jmellor commented Jul 21, 2016

(sorry, it looks like the github mail service is losing images. Here is the error image:
amazon-error

@yuvraj2987
Copy link

I am seeing this issue for macOS agent as well.
image

@repeatedly
Copy link
Member

@yuvraj2987 Sorry, we accidentally revoke permission for this browser. We restore our setting and you now see the repository.

@silverbadge
Copy link

silverbadge commented Jan 9, 2018

Hi,

We use Artifactory to proxy/cache remote repositories. We're getting this the same issue:

Unable to load bucket: packages.treasuredata.com

Are you able to help with this?

@fujimotos
Copy link
Member

fujimotos commented Feb 19, 2018

I'm looking into this issue and poking around for few minutes.

Apparently, the web app (S3 Browser) is just a directory viewer and not designed to proxy files.
For actual contents of files, we need to access to the S3 endpoint directly.

In other words, we cannot retrieve a file from:

https://td-agent-package-browser.herokuapp.com/2/ubuntu/trusty/dists/trusty/contrib/binary-amd64/Release

Instead, we need to access to the following URL:

http://packages.treasuredata.com.s3.amazonaws.com/2/ubuntu/trusty/dists/trusty/contrib/binary-amd64/Release

This observation leads us to the following script (I modified the command posted by @jmellor):

debmirror --no-source --no-check-gpg --rsync-extra=none --progress --nocleanup \
          --method=https --disable-ssl-verification \
          -h packages.treasuredata.com.s3.amazonaws.com \
          -a amd64 -s contrib -d trusty -r /2/ubuntu/trusty \
          /path/to/designed/dir/

This should properly mirror the remote files to you local directory.

(Beware the "--disable-ssl-verification" option in the script, though. We need this opt
because Amazon does not provide us a proper SSL certificate for S3 endpoints; So
it is not fully secure)

@chicofranchico
Copy link

For those using Aptly, you can use the following URL e.g. (for xenial v3)

http://packages.treasuredata.com.s3.amazonaws.com/3/ubuntu/xenial/ xenial contrib

And then:

aptly mirror create -architectures=amd64 <mirror_name> http://packages.treasuredata.com.s3.amazonaws.com/3/ubuntu/xenial/ xenial contrib

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants