Detecting GCE sometimes takes forever.

[craigcitro]

on some networks (@dhermes has one at home), doing a lookup on a bogus URL takes ~20-30s. this means that the default way of discovering the current environment (say for fetching the application default credential) can be insanely slow.

[craigcitro]

this was originally misdiagnosed in #92

[dhermes]

Now running gcloud auth and other commands I realize it has always stalled for a really long time on my machines. It seems this could be the cause there as well.

[craigcitro]

we definitely need to fix this, but there's no clear answer; deferring to @anthmgoogle to decide how to do this consistently for usable auth libraries.

[dhermes]

FYI this didn't actually fix the issue, I think the slowdown is something other than timeout (DNS related).

#100 will be fine I guess?

[craigcitro]

😢 #100 won't help you when you still need to fall through to 3LO, i.e. all the time.

just for kicks, can you try installing requests and doing the same thing? does it have the smarts to avoid this? (i'm guessing "no", but want to confirm.)

[craigcitro]

or urllib for that matter.

[dhermes]

In [3]: %paste
start = time.time()
try:
    requests.get('http://github.com/', timeout=0.001)
except:
    pass
print time.time() - start
## -- End pasted text --
0.0234730243683

In [4]: %paste
start = time.time()
try:
    requests.get('http://metadata.google.internal/', timeout=0.001)
except:
    pass
print time.time() - start
## -- End pasted text --
20.0183758736

I'm trying to read about ways to avoid this but not having much luck.

Seemed promising, but meh: http://askubuntu.com/questions/272358/extrememly-slow-dns-lookup

[craigcitro]

yeah, there's no way around doing some refactoring and just doing this check at the veeeery last minute. that's also what the AWS SDK does, it seems.

in the short term, here's a tweak to fix it for now ... thoughts?

[dhermes]

It seems very hacky. You should keep trying to send this up the chain.

There should be either

• a dependable and fast way to check if running in GCE
• no emphasis on code which can guess which environment it's in

[erwanor]

Just a suggestion but if it is a network dependent issue then maybe using custom DNS servers (8.8.8.8 for example) would solve the problem. The "technique" looks well documented and not too difficult to implement (http://stackoverflow.com/questions/2236498/tell-urllib2-to-use-custom-dns).

[dhermes]

@aaronwinter Thanks for the suggest. AFAICT using a custom nameserver depends on some system internals like dig or host which may not be on every machine with a valid Python install.

Maybe the source of dnspython will reveal something.

[erwanor]

I am not too sure if you are comfortable with using a different API but we can swap out the resolver trivially with twisted for example. Let me know if it is ok and I will implement a solution using it asap.

[craigcitro]

@aaronwinter in this case, we know what IP the hostname resolves to, so i think we don't want to go the twisted/dnspython route.

i think the best solution on the table is a three-pronged attack:

• refactor the code a bit so that we can avoid the GCE check in more cases,
• possibly switch to using the IP with a short timeout, and
• add an env var that says "i'm definitely not on gce, don't bother checking".

[craigcitro]

The first step (switch to the IP for the metadata service) is bf9d46e.

[erwanor]

Ok, thanks for the directions! Do you want the env var to be set to ~"no need to check" once a negative check has been done or something that is by default to "not check gce" and that the user will need to modify to enable gce checking?

[dhermes]

@craigcitro Care to close this out?

[craigcitro]

@dhermes i figured we'd leave it until all the steps above were done, but maybe we should track those separately ...

[dhermes]

Which steps remain? You referring to #111?

I like to use check-marks to track a plan of work in progress (see googleapis/google-cloud-python#451 for example).

[erwanor]

@dhermes I think @craigcitro is referring to this one:

refactor the code a bit so that we can avoid the GCE check in more cases,

For future reference:

• refactor the code a bit so that we can avoid the GCE check in more cases,
• possibly switch to using the IP with a short timeout
• add an env var that says "i'm definitely not on gce, don't bother checking".

[theacodes]

Thank you for creating this issue, however, this project is deprecatedand we will only be addressing critical security issues. You can read moreabout this deprecation here.

If you need support or help using this library, we recommend that you ask yourquestion on StackOverflow.

If you still think this issue is relevant and should be addressed, pleasecomment and let us know!