Installing bower dependencies by URL from Github Enterprise stopped working after upgrade to 1.4.1

[samny]

This is how the dependencies section of my bower.json looks:

"dependencies": {
  "repo-name":"https://github.domain.com/user-name/repo-name.git"
}

Running bower install -V gives me this output:

bower repo-name#*   not-cached https://github.domain.com/user-name/repo-name.git#*
bower repo-name#*      resolve https://github.domain.com/user-name/repo-name.git#*
bower repo-name#*     checkout master
bower repo-name#* detect-smart-git Smart Git host detected: true

Then it just hangs indefinitely.

The output suggests it would be related to this in release notes for 1.4: Automatically detecting smart Git hosts (#1628).

I only have this problem when referencing a Github Enterprise repository, the same config works fine with ordinary Github. If I downgrade to Bower 1.3.12 it works again

[sheerun]

Hmm. You you provide some repository to test against, so we can fix it?

[sheerun]

Ping @nwinkler

[samccone]

👍 would love on a test case on this also, I kinda figured we would get caught on some weird edges cases like this

[samny]

Unfortunately I can't provide you with a repository since it's only available internally at the company where I work. So I understand it's difficult to test and verify... could that be part of the problem though?

[anwalkers]

I am also experiencing the same issues running against a private registry. We are using the private-bower registry. https://github.com/Hacklone/private-bower

[patrickkettner]

for what its worth, I am not able to repo this on my enterprise github on bower 1.4.1, running GitHub Enterprise Version 11.10.352 (you can find this out by hovering over the octocat at the bottom of the page)

The dude from huboard mentioned that github gives devs an instance of github enterprise to develop against. Perhaps we could ask them for a dev copy to debug?

[nwinkler]

Ah, dang. I was really hoping that this would work without any issue, but I didn't have a GitHub Enterprise instance to test.

Can you run the following and post the full output here (replace the URL with yours, feel free to anonymize the result):

GIT_CURL_VERBOSE=2 git ls-remote --heads https://github.domain.com/user-name/repo-name.git

The code checks for the following to be present in the output:

Content-Type: application/x-git-upload-pack-advertisement

All of the sources that I have found indicate that the presence of this value indicates that the server supports shallow cloning, i.e. is a smart Git host.

Maybe we could get a sampling from several people using GitHub Enterprise and see if there's a pattern.

[nwinkler]

@patrickkettner @anwalkers @samny It would be great if each of you could run that command and post the output here.

[nwinkler]

Sources for my approach:

• http://stackoverflow.com/questions/9270488/is-it-possible-to-detect-whether-a-http-git-remote-is-smart-or-dumb
• https://github.com/dimitri/el-get/pull/1921/files

[patrickkettner]

here ya go

[nwinkler]

Here's the output when running this against GitHub:

$ GIT_CURL_VERBOSE=2 git ls-remote --heads https://github.com/bower/bower.git
* Couldn't find host github.com in the .netrc file; using defaults
* Hostname was NOT found in DNS cache
*   Trying 192.30.252.129...
* Connected to github.com (192.30.252.129) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
* Server certificate: github.com
* Server certificate: DigiCert SHA2 Extended Validation Server CA
* Server certificate: DigiCert High Assurance EV Root CA
> GET /bower/bower.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/2.3.2
Host: github.com
Accept: */*
Accept-Encoding: gzip
Pragma: no-cache

< HTTP/1.1 200 OK
* Server GitHub Babel 2.0 is not blacklisted
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-advertisement
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< 
* Connection #0 to host github.com left intact
2f6d680b6c72cdcde66775856e7f29a847cfc03e    refs/heads/alexanderGugel-save-exact
46560219027ba482b665083260425fd5b6d6727a    refs/heads/analytics_fix
e0a204d8061fa3726f8bda6798f60ed1b1d3f447    refs/heads/bitbucket-resolver
447c2e3c0427b9abbd5663b0eee20f9ce12b0107    refs/heads/feature/deplinks
b15acd2fea72cefebdfdb292c0ff9baeb948f988    refs/heads/feature/shrinkwrap
d82dda543681d5684442d29193927dd30a100f78    refs/heads/feature/windows
182d92f9bd9a719cbe0aa525c1e7e1ec29874389    refs/heads/fix/windows-tests
a1287416d40f944d58cb8f2e774e2076bfffd32f    refs/heads/master
e548d8b1a55be095abea400a0356c8957d706f8f    refs/heads/next

You can clearly see the Content-Type in there. The same results are provided by BitBucket and our internal Atlassian Stash instance, all of which support shallow cloning.

[sheerun]

I've requested GitHub Enterprise instance from GitHub as @patrickkettner suggested.

[nwinkler]

Thanks @patrickkettner - that's interesting.

I can see that there are two requests made to your server. The first one returns Content-Type: text/plain (indicating a non-smart host), while the second request returns Content-Type: application/x-git-upload-pack-advertisement.

It looks like the check needs to be changed to test the first occurrence of Content-Type in the response.

[anwalkers]

Here you go. I change companyName, userName, and IP address to protect the innocent...

$ GIT_CURL_VERBOSE=2 git ls-remote --heads https://vsts.companyName.com/tfs/RND-INA/I
PL-Smart-Admin/_git/ipl-companyName-font/
* Couldn't find host vsts.companyName.com in the _netrc file; using defaults
*   Trying 10.0.0.1...
* Connected to vsts.companyName.com (10.0.0.1) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: C:\Program Files (x86)\Git/bin/curl-ca-bundle.crt
  CApath: none
* SSL connection using TLSv1.0 / AES256-SHA
* Server certificate:
*        subject: OU=Domain Control Validated; CN=*.companyName.com
*        start date: 2013-04-24 22:26:53 GMT
*        expire date: 2016-04-24 22:26:53 GMT
*        subjectAltName: vsts.companyName.com matched
*        issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=http://
certificates.godaddy.com/repository; CN=Go Daddy Secure Certification Authority;
 serialNumber=07969287
*        SSL certificate verify ok.
> GET /tfs/RND-INA/IPL-Smart-Admin/_git/ipl-companyName-font/info/refs?service=git-upl
oad-pack HTTP/1.1
User-Agent: git/1.9.5.msysgit.1
Host: vsts.companyName.com
Accept: */*
Accept-Encoding: gzip
Pragma: no-cache
< HTTP/1.1 401 Unauthorized
< Content-Type: text/html
< Server: Microsoft-IIS/8.5
< X-TFS-ProcessId: 7afeae7a-93e4-433d-947c-36bff7123d27
< X-FRAME-OPTIONS: SAMEORIGIN
< WWW-Authenticate: NTLM
< X-Powered-By: ASP.NET
< P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM
NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
< X-Content-Type-Options: nosniff
< Date: Wed, 01 Apr 2015 17:30:43 GMT
< Content-Length: 1293
<
* Connection #0 to host vsts.companyName.com left intact
* Couldn't find host vsts.companyName.com in the _netrc file; using defaults
* Found bundle for host vsts.companyName.com: 0x2135a18
* Hostname vsts.companyName.com was found in DNS cache
*   Trying 10.0.0.1...
* Connected to vsts.companyName.com (192.168.19.122) port 443 (#1)
* successfully set certificate verify locations:
*   CAfile: C:\Program Files (x86)\Git/bin/curl-ca-bundle.crt
  CApath: none
* SSL re-using session ID
* SSL connection using TLSv1.0 / AES256-SHA
* Server certificate:
*        subject: OU=Domain Control Validated; CN=*.companyName.com
*        start date: 2013-04-24 22:26:53 GMT
*        expire date: 2016-04-24 22:26:53 GMT
*        subjectAltName: vsts.companyName.com matched
*        issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=http://
certificates.godaddy.com/repository; CN=Go Daddy Secure Certification Authority;
 serialNumber=07969287
*        SSL certificate verify ok.
> GET /tfs/RND-INA/IPL-Smart-Admin/_git/ipl-companyName-font/info/refs?service=git-upl
oad-pack HTTP/1.1
User-Agent: git/1.9.5.msysgit.1
Host: vsts.companyName.com
Accept: */*
Accept-Encoding: gzip
Pragma: no-cache
< HTTP/1.1 401 Unauthorized
< Content-Type: text/html
< Server: Microsoft-IIS/8.5
< X-TFS-ProcessId: 7afeae7a-93e4-433d-947c-36bff7123d27
< X-FRAME-OPTIONS: SAMEORIGIN
< WWW-Authenticate: NTLM
< X-Powered-By: ASP.NET
< P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM
NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
< X-Content-Type-Options: nosniff
< Date: Wed, 01 Apr 2015 17:30:43 GMT
< Content-Length: 1293
<
* Ignoring the response-body
* Connection #1 to host vsts.companyName.com left intact
* Issue another request to this URL: 'https://vsts.companyName.com/tfs/RND-INA/IPL-Sma
rt-Admin/_git/ipl-companyName-font/info/refs?service=git-upload-pack'
* Couldn't find host vsts.companyName.com in the _netrc file; using defaults
* Found bundle for host vsts.companyName.com: 0x2135a18
* Re-using existing connection! (#1) with host vsts.companyName.com
* Connected to vsts.companyName.com (192.168.19.122) port 443 (#1)
* Server auth using NTLM with user 'userName'
> GET /tfs/RND-INA/IPL-Smart-Admin/_git/ipl-companyName-font/info/refs?service=git-upl
oad-pack HTTP/1.1
Authorization: NTLM TlRMTVNTUAABAAAAB4IIogAAAAAAAAAAAAAAAAAAAAAGA4AlAAAADw==
User-Agent: git/1.9.5.msysgit.1
Host: vsts.companyName.com
Accept: */*
Accept-Encoding: gzip
Pragma: no-cache
< HTTP/1.1 401 Unauthorized
< Content-Type: text/html; charset=us-ascii
< Server: Microsoft-HTTPAPI/2.0
< WWW-Authenticate: NTLM TlRMTVNTUAACAAAACgAKADgAAAAFgomihb30Slfo/n8AAAAAAAAAAJo
AmgBCAAAABgOAJQAAAA9JAFQAUgBPAE4AAgAKAEkAVABSAE8ATgABABoAUwBQAE8ALQBUAEYAUwAtAFc
ARgBFAC0AMgAEABIAaQB0AHIAbwBuAC4AYwBvAG0AAwAuAHMAcABvAC0AdABmAHMALQB3AGYAZQAtADI
ALgBpAHQAcgBvAG4ALgBjAG8AbQAFABIAaQB0AHIAbwBuAC4AYwBvAG0ABwAIAM7CIZWhbNABAAAAAA=
=
< Date: Wed, 01 Apr 2015 17:30:43 GMT
< Content-Length: 341
<
* Ignoring the response-body
* Connection #1 to host vsts.companyName.com left intact
* Issue another request to this URL: 'https://vsts.companyName.com/tfs/RND-INA/IPL-Sma
rt-Admin/_git/ipl-companyName-font/info/refs?service=git-upload-pack'
* Couldn't find host vsts.companyName.com in the _netrc file; using defaults
* Found bundle for host vsts.companyName.com: 0x2135a18
* Re-using existing connection! (#1) with host vsts.companyName.com
* Connected to vsts.companyName.com (192.168.19.122) port 443 (#1)
* Server auth using NTLM with user 'userName'
> GET /tfs/RND-INA/IPL-Smart-Admin/_git/ipl-companyName-font/info/refs?service=git-upl
oad-pack HTTP/1.1
Authorization: NTLM TlRMTVNTUAADAAAAGAAYAIQAAAAiASIBnAAAAAAAAABYAAAAEAAQAFgAAAAc
ABwAaAAAAAAAAAC+AQAABYKIogYDgCUAAAAPTtWHTbbU97ByBMEnWjzS1WEAbgB3AGEAbABrAGUAcgBT
AFAATwAtAEEATgBXAEEATABLAC0AVwA4ADEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAraTa445y8nCP
X6gZYnD9PQEBAAAAAAAAzsIhlaFs0AHSpMw2ni50GgAAAAACAAoASQBUAFIATwBOAAEAGgBTAFAATwAt
AFQARgBTAC0AVwBGAEUALQAyAAQAEgBpAHQAcgBvAG4ALgBjAG8AbQADAC4AcwBwAG8ALQB0AGYAcwAt
AHcAZgBlAC0AMgAuAGkAdAByAG8AbgAuAGMAbwBtAAUAEgBpAHQAcgBvAG4ALgBjAG8AbQAHAAgAzsIh
laFs0AEGAAQAAgAAAAgAMAAwAAAAAAAAAAAAAAAAMAAAn4gEnS/yikCLdcHGqciBE+tRbDb/jbwyMUng
akBJN+0KABAAAAAAAAAAAAAAAAAAAAAAAAkAAAAAAAAAAAAAAAAAAAA=
User-Agent: git/1.9.5.msysgit.1
Host: vsts.companyName.com
Accept: */*
Accept-Encoding: gzip
Pragma: no-cache
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Type: application/x-git-upload-pack-advertisement
< Server: Microsoft-IIS/8.5
< X-TFS-ProcessId: 7afeae7a-93e4-433d-947c-36bff7123d27
< X-FRAME-OPTIONS: SAMEORIGIN
< X-AspNet-Version: 4.0.30319
< Persistent-Auth: true
< X-Powered-By: ASP.NET
< P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM
NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
< X-Content-Type-Options: nosniff
< Date: Wed, 01 Apr 2015 17:30:43 GMT
< Content-Length: 481
<
* Connection #1 to host vsts.companyName.com left intact
3e4fae7dedb9a882849a8a76303a08234f4b435b        refs/heads/master

[anwalkers]

Just to check my insanity I did downgrade to 1.3.12 and it does work there...

[nwinkler]

Thanks for the data, @anwalkers - that's helpful.

It looks like it's really the first Content-Type that needs to be verified, as it's returning the smart content type in a later request.

Update: This is probably a wrong assumption.

[patrickkettner]

@anwalkers @samny what version of GHE are you running?

[samny]

I'm on Github Enterprise 11.10.348

[samny]

And this is my output from the script: https://gist.github.com/samny/2a189cc4012a0c14c6d8

[nwinkler]

Thanks, @samny

Unfortunately, this is not consistent with the other GitHub Enterprise responses. Your response shows the correct Content-Type.

[nwinkler]

It shows exactly the same response fields as the one from GitHub. I'm not sure how to solve this at the moment, it looks like the server returns the expected content type, but doesn't act like a smart Git host.

[nwinkler]

I've created a PR that includes the check for the first content type here: #1766.

But I'm not sure whether it covers all of the GitHub Enterprise use cases - it looks like it won't help @samny for example.

Could the content type be dependent on the server that's used for hosting the Git repo, and the kind of authentication that's used?

I can spend some more time on this tomorrow, but it would really help if we had access to a GitHub Enterprise instance, or could talk to someone at GitHub who understands the ins and outs of their GitHub Enterprise product and its support for shallow cloning.

[samny]

Thanks for getting on this so quickly, awesome work! As expected though, it doesn't seem to solve the issue for my Enterprise version (I tried installing your pull request @nwinkler).

[nwinkler]

Some further reading:

• https://github.com/git/git/blob/master/Documentation/technical/protocol-capabilities.txt
• http://git-scm.com/book/es/v2/Git-Internals-Transfer-Protocols

[nwinkler]

BTW: Did anyone of you report this issue to GitHub Enterprise support? What's their take on it? Is it recognized as an official bug from their side? I have tried to read through the GitHub Enterprise release notes, but haven't found anything on this topic there (it's not very detailed).

Since I don't have a support contract with GitHub Enterprise, can maybe one of you @samny @patrickkettner @anwalkers contact your support and ask them about the issue? I would like to understand why this seems to happen with some versions of GitHub Enterprise, but works fine for others.

[patrickkettner]

@sheerun said he had contacted them.

[nwinkler]

Yes, he contacted them about a test instance that we could use. I would still like to hear what they have to say about not supporting shallow clones, it sounds like a bug on their end to me. Every other Git hosting service seems to support it just fine.

[sheerun]

GitHub says "the bug regarding shallow clones has been fixed on the GitHub Enterprise 2.1-series.".

Also, they'll give me access to Github Enterprise, so yay :)

[nwinkler]

This means the following: Once we have established whether the server is a smart server in GitRemoteResolver, we would need to fire off an additional request to the api/v3 URL to verify whether we get a valid response. From there it's either not GitHub Enterprise (no valid response/404), or GitHub Enterprise. In the latter case, we would have to run one or more additional requests to ports 80, 443, 8443 and 8080 to determine the version of GitHub Enterprise.

That's a lot of additional requests.

Could one of the folks running GitHub Enterprise verify these suggestions against their instance:

• Open the api/v3 URL, e.g. https://github.domain.com/api/v3
• Try to connect to ports 80, 443, 8443 and 8080 to see whether the admin console is running there

Detecting this reliably would involve quite a bit of extra work, all for supporting an older, faulty version of a product. We might be better off with an on/off command line switch or configuration option...

[sheerun]

@nwinkler Maybe we could implement "retry" mode for cmd util:

When running git clone with --progress flag it outputs progress on stderr. We could implement auto-retrying on no-output-after-n-seconds scenario.

API could be something like:

return cmd('git', ['clone', '--progress', '--depth', 1, url], { timeout: 5 }).fail(function (e) {
  if (e.code == 'ETIMEOUT') {
    // set _noShallow for this host
    return cmd('git', ['clone', url]);
  } else {
    throw e;
  }
});

Maybe we should also consider configuration option, as timeout would occur every bower install.

Also, due to concurrency there would be multiple timeouts and retries per bundle install.

[nwinkler]

In the proposed timeout scenario, you would also have to do the following:

• Cache the failed hosts so they're not tried again - there is existing code for this in GitRemoteResolver, but it needs to be adjusted.
• Kill the stalled git clone process before reporting the failure.

I think either a configuration on/off switch or a blacklist/whitelist approach like in #1559 would help. Hopefully the need for this will go away over time, as more people update their GitHub Enterprise instance to a version that supports shallow cloning.

[samny]

@nwinkler I tried the requests you suggested, and it seems to work as expected. I get a JSON response on /api/v3/ and I get a response on ports 443/80, nothing on the other ports. So, I'm on Older GitHub Enterprise instance that doesn't support shallow cloning.

[nwinkler]

@samny Thanks for verifying. Now we have to decide whether it's worth implementing a complex fix/check for this...

[michaelwarren1106]

+1 for a fix.

this issue is messing with us at my work here also. We're going to be using 1.3.12 for now until this issue can get resolved, because we have little control over our Github Enterprise version.

[SyntaxRules]

+1

Same issue and problem as @michaelwarren1106

[TalPasi]

+1

[alvaromb]

I'm experiencing the same issue, but instead GitHub enterprise, we're using a private-hosted RhodeCode server in our company.

This is what I'm doing to install a private repo:

$ bower install git+https://USER:PASSWORD@privaterhodecode.net/some-angularjs-private-module --save --verbose

This is the output I get:

It completely hangs after detect-smart-git. Running the GIT_CURL_VERBOSE command:

$ GIT_CURL_VERBOSE=2 git ls-remote --heads git+https://USER:PASSWORD@privaterhodecode.net/some-angularjs-private-module

I get this output.

We're planning to switch from RhodeCode to another solution, but we're facing an enormous amount of work and it's not going to happen soon.

[nwinkler]

Looks like a bug with RhodeCode to me. The server is advertising that it supports smart Git features, i.e. it will allow shallow cloning, but in reality it does not, hence the stalled download.

The same issue is/was present in earlier versions of GitHub Enterprise.

@alvaromb Can you check with the RhodeCode project whether they support shallow cloning? It looks like a bug on their end. Or you could try cloning one of the projects you host on this server using git clone --depth 1 https://... yourself.

[alvaromb]

This is the output I'm getting while cloning with --depth 1 @nwinkler:

Cloning into 'angular-xxxxxxxx-api'...
error: RPC failed; result=22, HTTP code = 417
fatal: The remote end hung up unexpectedly

[nwinkler]

Thanks for verifying! This looks like an issue in RhodeCode. If they are advertising to support shallow cloning (through the application/x-git-upload-pack-advertisement content type), the server shouldn't fail with this error...

[nwinkler]

One more question: Do you get this error immediately, or does it take a while after starting the git clone --depth 1?

[alvaromb]

I get the error 2 or 3 seconds after the clone.

[nwinkler]

Some links I found to give more detail on this:

• https://rhodecode.tenderapp.com/help/discussions/problems/7628-git-shallow-clone-not-working
• Using --depth 1 breaks clones from RhodeCode 1.7 servers #744

[alvaromb]

Thanks @nwinkler! The thing is that we're hosting an old version of RhodeCode (1.7.2) and we cannot update it because it's no longer open source. The RhodeCode open version, Kallithea, may have solved the issue, but I think we'll move to another SCM.

[nwinkler]

Ah, right - that would explain the presence of this error in your environment.

[sheerun]

In bower 1.5 it's possible to create custom git resolver that supports shallow cloning by default. To maintain backward compatibility I'm putting this feature behind a flag in Bower's default resolvers.

[nwinkler]

Thanks! From looking at the code, it looks like the default behavior is to not use shallow cloning. You have to specify the hosts that support shallow cloning in the global config file, right?

The irony is that I had implemented that same approach about a year ago in #1559 and it was rejected because it was adding another configuration value. Now we're back to this.

Well, whatever...

[nwinkler]

@sheerun Have you updated the documentation with the new key? I'll have to share this info with my team so we can list our private Git repo in the config file.

[sheerun]

Hey @nwinkler, sorry about that.. It would be perfect if your solution worked for all hosts, but bower has huge user base and people keep complaining. I'm thinking about introducing --beta flag for Bower for potentially breaking changes, and make it default in 2.0.0. Or maybe adding some popular hosts to whitelist.

I've documented new configuration key in http://bower.io/docs/config/#shallow-clone-hosts

[nwinkler]

@sheerun Thanks - no hard feelings 😄 I understand the concerns with the user base.

The problem with whitelisting (or blacklisting) is that you can't possibly account for the myriad of privately hosted corporate Git repos. Someone will have to bite the bullet and either whitelist or blacklist their instance...