Info messages while pushing/fetching #106

juancampa · 2018-03-20T02:30:51Z

Is there a way to get the informational messages sent by the server while pushing/fetching? I'm talking about what is normally prefixed by remote: .... in regular command-line git.

The text was updated successfully, but these errors were encountered:

juancampa · 2018-03-20T17:56:26Z

This is what I'm getting from a successful push:

{
 "ok": [
  "unpack",
  "refs/heads/master"
 ]
}

But my git server is sending messages that are not shown here. It would be nice if the returned promise was also a readable stream for these messages. @wmhilton if you point me in the right direction I could help you get this done

billiegoose · 2018-03-21T03:37:49Z

OK so. I know I implemented this for pull... I'm not sure if I implemented it for push. (goes and looks at code) Nope didn't implement it for push yet.

For fetch, you need create an EventEmitter that listens to 'message' events and pass it as an argument to fetch (or clone). (I know that's kinda weird, but I tried a couple different ideas and this was the only one that still let me make fetch async function that returns when the operation is completed.)

Here's a snippet from __tests__/test-fetch.js to demonstrate:

...
    let emitter = new EventEmitter()
    emitter
      .on('message', output.push.bind(output))
      .on('progress', progress.push.bind(progress))
    // Test
    await fetch({
      fs,
      gitdir,
      emitter,
      depth: 1,
      singleBranch: true,
      remote: 'origin',
      ref: 'test-branch-shallow-clone'
    })
...

The 'message' events are the raw messages. The 'progress' events are a synthetic thing that I added in case you just want to use them for a progress bar.

Um... that should help you get started. I'll have to fix my code for "push" though so it also emits message events. Just out of curiosity, what messages is your git server sending on push that you're interested in?

juancampa · 2018-03-21T19:34:05Z

Thanks! I'll work on this when I get a moment

what messages is your git server sending on push that you're interested in?

My git server transpiles the code and install all dependencies using the server-side update hook. It's nice to see those messages as an indication of progress.

billiegoose · 2018-03-22T00:24:21Z

My git server transpiles the code and install all dependencies using the server-side update hook. It's nice to see those messages as an indication of progress.

Holy 🐄 that's awesome!! Is that something you wrote? That sounds like an awesome git server. I'm working on adding git server features to isomorphic-git, so it'll be possible to use it as an Express middleware to do stuff like that. I hadn't thought about git-hooks yet though. I guess that would be an interesting starting point for the API.

juancampa · 2018-03-22T00:42:12Z

Holy 🐄 that's awesome!! Is that something you wrote?

Yeah, thanks! it's for this sort of hosted OS that I'm working on. The repos you push are programs that run on your OS so they have the same structure. The git server rejects a push if the head ref of the master branch (a convention) can't be transpiled or doesn't have the right file structure. It works pretty nicely, I think Heroku works similarly, or it used to.

juancampa · 2018-03-22T18:42:43Z

Status update: spending some time today looking into this

juancampa · 2018-03-22T20:27:17Z

Hey @wmhilton. I misunderstood your first response and thought you were giving me instructions on how to implement it for push (as opposed to just using it), and I thought, hmm, looks simple enough, but of course, it's not :)

I'm still giving it a shot, I might be able to figure out how to implement the push version.

I added a test to server-only.test-push.js but I'm pretty sure it's not being run during npm run test. Is there something special I have to do? Also, is there a way to only run this test (as opposed to the whole suite) while implementing it?

billiegoose · 2018-03-23T02:52:30Z

Also, is there a way to only run this test (as opposed to the whole suite) while implementing it?

I guess I really do need a better "How to Contribute" section to the README. I never expected this many contributors before I hit 1.0! Heck, this repo's not even at 100 stars.

Short answer is: Yes! try jest --watch push. Long broad sort-of-helpful answer is check out the Jest documentation or just google "how to X with jest javascript". This is the first project I've used Jest with.

Even longer answer is there's ALSO Jasmine tests that run in karma. Because I didn't realize that Jest didn't run in the browser when I started writing tests.

All that being said......... lemme see if I can just knock this out for ya!

juancampa · 2018-03-23T02:56:24Z

All that being said......... lemme see if I can just knock this out for ya!

👏👏👏👏👏

billiegoose · 2018-03-23T03:42:51Z

I pushed a push-messages branch that I think adds support... but for some reason I'm not seeing any messages in the test scripts. I'm not sure if I should though, because I'm not sure what messages to look for. I might need your help making a test case for this.

juancampa · 2018-03-23T20:10:50Z

Hmm I'm not seeing any of the two events being fired. I wanna take a closer look but I have to leave for now. I'll run another test when I come back

billiegoose · 2018-03-23T23:57:43Z

To generate such messages, do I just use "echo" in a post-recieve git hook? (Is that what you're using?)

juancampa · 2018-03-24T00:02:46Z

Yeah that's exactly what I'm using

…

On Fri, Mar 23, 2018, 4:57 PM William Hilton ***@***.***> wrote: To generate such messages, do I just use "echo" in a post-recieve git hook? (Is that what you're using?) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#106 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABWF2J1PCpKUJkRzrWjDPIMIJgAr--Wgks5thYv3gaJpZM4SxM14> .

juancampa · 2018-03-24T00:03:25Z

Actually the hook is called "update"

…

On Fri, Mar 23, 2018, 5:02 PM Juan Campa ***@***.***> wrote: Yeah that's exactly what I'm using On Fri, Mar 23, 2018, 4:57 PM William Hilton ***@***.***> wrote: > To generate such messages, do I just use "echo" in a post-recieve git > hook? (Is that what you're using?) > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#106 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ABWF2J1PCpKUJkRzrWjDPIMIJgAr--Wgks5thYv3gaJpZM4SxM14> > . >

billiegoose · 2018-03-24T02:13:33Z

Ha! OK. I figured out how to do it without setting up apache. :) Now I will see if I can use this new knowledge to write a unit test using such a git remote.

billiegoose · 2018-03-25T02:21:10Z

I've now written my first karma plugin, https://npm.im/karma-git-http-server-middleware! It exposes the main directory so that tests can directly do git.clone on "http://localhost:9876/__tests__/__fixtures__/test-clone.git". Unfortunately I forgot I'd need a way to start and stop the git server for non-karma tests (yay isomorphic tests) so I still need to add that.

billiegoose · 2018-03-27T03:16:03Z

#113 is an attempt to add support for http-servable git repo fixtures. If it works, it should support creating new repo test fixtures complete with "update" and "post-receive" git hooks!

billiegoose · 2018-04-01T03:25:20Z

#116 changes the git response parser to a streaming parser. That's necessary for the git push messages to be streamed instead of returned all at once at the end. Much more useful to know what's going on and get some feedback during the push.

billiegoose · 2018-04-06T03:15:59Z

#128 adds emitting status messages during git push, but I still need to add real tests. I'm not sure if it works yet lol.

juancampa · 2018-04-06T03:38:36Z

Sorry I haven't been able to help you here. I've been working on other things. I'll try this out as soon as possible though

…

On Thu, Apr 5, 2018, 8:16 PM William Hilton ***@***.***> wrote: #128 <#128> adds emitting status messages during git push, but I still need to add real tests. I'm not sure if it works yet lol. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#106 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABWF2ASBMQjp5pXN5JEkw7uQQ9plTVBiks5tlt3vgaJpZM4SxM14> .

billiegoose · 2018-04-07T16:19:39Z

No worries @juancampa I think I've got everything in place now to add a unit test simulating your scenario. I've created a karma plugin that lets you push to a git repo, but when you start a push operation it makes temporary copy of the got directory and operates on that, leaving the original untouched. This way you can rerun the tests over and over (or have five browsers run the same test in parallel) and the target repository will be in the same state.

Quick question for ya though before I write the test for this: how much time are the messages sent back from your server? Are we measuring in seconds ("build succeeded!") or minutes ("build started... build completed... tests started... test 1 passed... test 2 passed... tests completed... starting server... server running")?

juancampa · 2018-04-07T17:17:27Z

That plugin sounds awesome. For my use case it's generally in the order 5s-20s depending on how much transpiling/bundling it has to do.

…

On Sat, Apr 7, 2018, 9:19 AM William Hilton ***@***.***> wrote: No worries @juancampa <https://github.com/juancampa> I think I've got everything in place now to add a unit test simulating your scenario. I've created a karma plugin that lets you push to a git repo, but when you start a push operation it makes temporary copy of the got directory and operates on that, leaving the original untouched. This way you can rerun the tests over and over (or have five browsers run the same test in parallel) and the target repository will be in the same state. Quick question for ya though before I write the test for this: how much time are the messages sent back from your server? Are we measuring in seconds ("build succeeded!") or minutes ("build started... build completed... tests started... test 1 passed... test 2 passed... tests completed... starting server... server running")? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#106 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABWF2KhCc7PieQP0w222o7rzIGRvcQYaks5tmOcbgaJpZM4SxM14> .

juancampa · 2018-04-07T19:25:54Z

I just tried to build the push-messages branch and failed with this message:

[build.sw] ERROR: Source map doesn't work with input from STDIN
[build.internalApis] ERROR: Source map doesn't work with input from STDIN
[build.umd] ERROR: Source map doesn't work with input from STDIN
[build.sw] The script called "build.sw" which runs "browserify            --entry dist/for-serviceworker/index.js            --standalone git            --debug | uglifyjs                      --compress                      --mangle                      --source-map "content=inline,url=service-workder-bundle.umd.min.js.map"                      -o dist/service-worker-bundle.umd.min.js" failed with exit code 1 https://github.com/kentcdodds/nps/blob/v5.9.0/other/ERRORS_AND_WARNINGS.md#failed-with-exit-code
[build.sw] nps build.sw exited with code 1
--> Sending SIGTERM to other processes..
[build.umd] nps build.umd exited with code null
[build.internalApis] nps build.internalApis exited with code null
The script called "build.browserify" which runs "node node_modules/concurrently/src/main.js --kill-others-on-fail --prefix-colors "bgBlue.bold,bgMagenta.bold,bgGreen.bold" --prefix "[{name}]" --names "build.sw,build.umd,build.internalApis" 'nps build.sw' 'nps build.umd' 'nps build.internalApis'" failed with exit code 1 https://github.com/kentcdodds/nps/blob/v5.9.0/other/ERRORS_AND_WARNINGS.md#failed-with-exit-code
The script called "build" which runs "nps build.rollup && nps build.browserify && nps build.indexjson" failed with exit code 1 https://github.com/kentcdodds/nps/blob/v5.9.0/other/ERRORS_AND_WARNINGS.md#failed-with-exit-code
error Command failed with exit code 1.

Any thoughts?

juancampa · 2018-04-07T19:28:13Z

False alarm, I was using yarn instead of npm to install dependencies. Probably had the wrong dependencies

juancampa · 2018-04-07T19:51:31Z

This doesn't seem like a false alarm though:

error TypeError: bufferstream.slice is not a function
    at read (models.js:629)
    at nextBit (managers.js:613)
    at GitRemoteHTTP.stream (managers.js:647)

Might be an error in mbostock's stream-source

Seems to happen in both Chrome and Firefox

juancampa · 2018-04-07T20:07:11Z

My guess is that the stream returned by simple-get looks like a WhatWG stream from the point of view of stream-source (i.e. it has a read function) but it's not fully compliant (i.e. it doesn't have a slice function)

Update: I'm getting this with v0.9.9 from npm as well. During pull.

billiegoose · 2018-04-07T23:54:31Z

Update: I'm getting this with v0.9.9 from npm as well. During pull.

Hmm. When it rains it pours. I'm not too worried you got errors trying to build the push-messages branch... I've been force-pushing to it multiple times today lol... but getting the error in the published version is sad.

error TypeError: bufferstream.slice is not a function
at read (models.js:629)
at nextBit (managers.js:613)
at GitRemoteHTTP.stream (managers.js:647)

Most likely it's not a problem in simple-get or in stream-source but rather something about how I'm using it. Maybe res is undefined or the stream is empty (no new commits?) or something like that. Let's open up a new issue and continue the discussion there, since this one is about to get closed when I merge #128 (Travis willing).

billiegoose · 2018-04-08T00:22:25Z

🎉 This issue has been resolved in version 0.10.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

billiegoose · 2018-04-08T00:29:20Z

Yay! I finally got it working and have tests to demonstrate it:
/__tests__/test-push.js
/__tests__/__fixtures__/test-push-server.git/hooks

Usage Example:

    let emitter = new EventEmitter()
      .on('message', msg => console.log('remote: ' + msg))
    // Test
    let res = await push({
      fs,
      gitdir,
      emitter,
      remote: 'karma',
      ref: 'refs/heads/master'
    })

juancampa · 2018-04-16T20:00:52Z

@wmhilton I tried testing this with 0.10 but I'm still getting the TypeError: bufferstream.slice is not a function :/ Any ideas? res doesn't seem to be undefined, it just doesn't have the slice function

billiegoose · 2018-04-17T00:43:14Z

Hmm. My tests must be doing something different than what your code is doing, because I haven't seen that error. I might have to see how you're calling isomorphic-git. Can you open an issue with a code example? 🐛

billiegoose · 2018-04-22T21:40:12Z

@juancampa Are you using Webpack?! I just found (and solved) a very similar problem, I think.

juancampa · 2018-04-22T22:26:45Z

I am! I'll try it out. Is it the latest on npm?

…

On Sun, Apr 22, 2018, 5:40 PM William Hilton ***@***.***> wrote: @juancampa <https://github.com/juancampa> Are you using Webpack?! I just found (and solved) a very similar problem, I think. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#106 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABWF2GMAukSgRt3jXkxDObqFHb0hukqPks5trPi9gaJpZM4SxM14> .

billiegoose · 2018-04-23T00:34:27Z

That could be it then! And no, it's still on a branch. It's taking me a while to get all the build tests passing and stuff.

billiegoose · 2018-04-23T04:14:20Z

I've got it working on everything except Safari. But the Safari bug is proving extremely tricky to hunt down. When I first converted it to webpack, I wasn't using Babel and was shocked that the build was nearly 100% smaller. So now I'm trying to minimize the amount of transpiling. I dropped support for the "Android" browser and added support for the "ChromeAndroid" browser, and then babel-preset-env thought that only two transforms were needed, both for Edge browser. But that has proven not to be the case... Safari needs just a little help... and I need to figure out which of the dozen or so transforms it needs.

juancampa · 2018-04-24T15:49:58Z

I thought those days were over after babel-preset-env came out but I guess it's not as foolproof as it's meant to be

juancampa · 2018-04-24T17:22:40Z

I just tried the webpack branch (assuming this is the one) but I'm still getting the same slice problem :(

juancampa · 2018-04-24T17:26:18Z

I take it back, wasn't building correctly. I'm doing more tests. Still trying

billiegoose · 2018-04-26T01:57:25Z

So what I had to do to get it to build was add an alias:

'stream-source': 'stream-source/index.node.js'

I would think that the compiled version would Just Work, but if people are going to be building it from source, maybe I should try a different fix. Like, maybe import stream-source/index.node.js directly?

billiegoose · 2018-04-26T15:09:47Z

OK try v0.11.0. I changed import from 'stream-source' to import from 'stream-source/index.node.js' so I think it should Just Work™️ for you now. If not, I'm going to open a new issue because it really bugs me that it's not working for you!

juancampa · 2018-04-26T20:04:28Z

Hey thanks again for being so helpful.

v0.11.0 is not working for me. Babel is complaining that it has object spread syntax which is not expecting in any node_modules file.

It seems like your .babelrc has transform-object-rest-spread which should do the trick but if you look at dist/for-future/isomorphic-git/index.js the spread operator is still there:

(line 454)

  parse () {
    return { message: this.message(), ...this.headers() }
  }

juancampa · 2018-04-26T20:14:24Z

The good news is that I manually passed dist/for-future/isomorphic-git/index.js through babel to remove the object spread stuff and it works!! I can see the messages coming in 👏👏🎉🎉

billiegoose · 2018-04-27T01:30:51Z

Wait... does Webpack or Babel or whatever use for-future by default? I thought it would be using bundle.umd.min.js... It's honestly called for-future for a reason lol... it's not transpiled for the present. I might have to tweak my package.json.

billiegoose · 2018-04-27T01:32:07Z

What version of Babel are you running? Maybe I need to make a note of that...

juancampa · 2018-04-27T14:18:19Z

I'm using next@5.1.0 which uses babel-core@6.26.0 and webpack@3.10.0

It doesn't seem to matter if I use import or require. Webpack still decides to use for-future. Maybe to enable tree-shaking?

juancampa · 2018-04-27T14:22:42Z

I found this comment by Dan Abramov webpack/webpack#1979 (comment)

The module entry point should point to code with ES6 modules but not necessarily other ES6 features. People who want their libraries to be in ESM format can/should still precompile other ES6 features so that older browsers don't break.

juancampa · 2018-04-27T14:33:11Z

Oh I know what's going on here. Next.js has server-side-rendering so it transpiles everything for both environments!

juancampa · 2018-04-27T15:56:34Z

I'm currently working around the issue by using webpack's resolve.alias setting.

server-side:
config.resolve.alias['isomorphic-git$'] = './empty.js';

client-side:
config.resolve.alias['isomorphic-git$'] = 'isomorphic-git/dist/bundle.umd.min.js';

The server-side one is good and I'm keeping it because I don't really use git on the server but it shouldn't be needed. I'm not sure however why the client-side is needed, webpack should be looking at your package.json's browser field but it seems to be looking at module instead (even when target is web)

billiegoose · 2018-04-28T04:21:52Z

Rest/Spread Properties are actually a Stage 4 proposal and will be officially part of ECMAScript 2018 in June. But I shall remove them from now, since they are causing this issue and are only used in a couple places.

…c-git#106) (isomorphic-git#128)

billiegoose added the enhancement label Mar 27, 2018

billiegoose added a commit that referenced this issue Apr 7, 2018

feat: Emit 'remote:' server messages during git push (close #106)

13097a3

billiegoose added a commit that referenced this issue Apr 7, 2018

feat: Emit 'remote:' server messages during git push (close #106)

b80e8e7

billiegoose added a commit that referenced this issue Apr 7, 2018

feat: Emit 'remote:' server messages during git push (close #106)

9248fcc

billiegoose closed this as completed in 0e1bc77 Apr 8, 2018

billiegoose mentioned this issue Apr 28, 2018

Make work in Next.js without hacks. #146

Closed

billiegoose mentioned this issue May 22, 2018

No messages events in push #214

Closed

modesty pushed a commit to modesty/isomorphic-git that referenced this issue Apr 9, 2024

feat: Emit 'remote:' server messages during git push (close isomorphi…

c06a730

…c-git#106) (isomorphic-git#128)

Info messages while pushing/fetching #106

Info messages while pushing/fetching #106

Comments

juancampa commented Mar 20, 2018

juancampa commented Mar 20, 2018

billiegoose commented Mar 21, 2018

juancampa commented Mar 21, 2018

billiegoose commented Mar 22, 2018

juancampa commented Mar 22, 2018

juancampa commented Mar 22, 2018

juancampa commented Mar 22, 2018 • edited

billiegoose commented Mar 23, 2018

juancampa commented Mar 23, 2018

billiegoose commented Mar 23, 2018

juancampa commented Mar 23, 2018

billiegoose commented Mar 23, 2018

juancampa commented Mar 24, 2018 via email

juancampa commented Mar 24, 2018 via email

billiegoose commented Mar 24, 2018

billiegoose commented Mar 25, 2018

billiegoose commented Mar 27, 2018

billiegoose commented Apr 1, 2018

billiegoose commented Apr 6, 2018

juancampa commented Apr 6, 2018 via email

billiegoose commented Apr 7, 2018

juancampa commented Apr 7, 2018 via email

juancampa commented Apr 7, 2018

juancampa commented Apr 7, 2018 • edited

juancampa commented Apr 7, 2018 • edited

juancampa commented Apr 7, 2018 • edited

billiegoose commented Apr 7, 2018

billiegoose commented Apr 8, 2018

billiegoose commented Apr 8, 2018

juancampa commented Apr 16, 2018 • edited

billiegoose commented Apr 17, 2018

billiegoose commented Apr 22, 2018

juancampa commented Apr 22, 2018 via email

billiegoose commented Apr 23, 2018

billiegoose commented Apr 23, 2018

juancampa commented Apr 24, 2018 • edited

juancampa commented Apr 24, 2018

juancampa commented Apr 24, 2018 • edited

billiegoose commented Apr 26, 2018

billiegoose commented Apr 26, 2018

juancampa commented Apr 26, 2018

juancampa commented Apr 26, 2018 • edited

billiegoose commented Apr 27, 2018

billiegoose commented Apr 27, 2018

juancampa commented Apr 27, 2018 • edited

juancampa commented Apr 27, 2018 • edited

juancampa commented Apr 27, 2018 • edited

juancampa commented Apr 27, 2018

billiegoose commented Apr 28, 2018

juancampa commented Mar 22, 2018 •

edited

juancampa commented Apr 7, 2018 •

edited

juancampa commented Apr 7, 2018 •

edited

juancampa commented Apr 7, 2018 •

edited

juancampa commented Apr 16, 2018 •

edited

juancampa commented Apr 24, 2018 •

edited

juancampa commented Apr 24, 2018 •

edited

juancampa commented Apr 26, 2018 •

edited

juancampa commented Apr 27, 2018 •

edited

juancampa commented Apr 27, 2018 •

edited

juancampa commented Apr 27, 2018 •

edited