'ng scalafmt --stdin' throws exception 'java.io.IOException: Unknown stream type: H' #1115

hacker-works · 2018-02-27T10:24:14Z

Version: scalafmt 1.4.0, vim 8, nailgun 0.9.3
Integration: Vim 8, Chiel92/vim-autoformat, nailgun 0.9.3
Configuration:

I'm using scalafmt through nailgun:

" Enable vim-autoformat for Scala
function! StartNailgunScalafmt()
    silent execute '!scalafmt_ng 2>/dev/null 1>/dev/null &'
    sleep 1
    silent execute '!ng ng-alias scalafmt org.scalafmt.cli.Cli'
    execute(':redraw!')
endfunction

call StartNailgunScalafmt()

noremap <F5> :Autoformat<CR>
" let g:autoformat_verbosemode=1
let g:formatdef_scalafmt = "'ng scalafmt --stdin'"
let g:formatters_scala = ['scalafmt']
let g:formatters_sbt = ['scalafmt']

Steps

Given code like this:

import $ivy.{
  `org.apache.spark::spark-core:2.2.1`,
  `org.apache.spark::spark-sql:2.2.1`
}
import ammonite.ops._
import org.apache.spark.{SparkContext, SparkConf}
import org.apache.spark.sql.SQLContext

object TestSpark {
  def test(): Unit = {
    val conf = new SparkConf().setMaster("local[*]").setAppName("spike")
    val sc = new SparkContext(conf)

    sc.setLogLevel("WARN")

    sc.textfile("/Users/vladi/Work/pioneers-core/docs/README.md")
    val matchingLineAndLineNumberTuples = textFile.zipWithIndex().filter({ case (line, lineNumber) => line.toLowerCase.contains("scala") }).collect
    matchingLineAndLineNumberTuples.sortBy(_._2).foreach {
      case (l, i) => println(s"LINE ${i}:\t${l}")
    }
  }
}

When I run scalafmt like this:

cat spark.scala | scalafmt --stdin

The output on stdin is just as expected:

import $ivy.{
  `org.apache.spark::spark-core:2.2.1`,
  `org.apache.spark::spark-sql:2.2.1`
}
import ammonite.ops._
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.sql.SQLContext

object TestSpark {
  def test(): Unit = {
    val conf = new SparkConf().setMaster("local[*]").setAppName("spike")
    val sc = new SparkContext(conf)

    sc.setLogLevel("WARN")

    sc.textfile("/Users/vladi/Work/pioneers-core/docs/README.md")
    val matchingLineAndLineNumberTuples = textFile
      .zipWithIndex()
      .filter({ case (line, lineNumber) => line.toLowerCase.contains("scala") })
      .collect
    matchingLineAndLineNumberTuples.sortBy(_._2).foreach {
      case (l, i) => println(s"LINE ${i}:\t${l}")
    }
  }
}

But when I run scalafmt with nailgun:

scalafmt_ng 2>/dev/null 1>/dev/null &
[1] 58892

ng ng-alias scalafmt org.scalafmt.cli.Cli
[1]+  Done                    scalafmt_ng 2> /dev/null > /dev/null

cat spark.scala | ng scalafmt --stdin

Problem

I'm getting an exception that I didn't got before (my setup used to work just fine a week ago):

java.io.IOException: Unknown stream type: H
	at com.martiansoftware.nailgun.NGInputStream.readHeader(NGInputStream.java:73)
	at com.martiansoftware.nailgun.NGInputStream.read(NGInputStream.java:120)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
	at java.io.InputStreamReader.read(InputStreamReader.java:184)
	at java.io.BufferedReader.fill(BufferedReader.java:161)
	at java.io.BufferedReader.readLine(BufferedReader.java:324)
	at java.io.BufferedReader.readLine(BufferedReader.java:389)
	at scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:70)
	at scala.collection.Iterator.foreach(Iterator.scala:929)
	at scala.collection.Iterator.foreach$(Iterator.scala:929)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1417)
	at scala.collection.TraversableOnce.addString(TraversableOnce.scala:357)
	at scala.collection.TraversableOnce.addString$(TraversableOnce.scala:353)
	at scala.collection.AbstractIterator.addString(Iterator.scala:1417)
	at scala.collection.TraversableOnce.mkString(TraversableOnce.scala:323)
	at scala.collection.TraversableOnce.mkString$(TraversableOnce.scala:322)
	at scala.collection.AbstractIterator.mkString(Iterator.scala:1417)
	at scala.collection.TraversableOnce.mkString(TraversableOnce.scala:325)
	at scala.collection.TraversableOnce.mkString$(TraversableOnce.scala:325)
	at scala.collection.AbstractIterator.mkString(Iterator.scala:1417)
	at org.scalafmt.cli.InputMethod$StdinCode$.apply(InputMethod.scala:26)
	at org.scalafmt.cli.Cli$.getInputMethods(Cli.scala:106)
	at org.scalafmt.cli.Cli$.run(Cli.scala:156)
	at org.scalafmt.cli.Cli$.mainWithOptions(Cli.scala:66)
	at org.scalafmt.cli.Cli$.nailMain(Cli.scala:32)
	at org.scalafmt.cli.Cli.nailMain(Cli.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.martiansoftware.nailgun.NGSession.run(NGSession.java:280)

Expectation

My expectation is that instead of exception it'll output the formatted file.

Workaround

On the other hand ng scalafmt spark.scala formats the file as expected.

Notes

I have looked at the nailgun code indicated in the exception. This code is found only in nailgun 0.9.1. Mac OS X brew is installing only nailgun 0.9.3

Currently, Scalafmt is using nailgun 0.9.1 as a build dependency:

scalafmt/build.sbt

Line 61 in 3df7c03

"com.martiansoftware" % "nailgun-server" % "0.9.1",

The text was updated successfully, but these errors were encountered:

olafurpg · 2018-02-27T10:55:37Z

What happens if you include --no-stderr? I have not used the nailgun integration myself but there was a related discussion in the vim-autoformat issue tracker iirc

hacker-works · 2018-02-27T11:28:32Z

cat spark.scala | ng scalafmt --stdin --no-stderr
java.io.IOException: Unknown stream type: H
	at com.martiansoftware.nailgun.NGInputStream.readHeader(NGInputStream.java:73)
	at com.martiansoftware.nailgun.NGInputStream.read(NGInputStream.java:120)
...

Throws the same exception.

olafurpg · 2018-02-27T11:30:50Z

Anything in vim-autoformat/vim-autoformat#184 that could help?

hacker-works · 2018-02-27T11:38:29Z

@olafurpg: I have basically followed the instructions there... and it used to work just fine. But a week ago I have updated the Mac OS X with the recommended updates and in addition I ran brew upgrade

olafurpg · 2018-02-27T11:45:00Z

Can you try upgrading the nailgun dependency in the repo and cli/assembly to test if that helps?

…

On Tue, 27 Feb 2018 at 12:38, hacker-works ***@***.***> wrote: @olafurpg <https://github.com/olafurpg>: I have basically followed the instructions there... and it used to work just fine. But a week ago I have updated the Mac OS X with the recommended updates and in addition I ran brew upgrade — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1115 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABV8XcLHbci5FbJlVwi1NzdL82Y3UVssks5tY-k2gaJpZM4SUr_O> .

hacker-works · 2018-02-27T12:26:14Z

Maven repos provide only nailgun 0.9.1. So to build with 0.9.3 we need to build it locally and install it in the local Maven repository. :/

hacker-works · 2018-02-27T12:45:46Z

OK, so what I did was:

git clone git@github.com:facebook/nailgun.git
cd nailgun
mvn clean install

Now I have 0.9.3-SNAPSHOT in my local maven repository.

I've changed in the build.sbt to be able to use dependencies from maven local:

resolvers ++= Seq(Resolver.mavenLocal, Resolver.sonatypeRepo("releases"))

sbt compile passes without errors.

@olafurpg: So, how should I test the build?

hacker-works · 2018-02-27T14:57:57Z

Summary of what I've tried so far:

My changes to build.sbt:

line 13: resolvers ++= Seq(Resolver.mavenLocal, Resolver.sonatypeRepo("releases")),

line 61: "com.martiansoftware" % "nailgun-server" % "0.9.3-SNAPSHOT",

I have built and installed locally scalafmt-cli by running sbt cli/publishLocal

Then I have reinstalled scalafmt_ng with:

sudo coursier bootstrap --standalone com.geirsson:scalafmt-cli_2.12:1.4.0 \
    -r ivy2local \
    -o /usr/local/bin/scalafmt_ng -f --main com.martiansoftware.nailgun.NGServer

And I've restarted nailgun:

scalafmt_ng 2>/dev/null 1>/dev/null &
[1] 70424

ng ng-alias scalafmt org.scalafmt.cli.Cli

And then tested it again:

cat spark.scala | ng scalafmt --stdin
java.io.IOException: Unknown stream type: H
	at com.martiansoftware.nailgun.NGInputStream.readHeader(NGInputStream.java:73)
	at com.martiansoftware.nailgun.NGInputStream.read(NGInputStream.java:120)
        ...

So, seems changing the nailgun version to 0.9.3 is not fixing the issue.

NOTE: What is strange to me is that readHeader is no longer in the source of NGInputStream.java in nailgun 0.9.3 and still it is in the stack trace...

olafurpg · 2018-02-27T17:42:35Z

What does ng --nailgun-version print for you? It's 0.9.0 for me. I would be surprised if there was a breaking change in the protocol between 0.9.1 and 0.9.3

olafurpg · 2018-02-27T17:49:58Z

Just tried following installation nailgun installation for scalafmt and I am able to reproduce the problem. Not sure what to do. @pjrt do you still use scalafmt through vim?

pjrt · 2018-02-27T19:50:19Z

I do use it through vim, but not as part of the vim-autoformat plugin (was having some issues with it, but I never investigated it). Currently I just run it as :!scalafmt -i -f %.

I never tried to get nailgun running, though this issue makes me want to try it..

hacker-works · 2018-02-27T20:52:32Z

ng --nailgun-version
NailGun client version 0.9.0

Same here!

olafurpg · 2018-02-28T08:29:40Z

@hacker-works What happens if you use http://search.maven.org/#artifactdetails%7Cch.epfl.scala%7Cnailgun-server%7C4ecd1a7d%7Cjar for the nailgun-server and this nailgun client https://github.com/facebook/nailgun/blob/master/pynailgun/ng.py ?

jonmeredith · 2018-03-08T20:59:04Z

FWIW, checking out tag nailgun-all-0.9.1 of nailgun, building and installing the ng client fixed it for me.

olafurpg · 2018-05-14T17:54:49Z

Any actionable item that be done to close this ticket? I personally don't use the nailgun integration, but I am able to reproduce the bug and it would be great to have working nailgun.

OTOH, I am optimistic we will have Scala Native support in the coming few months #1172

vladimir-tsvetkov · 2018-05-14T20:48:51Z

@olafurpg: I don't think there is.
I still have to find the time to reproduce @jonmeredith's suggestion, but intuition tells me it'll work this way.

Thank you for the help, everyone!

olafurpg · 2018-05-16T16:03:21Z

OK, I'll close this ticket then in favor of #1172. I am optimistic scala native support will avoid the need for nailgun altogether

If you clone the nailgun repository, it builds just fine. However, if you try to use this version of nailgun with scalafmt [1], it doesn't actually work [2]. In [2], they suggest building the nailgun-all-0.9.1 tag. But, this tag doesn't build because of javadoc errors. This commit fixes those errors so that the code builds again. [1]: https://scalameta.org/scalafmt/#Nailgun [2]: scalameta/scalafmt#1115

ahakanbaba · 2018-09-17T06:14:25Z

This is not a solution but another workaround.

Setting the following in vim-autoformat settings worked for me.

let g:formatdef_scalafmt = '"ng org.scalafmt.cli.Cli ".expand("%:p")." --stdout 2> /dev/null"'

Works without using the --stdin mode. Vim function called expand(..) returns the full path of the file and the --stdout prints the formatted code to the output.

Probably vim-autoformat pipes the file contents to the stdin of this call. It looks like that gets ignored without problems.

olafurpg · 2018-09-17T07:59:00Z

For people using the nailgun integration, I recommend trying out the graalvm native image in #1266 Please report back how it works.

olafurpg closed this as completed May 16, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

'ng scalafmt --stdin' throws exception 'java.io.IOException: Unknown stream type: H' #1115

'ng scalafmt --stdin' throws exception 'java.io.IOException: Unknown stream type: H' #1115

hacker-works commented Feb 27, 2018 •

edited

olafurpg commented Feb 27, 2018

hacker-works commented Feb 27, 2018

olafurpg commented Feb 27, 2018

hacker-works commented Feb 27, 2018

olafurpg commented Feb 27, 2018 via email

hacker-works commented Feb 27, 2018

hacker-works commented Feb 27, 2018

hacker-works commented Feb 27, 2018 •

edited

olafurpg commented Feb 27, 2018

olafurpg commented Feb 27, 2018

pjrt commented Feb 27, 2018 •

edited

hacker-works commented Feb 27, 2018

olafurpg commented Feb 28, 2018

jonmeredith commented Mar 8, 2018

olafurpg commented May 14, 2018

vladimir-tsvetkov commented May 14, 2018

olafurpg commented May 16, 2018

ahakanbaba commented Sep 17, 2018

olafurpg commented Sep 17, 2018

'ng scalafmt --stdin' throws exception 'java.io.IOException: Unknown stream type: H' #1115

'ng scalafmt --stdin' throws exception 'java.io.IOException: Unknown stream type: H' #1115

Comments

hacker-works commented Feb 27, 2018 • edited

Steps

Problem

Expectation

Workaround

Notes

olafurpg commented Feb 27, 2018

hacker-works commented Feb 27, 2018

olafurpg commented Feb 27, 2018

hacker-works commented Feb 27, 2018

olafurpg commented Feb 27, 2018 via email

hacker-works commented Feb 27, 2018

hacker-works commented Feb 27, 2018

hacker-works commented Feb 27, 2018 • edited

olafurpg commented Feb 27, 2018

olafurpg commented Feb 27, 2018

pjrt commented Feb 27, 2018 • edited

hacker-works commented Feb 27, 2018

olafurpg commented Feb 28, 2018

jonmeredith commented Mar 8, 2018

olafurpg commented May 14, 2018

vladimir-tsvetkov commented May 14, 2018

olafurpg commented May 16, 2018

ahakanbaba commented Sep 17, 2018

olafurpg commented Sep 17, 2018

hacker-works commented Feb 27, 2018 •

edited

hacker-works commented Feb 27, 2018 •

edited

pjrt commented Feb 27, 2018 •

edited