Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

protoc-gen-grpc-java not available on apple m1 #7690

Open
reinmind opened this issue Dec 3, 2020 · 101 comments · Fixed by protocolbuffers/protobuf#8557 or #8680
Open

protoc-gen-grpc-java not available on apple m1 #7690

reinmind opened this issue Dec 3, 2020 · 101 comments · Fixed by protocolbuffers/protobuf#8557 or #8680

Comments

@reinmind
Copy link

reinmind commented Dec 3, 2020

What version of gRPC-Java are you using?

1.34.0

What is your environment?

osx 11.0.1

What did you expect to see?

build success

What did you see instead?

[INFO] --- protobuf-maven-plugin:0.6.1:compile (default-cli) @ grpc ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.720 s
[INFO] Finished at: 2020-12-03T16:37:02+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.xolstice.maven.plugins:protobuf-maven-plugin:0.6.1:compile (default-cli) on project grpc: Unable to resolve artifact: Missing:
[ERROR] ----------
[ERROR] 1) com.google.protobuf:protoc:exe:osx-aarch_64:3.12.0
[ERROR] 
[ERROR]   Try downloading the file manually from the project website.
[ERROR] 
[ERROR]   Then, install it using the command: 
[ERROR]       mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.12.0 -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=/path/to/file
[ERROR] 
[ERROR]   Alternatively, if you host your own repository you can deploy the file there: 
[ERROR]       mvn deploy:deploy-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.12.0 -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR] 
[ERROR]   Path to dependency: 
[ERROR]   	1) com.reinmind:grpc:jar:1.0-SNAPSHOT
[ERROR]   	2) com.google.protobuf:protoc:exe:osx-aarch_64:3.12.0
[ERROR] 
[ERROR] ----------
[ERROR] 1 required artifact is missing.
[ERROR] 
[ERROR] for artifact: 
[ERROR]   com.reinmind:grpc:jar:1.0-SNAPSHOT
[ERROR] 
[ERROR] from the specified remote repositories:
[ERROR]   aliyunmaven (https://maven.aliyun.com/repository/public, releases=true, snapshots=false)
[ERROR] 
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

Steps to reproduce the bug

mvn protobuf:compile

@ejona86
Copy link
Member

ejona86 commented Dec 3, 2020

Yes, we do not produce binaries for arm64 macs.


Edit: Summary of the long issue: The easiest thing to do as a consumer is to install Rosetta. With it installed, the Intel binary works fine. If you'd like to contribute, see #7690 (comment) . Easiest plan is to build protobuf and the grpc plugin as universal binaries.

@ejona86 ejona86 changed the title compile protobuf failed on apple m1 protoc-gen-grpc-java not available on apple m1 Dec 3, 2020
@ejona86
Copy link
Member

ejona86 commented Dec 3, 2020

Since this is the grpc repo, I've called out protoc-gen-grpc-java not being available. Although your actual error is protoc itself, which would be part of the protobuf project. If grpc gets to building it first, then we'd need to add support in protobuf as well.

@ejona86 ejona86 added this to the Next milestone Dec 3, 2020
@gaurav46
Copy link

gaurav46 commented Dec 17, 2020

Hi @ejona86
Is it possible for any workaround like to set osdetector.arch as x86_64 to get the complication done ?
I tried to set below in gradle.properties but it still looks for osx-aarch_64 binaries from maven.
org.gradle.jvmargs=-Dos.detected.name=osx -Dos.detected.arch=x86_64 -Dos.detected.classifier=osx-x86_64

@ejona86
Copy link
Member

ejona86 commented Dec 17, 2020

To manually specify the classifier, for Gradle:

protobuf {
  protoc {
    artifact = 'com.google.protobuf:protoc:3.14.0:osx-x86_64'
  }
}

I'm less familiar with the Maven plugin, but I think it may be something like:

<protocArtifact>com.google.protobuf:protoc:3.14.0:exe:osx-x86_64</protocArtifact>

You can do the same for protoc-gen-grpc-java.

@gaurav46
Copy link

Thanks @ejona86 . That worked out.

@zjc17
Copy link

zjc17 commented Jan 14, 2021

Thanks @ejona86 , But How can I keep other platform like linux-x86_64 when the project being compiled on other platform?

@ejona86
Copy link
Member

ejona86 commented Jan 14, 2021

@Jiachen-Zhang

protobuf {
  protoc {
    if (osdetector.os == "osx") {
      artifact = 'com.google.protobuf:protoc:3.14.0:osx-x86_64'
    } else {
      artifact = 'com.google.protobuf:protoc:3.14.0'
    }
  }
}

You can also use project properties to only change the behavior when explicitly requested.

@zjc17
Copy link

zjc17 commented Jan 15, 2021

@Jiachen-Zhang

protobuf {
  protoc {
    if (osdetector.os == "osx") {
      artifact = 'com.google.protobuf:protoc:3.14.0:osx-x86_64'
    } else {
      artifact = 'com.google.protobuf:protoc:3.14.0'
    }
  }
}

You can also use project properties to only change the behavior when explicitly requested.

Thanks!

Finally, I made it by

// for apple m1, please add protoc_platform=osx-x86_64 in $HOME/.gradle/gradle.properties
if (project.hasProperty('protoc_platform')) {
    artifact = "com.google.protobuf:protoc:3.13.0:${protoc_platform}"
} else {
    artifact = "com.google.protobuf:protoc:3.13.0"
}

@hu-chia
Copy link

hu-chia commented Feb 3, 2021

for me, add this to ~/.m2/settings.xml, it worked:

<settings>
  ...
  <activeProfiles>
    <activeProfile>
      apple-silicon
    </activeProfile>
    ...
  </activeProfiles>
  <profiles>
    <profile>
      <id>apple-silicon</id>
      <properties>
        <os.detected.classifier>osx-x86_64</os.detected.classifier>
      </properties>
    </profile>
    ...
  </profiles>
  ...
</settings>

@Uditmittal
Copy link

Can you share the full setting.xml
I tried it didn't work for me?
I am using M1 with protobuf.

@zjc17
Copy link

zjc17 commented Feb 23, 2021

@Uditmittal
in my build.gradle file

protobuf {
    protoc {
        // for apple m1, please add protoc_platform=osx-x86_64 in $HOME/.gradle/gradle.properties
        if (project.hasProperty('protoc_platform')) {
            artifact = "com.google.protobuf:protoc:3.13.0:${protoc_platform}"
        } else {
            artifact = "com.google.protobuf:protoc:3.13.0"
        }
    }
}

in my ~/.gradle/gradle.properties file

protoc_platform=osx-x86_64

I think you can define and use variables in maven to implement this feature. Good luck

@hu-chia
Copy link

hu-chia commented Feb 24, 2021

@Uditmittal My settings.xml is awfully long, the key is using your property overwrite kr.motd.maven:os-maven-plugin detected os classifier, if you share your settings.xml and pom.xml snippet, this may be help.

@Uditmittal
Copy link

Uditmittal commented Feb 24, 2021

<settings>
  <activeProfiles>
    <activeProfile>
      apple-silicon
    </activeProfile>
  </activeProfiles>
  <profiles>
    <profile>
      <id>apple-silicon</id>
      <properties>
        <os.detected.classifier>osx-x86_64</os.detected.classifier>
      </properties>
    </profile>
  </profiles>
</settings>

@guyeu my seetings.xml file

@hu-chia
Copy link

hu-chia commented Feb 25, 2021

@Uditmittal This is my org.xolstice.maven.plugins:protobuf-maven-plugin build plugin configuration, the ${os.detected.classifier} value could read from active profile from settings.xml.

      <configuration>
          <protocArtifact>
              com.google.protobuf:protoc:3.1.0:exe:${os.detected.classifier}
          </protocArtifact>
          <pluginId>grpc-java</pluginId>
          <pluginArtifact>io.grpc:protoc-gen-grpc-java:1.0.0:exe:${os.detected.classifier}</pluginArtifact>
      </configuration>

@Uditmittal
Copy link

Thanks, it works for me.

@bangbang2333
Copy link

谢谢老哥,搞定了

@ejona86
Copy link
Member

ejona86 commented May 4, 2021

I'm not at all impressed with how hard it was to find the appropriate flags to pass to clang to make a universal binary, but it appears to be -arch arm64 -arch x86_64. Unclear if cross-compiling is as simple as -arch arm64. We can start by figuring out how to cross-compile/universal binary for protobuf. Afterward we can fight Gradle. The goal would be to let us build M1 binaries from an x86 Mac. But I don't have a Mac which means I can't help much, as develop-via-CI is a form of torture.

@ghost
Copy link

ghost commented May 14, 2021

@ejona86 Have you had any success building from source for AArch64? I am interested in helping moving this along...

@ejona86
Copy link
Member

ejona86 commented May 14, 2021

@jjones-figure, no, as I said, I don't have a Mac, so I can't really help much here. We totally build from source for AArch64 already on Linux. But it is a different toolchain for M1.

@ejona86
Copy link
Member

ejona86 commented Jun 2, 2021

protocolbuffers/protobuf#8557 just went into protobuf based on my suggestion to just copy the current x86 binary to an arm64 name until we have actual M1 support. That doesn't "fix" this, but it does resolve the main user-visible issues and doesn't turn out to be too hacky.

I think that is as simple as making a copy of the "exe" and the hashes/signature in our upload_artifacts.sh script.

theimpulson added a commit to theimpulson/seedvault that referenced this issue Aug 2, 2023
Fixes compilation on M1 macOS. Check grpc/grpc-java#7690 for further info

Signed-off-by: Aayush Gupta <aayushgupta219@gmail.com>
theimpulson added a commit to theimpulson/seedvault that referenced this issue Aug 2, 2023
Fixes compilation on M1 macOS. Check grpc/grpc-java#7690 for further info

Signed-off-by: Aayush Gupta <aayushgupta219@gmail.com>
@glli
Copy link

glli commented Aug 14, 2023

It's easy to build the native protoc-gen-grpc-java-$VER-osx-aarch_64.exe with Bazel on a macbook m1/m2, even you can backfill the old versions for your project's backward compatibility, e.g. $VER=1.31.1

  1. Download https://github.com/grpc/grpc-java/archive/refs/tags/v1.31.1.tar.gz and extract it
  2. Download https://github.com/bazelbuild/bazel/releases/download/6.3.2/bazel-6.3.2-darwin-arm64, rename it to bazel and make it executable
  3. cd grpc-java-1.31.1/compiler and run bazel build grpc_java_plugin
  4. cd grpc-java-1.31.1/bazel-bin/compiler and cp/mv grpc_java_plugin to ~/bin (or what ever folder you like)
  5. make a symbolic link ln -s ~/bin/grpc_java_plugin ~/.gradle/caches/modules-2/files-2.1/io.grpc/protoc-gen-grpc-java/1.31.1/$SHA1/protoc-gen-grpc-java-1.31.1-osx-aarch_64.exe

(I didn't try building higher versions, please reply if you meet any issues in the above steps)

I also tried building grpc_java_plugin with Gradle. No problem of running ./gradlew :grpc-compiler:compileJava_pluginExecutableJava_pluginCpp (need to compile protobuf lib first based on the COMPILING.md). However I was stuck at running ./gradlew :grpc-compiler:build, always got

Task :grpc-compiler:linkJava_pluginExecutable FAILED
Undefined symbols for architecture arm64:
  "std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::at(unsigned long) const", referenced from:
      google::protobuf::io::Tokenizer::IsIdentifier(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) in libprotobuf.a(tokenizer.o)

Tried configuring --host=arm64 when compiling protobuf and it's not recognized. I think probably there's discrepancy between arm64 and aarch64 at the lib linking step.

@liuxuliangcumt
Copy link

liuxuliangcumt commented Aug 14, 2023 via email

tmancill added a commit to tmancill/JVoiceXML that referenced this issue Aug 22, 2023
Use x86_64 protobuf compilers on macOS until protobuf is updated to a newer version (~3.21.0).
See grpc/grpc-java#7690 for more background.
tmancill added a commit to tmancill/JVoiceXML that referenced this issue Aug 22, 2023
Use x86_64 protobuf compilers on macOS until protobuf is updated to a newer version (~3.21.0).
See grpc/grpc-java#7690 for more background.
stevesoltys pushed a commit to seedvault-app/seedvault that referenced this issue Sep 10, 2023
Fixes compilation on M1 macOS. Check grpc/grpc-java#7690 for further info

Signed-off-by: Aayush Gupta <aayushgupta219@gmail.com>
@JackyAnn
Copy link

zsh: bad CPU type in executable: ./protoc-gen-grpc-java-osx-aarch_64.exe

can not executable on mac m2 chips
version:
protoc-gen-grpc-java-1.58.0-osx-aarch_64.exe

@liuxuliangcumt
Copy link

liuxuliangcumt commented Oct 18, 2023 via email

@Kishan10008
Copy link

Kishan10008 commented Oct 19, 2023

for me, add this to ~/.m2/settings.xml, it worked:

<settings>
  ...
  <activeProfiles>
    <activeProfile>
      apple-silicon
    </activeProfile>
    ...
  </activeProfiles>
  <profiles>
    <profile>
      <id>apple-silicon</id>
      <properties>
        <os.detected.classifier>osx-x86_64</os.detected.classifier>
      </properties>
    </profile>
    ...
  </profiles>
  ...
</settings>

where i can find this .m2/settings.xml

@Kishan10008
Copy link

zsh: bad CPU type in executable: ./protoc-gen-grpc-java-osx-aarch_64.exe

can not executable on mac m2 chips version: protoc-gen-grpc-java-1.58.0-osx-aarch_64.exe

similar issue i am facing

@JackyAnn
Copy link

os.detected.classifier
@Kishan10008
I'm using gradle, and I'm not sure what the equivalent configuration is.
I downloaded the executable directly and it won't run directly on my mac m2 chip computer, but it works fine on the mac m1 chip

@ejona86
Copy link
Member

ejona86 commented Oct 20, 2023

@JackyAnn, it should work on M2 as well, as long as you have Rosetta installed.

@gelald
Copy link

gelald commented Nov 25, 2023

@hu-chia simple、useful、effective!

@liuxuliangcumt
Copy link

liuxuliangcumt commented Nov 25, 2023 via email

@hakusai22
Copy link

  <properties>
    <os.detected.classifier>osx-x86_64</os.detected.classifier>
  </properties>

@kdubb
Copy link
Contributor

kdubb commented Nov 26, 2023

This is going to have to be fixed. Apple has not shipped x86 processors for a year. Going by their past switch, from Power -> x86, it will only be a couple years before they stop shipping Rosetta.

Edit:
Let me clarify... Rosetta currently auto installs for macOS Applications, not CLI applications; we might be within a year of that not happening. When they stop relying on Rosetta they it will be abandoned. Currently installing it manually is a solution, which is already quite a nuisance for CI/CD, at some point things will stop working.

@isarang
Copy link

isarang commented Dec 5, 2023

<plugin>
  <groupId>org.xolstice.maven.plugins</groupId>
  <artifactId>protobuf-maven-plugin</artifactId>
  <version>0.5.1</version>
  <configuration>
    <protocArtifact>com.google.protobuf:protoc:3.24.4:exe:${os.detected.classifier}</protocArtifact>
    <pluginId>grpc-java</pluginId>
    <pluginArtifact>io.grpc:protoc-gen-grpc-java:1.43.3:exe:${os.detected.classifier}</pluginArtifact>
  </configuration>
  <executions>
    <execution>
      <goals>
        <goal>compile</goal>
        <goal>compile-custom</goal>
      </goals>
    </execution>
  </executions>
</plugin>

This works for me

AndreiFlorea04 added a commit to AndreiFlorea04/seedvault that referenced this issue Dec 19, 2023
Fixes compilation on M1 macOS. Check grpc/grpc-java#7690 for further info

Signed-off-by: Aayush Gupta <aayushgupta219@gmail.com>
@vvWantHealthLife
Copy link

thanks that's work

@martin-traverse
Copy link

March 2024. My build started working when I installed Rosetta. I'm using the Gradle protobuf plugin and did not need to change the Gradle build scripts. I echo the comments above, it would be good to have native binaries before the drop-dead date.

Another note, it seems this is already done for the protoc compiler itself, only the gRPC plugin is causing problems. I have three modules using protoc, only one of which also uses the gRPC plugin. The others just use base protoc and they built fine without needing to install Rosetta. Can the same approach that was used for the base protoc be replicated here?

@ejona86
Copy link
Member

ejona86 commented Mar 4, 2024

@martin-traverse, protobuf's release build used to be like ours. But they changed it dramatically and it is now based on Bazel. Looks like they are using --cpu= to build aarch_64 and x86_64 individually. They have quite specific toolchain configuration. On the release, they have three osx binaries: aarch_64, x86_64, and universal_binary. I don't see the code that makes the universal binary.

We are using autotools right now to build protobuf, and then using Gradle to build the plugin. But we've not been able to absorb the absl dependency in newer protobuf versions. I have a change to swap to cmake to build protobuf, but pkg-config is now required and that caused trouble on Windows. More recently pkgconf has taken off, so we might now have a solution there.

Since absl was added to protobuf, I have considered moving the plugin compilation out of Gradle. That could be a stepping-stone to using Bazel. But I think that is too much to depend on here. I still believe this just needs a few flags when building (like #7690 (comment), or the --cpu flags). But that really needs someone with a Mac to figure it out.

@Areeb786123
Copy link

@Uditmittal in my build.gradle file

protobuf {
    protoc {
        // for apple m1, please add protoc_platform=osx-x86_64 in $HOME/.gradle/gradle.properties
        if (project.hasProperty('protoc_platform')) {
            artifact = "com.google.protobuf:protoc:3.13.0:${protoc_platform}"
        } else {
            artifact = "com.google.protobuf:protoc:3.13.0"
        }
    }
}

in my ~/.gradle/gradle.properties file

protoc_platform=osx-x86_64

I think you can define and use variables in maven to implement this feature. Good luck

@Jiachen-Zhang

protobuf {
  protoc {
    if (osdetector.os == "osx") {
      artifact = 'com.google.protobuf:protoc:3.14.0:osx-x86_64'
    } else {
      artifact = 'com.google.protobuf:protoc:3.14.0'
    }
  }
}

You can also use project properties to only change the behavior when explicitly requested.

it didn't work for me

@gaofeiseu
Copy link

in my case, this will works well:
if you have a pom file and use maven plugin to complie the target package, for example:
image
so, what you need to do is specify a fixed value for the position pointed by the arrow, the fixed value is osx-x86_64 , it's x86 mac os actual value.

@amraupward
Copy link

amraupward commented Apr 3, 2024

com.google.protobuf:protoc:3.14.0:osx-x86_64
Title:
Build fails on macOS M2 with generateProto task error

Body:

Environment
OS: macOS M2
Gradle Version: (Specify your Gradle version here)
Java Version: (Specify your Java version here)
Protobuf Plugin Version: 0.9.4
gRPC Version: 1.62.2
Spring Boot Version: 3.2.3
OpenAPI Generator Version: 7.3.0
Issue Description
When attempting to build my project on a macOS M2 machine using Docker, the build fails during the generateProto task. The failure is related to the Protobuf and gRPC setup in my build.gradle.

Error Log
Screenshot 2024-04-03 at 17 29 28

Steps to Reproduce
Run gradle bootJar -x test --no-daemon inside a Docker container on a macOS M2 machine.

`protobuf {
protoc {
if (osdetector.os == 'osx' && osdetector.arch == 'aarch_64') {
artifact = 'com.google.protobuf:protoc:3.25.3:osx-x86_64'
} else {
artifact = 'com.google.protobuf:protoc:3.25.3'
}
}
plugins {
if (osdetector.os == 'osx'){
grpc {
artifact = 'io.grpc:protoc-gen-grpc-java:1.62.2:osx-x86_64'
}
} else {
grpc {
artifact = 'io.grpc:protoc-gen-grpc-java:1.62.2'
}
}
}
generateProtoTasks {
all()*.plugins {
grpc {}
}
}
}

dependencies {
// gRPC
implementation 'com.google.protobuf:protobuf-java:3.25.3'
implementation 'io.grpc:grpc-protobuf:1.62.2'
implementation 'io.grpc:grpc-stub:1.62.2'
runtimeOnly 'io.grpc:grpc-netty-shaded:1.62.2'
}

plugins {
id 'java'
id 'org.springframework.boot' version '3.2.3'
id 'io.spring.dependency-management' version '1.1.4'
id 'org.openapi.generator' version '7.3.0'
id 'maven-publish'
id 'com.google.protobuf' version '0.9.4'
id "io.github.lognet.grpc-spring-boot" version '5.0.0'
}`

Expected Behavior
The project builds successfully without errors related to the generateProto task.

Actual Behavior
The build fails with an error indicating an issue with executing protoc or the gRPC plugin.

@siddhsql
Copy link

siddhsql commented Apr 8, 2024

any update on this issue? everyone is on M1/M2 chips these days.

@amraupward
Copy link

any update on this issue? everyone is on M1/M2 chips these days.

M2 Macbook Air.

build.gradle

protobuf {
protoc {
artifact = "com.google.protobuf:protoc:3.25.1"
}
plugins {
grpc {
artifact = 'io.grpc:protoc-gen-grpc-java:1.52.1' // <----- this version can build your java
}
}
generateProtoTasks {
all()*.plugins {
grpc {}
}
}
}

if you want to create docker image.

FROM jdk as gradle_cache
RUN mkdir -p /tmp/gradle/cache
ENV GRADLE_USER_HOME /tmp/gradle/cache
ARG GPR_USERNAME
ARG GPR_TOKEN
ENV GPR_USERNAME $GPR_USERNAME
ENV GPR_TOKEN $GPR_TOKEN
ENV GRADLE_OPTS="-Xmx4g -XX:+UseContainerSupport -Dfile.encoding=UTF-8". // <----- this is important
COPY build.gradle /tmp
WORKDIR /tmp/
RUN gradle clean build -i --stacktrace --no-daemon

and

dockerbuild.sh
docker buildx build
--platform linux/amd64 --load \ //<------------ this too.
-t "$LOCAL_TAG"
-t "$AZURE_LOCAL_TAG"
--build-arg GPR_USERNAME=$GPR_USERNAME
--build-arg GPR_TOKEN=$GPR_TOKEN
--quiet=false
--output=type=docker
-f ./docker.dockerfile .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet