Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: hermetic build OS detection #1988

Merged
merged 7 commits into from Sep 13, 2023
Merged

feat: hermetic build OS detection #1988

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
2bf4704
feat: add OS detection to hermetic build scripts
diegomarquezp Sep 13, 2023
0b97a2c
skip os arch in integration test
diegomarquezp Sep 13, 2023
05f2a99
remove unused IT flag
diegomarquezp Sep 13, 2023
686da2a
dynamic architecture resolution
diegomarquezp Sep 13, 2023
2dae9f1
change function name
diegomarquezp Sep 13, 2023
287bfa8
use `case` in os detection func
diegomarquezp Sep 13, 2023
6308f9b
correct os detection for aarch64
diegomarquezp Sep 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 1 addition & 2 deletions .github/workflows/verify_library_generation.yaml
View file
Open in desktop
Expand Up @@ -28,8 +28,7 @@ jobs:
library_generation/test/generate_library_integration_test.sh \
-p google/bigtable/v2 \
-d google-cloud-bigtable-v2-java \
--googleapis_gen_url https://cloud-java-bot:${{ secrets.CLOUD_JAVA_BOT_GITHUB_TOKEN }}@github.com/googleapis/googleapis-gen.git \
--os_type ${{ matrix.os }}
--googleapis_gen_url https://cloud-java-bot:${{ secrets.CLOUD_JAVA_BOT_GITHUB_TOKEN }}@github.com/googleapis/googleapis-gen.git
unit_tests:
strategy:
matrix:
Expand Down
2 changes: 1 addition & 1 deletion library_generation/generate_library.sh
View file
Open in desktop
Expand Up @@ -78,7 +78,7 @@ if [ -z "${include_samples}" ]; then
fi

if [ -z "${os_architecture}" ]; then
os_architecture="linux-x86_64"
os_architecture=$(detect_os_architecture)
fi

mkdir -p "${destination_path}"
Expand Down
12 changes: 1 addition & 11 deletions library_generation/test/generate_library_integration_test.sh
View file
Open in desktop
Expand Up @@ -30,10 +30,6 @@ case $key in
googleapis_gen_url="$2"
shift
;;
--os_type)
os_type="$2"
shift
;;
*)
echo "Invalid option: [$1]"
exit 1
Expand Down Expand Up @@ -78,11 +74,6 @@ include_samples=$(get_config_from_BUILD \
"false"
)
echo "GAPIC options are transport=${transport}, rest_numeric_enums=${rest_numeric_enums}, include_samples=${include_samples}."
os_architecture="linux-x86_64"
if [[ "$os_type" == *"macos"* ]]; then
os_architecture="osx-x86_64"
fi
echo "OS Architecture is ${os_architecture}."
# generate GAPIC client library
echo "Generating library from ${proto_path}, to ${destination_path}..."
"${library_generation_dir}"/generate_library.sh \
Expand All @@ -93,8 +84,7 @@ echo "Generating library from ${proto_path}, to ${destination_path}..."
--grpc_version "${grpc_version}" \
--transport "${transport}" \
--rest_numeric_enums "${rest_numeric_enums}" \
--include_samples "${include_samples}" \
--os_architecture "${os_architecture}"
--include_samples "${include_samples}"

echo "Generate library finished."
echo "Checking out googleapis-gen repository..."
Expand Down
21 changes: 21 additions & 0 deletions library_generation/utilities.sh
View file
Open in desktop
Expand Up @@ -273,3 +273,24 @@ get_version_from_versions_txt() {
version=$(grep "$key:" "${versions}" | cut -d: -f3) # 3rd field is snapshot
echo "${version}"
}

detect_os_architecture() {
case "${OSTYPE}" in
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where does OSTYPE come from?

"linux-gnu"*|"freebsd"*)
os_architecture="linux-$(uname -m)"
;;
"darwin"*)
os_architecture="osx-$(uname -m)"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I run uname -m on my machine, the output is arm64.
However, the right os architecture should be osx-aarch_64.
You can reference the os architecture here

I also implemented a os architecture detection function:

get_os_architecture() {
  local os_type
  local os_architecture
  os_type=$(uname -sm)
  case "${os_type}" in
    *"Linux x86_64"*)
      os_architecture="linux-x86_64"
      ;;
    *"Darwin x86_64"*)
      os_architecture="osx-x86_64"
      ;;
    *)
      os_architecture="osx-aarch_64"
      ;;
  esac
  echo "${os_architecture}"
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, thanks for the function :) I couldn't confirm for the aarch64 case since I don't have such machine, but I switched the detection function to this one

;;
"cygwin"|"msys")
os_architecture="win32"
;;
*)
>&2 echo 'Could not detect OS. Please specify it with --os_architecture'
>&2 echo 'Also, see https://github.com/protocolbuffers/protobuf/releases for a list of available OS (e.g. linux-aarch_64)'
exit 1
;;
esac
>&2 echo "Detected OS architecture: ${os_architecture}" >2
echo "${os_architecture}"
}