Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

compilation issues #432

Closed
jwijffels opened this issue Dec 16, 2019 · 9 comments
Closed

compilation issues #432

jwijffels opened this issue Dec 16, 2019 · 9 comments
Labels
execution environment Any issues related to execution environment, installation

Comments

@jwijffels
Copy link

jwijffels commented Dec 16, 2019 •
edited

Hi,

I would like to upload the R wrapper (https://github.com/bnosac/sentencepiece) around sentencepiece to CRAN. CRAN is pretty strict on compilation issues. Would it be possible to fix the following issues which occur when checking on Debian, Solaris, ASAN/UBSAN

On Debian Linux, GCC, no long double, I'm getting:

Found the following significant warnings:
  sentencepiece/src/freelist.h:62:13: warning: ‘void* memset(void*, int, size_t)’ clearing an object of non-trivial type ‘struct sentencepiece::unigram::Lattice::Node’; use assignment or value-initialization instead [-Wclass-memaccess]
  sentencepiece/src/freelist.h:39:13: warning: ‘void* memset(void*, int, size_t)’ clearing an object of non-trivial type ‘struct sentencepiece::unigram::Lattice::Node’; use assignment or value-initialization instead [-Wclass-memaccess]

On Oracle Solaris, compilation fails: with the following error

Running `R CMD build`...
* checking for file ‘/export/home/XUABCjr/Rtemp/RtmpnMayqW/remotes60831f397749/sentencepiece/DESCRIPTION’ ... OK
* preparing ‘sentencepiece’:
* checking DESCRIPTION meta-information ... OK
* cleaning src
* checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories
* building ‘sentencepiece_0.1.0.tar.gz’
Installing package into ‘/export/home/XUABCjr/R’
(as ‘lib’ is unspecified)
* installing *source* package ‘sentencepiece’ ...
** using staged installation
** libs
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/arena.cc -o third_party/protobuf-lite/arena.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/arenastring.cc -o third_party/protobuf-lite/arenastring.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/bytestream.cc -o third_party/protobuf-lite/bytestream.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/coded_stream.cc -o third_party/protobuf-lite/coded_stream.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/common.cc -o third_party/protobuf-lite/common.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/extension_set.cc -o third_party/protobuf-lite/extension_set.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/generated_message_table_driven_lite.cc -o third_party/protobuf-lite/generated_message_table_driven_lite.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/generated_message_util.cc -o third_party/protobuf-lite/generated_message_util.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/implicit_weak_message.cc -o third_party/protobuf-lite/implicit_weak_message.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/int128.cc -o third_party/protobuf-lite/int128.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/io_win32.cc -o third_party/protobuf-lite/io_win32.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/message_lite.cc -o third_party/protobuf-lite/message_lite.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/repeated_field.cc -o third_party/protobuf-lite/repeated_field.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/status.cc -o third_party/protobuf-lite/status.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/statusor.cc -o third_party/protobuf-lite/statusor.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/stringpiece.cc -o third_party/protobuf-lite/stringpiece.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/stringprintf.cc -o third_party/protobuf-lite/stringprintf.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/structurally_valid.cc -o third_party/protobuf-lite/structurally_valid.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/strutil.cc -o third_party/protobuf-lite/strutil.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/time.cc -o third_party/protobuf-lite/time.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/wire_format_lite.cc -o third_party/protobuf-lite/wire_format_lite.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/zero_copy_stream.cc -o third_party/protobuf-lite/zero_copy_stream.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/protobuf-lite/zero_copy_stream_impl_lite.cc -o third_party/protobuf-lite/zero_copy_stream_impl_lite.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c third_party/absl/strings/string_view.cc -o third_party/absl/strings/string_view.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c sentencepiece/src/builtin_pb/sentencepiece.pb.cc -o sentencepiece/src/builtin_pb/sentencepiece.pb.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c sentencepiece/src/builtin_pb/sentencepiece_model.pb.cc -o sentencepiece/src/builtin_pb/sentencepiece_model.pb.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c sentencepiece/src/bpe_model.cc -o sentencepiece/src/bpe_model.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c sentencepiece/src/bpe_model_trainer.cc -o sentencepiece/src/bpe_model_trainer.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c sentencepiece/src/builder.cc -o sentencepiece/src/builder.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c sentencepiece/src/char_model.cc -o sentencepiece/src/char_model.o
/opt/csw/bin/g++ -std=gnu++11 -I"/opt/R/R-3.6.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I"/export/home/XUABCjr/R/Rcpp/include" -I/opt/cdw/include -I/usr/local/include  -fPIC  -O2  -c sentencepiece/src/char_model_trainer.cc -o sentencepiece/src/char_model_trainer.o
sentencepiece/src/char_model_trainer.cc: In member function ‘virtual sentencepiece::util::Status sentencepiece::character::Trainer::Train()’:
sentencepiece/src/char_model_trainer.cc:41:31: error: call of overloaded ‘log(uint64&)’ is ambiguous
   const float logsum = log(sum);
                               ^
In file included from /opt/csw/lib/gcc/i386-pc-solaris2.10/5.5.0/include-fixed/math.h:22:0,
                 from /opt/csw/include/c++/5.5.0/cmath:44,
                 from /opt/csw/include/c++/5.5.0/random:38,
                 from sentencepiece/src/util.h:24,
                 from sentencepiece/src/trainer_interface.h:29,
                 from sentencepiece/src/char_model_trainer.h:19,
                 from sentencepiece/src/char_model_trainer.cc:15:
/opt/csw/lib/gcc/i386-pc-solaris2.10/5.5.0/include-fixed/iso/math_iso.h:206:21: note: candidate: long double std::log(long double)
  inline long double log(long double __X) { return __logl(__X); }
                     ^
/opt/csw/lib/gcc/i386-pc-solaris2.10/5.5.0/include-fixed/iso/math_iso.h:167:15: note: candidate: float std::log(float)
  inline float log(float __X) { return __logf(__X); }
               ^
/opt/csw/lib/gcc/i386-pc-solaris2.10/5.5.0/include-fixed/iso/math_iso.h:67:15: note: candidate: double std::log(double)
 extern double log __P((double));
               ^
sentencepiece/src/char_model_trainer.cc:50:45: error: call of overloaded ‘log(const long long int&)’ is ambiguous
                                log(it.second) - logsum);
                                             ^
In file included from /opt/csw/lib/gcc/i386-pc-solaris2.10/5.5.0/include-fixed/math.h:22:0,
                 from /opt/csw/include/c++/5.5.0/cmath:44,
                 from /opt/csw/include/c++/5.5.0/random:38,
                 from sentencepiece/src/util.h:24,
                 from sentencepiece/src/trainer_interface.h:29,
                 from sentencepiece/src/char_model_trainer.h:19,
                 from sentencepiece/src/char_model_trainer.cc:15:
/opt/csw/lib/gcc/i386-pc-solaris2.10/5.5.0/include-fixed/iso/math_iso.h:206:21: note: candidate: long double std::log(long double)
  inline long double log(long double __X) { return __logl(__X); }
                     ^
/opt/csw/lib/gcc/i386-pc-solaris2.10/5.5.0/include-fixed/iso/math_iso.h:167:15: note: candidate: float std::log(float)
  inline float log(float __X) { return __logf(__X); }
               ^
/opt/csw/lib/gcc/i386-pc-solaris2.10/5.5.0/include-fixed/iso/math_iso.h:67:15: note: candidate: double std::log(double)
 extern double log __P((double));
               ^
At global scope:
cc1plus: warning: unrecognized command line option ‘-Wno-misleading-indentation’
gmake: *** [/opt/R/R-3.6.0/lib/R/etc/Makeconf:174: sentencepiece/src/char_model_trainer.o] Error 1

On Debian Linux, GCC, with Address Sanitizers (ASAN/UBSAN), I'm getting address sanitizer failures in protobuf-lite. See below:

sentencepiece_trainer.cc(116) LOG(INFO) Running command: --input=traindata.txt --model_prefix=sentencepiece --vocab_size=10 --character_coverage=0.9999 --model_type=char
sentencepiece_trainer.cc(49) LOG(INFO) Starts training with : 
TrainerSpec {
  input: traindata.txt
  input_format: 
  model_prefix: sentencepiece
  model_type: CHAR
  vocab_size: 10
  self_test_sample_size: 0
  character_coverage: 0.9999
  input_sentence_size: 0
  shuffle_input_sentence: 1
  seed_sentencepiece_size: 1000000
  shrinking_factor: 0.75
  max_sentence_length: 4192
  num_threads: 16
  num_sub_iterations: 2
  max_sentencepiece_length: 16
  split_by_unicode_script: 1
  split_by_number: 1
  split_by_whitespace: 1
  treat_whitespace_as_suffix: 0
  hard_vocab_limit: 1
  use_all_vocab: 0
  unk_id: 0
  bos_id: 1
  eos_id: 2
  pad_id: -1
  unk_piece: <unk>
  bos_piece: <s>
  eos_piece: </s>
  pad_piece: <pad>
  unk_surface:  ⁇ 
}
NormalizerSpec {
  name: nmt_nfkc
  add_dummy_prefix: 1
  remove_extra_whitespaces: 1
  escape_whitespaces: 1
  normalization_rule_tsv: 
}

trainer_interface.cc(267) LOG(INFO) Loading corpus: traindata.txt
trainer_interface.cc(287) LOG(WARNING) Found too long line (4825 > 4192).
trainer_interface.cc(289) LOG(WARNING) Too long lines are skipped in the training.
trainer_interface.cc(290) LOG(WARNING) The maximum length can be changed with --max_sentence_length=<size> flag.
trainer_interface.cc(315) LOG(INFO) Loaded all 3992 sentences
trainer_interface.cc(321) LOG(INFO) Skipped 8 too long sentences.
trainer_interface.cc(330) LOG(INFO) Adding meta_piece: <unk>
trainer_interface.cc(330) LOG(INFO) Adding meta_piece: <s>
trainer_interface.cc(330) LOG(INFO) Adding meta_piece: </s>
trainer_interface.cc(335) LOG(INFO) Normalizing sentences...
trainer_interface.cc(384) LOG(INFO) all chars count=2593880
trainer_interface.cc(392) LOG(INFO) Done: 99.9928% characters are covered.
trainer_interface.cc(402) LOG(INFO) Alphabet size=73
trainer_interface.cc(403) LOG(INFO) Final character coverage=0.999928
trainer_interface.cc(435) LOG(INFO) Done! preprocessed 3992 sentences.
trainer_interface.cc(507) LOG(INFO) Saving model: sentencepiece.model
third_party/protobuf-lite/google/protobuf/repeated_field.h:1537:35: runtime error: index 1 out of bounds for type 'void *[1]'
third_party/protobuf-lite/google/protobuf/repeated_field.h:1537:35: runtime error: index 3 out of bounds for type 'void *[1]'
third_party/protobuf-lite/repeated_field.cc:88:39: runtime error: index 4 out of bounds for type 'void *[1]'
third_party/protobuf-lite/google/protobuf/repeated_field.h:1507:28: runtime error: index 1 out of bounds for type 'void *[1]'
third_party/protobuf-lite/google/protobuf/repeated_field.h:1507:28: runtime error: index 1 out of bounds for type 'void *[1]'
trainer_interface.cc(531) LOG(INFO) Saving vocabs: sentencepiece.vocab
third_party/protobuf-lite/google/protobuf/repeated_field.h:1537:35: runtime error: index 1 out of bounds for type 'void *[1]'
third_party/protobuf-lite/google/protobuf/repeated_field.h:1507:28: runtime error: index 9 out of bounds for type 'void *[1]'
third_party/protobuf-lite/google/protobuf/repeated_field.h:1507:28: runtime error: index 1 out of bounds for type 'void *[1]'
third_party/protobuf-lite/google/protobuf/repeated_field.h:1507:28: runtime error: index 1 out of bounds for type 'void *[1]'

...

sentencepiece_trainer.cc(116) LOG(INFO) Running command: --input=traindata.txt --model_prefix=sentencepiece --vocab_size=10 --character_coverage=0.9999 --model_type=bpe
sentencepiece_trainer.cc(49) LOG(INFO) Starts training with : 
TrainerSpec {
  input: traindata.txt
  input_format: 
  model_prefix: sentencepiece
  model_type: BPE
  vocab_size: 10
  self_test_sample_size: 0
  character_coverage: 0.9999
  input_sentence_size: 0
  shuffle_input_sentence: 1
  seed_sentencepiece_size: 1000000
  shrinking_factor: 0.75
  max_sentence_length: 4192
  num_threads: 16
  num_sub_iterations: 2
  max_sentencepiece_length: 16
  split_by_unicode_script: 1
  split_by_number: 1
  split_by_whitespace: 1
  treat_whitespace_as_suffix: 0
  hard_vocab_limit: 1
  use_all_vocab: 0
  unk_id: 0
  bos_id: 1
  eos_id: 2
  pad_id: -1
  unk_piece: <unk>
  bos_piece: <s>
  eos_piece: </s>
  pad_piece: <pad>
  unk_surface:  ⁇ 
}
NormalizerSpec {
  name: nmt_nfkc
  add_dummy_prefix: 1
  remove_extra_whitespaces: 1
  escape_whitespaces: 1
  normalization_rule_tsv: 
}

trainer_interface.cc(267) LOG(INFO) Loading corpus: traindata.txt
trainer_interface.cc(287) LOG(WARNING) Found too long line (4825 > 4192).
trainer_interface.cc(289) LOG(WARNING) Too long lines are skipped in the training.
trainer_interface.cc(290) LOG(WARNING) The maximum length can be changed with --max_sentence_length=<size> flag.
trainer_interface.cc(315) LOG(INFO) Loaded all 3992 sentences
trainer_interface.cc(321) LOG(INFO) Skipped 8 too long sentences.
trainer_interface.cc(330) LOG(INFO) Adding meta_piece: <unk>
trainer_interface.cc(330) LOG(INFO) Adding meta_piece: <s>
trainer_interface.cc(330) LOG(INFO) Adding meta_piece: </s>
trainer_interface.cc(335) LOG(INFO) Normalizing sentences...
trainer_interface.cc(384) LOG(INFO) all chars count=2593880
trainer_interface.cc(392) LOG(INFO) Done: 99.9928% characters are covered.
trainer_interface.cc(402) LOG(INFO) Alphabet size=73
trainer_interface.cc(403) LOG(INFO) Final character coverage=0.999928
...
third_party/protobuf-lite/google/protobuf/repeated_field.h:1507:28: runtime error: index 110 out of bounds for type 'void *[1]'
third_party/protobuf-lite/google/protobuf/repeated_field.h:1507:28: runtime error: index 110 out of bounds for type 'void *[1]'
third_party/protobuf-lite/google/protobuf/repeated_field.h:1507:28: runtime error: index 89 out of bounds for type 'void *[1]'
third_party/protobuf-lite/google/protobuf/repeated_field.h:1507:28: runtime error: index 89 out of bounds for type 'void *[1]'
third_party/protobuf-lite/google/protobuf/repeated_field.h:1507:28: runtime error: index 269 out of bounds for type 'void *[1]'
third_party/protobuf-lite/google/protobuf/repeated_field.h:1507:28: runtime error: index 269 out of bounds for type 'void *[1]'
third_party/protobuf-lite/google/protobuf/repeated_field.h:1507:28: runtime error: index 248 out of bounds for type 'void *[1]'
third_party/protobuf-lite/google/protobuf/repeated_field.h:1507:28: runtime error: index 248 out of bounds for type 'void *[1]'
third_party/protobuf-lite/google/protobuf/repeated_field.h:1507:28: runtime error: index 248 out of bounds for type 'void *[1]'
third_party/protobuf-lite/google/protobuf/repeated_field.h:1537:35: runtime error: index 1 out of bounds for type 'void *[1]'
third_party/protobuf-lite/google/protobuf/repeated_field.h:1537:35: runtime error: index 1 out of bounds for type 'void *[1]'

Next it would be nice if we don't need to compile with -Wno-pedantic -Wno-misleading-indentation -Wno-sign-compare as that makes the compiler pretty noisy. Could such warnings also be removed.

thank you :)!
@jwijffels
Copy link
Author

Hello. Anyone willing to help on this?

@jwijffels jwijffels mentioned this issue Jan 22, 2020
Closed
@taku910 taku910 added the execution environment Any issues related to execution environment, installation label Apr 23, 2020
@jwijffels
Copy link
Author

Would be great if someone could look into these compilation issues such that the R community could also benefit from this package. Thanks!

@jwijffels
Copy link
Author

@taku910
I would like to upload the R package wrapping sentencepiece to CRAN but the The warning messages about the use of memset in freelist.h are the only blocking element here.
Can you fix this?

  sentencepiece/src/freelist.h:62:13: warning: 'void* memset(void*, int, size_t)' clearing an object of non-trivial type 'struct sentencepiece::unigram::Lattice::Node'; use assignment or value-initialization instead [-Wclass-memaccess]
  sentencepiece/src/freelist.h:39:13: warning: 'void* memset(void*, int, size_t)' clearing an object of non-trivial type 'struct sentencepiece::unigram::Lattice::Node'; use assignment or value-initialization instead [-Wclass-memaccess]

@jwijffels
Copy link
Author

@taku910
How about changing
https://github.com/google/sentencepiece/blob/master/src/freelist.h#L40 and
https://github.com/google/sentencepiece/blob/master/src/freelist.h#L63
with
memset(static_cast<void *>(chunk), 0, sizeof(*chunk) * chunk_size_);

this also seems to be done here to get rid of the warning: google/flatbuffers@2e865f4

@taku910
Copy link
Collaborator

taku910 commented May 24, 2020

Confirmed. Fixed in the head.

@jwijffels
Copy link
Author

Thanks. I'll leave this issue open for gradually fixing the remainder compilation issues. I've pushed the package to CRAN. That will provide me with further compilation traces with detailed descriptions. I'll get back on this within 2 weeks.

@jwijffels
Copy link
Author

jwijffels commented Jun 5, 2020 •
edited

FYI. R package wrapping sentencepiece was published to CRAN yesterday https://CRAN.R-project.org/package=sentencepiece.

The installation fails on Solaris however due to the issue which was reported here, namely the use of log in char_model_trainer.cc and word_model_trainer.cc as shown below

/opt/csw/bin/g++ -std=gnu++11 -I"/home/ripley/R/gcc/include" -DNDEBUG -D HAVE_PTHREAD=1 -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite -I'/home/ripley/R/Lib32/Rcpp/include' -I/opt/csw/include -I/usr/local/include  -fPIC  -O2  -c sentencepiece/src/char_model_trainer.cc -o sentencepiece/src/char_model_trainer.o
sentencepiece/src/char_model_trainer.cc: In member function ‘virtual sentencepiece::util::Status sentencepiece::character::Trainer::Train()’:
sentencepiece/src/char_model_trainer.cc:41:31: error: call of overloaded ‘log(uint64&)’ is ambiguous
   const float logsum = log(sum);
                               ^
In file included from /usr/include/math.h:15:0,
                 from /opt/csw/include/c++/5.2.0/cmath:44,
                 from /opt/csw/include/c++/5.2.0/random:38,
                 from /opt/csw/include/c++/5.2.0/bits/stl_algo.h:66,
                 from /opt/csw/include/c++/5.2.0/algorithm:62,
                 from ./third_party/protobuf-lite/google/protobuf/stubs/common.h:38,
                 from ./sentencepiece/src/builtin_pb/sentencepiece_model.pb.h:9,
                 from sentencepiece/src/char_model_trainer.h:18,
                 from sentencepiece/src/char_model_trainer.cc:15:
/opt/csw/lib/gcc/i386-pc-solaris2.10/5.2.0/include-fixed/iso/math_iso.h:200:21: note: candidate: long double std::log(long double)
  inline long double log(long double __X) { return __logl(__X); }
                     ^
/opt/csw/lib/gcc/i386-pc-solaris2.10/5.2.0/include-fixed/iso/math_iso.h:168:15: note: candidate: float std::log(float)
  inline float log(float __X) { return __logf(__X); }
               ^
/opt/csw/lib/gcc/i386-pc-solaris2.10/5.2.0/include-fixed/iso/math_iso.h:68:15: note: candidate: double std::log(double)
 extern double log __P((double));
               ^
sentencepiece/src/char_model_trainer.cc:50:45: error: call of overloaded ‘log(const long long int&)’ is ambiguous
                                log(it.second) - logsum);
                                             ^
In file included from /usr/include/math.h:15:0,
                 from /opt/csw/include/c++/5.2.0/cmath:44,
                 from /opt/csw/include/c++/5.2.0/random:38,
                 from /opt/csw/include/c++/5.2.0/bits/stl_algo.h:66,
                 from /opt/csw/include/c++/5.2.0/algorithm:62,
                 from ./third_party/protobuf-lite/google/protobuf/stubs/common.h:38,
                 from ./sentencepiece/src/builtin_pb/sentencepiece_model.pb.h:9,
                 from sentencepiece/src/char_model_trainer.h:18,
                 from sentencepiece/src/char_model_trainer.cc:15:
/opt/csw/lib/gcc/i386-pc-solaris2.10/5.2.0/include-fixed/iso/math_iso.h:200:21: note: candidate: long double std::log(long double)
  inline long double log(long double __X) { return __logl(__X); }
                     ^
/opt/csw/lib/gcc/i386-pc-solaris2.10/5.2.0/include-fixed/iso/math_iso.h:168:15: note: candidate: float std::log(float)
  inline float log(float __X) { return __logf(__X); }
               ^
/opt/csw/lib/gcc/i386-pc-solaris2.10/5.2.0/include-fixed/iso/math_iso.h:68:15: note: candidate: double std::log(double)
 extern double log __P((double));
               ^
*** Error code 1
make: Fatal error: Command failed for target `sentencepiece/src/char_model_trainer.o'

Mathematical functions such as sqrt or log are defined in C++11 for floating-point arguments: float, double, long double and possibly more. The standard specifies what happens with an argument of integer type but this is not always implemented, resulting in a report of ‘overloading ambiguity’: this is commonly seen on Solaris

I suggest to replace following code snippets which are in char_model_trainer.cc and word_model_trainer.cc in the following way:

  • log(sum) with static_cast<float>(log(static_cast<long double>(sum)))
  • log(it.second) with static_cast<float>(log(static_cast<long double>(it.second)))

Next to that I also had to update protobuf-lite/common.cc to avoid having the compilation issue on Solaris as shown below

/opt/developerstudio12.6/bin/CC -std=c++11 -library=stdcpp,CrunG3  -I"/opt/R/R-4.0.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I'/export/home/XzfJZVZ/R/Rcpp/include' -I/opt/csw/include   -KPIC  -O -xlibmil -xtarget=generic -xcache=generic -nofstore  -c third_party/protobuf-lite/common.cc -o third_party/protobuf-lite/common.o
CC: Warning: Option -pthread passed to ld, if ld is invoked, ignored otherwise
"third_party/protobuf-lite/common.cc", line 379: Error: Only a function may be called.

Code chunk which had to be updated in protobuf-lite/common.cc was as follows:

static void RunZeroArgFunc(const void* arg) {
  void (*func)() = reinterpret_cast<void (*)()>(const_cast<void*>(arg));
  func();
}

Next had to incorporate protobuf as fixed here on protocolbuffers/protobuf@be14bf2#diff-864b83a0332a5de58a78915daf987e1c
Otherwise had as well

/opt/developerstudio12.6/bin/CC -std=c++11 -library=stdcpp,CrunG3  -I"/opt/R/R-4.0.0/lib/R/include" -DNDEBUG -D HAVE_PTHREAD=1 -pthread -DSTRICT_R_HEADERS -I. -I./sentencepiece -I./sentencepiece/src -I./sentencepiece/src/builtin_pb -I./third_party/absl -I./third_party/darts_clone -I./third_party/esaxx -I./third_party/protobuf-lite  -I'/export/home/Xt5BSWw/R/Rcpp/include' -I/opt/csw/include   -KPIC  -O -xlibmil -xtarget=generic -xcache=generic -nofstore  -c third_party/protobuf-lite/generated_message_util.cc -o third_party/protobuf-lite/generated_message_util.o
CC: Warning: Option -pthread passed to ld, if ld is invoked, ignored otherwise
"third_party/protobuf-lite/generated_message_util.cc", line 667: Error: Only a function may be called.
"third_party/protobuf-lite/generated_message_util.cc", line 713: Error: Only a function may be called.

@taku910
Copy link
Collaborator

taku910 commented Jun 8, 2020

Hopefully they are fixed in v0.1.92. I cannot check it because I don't have the environment to test.
Please reopen this issue if you find the error again.

@taku910 taku910 closed this as completed Jun 8, 2020
@jwijffels
Copy link
Author

Only remaining issue is the UBSAN messages from protobuf-lite shown in the first message of this thread. It is the same as reported at protocolbuffers/protobuf#2073

@jwijffels jwijffels mentioned this issue Dec 8, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
execution environment Any issues related to execution environment, installation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jwijffels @taku910 and others