Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to openssl-1.0.2a (Part2: final) #1389

Closed
wants to merge 16 commits into from

Conversation

shigeki
Copy link
Contributor

@shigeki shigeki commented Apr 10, 2015

This is a final PR for upgrading to openssl-1.0.2a. I'm sorry for this is a really big patch for I could not succeed separating. This PR consist of 4 type of commits. I think we can focus to review 2nd and 3rd part.

1. source updates and private patches

e5624ef deps: upgrade openssl to 1.0.2a
2a01155 deps: fix openssl assembly error on ia32 win32
752636f deps: fixed asm build error of openssl in x86_win32
0e6fad7 deps: backport openssl patch of alt cert chains 1
6b3a7c9 openssl: fix keypress requirement in apps on win32

2. openssconf.h and openssl.gyp/gypi updates

c3e3c4d deps: replace all headers in openssl
b581ae2 deps: add x32 and arm64 support for opensslconf.h
c75888e deps: update opensslconf.h for 1.0.2a
0d19c08 deps: update openssl.gyp/gypi for openssl-1.0.2a

3. asm updates

f2327d0 deps: update asm Makefile for openssl-1.0.2a
208e06a deps, build: add support older asm compiler
e311459 deps: update asm files for openssl-1.0.2a

4. others

2a925d5 build: enable ssl support on arm64
5f532c2 test: fix error message check for openssl-1.0.2a
d4a0c27 deps: add docs to upgrade openssl

Ci is https://jenkins-iojs.nodesource.com/job/iojs+any-pr+multi/486/ and it seems fine . The test failures in armv7 and wins are not related this PR.

Here attached is the file of deps/openssl/doc/UPGRADING.md below
I hope this helps to review this PR.
R= @bnoordhuis @indutny

How to upgrade openssl library in io.js

This document is intended to describe the procedure to upgrade openssl
from 1.0.1m to 1.0.2a in io.js.

Build System and Upgrading Overview

The openssl build system is based on the perl script of Configure.
For example, running Configure linux_x86-64 in the openssl
repository generates Makefile and opensslconf.h for the target
architectures of linux_x86_64.
The Makefile contains the list of asm files which are generated by
perl scripts during build so that we can get the most of use of the
hardware performance according to the type of cpus.

Configure TABLE shows various build parameters that depend on each
os and arch.

In io.js, build target is defined as --dest-os and --dest-cpu in
configure options which are different from the one that is defined in
openssl and it's build system is gyp that is based on python,
therefore we cannot use the openssl build system directly.

In order to build openssl with gyp in iojs, files of opensslconf.h and
asm are generated in advance for several supported platforms.

Here is a map table to show conf(opensslconf.h) and asm between
the openssl target and configuration parameters of os and cpu in iojs.
The tested platform in CI are also listed.

--dest-os --dest-cpu conf asm openssl target CI
linux ia32 o o linux-elf o
linux x32 o x(*2) linux-x32 x
linux x64 o o linux-x86_64 o
linux arm o o linux-arm o
linux arm64 o o linux-aarch64 o
mac ia32 o o darwin-i386-cc o
mac x64 o o darwin64-x86_64-cc o
win ia32 o o(*3) VC-WIN32 x
win x64 o o VC-WIN64A o
solaris ia32 o o solaris-x86-gcc o
solaris x64 o o solaris64-x86_64-gcc o
freebsd ia32 o o BSD-x86 o
freebsd x64 o o BSD-x86_64 o
openbsd ia32 o o BSD-x86 x
openbsd x64 o o BSD-x86_64 x
others ia32 x(*1) o - x
others x64 x(*1) o - x
others arm x(*1) o - x
others arm64 x(*1) o - x
others others x(*1) x(*2) - x
  • (*1) use linux-elf as a fallback configuration
  • (*2) no-asm used
  • (*3) currently masm (Microsoft Macro Assembler) is used but it's no
    longer supported in opnessl. We need to move to use nasm or yasm.

All parameters such as sources, defines, cflags and others generated
in openssl Makefile are wrote down into deps/openssl/openssl.gypi.

The header file of deps/openssl/openssl/crypto/opensslconf.h are
generated by Configure and varies on each os and arch so that we
made a new deps/openssl/config/opensslconf.h, where it includes each
conf file from deps/openssl/config/archs/*/opensslconf.h by using
pre-defined compiler macros. This procedure can be processed
automatically with deps/openssl/config/Makefile

Assembler support is one of the key features in openssl, but asm files
are dynamically generated with
deps/openssl/openssl/crypto/*/asm/*.pl by perl during
build. Furthermore, these perl scripts check the version of asm
compiler and generate asm files according to the supported
instructions in each compiler.

Since perl is not a build requirement in iojs, they all should be
generated in advance and statically stored in the repository. We
provide two sets of asm files, one is asm_latest(avx2 and addx
supported) in deps/openssl/asm and the other asm_obsolete(without
avx1/2 and addx) in deps/openssl/asm_obsolute, which depends on
supported features in asm compilers. Each directory has a Makefile
to generate asm files with perl scripts in openssl sources.

configure and gyp check the version of asm compilers such as gnu
as(gas), llvm and Visual Studio. deps/openssl/openssl.gypi
determines what asm files should be used, in which the asm_latest
needs the version of gas >= 2.23, llvm >= 3.3 or MSVS_VERSION>='2012'
(ml64 >= 12) as defined in
https://github.com/openssl/openssl/blob/OpenSSL_1_0_2-stable/crypto/sha/asm/sha512-x86_64.pl#L112-L129,
otherwise asm_obsolete are used.

The following is the detail instruction steps how to upgrade openssl
version from 1.0.1m to 1.0.2a in iojs.

1. Replace openssl source in deps/openssl/openssl

Remove old openssl sources in deps/openssl/openssl .
Get original openssl sources from
https://www.openssl.org/source/openssl-1.0.2a.tar.gz and extract all
files into deps/openssl/openssl .

2. Apply private patches

There are three kinds of private patches to be applied in openssl-1.0.2a.

  • The two fixes of assembly error on ia32 win32. masm is no longer
    supported in openssl. We should move to use nasm or yasm in future
    version of iojs.
  • The fix of openssl-cli built on win. Key press requirement of
    openssl-cli in win causes timeout failures of several tests.
  • Backport patches for alt cert feature from openssl-1.1.x. Root certs
    of 1024bit RSA key length were deprecated in io.js. When a tls
    server has a cross root cert, io.js client leads CERT_UNTRUSTED
    error because openssl does not find alternate cert chains. This fix
    supports its feature but was made the current master which is
    openssl-1.1.x. We backported them privately into openssl-1.0.2 on
    iojs.

3. Replace openssl header files in deps/openssl/openssl/include/openssl

all header files in deps/openssl/openssl/include/openssl/*.h are
symbolic links in the distributed release tar.gz. They cause issues in
Windows. They are replaced into the files to include a real header
file such as

#include "../../crypto/aes/aes.h"

4. Change opensslconf.h so as to fit each platform.

The opensslconf.h in each target was created in advance by typing
deps/openssl/openssl/Configure {target} and copied
into deps/openssl/conf/archs/{target}/opensslconf.h.
deps/openssl/conf/openssconf.h includes each file according to its
target by checking pre-defined compiler macros. These can be generated
by using deps/openssl/conf/Makefile

We should remove OPENSSL_CPUID_OBJ define in opensslconf.h because it
causes build error when --openss-no-asm option is specified. Instead,
the OPENSSL_CPUID_OBJ is defined in deps/openssl/openssl.gypi
according to the configure options.

One fix of opensslconf.h is needed in 64-bit MacOS.
The current openssl release does not use RC4 asm since it explicitly
specified as $asm=~s/rc4\-[^:]+//; in
https://github.com/openssl/openssl/blob/OpenSSL_1_0_1-stable/Configure#L584
But iojs has used RC4 asm on MacOS for long time. Fix type of RC4_INT
into unsigned int in opensslconf.h of darwin64-x86_64-cc to work on
the RC4 asm.

5. Update openssl.gyp and openssl.gypi

Sources, cflags and define parameters that depends on each target can
be obtained via Configure TABLE. Its list is put in the table of
define and cflags changes in openssl-1.0.2a

There is no way to verify all necessary sources automatically. We can
only carefully look at the source list and compiled objects in
Makefile of openssl and compare the compiled objects that stored
stored under `out/Release/obj.target/openssl/deps/openssl/' in iojs.

6. ASM files for openssl

We provide two sets of asm files. One is for the latest asm compiler
and the other is the older one.

6.1. asm files for the latest compiler

This was made in deps/openssl/asm/Makefile

  • Updated asm files for each platforms which are required in
    openssl-1.0.2a.
  • Some perl files need CC and ASM envs. Added a check if these envs
    exist. Followed asm files are to be generated with CC=gcc and
    ASM=nasm on Linux. See
    deps/openssl/openssl/crypto/sha/asm/sha512-x86_64.pl
  • Added new 32bit targets/rules with a sse2 flag (OPENSSL_IA32_SSE2)
    to generate asm for use SSE2.
  • Generating sha512 asm files in x86_64 need output filename which
    has 512. Added new rules so as not to use stdout for outputs.
  • PERLASM_SCHEME of linux-armv4 is void as defined in openssl
    Configure. Changed its target/rule and all directories are moved
    from arm-elf-gas to arm-void-gas.
  • add a new rule for armv8 asm generation

With export environments of CC=gcc and ASM=nasm, then type make
command and check if new asm files are generated.

6.2.asm files for the older compiler

For older asm compiler, the version check of CC and ASM should be
skipped in generating asm file with perl scripts.
Copy files from deps/openssl/asm into
deps/openssl/asm/asm_obsolete and change rules to generate asm files
into this directories and remove the check of CC and ASM envs.

Without environments of CC and ASM, then type make command and check
if new asm files for older compilers are generated.

This PR fixes #589

@shigeki shigeki added the openssl Issues and PRs related to the OpenSSL dependency. label Apr 10, 2015
@shigeki shigeki self-assigned this Apr 10, 2015
@indutny
Copy link
Member

indutny commented Apr 10, 2015

Grammar (@bnoordhuis please double check me):

In certain situations the certificate chain provided by server may no
longer be valid. However the issuer of the leaf, or some intermediate
cert is in fact in the trust store.

@indutny
Copy link
Member

indutny commented Apr 10, 2015

So now you are generating asm files during the GYP build?

@indutny
Copy link
Member

indutny commented Apr 10, 2015

LGTM, @bnoordhuis PTAL too.

@bnoordhuis
Copy link
Member

In certain situations the certificate chain provided by server may no
longer be valid. However the issuer of the leaf, or some intermediate
cert is in fact in the trust store.

"In certain situations" is a little vague. I would write it as e.g.:

TLS servers sometimes present misconfigured or invalid certificate chains. That doesn't have to be a problem; as long as the server certificate is signed by a trusted certificate authority (CA), it will still be accepted.

@bnoordhuis
Copy link
Member

Left some comments but overall nice work, Shigeki!

@indutny
Copy link
Member

indutny commented Apr 10, 2015

Oh, looks like I forgot to say it... Thank you!

@bnoordhuis
Copy link
Member

By the way, I like how you made sure that the assembly files work with older toolchains but I think it's not unreasonable to bump the prerequisites if we have to jump through hoops every time to keep them working.

@shigeki shigeki force-pushed the upgrade_openssl102a branch 2 times, most recently from 74f6663 to a82def5 Compare April 11, 2015 14:43
@shigeki
Copy link
Contributor Author

shigeki commented Apr 11, 2015

Thanks for all reviewing my big patches. All fixes are made and Ci looks fine in https://jenkins-iojs.nodesource.com/job/iojs+any-pr+multi/488/
@indutny asm Makefile needs perl which is not requirements for iojs build. So I think we have to generate them beforehand.
@bnoordhuis The commit message of 5d789a7 comes from the original sentence and I just copied it. But author was changed to me because he is not aware of backporting. If the message or the author is improper, I will fix it. Please tell me it.
@jbergstroem White spaces are generated with openssl scripts.They are to be removed with git am --whitespace=fix at the time when they are merged.

If the updated commits have no problems, I will merge them after releasing v1.6.5. I will also summarize my benchmark results for updating 1.0.2a. After that, rootCA updates and RC4 removal with 1024-bit RSA deprecation will be submitted.

@shigeki
Copy link
Contributor Author

shigeki commented Apr 11, 2015

By the way, I like how you made sure that the assembly files work with older toolchains but I think it's not unreasonable to bump the prerequisites if we have to jump through hoops every time to keep them working.

@bnoordhuis Actually the current support for older toolchains are made to work in the CI of Ubuntu10/12 which have older binutils. And I found that clang in FreeBSD does not have a version banner of llvm so that assembler check was failed in to use obsoleted asm files. We can check them as long as the current CI cluster are used. I think we have to more clarify the build requirements of iojs with not only compiler versions but also assembler together with OS and CPU in order to deprecate asm_obsolete supports.

@bnoordhuis
Copy link
Member

LGTM

@shigeki
Copy link
Contributor Author

shigeki commented Apr 13, 2015

Minor typo: s/architectures/architecture/ (and perhaps the last part reads better as "for the linux_x86_64 target architecture".)
@bnoordhuis Thanks for checking typo. Fixed.
I also found a mistake of CI support of i386 Mac. The current CI only has x64 Mac so it was fixed.

@rvagg
Copy link
Member

rvagg commented Apr 13, 2015

I also found a mistake of CI support of i386 Mac. The current CI only has x64 Mac so it was fixed.

Happy to discuss this if it's seen as necessary, we could spin up VMs to run 32-bit tests. We're only shipping 64-bit binaries for OSX at the moment, not even universal binaries, and nobody has complained.

@shigeki
Copy link
Contributor Author

shigeki commented Apr 13, 2015

32-bit Mac is not necessary if it is out of our release distributions.
I would rather ask for 32-bit Windows for the future work to move masm to nasm/yasm. The current win2008r2/2012r2 on Ci runs on only 64-bit.

@rvagg
Copy link
Member

rvagg commented Apr 13, 2015

ah, yes, 64-bit only but we release as 32-bit, I guess we need to figure out how to best shoe-horn 32-bit in there somewhere

/cc @iojs/build

@indutny
Copy link
Member

indutny commented Apr 13, 2015

Let's land it?

@rvagg
Copy link
Member

rvagg commented Apr 14, 2015

wait until post 1.7.0 before landing, today if all's good

Shigeki Ohtsu and others added 6 commits April 14, 2015 23:48
This just replaces all sources in deps/openssl/openssl to originals in
https://www.openssl.org/source/openssl-1.0.2a.tar.gz
`x86masm.pl` was mistakenly using .486 instruction set, why `cpuid` (and
perhaps others) are requiring .686 .
This a backport of da084a5ec6cebd67ae27f2463ebe4a50bb840fa5 in
https://github.com/openssl/openssl by Matt Caswell <matt@openssl.org> as

In certain situations the server provided certificate chain may no
longer be valid. However the issuer of the leaf, or some intermediate
cert is in fact in the trust store.

When building a trust chain if the first attempt fails, then try to
see if alternate chains could be constructed that are trusted.

deps: backport openssl patch of alt cert chains 2

This a backport of 15dba5be6a4482a9ad7e5b846291f31e97e338ca in
https://github.com/openssl/openssl by Matt Caswell <matt@openssl.org> as

Add flag to inhibit checking for alternate certificate chains. Setting this
behaviour will force behaviour as per previous versions of OpenSSL
Change all openssl/include/openssl/*.h to include resolved symbolic
links and openssl/crypto/opensslconf.h to refer config/opensslconf.h
@MylesBorins MylesBorins mentioned this pull request Aug 17, 2018
rvagg pushed a commit that referenced this pull request Nov 23, 2018
`x86masm.pl` was mistakenly using .486 instruction set, why `cpuid` (and
perhaps others) are requiring .686 .

Fixes: #589
PR-URL: #1389
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Shigeki Ohtsu <ohtsu@iij.ad.jp>
rvagg pushed a commit that referenced this pull request Nov 23, 2018
See
https://mta.openssl.org/pipermail/openssl-dev/2015-February/000651.html

iojs needs to stop using masm and move to nasm or yasm on Win32.

Fixes: #589
PR-URL: #1389
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
rvagg pushed a commit that referenced this pull request Nov 23, 2018
Reapply b910613 .

Fixes: #589
PR-URL: #1389
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
rvagg pushed a commit that referenced this pull request Nov 23, 2018
`x86masm.pl` was mistakenly using .486 instruction set, why `cpuid` (and
perhaps others) are requiring .686 .

Fixes: #589
PR-URL: #1389
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Shigeki Ohtsu <ohtsu@iij.ad.jp>
rvagg pushed a commit that referenced this pull request Nov 23, 2018
See
https://mta.openssl.org/pipermail/openssl-dev/2015-February/000651.html

iojs needs to stop using masm and move to nasm or yasm on Win32.

Fixes: #589
PR-URL: #1389
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
rvagg pushed a commit that referenced this pull request Nov 23, 2018
Reapply b910613 .

Fixes: #589
PR-URL: #1389
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
rvagg pushed a commit that referenced this pull request Nov 24, 2018
`x86masm.pl` was mistakenly using .486 instruction set, why `cpuid` (and
perhaps others) are requiring .686 .

Fixes: #589
PR-URL: #1389
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Shigeki Ohtsu <ohtsu@iij.ad.jp>
rvagg pushed a commit that referenced this pull request Nov 24, 2018
See
https://mta.openssl.org/pipermail/openssl-dev/2015-February/000651.html

iojs needs to stop using masm and move to nasm or yasm on Win32.

Fixes: #589
PR-URL: #1389
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
rvagg pushed a commit that referenced this pull request Nov 24, 2018
Reapply b910613 .

Fixes: #589
PR-URL: #1389
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
shigeki pushed a commit to shigeki/node that referenced this pull request Feb 26, 2019
`x86masm.pl` was mistakenly using .486 instruction set, why `cpuid` (and
perhaps others) are requiring .686 .

Fixes: nodejs#589
PR-URL: nodejs#1389
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Shigeki Ohtsu <ohtsu@iij.ad.jp>
shigeki pushed a commit to shigeki/node that referenced this pull request Feb 26, 2019
See
https://mta.openssl.org/pipermail/openssl-dev/2015-February/000651.html

iojs needs to stop using masm and move to nasm or yasm on Win32.

Fixes: nodejs#589
PR-URL: nodejs#1389
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
shigeki pushed a commit to shigeki/node that referenced this pull request Feb 26, 2019
Reapply b910613 .

Fixes: nodejs#589
PR-URL: nodejs#1389
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
sam-github pushed a commit to sam-github/node that referenced this pull request Sep 5, 2019
`x86masm.pl` was mistakenly using .486 instruction set, why `cpuid` (and
perhaps others) are requiring .686 .

Fixes: nodejs#589
PR-URL: nodejs#1389
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Shigeki Ohtsu <ohtsu@iij.ad.jp>
sam-github pushed a commit to sam-github/node that referenced this pull request Sep 5, 2019
See
https://mta.openssl.org/pipermail/openssl-dev/2015-February/000651.html

iojs needs to stop using masm and move to nasm or yasm on Win32.

Fixes: nodejs#589
PR-URL: nodejs#1389
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
sam-github pushed a commit to sam-github/node that referenced this pull request Sep 5, 2019
Reapply b910613 .

Fixes: nodejs#589
PR-URL: nodejs#1389
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
BethGriggs pushed a commit that referenced this pull request Sep 19, 2019
`x86masm.pl` was mistakenly using .486 instruction set, why `cpuid` (and
perhaps others) are requiring .686 .

Fixes: #589
Backport-PR-URL: #28230
PR-URL: #1389
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Shigeki Ohtsu <ohtsu@iij.ad.jp>
BethGriggs pushed a commit that referenced this pull request Sep 19, 2019
See
https://mta.openssl.org/pipermail/openssl-dev/2015-February/000651.html

iojs needs to stop using masm and move to nasm or yasm on Win32.

Fixes: #589
Backport-PR-URL: #28230
PR-URL: #1389
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
BethGriggs pushed a commit that referenced this pull request Sep 19, 2019
Reapply b910613 .

Fixes: #589
Backport-PR-URL: #28230
PR-URL: #1389
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
@BethGriggs BethGriggs mentioned this pull request Sep 19, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
openssl Issues and PRs related to the OpenSSL dependency.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants