Initial VCPkg integration #4226

jediry · 2022-09-11T05:59:03Z

This change introduces the VCPkg builder integrating the vcpkg system (http://vcpkg.io) for acquiring and building 3rd party C/C++ libraries. This enables an SConstruct file to say, for example, "env.VCPkg('openjpeg')", and then vcpkg will handle downloading the source code and building it. The VCPkg() builder further ensures that the vcpkg include directory gets added to CPPPATH, and that its static lib directory gets added to LIBPATH.

One notable quirk of the VCPkg builder is that it does the actual package build during the initial execution of the SConstruct file, rather than occurring later, during SCons's normal build phase. While undesirable, this ensures that the VCPkg builder is able to accurately compute the set of files produced by building the package (as of this writing, vcpkg does not support any means to predict the files generated by building a package). I hope to improve this in the future.

Contributor Checklist:

I have created a new test or updated the unit tests to cover the new/changed functionality.
I have updated CHANGES.txt (and read the README.rst)
I have updated the appropriate documentation

… --silent mode.

…ersion. Also fix parsing of 'vcpkg depend-info'.

…eContents 'target' node

mwichmann · 2022-09-11T15:59:45Z

SCons/Tool/vcpkg.xml

+    <tool name="vcpkg">
+        <summary>
+            <para>
+                Sets construction variables for the &vcpkg; package manager


The generated entities for a tool named vcpkg will be &t-vcpkg; and &t-link-vcpgk; - thus this entity reference generated an error.

ERROR: SCons/Tool/vcpkg.xml fails to parse: Entity 'vcpkg' not defined, line 29, column 60 (vcpkg.xml, line 29)

mwichmann · 2022-09-11T16:03:23Z

SCons/Tool/VCPkgTests.py

@@ -0,0 +1,528 @@
+#
+# __COPYRIGHT__


For new files, please follow the templates in templates/file.py and templates/test.py - some files have not yet been converted, in particular Tool modules, so realize you probably got this form from an existing file...

bdbaddog · 2022-09-11T20:40:27Z

SCons/Tool/__init__.py

@@ -809,6 +809,8 @@ def tool_list(platform, env):
        'tar', 'zip',
        # File builders (text)
        'textfile',
+        # Package management
+        'vcpkg',


Seems like this should be a win32 only thing? Or does it work on all platforms?

Looks like win, linux, and macos. So it shouldn't be in default list. (BSD's not supported?, hpux, solaris, aix, other evil remnants of the past which are still in use(that's not referring to BSDs, but rather the others.. ;)

further review looks like your implementation is win32 only, so should be only in that platforms default tool list above. under other_plat_tools

My implementation is not Win32-only. I've tested on Ubuntu and Windows, and it should work on Mac as well, though I don't have the ability to test Mac, as I don't own one.

OK, so if I'm understanding correctly, I should conditionally add this to other_plat_tools on platforms where it's supported.

Yes. add it to other_plat_tools for each supported platform. Also I'd add a note which platforms it's supported on in the docs and CHANGES/RELEASE files.

Ummm...how do I detect "linux"? It seems like tools such as m4 and rpm are being added simply based on "not win32". Is there a positive test I can do to detect a linux system?

Ummm...how do I detect "linux"? It seems like tools such as m4 and rpm are being added simply based on "not win32". Is there a positive test I can do to detect a linux system?

Eh, that's a little complex. Depending on where you're asking... a simple way is to check sys.platform for linux. However, the name of the Platform module is actually posix, which is shared with some of the other choices (that originally comes from the value of os.name).

SCons/Tool/vcpkg.py

CHANGES.txt

mwichmann · 2022-09-11T20:52:51Z

Adding a reference to #4101 for tracking purposes.

jediry · 2022-09-11T22:47:21Z

@mwichmann @bdbaddog what are the expectations regarding code coverage? If I'm reading this correctly, my unit tests only exercise 41% of the lines of code. There are certainly more tests that could be written (I've left a TODO in the test file listing scenarios that I think would be reasonable to test), but 100% code coverage is not feasible.

bdbaddog · 2022-09-11T23:16:31Z

@mwichmann @bdbaddog what are the expectations regarding code coverage? If I'm reading this correctly, my unit tests only exercise 41% of the lines of code. There are certainly more tests that could be written (I've left a TODO in the test file listing scenarios that I think would be reasonable to test), but 100% code coverage is not feasible.

We don't have hard requirements on code coverage. That said if you can reasonable add more tests to cover the major code branches, please do.

mwichmann · 2022-09-12T14:04:19Z

SCons/Tool/vcpkg.py

+        """Returns the list of C/C++ header files belonging to the package.
+           If `transitive` is True, then files belonging to upstream dependencies of this package are also included."""
+        _vcpkg_print(Debug, str(self.descriptor) + ': headers')
+        files = self.FilesUnderSubPath('include/', transitive)


[Flake8] local variable 'files' is assigned to but never used (view)

Rule

F841

You can close this issue if no need to fix it. Learn more.

Looks like duplicated work, maybe just drop this line?

mwichmann · 2022-09-12T14:06:24Z

SCons/Tool/vcpkg.py

+            self.files = _read_vcpkg_file_list(self.env, self)
+            self.loaded = True
+
+        triplet = self.descriptor.get_triplet()


[Flake8] local variable 'triplet' is assigned to but never used (view)

Rule

F841

You can close this issue if no need to fix it. Learn more.

get_triplet called, then called again. maybe delete this line, or use triplet in the next?

bdbaddog · 2022-09-12T15:12:37Z

Probably worth addressing all the sider complaints.

jediry · 2022-09-18T20:18:07Z

SCons/Tool/VCPkgTests.py

+    # State modification functions used by contextmanager functions below
+    #
+
+    def addAvailablePackage(self, name, version, dependencies):


[Flake8] expected 1 blank line, found 0 (view)

Rule

E301

You can close this issue if no need to fix it. Learn more.

I don't understand this flake8 complaint. Is it asking me to put a blank line after "def addAvailablePackage(...):"?

If you google "flake8 e301" (The error indicated), you'll usually find a decent explanation.
see: https://www.flake8rules.com/rules/E301.html (for this case)

I did find that page...but it was unhelpful in this case. As you can see, there is a blank line before "def addAvailablePackage" and a blank line after the end of the function. So I don't know what to do to appease this rule...

We can mark it to skip, I don't see anything there to complain about

I closed a couple of the sider issues. The remaining ones are suggesting using def instead of lambda. @mwichmann - thoughts?

I don't personally have a strong opinion. The thinking these days seems to be if you want to use an anonymous function do it; if you're going to assign a name to it then use a real function instead - that's pretty much what the error message says anyway. lambdas work well for things like sort keys, so they definitely still have a place, it's just a smaller place than it used to be.

My inclination is that using "def" here is the wrong thing. I'm not just using the lambda to avoid declaring an extra function. I'm also using the fact that lambdas can capture state from the surrounding context (in this case, the suffix_filters variable). I would vote to leave this as-is.

If you feel strongly, the other options I see are:

Convert this into a single def, containing the entire "if...else" block on lines 527-532. Slightly less performant in the case where the user wants to enumerate all files without a filter, but that's probably not the common case.

Convert this into 3 defs (one for each case), assign the appropriate one to a local var, and then pass the suffix_filters variable as an argument when invoking. Feels clunky to me.

Let me know what you prefer.

…le by multiple paths through the dependency graph get their files reported multiple times.

…pass unit tests

mwichmann · 2022-09-25T19:01:15Z

Sigh. The doc build failure is one I started to run into as well. I was able to stave it off by backing out an example - that was the thing that triggered it, but there was nothing wrong with my snippets, it just someone was one piece too much for lxml (see #4235). In other words, it's going to be tricky to figure out what's going wrong here and in that PR.

jediry · 2022-09-25T20:24:07Z

Bummer. Also, on a related note, I noticed that the SconsDoc stuff tries to load lxml, and if that fails, tries to fall back to a different implementation, but that implementation apparently doesn't support XSD schema validation? In any case, it falls down pretty hard, with no clue on how to fix it. I finally figured out I needed to "pip install lxml". IMO, we should rip out the fallback XML parser and instead have a friendly failure message to the person running SconsDoc that they need to install lxml. I'll log an issue on this.

jediry · 2022-09-25T20:26:07Z

Bummer. Also, on a related note, I noticed that the SconsDoc stuff tries to load lxml, and if that fails, tries to fall back to a different implementation, but that implementation apparently doesn't support XSD schema validation? In any case, it falls down pretty hard, with no clue on how to fix it. I finally figured out I needed to "pip install lxml". IMO, we should rip out the fallback XML parser and instead have a friendly failure message to the person running SconsDoc that they need to install lxml. I'll log an issue on this.

The "new issue" text tells me to raise this on Discord or the mailing list first. Should I log this issue?

bdbaddog · 2022-09-25T20:35:53Z

Bummer. Also, on a related note, I noticed that the SconsDoc stuff tries to load lxml, and if that fails, tries to fall back to a different implementation, but that implementation apparently doesn't support XSD schema validation? In any case, it falls down pretty hard, with no clue on how to fix it. I finally figured out I needed to "pip install lxml". IMO, we should rip out the fallback XML parser and instead have a friendly failure message to the person running SconsDoc that they need to install lxml. I'll log an issue on this.

The "new issue" text tells me to raise this on Discord or the mailing list first. Should I log this issue?

Please bring it to the users mailing list or the discord server.. :)

mwichmann · 2022-09-25T22:50:52Z

So, yes, there was a transitional phase... lxml was the "second choice" in the Python 2 days, but the other approach hasn't been supported for a while, and doesn't wotk on Py3. Not everything relating to "the old days" has been ripped out completely, just because... we're (somewhat) human, I guess. There's a script to set up for development (bin/scons_dev_master.py) which is documented somewhere, and which also has an error in it, so it will blow up after installing the deps - just found that the other day. Meanwhile, in the current source tree (not in any released version) there are now three requirements files: the base one for running scons, requirements-dev.txt which adds the stuff you need basically to run the full test suite, and requirements-pkg.txt, which is "everything" - basically adding the stuff needed to build the docs. lxml is in the requirements-dev.txt since it's used by some of the docbook and xml tests.

mwichmann · 2022-09-25T22:51:55Z

Oh - and scons_dev_master is only set up for Debian-ish systems.

jediry · 2022-10-11T08:50:06Z

Hmmm...so maybe add this to an "else if posix" clause at the end of the "if hpux...else if aix...else if darwin..." structure? Sent from ProtonMail mobile

…

-------- Original Message --------

On Sep 18, 2022, 9:19 AM, Mats Wichmann wrote: @mwichmann commented on this pull request. --------------------------------------------------------------- In [SCons/Tool/__init__.py](#4226 (comment)): > @@ -809,6 +809,8 @@ def tool_list(platform, env): 'tar', 'zip', # File builders (text) 'textfile', + # Package management + 'vcpkg', > Ummm...how do I detect "linux"? It seems like tools such as m4 and rpm are being added simply based on "not win32". Is there a positive test I can do to detect a linux system? Eh, that's a little complex. Depending on where you're asking... a simple way is to check sys.platform for linux. However, the name of the Platform module is actually posix, which is shared with some of the other choices (that originally comes from the value of os.name). — Reply to this email directly, [view it on GitHub](#4226 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AAMZQHO36VQLNK3633VLEB3V646JXANCNFSM6AAAAAAQJUGTPE). You are receiving this because you authored the thread.Message ID: ***@***.***>

jediry · 2022-12-15T07:48:17Z

Sigh. The doc build failure is one I started to run into as well. I was able to stave it off by backing out an example - that was the thing that triggered it, but there was nothing wrong with my snippets, it just someone was one piece too much for lxml (see #4235). In other words, it's going to be tricky to figure out what's going wrong here and in that PR.

Hi @mwichmann . I'm finally getting back to this project. Since the doc generation is blocking the PR, I think I might take a stab at fixing that problem. Have you done any investigation on this issue yet? My suspicion is that this might be an out-of-memory issue, either in Python or in FOP (a Java tool, IIRC). I'm wondering if it might be as simple as allocating more "max" heap memory to the JVM that runs FOP. Also, have you been able to get a local repro for this issue? I tried to repro it myself on my Ubuntu WSL virtual machine back in October, but as I recall, I was not able to hit this, at least not on the few attempts I've made.

mwichmann · 2022-12-15T13:24:41Z

The doc build part appears to be a built-in limit that turns out to now be tunable thanks to someone's efforts about six months ago. I bumped the limit again for #4263, as those changes pushed me into the "trouble" area, but that has not been merged yet. Whether it triggers does seem to be situational. In any case, if you want, you can try the same bit:

diff --git a/SCons/Tool/docbook/__init__.py b/SCons/Tool/docbook/__init__.py
index 5cf5e6198..52e291144 100644
--- a/SCons/Tool/docbook/__init__.py
+++ b/SCons/Tool/docbook/__init__.py
@@ -69,7 +69,7 @@ re_refname = re.compile(r"<refname>([^<]*)</refname>")
 # lxml etree XSLT global max traversal depth
 #
 
-lmxl_xslt_global_max_depth = 3100
+lmxl_xslt_global_max_depth = 3600
 
 if has_lxml and lmxl_xslt_global_max_depth:
     def __lxml_xslt_set_global_max_depth(max_depth):

mwichmann · 2022-12-15T13:51:00Z

Bummer. Also, on a related note, I noticed that the SconsDoc stuff tries to load lxml, and if that fails, tries to fall back to a different implementation, but that implementation apparently doesn't support XSD schema validation? In any case, it falls down pretty hard, with no clue on how to fix it. I finally figured out I needed to "pip install lxml". IMO, we should rip out the fallback XML parser and instead have a friendly failure message to the person running SconsDoc that they need to install lxml. I'll log an issue on this.

FWIW, there's now a straightforward way to get all the deps in place if you intend to "build scons", which includes building the doc set:

pip install -r requirements-pkg.txt

jediry · 2023-01-16T22:04:08Z

The doc build part appears to be a built-in limit that turns out to now be tunable thanks to someone's efforts about six months ago. I bumped the limit again for #4263, as those changes pushed me into the "trouble" area, but that has not been merged yet. Whether it triggers does seem to be situational. In any case, if you want, you can try the same bit:
diff --git a/SCons/Tool/docbook/__init__.py b/SCons/Tool/docbook/__init__.py
index 5cf5e6198..52e291144 100644
--- a/SCons/Tool/docbook/__init__.py
+++ b/SCons/Tool/docbook/__init__.py
@@ -69,7 +69,7 @@ re_refname = re.compile(r"<refname>([^<]*)</refname>")
 # lxml etree XSLT global max traversal depth
 #
 
-lmxl_xslt_global_max_depth = 3100
+lmxl_xslt_global_max_depth = 3600
 
 if has_lxml and lmxl_xslt_global_max_depth:
     def __lxml_xslt_set_global_max_depth(max_depth):

OK! I've added this change (I upped it to 4000 for good measure), and that seems to have fixed the issue for me. Thank you!

bdbaddog · 2023-10-04T16:46:11Z

@jediry - Sorry for the delay. Finally got a big chunk of time to review this and read up on vcpkg and (hopefully) better understand.

Let me ask you a fundamental question about the goal of this functionality.
Is it really:

Check and see if this package is already installed, and if not install it and configure scons to be able to use it's libraries and header files (and binaries)?
Build this package as part of my build?

I'm thinking it's really #1?

jediry · 2023-10-04T18:07:28Z

I'm thinking it's really #1?

Correct. Though, if the specified package is not currently installed, its binaries will be built as part of the install process, so #2 kind of applies as well.

jediry added 12 commits February 25, 2022 23:12

Initial implementation of vcpkg builder

217ad69

Respect --silent in vcpkg spew and support a --vcpkg-debug flag

cccb361

Fix break caused by reading old-style CONTROL files. Tweak output for…

d3e983b

… --silent mode.

Avoid re-downloading vcpkg.exe from GitHub when we have the correct v…

b5f070c

…ersion. Also fix parsing of 'vcpkg depend-info'.

Merge branch 'master' into vcpkg

52de2a6

Enable enumeration of package contents in SConstruct file, via Packag…

282785e

…eContents 'target' node

Fix faulty use of AppendUnique

12d1fdd

Updates to VCPkg documentation and tests

705be7b

Merge remote-tracking branch 'upstream/master'

16b87e6

Merge branch 'master' into vcpkg

a8fecc5

Significantly expanded unit tests

a7d5fd8

Added CHANGES.txt entry

a560ee0

mwichmann reviewed Sep 11, 2022

View reviewed changes

mwichmann added enhancement Tools Issues related to tools subsystem labels Sep 11, 2022

mwichmann reviewed Sep 11, 2022

View reviewed changes

bdbaddog reviewed Sep 11, 2022

View reviewed changes

SCons/Tool/vcpkg.py Show resolved Hide resolved

bdbaddog reviewed Sep 11, 2022

View reviewed changes

CHANGES.txt Outdated Show resolved Hide resolved

Updated description in CHANGES.txt and replicated in RELEASE.txt

7ce7088

mwichmann reviewed Sep 12, 2022

View reviewed changes

jediry added 2 commits September 18, 2022 08:36

Updated file headers to match current templates.

2f3c24f

Only init vcpkg tool on supported platforms (Win/MacOS/Linux)

70fef43

jediry commented Sep 18, 2022

View reviewed changes

Fix code review feedback and add tests for package contents

20ce478

jediry added 8 commits September 18, 2022 22:52

Fix a flaw in enumerating package contents, in which packages reachab…

2ffe5d4

…le by multiple paths through the dependency graph get their files reported multiple times.

Add disabled test for the list-of-packages case

076d2d9

Fix multi-package case. Update documentation.

4ec11b5

Add missing XML xlink namespace

8e22ef5

Fix missed variable rename

93630d3

Convert VCPkgBuilder to its own class overriding __eq__, in order to …

93233b0

…pass unit tests

Fix naming of cv- link

9147eab

Fix doc generation

7e7727f

Merge remote-tracking branch 'upstream/master' into vcpkg

3b89b13

jediry added 2 commits January 16, 2023 13:42

Merge remote-tracking branch 'upstream/master' into vcpkg

ce40138

Update xml traversal limit

a7f5858

Initial VCPkg integration #4226

Are you sure you want to change the base?

Initial VCPkg integration #4226

Conversation

jediry commented Sep 11, 2022

Contributor Checklist:

mwichmann Sep 11, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bdbaddog Sep 11, 2022 • edited

Choose a reason for hiding this comment

jediry Sep 11, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mwichmann commented Sep 11, 2022

jediry commented Sep 11, 2022

bdbaddog commented Sep 11, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bdbaddog commented Sep 12, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mwichmann commented Sep 25, 2022

jediry commented Sep 25, 2022

jediry commented Sep 25, 2022

bdbaddog commented Sep 25, 2022

mwichmann commented Sep 25, 2022

mwichmann commented Sep 25, 2022

jediry commented Oct 11, 2022 via email

jediry commented Dec 15, 2022

mwichmann commented Dec 15, 2022

mwichmann commented Dec 15, 2022

jediry commented Jan 16, 2023 • edited

bdbaddog commented Oct 4, 2023

jediry commented Oct 4, 2023

mwichmann Sep 11, 2022 •

edited

bdbaddog Sep 11, 2022 •

edited

jediry Sep 11, 2022 •

edited

bdbaddog commented Sep 12, 2022 •

edited

jediry commented Jan 16, 2023 •

edited