Move to sphinx-doc

[AA-Turner]

Hi Brian,

Would you consider moving this project to @sphinx-doc? I'd like to integrate support for intersphinx lookup into core Sphinx, and it makes sense not to duplicate efforts.

Regardless of your thoughts, thank you for the great project!

Thanks,
Adam

[bskinn]

Thanks for the kind words, @AA-Turner!

I'm definitely interested in discussing moving the project under sphinx-doc, with a final decision depending quite a bit on the exact shape of things.

Completely agree about not duplicating efforts. But, sphobjinv is more than just lookup/search into objects.inv files, and I'm not sure if all of those other pieces of the project would be most naturally housed in sphinx-doc. So, e.g., it might make more sense to slice out some of the pieces of functionality you're wanting to bring into core Sphinx, and then sphobjinv would migrate over to using those sphinx-core pieces over time. Depends a lot on what you have in mind.

Other questions that come to mind include aspects of ownership/governance, and technical direction. For example, one of my big goal items is a plugin system to let users write and use custom suggest scorers (see #207). If suggest were to become a core Sphinx feature, then those scorers could be wired in using Sphinx's existing extensions system, which (at the moment, at least) seems very attractive.

I'm guessing this overture is coming after some internal discussion on the part of the Sphinx team -- if so, is that also something I could learn more about before making a decision?

Happy to find a time for a call if that would be more efficient. Else, async in the thread here is good too.

Thanks again!

[AA-Turner]

Completely agree about not duplicating efforts. But, sphobjinv is more than just lookup/search into objects.inv files, and I'm not sure if all of those other pieces of the project would be most naturally housed in sphinx-doc. So, e.g., it might make more sense to slice out some of the pieces of functionality you're wanting to bring into core Sphinx, and then sphobjinv would migrate over to using those sphinx-core pieces over time. Depends a lot on what you have in mind.

We have several changes planned for the inventory format in general (see sphinx-doc/sphinx#8929, sphinx-doc/sphinx#10128, sphinx-doc/sphinx#10127), so part of this is trying to future-proof a v3 serialised inventory format and the v2 internal representation, etc.

As part of the new sphinx command work, a possibility (especially if sphobjinv is more closely integrated) is a new sphinx inventory or sphinx inv sub-command, which would likely as a minimum contain the suggest functionality, but may also expose the inventory editing features sphobjinv currently has.

Other questions that come to mind include aspects of ownership/governance, and technical direction. For example, one of my big goal items is a plugin system to let users write and use custom suggest scorers (see #207). If suggest were to become a core Sphinx feature, then those scorers could be wired in using Sphinx's existing extensions system, which (at the moment, at least) seems very attractive.

I would envision (at least to start) very little changing from the sphobjinv user's perspective -- the changes would be that the Sphinx team has commit access to the repo, and changes in the inventory format can be reflected when made upstream.

Another option is full integration into the core sphinx-doc/sphinx repo, which I would see as less likely (if nothing else as Sphinx is BSD-2-Clause and sphobjinv is MIT, and it would make licencing complicated!).

I'm guessing this overture is coming after some internal discussion on the part of the Sphinx team -- if so, is that also something I could learn more about before making a decision?

Happy to find a time for a call if that would be more efficient. Else, async in the thread here is good too.

I've got a very busy few weeks (& as such I'm taking a break from GH), but could speak at the end of April / start of May, if useful?

Thanks,
Adam

[bskinn]

I've got a very busy few weeks (& as such I'm taking a break from GH), but could speak at the end of April / start of May, if useful?

No problem to keep this async for a while. I don't specifically see a need for a chat in the short term.

On the topic of time, can I ask—what's the timeline on all of these improvements? sphobjinv dates from my earlier days as a Python developer, and as such there's quite a bit in there that could really use cleanup. Would that cleanup need to happen before a move under the sphinx-doc umbrella? Some of the kludgy stuff might interfere with integration—e.g., sphobjinv desperately needs its own custom exception hierarchy (#118), and the CLI code is not pretty. Also, I've got a major docs rework on the radar (#270). I don't know how quickly I would be able to execute any cleanup plans, whether my own or developed in tandem with the Sphinx team.

---

As part of the new sphinx command work, a possibility (especially if sphobjinv is more closely integrated) is a new sphinx inventory or sphinx inv sub-command, which would likely as a minimum contain the suggest functionality, but may also expose the inventory editing features sphobjinv currently has.

...

I would envision (at least to start) very little changing from the sphobjinv user's perspective -- the changes would be that the Sphinx team has commit access to the repo, and changes in the inventory format can be reflected when made upstream.

Another option is full integration into the core sphinx-doc/sphinx repo, which I would see as less likely (if nothing else as Sphinx is BSD-2-Clause and sphobjinv is MIT, and it would make licencing complicated!).

If direct integration of sphobjinv's Inventory, suggest, convert, etc. functionality into Sphinx core—which truly would avoid the need to make coupled edits in multiple places—isn't the primary intent, due to the license collision... what are the functional advantages of pulling sphobjinv under the sphinx-doc umbrella? Without the sphobjinv code living directly in the Sphinx codebase, it's not clear to me how Sphinx can leverage sphobjinv's machinery other than by following a typical import-and-use pattern. And for that, the project relocation doesn't seem necessary to me.

Along these same lines, if the main thing is providing Sphinx maintainers with commit bits, wouldn't granting them commit bits on bskinn/sphobjinv serve the same purpose?

Following the train of thought further... logistically, wouldn't I need to be involved with most or all of the changes to sphobjinv machinery, at least at first, to help make sure that those will do the right things, and integrate correctly with the surrounding codebase? It seems to me that starting out with a standard open source PR flow for changes to sphobjinv would be a more cautious, lower-impact path, versus granting direct commit rights. (I've had negative effects on the state of the codebase result from an otherwise positive interaction, after granting a commit bit to a collaborator on a different project, so I'm hesitant to open that door too quickly.)

I'm not that familiar with how GitHub organizations work, so I may well be ignorant of some obvious benefits. If nothing else, I could see some potential advantages in terms of logistical tooling—gaining access to sphinx-doc's user roles, teams, etc. Perhaps there's more there than what I know of—I'm happy to be educated.

Ultimately, I think it boils down to wanting to learn more about what to expect from this experience and about what I would be opening the codebase and the project to, and wanting to get to know the team I'd be de facto joining a bit better, before making a decision.

[AA-Turner]

A quick reply to some key points:

What's the timeline on all of these improvements? ... Would that cleanup need to happen before a move under the sphinx-doc umbrella?

Timeline: perhaps Summer 2023? The tentative release timeline is Sphinx 6.2 around early May, and Sphinx 7.0 shortly afterwards (only removing deprecations).

I'd be happy to help with any clean-up, as you feel is appropriate.

If direct integration of sphobjinv's Inventory, suggest, convert, etc. functionality into Sphinx core—which truly would avoid the need to make coupled edits in multiple places—isn't the primary intent, due to the license collision... what are the functional advantages of pulling sphobjinv under the sphinx-doc umbrella? Without the sphobjinv code living directly in the Sphinx codebase, it's not clear to me how Sphinx can leverage sphobjinv's machinery other than by following a typical import-and-use pattern. And for that, the project relocation doesn't seem necessary to me.

Along these same lines, if the main thing is providing Sphinx maintainers with commit bits, wouldn't granting them commit bits on bskinn/sphobjinv serve the same purpose?

Following the train of thought further... logistically, wouldn't I need to be involved with most or all of the changes to sphobjinv machinery, at least at first, to help make sure that those will do the right things, and integrate correctly with the surrounding codebase? It seems to me that starting out with a standard open source PR flow for changes to sphobjinv would be a more cautious, lower-impact path, versus granting direct commit rights. (I've had negative effects on the state of the codebase result from an otherwise positive interaction, after granting a commit bit to a collaborator on a different project, so I'm hesitant to open that door too quickly.)

A partial reply to all of these points:

1. I admit I haven't thoroughly reviewed the code, and didn't realise that true integration would be the 'best' way of de-duplicating efforts. If so (and conversely that a simple move would offer no real benefits), we could try and focus on that aspect.

2. Following from (1), There are four commits not by you in this repo¹ (b2d2e18, dba67e3, 359a645, and e3b7938). The first two are trivial enough so as not to be copyrightable, the latter two are more questionable -- therefore (if you want to) you might be able to re-licence the repo to BSD-2-Clause either unilaterally or with Oscar's agreement -- should we want to integrate properly.

3. A possible halfway-house position (as you alluded to in your original reply) would be to integrate the 'suggest' functionality into Sphinx, and explore providing better APIs for the 'convert' subcommand.

4. The benefits I see of the organisation are that should any one person be suddenly indisposed or unavailable, others can continue maintainership -- I entirely agree though that this can be achieved by granting users write/collaborator permissions on this repository.

5. The downside of integration would be that improvements become coupled to versions of Sphinx, and if one is using an older version then the benefits aren't seen.

Thanks,
Adam

Footnotes

1. git log --perl-regexp --author='^((?!Brian Skinn).*)$' ↩

[bskinn]

Sorry for the long delay on this, @AA-Turner—whole bunch of stuff, including PyCon US, hung me up.

In any event, while mulling this over, I thought of something that makes integration of sphobjinv into Sphinx core a non-starter.

I've built sphobjinv specifically to have a light install footprint, since it's working with a static artifact published by Sphinx, and not actual Sphinx source or live Sphinx data structures. It doesn't need Sphinx installed to function, and I know of multiple uses of sphobjinv without Sphinx present. One example is inventory-search chatbots for Discord and such.

I don't want to have to install Sphinx everywhere I want to use sphobjinv, and I'm confident that these users won't want to either.

On top of this, I agree—the coupling of Sphinx and sphobjinv release cadences would almost certainly be undesirable.

---

Given these blockers to integration, I don't see any particular benefits to moving sphobjinv under sphinx-doc at this time. I truly appreciate the offer and what it signifies in terms of the perceived value of the project. But, for now, I think the best course forward is for Sphinx core devs to collaborate with sphobjinv by submitting PRs. While my independent development pace on sphobjinv is bursty but overall slow, I'll be able and motivated to work on any time-critical external contributions in prompt fashion.

---

That said, I wonder if vendoring might be a viable ~middle path for integration. I don't know what Sphinx's position is on vendoring dependencies, but it seems like it might work ok.

It would lock a given version of Sphinx to a given version of sphobjinv... but that actually might provide more benefits than downsides. It would freeze the sphobjinv API exposed internally to Sphinx while not pinning the sphobjinv version installed in a containing environment. And, if I understand correctly how Python imports work, a vendored sphobjinv inside Sphinx would not import-collide with a sphobjinv in the environment, because they would be imported from different locations.

I'd be willing to refactor the intra-project imports to be relative in future releases, to simplify vendoring.

I'd also be willing to go to the work of a license change, if it were needed; but I'd think it would suffice to include the MIT LICENSE.txt in the directory tree with the vendored code?

---

I recognize either of these approaches leaves some questions/challenges in place. One you've mentioned; the other may not be a big deal but bears considering:

The bus factor on sphobjinv

I'm willing to work toward bringing one or more Sphinx maintainers on as sphobjinv maintainer(s), but I would insist on it following a ~typical arc: a series of successful PRs leading to a good working rapport, and then a commit bit.

Circular imports in testing

While sphobjinv doesn't use Sphinx directly in its functionality, its test suite does import Sphinx. If Sphinx takes on sphobjinv as a dependency, then the risk emerges for circular imports in sphobjinv's test suite. I'm not sure what the likelihood of this problem emerging is, or how challenging fixes would be, but it's something to keep in mind.

One approach to avoid this possibility would be to have Sphinx only declare sphobjinv as part of an extras_require dependency set. sphobjinv could then install Sphinx without this extra in testing environments.