documenting punctuation manipulation rules for citations

[bdarcus]

I don't believe this is in the docs ATM.

As it just came up in discussions around the new org-mode citation support, it strikes me we should.

https://lists.gnu.org/archive/html/emacs-orgmode/2021-04/msg00497.html

Also, see this issue over at pandoc, which I think suggests this should be configurable?

jgm/pandoc-citeproc#256

@fbennett @bwiernik @denismaier @jgm

[denismaier]

Question is if this falls within the scope of CSL proper. While it is possible to support these things in a batch workflow (i.e., with org or pandoc), I don't think there will be proper support for this in interactive environments as a citeproc won't know anything about the context of a given citation.

OTOH, we can of course add some hints to the docs how these things should happen in systems where it is possible.

Regarding configurability: Yes, definitely.

There's one major caveat: I actually think the discussion shouldn't be about punctuation manipulation for note-based styles but the other way round. The way it's currently implemented in pandoc is a bit problematic as it results in semantical loss. I've the described the issue here, and I've also written a lua filter to circumenvent the problem.

Let's start with a citation in this form:

"A simple quote" [@doe].

This will result in:

"A simple quote" (Doe 2021).
Or, with a note style:
"A simple quote."[^1]

While this is perfectly acceptable in American English, it is not in German, or even in British English. Here we have to know whether the final period is part of the original quotation. If yes, it will be put inside the quotes, otherwise outside, so:

1. Quotation ending with a period:

"A simple quote."[^1]

2. Quotation not ending with a period:

• Variant a)

  "A simple quote".[^1]

• Variant b)

  "A simple quote"[^1].

Doing the punctuation manipulation from in-text to note-based styles is impossible to do correctly without additional markup. Going from note-based styles to the in-text version is much easier as you'd just have to pull punctuation out of a quotation.

[bdarcus]

I don't think there will be proper support for this in interactive environments as a citeproc won't know anything about the context of a given citation.

Obviously we'd want to confirm this.

I actually think the discussion shouldn't be about punctuation manipulation for note-based styles but the other way round.

Well, it probably needs to not assume either.

I'm in a field that is predominantly author-date, and occasionally note.

So not surprisingly, I'm going to prefer the former.

But it works the other way too, for people more firmly in the humanities.

But any documentation can clarify all that, much like we do with title vs sentence casing for titles.

[denismaier]

I'm in a field that is predominantly author-date, and occasionally note.

So not surprisingly, I'm going to prefer the former.

To be clear, as far as aestetics are concerned I also prefer the former. My argument is solely based on semantics and language conventions. If you want to convert between author-date and note-styles without semantic loss and without additional markup, I'd say the only way is to write as if you were using notes. (With LaTeX there's the csquotes package that goes at great lengths to make these kind of transformations possible. But I don't think you'll want to introduce that kind of syntactic overhead.

[bdarcus]

Generalized the issue description.

[bwiernik]

This issue has been discussed a few times over the years. There are several big issues:

1. The interaction of quotation marks, final punctuation, and citation markers (parentheticals or notes) is complex and highly dependent on document context in a way that isn't accessible to many citation processor contexts (eg, confirmed for Word, LO, or Google Docs).

2. There are 1000s of author-date styles written assuming citations contain only the parenthetical and not, eg, a preceding space. There are many many existing documents written with that structure in mind. Changing it would be extremely disruptive.

3. This isn't unique to CSL--most (all?) other citation managers similarly don't try to manage this between in text and note styles.

I think this is a design decision that is cast and way more construct to change to any benefit. There are almost always some edits needed when changing between a note and in text style, even if it is mostly automated. I don't think we can conceivably eliminate that.

[georgd]

3. This isn't unique to CSL--most (all?) other citation managers similarly don't try to manage this between in text and note styles.

Citavi (in my very personal view the only real competitor to Zotero in terms of usability and features) does the following (in Word):

• Citation entry works much like with most other reference managers: it adds the footnote if a note style and it doesn't add extra space before a parenthetical citation.
• When changing between a parenthetical and a note style, there's an extra step. Besides changing styles, the user has to explicitly operate a plugin functionality that switches reference positions. This process will remove a preceding space when switching from in-text to note position and it adds one she switching from note to in-text.

[denismaier]

When changing between a parenthetical and a note style, there's an extra step. Besides changing styles, the user has to explicitly operate a plugin functionality that switches reference positions. This process will remove a preceding space when switching from in-text to note position and it adds one she switching from note to in-text.

Will citavi move punctuation? Don't think so. But that could be easily accomplished with a macro.
That aside, Citavi is closer connected to word than Zotero will ever be, which makes things posdible that won't work with Zotero.

[bdarcus]

• The interaction of quotation marks, final punctuation, and citation markers (parentheticals or notes) is complex and highly dependent on document context in a way that isn't accessible to many citation processor contexts (eg, confirmed for Word, LO, or Google Docs).
• There are 1000s of author-date styles written assuming citations contain only the parenthetical and not, eg, a preceding space. There are many many existing documents written with that structure in mind. Changing it would be extremely disruptive.

To be clear, I would not advocate any change to styles to address this.

But while there may be barriers to adding this functionality to interactive GUI apps, I do think batch processors need to do it. Expecting people to edit final output documents in that context, which could be PDFs, is not practical, and both pandoc and citeproc-org do it already.

Maybe we could put in a "note" somewhere that explains the issue, without hard rules or recommendations?

[georgd]

When changing between a parenthetical and a note style, there's an extra step. Besides changing styles, the user has to explicitly operate a plugin functionality that switches reference positions. This process will remove a preceding space when switching from in-text to note position and it adds one she switching from note to in-text.

Will citavi move punctuation? Don't think so. But that could be easily accomplished with a macro.

That’s right, it doesn’t.

That aside, Citavi is closer connected to word than Zotero will ever be, which makes things posdible that won't work with Zotero.

Is it really that much closer connected? They surely concentrate much more on Word, not offering any other direct integration, but technically, I’m not sure, as they are now offering Word for Mac support too.

[bdarcus]

Will citavi move punctuation? Don't think so. But that could be easily accomplished with a macro.

Seems like GUI apps should include such a macro, with a configuration option.

[denismaier]

Is it really that much closer connected? They surely concentrate much more on Word, not offering any other direct integration, but technically, I’m not sure, as they are now offering Word for Mac support too.

See citation-style-language/schema#68 (comment)

According to @adomasven the integration is indeed tighter. Citavi can do things like suppressing ibids on the first citation on a page or in a paragraph, which won't be posdible with Zotero.

[bwiernik]

But while there may be barriers to adding this functionality to interactive GUI apps, I do think batch processors need to do it. Expecting people to edit final output documents in that context, which could be PDFs, is not practical, and both pandoc and citeproc-org do it already.

Maybe we could put in a "note" somewhere that explains the issue, without hard rules or recommendations?

Can you give an example of what pandoc or citeproc-org do? In my perspective, the editing situation isn't that different from GUI apps. If I were writing a document with a note style and had the citations entered like: Something profound.[@author2006] and then wanted to change to an author-date style, I would change it to something like: Something profound [@author2006]. This would happen in the source code, not the output PDF.

A program could implement some sort of macro or similar to rearrange the spaces and punctuation, but I think that would be outside of the scope of CSL at this point. If we wanted to add a note to implementers, that would be as far as I think we should go.

[denismaier]

Can you give an example of what pandoc or citeproc-org do?

In pandoc you write your citations like this:

Bla Bla [@doe2020].
Or:
"Bla Bla" [@doe2020].

Now, with a author-date style this will render as:

Bla Bla (Doe 2020).
"Bla Bla" (Doe 2020).

When you switch to a note style, pandoc will move the punctuation (unless you disable that):

Bla Bla.^[Doe 2020.]
"Bla Bla."^[Doe 2020.]

See https://pandoc.org/MANUAL.html#citations-in-note-styles

[jgm]

Yes, it's a design decision (in pandoc) to support seamless switching between different styles (including between author-date and note styles) without modifying anything in the source itself.

[bdarcus]

By way of update, one of the lead org developers posted POC code for this, which @denismaier has noted implements how pandoc does this.

https://lists.gnu.org/archive/html/emacs-orgmode/2021-04/msg00882.html

Denis and I are going to try to work out a suggested alternative to this on this google doc, if anyone is curious or wants to help (I'll probably remove this link once it's done).

@jgm - if we figure this out, it would address jgm/pandoc-citeproc#256 also if you adopt it.

Basically, what I'm thinking is the code to do this (in this case in org) needs two parameters:

1. move-punctuation
2. punctuation-inside-quotes

If we're happy with the result, we can adapt it for inclusion as a note in CSL.

BTW, notwithstanding any technical challenges of implementing this sort of thing in a GUI context, the lack of support for this is almost surely a PITA for anyone that does need to switch between in-text and note styles. It's really only not an issue if you exclusively work in one or the other.

[bdarcus]

Basically, what I'm thinking is the code to do this (in this case in org) needs two parameters:

In the (almost) end, looks to be three.

https://lists.gnu.org/archive/html/emacs-orgmode/2021-05/msg00714.html

[bdarcus]

And org and Emacs now have CSL support!

https://lists.gnu.org/archive/html/emacs-orgmode/2021-05/msg01112.html

[bwiernik]

@bdarcus about your question on list locators, pandoc accepts locators encapsulated in {} to prevent ambiguity in such situations.

Do you think emacs could also support CSL YAML input (and perhaps encourage YAML or JSON as the preferred format)?

[bdarcus]

@bdarcus about your question on list locators, pandoc accepts locators encapsulated in {} to prevent ambiguity in such situations.

OK, thanks; I'll pass that on if it seems to be a continued issue.

Do you think emacs could also support CSL YAML input (and perhaps encourage YAML or JSON as the preferred format)?

I think that would depend on citeproc-el, which is really doing the work there I think.

I need to test how emacs native json parsing handles YAML.

[bwiernik]

See here for the pandoc details https://pandoc.org/MANUAL.html#extension-citations

Supporting that would be good for when CSL 1.1 drops with multiple-locator support

[bdarcus]

I forgot about that, but just posted it to that thread.

Yeah, @denismaier and I were wondering about multiple locators.