Ambiguity for Snippet location in tag-value documents

[swinslow]

According to Section 5.2.4 of the 2.2 spec, in tag-value documents:

If a File contains Snippets, the Snippet Information section shall follow a related File Information section (if it exists in the document).
Presence of a new file or package section signals the end of the set of snippets associated with the original file, unless an explicit Relationship is used.

I had read this to say that, for tag-value documents, one or more Snippets should be listed immediately after the File that contains them. This would be equivalent to the way that Files contained in a Package are listed immediately after that Package.

However, the sample 2.2 tag-value document doesn't follow this practice: the Snippet SPDXRef-Snippet (line 194) does not appear immediately after the File which appears to contain it, SPDXRef-DoapSource (line 144). There are several Package sections defined between these two. Is this intentional and correct?

This might be intended behavior, since the Snippet section contains a SnippetFromFileSPDXID field. So I'm not marking this as a "bug" yet. :)

However, if the intent is not that Snippets need to appear immediately after the corresponding File, then that might be helpful to clarify in Section 5.2.4 to avoid ambiguity here.

I'm putting this as a 2.2.2 issue, but feel free to push to 2.3 as appropriate.

(Reference: spdx/tools-golang#124 where it seems I've gotten bitten with this, by assuming that Snippets would come immediately after the corresponding Files...)

[zvr]

I also agree that the sample file is wrong.

Since Tag-Value format does not have hierarchical nesting (Package/File/Snippet), we've always maintained that this was expressed by the order that entries appear and the inclusion is implicit, i.e., everything is in the previous unit until a new unit of higher level is encountered.

I do not remember the discussions that led to the inclusion of the mandatory SnippetFromFileSPDXID field, nor why we do not have the respective FileFromPackageSPDXID.

[goneall]

I thought we added the SnippetFromFileSPDXID to loosen the requirement for the snippet to follow the file. That being said, the field was added a very long time ago and my memory is not entirely reliable.

Although I don't recall discussing a FileFromPackgeSPDXID field, I'm guessing the files following the package was a convention used in SPDX 1.0 and carried forward for 2.0 for compatibility. Since Snippets were introduced post 1.0, it didn't have the same compatibility constraint.

The SPDX Java Tag/Value parser does not place the snippets immediately after the file, so if we want to impose that restriction, we will need to change the Java code. I'm not sure how the Python code works, but it should be reviewed as well to see which conventions it follows.

[ianling]

The Python library also does not put snippets alongside/within their associated file. The Go library does.

Snippets in tag-value files either need to use SnippetFromFileSPDXID or follow the implicit hierarchical ordering of the tag-value format. The example file does the former, which should be supported.

(slightly tangential, but) In my opinion, while it is important for these tools to provide a reference implementation for WRITING the various file types with perfect adherence to the spec, these tools should not assume that they will also be READING perfectly valid SPDX files and should instead aim to be useful to developers who might need to read slightly invalid files. There are places where it is safe to make assumptions about what the document creator's intent was (e.g. that a snippet is related to a file using the SnippetFromFileSPDXID and not by position within the tag-value file), even if they didn't exactly adhere to the spec.

In other words, while the spec should take a clear stance on what is correct, parsers should support common incorrect uses, as not doing so will simply make it more difficult for people to write applications that can parse an SPDX file, and therefore reduce adoption the spec.

[goneall]

In my opinion, while it is important for these tools to provide a reference implementation for WRITING the various file types with perfect adherence to the spec, these tools should not assume that they will also be READING perfectly valid SPDX files and should instead aim to be useful to developers who might need to read slightly invalid files.

I also adhere to this philosophy and have attempted to implement them in the Java tools implementation. In the SPDX-Java-Library there is strict flag. If set to true, it will fail for any non-compliant SPDX document. If it is set to false, it will be more lenient.

I also added an issue for the Java tag/value parser to allow snippets without the SnippetFromFileSPDXID.

Since I don't use the tag/value format myself, I don't plan to implement this feature, but I would support and encourage anyone else who would like to implement it.

[goneall]

Since we are not implementing tag/value in the 3.0 release and the current proposed tag/value replacements are significantly different, I'm going to close this issue.

cc: @kestewart