[SEC] potential security issue related to GitHub CI servers?

[fkiraly]

There have been multiple strange occurrences around GitHub CI servers that may be security related:

• sporadically, logs showing unexpected package checksums, e.g., here: [ENH] Refactor base series annotator #6265 (comment)

ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    unknown package:
        Expected sha256 55b380cff92d493e465b23e96a4bce8eec1a7b02db92c7718bb0d883f244c2b9
             Got        f8a96f8553d80288955fd9fa54bb4f9e6aec5df2028074a8cc65a11510af611a

These also do not always seem to be the same packages, or same sha.

• internet security flagging seemingly trustworthy microsoft URLs with GitHub logs as compromised by malware: [MNT] GitHub log server on AVG internet security blocklist skpro#304

I also wonder whether the two are correlated.

What is going on here? I really hope Microsoft don't have hackers in there...

[fkiraly]

GitHub support reply on the second issue (blocklist)

Hi Franz,

Thanks for that (and apologies about the upload.) I can see the screenshots via that issue you created.

The windows.net domain points to an Azure blob storage resource. On the surface this shouldn't indicate anything suspicious, but if the resource is ephemeral and/or not recognised as a reputable domain then I can see why AVG might be inclined to flag it.

In any case, thanks again for bringing this up. I'll raise this with the rest of the Actions team and close this out, but feel free to reopen if you have any other questions.

No clear "next step" or instructions how to resolve were given so far.

[achieveordie]

Related: pypa/pip#12680

It doesn't seem like something we should be worried about.

[achieveordie]

On second thoughts, we should keep an eye out, just in case (I'm looking at you xz)

[fkiraly]

Related: pypa/pip#12680

It doesn't seem like something we should be worried about.

I think our problem is not an instance of this - the hashes in the error logs are very different, not just lower/uppercase variants.

[fkiraly]

Reply from GitHub support:

Please rest assured that this has not been a widespread issue impacting GitHub services or resources. Nevertheless, while this has not been a widespread issue, we have communicated this with AVG directly. But with respect to next steps regarding the blacklisted log URL, the only other thing we can advise right now is that you send your feedback to AVG to notify them of the false positive too.

And on the topic of the more recent issue regarding the package SHAs:

This doesn't appear to be related to AVG's blacklisting; instead, it's likely related to some kind of build, deployment, or CICD workflow process expecting one version of a dependency that's pinned to a particular commit, but receiving a different one.

Do you happen to be using a requirements.txt file (or something similar)?

The reason I ask is because this error is typically seen when using Python's pip package manager with a requirements file that includes hashes for each package. The hash is a way to verify the integrity of the downloaded package, ensuring it has not been tampered with. The error message is suggests that the hash of the downloaded package does not match the expected hash listed in the requirements file. This could be due to:

• The package version has been updated, and the new version has a different hash. In this case, you should update the hashes in your requirements file.
• The downloaded package has been tampered with or corrupted during download. In this case, you should delete the package and try downloading it again.

To update the hashes in your requirements file, you can use the pip freeze command to get the current versions and hashes of your installed packages, and then update your requirements file with this information. But please also note that this will overwrite your current requirements file, including any packages you've installed that are not in your requirements file.

[fkiraly]

My analysis of this is:

• they consider the AVG blocklisting as a false positive, although no reasoning is given here. I wonder if this is jumping to conclusions, or if there is an argument they are not sharing.
• regarding package SHA, we are not using requirements.txt or anything similar that specifies SHA explicitly, afaik. This should be clear from inspecting this GitHub repository? Or is there sth that I am missing?
• in particular, we are using pyproject with ranges rather than pins, and GHA VMs are set up dynamically. The advice seems to be geared at a "end user project" rather than a package that one would expect to be consumed by end users?

I will follow up.

[fkiraly]

My reply to GH support:

Thanks for your update.

• regarding AVG blacklisting, can you confirm that you consider this a false positive? If so, can you explain how you know that it is a false positive, i.e., not actually flagging malicious behaviour?

• regarding requirements: we do not pin SHA or similar, sktime's dependencies are specified by version ranges in the pyproject.toml, the entire repository is public and you can inspect it: https://github.com/sktime/sktime

• as a consequence of this, I would conclude that your first suspicion does not apply, since we do not pin SHAs. Hence, by exclusion, your second one becomes more likely, i.e., corruption or tampering. It has become very frequent now, over hte last months, so I think it warrants a proper investigation?