Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add decompression flag to openstack_images_image_v2 resource #1482

Conversation

spnngl
Copy link
Contributor

@spnngl spnngl commented Feb 16, 2023

Some image are compressed and only filling Content-Type header resulting if uploading a compressed image to openstack.

With this we automatically detects and uncompress images in this case.

Example:

$ curl -I "https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_openstack_image.img.bz2"
HTTP/2 200
server: nginx
date: Thu, 16 Feb 2023 15:54:03 GMT
content-type: application/x-bzip2
content-length: 371535602
etag: "rq4xkz657ape"
last-modified: Wed, 15 Feb 2023 18:48:35 GMT
accept-ranges: bytes

Resolves #1481

Copy link
Collaborator

@kayrus kayrus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for your PR. I left a couple of comments.

var reader io.ReadCloser
decompress := d.Get("decompress").(bool)

if decompress && !resp.Uncompressed {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

checking for resp.Uncompressed doesn't make sense, since it is used only for Content-Encoding

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to check if we're not trying to decompress something already decompressed by golang stack or which was never compressed to begin with

openstack/images_image_v2.go Outdated Show resolved Hide resolved
openstack/images_image_v2.go Outdated Show resolved Hide resolved
@kayrus
Copy link
Collaborator

kayrus commented Feb 16, 2023

In addition please update the CHANGELOG.md:

## 1.50.0 (Not released yet)

FEATURES

IMPROVEMENTS

* ...

@spnngl spnngl force-pushed the feat/resource/image_v2_decompress branch from c4d7033 to 7fbd8cc Compare February 17, 2023 10:10
@spnngl
Copy link
Contributor Author

spnngl commented Feb 17, 2023

Did the modifications in the last version

Copy link
Collaborator

@kayrus kayrus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. LGTM. Please fix the comments. And I think it's ready to be merged.

CHANGELOG.md Outdated Show resolved Hide resolved
website/docs/r/images_image_v2.html.markdown Outdated Show resolved Hide resolved
@spnngl spnngl force-pushed the feat/resource/image_v2_decompress branch from 7fbd8cc to cc2f094 Compare February 17, 2023 10:40
@spnngl
Copy link
Contributor Author

spnngl commented Feb 17, 2023

Updated

@kayrus
Copy link
Collaborator

kayrus commented Feb 20, 2023

@spnngl can you fix golang linter issues?

@kayrus
Copy link
Collaborator

kayrus commented Feb 20, 2023

It also looks like OpenStack test env doesn't support web-download anymore. Can you add an exception and skip these tests for a while?

@spnngl spnngl force-pushed the feat/resource/image_v2_decompress branch 2 times, most recently from ce7f87a to 4f7becb Compare February 20, 2023 13:38
@spnngl
Copy link
Contributor Author

spnngl commented Feb 20, 2023

Pushed a new version with lint fix + web_download tests deactivated

@maxime1907
Copy link

maxime1907 commented Feb 28, 2023

@kayrus any update? We really need this to be merged! 🙂

@kayrus
Copy link
Collaborator

kayrus commented Feb 28, 2023

I re-triggered glance tests, once they pass, the PR can be merged.

@kayrus
Copy link
Collaborator

kayrus commented Feb 28, 2023

can you add a "decompress" exception into the import tests? Tests fail because of

ImportStateVerify attributes not equivalent. Difference is shown below. Top is actual, bottom is expected.

(map[string]string) {
}


(map[string]string) (len=1) {
 (string) (len=10) "decompress": (string) (len=5) "false"
}

"region",
"local_file_path",
"image_cache_path",
"image_source_url",
"verify_checksum",

@spnngl spnngl force-pushed the feat/resource/image_v2_decompress branch from 4f7becb to bac0376 Compare February 28, 2023 10:40
@spnngl
Copy link
Contributor Author

spnngl commented Feb 28, 2023

I added the decompress ignore + rebase on main

Some image are compressed and only filling Content-Type header resulting
if uploading a compressed image to openstack.

With this we automatically detects and uncompress images in this case.

Example:
```
$ curl -I "https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_openstack_image.img.bz2"
HTTP/2 200
server: nginx
date: Thu, 16 Feb 2023 15:54:03 GMT
content-type: application/x-bzip2
content-length: 371535602
etag: "rq4xkz657ape"
last-modified: Wed, 15 Feb 2023 18:48:35 GMT
accept-ranges: bytes
```
OpenStack dev env seems to not support it anymore.
@spnngl spnngl force-pushed the feat/resource/image_v2_decompress branch from bac0376 to fead509 Compare February 28, 2023 15:07
@nikParasyr
Copy link
Member

nikParasyr commented Feb 28, 2023

Since changelog is updated this will trigger all the ci jobs.

I think recently opendev,org added some limits to pulling repos etc and as we also have our daily check failing with a timeout. I need to have a look to make sure that the timeout is due to limits and think of a way to bypass it.

Im addressing this in #1497

@kayrus
Copy link
Collaborator

kayrus commented Feb 28, 2023

@nikParasyr @ozerovandrei I was not aware about the behavior when changelog is changed. I think we can skip failed functional-images test, it has some issues with the data source. other tests are fine.
I think we can merge this PR.

@maxime1907 maxime1907 mentioned this pull request Feb 28, 2023
@nikParasyr
Copy link
Member

@kayrus the idea behind this behavior with changelog is that we update it only before a release (or after multiple MRs are changed) and the whole ci runs to make sure all services are ok. But it can be a bit annoying if changelog is updated on each MR (also often we have conflicts due to that).

From my side this can be merged as well. The failing ci jobs are not related to this change.

@nikParasyr nikParasyr self-requested a review March 1, 2023 06:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

openstack_images_image_v2 resource not uncompressing dowloaded image
4 participants