Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix mixed image format for file, S3 and azureblob caches #796

Merged
merged 4 commits into from
Jan 4, 2024

Conversation

shimoncohen
Copy link
Contributor

@shimoncohen shimoncohen commented Nov 29, 2023

Fix the usage of mixed image format with the File, S3 and azureblob caches.
Fixes #605.

@shimoncohen shimoncohen changed the title Fix mixed image format for file, S3 and azureblob caches (WIP) Fix mixed image format for file, S3 and azureblob caches Dec 7, 2023
@shimoncohen
Copy link
Contributor Author

@weskamm Could you please review this PR?

@shimoncohen
Copy link
Contributor Author

@weskamm Could you please take a look? If there are changes that need to be made I will fix them right away.

@weskamm
Copy link
Member

weskamm commented Jan 4, 2024

Looks good to me, thanks for the fixup.

@weskamm weskamm merged commit 3f89bc6 into mapproxy:master Jan 4, 2024
4 checks passed
@simonseyock
Copy link
Contributor

We are reverting this PR because it was causing problems in production - especially with existing caches. I think we need backwards compatibility and unit tests to assure that everything works correctly.

@shimoncohen
Copy link
Contributor Author

shimoncohen commented Jan 10, 2024

@simonseyock What are the problems that this causes? I will work on the issues ASAP.

@simonseyock
Copy link
Contributor

simonseyock commented Jan 11, 2024

There was the problem that mapproxy did only return blank images for caches that were created before the update.

In the PR itself I see this problems:

  1. It does not introduce a test that demonstrates the failing case
  2. In the existing tests, it is ensured that every tile is stored with a .png ending. I do not know exactly why that is, but it feels not right to me. See
  3. The new tile_location function has this code:
        if self.is_mixed:
            location = self._tile_location(tile, self.cache_dir, 'jpeg', create_dir=create_dir, dimensions=dimensions)
            if not os.path.exists(location):
                tile.location = None
                location = self._tile_location(tile, self.cache_dir, 'png', create_dir=create_dir, dimensions=dimensions)

This will never cause a file with .jpeg to be created. Might be intended so
4. In the tile_location function of the file cache there is code that does nothing. This was there before and maybe is unrelated to the problem:

cache_dir = os.path.join(self.cache_dir, '_'.join(dimensions_str))

So, if MapProxy should supported reading caches that mix .jpeg and .png files I think it should also create .jpeg and .png files. Furthermore we need to ensure that old caches are still usable.

@simonseyock
Copy link
Contributor

I do not have a reproducible example, but I hope to get one.

We reverted the commit because we needed a clean working release before merging all of the linting and deprecation issues.

@shimoncohen
Copy link
Contributor Author

shimoncohen commented Jan 13, 2024

There was the problem that mapproxy did only return blank images for caches that were created before the update.

In the PR itself I see this problems:

  1. It does not introduce a test that demonstrates the failing case
  2. In the existing tests, it is ensured that every tile is stored with a .png ending. I do not know exactly why that is, but it feels not right to me. See
  3. The new tile_location function has this code:
        if self.is_mixed:
            location = self._tile_location(tile, self.cache_dir, 'jpeg', create_dir=create_dir, dimensions=dimensions)
            if not os.path.exists(location):
                tile.location = None
                location = self._tile_location(tile, self.cache_dir, 'png', create_dir=create_dir, dimensions=dimensions)

This will never cause a file with .jpeg to be created. Might be intended so 4. In the tile_location function of the file cache there is code that does nothing. This was there before and maybe is unrelated to the problem:

cache_dir = os.path.join(self.cache_dir, '_'.join(dimensions_str))

So, if MapProxy should supported reading caches that mix .jpeg and .png files I think it should also create .jpeg and .png files. Furthermore we need to ensure that old caches are still usable.

Thank you very much for your review! I will add a failing test as you requested and fix the existing tests.
As for your third point, I see that when dealing with file caches I have ignored the create_dir parameter that indicates wanting to store a tile instead of just getting it from the cache. I will look into the issue for S3 and azureblob as well.
I agree that storing mixed tiles is what was intended.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Mixed image formats doesn't work with s3/local directories tiles cache
3 participants