Support for "data" URIs in ImageConverter.guess_mimetypes()

[mgeier]

Is your feature request related to a problem? Please describe.

This problem came up when using the sphinxcontrib.rsvgconverter extension, which is supposed to automatically convert SVG images to PDF when using the LaTeX builder, which it does with the help of the external program rsvg-convert.

When using the HTML builder, the extension is supposed to do nothing.
Previously, there was a problem with this (missinglinkelectronics/sphinxcontrib-svg2pdfconverter#8), which was fixed in #8649.

However, there is still one error case remaining:

When using data URIs instead of image files, this is still broken.
The data type is not recognized in this case and therefore it is handled as unsupported by the builder, even if the builder actually supports it.

Example reST code:

.. image:: data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAYAAACqaXHeAAAACXBIWXMAAAsTAAALEwEAmpwYAAAHv0lEQVR42u2ae2xbZxnGf+f4OLbjOHGcNq2bpqF12tJmZGtJttK1m7Z1XMomVQyp4zYESEMwMNLEKEgIbQJNFKkIVUNogJj2B9u6cYkETBUtl5UNWDt10NGMbiU0G+klpE6aq9PYMX+c5xRjEl8SJ3WcvdKR7ePvXN7ney/P+34fvCWLW4x5eIYHCAGV+v2PcgcgBFTpewB4J/AxYDmQAKLAaeBcOQLQKgW3AymgETgL1AJHgAlgI9ADPAQcB8bLxZ1WAZ3AYaALuBdo0flrgTogLID6gDc0xlMOynuAm4DXgU9plj1Zxt4B/BM4JYAWvPL3AX8VCPnO6A5gRFayoKUFiAOvFTibzcDfgTuvphuYRbjHkPz518BAAVbTCkwCr17NQGgW6T5J+fV1BQCwRFnhcjnEgE/KnDs1s6vEB6aSauDzwN+ALeWSAquV0rqApwXG9wXGBoERBtqkfB8wCqwtJyLkEevbI8Vf1Kcf+JNA2CzljwPfBl4qRyocltIjwBdEfA4ClwA38Czwu1KhwnMpzcAJ4CsCpU5HSTE/aw4D463KMs8shtnOVP5dmv0DQH0pv6yriEonFQRvBR5QKhwS5+8W6Sm7hkgIWA2skDs9CgwCPs38oAqku4EzpQiANYOZ9qu+r1VO3ykKvAL4I/BlBbuvqyFyXkCUhW9/GHhMjG8UeAL4lUx/raK9B7gfeEXjy6Lic7o9J4CngJeB29KaHOnSJo7/jK7pFBCeAtwqXGrKR4BHRG+bs1BYj/i9o3QYeBz4XpbaIPM5PwJ+Pl8gWHnOyEOq+b+J3dCcTvl3Aw+rOHpZ8WJCDDCXbAc+I6sx8rxmXsrhNcBu7LZ2LMu4cY1ZA1xMq/ETOZSJAB8EfgbcJTf7kHoMJSHbsXt3ufw43fdDGYHzpFwns2Z4P3Yr7d/AT4CtiislIxv0go/n4ZOblBnap6gJOoEPCJCIssZBoFfl871XK/BZOdLeJqBBFd25HP2Aj4jxXcj477JYYEQz/B2gQue+IYuJUaLrA+/Ns3HRKjN+aopoH1JNMKEAekIz3kyJrwlEFMmfziOFNWN3haeLE3XAjdgd4I2lpHiuNFghJpdP6XsZODqNKV8EXtDYkjL1bGlwQn56KAcIK4HPYjc5YzmeV3J+ns0C3lCaupDjxR3SEl+I/D6XC+RLRlzYCyMjCw2A2S6MpK/wHGUBLnUXY2XInZbvWYwAFKOztOAB8CplLjoAnC0wg7rXChbYjo+Zmm4IuBnYh935jYoP7AZ+KtITw+4b9k9zj9hsXrznc22pzHMNj7xkzAcAEeBBKXsEey3QqfbukVVMihhtVDWZmR3iwNdmAsJUis8GCGMGM/8tfX9SijtVok//+4DrsbfAtE0BgEOHv1QoAPkoXygIRgGKh+Tj+1TRPZBFAQ/2XsFQxpha7LY5hQJQiPKFgJBvT3CfCp4B+Xsu3j+u42LG+RR2i8w3VzOfeV0uEPIBIKCmSBJ7IeTILGLXiK7/qixkShCjHe0GwP5dx1IUWaId7Vcy3/5dxyaNaEe7S65gKHilgJQeHgZukblGZ6m8Iy3AMezNk6+nvZhbE2Ly3zXLxJ7DqVnXF3t3GC4xVrfc0wHhsgWs0x+OeY4Dw+tvClWc+kPsYVKMKLqf0MyYafFjMg2sfMWhzDUC36vYUIe9cSqgc5PqRhVD/Lr3crm017FIC/iEzNGUifYD55dG/K7Xno/dabqNQ81bamPvuT9Sqxt500AYB8aiHe2jUiyRByAXgV+YLuOev/zywuh1dywLaRLWiUvU6DkUEYA12DvY1qp75ZMOMQt75dZBJAmMAb3V9RXjHr+LyJbQ4NaPNtyoC5ZiN0AtKXsJuxf4L/UNBoB4tKM9ofs5LpUEJvfvOpaq8LuqJ0aTrcvW+X/89pvrtgLX9nWP3mC6jKWhlb7KDICLFQN2qiW3WpbglgXHLQU4x6RTAD0nhxr//MRZ831fbO4Pb6jaZlWY16SlN2/a2GExwR5R4h7R4iFZxLA+B4GRaEf72IsHehqPd5z3N7ZWbxofTS7p7RpZ+/xjby4DjJ17mlPBsHcuCqvdQJMsy9SRAgJWRj1gxN4c47kfdJuJ8ST+kDtkVZhBDZ6cgjs4ceNK7JAbjelzQGD0K+InVrcFm04e6gsaBtuPHugJ9p4etUKNXsPtc+HxW8YUAYw9h1OzCYAXxEitjPc3/i8NJiYmOfvqMKFGHy63QWXQbaRF5NQ05Mmdli6XZJh9QhYQd9igp8qqtDymv/O3fVUrrwkYbXeFWfmOAKZl4queky1L9dkInxHtaL8Cb8/JIZ7de5qmzTVcv3sFwbB3Ng9OZXx3DnPgXNzoOzNqhNdX4QlYWO7cRelMrGDvjtzedOXJ8aEEXUf7adpcQ8vtS/HVzHpx1kg7nNxuAWYw7DXe1hbEH6rIS/l8lZnJeNcNdzc8CGB5TOojfupW+fj9o92MXZrAG7BwWQaWxyy6XZquwmPdC2sMtnUVF6z/cQFHzp0a5jffPQOpFMvXV7Ht4414A3O1pXBmMpVLFGol09YC4fVV3PLpJp77YTfJiRTx4UTJATATZQtqiTW0BLjtvtVUVLp45WAv8aEE5ShZp3VZs5/qervFV2oWMC8AAHOVm0tGTBa5vAXAYgfgP6N1EOJNty18AAAAAElFTkSuQmCC

Here is a test repo that shows the problem: https://github.com/mgeier/rsvgconverter-bug.

Describe the solution you'd like

The MIME type of data URIs should be checked and correctly recognized so that sphinxcontrib.rsvgconverter (and probably other extensions) knows if a supported data type is used.

Describe alternatives you've considered

One could just ignore this problem, but then the program rsvg-convert has to be installed and it is executed even if it is absolutely not necessary (e.g. for the HTML builder).

Additional context

This is where the MIME type is guessed:

sphinx/sphinx/transforms/post_transforms/images.py

Lines 229 to 235 in 08a87d9

	def guess_mimetypes(self, node: nodes.image) -> List[str]:
	if '?' in node['candidates']:
	return []
	elif '*' in node['candidates']:
	return [guess_mimetype(node['uri'])]
	else:
	return node['candidates'].keys()

The value of node['candidates'] is {'?': 'data:image/png;base64,...'} in my case.
I'm not sure what the question mark means exactly.

There is already a helper function parse_data_uri(), which can probably be used to implement this:

sphinx/sphinx/util/images.py

Lines 85 to 103 in 5bf0e3a

	def parse_data_uri(uri: str) -> Optional[DataURI]:
	if not uri.startswith('data:'):
	return None
	# data:[<MIME-type>][;charset=<encoding>][;base64],<data>
	mimetype = 'text/plain'
	charset = 'US-ASCII'
	properties, data = uri[5:].split(',', 1)
	for prop in properties.split(';'):
	if prop == 'base64':
	pass # skip
	elif prop.startswith('charset='):
	charset = prop[8:]
	elif prop:
	mimetype = prop
	image_data = base64.b64decode(data)
	return DataURI(mimetype, charset, image_data)

[mgeier]

I've created a possible fix: #10074.

[tk0miya]

If my understanding is correct, data URIs are extracted to temporary files and detected its mimetype by DataURIExtractor before a process of image converters.

sphinx/sphinx/transforms/post_transforms/images.py

Lines 114 to 144 in 5bf0e3a

	class DataURIExtractor(BaseImageConverter):
	default_priority = 150
	def match(self, node: nodes.image) -> bool:
	if self.app.builder.supported_remote_images == []:
	return False
	elif self.app.builder.supported_data_uri_images is True:
	return False
	else:
	return node['uri'].startswith('data:')
	def handle(self, node: nodes.image) -> None:
	image = parse_data_uri(node['uri'])
	ext = get_image_extension(image.mimetype)
	if ext is None:
	logger.warning(__('Unknown image format: %s...'), node['uri'][:32],
	location=node)
	return
	ensuredir(os.path.join(self.imagedir, 'embeded'))
	digest = sha1(image.data).hexdigest()
	path = os.path.join(self.imagedir, 'embeded', digest + ext)
	self.app.env.original_image_uri[path] = node['uri']
	with open(path, 'wb') as f:
	f.write(image.data)
	node['candidates'].pop('?')
	node['candidates'][image.mimetype] = path
	node['uri'] = path
	self.app.env.images.add_file(self.env.docname, path)

Therefore, no additional code is needed to process them, AFAIK.

For example, I succeeded to embed SVG image into PDF document using sphinx.ext.imgconverter without any changes.

.. image:: data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9Im5vIj8+CjxzdmcgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB4bWxuczp4bGluaz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94bGluayIgaGVpZ2h0PSI2MCIgd2lkdGg9IjYwIj4KICAgIDxjaXJjbGUgY3g9IjQwIiBjeT0iNDAiIHI9IjI0IiBzdHlsZT0ic3Ryb2tlOiMwMDAwMDA7IGZpbGw6I2ZmZmZmZiIvPgo8L3N2Zz4K

[mgeier]

Thanks for looking into this @tk0miya!

The DataURIExtractor is only used when data: URIs are not supported by the builder (via supported_data_uri_images).

In my case it is not used, because the HTML builder can handle data: URIs.

I succeeded to embed SVG image into PDF document using sphinx.ext.imgconverter without any changes.

I know that it works for unsupported types (e.g. SVG) in the LaTeX builder.

I probably wasn't clear in my description, but the the problem is with supported types (in my case PNG) in the HTML builder.

In this case, the converter is supposed to do nothing, but it is called nevertheless, which is wrong.
This is causing my original problem (in case you are interested: spatialaudio/nbsphinx#559).

[tk0miya]

I think the expected behavior is that rsvgconverter will not be invoked for the embedded PNG image, right? If so, no reason to "guess mimetypes". How about skipping without guessing?

[mgeier]

the expected behavior is that rsvgconverter will not be invoked for the embedded PNG image, right?

Yes.

If so, no reason to "guess mimetypes". How about skipping without guessing?

Don't we have to guess first to be able to check whether the guessed MIME type is supported or not?

sphinx/sphinx/transforms/post_transforms/images.py

Lines 200 to 202 in 08a87d9

	elif set(self.guess_mimetypes(node)) & set(self.app.builder.supported_image_types):
	# builder supports the image; no need to convert
	return False

That's what I'm trying to implement in #10074.

[tk0miya]

I think all embedded images should be skipped. So it's better to skip them all:

if node["uri"].startswith("data:"):
    return False

Is any reason to obtain the mime-type from the image? What do you do if the builder does not support the image?

[mgeier]

Yes, thanks, I think you are right!

Nobody would use an unsupported MIME type in a "data" URI anyway.

I've created an alternative fix based on your suggestion in #10099.