'There is a memory type that is host visible, but not host coherent.' warning on MacOS

[expenses]

So I'm not entirely sure why this warning exists:

15:04:03 [WARN] There is a memory type that is host visible, but not host coherent. It's time to upgrade our memory allocator to take advantage of this type of memory :)

but just FYI, this is generally how memory is set up in MoltenVK, atleast in version 1.1.6.

Here are some reports: https://vulkan.gpuinfo.org/displayreport.php?id=13266#memory https://vulkan.gpuinfo.org/displayreport.php?id=13265#memory

[Jasper-Bekkers]

That makes me suspect that MoltenVK is incorrect here - AMD has had host coherent PCI-e memory on PC and Linux since forever so it would seem that it'd be the same on macOS. Before anyone does any work resolving this, I'd like to see why MoltenVK couldn't mark that memory as host coherent.

Intel on windows (same GPU) has it's host memory marked as coherent on windows as well; https://vulkan.gpuinfo.org/displayreport.php?id=1732#memory

Amd on windows (same GPU), also marks all of it's memory as coherent; https://vulkan.gpuinfo.org/displayreport.php?id=13241#memory

The reason we're not currently supporting this (though there's no reason we couldn't) is because non-coherent memory requires manual vkFlushMappedMemoryRanges of vkUnmapMemory calls after data writes to buffers that are mapped.

The allocator internally does a persistent mapping of all host-visible buffers for ease of use and because it's the fast-path on PC/Linux compared to calling vkMapMemory repeatedly (vkMapMemory / vkUnmapMemory need to reserve and release virtual memory ranges every time, whereas persistent mapping doesn't need it). If anybody attempts a fix for this issue, persistent mapping should (imho) be the default behavior, but it could be made into something that one opts out of.

[Jasper-Bekkers]

On macOS versions prior to macOS 10.15.6, native host-coherent image device memory is not available. Because of this, changes made to VkImage VK_MEMORY_PROPERTY_HOST_COHERENT_BIT device memory by the CPU or GPU will not be available to the GPU or CPU, respectively, until the memory is flushed or unmapped by the application. Applications using vkMapMemory() with VkImage VK_MEMORY_PROPERTY_HOST_COHERENT_BIT device memory on macOS versions prior to macOS 10.15.6 must call either vkUnmapMemory(), or vkFlushMappedMemoryRanges() / vkInvalidateMappedMemoryRanges() to ensure memory changes are coherent between the CPU and GPU. This limitation does not apply to VKImage device memory on macOS starting with macOS 10.15.6, does not apply to VKImage device memory on any version of iOS or tvOS, and does not apply to VKBuffer device memory on any platform.

From the MoltenVK manual - https://github.com/KhronosGroup/MoltenVK/blob/e06eb3a8925d1d517034153461979cc1c1007ad0/Docs/MoltenVK_Runtime_UserGuide.md

What version of macOS are you running on, is it older then 10.15.6?

[expenses]

What version of macOS are you running on, is it older then 10.15.6?

Nope, I'm on 12.0.1. This is also happening with MoltenVK 1.1.2 and MacOS 11.5: http://vulkan.gpuinfo.org/displayreport.php?id=13263#device and MoltenVK 1.1.6, also 11.5: http://vulkan.gpuinfo.org/displayreport.php?id=13264

This also only applies to one of the memory types - both GPUs on MoltenVK do have a memory type with HOST_VISIBLE_BIT and HOST_COHERENT_BIT, they just also have this wierd one.

[Jasper-Bekkers]

Alright, thanks for the extra information! In that case, this warning can probably be disregarded entirely since I suspect we'll just persistently map the coherent memory, and that'd be enough to keep everything working.

@max-traverse IIRC you added this warning more as a reminder to take a look at this? Maybe it's time to change it to "if we /only/ find a cpu visible memory type that's non-coherent" instead of "if we find a cpu visible memory type that's non-coherent"?

[manon-traverse]

@Jasper-Bekkers
I vaguely recall putting something like this in for a special type of memory where we could do something more optimal but left it out until we had a GPU that actually had it so we can test it. Although I am not sure it's the case for this since host visible memory that's not coherent just means it needs manual flushing.
The case of only finding "only host visible memory that's not host coherent" is not valid though, since the Vulkan spec requires that there should always be a memory type that is host visible and host coherent.

There must be at least one memory type with both the VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT and VK_MEMORY_PROPERTY_HOST_COHERENT_BIT bits set in its propertyFlags. There must be at least one memory type with the VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT bit set in its propertyFlags. If the deviceCoherentMemory feature is enabled, there must be at least one memory type with the VK_MEMORY_PROPERTY_DEVICE_COHERENT_BIT_AMD bit set in its propertyFlags.

I'll spend some time digging into this while I wait for unreal compiles.

[manon-traverse]

I have tried to do some 'code archeology', but it was in there in the initial commit of gpu_allocator. I don't see why this memory type would give any possibilities for extra performance, so I guess we can just ignore those memory types instead.

[Jasper-Bekkers]

@expenses Is this an issue that is blocking you in some way, or did you mostly file this because of the odd warning? @max-traverse If that's the case, maybe we should just remove the warning.

[expenses]

@expenses Is this an issue that is blocking you in some way, or did you mostly file this because of the odd warning? @max-traverse If that's the case, maybe we should just remove the warning.

Nope, this isn't blocking my in any way. I haven't gotten my code to work fully on MoltenVK yet, but I don't think it'll be an issue.