Supporting Apple Silicon

[kevzhao2]

I spent some time debugging over the weekend and figured out the necessary changes for supporting Apple Silicon. I can try to restructure the code a bit and send in a pull request, but I just wanted to write down the things that I encountered.

W^X

Apple requires all pages to be W^X, and requires pages to be mapped in with the MAP_JIT flag.

1. When applying detours, the relevant pages are already mapped in with the MAP_JIT flag. We just need to do the following:
   • Call pthread_jit_write_protect_np(false) (to enable write and disable execute)
   • Make the changes
   • Call pthread_jit_write_protect_np(true) (to disable write and enable execute)
2. When creating the trampoline for the JIT hook, we need to mmap in memory with the MAP_JIT flag instead of calling the equivalent of malloc.

The first item has to be done entirely in native code because calling pthread_jit_write_protect_np(false) from managed code causes a bus error -- this is because pthread_jit_write_protect_np controls every page!

libclrjit.dylib

libclrjit.dylib doesn't seem to be one of the modules loaded into the process, which means that the JIT hooks don't work. This is, however, a pretty easy change to make.

Virtual Method Function Pointers

Hooking virtual methods seems to be a bit fraught (with arm64, at least?) because the function pointers that we get with DetourRuntimeNETPlatform.GetFunctionPointer don't seem to related to the entries found in the vtable, and I'm not exactly sure why this isn't the case for amd64...

My quick and dirty solution was to just read the vtable to get the function pointer:

var methodTable = method.DeclaringType.TypeHandle.Value;
var methodDesc = method.MethodHandle.Value;

var m_wSlotNumber = (ushort) Marshal.ReadInt16((IntPtr) (methodDesc.ToInt64() + 4));
var m_wFlags = (ushort) Marshal.ReadInt16((IntPtr) (methodDesc.ToInt64() + 6));
if ((m_wFlags & 0x8000) == 0) {
    m_wSlotNumber &= 0xff;
}

var chunkNumber = m_wSlotNumber / 8;
var chunkOffset = m_wSlotNumber % 8;

var vtablePtr = Marshal.ReadIntPtr((IntPtr) (methodTable.ToInt64() + 0x40 + 8 * chunkNumber));
var vtableSlotPtr = (IntPtr) (vtablePtr.ToInt64() + 8 * chunkOffset);

ptr = Marshal.ReadIntPtr(vtableSlotPtr);

Additionally, because the devirtualization code expects the entries to point to things that look like precode, I had to change the detour code a bit to match.

[nike4613]

W^X stuff

I find it hard to believe that the only way to adjust protections is to adjust the protections for every JIT-mapped page. Does mprotecting the relevant pages with the correct permissions not work?

libclrjit.dylib

Where is the JIT loaded then? As far as I'm aware, CoreCLR always loads the JIT as a separate binary.

Virtual Functions

GetFunctionPointer getting a value different from reading the VTable is expected. Pretty much every managed call in the runtime, until it's fully optimized, goes through at least one stub. GetFunctionPointer skips through those to find the actual function bodies, and apply the detour there. If virtual calls aren't working, that's likely some other issue which is also present.

[kevzhao2]

W^X stuff

I find it hard to believe that the only way to adjust protections is to adjust the protections for every JIT-mapped page. Does mprotecting the relevant pages with the correct permissions not work?

The mprotect calls failed with EACCES. Apparently this is fairly recent behavior?

libclrjit.dylib

Where is the JIT loaded then? As far as I'm aware, CoreCLR always loads the JIT as a separate binary.

I'm.. not really quite familiar enough with what's going on. But this was always null, and there only seemed to be a single module loaded:

                ProcessModule clrjitModule = currentProc.Modules.Cast<ProcessModule>()
                    .FirstOrDefault(m => Path.GetFileNameWithoutExtension(m.FileName).EndsWith("clrjit", StringComparison.Ordinal));

Virtual Functions

GetFunctionPointer getting a value different from reading the VTable is expected. Pretty much every managed call in the runtime, until it's fully optimized, goes through at least one stub. GetFunctionPointer skips through those to find the actual function bodies, and apply the detour there. If virtual calls aren't working, that's likely some other issue which is also present.

Yeah that makes sense. So really the problem is that we didn't fill out any precode walkers here, and the ngen comment is unrelated:

            if (PlatformHelper.Is(Platform.ARM)) {
                // TODO: Debug detouring NGEN'd methods on ARM.

            }

Shouldn't this have been noticed earlier though??

[nike4613]

Shouldn't this have been noticed earlier though??

Until fairly recently, there simply wasn't a (non-Mono) runtime which targeted ARM and was widely used. As such, ARM walking was never implemented. Now would likely be the time to do it, though I would not blame you at all for not wanting to given the amount of pain that has gone into the x86 walking.

there only seemed to be a single module loaded

Could you enumerate the modules loaded in a CoreCLR process from outside? On Windows this would be done with something like Process Hacker, but I'm not sure what the equivalents would be for MacOS. It would likely be the same as on Linux.

The mprotect calls failed with EACCES.

Sigh. Personally, I'd like to avoid pulling in more shellcode, for hopefully obvious reasons. Is there any way to change a mapping away from MAP_JIT? Otherwise, it may be possible to coerce the JIT to generate code which we could just copy verbatim into another block to perform the detours.

[0x0ade]

Thank you for compiling all that information. I'll copy paste what I've sent someone else on the MonoMod Discord server when they asked about Apple Silicon support earlier this month:

macOS in general is just close enough™ to Linux, and Android ARM support ended up being "works on my machine, sorry I can't get it working on your machine" as well
I'll happily try to review any community-maintained efforts at supporting macOS on Apple ARM, but can't start working on it myself, thus have no real plans
maybe once I'll be a few months into my new job, I'll maybe be able to afford a M1 testing device, but that's a huge maybe.
... and by then, I'll probably have less free time available than I've had for the past few years.

Having said that, I'll happily review your (and anyone else's) efforts whenever I can, and highly appreciate that you've put your time into this to begin with.

---

there only seemed to be a single module loaded

... and after checking the dotnet runtime repo, that seems to be an intended workaround, given BSD limitations / lack of standardization..? Compare the following:

• ProcessManager.BSD.cs
• ProcessManager.Linux.cs

I am only familiar with Linux, where procfs is a given. I've recently had to work around this in another project of mine, and had to resort to calling a command line helper and parsing its output. In this scenario, on Linux that'd be pmap, on macOS it sounds like it'd be vmmap, but I can't test that and don't know how reliably parseable its output is. And more importantly...

The first item has to be done entirely in native code because calling pthread_jit_write_protect_np(false) from managed code causes a bus error -- this is because pthread_jit_write_protect_np controls every page!

Personally, I'd like to avoid pulling in more shellcode, for hopefully obvious reasons.

I can't even imagine how we'd embed shellcode like we already do for icache flushing (which was only tested on Android and would also require updating, see LLVM's clear_cache.c), or detour non-JITed code, given how...

Is there any way to change a mapping away from MAP_JIT?

The only way of achieving this that I was made aware of by Hello71 on Discord is to copy it out, unmap it, remap it as MAP_JIT and copy it back. I'd love to get my hands on Apple Silicon to check if that really is necessary - I hope it isn't.

As much as I hate to say it, at this point I'd be willing to let Azure Pipelines / GitHub Actions compile a helper native library, embed it, unpack it onto the disk and load it at runtime, as long as size and per-platform code can be kept to an absolute minimum, and only if absolutely necessary.
I mention embedding because RuntimeDetour can be used in environments where shipping a native library alongside the managed library is impossible, and where the managed library is copied between platforms. For example, a mod loader for a game loads mods from .zips. You can load managed .dlls from .zips, but native libs must exist on the disk, the mod loader might not be able to unpack and resolve native dependencies like that, and modders certainly won't create per-platform .zips anyway.
At least I'd rather ship that than "let's download CLR function offsets from a selfhosted server that parses PDBs".

Getting back to the point of actually iterating through loaded modules, the docs for mach_vm_region_recurse (which, according to some random comments on the internet™️, is what vmmap might be using) look very... promising. And to me, it still feels like "dotnet on macOS can't list all loaded modules" should be fixed by the dotnet team, no matter how much we'll work around that. Let's first see if that function - or vmmap in general - is in any way helpful though. Or maybe pmap instead of vmmap. Once again, I wish I could get my hands on Apple Silicon.

and the ngen comment is unrelated:

For some context: I began adding precode walkers when someone reported problems related to ngen'd assemblies, and it's all gotten much complex over time, to the point where RuntimeDetour now needs to works around Wine behavior and I've given up and added a generic catch-all. I should go back and fix those misnomers at some point though.

Until fairly recently, there simply wasn't a (non-Mono) runtime which targeted ARM and was widely used.

I remember having tested dotnet 2.x on one of my older rooted phones a few years ago, everything seemed fine enough to meet my personal needs, interest was lost eventually, thus yea. RuntimeDetour is heavily underutilized on ARM, both inside and outside mono, and as mentioned previously (icache), I wouldn't be surprised if other parts required updating as well.

[hakusaro]

Having said that, I'll happily review your (and anyone else's) efforts whenever I can, and highly appreciate that you've put your time into this to begin with.
[...]
I'd love to get my hands on Apple Silicon to check if that really is necessary

Hi @0x0ade! Nice to meet you. I was actually made aware of this discussion thanks to input from @SignatureBeef. In a conversation with them, I reached the conclusion that providing money or hardware for Apple Silicon is definitely something we could consider. My biggest problem with doing this is that it might create psychological obligations for you, and if you don't have the time/energy to do it, it isn't worth it anyways. It's an option that I should probably put on the table though -- I don't know how much time/energy/etc outside of just the raw issue of getting access to the hardware you want to spend on it. It sounds like you want AS to test on, but it also sounds like you don't realistically have the time. Consider it food for thought: we would love to get arm working and Apple Silicon working, and if having the hardware will unblock you, that's something we should consider. But if it's just going to sit in a closet or create stress for you, I don't want you to shoulder that burden necessarily.

[0x0ade]

If this had happened half a year ago, time wouldn't have been a concern 😅 My first day at my new job is tomorrow. I'll let you know when I'll know if I'll even have the time and energy to work on this in my time off, as I have yet to see what stress awaits me. Thank you for the offer though!

[kevzhao2]

I think I can send in a PR to implement some of the precode walking for arm64 some time this week, which should unblock @hakusaro and @SignatureBeef's issues with general arm64 support.

As much as I hate to say it, at this point I'd be willing to let Azure Pipelines / GitHub Actions compile a helper native library

Embedding the native library seems like the right choice, because I can't see any ways to not use pthread_jit_write_protect_np:

1. We could try to save the page, map the page out, map it in again without the MAP_JIT flag, and restore the page.

   a. But leaving the page without the MAP_JIT flag is problematic if coreclr wants to change something in that page, since it will use pthread_jit_write_protect_np instead of mprotect for changing the page protections!

   b. ...and mapping it in again with the MAP_JIT flag is problematic since we'd have to initialize the page, or somehow have the old values lying around...

2. ...or we could try to map in a new page without the MAP_JIT flag, but this would involve us repointing stuff to the new page, which just runs into the same issues.

Also, by embedding a native library, we'll at least be able to use sys_icache_invalidate.

Getting back to the point of actually iterating through loaded modules, the docs for mach_vm_region_recurse (which, according to some random comments on the internet™️, is what vmmap might be using) look very... promising. And to me, it still feels like "dotnet on macOS can't list all loaded modules" should be fixed by the dotnet team, no matter how much we'll work around that. Let's first see if that function - or vmmap in general - is in any way helpful though. Or maybe pmap instead of vmmap. Once again, I wish I could get my hands on Apple Silicon.

vmmap definitely provides the right output:

__TEXT                      105f60000-106130000    [ 1856K  1856K     0K     0K] r-x/rwx SM=COW          /usr/local/share/dotnet/shared/Microsoft.NETCore.App/6.0.2/libclrjit.dylib
__DATA_CONST                106130000-106144000    [   80K    48K    48K    32K] r--/rwx SM=COW          /usr/local/share/dotnet/shared/Microsoft.NETCore.App/6.0.2/libclrjit.dylib
__LINKEDIT                  106160000-1061cc000    [  432K   432K     0K     0K] r--/rwx SM=COW          /usr/local/share/dotnet/shared/Microsoft.NETCore.App/6.0.2/libclrjit.dylib
__DATA                      106144000-106148000    [   16K    16K    16K     0K] rw-/rwx SM=COW          /usr/local/share/dotnet/shared/Microsoft.NETCore.App/6.0.2/libclrjit.dylib
__DATA                      106148000-106160000    [   96K    16K    16K    32K] rw-/rwx SM=PRV          /usr/local/share/dotnet/shared/Microsoft.NETCore.App/6.0.2/libclrjit.dylib

No idea how stable this, so I would prefer mach_vm_* functions over trying to parse this output, but searching for "libclrjit.dylib" seems reasonable as a last resort.

My first day at my new job is tomorrow.

Good luck! Hope everything goes smoothly :)

[0x0ade]

Initial work on ARM64 precode walking by @kevzhao2 merged with 69a2aa1. I'll keep this issue open for further progress tracking.

I've already mentioned this on the Discord server in more detail, but will recap here. I'm currently unable to use "Mac M1 as a service" services because:

• Scaleway hates my credit card, or my credit card hates Scaleway, as Mastercard's "ID check" popup tells me to contact my credit card client service despite it working everywhere else.
• AWS M1 instances are still in preview, I can only request an invitation and hope for the best.
• MacStadium's prices aren't something I feel comfortable randomly throwing money at, and they don't offer free trials.

My current plan is: Given that I'll need a laptop when I'll relocate closer to my new workplace sooner or later anyway, I'll probably save up until I can afford a M1 MacBook, hitting two birds with one stone. This is to be seen as completely independent from MonoMod - getting it won't mean that I'll immediately work on M1 support for MonoMod, and I want to pay it from my own income, not from community donations which could introduce pressure. By the time I'll get it, I'll hopefully have settled a bit more at my new job and will be able to work on this in my spare time at my own pace. If the community ends up creating solid M1 support for MonoMod before that happens, I'll finally be able to review it by then, and continue maintaining it. And if something happens to my new employment (I'm only one week in, and it's going well so far), I'll see how to move forward from there.

[hakusaro]

@0x0ade hey! how's the new job? I joined your patreon a while back, but I'd be more than happy to throw some gas on the fire and send you an M1 MacBook to an address you send me if you need one.

[0x0ade]

Saw this pop up in my email inbox again, sorry for the lack of updates, sadly lacking time right now thus keeping it short: I'm relocating next month. All my previous plans were shot down by stuff going on in the household I'm living in right now that I won't dwell much on, which I can't wait to finally get out of.

But I was already very close to buying a M1 MacBook multiple times now. If everything goes well, I might finally press the buy button this or next month.

At this point I can't promise that I'll have the free time to look into M1 support though.

[hakusaro]

No problemo, don't let not having the product stop you though. Once you're settled, hit me up!

[YogurtTheHorse]

Hey! Just wanted to know how is your progress on getting Silicon Mac and the issue itself? :)

[nike4613]

I have finally started working on x64 Mac support on reorganize, after having finally gotten access to one. I'd like to get that working, then I'll take a look at working on Arm, though I don't currently have access to one.

[GrimLothar]

Please ignore this comment, I just want to remember in the future that I'm following this issue as it looks like it's a requirement to get Harmony working on Silicon, which is also the requirement to get FastScriptReload working on Silicon. 🙏

[karanshah-browserstack]

Any ETA when can we expect this to be released?

[nike4613]

There is currently no ETA. I simply don't have the hardware to be able to work on this.

[hakusaro]

@nike4613 you can hit me up in the MonoMod discord (i’m antiparticles) for hardware if you’re seriously only limited by hardware for this issue

[pardeike]

I could contribute something too if necessary. Especially if I'm not the only one.

[Nihlus]

I've run into an instance where I need Harmony on an ARM platform as well, though I'd be happy with any sort of ARM support, not just M1. I'm happy to offer any sort of support I can.

Edit: reading back... MonoMod should have some level of support on ARM already, right? Am I not understanding that correctly? If so, I should perhaps open up a separate issue for the platform I'm currently having trouble with.

[nike4613]

It currently does not have any support for ARM.

[Nihlus]

@nike4613 Gotcha. How does https://github.com/MonoMod/MonoMod.Common/blob/master/RuntimeDetour/Platforms/Native/DetourNativeARMPlatform.cs relate to this?

[nike4613]

That (and all of MMC) are part of legacy MonoMod. Current development (and broadest support) is on the reorganize branch, which is a nearly full rewrite, and that does not have any ARM support.

Legacy's ARM support was all "it works on my machine" as-is, so it got dropped along with the rest of legacy, pending a return when I or someone else has the time/resources to work on it.

[Nihlus]

Okay, I see - thanks for the explanation :) Is there some way I can support or aid the ARM implementation? I'm not familiar enough with MonoMod's architecture or the ABI of ARM processors to take a crack at it all on my lonesome, but I'm more than willing to help out any way I can.

[kirilllysenko]

I would also like to aid in making ARM implementation. I have a hardware to test on, but don't have any experience with MonoMod

[tyronx]

Vintage Story dev here. ARM support would enable us to port our game to Mac. I've doubled my patreon pledge. If additional hardware is needed to accelerate development of this, do let me know.

[MisutaaAsriel]

Forgive me for asking about this here, but has there been any progress made in this regard? VRChat just released their Unity 2022 SDK in beta, and their scripting tool, Udon Sharp, uses Harmony Lib as a key component. Unity 2022 itself brings with it native Apple Silicon support, which greatly improves performance and stability over running Intel builds in Rosetta. However, this issue in particular seems to effect Harmony, which in turn effects Udon Sharp, which in turn effects the VRChat SDK.

Issues regarding Apple Silicon support for Harmony point here. With a major platform having a shift towards newer software, which provide Apple Silicon support, and said platform having plans for Apple Silicon support themselves in the extended pipeline (iOS/macOS), I fear this issue might become more pressing as time goes on.

[hakusaro]

I'm still happy to sponsor hardware for people who want to work on getting this (now, reorganize) working on Apple Silicon (including arm64/arm64e) architectures.

[Chicken-Bones]

I'm still happy to sponsor hardware for people who want to work on getting this (now, reorganize) working on Apple Silicon (including arm64/arm64e) architectures.

Likewise

[zed-0xff]

I'm still happy to sponsor hardware for people who want to work on getting this (now, reorganize) working on Apple Silicon (including arm64/arm64e) architectures.

+1

[Guztaver]

I would also like to aid in making ARM implementation. I have a hardware to test on, but don't have any experience with MonoMod

Me neither, I am interested in running tMod on the Arm64 CPU, so, I can help testing, since I don't know how MonoMod works too

[Fadenfire]

I've experimented with this issue to try to figure out what all needs to be done for arm64 support. Some of what I've discovered has already been mentioned here, but there are also some things that I have not seen mentioned that may be helpful for creating a proper port. I've mostly been trying to get my changes to work on .NET core 6, so I don't know how well my changes work for other versions of .NET or other runtimes. Here's a list of my major observations:

1. As mentioned before in this issue, apple silicon requires that pages are never mapped write and exec at the same. It instead requires you to map the memory with MAP_JIT and then use pthread_jit_write_protect_np to toggle all MAP_JIT pages between read-write and read-exec for the current thread. This means that PatchData in MacOSSystem needs to be modified to use pthread_jit_write_protect_np when copying to executable memory. Unfortunately, the memory copy itself can't be done in managed code, as calling pthread_jit_write_protect_np(0) will disable execution for ALL MAP_JIT pages, which includes the JIT compiled code of PatchData. This means trying to use pthread_jit_write_protect_np inside of managed code will always result in BAD_ACCESS. I solved this by created a small C++ helper lib with a simple function like this:

   extern "C" void copy_to_jit(void* from, void* to, size_t length)
   {
       pthread_jit_write_protect_np(0);
   
       std::memcpy(to, from, length);
   
       pthread_jit_write_protect_np(1);
   }

   Which is then called from managed code.

2. MonoMod hooks into CILJit::compileMethod by replacing the corresponding entry in the CILJit vtable with a pointer to a managed function. However, on apple silicon .NET core sometimes calls compileMethod with pthread_jit_write_protect_np turned off, preventing the execution of any JIT compiled managed code. This is a problem as any managed hooks that we insert into the vtable will crash as soon as compileMethod is called with JIT write protection turned off. The only solution I could find was to create another function in my C++ helper library and then point the vtable entry for compileMethod to my C++ helper function. The original compileMethod method pointer is placed in a global variable inside of the C++ helper library. When compileMethod is called, it'll call my C++ function, which will then use the global variable to call the original compileMethod method along with a call back into managed code to allow MonoMod to manipulate the JIT result. My C++ function can be found here.

3. On arm64, .NET core 6 seems to use a similar JIT system as .NET core 7, placing the compiled JIT code into a separate area instead of entryAddress. This means that I had to backport the fix in Core70Runtime to Core60Runtime. Unfortunately, I ran into a similar problem as above, as the Core70Runtime fix adds a managed hook to ICorJitInfo::allocMem. Since calling any JIT compiled managed code during CILJit::compileMethod crashes, this doesn't work for apple silicon. I was able to fix this by using the same strategy as above, by creating another function in my C++ helper library, overwriting the vtable entry of ICorJitInfo::allocMem with my C++ function, and then storing the original allocMem method in a C++ global variable. There is probably a better way to fix this, but this is what I got to work.

The code for my attempt at adding arm64 support can be found here, if anyone's interested.

[nike4613]

Nice work.

1. ...

I'm glad the new PatchData method interface is proving useful. :)

2. ...

This is slightly concerning. I wonder what causes this?

We already use (if available) a native exception helper stub between native and managed code everywhere we inject ourselves into the JIT/EE boundary. Could we use those stubs to make sure pthread_jit_write_protect_np is in the correct state? (Is it possible to query the state?)

3. ...

This sounds to me like we should inject another type in between Core70 and Core60 (some Core60WXORXRuntime maybe? Also worth double checking with the release branch of dotnet/runtime for it, see what's different) to move much of the .NET 7 support code into.

---

For an actual release, I'd like to avoid having C++-compiled binaries if possible, to avoid needing platform-specific toolsets to build those binaries. Currently, we're able to avoid it by using NASM and LLD-Link; I suspect a similar approach with a portable assembler + linker would probably be the approach here, though the amount of code necessary here is rather unfortunate.

[Fadenfire]

We already use (if available) a native exception helper stub between native and managed code everywhere we inject ourselves into the JIT/EE boundary. Could we use those stubs to make sure pthread_jit_write_protect_np is in the correct state?

I'm pretty sure that is an option. The main issue is that currently PosixExceptionHelper uses CreateSpecialEntryStub to pass the managed function pointer to the exception helper. The generated stub would have to have been created with MAP_JIT, which would cause a crash as soon as it is called from native code. This can probably be fixed by using some sort of global state to pass information to the exception helper instead of a generated stub.

(Is it possible to query the state?)

From what I can find there isn't a way. (although Apple's documentation is very sparse on the subject) This is especially problematic, as sometimes CILJit::compileMethod is called with write protection is disabled and sometimes it's called with it enabled. I just pretended that it was always disabled and re-enabled it accordingly. This seems very fragile though and could lead to crashes later on, so there's probably a better way.

[nike4613]

The generated stub would have to have been created with MAP_JIT

Can we not allocate those in a page marked RW, then remap them as RX? We need few enough of them that wasting most of a page is probably not a huge issue.

[Fadenfire]

Yeah, that's potentially an option as well. I wasn't sure if Apple Silicon allows remapping a page to be exec, but if it does then that would be a much better option than using global state.