Homemenu Memory Shenanigans - Part 3 - Finalle!

Where we left off

If you missed the first part If you missed the second part

tl;dr

I wanted to know how much memory the real homemenu is requesting on real hardware
I patched the relevant svcControlMemory call to crash instead, which would dump the stack and register values - that contain the args passed to svcControlMemory, including size and etc...
The real homemenu is requesting ~13MB of linear heap, i can't even allocate 4MB

External Guidance

I asked about my issue in a couple of forums, and @neobrain (one of the authors of mikage), suggested something interesting.

At this point in time i wasn't sure if my issue was the program state, or the program settings, i will explain.

When i say program state, i mean that i might have to setup the 3ds state in a different way to be able to allocate the linear heap. Maybe call some other IPC call, change the order of the IPC calls that i am performing, or something of that sort.

Or it could be an issue with the program settings, aka, the headers of the homemenu that i am building. So @neobrain suggested i do an experiment to check if the issue is with the program itself, or maybe the headers.

To explain that, we need to talk a bit about exheaders!

ExHeaders

Most 3ds titles are packaged into CXI files, also known as NCCH files. Those files contains multiple important pieces, but we will focus on 2 of them.

The first of which is the code.bin file, this is the executable itself. It's a file that contains the compiled instructions without any wrapper format (ELF or PE) at all, simply compiled instructions.

The second interesting piece are the ExHeaders, these headers are signed by nintendo, and contain everything the 3ds system needs to know about the title to execute it. It contains a wide range of details such as:

Section Sizes - the size of the TEXT section, BSS section and etc...
IPC & Syscall Access - to which services can i perform IPC calls? which syscalls can i call?
FCRAM Memory Region - in which region we would allocate your requested memory? This is the field i changed in the first part
A lot more random stuff - TitleId, Title Dependencies, FS Access...

The Experiment

Let's build a code.bin file that 1 thing exactly, allocate a linear heap using svcControlMemory. Then run that code using the real homemenu exheaders, and using pomelo exheaders. That should tell us if the issue is in the code or in the exheaders.

I am using Luma3ds "game patching" for replacing the real homemenu code.bin and exheader.bin files with pomelo ones. Luma3ds does something interesting when replacing the original code.bin and exheader.bin files with the custom ones, and it is patching them in some cases. For examples when loading the homemenu code.bin file (doesn't matter if the original or a patched one), it patches out some of the security checks that the homemenu performs, such as region checks, or DS flashcart whitelists. You can see the patching that luma3ds performs in patcher.c, specifically in the function patchCode.

Since the code.bin i am building doesn't have the original region check logic, or the DS flashcart whitelist, the loader fails to patch the code.bin file and crashes the system as a result. This is something i also encountered when building pomelo, and i fixed it in pomelo by injecting fake code that looks like the logic that luma3ds is trying to patch but is never actually executed, just so luma3ds would think that it was able to patch the homemenu.

Since i wanted to keep this payload as simple as possible, in this case i rebuild luma3ds without the security checks, just for running the test payload.

Code snippet

I ran the following code that does 3 things:

Call svcControlMemory and tells it to allocate a 4MB linear heap
Copy the return code to R12
Crash the system using svcBreak, so we could read the return code

.arm
.global _start

.equ MEMOP_ALLOC_LINEAR, 0x10003   @ MEMOP_LINEAR (0x10000) | MEMOP_ALLOC (3)
.equ MEMPERM_RW,         0x3       @ MEMPERM_READ (1) | MEMPERM_WRITE (2)
.equ ALLOC_SIZE,         0x400000  @ 4 MB

_start:
    ldr r0, =MEMOP_ALLOC_LINEAR @ op
    mov r1, #0                  @ addr0
    mov r2, #0                  @ addr1
    ldr r3, =ALLOC_SIZE         @ size
    ldr r4, =MEMPERM_RW         @ perm
    svc 0x01                    @ svcControlMemory
    mov r12, r0                 @ stash result code where the crash screen shows it

    mov r0, #0                  @ breakReason = BREAKREASON_PANIC
    svc 0x3C                    @ svcBreak

Results

I ran the code using the original homemenu exheaders, and using pomelo exheaders, and here is what i got

Original Homemenu ExHeaders

Pomelo ExHeaders

AHA!

We got our answer! The issue is in the ExHeaders!

You can see in the crash dump of the original homemenu exheaders that the svcControlMemory call succeeded, indicated by the value 0 in R12. And you can see that when i used pomelo ExHeaders, the value in R12 was 0xD86007F3, which means the svcControlMemory FAILED when using pomelo ExHeaders files.

What Now?

This really narrows down the root cause of our issue. Now i know i just need to make the ExHeaders of pomelo closer to the real ones and finally the heap allocation should work.

I am using makerom to build the exheaders of pomelo. And i am using a template.rsf file to tell makerom which values to put in the pomelo exheaders.

I dumped the CXI file of the original homemenu from my own console, and ran ctrtool on it to extract the headers. I gave claude my current template.rsf file and the desired homemenu exheaders, and told it to make my headers as close as possible to the real ones.

Then, A couple of claude prompts later, I finally found the culprit! The culprit was the KernelVersion value, which indicates for which kernel version that title was built and tested against. After setting the kernel version to 2.50, the linear heap allocation worked!

I started having other issues related to copying buffers from the linear heap to VRAM, but i think that is a story for another time :)

The reason the kernelversion even makes a difference is that there were some changes to the virtual memory allocation in kernel version 2.44. The virtual address to which the physical memory was mapped, was changed in kernel version 2.44. Before 2.44, the virtual address to which heaps were mapped was 0x14000000, and after 2.44 it was 0x30000000.

Even after understanding the connection between the kernel version and why it affects linear heap allocations, i am still not entirely sure why not setting a kernel version causes the allocation to fail. I would expect the OS to be backwards compatible with old games, taht were built for older kernels.

Conclusion

That was a pretty fun research, even tho it was pretty frustrating at times, but i think the most interesting part to take from there is the research itself, and hopefully you learned something new about 3ds heap allocation :)

Like i said, i am now having other issues related to linear memory and VRAM, so i might write some more pretty soon :)