Code Relocation to ITCM on NXP RT1060

[ofirshe]

Hey,

I am currently using zephyr_code_relocate() and CONFIG_CODE_DATA_RELOCATION to enhance system performance by relocating certain modules to run from ITCM. I typically load the code in two ways:

1. Using a debugger.
2. Via USB serial, utilizing NXP-MCUBootUtility-3.5.0. For this, I set the board’s switch to “Serial Downloader” mode, load the code (I use the zephyr.s19 file), and upon completion, I disconnect the power, switch to “Internal Boot,” and then reboot the system.

However, code relocation isn’t feasible in the second case due to ITCM being volatile memory. I assume this means I need to use a dedicated bootloader when loading the code via USB serial, whereas it's not necessary when using the debugger.

Is there an example of compiling the code with such a bootloader that loads the required modules into ITCM before the application starts executing?

Thank you for your assistance.

[ofirshe]

Hey @DerekSnell,

I recently came across your discussion #75424 and found your comments on code relocation for specified project modules.

I’m wondering how to achieve code relocation when loading via USB serial (using NXP-MCUBootUtility) instead of a debugger. You mentioned "configuring the bootloader"—is there any guidance or documentation on how to set this up?

Best regards

[DerekSnell]

Hi @ofirshe ,
To summarize the i.MX RT boot flow when any code is located in RAM, the code must be stored in non-volatile memory, like the FlexSPI flash. The bootloader in ROM will read from the boot memory, configure the SOC and memory, read the boot header stored in that memory to get details about the boot image, and jump to the reset entry function. The relocation to RAM can be done by the bootloader before the app executes, or the app can relocate code to RAM before the app executes that code. Regardless of the method that programs the boot image in the memory, including the USB Serial Download mode you are using, this code will need to be relocated after POR. The debugger gets around this temporarily, because it will also program the code directly to RAM. But that RAM will be lost when power cycled.

The ROM bootloader does have a RAM Loader feature (and FYI, MCUboot also has a RAM Loader feature). The boot header for the image includes the RAM load address and the size to relocate. The bootloader will then copy that portion of the image from the IVT at the start of the image to the RAM load address. Then it will jump to the reset entry address in the vector table, and the app executes. There is a Kconfig symbol that makes this easy if "zephyr,flash" is in RAM, like ITCM. The Kconfig was recently updated to CONFIG_NXP_FLEXSPI_ROM_RAMLOADER. In Zephyr v3.7.0 and earlier, it was called CONFIG_NXP_IMX_RT_ROM_RAMLOADER.

The Zephyr Code And Data Relocation APIs can also help relocate code to RAM, especially if the whole app does not fit in the RAM, and you want to relocate parts of the app. With zephyr_code_relocate(), the memory is copied after the app starts executing, but early in the startup code.

code relocation isn’t feasible in the second case due to ITCM being volatile memory

That is not true, zephyr_code_relocate() does enable relocating Code (.text) and rodata to RAM. Running the Zperf sample on the i.MX RT boards uses zephyr_code_relocate() to move code to ITCM for performance. And in the discussion #75424 (comment) you referenced, I used zephyr_code_relocate() to move more kernel files and libraries to ITCM. You can reference these as an example. I am not sure why you feel this does not work without the debugger. Perhaps you have the NOCOPY flag set?

From experience I know that zephyr_code_relocate() does not enable relocating everything from flash to RAM. For relocating the memory that is typically non-volatile, zephyr_code_relocate() can relocate the TEXT and RODATA linker sections. But Zephyr uses other linker sections which go into the "zephyr,flash" memory, and zephyr_code_relocate() will not relocate those. Also, some code/rodata in the kernel or SOC files are used in the startup process before this Relocation occurs. Meaning the Relocated code may need to execute before that code was copied to RAM, which will not work and prevents the app startup from finishing. When I wrote up the list of files and libraries I relocated in #75424 (comment), there are several more kernel files I tried relocating, but found the app failed. So if you try to relocate some code and the app no longer works, that is likely the reason. If you need to relocate that code, you will need to do it before the app executes, like with a bootloader. Or you could use a custom reset vector to do some relocation, and then jump to the Zephyr reset entry.

Best regards

[ofirshe]

Hey @DerekSnell,

First thank you for your detailed response. In my case, the entire application doesn't fit in the ITCM, so I’ve tried relocating the major modules (including those you mentioned in #75424 (comment), such as fsl_enet and other networking components), which has significantly improved system performance.

I apologize if my previous wording was unclear. I didn’t mean to imply that loading code via USB serial with code relocation enabled wouldn’t work. My question is more about how to build the final application, which will be loaded into flash and includes both the bootloader and the original application to manage the relocation process of the specified modules after a reset.

As you mentioned, when using a debugger, the bootloader isn’t needed since the ELF file is parsed, and the content is written directly to the appropriate addresses. However, I’d like to understand how to build a "concatenated" application that combines my original app and the bootloader, allowing me to download the zephyr.s19 file to 0x60000000 using the NXP-MCUboot utility. This approach requires a power cycle to switch from serial downloader mode to internal boot mode.

Regarding the issue you mentioned where the application didn’t work after relocating some modules, did you investigate the cause? What could lead to an application that runs from both FlexSPI flash and ITCM failing? Also, what symptoms did you observe during this failure?

Best regards

[DerekSnell]

Hi @ofirshe ,
Since you do not want to relocate all of the app to SRAM, I think your best option is using Zephyr's Code And Data Relocation APIs. The RAM Load feature in both the ROM bootloader in the SOC and in MCUboot are intended to load the entire image to RAM before the app executes. But since you are using zephyr_code_relocate() already to move specific files to RAM, you do not need to integrate a bootloader with the app for this relocation. The relocation is managed in the Zephyr startup code. The build image will store the code in flash, and the app will relocate to RAM. And this image can be programmed to flash using any method: west flash, Serial Downloader mode, a debugger, etc..

For a reference, you can build the Zperf sample for the mimxrt1060_evk board. That sample uses zephyr_code_relocate() to relocate a few files to ITCM. You can program it to flash using any method above, and it will boot from flash. If you still have issues with your app, you can compare with this sample to see what is different.

Regarding the issue you mentioned where the application didn’t work after relocating some modules, did you investigate the cause?

Yes, as I shared before, the cause is because the CPU attempts to access the code or RODATA before the relocation function moves it to RAM. That will not work.

Also, what symptoms did you observe during this failure?

The app fails in the startup code. It never gets to main(), and the boot banner does not print. Since the CPU accesses RAM that has not been loaded yet, it usually manifests as a memory access issue, like a hard fault, or an illegal instruction.

Best regards

[ofirshe]

Since you do not want to relocate all of the app to SRAM.

Are you referring to external SDRAM? If so, has anyone attempted to run the entire application from that memory without encountering the issues you mentioned? If you have an example, that would be helpful. Do you think running from SDRAM would improve performance compared to QSPI flash? We're currently seeing very poor networking performance with the default configuration.

the cause is because the CPU attempts to access the code or RODATA before the relocation function moves it to RAM.

Why is the CPU accessing RAM that hasn't been loaded yet? Shouldn't it only access the RAM after the relocation is complete?

Best Regards

[DerekSnell]

Hi @ofirshe

I was referring to the internal SRAM, since you were asking about ITCM. But yes, you can execute the code from external SDRAM. Since SDRAM is quite large, you can use the ROM bootloader to load the entire zephyr,flash into SDRAM. And yes, QSPI latencies are pretty slow, running from SDRAM can be higher performance. NXP has appnote AN12437 that details the different memory options and performance considerations.

Attached is a simple example SDRAM_hello_world.zip that moves the entire app to SDRAM, and uses the ROM bootloader to load the RAM at boot, before the app executes. This example was tested on the mimxrt1060_evkb board, using the latest main branch with commit e11ddd5.

Some notes about this example: the devicetree overlay splits the SDRAM memory into two regions:

	/* Divide SDRAM into two partitions for Code and Data */
	/* Micron MT48LC16M16A2B4-6AIT:G */
	sdram_code: memory@80000000 {
		device_type = "memory";
		reg = <0x80000000 DT_SIZE_M(16)>;
	};
	sdram_data: memory@81000000 {
		device_type = "memory";
		reg = <0x81000000 DT_SIZE_M(16)>;
	};

Then the entire app is placed in SDRAM:

	chosen {
		zephyr,flash = &sdram_code;
		zephyr,sram = &sdram_data;
	};

The board .conf file configures the ROM bootloader to load the image to RAM at boot:

CONFIG_NXP_FLEXSPI_ROM_RAMLOADER=y

The LMA is adjusted in the app's Kconfig file. The linker builds the code/rodata for SDRAM at base address 0x8000_0000, but stores in FlexSPI flash at base address 0x1800_0000. So the LMA adjustment is +0x9800_0000.

if NXP_FLEXSPI_ROM_RAMLOADER

# Adjust LMA when zephyr,flash is in SDRAM
FLASH_CHOSEN := zephyr,flash
FLASH_BASE := $(dt_chosen_reg_addr_hex,$(FLASH_CHOSEN))
FLEXSPI_BASE := $(dt_nodelabel_reg_addr_hex,flexspi,1)
config BUILD_OUTPUT_ADJUST_LMA
	default "0x100000000 - $(FLASH_BASE) + $(FLEXSPI_BASE)"

endif # NXP_FLEXSPI_ROM_RAMLOADER

Best regards