Code for function is too large、High memory usage

[crowforkotlin]

"Code for function is too large" error and 1.6GB Memory Spike during JIT compilation of a 781KB Kotlin/Wasm module

Environment:

• Wasmtime version: v40.0.0 (Release Release Wasmtime 40.0.0 #12192)
• Host Machine (Build): Mac M2 (aarch64)
• Target Device (Runtime): Android (aarch64)
• Compiler: Kotlin/Wasm (WASI) via compileProductionExecutableKotlinWasmWasiOptimize
• Project Link: crowforkotlin/wasmline
• Source File: Plugin.kt

Description:
I am reporting a Compilation error: Code for function is too large that occurs during the JIT loading phase on Android.

Despite the generated .wasm binary being only 781.63 KB, the compilation fails when a single Kotlin function contains a large amount of logic/content. During the attempt to compile this module, I observed a massive RSS memory spike on the Android device, jumping from ~235 MB to over 1.6 GB in just 4 seconds, eventually resulting in the "function too large" error.

Wasmtime Configuration:
The engine is configured specifically for Android compatibility to avoid signal conflicts with ART and minimize VSS overhead:

/**
 * Creates and configures the Wasm Config object.
 *
 * Critical Android Settings:
 * 1. Signals-based traps DISABLED: To prevent conflicts with Android ART signal handlers (SIGSEGV).
 * 2. Memory Guard Size = 0: To prevent VSS (Virtual Set Size) OOM on 32-bit or limited devices.
 * 3. GC / Exceptions: Enabled for Kotlin/Wasm support.
 */
wasm_config_t *Engine::createConfig() {
    wasm_config_t *conf = wasm_config_new();

    // Feature Flags for Kotlin/Wasm support
    wasmtime_config_wasm_gc_set(conf, true);
    wasmtime_config_wasm_function_references_set(conf, true);
    wasmtime_config_wasm_exceptions_set(conf, true);

    // Optimization: Disable SIMD if not strictly needed (improves compatibility)
    wasmtime_config_wasm_simd_set(conf, false);
    wasmtime_config_wasm_relaxed_simd_set(conf, false);

    // [CRITICAL] Disable signal handlers to avoid crash conflicts with Android Runtime (ART)
    wasmtime_config_signals_based_traps_set(conf, false);

    // [CRITICAL] Set guard pages to 0 to minimize Virtual Memory usage (prevents OOM)
    wasmtime_config_memory_guard_size_set(conf, 0);

    // Set max stack size (512KB is usually sufficient for mobile logic)
    wasmtime_config_max_wasm_stack_set(conf, 512 * 1024);

    // Compiler Optimization Strategy: Optimize for Speed and Binary Size
    wasmtime_config_cranelift_opt_level_set(conf, WASMTIME_OPT_LEVEL_SPEED_AND_SIZE);
    wasmtime_config_cranelift_debug_verifier_set(conf, false);

    return conf;
}

Reproduction Steps:

1. Add a large amount of content/logic into the wasmWasiMain function in Plugin.kt.
2. Build the wasm file: ./gradlew wasmtime-core:plugin:compileProductionExecutableKotlinWasmWasiOptimize.
3. The resulting plugin.wasm is ~781 KB.
4. Load the module on an Android device using the configuration above.

Observed Behavior:

• Total Wasm Size: 781.63 KB (Small)
• Memory Usage: Spikes from 234MB to 1.6GB.
• Result: Compilation error: Code for function is too large.

Full Android Logs:

00:49:34.204  I  [MotionEvent] ViewRootImpl windowName 'crow.wasmtime.wasmline/crow.mordecai.wasmline.MainActivity', { action=ACTION_DOWN, id[0]=0, pointerCount=1, eventTime=51746485, downTime=51746485, phoneEventTime=00:49:34.191 } moveCount:0
00:49:34.206  D  getMiuiFreeformStackInfo mTmpFrames.miuiFreeFormStackInfo: null
00:49:34.217  I  CreateGraphicsPipeline pipeline cache hit. elpased time: 0.32 ms.
00:49:34.218  E  FrameInsert open fail: No such file or directory
00:49:34.230  I  This is non sticky GC, maxfree is 33554432 minfree is 8388608
00:49:34.227  W  type=1400 audit(0.0:2008818): avc:  denied  { getopt } for  path="/dev/socket/usap_pool_primary" scontext=u:r:untrusted_app:s0:c190,c257,c512,c768 tcontext=u:r:zygote:s0 tclass=unix_stream_socket permissive=0 app=crow.wasmtime.wasmline
00:49:34.322  I  [MotionEvent] ViewRootImpl windowName 'crow.wasmtime.wasmline/crow.mordecai.wasmline.MainActivity', { action=ACTION_UP, id[0]=0, pointerCount=1, eventTime=51746611, downTime=51746485, phoneEventTime=00:49:34.316 } moveCount:0
00:49:34.332  I  ==============================================
00:49:34.337  I  [Android] Wasm file : plugin.wasm    ||    wasm file exits : true    ||    wasm file path :  /data/user/0/crow.wasmtime.wasmline/cache/plugin.wasm
00:49:34.338  I  [Android] Cwasm cache file : plugin.cwasm    ||    cache file exits : false    ||    cache file path :  /data/user/0/crow.wasmtime.wasmline/cache/plugin.cwasm
00:49:34.345  I  [Wasmtime] Module --> Jit Compiling for /data/user/0/crow.wasmtime.wasmline/cache/plugin.wasm...
00:49:34.363  W  type=1400 audit(0.0:2008819): avc:  denied  { read } for  name="cpu.cfs_quota_us" dev="cgroup" ino=64 scontext=u:r:untrusted_app:s0:c190,c257,c512,c768 tcontext=u:object_r:cgroup:s0 tclass=file permissive=0 app=crow.wasmtime.wasmline
00:49:34.363  W  type=1400 audit(0.0:2008820): avc:  denied  { read } for  name="cpu.cfs_period_us" dev="cgroup" ino=65 scontext=u:r:untrusted_app:s0:c190,c257,c512,c768 tcontext=u:object_r:cgroup:s0 tclass=file permissive=0 app=crow.wasmtime.wasmline
00:49:34.363  W  type=1400 audit(0.0:2008821): avc:  denied  { read } for  name="cpu.cfs_quota_us" dev="cgroup" ino=10 scontext=u:r:untrusted_app:s0:c190,c257,c512,c768 tcontext=u:object_r:cgroup:s0 tclass=file permissive=0 app=crow.wasmtime.wasmline
00:49:34.363  W  type=1400 audit(0.0:2008822): avc:  denied  { read } for  name="cpu.cfs_period_us" dev="cgroup" ino=11 scontext=u:r:untrusted_app:s0:c190,c257,c512,c768 tcontext=u:object_r:cgroup:s0 tclass=file permissive=0 app=crow.wasmtime.wasmline
00:49:34.441  E  ->pid:19641,processName:"crow.wasmtime.wasmline",Rss Memory Size Change 234952kb -> 342100kb
00:49:34.494  E  ->pid:19641,processName:"crow.wasmtime.wasmline",Rss Memory Size Change 342100kb -> 428844kb
00:49:34.544  E  ->pid:19641,processName:"crow.wasmtime.wasmline",Rss Memory Size Change 428844kb -> 541612kb
00:49:34.602  E  ->pid:19641,processName:"crow.wasmtime.wasmline",Rss Memory Size Change 541612kb -> 657240kb
00:49:34.928  E  ->pid:19641,processName:"crow.wasmtime.wasmline",Rss Memory Size Change 657240kb -> 860432kb
00:49:35.761  E  ->pid:19641,processName:"crow.wasmtime.wasmline",Rss Memory Size Change 860432kb -> 959324kb
00:49:36.012  E  ->pid:19641,processName:"crow.wasmtime.wasmline",Rss Memory Size Change 959324kb -> 1075892kb
00:49:36.184  E  ->pid:19641,processName:"crow.wasmtime.wasmline",Rss Memory Size Change 1075892kb -> 1183820kb
00:49:37.835  E  ->pid:19641,processName:"crow.wasmtime.wasmline",Rss Memory Size Change 1183820kb -> 1306888kb
00:49:38.482  E  ->pid:19641,processName:"crow.wasmtime.wasmline",Rss Memory Size Change 1306888kb -> 1450064kb
00:49:38.639  E  ->pid:19641,processName:"crow.wasmtime.wasmline",Rss Memory Size Change 1450064kb -> 1612484kb
00:49:38.700  E  ->pid:19641,processName:"crow.wasmtime.wasmline",Rss Memory Size Change 1612484kb -> 1389664kb
00:49:38.700  E  ->pid:19641,processName:"crow.wasmtime.wasmline",Rss Memory Size Change 1389664kb -> 1011816kb
00:49:38.702  E  [Wasmtime] Module --> Error loading module /data/user/0/crow.wasmtime.wasmline/cache/plugin.wasm: Compilation error: Code for function is too large
00:49:38.708  I  [Wasmline] Load failure, because native load return false, file path is :  /data/user/0/crow.wasmtime.wasmline/cache/plugin.wasm
00:49:38.790  E  ->pid:19641,processName:"crow.wasmtime.wasmline",Rss Memory Size Change 1011816kb -> 727648kb

Artifacts:
Wasmline-wasmline-core-plugin.wasm.zip

• Wasm File Size (781.63 KB):
  
• Error Detail:
  
• Success state (after reducing content):
  

Analysis & Questions:

1. Why does a relatively small Wasm file (under 1MB) trigger a "Code too large" error? It seems the Kotlin/Wasm compiler is generating a single function with extremely high complexity.
2. The memory usage spike to 1.6GB suggests that Cranelift might be hitting a worst-case scenario in its e-graph optimization or register allocation phase.
3. Are there any specific Cranelift settings to mitigate this, or is this a fundamental limit on function complexity regardless of the total .wasm file size?

[crowforkotlin]

#11682 @cfallin

[crowforkotlin]

Wasmline-wasmline-core-plugin.wasm.zip

./wasmtime compile Wasmline-wasmline-core-plugin.wasm -o plugin.cwasm \
    --target aarch64-linux-android \
    -W gc=y \
    -W function-references=y \
    -W exceptions=y \
    -W simd=n \
    -W relaxed-simd=n \
    -O static-memory-guard-size=0 \
    -O dynamic-memory-guard-size=0 \
    -O signals-based-traps=n \
    -O opt-level=2 \
    -C cranelift-debug-verifier=no

[crowforkotlin]

wasm.zip

[cfallin]

Just an FYI -- I am on PTO and then at a conference, so I will be able to investigate this starting Mon, Jan 12. (Thanks for the report!)

[cfallin]

Actually, this was bothering me and I at least wanted to verify, so I double-checked with your provided Wasm module and the command

./wasmtime compile Wasmline-wasmline-core-plugin.wasm -o plugin.cwasm \
    --target aarch64-linux-android \
    -W gc=y \
    -W function-references=y \
    -W exceptions=y \
    -W simd=n \
    -W relaxed-simd=n \
    -O static-memory-guard-size=0 \
    -O dynamic-memory-guard-size=0 \
    -O signals-based-traps=n \
    -O opt-level=2 \
    -C cranelift-debug-verifier=no

However, both with a wasmtime built from main (b5272a5) and from just before the 40.0 branch was cut (ac3358b), running on macOS/aarch64 as you specify, I see the command complete just fine with minimal time taken, and definitely no "Code for function is too large" error -- e.g. from time,

target/release/wasmtime compile Wasmline-wasmline-core-plugin.wasm -o    -W    0.38s user 0.03s system 328% cpu 0.126 total

Am I missing some setting? I'm inclined to close as "cannot reproduce" if not but want to make sure I'm running exactly what you are.

[crowforkotlin]

Actually, this was bothering me and I at least wanted to verify, so I double-checked with your provided Wasm module and the command

./wasmtime compile Wasmline-wasmline-core-plugin.wasm -o plugin.cwasm \
    --target aarch64-linux-android \
    -W gc=y \
    -W function-references=y \
    -W exceptions=y \
    -W simd=n \
    -W relaxed-simd=n \
    -O static-memory-guard-size=0 \
    -O dynamic-memory-guard-size=0 \
    -O signals-based-traps=n \
    -O opt-level=2 \
    -C cranelift-debug-verifier=no

However, both with a wasmtime built from main (b5272a5) and from just before the 40.0 branch was cut (ac3358b), running on macOS/aarch64 as you specify, I see the command complete just fine with minimal time taken, and definitely no "Code for function is too large" error -- e.g. from time

target/release/wasmtime compile Wasmline-wasmline-core-plugin.wasm -o    -W    0.38s user 0.03s system 328% cpu 0.126 total

Am I missing some setting? I'm inclined to close as "cannot reproduce" if not but want to make sure I'm running exactly what you are.

The file download was incorrect. You used a WASM file generated from a small amount of code; it was my oversight for sending it to the wrong location. While the CWASM generated from that file works fine, using the larger WASM file (700KB+) to generate a CWASM triggers an exception. I’m sending you the latest version now for you to check.

wasm.zip

The Kotlin/WASI build process uses Gradle tasks to generate WASM artifacts. The output ZIP includes 'kotlin' and 'optimize' folders, which seem to act as a size-optimization toggle similar to a release environment switch.[1] Yesterday, I inspected the WASM file with wasmtime-tools and discovered it contains over 22,000 global variables.[1] I realize this is not an ideal implementation, but I'm concerned that if a user implements extensive logic in the future, the resulting large file size might trigger exceptions during CWASM generation.

[cfallin]

Thanks for that. A few stats from your optimized/Wasmline-wasmline-core-plugin.wasm in your latest zip file, function 127 in the module:

• There are 4005519 (4 million) SSA values in the CLIF generated from that function body;
• There are 488081 (488k) basic blocks in that CLIF;
• There are 61796 Wasm bytecode operators in the function body;
• Those operators are distributed according to this histogram:

22433 global.get
16819 call
11238 i32.const
5612 global.set
5608 struct.new
  20 local.get
  12 struct.set
   9 end
   7 local.set
   7 drop
   6 ref.null
   6 ref.is_null
   6 local.tee
   6 if
   2 block
   1 unreachable
   1 try_table
   1 throw_ref
   1 br
   1 array.new_data

So, my thoughts:

• It's not surprising at all that feeding a CFG of 488k basic blocks into an optimizing compiler causes memory usage to grow, potentially to over a gigabyte. That's the nature of the beast: we build IR in memory and do passes over it. Just representing the IR itself, nevermind the analysis results, is going to take a large fraction of the usage you observed. Our optimizing compiler is not designed to have linear-time overhead (that would be impossible while still performing the kinds of optimizations we want to do), so any function this large is likely to blow up.
• struct.new in particular is expanding to a lot of IR, because of its inlined fast-path. I don't think that's the wrong choice though -- it makes sense to make allocation fast for the usual program structure that has maybe a few allocation sites per function, rather than try to optimize for code size for the uncommon case of a single function with 5608 struct.news (!).

The immediate issue is that lowering runs out of VRegs, for which we allow only 2M (2^21), because of bitpacking for efficiency. We could in theory look at increasing that. But at least we are returning a clean error ("function too large") rather than panic'ing.

On a broader scale, I don't think Cranelift (or any optimizing backend) is going to meet your requirements/expectations if that is the code that you need to compile. I would thus recommend either:

• Using Winch, once it supports exceptions and GC (it currently does not support either); or
• Modifying your codegen to not produce such a function, e.g. by splitting it up, using table-driven setup, using initializer expressions for the globals if the graph is acyclic, or something else.

cc @fitzgen for any thoughts on struct.new...

[crowforkotlin]

Interestingly, after refining my Kotlin/Wasm (WASI) implementation, a 700KB Wasm file successfully compiled into a 28MB CWASM via JIT without triggering the "Code for function is too large" error. I suspect the Kotlin/Wasm compiler might be performing aggressive inlining, potentially collapsing numerous function implementations into a single entry point (e.g., the main function). If this aggressive inlining could be disabled or tuned, the issue might be resolved. For context, I have successfully tested the build and execution on an Android app.
Additionally, I have a question regarding Wasmtime: is there a hard limit of 2 million virtual registers? I’m trying to understand if this issue stems from the generation of approximately 4 million (4M) virtual register values.
I am planning to report this to the Kotlin team with the relevant details. While I am not entirely certain about the underlying mechanism, the issue is consistently reproducible.

wasm-success-size28mb.zip

@cfallin

[fitzgen]

struct.new in particular is expanding to a lot of IR, because of its inlined fast-path. I don't think that's the wrong choice though -- it makes sense to make allocation fast for the usual program structure that has maybe a few allocation sites per function, rather than try to optimize for code size for the uncommon case of a single function with 5608 struct.news (!).

We could possibly have a fuel counter for inlining the GC allocation fast path, and start deferring to out-of-line calls once the fuel is spent. This is nice because we already have out-of-line paths for allocation that we can reuse.

Except... the (default) DRC collector already does allocation out-of-line; it only does field initialization inline. I expect that it is this field initialization that is what is blowing up the CLIF size relative to the input Wasm size. The DRC collector's barriers are not small.

That said, we don't have any existing code path to do allocation and field initialization out of line, so adding such a path would be purely new maintenance burden.

Chris, did you check that this function doesn't blow any implementation limits? We should be able to compile anything within those limits in theory (even if it hits slow paths or whatever). Those limits should in theory be caught by wasmparser's validation, but it is always possible we missed something... Looking into this a little more now.

[fitzgen]

Chris, did you check that this function doesn't blow any implementation limits? We should be able to compile anything within those limits in theory (even if it hits slow paths or whatever). Those limits should in theory be caught by wasmparser's validation, but it is always possible we missed something... Looking into this a little more now.

I'm not seeing this break any limits, fwiw.

[cfallin]

is there a hard limit of 2 million virtual registers? I’m trying to understand if this issue stems from the generation of approximately 4 million (4M) virtual register values.

That's correct. (And yes, this doesn't hit any implementation limits; there are "only" 61k Wasm instructions in the function, which is far below the ~7MB limit.)

I agree that "should be able to compile any function at all within limits without a 'too large' error" is a good goal to have, and is the ideal state to end up in. I also think that's a large project in general: not only the VReg limit (which we could raise today, at a ~few-percent compile time cost) but the blowup in the IR in general is likely going to lead to OOMs etc in worst-cases as long as we have a lot of inlined logic for some operators.

I wonder if it would make sense to have a "small operator footprint" mode where anything nontrivial (more than one or two CLIF operators) becomes a libcall? And then select this mode if the function is too large or contains too many of a certain kind of operator? I guess that's an eager form of your "fuel" idea; starting that way rather than switching partway through. Though fuel seems workable too.

For what it's worth, to ask the usual "what would SpiderMonkey do?" question, it seems that a tier-up heuristic that estimates compilation cost will effectively prevent optimizing compilation if the function body is "too large". So we're in kind of a unique position where the standard Wasm implementation limits are agreed upon by engines that can always fall back to baseline but we must always use the optimizing compiler on every function if we're set up to use it at all. So we're trying to solve a much harder problem (and one that the implementation limits are arguably not designed to guard against).

[crowforkotlin]

I have reported the upstream issue regarding the IR explosion to the Kotlin team (see KT-83497). While we await a fix on their end, I would like to highlight a critical bottleneck in Wasmtime's JIT compilation.
currently, the JIT behavior is unstable for our use case. I observing exponential memory growth when processing these large initialization functions, which rapidly leads to OOM errors. Consequently, I have been forced to migrate my project entirely to AOT compilation to maintain stability.