Back to blog

DataWeave under load: streaming, deferred output, and when Batch wins

2026-09-17

DataWeave in Mule 4 can read data three ways: in-memory (whole document in RAM), indexed (disk + index, random access), or streaming (sequentially; unit = CSV record / JSON array element / XML collection). Combining source streaming with deferred=true on output lets you push data end-to-end without a full index and without writing output to disk first — faster and cheaper than the default read/write paths.

What this piece is not: a “Top 10 DataWeave functions” list, a Batch Job replacement for ETL with resume and per-record errors, or a promise that streaming=true “always speeds up map/filter.” Streaming is not on by default. It only works with sequential access to stream units — and fails quietly when a script needs random access to the whole document.

Audience: developers and integration leads with GB files, memory pressure on CloudHub / workers, and transforms on the hot path. Below: three read strategies, pitfall cards (symptom → cause → docs → pattern), a DW vs Batch vs Cache decision tree, checklist, and AEO FAQ.

Three read strategies (before you flip streaming=true)

The official DataWeave formats page describes three read strategies: in-memory, indexed, and streaming (Supported Data Formats).

In-memory

The whole document lands in memory. You get full random access — any selector, any order. On large payloads that is a short road to OOM or aggressive GC. Works for all formats, but is not the “under load” path.

Indexed

The parser builds an index and may spill content to disk while keeping random access like in-memory. Docs: indexed readers support files up to about 20 GB; larger → streaming (faster, no max input size documented) (Indexed Readers).

The threshold for disk buffers is com.mulesoft.dw.max_memory_allocation — default 1,572,864 bytes (1.5 MB). Above that you see dw-buffer-input-*.tmp, dw-buffer-output-*.tmp, and dw-buffer-index-*.tmp (Memory Management).

Streaming

Data flows sequentially; only the current unit is in memory. Unit depends on format: CSV row, JSON array element, XML collection element. No random access to the whole document (Streaming in DataWeave). Formats with streaming: CSV, JSON, XML, Excel (XLSX) — this article focuses on CSV/JSON/XML.

Enabling streaming on the source (not inside the script alone)

The switch is the MIME type on the data source — outputMimeType / mimeType on HTTP Listener, Request, File, Set Payload, etc. Transform Message without a source flag does not “magically enable streaming.”

<http:listener doc:name="Listener"
    outputMimeType="application/json; streaming=true"
    config-ref="HTTP_Listener_config" path="/input"/>
  • CSV: unit = row; random access within a record is OK.
  • JSON: unit = array element. From Mule 4.3 streaming also covers arrays outside the root (in 4.2 the root had to be an array).
  • XML: both required: streaming=true and collectionPath (collection location). Missing either = no stream.

Example from docs (note casing in the official example — collectionpath):

<http:listener
    outputMimeType="application/xml; collectionpath=order.order-items; streaming=true"
    config-ref="HTTP_Listener_config" path="/input"/>

Source: Streaming in DataWeave.

What breaks streaming (even when it “looks fine”)

Streaming = sequential access to units. The patterns below force random access or “rewinding” the document.

Negative indexes and reordering

[payload[-2], payload[-1], payload[3]]

This script needs access to the whole document in a different order than arrival — streaming will not work. Random access within a single CSV/JSON record is allowed.

Double reference to payload / bad JSON key order

You want both payload.family (streamed array) and payload.name, or { a: payload.age, b: payload.family } when age comes after family in the object — the stream does not go backwards. JSON does not guarantee key order: a script may work on this file and fail on another.

Reference from a nested lambda

[1,2,3] map ((item, index) -> payload) — validator and runtime treat this as a reference outside the variable’s definition scope. @StreamCapable criteria (below) catch this explicitly.

orderBy / groupBy / distinctBy (and similar reductions)

Functions that need the entire set before the first result force materialization — sequential streaming does not survive a global sort/group. Docs: streaming = sequential access, no whole-document random access. For GB files, sort/group → Indexed (consciously), push work to a database/downstream, or Batch — not a “cleverer” script with deferred=true.

Pattern: single-pass map / filter; metadata before the collection in the data model; last-element / reorder / global sort-group → indexed or Batch / two passes.

@StreamCapable() — validator, not a runtime guarantee

Experimental annotation: checks whether a script can sequentially read a variable (usually payload). Criteria:

  1. variable referenced once,
  2. no negative index ([-1], etc.),
  3. no reference from a nested lambda.

Requires an input directive with MIME type, e.g. input payload application/json.

False fail: a script may stream on a concrete file while the validator fails — because JSON does not guarantee key order, and the annotation processor cannot assume fixed order. Treat @StreamCapable as a sequentiality linter, not a production certificate.

Source: Streaming in DataWeave — Validate a Script.

deferred=true: handoff without disk — and without normal error handling

Writer property deferred=true in the output directive generates output as a stream and defers script execution until the next processor consumes it:

output application/json deferred=true

Official NOTE: exceptions aren’t handled when deferred=true. In Studio debug: the exception logs to the console, but the flow does not stop at Transform Message — problems surface at the consumer (File Write, HTTP, next component).

End-to-end from docs: File listener streaming=true → Transform with deferred=true → File write. That is a sound pattern when the next hop truly consumes the stream. When you need synchronous fail-fast on Transform — skip deferred.

deferred is also a binary format writer property (Binary Format).

Mule memory vs DataWeave memory (two buffer layers)

Repeatable streams (Mule 4)

By default Mule 4 uses repeatable streams (EE: file-store, starts 512 KB in-memory, then disk). Larger inMemorySize = less disk I/O, but fewer concurrent requests. An unused stream holds file handles, DB cursors, HTTP connections until the event ends — pool exhaustion / OOM risk. set-payload value="#[payload]" does not consume the stream (Streaming in Mule Apps).

non-repeatable-stream only when a single read is enough and you know Cache / For Each / some Transforms need full consumption.

DataWeave buffers

Separate layer: dw-buffer-*.tmp in java.io.tmpdir, off-heap pool (com.mulesoft.dw.memory_pool_size, com.mulesoft.dw.directbuffer.disable). Strings >1.5 MB in JSON/XML are split into chunks at the same max_memory_allocation threshold — a performance cost; disable com.mulesoft.dw.buffered_char_sequence.enabled only with deliberate RAM headroom (Indexed Readers, Memory Management).

Performance antipatterns (before you enable streaming)

Transform inside foreach

Official antipattern: iterate + per-item Transform instead of one collection map, then foreach for side-effects (tuning-app-design — DataWeave).

Needless format hops and excess fields

Help article How To Improve Dataweave Performance: avoid unnecessary JSON↔XML conversions; map only needed fields (Help).

indent=false and logging

On large outputs indent=false reduces size and client load. Do not log heavy DW expressions on every request (tuning-app-design).

Parallel For Each

Buffers results of all routes into a list — OOM risk on large element counts. Docs say plainly: large payloads → Batch (Parallel For Each). Help repeats the warning.

Cache scope

Helps for frequently repeated, rarely changing data. Caches repeatable streams; not non-repeatable. In prod avoid default in-memory OS — Object Store + TTL / max entries (Cache Scope, Tuning Caching).

When Batch wins over DataWeave

Batch Job (EE) = reliable, asynchronous processing larger-than-memory: persistent queues, resume after crash/redeploy, per-record errors, steps + aggregator/bulk to SaaS (Batch Processing).

DW streaming = fast single-pass transform of a large document → next processor (write/HTTP). No resume guarantee and no native per-record error model like Batch.

Common production pattern: DW prepares/splits shape (or streams shape) → Batch processes records. Help suggests Batch for very large payloads; Parallel For Each docs say the same under OOM risk.

Do not shove every GB of CSV into Batch “just in case” — and do not push ETL that needs resume into pure DW.

Mini decision tree

Question Path
Need random access on payload >1.5 MB? Indexed (consciously) or redesign the script
Single-pass CSV / JSON array / XML collection → write/next hop? streaming=true on source + consider deferred=true
Resume / DLQ / bulk API / multi-step per record? Batch Job
Same lookup/response repeatedly, data rarely changes? Cache (+ selective map); OS + TTL in prod
Independent I/O on items, but not “millions”? Parallel For Each with concurrency limit — not for GB sets

Practical checklist

  1. Source: set streaming=true on listener/request/file — do not assume Transform “streams by itself.”
  2. XML: collectionPath and streaming=true (check property casing in Studio vs docs example).
  3. Script: one reference to the streamed variable; no payload[-1] / whole-document reorder; metadata before the collection.
  4. Output: deferred=true only when the next hop consumes the stream; test the failure path at the consumer, do not assume fail-fast on Transform.
  5. Measure: heap, /tmp (dw-buffer-*.tmp), concurrency after changing inMemorySize / max_memory_allocation.
  6. Antipatterns: one collection map instead of Transform in foreach; zero needless JSON↔XML; indent=false on large outputs.
  7. Parallel For Each / Cache: deliberately — Batch for large sets; Cache not on non-repeatable; OS strategy in prod.
  8. Durability: if you need resume / per-record errors → Batch, not a “cleverer” DW script.

FAQ

1. Is DataWeave streaming enabled by default?

No. You must set reader property streaming=true on the source (outputMimeType / mimeType). Without it DataWeave may take the in-memory or indexed path. Separately: deferred=true on output defers write and hands the stream onward.

2. How does indexed read differ from streaming?

Indexed parses the document, builds an index (often on disk), and gives random access — about a 20 GB cap in docs. Streaming reads sequentially by format unit, with no max input size in docs, but without whole-document random access. Indexed is the memory↔disk compromise; streaming is fastest for single-pass transforms.

3. What does deferred=true do and why can an error “disappear” from Transform Message?

deferred=true generates output as a stream and defers execution until the next component consumes it. Officially exceptions are not handled as usual — in Studio debug the flow does not stop on Transform; the error shows in the log / at the consumer. Use when the next hop reads the stream; skip deferred when you need synchronous fail-fast.

4. Why does XML streaming need both collectionPath and streaming=true?

XML has no arrays like JSON. collectionPath points at the collection location (e.g. order.order-items); only then do elements under that path become the stream unit. Docs: missing either setting = no stream.

5. When does a script pass runtime but @StreamCapable fails (or the reverse)?

The validator checks sequentiality rules (one reference, no negative index, no nested-lambda). It may fail even though a given JSON file streams — because JSON key order is not guaranteed. Conversely: a script without the annotation may “work” on a small file and under load fall into indexed/in-memory and eat memory.

6. How do payload[-1] / double payload break streaming?

A negative index and reordering elements require random access to the whole document. Double payload (e.g. family + name) requires “rewinding” the stream. In both cases you lose the sequential model — you fall back to indexed/in-memory or get an @StreamCapable validation error.

7. When should I choose Batch Job over DataWeave streaming?

When you need reliable async ETL: persistent queues, resume after crash/redeploy, per-record error handling, aggregator/bulk to external systems. DW streaming wins for single-pass document-shape transforms to the next hop. Often you combine both: DW → Batch.

8. What do dw-buffer-*.tmp files and com.mulesoft.dw.max_memory_allocation mean?

Those are DataWeave disk buffers (input/output/index) when payload exceeds the threshold — default 1.5 MB. Files live in java.io.tmpdir until streams close / the event ends. Raise the threshold only when you have RAM; for sequential GB files prefer streaming over indexed windowing.

9. Does Cache scope help with large streams?

Cache helps for repeatable, rarely changing data (lookup/reference). It caches repeatable streams; not non-repeatable. A large stream in default in-memory OS in prod can eat heap — use Object Store with expiry / max entries. It is not a substitute for streaming or Batch.

10. How do I avoid Transform inside foreach on large collections?

Do one transform of the whole collection (map / mapObject), then foreach if you need side-effects (HTTP per item, DB write). Per-element Transform in a loop creates needless events and CPU — an antipattern from the official tuning guide.

Soft CTA

Designing GB-file flows or hot-path transforms and want streaming vs indexed vs Batch without experimenting on production? Solita is a Nordic MuleSoft partner with delivery from Poland (EU-shoring) — we help teams pick the read model and reliability shape under real load. No marketing checklist and no “#1” claims: a concrete review of scripts, buffers, and the line where DataWeave hands off to Batch.

Sources

DataWeave / Mule documentation

Help

Version path twin

How-to video (verified titles / oEmbed)

Editorial note: a dedicated, stable YouTube how-to for “DataWeave streaming=true + deferred” was not found in this pass — primary source remains Streaming in DataWeave.