Skip to content

refactor(execution): Move Forwarding Into Dedicated Crate - #4232

Draft
refcell wants to merge 4 commits into
mainfrom
rf/refactor/move-forwarding-into-crate
Draft

refactor(execution): Move Forwarding Into Dedicated Crate#4232
refcell wants to merge 4 commits into
mainfrom
rf/refactor/move-forwarding-into-crate

Conversation

@refcell

@refcell refcell commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Summary

This moves transaction forwarding consumers, forwarders, configuration, metrics, and lifecycle ownership from txpool into base-tx-forwarding. It replaces the separate spawned consumer and forwarder APIs with a unified service and shutdown handle. Each destination now has independent deduplication and bounded delivery so a slow builder cannot cause transactions to be lost through shared broadcast lag.

@refcell refcell added the execution Area: execution label Jul 31, 2026
@refcell refcell self-assigned this Jul 31, 2026
@cb-heimdall

Copy link
Copy Markdown
Collaborator

🟡 Heimdall Review Status

Requirement Status More Info
Reviews 🟡 0/1
Denominator calculation
Show calculation
1 if user is bot 0
1 if user is external 0
2 if repo is sensitive 0
From .codeflow.yml 1
Additional review requirements
Show calculation
Max 0
0
From CODEOWNERS 0
Global minimum 0
Max 1
1
1 if commit is unverified 0
Sum 1

Comment on lines +80 to +94
let mut pending = tx;
loop {
match self.sender.try_send(pending) {
Ok(()) => {
self.recently_sent.mark_sent(hash);
txs_sent += 1;
self.emit_builder_consumed_event(hash, iterator_index);
break;
}
Err(mpsc::error::TrySendError::Full(tx)) => {
if self.cancel.is_cancelled() {
return;
}
pending = tx;
std::thread::sleep(self.config.poll_interval);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The try_sendFull spin loop blocks the entire consumer for this destination, including the cancellation check at line 66 (which only runs once per transaction, before entering this loop). If the mpsc channel is persistently full (e.g. the forwarder is rate-limited), this loop burns CPU on thread::sleep wakeups, and the outer for tx in best_txs iterator is stalled — meaning the pool snapshot grows stale while we wait for a single slot.

A simpler design: on Full, break out of the inner for tx loop entirely and fall through to the outer while !self.cancel.is_cancelled() loop, which already sleeps on poll_interval when txs_sent == 0. The transaction would be retried from a fresh best_transactions() snapshot on the next iteration. This avoids holding a stale iterator snapshot while blocked and is more consistent with the "per-iteration snapshot" model documented in the doc comment.

Comment on lines +113 to +115
let buffer_limit =
if config.max_batch_size == 0 { queue_capacity } else { config.max_batch_size };
let buffer = Vec::with_capacity(buffer_limit);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When max_batch_size == 0 (unlimited), buffer_limit is set to queue_capacity (default 10,000), and Vec::with_capacity(buffer_limit) eagerly allocates space for 10,000 entries. Each BufferedTransaction is roughly ~100+ bytes so this is ~1MB upfront per destination.

More importantly, the buffer_limit controls the "buffer full" backpressure in the run loop (line 136: self.buffer.len() >= self.buffer_limit). When max_batch_size == 0, the buffer can grow to queue_capacity entries before backpressure kicks in — but the mpsc channel also has capacity queue_capacity. Combined, this means up to 20,000 transactions can be buffered per destination (10k in the channel + 10k in the buffer) before any backpressure is applied. Consider using a smaller cap for buffer_limit when max_batch_size is unlimited.

_ = self.cancel.cancelled() => return,
_ = time::sleep(backoff) => {}
}
time::sleep(backoff).await;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After removing the cancel token, the retry backoff sleep is no longer cancellation-aware. During shutdown, if the RPC endpoint is down the forwarder will retry up to max_retries (default 3) times with exponential backoff (100ms, 200ms, 400ms) before the task can check channel closure. In the worst case this adds ~700ms of latency per in-flight batch before the forwarder notices it should stop.

This is likely acceptable for the 30s shutdown timeout, but worth documenting as a known shutdown delay. An alternative is to use tokio::select! with the receiver's closed future to abort retries early when the consumer side has been dropped.

fn default() -> Self {
Self {
max_rps: 200,
max_batch_size: 500,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ForwarderConfig::default() sets max_batch_size: 500, but TxForwardingConfig::default() uses DEFAULT_MAX_BATCH_SIZE = 100 and passes it through forwarder_config(). The ForwarderConfig::default() value of 500 is never actually used in production because the CLI config always overrides it. This creates a confusing mismatch — the test defaults() below asserts 500, but production will always see 100. Consider aligning these or removing the dead default.

refcell and others added 2 commits July 31, 2026 14:12
`DestinationForwarder` was welded to the transaction pool in two places: its
queue carried `Arc<ValidPoolTransaction<T>>`, and `send_batch` hardcoded
`base_insertValidatedTransaction`. Everything else it does — batching, the
sliding-window rate limiter, retries with backoff, metrics, event emission,
drain-on-shutdown — only ever needed "a thing I can turn into a JSON-RPC call".

Introduce `ForwardRequest` to name exactly that: a per-request method, params
and optional tx hash. `DestinationForwarder<R>` is now generic over it, and the
`ValidPoolTransaction` -> `ValidatedTransaction<E>` conversion moves up into
`DestinationConsumer`, which is the component that legitimately holds one. That
also puts the 2718 encoding on the consumer's blocking thread instead of the
async runtime.

Because the method name is read per request rather than per batch, one
destination queue can carry several kinds of call, and a batch preserves
submission order between them — the property a producer needs when a removal
supersedes an earlier insert.

`spawn_requests` exposes the transport to producers that are not the pool: the
caller owns the queues and its own overflow policy, which matters for a producer
that enqueues under a lock and must drop rather than wait. Unlike `spawn`, an
unusable endpoint is a `ForwardingSetupError` rather than a logged skip, so a
caller never silently forwards to fewer destinations than it asked for.

`spawn` and `spawn_with_extensions` keep byte-identical signatures; the node
path is unchanged, and the integration test covering it still passes.

Co-Authored-By: Claude <noreply@anthropic.com>
executor: &TaskExecutor,
) -> Result<TxForwardingHandle, ForwardingSetupError> {
let forwarder_config = Arc::new(self.config.forwarder_config());
let queue_capacity = self.config.consumer_config().channel_capacity;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

queue_capacity is read from ConsumerConfig::default().channel_capacity (10,000), but in spawn_requests there is no consumer — the caller created the mpsc::Receiver with their own channel capacity. This value flows into DestinationForwarder::new where it sets buffer_limit (the internal buffer cap when max_batch_size == 0) and the Vec::with_capacity preallocation.

This means a caller who creates a channel with capacity 16 still gets a forwarder that eagerly allocates for 10,000 entries and won't apply buffer backpressure until 10,000 items have accumulated — a disconnect from the caller's intended backpressure semantics. Consider either accepting queue_capacity as a parameter of spawn_requests, or deriving buffer_limit from max_batch_size alone (the queue_capacity fallback for the unlimited case could use a smaller default).

`zepter format features` has been failing on this branch since the crate gained
a `[features]` section in the move commit; it wants inner spacing on feature
arrays. Formatting only — no feature is added, removed or repointed.

Co-Authored-By: Claude <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Review Summary

The refactoring is well-structured — moving consumer/forwarder ownership from base-execution-txpool into base-tx-forwarding is a clear improvement in crate responsibility. The per-destination pipeline design (independent dedup caches, bounded queues, isolated backpressure) directly addresses the "slow builder blocks all destinations" problem. The ForwardRequest trait is a good abstraction for extensibility. Shutdown lifecycle with ShutdownReport is thorough.

Existing inline comments (from prior review pass)

The five inline comments already posted cover the main findings:

  1. Consumer spin loop (consumer/task.rs): try_sendFull blocks the entire consumer for one destination, stalling the pool iterator on a stale snapshot. Breaking out and re-polling on the next iteration would be more consistent with the per-iteration snapshot model.

  2. Buffer preallocation + backpressure gap (forwarder/task.rs): When max_batch_size == 0, buffer_limit defaults to queue_capacity (10k), pre-allocating ~1MB and allowing 20k total buffered transactions (channel + buffer) before backpressure kicks in.

  3. Retry backoff not cancellation-aware (forwarder/task.rs): Up to ~700ms shutdown delay per in-flight batch when the RPC endpoint is down. Acceptable given the 30s timeout, but worth documenting.

  4. Default max_batch_size mismatch (forwarder/config.rs): ForwarderConfig::default() uses 500, but production always receives 100 from TxForwardingConfig. The test asserts 500, creating a confusing split.

  5. queue_capacity disconnect in spawn_requests (service.rs): Hardcoded 10k from ConsumerConfig::default() regardless of the caller's actual channel capacity, leading to mismatched backpressure semantics.

Additional observations (not blocking)

  • Cancellation safety: The tokio::select! in the forwarder run loop (line 129) is cancel-safe — both time::sleep and mpsc::Receiver::recv are cancel-safe per tokio guarantees.
  • expect("valid method name") (forwarder/task.rs:338): Safe because ForwardRequest::method() returns &'static str, and all implementations use hardcoded valid JSON-RPC method names.
  • Test coverage: Good coverage of edge cases — queue-full retries, cancellation, mixed-method batch ordering, shutdown timeout abort, and the integration test proving destination isolation under load.

@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Base Std historical fork tests

Fork Result Passed Failed Skipped base/base base-anvil base-std
Beryl pass 616 0 13 f178cb40 6d744e03 4658f1b7
Cobalt pass 703 0 14 f178cb40 ae7557c4 96e96870

View run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

execution Area: execution

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants