refactor(execution): Move Forwarding Into Dedicated Crate - #4232
Conversation
Amp-Thread-ID: https://ampcode.com/threads/T-019fb92a-6c1e-76c9-af4b-418e719339be Co-authored-by: Amp <amp@ampcode.com>
🟡 Heimdall Review Status
|
| let mut pending = tx; | ||
| loop { | ||
| match self.sender.try_send(pending) { | ||
| Ok(()) => { | ||
| self.recently_sent.mark_sent(hash); | ||
| txs_sent += 1; | ||
| self.emit_builder_consumed_event(hash, iterator_index); | ||
| break; | ||
| } | ||
| Err(mpsc::error::TrySendError::Full(tx)) => { | ||
| if self.cancel.is_cancelled() { | ||
| return; | ||
| } | ||
| pending = tx; | ||
| std::thread::sleep(self.config.poll_interval); |
There was a problem hiding this comment.
The try_send → Full spin loop blocks the entire consumer for this destination, including the cancellation check at line 66 (which only runs once per transaction, before entering this loop). If the mpsc channel is persistently full (e.g. the forwarder is rate-limited), this loop burns CPU on thread::sleep wakeups, and the outer for tx in best_txs iterator is stalled — meaning the pool snapshot grows stale while we wait for a single slot.
A simpler design: on Full, break out of the inner for tx loop entirely and fall through to the outer while !self.cancel.is_cancelled() loop, which already sleeps on poll_interval when txs_sent == 0. The transaction would be retried from a fresh best_transactions() snapshot on the next iteration. This avoids holding a stale iterator snapshot while blocked and is more consistent with the "per-iteration snapshot" model documented in the doc comment.
| let buffer_limit = | ||
| if config.max_batch_size == 0 { queue_capacity } else { config.max_batch_size }; | ||
| let buffer = Vec::with_capacity(buffer_limit); |
There was a problem hiding this comment.
When max_batch_size == 0 (unlimited), buffer_limit is set to queue_capacity (default 10,000), and Vec::with_capacity(buffer_limit) eagerly allocates space for 10,000 entries. Each BufferedTransaction is roughly ~100+ bytes so this is ~1MB upfront per destination.
More importantly, the buffer_limit controls the "buffer full" backpressure in the run loop (line 136: self.buffer.len() >= self.buffer_limit). When max_batch_size == 0, the buffer can grow to queue_capacity entries before backpressure kicks in — but the mpsc channel also has capacity queue_capacity. Combined, this means up to 20,000 transactions can be buffered per destination (10k in the channel + 10k in the buffer) before any backpressure is applied. Consider using a smaller cap for buffer_limit when max_batch_size is unlimited.
| _ = self.cancel.cancelled() => return, | ||
| _ = time::sleep(backoff) => {} | ||
| } | ||
| time::sleep(backoff).await; |
There was a problem hiding this comment.
After removing the cancel token, the retry backoff sleep is no longer cancellation-aware. During shutdown, if the RPC endpoint is down the forwarder will retry up to max_retries (default 3) times with exponential backoff (100ms, 200ms, 400ms) before the task can check channel closure. In the worst case this adds ~700ms of latency per in-flight batch before the forwarder notices it should stop.
This is likely acceptable for the 30s shutdown timeout, but worth documenting as a known shutdown delay. An alternative is to use tokio::select! with the receiver's closed future to abort retries early when the consumer side has been dropped.
| fn default() -> Self { | ||
| Self { | ||
| max_rps: 200, | ||
| max_batch_size: 500, |
There was a problem hiding this comment.
ForwarderConfig::default() sets max_batch_size: 500, but TxForwardingConfig::default() uses DEFAULT_MAX_BATCH_SIZE = 100 and passes it through forwarder_config(). The ForwarderConfig::default() value of 500 is never actually used in production because the CLI config always overrides it. This creates a confusing mismatch — the test defaults() below asserts 500, but production will always see 100. Consider aligning these or removing the dead default.
Amp-Thread-ID: https://ampcode.com/threads/T-019fb92a-6c1e-76c9-af4b-418e719339be Co-authored-by: Amp <amp@ampcode.com>
`DestinationForwarder` was welded to the transaction pool in two places: its queue carried `Arc<ValidPoolTransaction<T>>`, and `send_batch` hardcoded `base_insertValidatedTransaction`. Everything else it does — batching, the sliding-window rate limiter, retries with backoff, metrics, event emission, drain-on-shutdown — only ever needed "a thing I can turn into a JSON-RPC call". Introduce `ForwardRequest` to name exactly that: a per-request method, params and optional tx hash. `DestinationForwarder<R>` is now generic over it, and the `ValidPoolTransaction` -> `ValidatedTransaction<E>` conversion moves up into `DestinationConsumer`, which is the component that legitimately holds one. That also puts the 2718 encoding on the consumer's blocking thread instead of the async runtime. Because the method name is read per request rather than per batch, one destination queue can carry several kinds of call, and a batch preserves submission order between them — the property a producer needs when a removal supersedes an earlier insert. `spawn_requests` exposes the transport to producers that are not the pool: the caller owns the queues and its own overflow policy, which matters for a producer that enqueues under a lock and must drop rather than wait. Unlike `spawn`, an unusable endpoint is a `ForwardingSetupError` rather than a logged skip, so a caller never silently forwards to fewer destinations than it asked for. `spawn` and `spawn_with_extensions` keep byte-identical signatures; the node path is unchanged, and the integration test covering it still passes. Co-Authored-By: Claude <noreply@anthropic.com>
| executor: &TaskExecutor, | ||
| ) -> Result<TxForwardingHandle, ForwardingSetupError> { | ||
| let forwarder_config = Arc::new(self.config.forwarder_config()); | ||
| let queue_capacity = self.config.consumer_config().channel_capacity; |
There was a problem hiding this comment.
queue_capacity is read from ConsumerConfig::default().channel_capacity (10,000), but in spawn_requests there is no consumer — the caller created the mpsc::Receiver with their own channel capacity. This value flows into DestinationForwarder::new where it sets buffer_limit (the internal buffer cap when max_batch_size == 0) and the Vec::with_capacity preallocation.
This means a caller who creates a channel with capacity 16 still gets a forwarder that eagerly allocates for 10,000 entries and won't apply buffer backpressure until 10,000 items have accumulated — a disconnect from the caller's intended backpressure semantics. Consider either accepting queue_capacity as a parameter of spawn_requests, or deriving buffer_limit from max_batch_size alone (the queue_capacity fallback for the unlimited case could use a smaller default).
`zepter format features` has been failing on this branch since the crate gained a `[features]` section in the move commit; it wants inner spacing on feature arrays. Formatting only — no feature is added, removed or repointed. Co-Authored-By: Claude <noreply@anthropic.com>
Review SummaryThe refactoring is well-structured — moving consumer/forwarder ownership from Existing inline comments (from prior review pass)The five inline comments already posted cover the main findings:
Additional observations (not blocking)
|
Summary
This moves transaction forwarding consumers, forwarders, configuration, metrics, and lifecycle ownership from txpool into base-tx-forwarding. It replaces the separate spawned consumer and forwarder APIs with a unified service and shutdown handle. Each destination now has independent deduplication and bounded delivery so a slow builder cannot cause transactions to be lost through shared broadcast lag.