New Features

• Adds race_mirai(), which accepts a list of mirai and waits efficiently for the next mirai amongst them to resolve (@t-kalinowski, #448).
• New synchronous mode: daemons(sync = TRUE) causes mirai to run synchronously within the current process.
  This facilitates testing and debugging, e.g. via interactive browser() instances (#439, @kentqin-cve #442).

Updates

• everywhere() adds argument .min to specify a minimum number of daemons on which to evaluate the expression (when using dispatcher).
  This creates a synchronization point and can be useful when launching remote daemons to ensure that the expression has run on all daemons to connect (@louisaslett, #330).
• OpenTelemetry span names have been updated.
  Spans for long-running daemons are now split into short spans when they are created and when they end - refer to the updated vignette for more details (#464, #471).
• Removes the following developer features:
  • nextget("pid") is no longer a supported option.
  • Argument id is removed at daemon(). This means that status() no longer reports daemon connection or disconnection events.
• Removes deprecated argument tls at daemon(), launch_local() and launch_remote().
• Removes deprecated dispatcher argument 'none' at daemons(), deprecated in mirai v2.1.0.
• Fixes a phenomenon where stop_mirai() or mirai_map()[.stop] could on occasion cause (dispatcher) daemons to be interrupted and exit on subsequent runs (#459).
• Non-dispatcher daemons now synchronize upon timeout or task-out, ensuring that they exit safely only after all data has been sent (#458).
• Requires nanonext >= 1.7.1.

Behavioural Changes

• Behavioural changes for daemons():
  • Returns invisibly logical TRUE when creating daemons and FALSE when resetting, for simplicity and consistency (#384).
  • Creating new daemons resets any existing daemons for the compute profile rather than error.
    This means that an explicit daemons(0) is no longer required before applying new settings (thanks @eliocamp, #383).
  • Calling without supplying any arguments now errors rather than return the value of status().

New Features

• Complete observability of mirai requests by emitting OpenTelemetry traces when tracing is enabled by the otelsdk package (#394).
• Adds info() as an alternative to status() for retrieving more succinct information statistics, more convenient for programmatic use (thanks @wlandau, #410).
• Adds with_daemons() and local_daemons() helper functions for using a particular compute profile.
  These work with daemons that are already set up unlike the existing with.miraiDaemons() method, which creates a new scope and tears it down when finished (#360).
• A mirai now has an attribute id, which is a monotonically increasing integer identifier unique to each session.

Updates

• stop_mirai() is more efficient and responsive, especially for 'mirai_map' objects (#417).
• miraiError enhancements:
  • The original condition classes are preserved as $condition.class (thanks @sebffischer, #400).
  • The print method includes the customary additional line break (thanks @sebffischer, #399).
• Fixes daemons(n) failing to launch local daemons if mirai was installed in a custom user library set by an explicit .libPaths() call in '.Rprofile' (thanks @erydit and @dpastoor, #390).
• Improved behaviour for serial_config() custom serialization.
  If the serialization hook function errors or otherwise fails to return a raw vector, this will error out rather than be silently ignored (thanks @dipterix, #378).
• as.promise() method for mirai made robust for high-throughput scenarios (#377).
• mirai_map() now supports Arrow Tables and Polars DataFrames (#366).
• require_daemons() arguments are swapped so that .compute comes before call for ease of use.
  Previous usage will work for the time being, although is deprecated and will be defunct in a future version.
• Enhancements to everywhere():
  • Consecutive everywhere() calls are permissible again when using dispatcher (behaviour update in v2.4.1) (#354).
  • No longer has any effect on the RNG stream when using a reproducible seed value at daemons() (#356).
• A mirai() evaluated on an ephemeral daemon returns invisibly, consistent with other cases (#351).
• daemon() gains a tlscert argument for custom TLS certificates.
  The change in argument name lets this be passed when making a daemons() call (#344).
• The tls argument at daemon(), launch_local() and launch_remote() is deprecated.
• Requires nanonext >= 1.7.0.

New Features

• Reproducible parallel RNG by setting the seed argument to daemons():
  • The default NULL uses L'Ecuyer-CMRG RNG streams advanced per daemon, the same as base R's parallel package, which produces statistically-sound yet generally non-reproducible results.
  • Setting an integer seed now initializes a L'Ecuyer-CMRG RNG stream for the compute profile, which is advanced for each mirai evaluation, which does provide reproducible results.

Updates

• everywhere() has been updated for robustness and ease of use:
  • Returns a mirai_map object for easier handling (rather than just a list of mirai).
  • When using dispatcher, no longer has the potential to fail if sending large data (#326).
• dispatcher() function signature simplified with rs, tls and pass arguments removed (no user-facing impact).
• Fixes a bug where using non-dispatcher daemons, an unresolvedValue could be returned as the fulfilled value of a promise in extremely rare cases (thanks @James-G-Hill and @olivier7121, #243 and #317).
• Fixes a regression in mirai 2.4.0 where the L'Ecuyer-CMRG seed was not being passed correctly for remote daemons (#333).
• Requires nanonext >= 1.6.2.

Behavioural Changes

• An ephemeral daemon started by mirai() without setting daemons now exits as soon as the parent process does rather than finish the task.
• Change in daemon() defaults:
  • Argument autoexit default of TRUE now ensures daemons are terminated along with the parent process.
    Set to NA to retain the previous behaviour of having them automatically exit after completing any in-progress tasks.
  • Argument dispatcher now defaults to TRUE.
• Calling daemons() to create local daemons now errors if performed within a mirai_map() call.
  This guards against excessive spawning of local processes on a single machine.

New Features

• Adds cluster_config() to launch remote daemons via HPC resource managers for Slurm, SGE, Torque, PBS and LSF clusters.
• Adds require_daemons() as a developer function that prompts the user to set daemons if not already set, with a clickable function link if the cli package is available.

Updates

• Simplifies launches when using dispatcher - launch_remote() commands are now the same irrespective of the number of launches.
  This is as daemons now retrieve the next RNG stream from dispatcher rather than the rs argument to daemon().
• Deprecated call_mirai_() is now removed.
• Requires nanonext >= 1.6.1.

Behavioural Changes

• mirai() argument .timeout is upgraded to automatically cancel ongoing mirai upon timeout when using dispatcher (thanks @be-marc, @sebffischer #251).
• serial_config() now accepts vector arguments to register multiple custom serialization configurations. Argument vec is dropped as internal optimizations mean this option no longer needs to be set.

New Features

• host_url() is upgraded to return all local IP addresses (named by network interface), which provides a more comprehensive solution than just using a hostname.
• Adds register_serial() to register serialization configurations for all daemons() calls (may be used by package authors as a convenience).
• Adds on_daemon() which returns a logical value, whether or not evaluation is taking place within a mirai call on a daemon.
• Adds daemons_set() which returns a logical value, whether or not daemons are set for a given compute profile.
• daemons() now supports initial synchronization exceeding 10s (between host/dispatcher/daemons). This is particularly relevant for HPC setups (thanks @sebffischer, #275).

Updates

• For all functions that use .compute, this argument has a new default of NULL, which continues to use the default profile (and hence should not result in any change in behaviour).
• Fixes stop_mirai() failing to interrupt in certain cases on non-Windows platforms, and more robust interruption if tools::SIGINT is supplied or passed through to the autoexit argument of daemon() (thanks @LennardLux, #240).
• daemons() dispatcher argument 'process', deprecated in mirai v2.1.0, is removed.
• Requires nanonext >= 1.6.0.
• Package is re-licensed under the MIT license.

Behavioural Changes

• Simplified SSH tunnelling for distributed computing:
  • ssh_config() argument 'port' is removed, with the tunnel port now inferred at the time of launch, and no longer set by the configuration.
  • local_url() adds logical argument 'tcp' for easily constructing an automatic local TCP URL when setting daemons() for SSH tunnelling.

New Features

• Adds as.promise() method for 'mirai_map' objects. This will resolve upon completion of the entire map operation.

Updates

• mirai (in R >= 4.5) is now one of the official base R parallel cluster types.
  • register_cluster() is removed as no longer required.
  • Directly use parallel::makeCluster(type = "MIRAI") to create a 'miraiCluster'.
• call_mirai() is now user-interruptible, consistent with all other functions in the package.
  • call_mirai_() is hence redundant and now deprecated.
• with() method for daemons() now propagates ".compute" so that this does not need to be specified in functions such as mirai() within the with() clause.
• mirai() arguments ... and .args now accept environments containing variables beginning with a dot . (#207).
• 'miraiError' stack traces no longer sometimes contain an additional (internal) call (#216).
• 'miraiError' condition $call objects are now stripped of 'srcref' attributes (thanks @lionel-, #218).
• A mirai promise now rejects in exactly the same way whether or not the mirai was already resolved at time of creation.
  This avoids Shiny deep stack trace errors when the mirai had already resolved (#229).
• daemons() calls that error due to the remote launcher no longer leave the compute profile set up (#237).

Behavioural Changes

• daemons() now requires an explicit reset before providing revised settings for a compute profile, and will error otherwise.
• mirai_map() now errors if daemons have not yet been set (rather than warn and launch one local daemon).
• Removal of mirai v1 compatibility features:
  • saisei() is now removed as no longer required.
  • daemons() dispatcher argument "thread" is removed.
  • daemons() dispatcher arguments "process" and "none" are formally deprecated and will be removed in a future version.

Updates

• 'miraiError' evaluation errors now return the call stack at $stack.trace as a list of calls (with srcrefs removed) without deparsing to character strings.
• mirai_map() improvements:
  • Multiple map on a dataframe or matrix now correctly preserves the row names of the input as the names of the output.
  • Fixes language objects being evaluated before the map function is applied (#194).
  • Fixes classes of objects in a dataframe being dropped during a multiple map (#196).
  • Better cli errors when collecting a 'mirai_map'.
• Fixes daemons(NULL) not causing all daemons started with autoexit = FALSE to quit, regression introduced in mirai v2.0.0.
• Requires nanonext >= 1.5.0.

Updates

• mirai_map() collection option improvements:
  • The cli package is used, if installed, for richer progress bars and error messages.
  • [.progress_cli] is no longer a separate option.
  • [.stop] now reports the index number that errored.
• collect_mirai() replaces '...' with an 'options' argument, to which collection options should be supplied as a character vector. This avoids non-standard evaluation in this function.
• daemon() now returns an integer exit code to indicate the reason for termination.
• Adds nextcode() to provide a human-readable translation of the exit codes returned by daemon().
• everywhere() now returns a list of at least one mirai regardless of the number of actual connections.

New Architecture

• Distributed computing now uses a single socket and URL at which all daemons connect (with or without dispatcher).
  • Allows a more efficient tcp:// or tls+tcp:// connection in all cases.
  • The number of connected daemons may be upscaled or downscaled at any time without limit.

New Features

• daemons(dispatcher = TRUE) provides a new and more efficient architecture for dispatcher. This argument reverts to a logical value, although 'process' is still accepted and retains the previous behaviour of the v1 dispatcher.
• daemons() gains argument 'serial' to register serialization configurations when using dispatcher. These automatically apply to all daemons that connect.
• stop_mirai() is now able to cancel remote mirai tasks (when using dispatcher), returning a logical value indicating whether cancellation was successful.
• A 'miraiError' now preserves the original condition object. This means that rlang::abort() custom metadata may now be accessed using $ on the 'miraiError' (thanks @James-G-Hill #173).

Updates

• status() using the new dispatcher is updated to provide more concise and insightful information.
• everywhere() updates:
  • Enhanced to return a list of mirai, which may be waited for and inspected (thanks @dgkf #164).
  • Drops argument '.serial' as serialization configurations are now registered via an argument at daemons().
• daemon() updates:
  • Gains the new argument 'dispatcher', which should be set to TRUE when connecting to dispatcher and FALSE when connecting directly to host.
  • Gains argument 'id' which accepts an integer value that allows status() to track connection and disconnection events.
  • '...' has been moved up to prevent partial matching on any of the optional arguments.
  • 'cleanup' argument simplified to a TRUE/FALSE value.
  • 'timerstart' argument removed.
• launch_local() and launch_remote() updates:
  • Enhanced to now launch daemons with the originally-supplied arguments by default.
  • Simplified to take the argument 'n' instead of 'url' for how many daemons to launch.
  • launch_local() now returns the number of daemons launched rather than invisible NULL.
• collect_mirai() is now interruptible and takes a '...' argument accepting the collection options provided to the 'mirai_map' [] method, such as .flat etc.
• ssh_config() simplified to take the argument 'port' instead of 'host'. For SSH tunnelling, this is the port that will be used, and the hostname is now required to be '127.0.0.1' (no longer accepting 'localhost').
• host_url() argument 'ws' is removed as a TCP URL is now always recommended (although websocket URLs are still supported).
• saisei() is defunct as no longer required, but still available for use with the old v1 dispatcher.
• daemons(dispatcher = "thread") (experimental threaded dispatcher) has been retired - as this was based on the old dispatcher architecture and future development will focus on the current design. Specifying 'dispatcher = thread' is defunct, but will point to 'dispatcher = process' for the time being.
• Requires nanonext >= 1.4.0.

Updates

• Cleanup of packages only detaches them from the search path and does not attempt to unload them, as it is not always safe to do so. Fixes daemon crashes using packages such as data.table (thanks @D3SL, #166).
• serialization() deprecated in mirai 1.2.0 is now removed.