Rewrite DataTree.to_netcdf and support netCDF4 in-memory #10624
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I am thinking I might try to split this into a separate PR, because it's unrelated to the netCDF4 in-memory changes. |
This PR includes a handful of significant changes: 1. It refactors the internal structure of `DataTree.to_netcdf()` and `DataTree.to_zarr()` to use lower level interfaces, rather than calling `Dataset` methods. This allows for properly supporting `compute=False` (and likely various other improvements). 2. Reading and writing in-memory data with netCDF4-python is now supported, including DataTree. 3. The `engine` argument in `DataTree.to_netcdf()` is now set consistently with `Dataset.to_netcdf()`, preferring `netcdf4` to `h5netcdf`. 3. Calling `Dataset.to_netcdf()` without a target now always returns a `memoryview` object, *including* in the case where `engine='scipy'` is used (which currently returns `bytes`). This is a breaking change, rather than merely issuing a warning as is done in pydata#10571. I believe it probably makes sense to do as a this breaking change because (1) it offers significant performance benefits, (2) the default behavior without specifying an engine will already change (because `netcdf4` is preferred to the `scipy` backend) and (3) restoring previous behavior is easy (by wrapping the memoryview with `bytes()`). mypy
2 tasks
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR includes a handful of significant changes:
DataTree.to_netcdf()andDataTree.to_zarr()to use lower level interfaces, rather than callingDatasetmethods. This allows for properly supportingcompute=False(and likely various other improvements).load_datatreefunction, for consistentcy withload_datasetandload_dataarray.engineargument inDataTree.to_netcdf()is now set consistently withDataset.to_netcdf(), preferringnetcdf4toh5netcdf.Dataset.to_netcdf()without a target now always returns amemoryviewobject, including in the case whereengine='scipy'is used (which currently returnsbytes). This is a breaking change, rather than merely issuing a warning as is done in Support for DataTree.to_netcdf to write to a file-like object or bytes #10571. I believe it probably makes sense to do as a this breaking change because (1) it offers significant performance benefits, (2) the default behavior without specifying an engine will already change (becausenetcdf4is preferred to thescipybackend) and (3) restoring previous behavior is easy (by wrapping the memoryview withbytes()).whats-new.rst