All Tools
T
DataFreeOpen Source
TENSORSTORE
High-performance library for large multi-dimensional array storage and I/O
Apache-2.0
ABOUT
Machine learning and scientific computing workloads increasingly produce massive multi-dimensional arrays that exceed available memory and require efficient persistent storage, but standard tools like NumPy and HDF5 lack support for asynchronous streaming, cloud-native storage backends, and fine-grained sub-array access. TensorStore solves this by providing a high-performance, scalable array storage library with chunked compression, asynchronous I/O, virtual views, and support for multiple storage backends including cloud object stores.
INSTALL
pip install tensorstoreINTEGRATION GUIDE
1. Store and access large-scale neural network weight checkpoints with chunked compression for fast partial loading
2. Read sub-regions of massive 3D/4D scientific datasets without loading the full array into memory
3. Stream training data from cloud object stores with asynchronous I/O for GPU-accelerated training pipelines
4. Manage multi-petabyte geospatial or biomedical array datasets with virtual views for efficient access
5. Share and version large array datasets across teams using cloud-native storage backends
TAGS
array-storagenumerical-computingtensorpythonhpcio