xet-core is xet client technology used by huggingface_hub for uploading and downloading files to the HF Hub using xet storage. It provides chunk-based deduplication, efficient storage and retrieval with local disk caching, and Python bindings, including the hf-xet package, plus a Git LFS compatible git-xet CLI tool.
Project status
- Actively maintained: the upstream was pushed on 2026-11-04, and the latest listed update is from 2026-10-30.
- Updates have been frequent, with multiple updates in October and earlier updates in June through August.
AI summary generated
Recent updates
v1.8.0
v1.8.0 adds experimental direct-to-S3 xorb uploads using CAS-issued grants, with fallback to the existing upload path, and fixes reconstruction requests that extend beyond a known file size. It also fixes publishing partially written files after write errors. The diff includes additional telemetry, logging, and retry behavior that is not described in the release notes.
Featuresv1.7.1
v1.7.1 fixes global dedup indexing behavior at the chunk-index size limit, prevents failed commit or download-group construction from poisoning later attempts, and applies the configured minimum spacing between global dedup queries. The supplied diff also contains source-level API changes to shard uploads and shard-index management that are not called out in the release notes.
v1.8.0-dev0
v1.8.0-dev0 adds support for returning the server stored shard hash after shard uploads, and includes fixes related to session poisoning when commit or group construction fails. It also introduces types to support XORB upload grants, along with related client-side logic and dependency bumps.
BreakingFeaturesv1.7.0
hf-xet v1.7.0 adds client-side telemetry reporting for uploads and downloads and introduces dynamic download buffers that adapt to available RAM. The release also includes multiple runtime and simulation fixes, including releasing the GIL while waiting for download chunks.
Featuresv1.7.0-dev1
This release note entry provides only a link to the full changelog and does not enumerate any specific functional changes. The provided code diff shows only internal crate version and lockfile metadata updates to move from 1.6.0 to 1.7.0-dev1 across the workspace and related WASM crates.
v1.7.0-dev0
v1.7.0-dev0 introduces a new client telemetry pipeline, including a dedicated /v1/telemetry ingestion API, transfer-level aggregation, and additional Python stream-group semantics to ensure telemetry is emitted reliably. The release also updates simulation support for S3-style object tag sets and includes several fixes and dependency/security bumps (TLS-related).
SecurityFeaturesv1.6.1-a0
Release v1.6.1-a0 has no substantive release-note content beyond a changelog link. The actual diff provided shows only Cargo.lock changes, indicating dependency version bumps rather than direct source code changes.
v1.6.0
hf-xet v1.6.0 adds streaming progress visibility for shard upload finalization by introducing an NDJSON progress stream on POST /v2/shards. It also introduces richer shard-upload progress event types and exposes them through Python progress reporting, along with some upload efficiency and build-related improvements.
BreakingFeaturesv1.5.2
Release v1.5.2 is described as a fix for a bug in HTTP request retry configuration that could cause hangs when a request was retried more than twice. However, the actual diff for this release includes multiple unrelated changes across Python bindings, Rust dependencies, wasm build tooling, and CI behavior that are not mentioned in the release notes.
Featuresv1.5.1
v1.5.1 improves hf-xet Python progress reporting by ensuring a final progress snapshot is emitted even if uploads/downloads complete between poll intervals, preventing UIs from getting stuck below 100%. It also addresses shutdown-time panics by changing how the background progress thread interacts with the Python interpreter during teardown.
Features1.5.0
hf-xet v1.5.0 introduces a session based, object oriented Python API centered around XetSession and its child objects (upload commit, download group, streaming download). The legacy module level functions (upload_files, download_files, hash_files) are kept under hf_xet/src/legacy and remain importable, but they now emit DeprecationWarning.
Featuresv1.4.3
This release introduces a hotfix for file corruption by ensuring the local output file is truncated appropriately during full-file downloads. A new boolean flag controls whether the output file is truncated to the reconstructed length.
v1.4.2
Despite the extensive release note content, the actual code diff between v1.4.1 and v1.4.2 shows only a version bump. No functional logic changes are present in the provided diff.
v1.4.1
hf-xet v1.4.1 focuses on upload and download performance improvements, plus fixes for stalls and incorrect error propagation during large file transfers. The release notes also mention an upload optimization around SHA-256 computation and new session/task APIs.
BreakingSecurityFeaturesv1.3.2
v1.3.2 is a download performance and stability release focused on faster multi-file downloads and reducing progress callback overhead. It also includes a wasm dependency bump and changes intended to address intermittent Windows test failures.
v1.3.1
This release adds Windows 11 ARM64 build support to the CI and Windows installer build workflow. It also bumps the hf-xet crate version from 1.3.0 to 1.3.1.
Featuresv1.3.0
v1.3.0 introduces adaptive concurrency control for uploads and downloads and improves progress reporting. The implementation refactors progress reporting to use byte-based callbacks and replaces the prior upload tracking mechanism, along with some HTTP client header handling changes.
Featuresgit-xet-v0.2.1
git-xet v0.2.1 focuses on installation workflow improvements for Homebrew, a macOS CI dependency tweak (removing openssl@3), and enabling adaptive concurrency. However, the code diff shows a much larger internal refactor inside cas_client, including changes to the public client interface and HTTP/retry plumbing that are not mentioned in the release notes.
BreakingFeatures