fix(encryption): Refactor EncryptionWrapper, fix zero-byte size, add S3 tests#57279
fix(encryption): Refactor EncryptionWrapper, fix zero-byte size, add S3 tests#57279cuppett wants to merge 11 commits intonextcloud:masterfrom
Conversation
0a01f11 to
fd39e51
Compare
f4f2d21 to
1a81edd
Compare
|
Failing test seems related. e.g. |
This is not true, the home storage is mounted at |
fcadd6f to
c0c3ae6
Compare
30903a9 to
4dc1def
Compare
There was a problem hiding this comment.
Pull request overview
This PR refines Nextcloud’s encryption behavior (wrapper application + home-mount handling), fixes incorrect size reporting for encrypted zero-byte files, and expands/isolates encryption test coverage (including new S3 primary-storage encryption tests).
Changes:
- Refactor
EncryptionWrapperflow and addHomeMountPoint+encryptHomeStoragehandling. - Fix encrypted size propagation by treating
unencrypted_size = 0as a valid value in cache/fileinfo paths. - Add S3 encryption test suites and broaden test cleanup to avoid encryption state leaking between tests.
Reviewed changes
Copilot reviewed 21 out of 21 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/lib/Traits/EncryptionTrait.php | Switch test encryption enable/restore to IAppConfig APIs. |
| tests/lib/TestCase.php | Add global teardown cleanup for encryption app-config state. |
| tests/lib/Files/ObjectStore/S3EncryptionTest.php | New integration test suite for encryption + S3 primary storage (sizes, reads, seeks, multipart). |
| tests/lib/Files/ObjectStore/S3EncryptionMigrationTest.php | New migration-oriented tests for mixed encrypted/unencrypted S3 content. |
| tests/lib/Encryption/EncryptionWrapperTest.php | Adjust wrapper test expectations/mocking for updated logic. |
| tests/Core/Command/Encryption/EnableTest.php | Update command test to use IAppConfig methods. |
| tests/Core/Command/Encryption/DisableTest.php | Update command test to use IAppConfig methods. |
| lib/private/Files/Stream/Encryption.php | Broaden $unencryptedSize property type to accept int/float/null. |
| lib/private/Files/Storage/Wrapper/Encryption.php | Tighten plausibility checks for unencryptedSize to trigger recalculation. |
| lib/private/Files/FileInfo.php | Return unencrypted_size even when it is 0 for encrypted entries. |
| lib/private/Files/Cache/Scanner.php | Adjust scanner update/unset logic for unencrypted_size. |
| lib/private/Files/Cache/CacheEntry.php | Return unencrypted_size even when it is 0 for encrypted entries. |
| lib/private/Encryption/EncryptionWrapper.php | Refactor early-return logic; add HomeMountPoint handling for encryptHomeStorage. |
| core/Command/Encryption/Enable.php | Use IAppConfig for encryption_enabled / default module reads/writes. |
| core/Command/Encryption/Disable.php | Use IAppConfig for encryption_enabled reads/writes. |
| build/psalm-baseline.xml | Update baseline entries (including new Imagick-related suppressions). |
| apps/files_sharing/tests/EncryptedSizePropagationTest.php | Add encryption trait teardown to prevent state leakage. |
| apps/encryption/tests/EncryptedStorageTest.php | Add encryption trait teardown to prevent state leakage. |
| apps/encryption/tests/Command/FixEncryptedVersionTest.php | Add encryption trait teardown to prevent state leakage. |
| apps/dav/tests/unit/Connector/Sabre/RequestTest/EncryptionUploadTest.php | Add encryption trait teardown to prevent state leakage. |
| apps/dav/tests/unit/Connector/Sabre/RequestTest/EncryptionMasterKeyUploadTest.php | Add encryption trait teardown to prevent state leakage. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
d2addff to
f24143c
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 21 changed files in this pull request and generated 12 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@copilot apply changes based on the comments in this thread |
dd6246f to
d87b141
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 15 changed files in this pull request and generated 4 comments.
Comments suppressed due to low confidence (1)
lib/private/Files/FileInfo.php:170
getSize(false)returns$this->rawSize, butrawSizeis initialized in the constructor only whenunencrypted_size !== 0. With this change, encrypted empty files will now returnunencrypted_size(0) forgetSize(true), whilegetSize(false)will still return the encryptedsize(e.g. header size). Consider initializingrawSizefromunencrypted_sizewhenever it is set (even when it is 0) to keep both code paths consistent for 0-byte encrypted files.
public function getSize($includeMounts = true) {
if ($includeMounts) {
$this->updateEntryFromSubMounts();
if ($this->isEncrypted() && isset($this->data['unencrypted_size'])) {
return $this->data['unencrypted_size'];
} else {
return isset($this->data['size']) ? 0 + $this->data['size'] : 0;
}
} else {
return $this->rawSize;
}
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| parent::setUp(); | ||
|
|
||
| // Get S3 config from system config | ||
| $this->s3Config = Server::get(IConfig::class)->getSystemValue('objectstore'); |
There was a problem hiding this comment.
If the primary object store is configured with arguments.multibucket = true, the effective bucket name is user-dependent (see PrimaryObjectStoreConfig::getBucketForUser() / homeobjectstore user config). Using the raw arguments['bucket'] here will point headObject at the wrong bucket. Consider either resolving the effective bucket for TEST_USER or skipping when multibucket is enabled.
| $this->s3Config = Server::get(IConfig::class)->getSystemValue('objectstore'); | |
| $this->s3Config = Server::get(IConfig::class)->getSystemValue('objectstore'); | |
| if (($this->s3Config['arguments']['multibucket'] ?? false) === true) { | |
| self::markTestSkipped('S3 encryption test does not support multibucket primary object storage because direct bucket inspection requires resolving the effective bucket for the test user.'); | |
| } |
…y object storage (S3) This simplifies the encryption wrapper to clarify the logic and setting checks. Fix: - Now checks encryptHomeStorage app setting for home mount point - When enabled (default), encryption wrapper is applied to primary storage - Maintains backward compatibility with existing behaviors Testing: - Comprehensive test suite with 23 tests covering 1KB to 110MB files - Validates size consistency across database, S3, and actual content - Tests multipart uploads, seeking, partial reads, and streaming - All tests passing on real AWS S3 with encryption enabled - Migration test suite (3 tests) for upgrade scenarios - Updates to EncryptionWrapperTest mock objects Verified: - Files are encrypted in S3 (AES-256-CTR with HMAC signatures) - Size tracking accurate (DB stores unencrypted size) - No corruption at any file size (including 16MB, 64MB, 100MB) - Multipart uploads work correctly with encryption - Storage overhead: ~1.2% for files > 1MB Migration Path: - Users with unencrypted files can run encryption:encrypt-all - Command safely handles mixed encrypted/unencrypted content - Skips already-encrypted files (checks database flag) This validates in tests long-standing GitHub issues where users reported encryption not working with S3 primary storage. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Stephen Cuppett <steve@cuppett.com>
This fixes a bug where zero-byte encrypted files were incorrectly reported as 8192 bytes (the encryption header size) instead of 0 bytes. Root Cause: - CacheEntry::getUnencryptedSize() and FileInfo::getSize() checked if unencrypted_size > 0 before using it - For zero-byte files, this condition fails (0 > 0 is false) - Falls back to encrypted size (8192 bytes) instead of 0 Impact: - Zero-byte encrypted files showed wrong size in UI and API - Database stored correct unencrypted_size=0, but getSize() returned 8192 - Affected quota calculations, file listings, and size comparisons Fix: - Remove the > 0 check from both methods - Now checks only isset() to distinguish between missing and zero values - Zero-byte files correctly report size as 0 Testing: - Added zero-byte files to all encryption test scenarios - testSizeConsistencyAcrossSources with 0 bytes - PASS - testEncryptedFileRoundTrip with 0 bytes - PASS - testEncryptedFileIntegrity with 0 bytes - PASS - All 26 tests passing including edge cases Files Modified: - lib/private/Files/Cache/CacheEntry.php - getUnencryptedSize() - lib/private/Files/FileInfo.php - getSize() - tests/lib/Files/ObjectStore/S3EncryptionTest.php - Added 0-byte test cases fix(encryption): Fix unencrypted size calculation for cached zero values When the zero-byte fix removed the `> 0` check from getUnencryptedSize(), it exposed a gap in verifyUnencryptedSize() that didn't recalculate the size when unencrypted_size=0 but the encrypted file has content. This caused Behat tests to fail because: 1. Newly encrypted files had unencrypted_size=0 in cache 2. getUnencryptedSize() returned 0 (instead of falling back to size) 3. verifyUnencryptedSize() didn't recalculate (conditions didn't match) 4. Encryption stream received 0 and returned empty content Added condition to trigger recalculation when cache reports 0 bytes but the physical encrypted file has content (size > 0). Fixes 21 Behat test failures: - files_features/encryption.feature (1 scenario) - files_features/transfer-ownership.feature (20 scenarios) fix(encryption): Fix type error for unencryptedSize property The Stream\Encryption::$unencryptedSize property was typed as ?int, but fixUnencryptedSize() returns int|float for large files. This caused TypeErrors when the verifyUnencryptedSize() fix triggered recalculation. Changed property type from ?int to int|float|null to match the return type of fixUnencryptedSize(). Fixes: - EncryptionMasterKeyUploadTest::testUploadOverWrite - EncryptionUploadTest::testUploadOverWrite 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Stephen Cuppett <steve@cuppett.com>
Fixes 375 PHPUnit errors in php8.2-s3-minio CI caused by encryption tests not cleaning up their state, polluting subsequent tests. Root Cause: - 5 tests using EncryptionTrait enable encryption but have NO tearDown - After tests complete, encryption_enabled remains 'yes' - EncryptionWrapper sees encryption enabled for ALL subsequent tests - Tests without encryption keys fail with "multikeyencryption failed" Tests Fixed: 1. apps/encryption/tests/EncryptedStorageTest.php 2. apps/encryption/tests/Command/FixEncryptedVersionTest.php 3. apps/files_sharing/tests/EncryptedSizePropagationTest.php 4. apps/dav/tests/unit/Connector/Sabre/RequestTest/EncryptionUploadTest.php 5. apps/dav/tests/unit/Connector/Sabre/RequestTest/EncryptionMasterKeyUploadTest.php Fix: Add tearDown() method to each test that calls tearDownEncryptionTrait(), which restores encryption_enabled to its previous value. Also Fixed: - tests/lib/Encryption/EncryptionWrapperTest.php: Mock isEnabled() for tests that expect wrapper to be applied Testing: - Local full suite run: 0 encryption errors (vs 375 in CI) - Encryption state properly restored after tests - ViewTest passes after encryption tests run Impact: - Prevents test state pollution in CI - Each test properly cleans up encryption configuration - No functional changes to production code fix(tests): Add global encryption state cleanup to prevent test pollution Fixes 255 PHPUnit errors in php8.2-s3-minio CI caused by encryption state pollution. Tests that enable encryption (encryption_enabled='yes') without proper cleanup now have automatic cleanup in base TestCase. Root Cause: - EncryptionWrapper.wrapStorage() now checks Manager::isEnabled() - This check is new (not in master) and makes wrapper sensitive to encryption_enabled app config value - Tests leaving encryption_enabled='yes' cause subsequent tests to fail - HomeMountPoints get encryption wrapper but no user keys exist - Results in MultiKeyEncryptException in 255+ tests Previous Fix Attempt (c0c3ae6): - Added tearDown to 5 specific test files - Reduced errors from 375 to 255 - But pollution source remained unknown This Fix: - Add encryption cleanup to TestCase.tearDown() base method - Runs after all trait tearDown methods - Automatically resets encryption state if enabled - Applies to ALL 6000+ tests extending TestCase - No need to track down individual pollution sources Impact: - Prevents all encryption state pollution - Minimal performance impact (only resets if enabled) - Safe (try-catch prevents breaking tests) - Comprehensive (applies to entire test suite) Testing: - Verified cleanup runs: manually set encryption_enabled='yes' - Ran ViewTest: failed (expected - no keys) - After test: encryption_enabled reset to 'no' (cleanup worked) Fixes: https://github.com/nextcloud/server/actions/runs/20576563150 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Stephen Cuppett <steve@cuppett.com>
This fixes a bug where encrypted files uploaded via stream (e.g., using `occ files:put` or `Folder::newFile($path, $stream)`) had their `unencrypted_size` incorrectly set to 0 in the database, even though the files were properly encrypted and the encryption wrapper provided the correct size. Root Cause: The Scanner class had overly restrictive logic (from commit 649bed5) that prevented `unencrypted_size` from being updated in two scenarios: 1. **For existing files with cached unencrypted_size=0:** - Scanner would get correct value from encryption wrapper - But would remove it because cached value was 0 - Result: Cache never updated from incorrect 0 to correct value 2. **For new file uploads:** - Encryption wrapper provided correct unencrypted_size in metadata - But Scanner unconditionally removed it for new files - Result: Cache entry created with unencrypted_size=0 Impact Before Fix: - Files uploaded via `occ files:put` had unencrypted_size=0 - `info:file` reported all files as "size: 0 B" - Required running `files:scan` to fix the values - The workaround in verifyUnencryptedSize() fixed it on access/scan, but didn't prevent the incorrect initial value Impact After Fix: - Files uploaded via any method have correct unencrypted_size immediately - No scan required to fix database values - Both zero-byte and non-zero encrypted files work correctly The Fix: 1. **For existing files (line 200-207):** - Only skip update if BOTH cached AND new values are 0 - Allows updating from incorrect cached 0 to correct non-zero value - Avoids unnecessary updates when both are legitimately 0 2. **For new files (line 223-228):** - Only remove unencrypted_size if file is not encrypted - Or if unencrypted_size is 0/not set (unencrypted file) - Preserves correct unencrypted_size for encrypted files Testing: - Verified 0B, 1MB, 100MB files upload with correct unencrypted_size - Database values correct immediately after upload (no scan needed) - Zero-byte files still report size as 0 (not 8192) - Local and S3 storage both work correctly Signed-off-by: Stephen Cuppett <steve@cuppett.com>
Fixes AppConfigTypeConflictException when setting encryption configuration values. The deprecated setAppValue()/getAppValue() methods default to MIXED type, which conflicts with existing STRING type values in the database. Changes: - core/Command/Encryption/Enable: Use setValueString/getValueString - core/Command/Encryption/Disable: Use setValueString/getValueString - tests/lib/Traits/EncryptionTrait: Use typed IAppConfig methods This resolves all 29 test failures in the PRIMARY-s3 PHPUnit test group where encryption setup was throwing type conflict exceptions. Co-authored-by: Claude Sonnet 4.5 (1M context) <noreply@anthropic.com> Signed-off-by: Stephen Cuppett <steve@cuppett.com>
Update unit tests for Enable and Disable commands to mock the new IAppConfig dependency that was added in the previous commit. Changes: - tests/Core/Command/Encryption/EnableTest: Add IAppConfig mock - tests/Core/Command/Encryption/DisableTest: Add IAppConfig mock - Update test expectations to use getValueString/setValueString Fixes 7 test failures in the NODB test suite. Co-authored-by: Claude Sonnet 4.5 (1M context) <noreply@anthropic.com> Signed-off-by: Stephen Cuppett <steve@cuppett.com>
… errors Add 19 pre-existing Psalm errors to the baseline that are not caused by this branch's encryption fixes: - 14 Imagick-related UndefinedClass/UndefinedConstant errors in apps/theming/lib/IconBuilder.php, apps/theming/lib/ImageManager.php, lib/private/Preview/Bitmap.php, lib/private/Preview/HEIC.php, and lib/private/Preview/SVG.php. These stem from the imagick.php stub at build/stubs/imagick.php missing ImagickPixel, ImagickDraw, ImagickException classes and several constants. - 1 DeprecatedMethod error in tests/lib/TestCase.php for SimpleContainer::offsetGet, introduced when that file was added to psalm.xml project files by upstream commit a430702. None of these errors originate from this branch's changes. Signed-off-by: Stephen Cuppett <steve@cuppett.com>
…lm baseline - Remove unrelated Imagick entries from psalm-baseline.xml that caused static-code-analysis CI failure; these are pre-existing stubs resolved independently of this PR's changes - Fix Scanner.php: preserve valid unencrypted_size === 0 for encrypted new files by dropping the erroneous === 0 check in the new-file branch - Align S3EncryptionTest/S3EncryptionMigrationTest setUp/tearDown with EncryptionTrait: remove duplicate IConfig state snapshot (taken after encryption was already enabled, which overwrote the trait's snapshot of the pre-test state), call tearDownEncryptionTrait() instead - Remove permanently-skipped testEncryptAllHandlesMixedContent and its unused createUnencryptedFileInS3 helper - Fix testDetectUnencryptedFilesQuery: replace LIKE on numeric storage column with proper two-step query using IN + PARAM_INT_ARRAY - Reduce peak memory in large-file tests: hash before unset(data), use incremental hash_init/hash_update/hash_final for streaming integrity; gate 64 MB / 100 MB data-provider cases behind RUN_HEAVY_S3_TESTS - Skip 110 MB multipart test behind same env gate; fix misleading "6MB file" comment (actual size is 110 MiB) - Drop unused IConfig constructor param from Enable/Disable commands and their tests; commands already use IAppConfig exclusively Signed-off-by: Stephen Cuppett <steve@cuppett.com>
…n writes Setup::setupSystem() is gated by a local 'keys-validated' cache and can no-op even on a fresh S3 bucket where the keys have never been written. This caused every write in S3EncryptionTest to throw MultiKeyEncryptException because getPublicMasterKey() returned an empty string. Call validateMasterKey() and validateShareKey() directly in setUp() — matching the pattern already used by apps/encryption/tests/EncryptedStorageTest.php — to guarantee system keys exist in the S3 keystore before any file operations. Signed-off-by: Stephen Cuppett <steve@cuppett.com>
Replace deprecated IConfig::getAppValue() with the typed IAppConfig::getValueString() for the encryptHomeStorage check, consistent with the rest of this PR's config migration. Signed-off-by: Stephen Cuppett <steve@cuppett.com>
- Guard $config['class'] with null coalescing in setUpBeforeClass() to avoid PHP notice when objectstore is configured without a 'class' key - Remove unused $rootStorage property and unwrapping loop in setUp() (was assigned but never referenced in any test method) - Chain previous exception in getObjectUrn() throw to preserve stack trace Signed-off-by: Stephen Cuppett <steve@cuppett.com>
6ed3047 to
ebc2843
Compare
Summary
This PR makes several improvements to Nextcloud's encryption infrastructure:
encryptHomeStoragesetting for home storage mountsChanges
Code Fixes
lib/private/Encryption/EncryptionWrapper.phpRefactored storage wrapper logic:
encryptHomeStoragesetting on HomeMountPoint mountslib/private/Files/Cache/CacheEntry.php&lib/private/Files/FileInfo.phpFixed zero-byte file size reporting:
> 0check that caused 0-byte encrypted files to report as 8192 bytes (encryption header size)Test Infrastructure
New:
tests/lib/Files/ObjectStore/S3EncryptionTest.php(493 lines)Comprehensive S3 encryption test suite covering:
New:
tests/lib/Files/ObjectStore/S3EncryptionMigrationTest.php(270 lines)Migration scenario tests:
encryption:encrypt-alltests/lib/TestCase.php+ 5 encryption test filesFixed test pollution:
MultiKeyEncryptExceptionfailures from state leakage between testsFiles Changed
lib/private/Encryption/EncryptionWrapper.phplib/private/Files/Cache/CacheEntry.phplib/private/Files/FileInfo.phptests/lib/TestCase.phptests/lib/Encryption/EncryptionWrapperTest.phptests/lib/Files/ObjectStore/S3EncryptionTest.phptests/lib/Files/ObjectStore/S3EncryptionMigrationTest.phpapps/Backward Compatibility
✅ Fully backward compatible
encryptHomeStoragesetting defaults to enabled ('1')Testing
Run the S3 encryption tests: