FLINK-37686 - Do not fail on NOT_OWNED deleted files

[francis-a]

What is the purpose of the change

This PR aims to fix the issue documented in https://issues.apache.org/jira/browse/FLINK-37686 where remote objects cleaned up by the JM cause checkpoint failures even when the object is marked as NOT_OWNED.

Brief change log

• Added a FileNotFoundException catch block to ForStFlinkFileSystem::listStatus
• In this catch block we verify the object is NOT_OWNED
• If true, we assume the JM removed this object and delete it using the FileMappingManager
• Tests have been added to cover this case as well as ensuring the exception still bubbles up if the object is not NOT_OWNED

Verifying this change

This change added tests and can be verified as follows:

• Added two new tests to ForStFlinkFileSystemTest, testListStatusDropsDeletedNotOwnedEntryAfterGiveUpOwnership and testListStatusStillFailsForDeletedDbOwnedFile
• These tests cover the NOT_OWNED and verify the removed object is also removed from the FileMappingManager, the other test verifies the baseline hasn't changed for other object types

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): (no)
• The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
• The serializers: (yes / no / don't know)
• The runtime per-record code paths (performance sensitive): (don't know)
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
• The S3 file system connector: (no)

Documentation

• Does this pull request introduce a new feature? (no)

---

Was generative AI tooling used to co-author this PR?

• Yes

Generated-by: [Claude Opus]

[flinkbot]

CI report:

• b85bade Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[AlexYinHan]

@francis-a Thanks for the PR. LGTM. The orphan files deleted by the JM are abandoned by ForSt. They should be removed and not cause failures during a full scan. @Zakelly Could you take a look?

[Zakelly]

LGTM, @francis-a thanks for the fix

[Zakelly]

@flinkbot run azure

[francis-a]

thanks @Zakelly, do you plan on handling merging? Is there anything else I should do?

[Zakelly]

thanks @Zakelly, do you plan on handling merging? Is there anything else I should do?

@francis-a yeah, I was about to merge but the CI failed. I think it's irrelevant but the CI bot is not responding to the re-run request. It would be nice if you could amend the commit and force push once, to re-trigger the CI.

[francis-a]

@Zakelly done

[francis-a]

It doesn't look like checks are running, I'll push a minor change