IncrementalLoad pull handler work queue manager to import and delete items in parallel in some situations#973
Open
raymond-rebbeck wants to merge 4 commits into
Conversation
…en importing items to improve performance Only used when decomposed productions are not in use, these will very likely require further locking being implemented to ensure that productions are created and updated safely.
… a work queue manager These are not used anywhere else outside of this class and don't do anything that requires them to be methods as opposed to class methods.
…when deleting items to improve performance
raymond-rebbeck
requested review from
isc-dchui,
isc-pbarton and
isc-tleavitt
as code owners
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Use a work queue manager in the Incremental Load pull event handler for both importing and deleting files. Similar to what was done in #970
Results in a pleasing performance improvement to anything that uses the pull event handler such as revert, switching branches and this also now covers Import All when
compileOnImportis enabled - something that was not covered in #970The work queue manager is not used when
decomposeProductionsis enabled as I do not use this feature, have limited experience with it and I suspect that additional locking or other more complicated changes would be required to allow decomposed production items to safely create/update/delete productions in parallel.Testing
Tested locally with pulls of various sizes (including 1000+ files) and other actions that use the pull event handler such as: revert, switching branches and Import All with
compileOnImportenabled.Checklist
mainbranch rebased or merged.git-webui/srchave matching changes ingit-webui/release)