[larrymou9]

Do not merge

This is a draft opened only to run pulpcore's CI and tests. Per team policy, performance fixes for the hosted Pulp service ship as patches in pulp-service on top of the released pulpcore. The maintained fix is patch 0079 plus a migration in pulp-service. This PR exists to get CI signal on the same change.

Ticket: PULP-2346 (HEAD requests to the lightwell Java repositories time out at 30 s).

Problem

For a file path, the content handler asks the database three questions, each shaped as "find rows for this path among the repository version's items":

	Question	Where
Q1	is there an index.html?	_match_and_stream
Q2	does anything exist under the folder x.jar/?	list_directory_blocking()
Q3	fetch the file	_match_and_stream

All three use content__in=<version content>. For a repository with ~1.6M items, Postgres walks every item before answering "no". Measured on prod (read-only EXPLAIN ANALYZE, java/validated, 1,592,450 items, path that does not exist): Q1 13.5 s, Q2 14.0 s, Q3 13.0 s. Together that is over Akamai's 30 s limit.

What this PR changes (Q2 only)

• list_directory_blocking() filters on relative_path first and checks membership per matching row with a correlated EXISTS.
• New text_pattern_ops index on ContentArtifact.relative_path (migration 0162, AddIndexConcurrently). The existing relative_path btree uses the database collation and cannot serve LIKE 'prefix%'.
• Changelog corrected. An earlier version claimed constant time, which is wrong without the index.

Without the index the new query is worse on prod (parallel seq scan of core_contentartifact, ~16.8 s against ~14 s today). The query change and the index only work together.

Results (test database, repositories sized like the ticket's)

What was measured: the time of one database lookup, the folder question (Q2), as reported by EXPLAIN ANALYZE. Lower is better. Three versions of the code are compared:

• Before: the code in released pulpcore today.
• Query change only: this PR's new query, without the index.
• Query change + index (this PR): the new query and the new index together.

Tested on a test database with repositories sized like the two in the ticket. Units are shown in each cell (s = seconds, ms = milliseconds).

java/validated (1.59M items)

Folder being requested	Before	Query change only	Query change + index (this PR)
Does not exist	17.4 s	7.0 s	0.04–0.07 ms
Exists, specific folder	17.5 s	7.0 s	0.06 ms
Exists, very large folder (org/codehaus/)	17.4 s	7.1 s	7.1 s (index not used)

java/remediated (108K items)

Folder being requested	Before	Query change only	Query change + index (this PR)
Does not exist	0.89 s	0.58 s	0.04 ms
Exists, specific folder	0.89 s	0.60 s	0.06–0.10 ms
Exists, very large folder	0.89 s	0.58 s	0.58 s (index not used)

Small test repository (2K items)

Folder being requested	Before	Query change only	Query change + index (this PR)
Does not exist	13 ms	24.5 ms	0.05 ms
Exists	13 ms	24.5 ms	17.5 ms (slower than before)

Index build: 46 s, 1.79 GB on 28M rows.

Not fixed here

• Q1 and Q3 keep the old shape and still cost ~13 s each on prod's largest repository. A missing-file HEAD on java/validated stays near the limit. Follow-up PR.
• Broad prefixes do not use the index.
• Small repositories with a folder that exists are ~4.5 ms slower (17.5 ms against 13 ms).

Caveats

• The test database is smaller than prod (artifact table 3.5 GB against 22 GB, smaller instance). Its planner also takes a different route for Q1 and Q3 because relative_path statistics differ. The numbers show the mechanism, not prod timings.
• Only the latest version of each test repository has content_ids; no removed-membership rows.
• The migration was written by hand (Django is not installed in my environment). CI's migration check is its first test.

Test plan

• Query plans and timings on a prod-sized test database (above).
• CI: test_handler.py unit tests (test_list_directory_metadata, test_publication_listing_uses_repository_content_date) pass with the new query.
• CI: migration matches the model (makemigrations --check), lint, sanity.
• End to end: HEAD requests through a content app running this change.
• Index write cost on bulk inserts.
• CREATE INDEX CONCURRENTLY duration and statement_timeout behavior on a prod-sized table.
• Follow-up: same membership shape for Q1 and Q3.

[kdelee]

I tested the old and proposed query shapes in GABI against the production-shaped dataset.

For a nonexistent directory prefix, the SQL plans were:

-- Existing shape
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT ca.content_id, ca.relative_path
FROM core_contentartifact ca
WHERE ca.relative_path LIKE '<directory-prefix>%'
  AND ca.content_id IN (
    SELECT unnest(content_ids)
    FROM core_repositoryversion
    WHERE pulp_id = '<repository-version-uuid>'::uuid
  );

-- Proposed shape
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT ca.content_id, ca.relative_path
FROM core_contentartifact ca
WHERE ca.relative_path LIKE '<directory-prefix>%'
  AND EXISTS (
    SELECT 1
    FROM core_repositorycontent rc
    JOIN core_repositoryversion va
      ON va.pulp_id = rc.version_added_id
    LEFT JOIN core_repositoryversion vr
      ON vr.pulp_id = rc.version_removed_id
    WHERE rc.repository_id = '<repository-uuid>'::uuid
      AND rc.content_id = ca.content_id
      AND va.number <= <version-number>
      AND (
        rc.version_removed_id IS NULL
        OR vr.number > <version-number>
      )
  );

Observed GABI results for the same nonexistent path:

Existing query:  ~0.6 ms
Proposed query:  ~16,788 ms

The proposed plan performed a parallel sequential scan of core_contentartifact, removing approximately 9.5M rows per worker. The existing content-artifact index is content-id-leading, so it cannot efficiently satisfy a relative_path-only prefix lookup.

The PR currently does not add a migration or a relative_path-leading index. hypopg is not enabled in GABI, so I could not test a hypothetical index.

Please recheck the index assumption and test the proposed query with the actual production index schema. A suitable relative_path-leading index may be required before this query shape is beneficial.

[kdelee]

I compared this with Jan's #8262. The indexes are attribute-equivalent: B-tree on core_contentartifact(relative_path) using the text_pattern_ops operator class.

I added the shared pulp-service migration tracked in pulp/pulp-service#1641. Please use the shared index name core_ca_relpath_pattern_idx and avoid creating a second equivalent index.

[larrymou9]

Closing this in favor of the combined work:

• pulpcore [DRAFT - DO NOT MERGE] PULP-2346: combined read-only content query optimizations (needs pulp-service #1642 first) #8270 (draft, for CI): Experiment: optimize read-only content queries #8262 and DRAFT: Add interval-table option to RepositoryVersion.get_content() #8167 combined, without index creation.
• pulp-service Use new-style urlpatterns #1643: the same changes as patches 0080 and 0081.
• Both depend on pulp-service Fix a formatting typo in the changelog #1642, which creates the index (core_ca_relpath_pattern_idx) and merges first.

The perf results for the directory-listing query stay in this PR's description for reference.