HIVE-29668: Add -rebuildIndexes utility to reconstruct backend Metast… by soumyakanti3578 · Pull Request #6636 · apache/hive

soumyakanti3578 · 2026-07-23T23:30:53Z

…ore indexes

What changes were proposed in this pull request?

Add a rebuildIndexes option to schemaTool
For postgres, mysql, oracle, and mssql, add a rebuild-indexes sql file that drops and recreates all HMS indexes.
Tests

Why are the changes needed?

Gives users the ability to drop and recreate indexes easily through schematool.

Does this PR introduce any user-facing change?

Yes, adds a new option to schematool.

How was this patch tested?

mvn -pl standalone-metastore/metastore-server -Dtest=TestRebuildIndexesScriptConsistency  test
mvn -pl standalone-metastore/metastore-server -Dtest="TestSchemaToolForMetastore#testRebuildIndexes" -Dtest.groups="" -DfailIfNoTests=false test

…ore indexes

…ripts

sonarqubecloud · 2026-07-28T01:33:28Z

Quality Gate passed

Issues
6 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

nrg4878

I wasnt expecting that we will be checking in rebuild-indexes scripts as well. But I do not have a problem per se as long as these get updated when new indices are added.

nrg4878 · 2026-07-30T22:05:56Z

+
+@Category(MetastoreUnitTest.class)
+@RunWith(Parameterized.class)
+public class TestRebuildIndexesScriptConsistency {


can we add a test that fails if we add a new index to the schema file that is not found in the rebuild-indexes file ? So that way developers will remember to update these files as well?

Yes, that is exactly what's being tested in rebuildScriptMatchesInitScript()

Pattern pattern = PATTERNS.get(dbType); Set<String> initIndexes = extractIndexNames(schemaScript, pattern); Set<String> rebuildIndexes = extractIndexNames(rebuildScript, pattern); Set<String> missing = new HashSet<>(initIndexes); missing.removeAll(rebuildIndexes); Set<String> extra = new HashSet<>(rebuildIndexes); extra.removeAll(initIndexes); assertTrue("Indexes in init script missing from rebuild script: " + missing, missing.isEmpty()); assertTrue("Indexes in rebuild script not found in init script: " + extra, extra.isEmpty());

and it tests extra indexes in the rebuild scripts too, so devs can remember to remove them if we ever remove an index from the schema file.

Copilot

Pull request overview

Adds a new -rebuildIndexes operation to the standalone metastore SchemaTool to (re)create HMS backend indexes via DB-specific SQL scripts, with accompanying tests to validate both execution and script/index consistency across supported DBs.

Changes:

Adds -rebuildIndexes command-line option and wires it to a new SchemaToolTaskRebuildIndexes.
Introduces per-DB rebuild-indexes.<db>.sql scripts (postgres/mysql/oracle/mssql) for dropping/recreating indexes.
Adds tests to execute the new task and to ensure rebuild scripts stay consistent with init schema index definitions.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/tools/schematool/TestSchemaToolForMetastore.java	Adds an integration-style test invoking the new `-rebuildIndexes` task.
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/tools/schematool/TestRebuildIndexesScriptConsistency.java	New test validating rebuild scripts’ index sets match init schema scripts for supported DBs.
standalone-metastore/metastore-server/src/main/sql/postgres/rebuild-indexes.postgres.sql	Adds PostgreSQL index rebuild script.
standalone-metastore/metastore-server/src/main/sql/mysql/rebuild-indexes.mysql.sql	Adds MySQL/MariaDB index rebuild script using `ALTER TABLE` for drop+add.
standalone-metastore/metastore-server/src/main/sql/oracle/rebuild-indexes.oracle.sql	Adds Oracle index rebuild script.
standalone-metastore/metastore-server/src/main/sql/mssql/rebuild-indexes.mssql.sql	Adds MSSQL index rebuild script using `DROP_EXISTING = ON`.
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolTaskRebuildIndexes.java	New SchemaTool task that locates and executes the rebuild script (or logs it in dry-run).
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolCommandLine.java	Registers the new `rebuildIndexes` CLI option and fixes the Postgres constant reference.
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/MetastoreSchemaTool.java	Routes `-rebuildIndexes` to the new task implementation.
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/HiveSchemaHelper.java	Renames the Postgres DB type constant to `DB_POSTGRES` and updates parser selection.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

saihemanth-cloudera · 2026-08-03T19:14:42Z

+-- See the License for the specific language governing permissions and
+


nit: incomplete license info.

Ack, will update in the next commit.

soumyakanti3578 · 2026-08-03T20:56:00Z

  public static final String DB_MSSQL = "mssql";
  public static final String DB_MYSQL = "mysql";
-  public static final String DB_POSTGRACE = "postgres";
+  public static final String DB_POSTGRES = "postgres";
  public static final String DB_ORACLE = "oracle";


I don't think this change is very risky, but let me know what others think.

+        LOG.info("Dry run: would execute {}", script.getAbsolutePath());
+        LOG.info(new String(Files.readAllBytes(script.toPath())));
+      } catch (IOException e) {


Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (3)

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/HiveSchemaHelper.java:46

Renaming/removing the public constant DB_POSTGRACE is a source/binary compatibility break for any downstream code that references it. Consider keeping it as a deprecated alias to DB_POSTGRES to preserve compatibility while still fixing the typo.

  public static final String DB_DERBY = "derby";
  public static final String DB_HIVE = "hive";
  public static final String DB_MSSQL = "mssql";
  public static final String DB_MYSQL = "mysql";
  public static final String DB_POSTGRES = "postgres";
  public static final String DB_ORACLE = "oracle";

standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolTaskRebuildIndexes.java:54

new String(Files.readAllBytes(...)) uses the platform default charset, which can make dry-run output vary by environment. Prefer reading with an explicit charset.

        LOG.info("Dry run: would execute {}", script.getAbsolutePath());
        LOG.info(new String(Files.readAllBytes(script.toPath())));
      } catch (IOException e) {

standalone-metastore/metastore-server/src/main/sql/oracle/rebuild-indexes.oracle.sql:20

As written, this Oracle rebuild script will fail immediately if any index is already missing (no DROP INDEX IF EXISTS), which undermines -rebuildIndexes as an unattended recovery utility. Consider making each DROP resilient (e.g., PL/SQL block that ignores ORA-01418) so the tool can recreate missing indexes without manual edits.

-- Oracle has no DROP INDEX IF EXISTS; this script uses plain DROP INDEX.
-- If an index is already missing, comment out its DROP line before running.
-- Update this file whenever a new upgrade script adds an index.

saihemanth-cloudera

For databases with native support, should we prefer REINDEX, ALTER INDEX ... REBUILD, or engine-specific online rebuild options rather drop/create semantics?

saihemanth-cloudera · 2026-07-29T00:00:44Z

+-- Oracle has no DROP INDEX IF EXISTS; this script uses plain DROP INDEX.
+-- If an index is already missing, comment out its DROP line before running.
+-- Update this file whenever a new upgrade script adds an index.


If the utility is run because an index is missing, Oracle fails on the first missing index and never reaches the create. The comment telling users to manually edit the script defeats the purpose of -rebuildIndexes. Can we use use PL/SQL blocks that ignore only “index does not exist”?

Yes, PL/SQL should work for this. I wanted to go with the simpler approach for now given that this feature would be used rarely, and maintainability of a PL/SQL script is higher. I will update the script with PL/SQL.

saihemanth-cloudera · 2026-08-03T19:10:32Z

+-- Update this file whenever a new upgrade script adds an index.
+
+DROP INDEX IF EXISTS "NOTIFICATION_LOG_EVENT_ID";
+CREATE UNIQUE INDEX "NOTIFICATION_LOG_EVENT_ID" ON "NOTIFICATION_LOG" USING btree ("EVENT_ID");


hive-schema-4.3.0.postgres.sql defines important uniqueness objects like UNIQUE_DATABASE, UNIQUETABLE, UNIQUEPARTITION, and UNIQUE_TYPE. This script only covers CREATE INDEX objects, so it cannot fully reconstruct the backend indexing/constraint state?
The new consistency test also misses this because it only extracts CREATE INDEX names from the init script.

saihemanth-cloudera · 2026-08-03T19:12:42Z

+
+@Category(MetastoreUnitTest.class)
+@RunWith(Parameterized.class)
+public class TestRebuildIndexesScriptConsistency {


This would pass if a rebuild script recreated an index with the same name but wrong table, wrong columns, wrong order, wrong uniqueness, or wrong access method.
I think the right way to test this is that, it should parse and compare at least index name, table, column list/order, and uniqueness; for PostgreSQL it also needs constraint-backed indexes. What do you think?

Yes but it would also make it more complex, and we would have to rely more on regex. Moreover, the pattern will be different for all DBs so even more regex!

Since this feature would be rarely used, and addition/deletion of indexes to the schema file is not super common and restricted to developers, we can probably rely on these tests?

What you are suggesting is probably the right thing to do but since we are relying on regex, it will make the code more error-prone. It would probably have been easier to do with the other approach #6545.

Do let me know what you think

saihemanth-cloudera · 2026-08-03T19:14:42Z

+-- See the License for the specific language governing permissions and
+


nit: incomplete license info.

soumyakanti3578 · 2026-08-03T22:31:24Z

@saihemanth-cloudera

For databases with native support, should we prefer REINDEX, ALTER INDEX ... REBUILD, or engine-specific online rebuild options rather drop/create semantics?

From what I understand, if an index was dropped accidentally, these statements won't rebuild it.

asf-ci-hive added tests pending tests passed and removed tests pending labels Jul 23, 2026

soumyakanti3578 marked this pull request as ready for review July 24, 2026 19:46

asf-ci-hive added tests pending and removed tests passed labels Jul 27, 2026

soumyakanti3578 changed the title ~~[WIP]HIVE-29668: Add -rebuildIndexes utility to reconstruct backend Metast…~~ HIVE-29668: Add -rebuildIndexes utility to reconstruct backend Metast… Jul 27, 2026

asf-ci-hive added tests unstable and removed tests pending labels Jul 27, 2026

soumyakanti3578 added 2 commits July 27, 2026 17:35

HIVE-29668: Add -rebuildIndexes utility to reconstruct backend Metast…

4f81362

…ore indexes

Add an integration test that actually runs all the rebuild indexes sc…

d42f9cf

…ripts

soumyakanti3578 force-pushed the HIVE-29668-index-gen branch from e0821f6 to d42f9cf Compare July 28, 2026 00:35

asf-ci-hive added tests pending and removed tests unstable labels Jul 28, 2026

asf-ci-hive added tests unstable and removed tests pending labels Jul 28, 2026

nrg4878 requested review from dengzhhu653 and saihemanth-cloudera July 30, 2026 21:58

nrg4878 reviewed Jul 30, 2026

View reviewed changes

dengzhhu653 requested a review from Copilot August 1, 2026 01:17

Copilot started reviewing on behalf of dengzhhu653 August 1, 2026 01:17 View session

Copilot AI reviewed Aug 1, 2026

View reviewed changes

dengzhhu653 requested a review from Copilot August 3, 2026 06:58

Copilot started reviewing on behalf of dengzhhu653 August 3, 2026 06:59 View session

Copilot AI reviewed Aug 3, 2026

View reviewed changes

saihemanth-cloudera reviewed Aug 3, 2026

View reviewed changes

		-- See the License for the specific language governing permissions and

Uh oh!

Conversation

soumyakanti3578 commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

sonarqubecloud Bot commented Jul 28, 2026

Quality Gate passed

Uh oh!

nrg4878 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

saihemanth-cloudera left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

soumyakanti3578 commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

soumyakanti3578 commented Jul 23, 2026 •

edited

Loading