Describe the bug / cleanup
In #4693, datetime expression collation tests were added to spark/src/test/spark-4.1/org/apache/spark/sql/CometCollationSuite.scala. Although these tests behave identically on Spark 4.1 and 4.2+, they currently reside in the spark-4.1 test shim, which applies only to the 4.1 profile. As noted in the review (#4693 (comment)), the recently introduced spark-4.1+ shim is the appropriate location for code shared across Spark 4.1 and all later 4.x releases. Keeping the suite in spark-4.1 would necessitate duplication once the spark-4.2 test profile is configured.
Additional context
Two things are needed, and neither is a trivial file copy:
- The test build does not pick up the
4.1+ shim yet. add-test-source in spark/pom.xml only adds ${shims.majorVerSrc} and ${shims.minorVerSrc}; ${shims.minorPlusVerSrc} is wired into add-source (main sources) but not into the test sources.
CometCollationSuite already exists in the spark-4.1 shim, so the suite has to be moved rather than copied — having the same class in both spark-4.1 and spark-4.1+ would put two org.apache.spark.sql.CometCollationSuite classes on the 4.1 classpath and fail to compile.
The spark-4.0 suite should be left as-is: it carries 4.0-specific join tests and does not pick up the 4.1+ shim.
Proposed change
- Add
<source>src/test/${shims.minorPlusVerSrc}</source> to theadd-test-source execution in spark/pom.xml.
git mv the suite from spark/src/test/spark-4.1/... to spark/src/test/spark-4.1+/... (content unchanged).
Verified with:
./mvnw test -Pspark-4.1 -Dtest=none -Dsuites=org.apache.spark.sql.CometCollationSuite
./mvnw compile test-compile -Pspark-4.2 -DskipTests
Describe the bug / cleanup
In #4693, datetime expression collation tests were added to
spark/src/test/spark-4.1/org/apache/spark/sql/CometCollationSuite.scala. Although these tests behave identically on Spark 4.1 and 4.2+, they currently reside in thespark-4.1test shim, which applies only to the 4.1 profile. As noted in the review (#4693 (comment)), the recently introducedspark-4.1+shim is the appropriate location for code shared across Spark 4.1 and all later 4.x releases. Keeping the suite inspark-4.1would necessitate duplication once thespark-4.2test profile is configured.Additional context
Two things are needed, and neither is a trivial file copy:
4.1+shim yet.add-test-sourceinspark/pom.xmlonly adds${shims.majorVerSrc}and${shims.minorVerSrc};${shims.minorPlusVerSrc}is wired intoadd-source(main sources) but not into the test sources.CometCollationSuitealready exists in thespark-4.1shim, so the suite has to be moved rather than copied — having the same class in bothspark-4.1andspark-4.1+would put twoorg.apache.spark.sql.CometCollationSuiteclasses on the 4.1 classpath and fail to compile.The
spark-4.0suite should be left as-is: it carries 4.0-specific join tests and does not pick up the4.1+shim.Proposed change
<source>src/test/${shims.minorPlusVerSrc}</source>to theadd-test-sourceexecution inspark/pom.xml.git mvthe suite fromspark/src/test/spark-4.1/...tospark/src/test/spark-4.1+/...(content unchanged).Verified with:
./mvnw test -Pspark-4.1 -Dtest=none -Dsuites=org.apache.spark.sql.CometCollationSuite./mvnw compile test-compile -Pspark-4.2 -DskipTests