[SPARK-58220][SQL] Fix scaladoc and comment errors across sql/api#57377
Closed
LuciferYang wants to merge 1 commit into
Closed
[SPARK-58220][SQL] Fix scaladoc and comment errors across sql/api#57377LuciferYang wants to merge 1 commit into
LuciferYang wants to merge 1 commit into
Conversation
Corrects documentation/comments in sql/api that contradict the code (copy-paste artifacts, wrong parameter names, examples calling the wrong symbol, stale or incorrect statements). Documentation only; no behavior change. 19 files touched, e.g.: - SQLContext.tables() documented a `database` column; the output schema emits `namespace`. - DataFrameStatFunctions.corr: the 2-arg and 3-arg examples were swapped. - Dataset.groupingSets: the second example was malformed and would not compile. - SparkSession.config(String, String): dropped the wrong "@note only supported in Connect mode" (it works in Classic too). - SparkDateTimeUtils.truncateTimeToPrecision: example used microseconds for a nanosecond method. - catalog.interface Database/Function @param docs said "table"; Catalog/SQLContext createExternalTable options overloads said "from the given path" though they take no path. - Also DataTypes.createMapType (keyType/valueType), TimestampNTZType, NonEmptyNamespaceException, QueryParsingErrors.cursorInvalidQualifierError, StructType, PositionMapper, DateTimeFormatterHelper, StringConcat, SinkProgress, UserDefinedAggregateFunction.bufferSchema, DataTypeAstBuilder.
HyukjinKwon
approved these changes
Jul 20, 2026
LuciferYang
added a commit
that referenced
this pull request
Jul 21, 2026
### What changes were proposed in this pull request?
This PR corrects scaladoc/comments across the `sql/api` module that contradict the code -- copy-paste artifacts, wrong parameter names, examples calling the wrong symbol, and stale or incorrect statements. It is documentation-only; no behavior changes. 19 files are touched. Highlights:
- `SQLContext.tables()` / `tables(databaseName)` documented a `database` column, but the output schema (via `mapTableDatasetOutput`) emits `namespace`.
- `DataFrameStatFunctions.corr`: the examples on the 2-arg and 3-arg overloads were swapped (each demonstrated the other overload).
- `Dataset.groupingSets`: the second example was missing the outer `Seq(...)` wrapper, so it had the wrong argument type and unbalanced parentheses -- it would not compile.
- `SparkSession.config(String, String)`: removed the incorrect `note` "this is only supported in Connect mode" -- it works in Classic too (the `ClassicOnly` `appName`/`master`/`enableHiveSupport` all delegate to it).
- `SparkDateTimeUtils.truncateTimeToPrecision`: the example used microseconds for a method that operates on nanoseconds.
- `catalog.interface`: `Database.catalog`, `Function.catalog`, and `Function.namespace` `param` docs said "table" (pasted from `Table`). `Catalog.createExternalTable` and `SQLContext.createExternalTable` options-based overloads said "from the given path" although they take no path argument.
- `DataTypes.createMapType` (values documented as `keyType`), `TimestampNTZType` (byte-code type name missing `$`), `NonEmptyNamespaceException` ("already exists" vs not-empty namespace), `QueryParsingErrors.cursorInvalidQualifierError` (`throws`/"Always throws" but it returns the exception), `StructType` (class doc claimed missing names are ignored / `null` returned, but `apply` throws), `PositionMapper` ("binary search / O(log k)" over a linear `.find`; one wrong example offset), `DateTimeFormatterHelper` (stale "synchronised omitted" comment vs `synchronizedMap.computeIfAbsent`), `StringConcat.append` ("Returns true" on a `Unit` method), `SinkProgress.description` ("source" vs sink), `UserDefinedAggregateFunction.bufferSchema` ("input arguments" vs buffer values), and `DataTypeAstBuilder` (a `GEOGRAPHY` branch comment saying "geometry"; `visitIdentityColumn` `return` "Tuple" vs `IdentityColumnSpec`).
### Why are the changes needed?
The documentation misdescribes these APIs and internals. Some are user-facing (e.g. `SQLContext.tables` names a column that does not exist, so `.select("database")` would fail; `config`'s note wrongly restricts it to Connect), and the rest are misleading comments that can send readers to the wrong behavior.
### Does this PR introduce _any_ user-facing change?
No. Documentation/comment-only changes.
### How was this patch tested?
No tests; documentation-only. `sql-api` compiles, and `scalafmt` validation passes on the module. Each correction was verified against the corresponding signature/body (and, where semantics were involved, against the catalyst expression or builder it maps to). Changed lines stay within 100 chars and introduce no non-ASCII characters.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57377 from LuciferYang/SPARK-sqlapi-scaladoc.
Authored-by: YangJie <yangjie01@baidu.com>
Signed-off-by: yangjie01 <yangjie01@baidu.com>
(cherry picked from commit 8fb0675)
Signed-off-by: yangjie01 <yangjie01@baidu.com>
Contributor
Author
Contributor
Author
|
Thank you @HyukjinKwon |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR corrects scaladoc/comments across the
sql/apimodule that contradict the code -- copy-paste artifacts, wrong parameter names, examples calling the wrong symbol, and stale or incorrect statements. It is documentation-only; no behavior changes. 19 files are touched. Highlights:SQLContext.tables()/tables(databaseName)documented adatabasecolumn, but the output schema (viamapTableDatasetOutput) emitsnamespace.DataFrameStatFunctions.corr: the examples on the 2-arg and 3-arg overloads were swapped (each demonstrated the other overload).Dataset.groupingSets: the second example was missing the outerSeq(...)wrapper, so it had the wrong argument type and unbalanced parentheses -- it would not compile.SparkSession.config(String, String): removed the incorrect@note"this is only supported in Connect mode" -- it works in Classic too (the@ClassicOnlyappName/master/enableHiveSupportall delegate to it).SparkDateTimeUtils.truncateTimeToPrecision: the example used microseconds for a method that operates on nanoseconds.catalog.interface:Database.catalog,Function.catalog, andFunction.namespace@paramdocs said "table" (pasted fromTable).Catalog.createExternalTableandSQLContext.createExternalTableoptions-based overloads said "from the given path" although they take no path argument.DataTypes.createMapType(values documented askeyType),TimestampNTZType(byte-code type name missing$),NonEmptyNamespaceException("already exists" vs not-empty namespace),QueryParsingErrors.cursorInvalidQualifierError(@throws/"Always throws" but it returns the exception),StructType(class doc claimed missing names are ignored /nullreturned, butapplythrows),PositionMapper("binary search / O(log k)" over a linear.find; one wrong example offset),DateTimeFormatterHelper(stale "synchronised omitted" comment vssynchronizedMap.computeIfAbsent),StringConcat.append("Returns true" on aUnitmethod),SinkProgress.description("source" vs sink),UserDefinedAggregateFunction.bufferSchema("input arguments" vs buffer values), andDataTypeAstBuilder(aGEOGRAPHYbranch comment saying "geometry";visitIdentityColumn@return"Tuple" vsIdentityColumnSpec).Why are the changes needed?
The documentation misdescribes these APIs and internals. Some are user-facing (e.g.
SQLContext.tablesnames a column that does not exist, so.select("database")would fail;config's note wrongly restricts it to Connect), and the rest are misleading comments that can send readers to the wrong behavior.Does this PR introduce any user-facing change?
No. Documentation/comment-only changes.
How was this patch tested?
No tests; documentation-only.
sql-apicompiles, andscalafmtvalidation passes on the module. Each correction was verified against the corresponding signature/body (and, where semantics were involved, against the catalyst expression or builder it maps to). Changed lines stay within 100 chars and introduce no non-ASCII characters.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)