Replace Newtonsoft.Json with System.Text.Json in tests and samples by paulmedynski · Pull Request #4323 · dotnet/SqlClient

paulmedynski · 2026-05-29T11:59:23Z

Description

Removes the Newtonsoft.Json dependency from all test and sample projects, replacing usage with System.Text.Json. We don't need 2 JSON library dependencies.

Based on #4297 which already migrated TestUtilities/Config.cs.

Changes

JsonBulkCopyTest / JsonStreamTest — migrated serialization and deep-comparison to System.Text.Json + new JsonTestHelper
JsonTestHelper — shared helper using JsonNode.DeepEquals on .NET 9+ and a recursive JsonElement comparison on .NET 8/462 (with max-depth guard)
SqlExceptionTest — rewrote to serialize key properties via STJ; removed JSONSerializationTest (tested Newtonsoft-specific ISerializable deserialization)
AzureKeyVaultProviderLegacyExample_2_0.cs — deleted (was already commented out and marked TODO for removal)
Project files — removed Newtonsoft.Json from ManualTests, FunctionalTests, Samples csprojs
Directory.Packages.props — removed Newtonsoft.Json PackageVersion entry

Follow-up

Remove orphaned legacy AKV sample reference from MS Docs #4322 tracks removal of the orphaned legacy sample section from MS Docs

Testing

FunctionalTests SqlExceptionTest passes on net8.0, net9.0, net10.0
ManualTests builds clean on net8.0 and net9.0
Samples project builds clean
Zero remaining Newtonsoft references in *.cs, *.csproj, *.props

Note: Will re-target to main once #4297 is squash-merged.

Copilot

Pull request overview

Removes the Newtonsoft.Json dependency from all test and sample projects in favor of System.Text.Json, deletes a long-commented legacy AKV sample, and drops the central Newtonsoft.Json PackageVersion. The PR also bundles a number of unrelated pipeline, flaky-test, and unit-test reliability changes that are not described in the PR text.

Changes:

Migrate JsonBulkCopyTest, JsonStreamTest, and SqlExceptionTest to STJ; add shared JsonTestHelper with JsonNode.DeepEquals on .NET 9+ and a recursive JsonElement fallback (depth-limited) on earlier TFMs.
Remove Newtonsoft.Json PackageReferences from ManualTests/FunctionalTests/Samples csprojs, remove its central PackageVersion, and delete AzureKeyVaultProviderLegacyExample_2_0.cs.
Unrelated extras: reset cached switches in LocalAppContextSwitchesTest, mark several simulated-server tests flaky, gate AsyncCancelledConnectionsTest against Kerberos/Managed Instance, add retryCountOnTaskFailure to dotnet install tasks, switch OneBranch symbols variable group/casing, add restore-dotnet-tools.yml to OneBranch and Kerberos pipelines, and change Kerberos code-coverage stage condition.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
Directory.Packages.props	Drop central `Newtonsoft.Json` `PackageVersion`.
doc/samples/Microsoft.Data.SqlClient.Samples.csproj	Remove `Newtonsoft.Json` `PackageReference`.
doc/samples/AzureKeyVaultProviderLegacyExample_2_0.cs	Delete already-commented legacy AKV sample.
src/.../tests/FunctionalTests/Microsoft.Data.SqlClient.FunctionalTests.csproj	Remove `Newtonsoft.Json` `PackageReference` for net8/net462.
src/.../tests/FunctionalTests/SqlExceptionTest.cs	Replace Newtonsoft round-trip with STJ property-projection serialization; remove `JSONSerializationTest`.
src/.../tests/ManualTests/Microsoft.Data.SqlClient.ManualTests.csproj	Add `JsonTestHelper.cs`; drop `Newtonsoft.Json` `PackageReference`s.
src/.../tests/ManualTests/SQL/JsonTest/JsonBulkCopyTest.cs	Switch JSON serialization and file comparison to STJ + `JsonTestHelper`.
src/.../tests/ManualTests/SQL/JsonTest/JsonStreamTest.cs	Same STJ migration for the stream test.
src/.../tests/ManualTests/SQL/JsonTest/JsonTestHelper.cs	New depth-limited deep-equals helper (delegates to `JsonNode.DeepEquals` on .NET 9+).
src/.../tests/ManualTests/DataCommon/DataTestUtility.cs	Add `IsNotKerberosTest` helper used by gated tests.
src/.../tests/ManualTests/SQL/AsyncTest/AsyncCancelledConnectionsTest.cs	Gate `CancelAsyncConnections` against MI and Kerberos environments.
src/.../tests/UnitTests/Microsoft/Data/SqlClient/LocalAppContextSwitchesTest.cs	Reset cached switch fields via helper before asserting defaults.
src/.../tests/UnitTests/SimulatedServerTests/ConnectionFailoverTests.cs	Mark two transient-fault retry-disabled tests `flaky`.
src/.../tests/UnitTests/SimulatedServerTests/ConnectionRoutingTestsAzure.cs	Mark routed-location timeout test `flaky`.
eng/pipelines/steps/install-dotnet.yml	Add `retryCountOnTaskFailure: 3` to dotnet install tasks.
eng/pipelines/onebranch/variables/onebranch-variables.yml	Switch symbols group to `symbols-variables-v3` with `Ppe` casing.
eng/pipelines/onebranch/sqlclient-non-official.yml	Use new `SymbolsPublishServerPpe` / `SymbolsPublishTokenUriPpe`.
eng/pipelines/onebranch/jobs/build-buildproj-job.yml	Restore dotnet local tools before build.
eng/pipelines/kerberos/sqlclient-kerberos.yml	Restore dotnet local tools; tighten coverage stage condition to `succeeded()`.

paulmedynski · 2026-05-29T12:10:06Z

Issue #4322 tracks the corresponding MSDocs removal of references to this file's snippets.

paulmedynski · 2026-05-29T12:12:52Z

-            Assert.Equal(e.ClientConnectionId, sqlEx.ClientConnectionId);
-            Assert.Equal(e.StackTrace, sqlEx.StackTrace);
-        }
+            // Serialize the properties we want to validate round-trip through JSON.


The old test was confirming that Newtonsoft round-tripping functioned, which isn't our responsibility. Now we're checking that specific fields from our SqlException can be serialized, which is still not really our responsibility, but is at least more targeted.

paulmedynski · 2026-05-29T12:14:01Z

+            });

-        [Fact]
-        public void JSONSerializationTest()


This test was actually a Newtonsoft unit test, which isn't something we should be testing.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Wraith2 · 2026-06-02T22:17:22Z

Any specific reason for removing newtonsoft.json?

paulmedynski · 2026-06-04T10:38:11Z

Any specific reason for removing newtonsoft.json?

@Wraith2 - We already have System.Text.Json with suitable functionality. I didn't see the need for a second JSON lib, and I also don't want anyone to get the impression that we support some specific functionality from a 3rd party lib (like a particular serialization implementation).

- Migrate JsonBulkCopyTest and JsonStreamTest to System.Text.Json - Add JsonTestHelper with DeepEquals supporting net8.0+ and net9.0+ - Rewrite SqlExceptionTest to serialize key properties via STJ - Remove JSONSerializationTest (tested Newtonsoft-specific ISerializable) - Delete legacy AzureKeyVaultProviderLegacyExample_2_0.cs (marked TODO) - Remove Newtonsoft.Json from all csproj files and Directory.Packages.props Tracks: #4322

System.Text.Json serializes supplementary-plane characters (e.g. U+29E3D) as \uD867\uDE3D escape sequences, while SQL Server returns them as literal UTF-8. The DeepEqualsCore fallback compared GetRawText() which preserves escaping differences. Add explicit JsonValueKind.String case using GetString() which decodes both representations to the same .NET string.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

- Wrap 'using System.Text.Json.Nodes' in #if NET9_0_OR_GREATER (only used in that branch) - Use UnsafeRelaxedJsonEscaping in JsonStreamTest and JsonBulkCopyTest to preserve intentional non-ASCII characters in test payloads - Remove unused System.IO and System.Runtime.Serialization.Formatters.Binary from SqlExceptionTest

codecov · 2026-06-08T16:45:09Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 64.11%. Comparing base (5ac26c9) to head (a5e87b4).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4323      +/-   ##
==========================================
- Coverage   66.50%   64.11%   -2.40%     
==========================================
  Files         285      280       -5     
  Lines       43311    66160   +22849     
==========================================
+ Hits        28806    42420   +13614     
- Misses      14505    23740    +9235

Flag	Coverage Δ
CI-SqlClient	`?`
PR-SqlClient-Project	`64.11% <ø> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Copilot AI review requested due to automatic review settings May 29, 2026 11:59

github-project-automation Bot added this to SqlClient Board May 29, 2026

github-project-automation Bot moved this to To triage in SqlClient Board May 29, 2026

Copilot started reviewing on behalf of paulmedynski May 29, 2026 11:59 View session

paulmedynski added Area\Tests Issues that are targeted to tests or test projects Code Health 💊 Issues/PRs that are targeted to source code quality improvements. labels May 29, 2026

paulmedynski added this to the 7.1.0-preview2 milestone May 29, 2026

paulmedynski mentioned this pull request May 29, 2026

Remove orphaned legacy AKV sample reference from MS Docs #4322

Open

Copilot AI reviewed May 29, 2026

View reviewed changes

Comment thread eng/pipelines/kerberos/sqlclient-kerberos.yml

paulmedynski force-pushed the dev/paul/newtonsoft branch from 4c5e6d7 to b842414 Compare May 29, 2026 12:08

paulmedynski commented May 29, 2026

View reviewed changes

paulmedynski moved this from To triage to In progress in SqlClient Board May 29, 2026

Copilot AI review requested due to automatic review settings May 29, 2026 17:04

Copilot started reviewing on behalf of paulmedynski May 29, 2026 17:04 View session

Copilot AI reviewed May 29, 2026

View reviewed changes

paulmedynski mentioned this pull request Jun 3, 2026

Move Test Config to Jsonc #4297

Merged

Base automatically changed from dev/russellben/config-jsonc to main June 8, 2026 09:26

paulmedynski added 2 commits June 8, 2026 08:19

paulmedynski force-pushed the dev/paul/newtonsoft branch from 312ca72 to 77fa2cf Compare June 8, 2026 11:20

paulmedynski marked this pull request as ready for review June 8, 2026 11:23

paulmedynski requested a review from a team as a code owner June 8, 2026 11:23

Copilot AI review requested due to automatic review settings June 8, 2026 11:23

paulmedynski enabled auto-merge (squash) June 8, 2026 11:23

Copilot started reviewing on behalf of paulmedynski June 8, 2026 11:23 View session

github-actions Bot assigned apoorvdeshmukh and priyankatiwari08 Jun 8, 2026

paulmedynski requested a review from apoorvdeshmukh June 8, 2026 11:24

paulmedynski requested a review from priyankatiwari08 June 8, 2026 11:24

paulmedynski moved this from In progress to In review in SqlClient Board Jun 8, 2026

Copilot AI reviewed Jun 8, 2026

View reviewed changes

Merge branch 'main' into dev/paul/newtonsoft

a5e87b4

Copilot AI review requested due to automatic review settings June 8, 2026 18:05

Copilot started reviewing on behalf of paulmedynski June 8, 2026 18:05 View session

Copilot AI reviewed Jun 8, 2026

View reviewed changes

Comment thread src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/JsonTest/JsonStreamTest.cs

apoorvdeshmukh approved these changes Jun 9, 2026

View reviewed changes

cheenamalhotra approved these changes Jun 16, 2026

View reviewed changes

Conversation

paulmedynski commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Follow-up

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

paulmedynski May 29, 2026

Choose a reason for hiding this comment

Uh oh!

paulmedynski May 29, 2026

Choose a reason for hiding this comment

Uh oh!

paulmedynski May 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Wraith2 commented Jun 2, 2026

Uh oh!

paulmedynski commented Jun 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

paulmedynski commented May 29, 2026 •

edited

Loading

codecov Bot commented Jun 8, 2026 •

edited

Loading