2.2 compatibility tracker

[cmp0xff]

This is the list of things that are in pandas 2.2 release notes that need to be addressed in pandas-stubs. I have removed the sections Performance improvements and Bug fixes.

PR's welcome. If you do a PR, check off the item and put a link to the PR that closed it. One PR can address multiple issues.

Some of these may already have been taken care of, so if so, check them off and indicate with a comment such as "previously complete"

Upcoming changes in pandas 3.0

• Dedicated string data type (backed by Arrow) by default

Enhancements

• ADBC Driver support in to_sql and read_sql
• Create a pandas Series based on one or more conditions
• Series.struct accessor for PyArrow structured data
• Series.list accessor for PyArrow list data
• Calamine engine for read_excel
• ~DataFrame.to_sql with method parameter set to multi works with Oracle on the backend
• :attr:Series.attrs / :attr:DataFrame.attrs now uses a deepcopy for propagating attrs (/BUG: copy.deepcopy() doesn't deepcopy the metadata in .attrs pandas#54134).
• get_dummies now returning extension dtypes boolean or bool[pyarrow] that are compatible with the input dtype (/BUG: pd.get_dummies should return bool[pyarrow] types pandas#56273)
• read_csv now supports on_bad_lines parameter with engine="pyarrow" (/ENH: Pandas 2.0 with pyarrow engine add the argument like 'skip_bad_lines=True' pandas#54480)
• read_sas returns datetime64 dtypes with resolutions better matching those stored natively in SAS, and avoids returning object-dtype in cases that cannot be stored with datetime64[ns] dtype (/ENH: non-nano datetime64s for read_sas pandas#56127)
• read_spss now returns a DataFrame that stores the metadata in :attr:DataFrame.attrs (/read_spss doens't return the metadata pandas#54264)
• tseries.api.guess_datetime_format is now part of the public API (/ENH: make guess_datetime_format public pandas#54727)
• DataFrame.apply now allows the usage of numba (via engine="numba") to JIT compile the passed function, allowing for potential speedups (/ENH: numba engine in df.apply pandas#54666)
• ExtensionArray._explode interface method added to allow extension type implementations of the explode method (/ENH: make _explode a method of the ExtensionArray interface pandas#54833)
• ExtensionArray.duplicated added to allow extension type implementations of the duplicated method (/ENH/PERF: add ExtensionArray.duplicated pandas#55255)
• Series.ffill, Series.bfill, DataFrame.ffill, and DataFrame.bfill have gained the argument limit_area; 3rd party .ExtensionArray authors need to add this argument to the method _pad_or_backfill (/ENH: add limit_area argument to ffill() method as interpolate( method='ffill', limit_area='inside') as been deprecated pandas#56492)
• Allow passing read_only, data_only and keep_links arguments to openpyxl using engine_kwargs of read_excel (/ENH: _openpyxl.py load_workbook allow to modify the read_only, data_only and keep_links parameters using engine_kwargs pandas#55027)
• Implement Series.interpolate and DataFrame.interpolate for ArrowDtype and masked dtypes (/.interpolate() Method Incompatible with float[pyarrow] Dtype pandas#56267)
• Implement masked algorithms for Series.value_counts (/ENH: Implement masked algorithm for value_counts pandas#54984)
• Implemented Series.dt methods and attributes for ArrowDtype with pyarrow.duration type (/BUG: Cannot access Timedelta properties with Arrow Backend pandas#52284)
• Implemented Series.str.extract for ArrowDtype (/BUG: str.extract Method Not Implemented for pd.ArrowDtype(pa.string()) pandas#56268)
• Improved error message that appears in DatetimeIndex.to_period with frequencies which are not supported as period frequencies, such as "BMS" (/ENH: Raise TypeError when converting DatetimeIndex to PeriodIndex with invalid period frequency pandas#56243)
• Improved error message when constructing Period with invalid offsets such as "QS" (/BUG: QuarterBegin Does not work with Period pandas#55785)
• The dtypes string[pyarrow] and string[pyarrow_numpy] now both utilize the large_string type from PyArrow to avoid overflow for long columns (/BUG: new string dtype fails with >2 GB of data in a single column pandas#56259)

Notable bug fixes

• check_exact now only takes effect for floating-point dtypes in testing.assert_frame_equal and testing.assert_series_equal. In particular, integer dtypes are always checked exactly (/BUG: assert_series_equal not raising on unequal series? pandas#55882)

Deprecations

• Chained assignment
• Deprecate aliases M, Q, Y, etc. in favour of ME, QE, YE, etc. for offsets
• Deprecated automatic downcasting
• Changed Timedelta.resolution_string to return h, min, s, ms, us, and ns instead of H, T, S, L, U, and N, for compatibility with respective deprecations in frequency aliases (/BUG: Either incorrect unit validation for 'T' in to_timedelta() or incorrect documentation pandas#52536)
• Deprecated :attr:offsets.Day.delta, :attr:offsets.Hour.delta, :attr:offsets.Minute.delta, :attr:offsets.Second.delta, :attr:offsets.Milli.delta, :attr:offsets.Micro.delta, :attr:offsets.Nano.delta, use pd.Timedelta(obj) instead (/DEPR: Tick.delta pandas#55498)
• Deprecated pandas.api.types.is_interval and pandas.api.types.is_period, use isinstance(obj, pd.Interval) and isinstance(obj, pd.Period) instead (/DEPR: is_decimal, is_interval pandas#55264)
• Deprecated read_gbq and DataFrame.to_gbq. Use pandas_gbq.read_gbq and pandas_gbq.to_gbq instead https://pandas-gbq.readthedocs.io/en/latest/api.html (/DEPR: read_gbq, DataFrame.to_gbq pandas#55525)
• Deprecated .DataFrameGroupBy.fillna and .SeriesGroupBy.fillna; use .DataFrameGroupBy.ffill, .DataFrameGroupBy.bfill for forward and backward filling or .DataFrame.fillna to fill with a single value (or the Series equivalents) (/DEPR: groupby.fillna pandas#55718)
• Deprecated DateOffset.is_anchored, use obj.n == 1 for non-Tick subclasses (for Tick this was always False) (/DOC: DateOffset.is_anchored track down intention, fix docstring pandas#55388)
• Deprecated DatetimeArray.__init__ and TimedeltaArray.__init__, use array instead (/DEPR: DTA/TDA.__init__ pandas#55623)
• Deprecated Index.format, use index.astype(str) or index.map(formatter) instead (/DEPR: Index.format? pandas#55413)
• Deprecated Series.ravel, the underlying array is already 1D, so ravel is not necessary (/DEPR: Deprecate Series.ravel pandas#52511)
  • DEPR: ravel in Index and Series pandas-dev/pandas#36900 pandas-dev/pandas#52511 #1613
• Deprecated Series.resample and DataFrame.resample with a PeriodIndex (and the 'convention' keyword), convert to DatetimeIndex (with .to_timestamp()) before resampling instead (/DEPR: Resample with PeriodIndex? pandas#53481). Note: this deprecation was later undone in pandas 2.3.3 (/QST: FutureWarning: Resampling with a PeriodIndex is deprecated, how to resample now? pandas#57033)
• Deprecated Series.view, use Series.astype instead to change the dtype (/DEPR: Series.view pandas#20251)
• Deprecated offsets.Tick.is_anchored, use False instead (/DOC: DateOffset.is_anchored track down intention, fix docstring pandas#55388)
• Deprecated core.internals members Block, ExtensionBlock, and DatetimeTZBlock, use public APIs instead (/DEPR: deprecate exposing blocks in core.internals pandas#55139)
• Deprecated year, month, quarter, day, hour, minute, and second keywords in the PeriodIndex constructor, use PeriodIndex.from_fields instead (/DEPR: PeriodIndex.__new__ accepting ordinals, fields pandas#55960)
• Deprecated accepting a type as an argument in Index.view, call without any arguments instead (/API/DEPR: Index.view _typ check, return type pandas#55709)
• Deprecated allowing non-integer periods argument in date_range, timedelta_range, period_range, and interval_range (/DEPR: fractional periods in date_range, timedelta_range, period_range… pandas#56036)
• Deprecated allowing non-keyword arguments in DataFrame.to_clipboard (/DEPR: Positional arguments in to_* I/O methods pandas#54229)
• Deprecated allowing non-keyword arguments in DataFrame.to_csv except path_or_buf (/DEPR: Positional arguments in to_* I/O methods pandas#54229)
• Deprecated allowing non-keyword arguments in DataFrame.to_dict (/DEPR: Positional arguments in to_* I/O methods pandas#54229)
• Deprecated allowing non-keyword arguments in DataFrame.to_excel except excel_writer (/DEPR: Positional arguments in to_* I/O methods pandas#54229)
• Deprecated allowing non-keyword arguments in DataFrame.to_gbq except destination_table (/DEPR: Positional arguments in to_* I/O methods pandas#54229)
• Deprecated allowing non-keyword arguments in DataFrame.to_hdf except path_or_buf (/DEPR: Positional arguments in to_* I/O methods pandas#54229)
• Deprecated allowing non-keyword arguments in DataFrame.to_html except buf (/DEPR: Positional arguments in to_* I/O methods pandas#54229)
• Deprecated allowing non-keyword arguments in DataFrame.to_json except path_or_buf (/DEPR: Positional arguments in to_* I/O methods pandas#54229)
• Deprecated allowing non-keyword arguments in DataFrame.to_latex except buf (/DEPR: Positional arguments in to_* I/O methods pandas#54229)
• Deprecated allowing non-keyword arguments in DataFrame.to_markdown except buf (/DEPR: Positional arguments in to_* I/O methods pandas#54229)
• Deprecated allowing non-keyword arguments in DataFrame.to_parquet except path (/DEPR: Positional arguments in to_* I/O methods pandas#54229)
• Deprecated allowing non-keyword arguments in DataFrame.to_pickle except path (/DEPR: Positional arguments in to_* I/O methods pandas#54229)
• Deprecated allowing non-keyword arguments in DataFrame.to_string except buf (/DEPR: Positional arguments in to_* I/O methods pandas#54229)
• Deprecated allowing non-keyword arguments in DataFrame.to_xml except path_or_buffer (/DEPR: Positional arguments in to_* I/O methods pandas#54229)
• Deprecated allowing passing BlockManager objects to DataFrame or SingleBlockManager objects to Series (/DEPR: accepting Manager objects in DataFrame/Series pandas#52419)
• Deprecated behavior of Index.insert with an object-dtype index silently performing type inference on the result, explicitly call result.infer_objects(copy=False) for the old behavior instead (/API: Index.insert too much dtype inference pandas#51363)
• Deprecated casting non-datetimelike values (mainly strings) in Series.isin and Index.isin with datetime64, timedelta64, and PeriodDtype dtypes (/DEPR: casting in DatetimeLikeArrayMixin.isin pandas#53111)
• Deprecated dtype inference in Index, Series and DataFrame constructors when giving a pandas input, call .infer_objects on the input to keep the current behavior (/DEPR: Series and Index shouldn't do inference on pandas objects pandas#56012)
• Deprecated dtype inference when setting a Index into a DataFrame, cast explicitly instead (/DEPR: Disallow dtype inference when setting Index into DataFrame pandas#56102)
• Deprecated including the groups in computations when using .DataFrameGroupBy.apply and .DataFrameGroupBy.resample; pass include_groups=False to exclude the groups (/API: way to exclude the grouped column with apply pandas#7155)
• Deprecated indexing an Index with a boolean indexer of length zero (/BUG: inconsistency in Index.__getitem__ for Numpy and non-numpy dtypes pandas#55820)
• Deprecated not passing a tuple to .DataFrameGroupBy.get_group or .SeriesGroupBy.get_group when grouping by a length-1 list-like (/Groupby on single key should be accessible by tuple of length 1 pandas#25971)
• Deprecated string AS denoting frequency in YearBegin and strings AS-DEC, AS-JAN, etc. denoting annual frequencies with various fiscal year starts (/DEPR: deprecate the alias 'A' in favour of 'Y' for year end frequency pandas#54275)
• Deprecated string A denoting frequency in YearEnd and strings A-DEC, A-JAN, etc. denoting annual frequencies with various fiscal year ends (/DEPR: deprecate the alias 'A' in favour of 'Y' for year end frequency pandas#54275)
• Deprecated string BAS denoting frequency in BYearBegin and strings BAS-DEC, BAS-JAN, etc. denoting annual frequencies with various fiscal year starts (/DEPR: deprecate the alias 'A' in favour of 'Y' for year end frequency pandas#54275)
• Deprecated string BA denoting frequency in BYearEnd and strings BA-DEC, BA-JAN, etc. denoting annual frequencies with various fiscal year ends (/DEPR: deprecate the alias 'A' in favour of 'Y' for year end frequency pandas#54275)
• Deprecated strings H, BH, and CBH denoting frequencies in Hour, BusinessHour, CustomBusinessHour (/BUG: Either incorrect unit validation for 'T' in to_timedelta() or incorrect documentation pandas#52536)
• Deprecated strings H, S, U, and N denoting units in to_timedelta (/BUG: Either incorrect unit validation for 'T' in to_timedelta() or incorrect documentation pandas#52536)
• Deprecated strings H, T, S, L, U, and N denoting units in Timedelta (/BUG: Either incorrect unit validation for 'T' in to_timedelta() or incorrect documentation pandas#52536)
• Deprecated strings T, S, L, U, and N denoting frequencies in Minute, Second, Milli, Micro, Nano (/BUG: Either incorrect unit validation for 'T' in to_timedelta() or incorrect documentation pandas#52536)
• Deprecated support for combining parsed datetime columns in read_csv along with the keep_date_col keyword (/DEPR: read_csv keywords: keep_date_col, delim_whitespace pandas#55569)
• Deprecated the :attr:.DataFrameGroupBy.grouper and :attr:SeriesGroupBy.grouper; these attributes will be removed in a future version of pandas (/DEPR: groupby.grouper pandas#56521)
• Deprecated the .Grouping attributes group_index, result_index, and group_arraylike; these will be removed in a future version of pandas (/DEPR: Certain Grouper and Grouping attributes pandas#56148)
• Deprecated the delim_whitespace keyword in read_csv and read_table, use sep="\\s+" instead (/DEPR: read_csv keywords: keep_date_col, delim_whitespace pandas#55569)
• Deprecated the errors="ignore" option in to_datetime, to_timedelta, and to_numeric; explicitly catch exceptions instead (/DEPR: deprecate errors='ignore' in to_datetime and make output dtype predictable pandas#54467)
• Deprecated the fastpath keyword in the Series constructor (/CLN: remove fastpath & verify_integrity from constructors pandas#20110)
• Deprecated the kind keyword in Series.resample and DataFrame.resample, explicitly cast the object's index instead (/DEPR: kind keyword in resample pandas#55895)
• Deprecated the ordinal keyword in PeriodIndex, use PeriodIndex.from_ordinals instead (/DEPR: PeriodIndex.__new__ accepting ordinals, fields pandas#55960)
• Deprecated the unit keyword in TimedeltaIndex construction, use to_timedelta instead (/DEPR: DatetimeIndex/TimedeltaIndex constructor keywords pandas#55499)
• Deprecated the verbose keyword in read_csv and read_table (/DEPR: read_csv keywords: keep_date_col, delim_whitespace pandas#55569)
• Deprecated the behavior of DataFrame.replace and Series.replace with CategoricalDtype; in a future version replace will change the values while preserving the categories. To change the categories, use ser.cat.rename_categories instead (/DEPR/API: Series[categorical].replace behavior pandas#55147)
• Deprecated the behavior of Series.value_counts and Index.value_counts with object dtype; in a future version these will not perform dtype inference on the resulting Index, do result.index = result.index.infer_objects() to retain the old behavior (/DEPR: dtype inference in value_counts pandas#56161)
• Deprecated the default of observed=False in DataFrame.pivot_table; will be True in a future version (/DEPR: observed=False default for DataFrame.pivot_table pandas#56236)
• Deprecated the extension test classes BaseNoReduceTests, BaseBooleanReduceTests, and BaseNumericReduceTests, use BaseReduceTests instead (/DEPR: BaseNoReduceTests pandas#54663)
• Deprecated the option mode.data_manager and the ArrayManager; only the BlockManager will be available in future versions (/DEPR: ArrayManager pandas#55043)
• Deprecated the previous implementation of DataFrame.stack; specify future_stack=True to adopt the future version (/DEPR/BUG: DataFrame.stack including all null rows when stacking multiple levels pandas#53515)

[Dr-Irv]

@cmp0xff I'm reopening this. I think it got closed by mistake when you merged in #1719

Unless you think we have handled all the issues here...

[cmp0xff]

@Dr-Irv indeed, I did not manage to unlink a PR with the issue if the latter is mentioned in the PR like "closes #xxxx", unless I edit the PR description myself 😕 cc @loicdiridollou