Deploying to gh-pages from @ c2882ab 🚀

Author: josix

_sources/howto/free-threading-python.rst.txt

@@ -165,3 +165,132 @@ to false.  If the flag is true then the :class:`warnings.catch_warnings`
 context manager uses a context variable for warning filters.  If the flag is
 false then :class:`~warnings.catch_warnings` modifies the global filters list,
 which is not thread-safe.  See the :mod:`warnings` module for more details.
+
+
+Increased memory usage
+----------------------
+
+The free-threaded build will typically use more memory compared to the default
+build.  There are multiple reasons for this, mostly due to design decisions.
+
+
+All interned strings are immortal
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+For modern Python versions (since version 2.3), interning a string (e.g. with
+:func:`sys.intern`) does not cause it to become immortal.  Instead, if the last
+reference to that string disappears, it will be removed from the interned
+string table.  This is not the case for the free-threaded build and any interned
+string will become immortal, surviving until interpreter shutdown.
+
+
+Non-GC objects have a larger object header
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The free-threaded build uses a different :c:type:`PyObject` structure.  Instead
+of having the GC related information allocated before the :c:type:`PyObject`
+structure, like in the default build, the GC related info is part of the normal
+object header.  For example, on the AMD64 platform, ``None`` uses 32 bytes on
+the free-threaded build vs 16 bytes for the default build.  GC objects (such as
+dicts and lists) are the same size for both builds since the free-threaded
+build does not use additional space for the GC info.
+
+
+QSBR can delay freeing of memory
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In order to safely implement lock-free data structures, a safe memory
+reclamation (SMR) scheme is used, known as quiescent state-based reclamation
+(QSBR).  This means that the memory backing data structures allowing lock-free
+access will use QSBR, which defers the free operation, rather than immediately
+freeing the memory.  Two examples of these data structures are the list object
+and the dictionary keys object.  See ``InternalDocs/qsbr.md`` in the CPython
+source tree for more details on how QSBR is implemented.  Running
+:func:`gc.collect` should cause all memory being held by QSBR to be actually
+freed.  Note that even when QSBR frees the memory, the underlying memory
+allocator may not immediately return that memory to the OS and so the resident
+set size (RSS) of the process might not decrease.
+
+
+mimalloc allocator vs pymalloc
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The default build will normally use the "pymalloc" memory allocator for small
+allocations (512 bytes or smaller).  The free-threaded build does not use
+pymalloc and allocates all Python objects using the "mimalloc" allocator.  The
+pymalloc allocator has the following properties that help keep memory usage
+low: small per-allocated-block overhead, effective memory fragmentation
+prevention, and quick return of free memory to the operating system.  The
+mimalloc allocator does quite well in these respects as well but can have some
+more overhead.
+
+In the free-threaded build, mimalloc manages memory in a number of separate
+heaps (currently four).  For example, all GC supporting objects are allocated
+from their own heap.  Using separate heaps means that free memory in one heap
+cannot be used for an allocation that uses another heap.  Also, some heaps are
+configured to use QSBR (quiescent-state based reclamation) when freeing the
+memory that backs up the heap (known as "pages" in mimalloc terminology).  The
+use of QSBR creates a delay between all memory blocks for a page being freed
+and the memory page being released, either for new allocations or back to the
+OS.
+
+The mimalloc allocator also defers returning freed memory back to the OS.  You
+can reduce that delay by setting the environment variable
+:envvar:`!MIMALLOC_PURGE_DELAY` to ``0``.  Note that this will likely reduce
+the performance of the allocator.
+
+
+Free-threaded reference counting can cause objects to live longer
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In the default build, when an object's reference count reaches zero, it is
+normally deallocated.  The free-threaded build uses "biased reference
+counting", with a fast-path for objects "owned" by the current thread and a
+slow path for other objects.  See :pep:`703` for additional details.  Any time
+an object's reference count ends up in a "queued" state, deallocation can be
+deferred.  The queued state is cleared from the "eval breaker" section of the
+bytecode evaluator.
+
+The free-threaded build also allows a different mode of reference counting,
+known as "deferred reference counting".  This mode is enabled by setting a flag
+on a per-object basis.  Deferred reference counting is enabled for the
+following types:
+
+* module objects
+* module top-level functions
+* class methods defined in the class scope
+* descriptor objects
+* thread-local objects, created by :class:`threading.local`
+
+When deferred reference counting is enabled, references from Python function
+stacks are not added to the reference count.  This scheme reduces the overhead
+of reference counting, especially for objects used from multiple threads.
+Because the stack references are not counted, objects with deferred reference
+counting are not immediately freed when their internal reference count goes to
+zero.  Instead, they are examined by the next GC run and, if no stack
+references to them are found, they are freed.  This means these objects are
+freed by the GC and not when their reference count goes to zero, as is typical.
+
+
+Per-thread reference counting can delay freeing objects
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To avoid contention on the reference count fields of frequently shared
+objects, the free-threaded build also uses "per-thread reference counting"
+for a few selected object types.  Rather than updating a single shared
+reference count, each thread maintains its own local reference count array,
+indexed by a unique id assigned to the object.  The true reference count is
+only computed by summing the per-thread counts when the object's local
+count drops to zero.  Per-thread reference counting is currently used for:
+
+* heap type objects (classes created in Python)
+* code objects
+* the ``__dict__`` of module objects
+
+Because the per-thread counts must be merged back to the object before it
+can be deallocated, objects using per-thread reference counting are
+typically freed later than they would be in the default build.  In
+particular, such an object is usually not freed until the thread that
+referenced it reaches a safe point (for example, in the "eval breaker"
+section of the bytecode evaluator) or exits.  Running :func:`gc.collect`
+will merge the per-thread counts and allow these objects to be freed.

_sources/library/base64.rst.txt

@@ -16,8 +16,10 @@
 This module provides functions for encoding binary data to printable
 ASCII characters and decoding such encodings back to binary data.
 This includes the :ref:`encodings specified in <base64-rfc-4648>`
-:rfc:`4648` (Base64, Base32 and Base16)
-and the non-standard :ref:`Base85 encodings <base64-base-85>`.
+:rfc:`4648` (Base64, Base32 and Base16), the :ref:`Base85 encoding
+<base64-base-85>` specified in `PDF 2.0
+<https://pdfa.org/resource/iso-32000-2/>`_, and non-standard variants
+of Base85 used elsewhere.
 
 There are two interfaces provided by this module.  The modern interface
 supports encoding :term:`bytes-like objects <bytes-like object>` to ASCII
@@ -189,19 +191,28 @@ POST request.
 Base85 Encodings
 -----------------
 
-Base85 encoding is not formally specified but rather a de facto standard,
-thus different systems perform the encoding differently.
+Base85 encoding is a family of algorithms which represent four bytes
+using five ASCII characters.  Originally implemented in the Unix
+``btoa(1)`` utility, a version of it was later adopted by Adobe in the
+PostScript language and is standardized in PDF 2.0 (ISO 32000-2).
+This version, in both its ``btoa`` and PDF variants, is implemented by
+:func:`a85encode`.
 
-The :func:`a85encode` and :func:`b85encode` functions in this module are two implementations of
-the de facto standard. You should call the function with the Base85
-implementation used by the software you intend to work with.
+A separate version, using a different output character set, was
+defined as an April Fool's joke in :rfc:`1924` but is now used by Git
+and other software.  This version is implemented by :func:`b85encode`.
 
-The two functions present in this module differ in how they handle the following:
+Finally, a third version, using yet another output character set
+designed for safe inclusion in programming language strings, is
+defined by ZeroMQ and implemented here by :func:`z85encode`.
 
-* Whether to include enclosing ``<~`` and ``~>`` markers
-* Whether to include newline characters
-* The set of ASCII characters used for encoding
-* Handling of null bytes
+The functions present in this module differ in how they handle the following:
+
+* Whether to include and expect enclosing ``<~`` and ``~>`` markers.
+* Whether to fold the input into multiple lines.
+* The set of ASCII characters used for encoding.
+* Compact encodings of sequences of spaces and null bytes.
+* The encoding of zero-padding bytes applied to the input.
 
 Refer to the documentation of the individual functions for more information.
 
@@ -212,17 +223,22 @@ Refer to the documentation of the individual functions for more information.
 
    *foldspaces* is an optional flag that uses the special short sequence 'y'
    instead of 4 consecutive spaces (ASCII 0x20) as supported by 'btoa'. This
-   feature is not supported by the "standard" Ascii85 encoding.
+   feature is not supported by the standard encoding used in PDF.
 
    *wrapcol* controls whether the output should have newline (``b'\n'``)
    characters added to it. If this is non-zero, each output line will be
    at most this many characters long, excluding the trailing newline.
 
-   *pad* controls whether the input is padded to a multiple of 4
-   before encoding. Note that the ``btoa`` implementation always pads.
+   *pad* controls whether zero-padding applied to the end of the input
+   is fully retained in the output encoding, as done by ``btoa``,
+   producing an exact multiple of 5 bytes of output. This is not part
+   of the standard encoding used in PDF, as it does not preserve the
+   length of the data.
 
-   *adobe* controls whether the encoded byte sequence is framed with ``<~``
-   and ``~>``, which is used by the Adobe implementation.
+   *adobe* controls whether the encoded byte sequence is framed with
+   ``<~`` and ``~>``, as in a PostScript base-85 string literal.  Note
+   that while ASCII85Decode streams in PDF documents *must* be
+   terminated with ``~>``, they *must not* use a leading ``<~``.
 
    .. versionadded:: 3.4
 
@@ -234,10 +250,12 @@ Refer to the documentation of the individual functions for more information.
 
    *foldspaces* is a flag that specifies whether the 'y' short sequence
    should be accepted as shorthand for 4 consecutive spaces (ASCII 0x20).
-   This feature is not supported by the "standard" Ascii85 encoding.
+   This feature is not supported by the standard Ascii85 encoding used in
+   PDF and PostScript.
 
-   *adobe* controls whether the input sequence is in Adobe Ascii85 format
-   (i.e. is framed with <~ and ~>).
+   *adobe* controls whether the ``<~`` and ``~>`` markers are
+   present. While the leading ``<~`` is not required, the input must
+   end with ``~>``, or a :exc:`ValueError` is raised.
 
    *ignorechars* should be a byte string containing characters to ignore
    from the input. This should only contain whitespace characters, and by
@@ -251,35 +269,40 @@ Refer to the documentation of the individual functions for more information.
    Encode the :term:`bytes-like object` *b* using base85 (as used in e.g.
    git-style binary diffs) and return the encoded :class:`bytes`.
 
-   If *pad* is true, the input is padded with ``b'\0'`` so its length is a
-   multiple of 4 bytes before encoding.
+   The input is padded with ``b'\0'`` so its length is a multiple of 4
+   bytes before encoding.  If *pad* is true, all the resulting
+   characters are retained in the output, which will always be a
+   multiple of 5 bytes, and thus the length of the data may not be
+   preserved on decoding.
 
    .. versionadded:: 3.4
 
 
 .. function:: b85decode(b)
 
    Decode the base85-encoded :term:`bytes-like object` or ASCII string *b* and
-   return the decoded :class:`bytes`.  Padding is implicitly removed, if
-   necessary.
+   return the decoded :class:`bytes`.
 
    .. versionadded:: 3.4
 
 
 .. function:: z85encode(s)
 
    Encode the :term:`bytes-like object` *s* using Z85 (as used in ZeroMQ)
-   and return the encoded :class:`bytes`.  See `Z85  specification
-   <https://rfc.zeromq.org/spec/32/>`_ for more information.
+   and return the encoded :class:`bytes`.
+
+   The `ZeroMQ specification <https://rfc.zeromq.org/spec/32/>`_
+   requires the length of Z85-encoded data to be a multiple of 5
+   bytes. To produce compliant data frames, you must pad the input
+   data to this function to a multiple of 4 bytes.
 
    .. versionadded:: 3.13
 
 
 .. function:: z85decode(s)
 
    Decode the Z85-encoded :term:`bytes-like object` or ASCII string *s* and
-   return the decoded :class:`bytes`.  See `Z85  specification
-   <https://rfc.zeromq.org/spec/32/>`_ for more information.
+   return the decoded :class:`bytes`.
 
    .. versionadded:: 3.13
 
@@ -352,3 +375,11 @@ recommended to review the security section for any code deployed to production.
       Section 5.2, "Base64 Content-Transfer-Encoding," provides the definition of the
       base64 encoding.
 
+   `ISO 32000-2 Portable document format - Part 2: PDF 2.0 <https://pdfa.org/resource/iso-32000-2/>`_
+      Section 7.4.3, "ASCII85Decode Filter," provides the definition
+      of the Ascii85 encoding used in PDF and PostScript, including
+      the output character set and the details of data length preservation
+      using zero-padding and partial output groups.
+
+   `ZeroMQ RFC 32/Z85 <https://rfc.zeromq.org/spec/32/>`_
+      The "Formal Specification" section provides the character set used in Z85.

_sources/library/binascii.rst.txt

@@ -158,9 +158,8 @@ The :mod:`!binascii` module defines the following functions:
    of hexadecimal digits (which can be upper or lower case), otherwise an
    :exc:`Error` exception is raised.
 
-   Similar functionality (accepting only text string arguments, but more
-   liberal towards whitespace) is also accessible using the
-   :meth:`bytes.fromhex` class method.
+   Similar functionality (but more liberal towards whitespace) is also accessible
+   using the :meth:`bytes.fromhex` class method.
 
 .. exception:: Error
 

_sources/library/codecs.rst.txt

@@ -1152,7 +1152,7 @@ particular, the following variants typically exist:
 +-----------------+--------------------------------+--------------------------------+
 | cp857           | 857, IBM857                    | Turkish                        |
 +-----------------+--------------------------------+--------------------------------+
-| cp858           | 858, IBM858                    | Western Europe                 |
+| cp858           | 858, IBM00858                  | Western Europe                 |
 +-----------------+--------------------------------+--------------------------------+
 | cp860           | 860, IBM860                    | Portuguese                     |
 +-----------------+--------------------------------+--------------------------------+
@@ -1189,7 +1189,7 @@ particular, the following variants typically exist:
 |                 |                                |                                |
 |                 |                                | .. versionadded:: 3.4          |
 +-----------------+--------------------------------+--------------------------------+
-| cp1140          | ibm1140                        | Western Europe                 |
+| cp1140          | IBM01140                       | Western Europe                 |
 +-----------------+--------------------------------+--------------------------------+
 | cp1250          | windows-1250                   | Central and Eastern Europe     |
 +-----------------+--------------------------------+--------------------------------+

_sources/library/copy.rst.txt

@@ -72,9 +72,13 @@ file, socket, window, or any similar types.  It does "copy" functions and
 classes (shallow and deeply), by returning the original object unchanged; this
 is compatible with the way these are treated by the :mod:`pickle` module.
 
-Shallow copies of dictionaries can be made using :meth:`dict.copy`, and
-of lists by assigning a slice of the entire list, for example,
-``copied_list = original_list[:]``.
+Shallow copies of many collections can be made using the corresponding
+:meth:`!copy` method (such as :meth:`list.copy`, :meth:`dict.copy` or
+:meth:`set.copy`), and of sequences (such as lists or bytearrays) by making
+a slice of the entire sequence (``sequence[:]``).
+However, these methods and slicing can create an instance of the base type
+when copying an instance of a subclass, whereas :func:`copy.copy` normally
+returns an instance of the same type.
 
 .. index:: pair: module; pickle
 

_sources/library/ctypes.rst.txt

@@ -16,6 +16,14 @@ used to wrap these libraries in pure Python.
 
 .. include:: ../includes/optional-module.rst
 
+.. warning::
+
+   :mod:`!ctypes` provides low-level access to native libraries and the
+   process's memory, bypassing Python's safety mechanisms and allowing
+   execution of arbitrary native code.
+   Incorrect use can corrupt data and objects, reveal sensitive information,
+   cause crashes, or otherwise compromise the running process.
+
 
 .. _ctypes-ctypes-tutorial:
 
@@ -200,10 +208,8 @@ argument values::
    OSError: exception: access violation reading 0x00000020
    >>>
 
-There are, however, enough ways to crash Python with :mod:`!ctypes`, so you
-should be careful anyway.  The :mod:`faulthandler` module can be helpful in
-debugging crashes (e.g. from segmentation faults produced by erroneous C library
-calls).
+The :mod:`faulthandler` module can help debug crashes,
+such as segmentation faults produced by erroneous C library calls.
 
 ``None``, integers, bytes objects and (unicode) strings are the only native
 Python objects that can directly be used as parameters in these function calls.