gh-105689: Parse only the current statement in the IDLE Shell by serhiy-storchaka · Pull Request #157594 · python/cpython

serhiy-storchaka · 2026-09-15T21:55:43Z

A bug in the conversion of a Shell indents from a literal tab to (default) 4 spaces (PR #25678, issue #82073) resulted in
HyperParser and newline_and_indent_event taking the editor path in the Shell and scanning back through previous output. Unbalanced quotes or parentheses in that output — a traceback for raise ValueError("""\n"""), a SyntaxError: '(' was never closed — then broke calltips, parenthesis matching and auto-indent for everything typed afterwards. Use an explicit is_shell attribute and parse only the current statement in the Shell, as before 3.10. This also fixes the Shell examples of #124451.

🤖 Generated with Claude Code

Issue: IDLE: Shell calltips bugged by printed unclosed "(''' " #105689

Since the prompts moved to the sidebar, sys.ps1 ends with a newline and prompt_last_line is empty, so HyperParser and newline_and_indent took the editor path in the Shell and parsed previous output. Use an explicit is_shell attribute instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

terryjreedy · 2026-09-17T01:28:55Z

The space indent patch was developed in parallel with and partially as an alternative to Tal Einat's proposed sidebar (issue #82084). The original intent of my patch was to move the default prompt '>>>\n', possibly replaced by sys.ps1, to it own line so as to enable the indent change. When the two patches worked so well together, we dropped the idea of adding an option for an own-line prompt instead of the sidebar. That idea is why the self.ps1 and self.prompt lines remain around the line removed. They can go in a separate cleanup patch as I am ready to merge this. I wish I had added a clear 'I am Shell' variable in my patch.

miss-islington-app · 2026-09-17T01:31:21Z

Thanks @serhiy-storchaka for the PR, and @terryjreedy for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14, 3.15.
🐍🍒⛏🤖

bedevere-app · 2026-09-17T01:31:39Z

GH-157650 is a backport of this pull request to the 3.15 branch.

bedevere-app · 2026-09-17T01:31:47Z

GH-157651 is a backport of this pull request to the 3.14 branch.

bedevere-app · 2026-09-17T01:31:54Z

GH-157652 is a backport of this pull request to the 3.13 branch.

…H-157594) (#157652) gh-105689: Parse only the current statement in the IDLE Shell (GH-157594) Since the prompts moved to the sidebar, sys.ps1 ends with a newline and prompt_last_line is empty, so HyperParser and newline_and_indent took the editor path in the Shell and parsed previous output. Use an explicit is_shell attribute instead. (cherry picked from commit b42dcf6) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…H-157594) (#157651) gh-105689: Parse only the current statement in the IDLE Shell (GH-157594) Since the prompts moved to the sidebar, sys.ps1 ends with a newline and prompt_last_line is empty, so HyperParser and newline_and_indent took the editor path in the Shell and parsed previous output. Use an explicit is_shell attribute instead. (cherry picked from commit b42dcf6) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

bedevere-bot · 2026-09-17T02:54:56Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 CentOS9 NoGIL Refleaks 3.x (tier-1) has failed when building commit b42dcf6.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1610/builds/4008) and take a look at the build logs.
Check if the failure is related to this commit (b42dcf6) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1610/builds/4008

Failed tests:

test_free_threading
test.test_multiprocessing_fork.test_processes

Failed subtests:

test_concurrent_gi_frame - test.test_free_threading.test_generators.TestFTGenerators.test_concurrent_gi_frame
test_interrupt - test.test_multiprocessing_fork.test_processes.WithProcessesTestProcess.test_interrupt

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/contextlib.py", line 116, in inner
    return func(*args, **kwds)
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/_test_multiprocessing.py", line 651, in test_interrupt
    exitcode = self._kill_process(multiprocessing.Process.interrupt)
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/contextlib.py", line 116, in inner
    return func(*args, **kwds)
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/_test_multiprocessing.py", line 632, in _kill_process
    self.assertEqual(join(), None)
                     ~~~~^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/_test_multiprocessing.py", line 305, in __call__
    return self.func(*args, **kwds)
           ~~~~~~~~~^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/multiprocessing/process.py", line 156, in join
    res = self._popen.wait(timeout)
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/multiprocessing/popen_fork.py", line 44, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/multiprocessing/popen_fork.py", line 28, in poll
    pid, sts = os.waitpid(self.pid, flag)
               ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/_test_multiprocessing.py", line 628, in handler
    raise RuntimeError('join took too long: %s' % p)
RuntimeError: join took too long: <Process name='Process-1' pid=3762413 parent=3762384 started daemon>


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/multiprocessing/process.py", line 320, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/_test_multiprocessing.py", line 579, in _sleep_some_event
    time.sleep(100)
    ~~~~~~~~~~^^^^^
KeyboardInterrupt
k


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/contextlib.py", line 116, in inner
    return func(*args, **kwds)
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/_test_multiprocessing.py", line 651, in test_interrupt
    exitcode = self._kill_process(multiprocessing.Process.interrupt)
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/contextlib.py", line 116, in inner
    return func(*args, **kwds)
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/_test_multiprocessing.py", line 632, in _kill_process
    self.assertEqual(join(), None)
                     ~~~~^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/_test_multiprocessing.py", line 305, in __call__
    return self.func(*args, **kwds)
           ~~~~~~~~~^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/multiprocessing/process.py", line 156, in join
    res = self._popen.wait(timeout)
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/multiprocessing/popen_fork.py", line 44, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/multiprocessing/popen_fork.py", line 28, in poll
    pid, sts = os.waitpid(self.pid, flag)
               ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/_test_multiprocessing.py", line 628, in handler
    raise RuntimeError('join took too long: %s' % p)
RuntimeError: join took too long: <Process name='Process-1771' pid=3716963 parent=3690306 started daemon>


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/multiprocessing/process.py", line 320, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/_test_multiprocessing.py", line 578, in _sleep_some_event
    event.set()
    ~~~~~~~~~^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/multiprocessing/synchronize.py", line 349, in set
    with self._cond:
         ^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/multiprocessing/synchronize.py", line 247, in __exit__
    return self._lock.__exit__(*args)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/multiprocessing/synchronize.py", line 100, in __exit__
    return self._semlock.__exit__(*args)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
KeyboardInterrupt
k


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/test_free_threading/test_generators.py", line 218, in test_concurrent_gi_frame
    self.assertEqual(len(frames), 1)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
AssertionError: 0 != 1

serhiy-storchaka requested a review from terryjreedy as a code owner September 15, 2026 21:55

serhiy-storchaka added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes needs backport to 3.15 pre-release feature fixes, bugs and security fixes labels Sep 15, 2026

bedevere-app Bot mentioned this pull request Sep 15, 2026

IDLE: Shell calltips bugged by printed unclosed "(''' " #105689

Closed

bedevere-app Bot added the awaiting core review label Sep 15, 2026

terryjreedy merged commit b42dcf6 into python:main Sep 17, 2026
59 checks passed

bedevere-app Bot removed the awaiting core review label Sep 17, 2026

bedevere-app Bot removed the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Sep 17, 2026

bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Sep 17, 2026

bedevere-app Bot removed the needs backport to 3.13 bugs and security fixes label Sep 17, 2026

Navigation Menu

Uh oh!

gh-105689: Parse only the current statement in the IDLE Shell - #157594

gh-105689: Parse only the current statement in the IDLE Shell#157594
terryjreedy merged 1 commit into
python:mainfrom
serhiy-storchaka:gh-105689-shell-hyperparser

serhiy-storchaka commented Sep 15, 2026 •

edited by terryjreedy

Loading

Uh oh!

terryjreedy commented Sep 17, 2026

Uh oh!

Uh oh!

miss-islington-app Bot commented Sep 17, 2026

Uh oh!

bedevere-app Bot commented Sep 17, 2026

Uh oh!

bedevere-app Bot commented Sep 17, 2026

Uh oh!

bedevere-app Bot commented Sep 17, 2026

Uh oh!

bedevere-bot commented Sep 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

serhiy-storchaka commented Sep 15, 2026 • edited by terryjreedy Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

terryjreedy commented Sep 17, 2026

Uh oh!

Uh oh!

miss-islington-app Bot commented Sep 17, 2026

Uh oh!

bedevere-app Bot commented Sep 17, 2026

Uh oh!

bedevere-app Bot commented Sep 17, 2026

Uh oh!

bedevere-app Bot commented Sep 17, 2026

Uh oh!

bedevere-bot commented Sep 17, 2026

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

serhiy-storchaka commented Sep 15, 2026 •

edited by terryjreedy

Loading