pgcli crashes when using use directive

[dlangille]

Description

pgcli crashes when using use directive

Your environment

PostgreSQL 16.5
pgcli-4.1.0
Python 3.11

• FreeBSD 14.1
• 4.1.0
• zsh: command not found: pip (I install from package).

dvl@/tmp:samdrucker> use freshports.dvl
You are now connected to database "freshports.dvl" as user "dvl"
Time: 0.022s
Traceback (most recent call last):
  File "/usr/local/bin/pgcli", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
^[[25;1R    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pgcli/main.py", line 1640, in cli
    pgcli.run_cli()
  File "/usr/local/lib/python3.11/site-packages/pgcli/main.py", line 971, in run_cli
    text = self.prompt_app.prompt()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/shortcuts/prompt.py", line 1035, in prompt
    return self.app.run(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/application/application.py", line 1002, in run
    return asyncio.run(coro)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/application/application.py", line 886, in run_async
Exception in thread completion_refresh:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    return await _run_async(f)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/application/application.py", line 739, in _run_async
    self.run()
  File "/usr/local/lib/python3.11/threading.py", line 982, in run
    self._redraw()
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/application/application.py", line 543, in _redraw
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.11/site-packages/pgcli/completion_refresher.py", line 67, in _bg_refresh
    self.context.copy().run(run_in_context)
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/application/application.py", line 526, in run_in_context
    refresher(completer, executor)
  File "/usr/local/lib/python3.11/site-packages/pgcli/completion_refresher.py", line 108, in refresh_schemata
    self.renderer.render(self, self.layout)
    completer.set_search_path(executor.search_path())
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/renderer.py", line 641, in render
  File "/usr/local/lib/python3.11/site-packages/pgcli/pgcompleter.py", line 328, in set_search_path
    layout.container.preferred_height(size.columns, size.rows).preferred,
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/layout/containers.py", line 318, in preferred_height
    self.search_path = self.escaped_names(search_path)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pgcli/pgcompleter.py", line 176, in escaped_names
    dimensions = [
                 ^
    return [self.escape_name(name) for name in names]
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/layout/containers.py", line 319, in <listcomp>
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pgcli/pgcompleter.py", line 176, in <listcomp>
    c.preferred_height(width, max_available_height) for c in self._all_children
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    return [self.escape_name(name) for name in names]
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/layout/containers.py", line 786, in preferred_height
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pgcli/pgcompleter.py", line 157, in escape_name
    return self.content.preferred_height(width, max_available_height)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (not self.name_pattern.match(name))
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/layout/containers.py", line 318, in preferred_height
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: cannot use a string pattern on a bytes-like object
    dimensions = [
                 ^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/layout/containers.py", line 319, in <listcomp>
    c.preferred_height(width, max_available_height) for c in self._all_children
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/layout/containers.py", line 2623, in preferred_height
    if self.filter():
       ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/filters/base.py", line 254, in __call__
    return self.func()
           ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/shortcuts/prompt.py", line 161, in has_before_fragments
    for fragment, char, *_ in get_prompt_text():
                              ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/shortcuts/prompt.py", line 1280, in _get_prompt
    return to_formatted_text(self.message, style="class:prompt")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/formatted_text/base.py", line 82, in to_formatted_text
    return to_formatted_text(value(), style=style)
                             ^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pgcli/main.py", line 1043, in get_message
    prompt = self.get_prompt(prompt_format)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pgcli/main.py", line 1314, in get_prompt
    string = string.replace("\\H", self.pgexecute.host or "(none)")
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: replace() argument 2 must be str, not bytes
[20:41 pg03 dvl ~] % pgcli samdrucker
Server: PostgreSQL 16.4
Version: 4.1.0
Home: http://pgcli.com
dvl@/tmp:samdrucker> use fres
connection failed: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL:  database "fres" does not exist
Previous connection kept
You are now connected to database "samdrucker" as user "dvl"
Time: 0.009s
dvl@/tmp:samdrucker> use freshports.freshports.dvl
connection failed: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL:  database "freshports.freshports.dvl" does not exist
Previous connection kept
You are now connected to database "samdrucker" as user "dvl"
Time: 0.012s
dvl@/tmp:samdrucker> use freshports.dvl
You are now connected to database "freshports.dvl" as user "dvl"
Time: 0.021s
Traceback (most recent call last):
  File "/usr/local/bin/pgcli", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
^[[50;1R    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pgcli/main.py", line 1640, in cli
    pgcli.run_cli()
  File "/usr/local/lib/python3.11/site-packages/pgcli/main.py", line 971, in run_cli
    text = self.prompt_app.prompt()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/shortcuts/prompt.py", line 1035, in prompt
    return self.app.run(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/application/application.py", line 1002, in run
    return asyncio.run(coro)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
Exception in thread completion_refresh:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    self.run()
  File "/usr/local/lib/python3.11/threading.py", line 982, in run
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/application/application.py", line 886, in run_async
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.11/site-packages/pgcli/completion_refresher.py", line 67, in _bg_refresh
    return await _run_async(f)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/application/application.py", line 739, in _run_async
    refresher(completer, executor)
  File "/usr/local/lib/python3.11/site-packages/pgcli/completion_refresher.py", line 108, in refresh_schemata
    completer.set_search_path(executor.search_path())
    self._redraw()
  File "/usr/local/lib/python3.11/site-packages/pgcli/pgcompleter.py", line 328, in set_search_path
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/application/application.py", line 543, in _redraw
    self.search_path = self.escaped_names(search_path)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pgcli/pgcompleter.py", line 176, in escaped_names
    self.context.copy().run(run_in_context)
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/application/application.py", line 526, in run_in_context
    return [self.escape_name(name) for name in names]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pgcli/pgcompleter.py", line 176, in <listcomp>
    return [self.escape_name(name) for name in names]
    self.renderer.render(self, self.layout)
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/renderer.py", line 641, in render
  File "/usr/local/lib/python3.11/site-packages/pgcli/pgcompleter.py", line 157, in escape_name
    (not self.name_pattern.match(name))
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    layout.container.preferred_height(size.columns, size.rows).preferred,
TypeError: cannot use a string pattern on a bytes-like object
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/layout/containers.py", line 318, in preferred_height
    dimensions = [
                 ^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/layout/containers.py", line 319, in <listcomp>
    c.preferred_height(width, max_available_height) for c in self._all_children
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/layout/containers.py", line 786, in preferred_height
    return self.content.preferred_height(width, max_available_height)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/layout/containers.py", line 318, in preferred_height
    dimensions = [
                 ^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/layout/containers.py", line 319, in <listcomp>
    c.preferred_height(width, max_available_height) for c in self._all_children
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/layout/containers.py", line 2623, in preferred_height
    if self.filter():
       ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/filters/base.py", line 254, in __call__
    return self.func()
           ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/shortcuts/prompt.py", line 161, in has_before_fragments
    for fragment, char, *_ in get_prompt_text():
                              ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/shortcuts/prompt.py", line 1280, in _get_prompt
    return to_formatted_text(self.message, style="class:prompt")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/formatted_text/base.py", line 82, in to_formatted_text
    return to_formatted_text(value(), style=style)
                             ^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pgcli/main.py", line 1043, in get_message
    prompt = self.get_prompt(prompt_format)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pgcli/main.py", line 1314, in get_prompt
    string = string.replace("\\H", self.pgexecute.host or "(none)")
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: replace() argument 2 must be str, not bytes

[dbaty]

The error does not seem to be related to use, as it happens on this line:

string = string.replace("\\H", self.pgexecute.host or "(none)")

It looks like self.pgexecute.host here is a bytes object, whereas we expect it to be a string. I cannot reproduce (yet).

As a temporary workaround, you could tweak your pgcli prompt and remove this \H. It's possible that other bindings fail the same (\h, etc.), so you may have to remove these, too.

[ebishop-retool]

Btw, I'm seeing the same error message and almost exactly the same stacktrace, but for me it happens immediately when I run pgcli -p $MY_PORT $MY_DB $MY_USER. In other words, I don't get a chance to enter use or any other command. I'm also on v4.3.0, but on MacOS 15.5.

The difference in the stacktrace for me is that the Python it's using is a Python 3.13 installation that seems to be packaged with pgcli in Homebrew, so everywhere that /usr/local/lib/python3.11 appears in the stacktrace above, I have /opt/homebrew/Cellar/pgcli/4.3.0/libexec/lib/python3.13/ instead.

[DiegoDAF]

Hi,

Thanks for the report. I can reproduce this and I think the three symptoms
(TypeError on the prompt, the b'...'-prefixed timezone message, and the
completion_refresh thread dying in parse_defaults) all share one root
cause.

When the client encoding cannot decode a text value (most commonly
SQL_ASCII), psycopg returns the text columns to us as raw bytes instead of
str. A few code paths were passing those values through unguarded:

• The socket directory query result was used to build the prompt, so a Unix
  socket connection raised TypeError: expected str, not bytes.
• The show timezone result was printed in the startup message, showing a
  b'...'-prefixed value.
• The function metadata query (used by the background completion refresh)
  selected text columns such as arg_defaults; parsing those bytes with
  parse_defaults() crashed the refresh thread with
  TypeError: can only concatenate str (not "int") to str.

The fix is to decode those values defensively when they come back as bytes
(same guard we already use in the completion metadata fix for #1405). It is a
small, self-contained change: a _decode_if_bytes() helper plus decoding the
socket directory, the timezone, and the function metadata rows. I have
reproduced it against a real SQL_ASCII cluster (3396 catalog functions, all
metadata fields come back as plain str after the fix) and added unit tests.

A small PR with tests is coming. I'll link it here once it is in.

its the same for #1518

[DiegoDAF]

Hi,

Thanks for the report. I can reproduce this and I think the three symptoms (TypeError on the prompt, the b'...'-prefixed timezone message, and the completion_refresh thread dying in parse_defaults) all share one root cause.

When the client encoding cannot decode a text value (most commonly SQL_ASCII), psycopg returns the text columns to us as raw bytes instead of str. A few code paths were passing those values through unguarded:

• The socket directory query result was used to build the prompt, so a Unix socket connection raised TypeError: expected str, not bytes.
• The show timezone result was printed in the startup message, showing a b'...'-prefixed value.
• The function metadata query (used by the background completion refresh) selected text columns such as arg_defaults; parsing those bytes with parse_defaults() crashed the refresh thread with TypeError: can only concatenate str (not "int") to str.

The fix is to decode those values defensively when they come back as bytes (same guard we already use in the completion metadata fix for #1405). It is a small, self-contained change: a _decode_if_bytes() helper plus decoding the socket directory, the timezone, and the function metadata rows. I have reproduced it against a real SQL_ASCII cluster (3396 catalog functions, all metadata fields come back as plain str after the fix) and added unit tests.

The fix is up as #1628.