Previously, a conn.Write would simply unlock pgconn, leaving the
connection as Idle and reusable while the multiResultReader would be
closed. From this state, calling multiResultReader.Close won't try to
receiveMessage and thus won't unwatch and close the connection since it
is already closed. This leaves the connection "open" and the next time
it's used, a "Watch already in progress" panic could be triggered.
This patch fixes the issue by unwatching and closing the connection on a
batch write error. The same was done on Sync.Encode error even if the
path is unreachable as Sync.Error never returns an error.
A single Connect("connstring") may actually make multiple connection
requests due to TLS or HA configuration. Previously, when all attempts
failed only the last error was returned. This could be confusing.
Now details of all failed attempts are included.
For example, the following connection string:
host=localhost,127.0.0.1,foo.invalid port=1,2,3
Will now return an error like the following:
failed to connect to `user=postgres database=pgx_test`:
lookup foo.invalid: no such host
[::1]:1 (localhost): dial error: dial tcp [::1]:1: connect: connection refused
127.0.0.1:1 (localhost): dial error: dial tcp 127.0.0.1:1: connect: connection refused
127.0.0.1:2 (127.0.0.1): dial error: dial tcp 127.0.0.1:2: connect: connection refused
https://github.com/jackc/pgx/issues/1929
The PostgreSQL server will reject messages greater than ~1 GB anyway.
However, worse than that is that a message that is larger than 4 GB
could wrap the 32-bit integer message size and be interpreted by the
server as multiple messages. This could allow a malicious client to
inject arbitrary protocol messages.
https://github.com/jackc/pgx/security/advisories/GHSA-mrww-27vc-gghv
Otherwise, it might be possible to panic when closing the pipeline if it
tries to read a connection that should be closed but still has a fatal
error on the wire.
https://github.com/jackc/pgx/issues/1920
OnPGError is called on every error response received from Postgres and can
be used to close connections on specific errors. Defaults to closing on
FATAL-severity errors.
Fixes#1803
stdlib_test.TestConnConcurrency had been flickering on CI deadline /
timeout errors. This was extremely confusing because the test deadline
was set for 2 minutes and the errors would occur much quicker.
The problem only manifested in an extremely specific and timing
sensitive situation.
1. The watchdog timer for deadlocked writes starts the goroutine to
start the background reader
2. The background reader is stopped
3. The next operation is a read without a preceding write (AFAIK only
CheckConn does this)
4. The deadline is set to interrupt the read
5. The goroutine from 1 actually starts the background reader
6. The background reader gets an error reading the connection with the
deadline
7. The deadline is cleared
8. The next read on the connection will get the timeout error
connect_timeout given in conn string was not obeyed if sslmode is not specified (default is prefer) or equals sslmode=allow|prefer. It took twice the amount of time specified by connect_timeout in conn string. While this behavior is correct if multi-host is provided in conn string, it doesn't look correct in case of single host. This behavior was also not matching with libpq.
The root cause was to implement sslmode=allow|prefer conn are tried twice. First with TLSConfig and if that doesn't work then without TLSConfig. The fix for this issue now uses the same context if same host is being tried out. This change won't affect the existing multi-host behavior.
This PR goal is to close issue [jackc/pgx/issues/1672](https://github.com/jackc/pgx/issues/1672)
This commit adds a background reader that can optionally buffer reads.
It is used whenever a potentially blocking write is made to the server.
The background reader is started on a slight delay so there should be no
meaningful performance impact as it doesn't run for quick queries and
its overhead is minimal relative to slower queries.
The non-blocking IO system was designed to solve three problems:
1. Deadlock that can occur when both sides of a connection are blocked
writing because all buffers between are full.
2. The inability to use a write deadline with a TLS.Conn without killing
the connection.
3. Efficiently check if a connection has been closed before writing.
This reduces the cases where the application doesn't know if a query
that does a INSERT/UPDATE/DELETE was actually sent to the server or
not.
However, the nbconn package is extraordinarily complex, has been a
source of very tricky bugs, and has OS specific code paths. It also does
not work at all with underlying net.Conn implementations that do not
have platform specific non-blocking IO syscall support and do not
properly implement deadlines. In particular, this is the case with
golang.org/x/crypto/ssh.
I believe the deadlock problem can be solved with a combination of a
goroutine for CopyFrom like v4 used and a watchdog for regular queries
that uses time.AfterFunc.
The write deadline problem actually should be ignorable. We check for
context cancellation before sending a query and the actual Write should
be almost instant as long as the underlying connection is not blocked.
(We should only have to wait until it is accepted by the OS, not until
it is fully sent.)
Efficiently checking if a connection has been closed is probably the
hardest to solve without non-blocking reads. However, the existing code
only solves part of the problem. It can detect a closed or broken
connection the OS knows about, but it won't actually detect other types
of broken connections such as a network interruption. This is currently
implemented in CheckConn and called automatically when checking a
connection out of the pool that has been idle for over one second. I
think that changing CheckConn to a very short deadline read and changing
the pool to do an actual Ping would be an acceptable solution.
Remove nbconn and non-blocking code. This does not leave the system in
an entirely working state. In particular, CopyFrom is broken, deadlocks
can occur for extremely large queries or batches, and PgConn.CheckConn
is now a `select 1` ping. These will be resolved in subsequent commits.
This ensures that a closed connection at the pgconn layer is not
considered okay when the background closing of the net.Conn is still in
progress.
This also means that CheckConn cannot be called when the connection is
locked (for example, by in an progress query). But that seems
reasonable. It's not exactly clear that that would have ever worked
anyway.
https://github.com/jackc/pgx/issues/1618#issuecomment-1563702231
The tests for cancelling requests were failing when using unix
sockets. The reason is that net.Conn.RemoteAddr() calls getpeername()
to get the address. For Unix sockets, this returns the address that
was passed to bind() by the *server* process, not the address that
was passed to connect() by the *client*. For postgres, this is always
relative to the server's directory, so is a path like:
./.s.PGSQL.5432
Since it does not return the full absolute path, this function cannot
connect, so it cannot cancel requests. To fix it, use the connection's
config for Unix sockets. I think this should be okay, since a system
using unix sockets should not have "fallbacks". If that is incorrect,
we will need to save the address on PgConn.
Fixes the following failed tests when using Unix sockets:
--- FAIL: TestConnCancelRequest (2.00s)
pgconn_test.go:2056:
Error Trace: /Users/evan.jones/pgx/pgconn/pgconn_test.go:2056
/Users/evan.jones/pgx/pgconn/asm_arm64.s:1172
Error: Received unexpected error:
dial unix ./.s.PGSQL.5432: connect: no such file or directory
Test: TestConnCancelRequest
pgconn_test.go:2063:
Error Trace: /Users/evan.jones/pgx/pgconn/pgconn_test.go:2063
Error: Object expected to be of type *pgconn.PgError, but was <nil>
Test: TestConnCancelRequest
--- FAIL: TestConnContextCanceledCancelsRunningQueryOnServer (5.10s)
pgconn_test.go:2109:
Error Trace: /Users/evan.jones/pgx/pgconn/pgconn_test.go:2109
Error: Received unexpected error:
timeout: context already done: context deadline exceeded
Test: TestConnContextCanceledCancelsRunningQueryOnServer
If multiple hostnames are provided and one cannot be resolved the others
should still be tried.
Longterm, it would be nice for the connect process to return a list of
errors rather than just one.
fixes https://github.com/jackc/pgx/issues/1464
Felix Röhrich
Anthonin Bonnefoy
zenkovev
zenkovev
Jack Christensen
James Hartig
Ryan Fowler
Samuel Stauffer
Ivan Posazhennikov
Anton Levakin
Ville Skyttä
Alexey Palazhchenko
smaher-edb
Adrian-Stefan Mares
Brandon Kauffman
Nicola Murino
Evan Jones