This commit adds a background reader that can optionally buffer reads.
It is used whenever a potentially blocking write is made to the server.
The background reader is started on a slight delay so there should be no
meaningful performance impact as it doesn't run for quick queries and
its overhead is minimal relative to slower queries.
The non-blocking IO system was designed to solve three problems:
1. Deadlock that can occur when both sides of a connection are blocked
writing because all buffers between are full.
2. The inability to use a write deadline with a TLS.Conn without killing
the connection.
3. Efficiently check if a connection has been closed before writing.
This reduces the cases where the application doesn't know if a query
that does a INSERT/UPDATE/DELETE was actually sent to the server or
not.
However, the nbconn package is extraordinarily complex, has been a
source of very tricky bugs, and has OS specific code paths. It also does
not work at all with underlying net.Conn implementations that do not
have platform specific non-blocking IO syscall support and do not
properly implement deadlines. In particular, this is the case with
golang.org/x/crypto/ssh.
I believe the deadlock problem can be solved with a combination of a
goroutine for CopyFrom like v4 used and a watchdog for regular queries
that uses time.AfterFunc.
The write deadline problem actually should be ignorable. We check for
context cancellation before sending a query and the actual Write should
be almost instant as long as the underlying connection is not blocked.
(We should only have to wait until it is accepted by the OS, not until
it is fully sent.)
Efficiently checking if a connection has been closed is probably the
hardest to solve without non-blocking reads. However, the existing code
only solves part of the problem. It can detect a closed or broken
connection the OS knows about, but it won't actually detect other types
of broken connections such as a network interruption. This is currently
implemented in CheckConn and called automatically when checking a
connection out of the pool that has been idle for over one second. I
think that changing CheckConn to a very short deadline read and changing
the pool to do an actual Ping would be an acceptable solution.
Remove nbconn and non-blocking code. This does not leave the system in
an entirely working state. In particular, CopyFrom is broken, deadlocks
can occur for extremely large queries or batches, and PgConn.CheckConn
is now a `select 1` ping. These will be resolved in subsequent commits.
Tests should timeout in a reasonable time if something is stuck. In
particular this is important when testing deadlock conditions such as
can occur with the copy protocol if both the client and the server are
blocked writing until the other side does a read.
This ensures that a closed connection at the pgconn layer is not
considered okay when the background closing of the net.Conn is still in
progress.
This also means that CheckConn cannot be called when the connection is
locked (for example, by in an progress query). But that seems
reasonable. It's not exactly clear that that would have ever worked
anyway.
https://github.com/jackc/pgx/issues/1618#issuecomment-1563702231
The tests for cancelling requests were failing when using unix
sockets. The reason is that net.Conn.RemoteAddr() calls getpeername()
to get the address. For Unix sockets, this returns the address that
was passed to bind() by the *server* process, not the address that
was passed to connect() by the *client*. For postgres, this is always
relative to the server's directory, so is a path like:
./.s.PGSQL.5432
Since it does not return the full absolute path, this function cannot
connect, so it cannot cancel requests. To fix it, use the connection's
config for Unix sockets. I think this should be okay, since a system
using unix sockets should not have "fallbacks". If that is incorrect,
we will need to save the address on PgConn.
Fixes the following failed tests when using Unix sockets:
--- FAIL: TestConnCancelRequest (2.00s)
pgconn_test.go:2056:
Error Trace: /Users/evan.jones/pgx/pgconn/pgconn_test.go:2056
/Users/evan.jones/pgx/pgconn/asm_arm64.s:1172
Error: Received unexpected error:
dial unix ./.s.PGSQL.5432: connect: no such file or directory
Test: TestConnCancelRequest
pgconn_test.go:2063:
Error Trace: /Users/evan.jones/pgx/pgconn/pgconn_test.go:2063
Error: Object expected to be of type *pgconn.PgError, but was <nil>
Test: TestConnCancelRequest
--- FAIL: TestConnContextCanceledCancelsRunningQueryOnServer (5.10s)
pgconn_test.go:2109:
Error Trace: /Users/evan.jones/pgx/pgconn/pgconn_test.go:2109
Error: Received unexpected error:
timeout: context already done: context deadline exceeded
Test: TestConnContextCanceledCancelsRunningQueryOnServer
ioutil.TempFile: Deprecated: As of Go 1.17, this function simply calls os.CreateTemp.
ioutil.ReadFile: Deprecated: As of Go 1.16, this function simply calls os.ReadFile.
If multiple hostnames are provided and one cannot be resolved the others
should still be tried.
Longterm, it would be nice for the connect process to return a list of
errors rather than just one.
fixes https://github.com/jackc/pgx/issues/1464
CopyFrom had to create a prepared statement to get the OIDs of the data
types that were going to be copied into the table. Every COPY operation
required an extra round trips to retrieve the type information. There
was no way to customize this behavior.
By leveraging the QueryExecMode feature, like in `Conn.Query`, users can
specify if they want to cache the prepared statements, execute
them on every request (like the old behavior), or bypass the prepared
statement relying on the pgtype.Map to get the type information.
The `QueryExecMode` behave exactly like in `Conn.Query` in the way the
data type OIDs are fetched, meaning that:
- `QueryExecModeCacheStatement`: caches the statement.
- `QueryExecModeCacheDescribe`: caches the statement and assumes they do
not change.
- `QueryExecModeDescribeExec`: gets the statement description on every
execution. This is like to the old behavior of `CopyFrom`.
- `QueryExecModeExec` and `QueryExecModeSimpleProtocol`: maintain the
same behavior as before, which is the same as `QueryExecModeDescribeExec`.
It will keep getting the statement description on every execution
The `QueryExecMode` can only be set via
`ConnConfig.DefaultQueryExecMode`, unlike `Conn.Query` there's no
support for specifying the `QueryExecMode` via optional arguments
in the function signature.
CockroachDB added support for COPY in version 20.2.
https://www.cockroachlabs.com/docs/v20.2/copy-from
There are some limitations on the implementation, that's why not all the
existing tests were enabled.
* Extract CONTRIBUTING.md
* Add instructions and scripts to setup standalone PostgreSQL server
that tests the various connection and authentication types.
Instead of using pgproto3.FieldDescription through pgconn and pgx. This
lets the lowest level pgproto3 still be as memory efficient as possible.
https://github.com/jackc/pgx/pull/1281
Previously, a batch with 10 unique parameterized statements executed
100 times would entail 11 network round trips. 1 for each prepare /
describe and 1 for executing them all. Now pipeline mode is used to
prepare / describe all statements in a single network round trip. So it
would only take 2 round trips.
Jack Christensen
Nicola Murino
Evan Jones
Alexey Palazhchenko
Mark Chambers
Alejandro Do Nascimento Mora
Tommy Reilly