The new logic checks for any type of nil at the beginning of Encode and
then either treats it as NULL or calls the driver.Valuer method if
appropriate.
This should preserve the existing nil normalization while restoring the
ability to encode nil driver.Valuer values.
pgx v5 introduced nil normalization for typed nils. This means that
[]byte(nil) is normalized to nil at the edge of the encoding system.
This simplified encoding logic as nil could be encoded as NULL and type
specific handling was unneeded.
However, database/sql compatibility requires Value to be called on a
nil pointer that implements driver.Valuer. This was broken by
normalizing to nil.
This commit changes the normalization logic to not normalize pointers
that directly implement driver.Valuer to nil. It still normalizes
pointers that implement driver.Valuer through implicit derefence.
e.g.
type T struct{}
func (t *T) Value() (driver.Value, error) {
return nil, nil
}
type S struct{}
func (s S) Value() (driver.Value, error) {
return nil, nil
}
(*T)(nil) will not be normalized to nil but (*S)(nil) will be.
https://github.com/jackc/pgx/issues/1566
This still solves the problem of negative numbers creating a line
comment, but this avoids breaking edge cases such as `set foo to $1`
where the substition is taking place in a location where an arbitrary
expression is not allowed.
https://github.com/jackc/pgx/issues/1928
pgx v5 was not vulnerable to CVE-2024-27289 do to how the sanitizer was
being called. But the sanitizer itself still had the underlying issue.
This commit ports the fix from pgx v4 to v5 to ensure that the issue
does not emerge if pgx uses the sanitizer differently in the future.
When a conn is going to execute a query, the first thing it does is to
deallocate any invalidated prepared statements from the statement cache.
However, the statements were removed from the cache regardless of
whether the deallocation succeeded. This would cause subsequent calls of
the same SQL to fail with "prepared statement already exists" error.
This problem is easy to trigger by running a query with a context that
is already canceled.
This commit changes the deallocate invalidated cached statements logic
so that the statements are only removed from the cache if the
deallocation was successful on the server.
https://github.com/jackc/pgx/issues/1847
QueryExecModeCacheDescribe actually is safe even when the schema or
search_path is modified. It may return an error on the first execution
but it should never silently encode or decode a value incorrectly. Add a
test to demonstrate and ensure this behavior.
Update documentation of QueryExecModeCacheDescribe to remove warning of
undetected result decoding errors.
Update documentation of QueryExecModeCacheStatement and
QueryExecModeCacheDescribe to indicate that the first execution of an
invalidated statement may fail.
The non-blocking IO system was designed to solve three problems:
1. Deadlock that can occur when both sides of a connection are blocked
writing because all buffers between are full.
2. The inability to use a write deadline with a TLS.Conn without killing
the connection.
3. Efficiently check if a connection has been closed before writing.
This reduces the cases where the application doesn't know if a query
that does a INSERT/UPDATE/DELETE was actually sent to the server or
not.
However, the nbconn package is extraordinarily complex, has been a
source of very tricky bugs, and has OS specific code paths. It also does
not work at all with underlying net.Conn implementations that do not
have platform specific non-blocking IO syscall support and do not
properly implement deadlines. In particular, this is the case with
golang.org/x/crypto/ssh.
I believe the deadlock problem can be solved with a combination of a
goroutine for CopyFrom like v4 used and a watchdog for regular queries
that uses time.AfterFunc.
The write deadline problem actually should be ignorable. We check for
context cancellation before sending a query and the actual Write should
be almost instant as long as the underlying connection is not blocked.
(We should only have to wait until it is accepted by the OS, not until
it is fully sent.)
Efficiently checking if a connection has been closed is probably the
hardest to solve without non-blocking reads. However, the existing code
only solves part of the problem. It can detect a closed or broken
connection the OS knows about, but it won't actually detect other types
of broken connections such as a network interruption. This is currently
implemented in CheckConn and called automatically when checking a
connection out of the pool that has been idle for over one second. I
think that changing CheckConn to a very short deadline read and changing
the pool to do an actual Ping would be an acceptable solution.
Remove nbconn and non-blocking code. This does not leave the system in
an entirely working state. In particular, CopyFrom is broken, deadlocks
can occur for extremely large queries or batches, and PgConn.CheckConn
is now a `select 1` ping. These will be resolved in subsequent commits.
The test was relying on sending so big a message that the write blocked.
However, it appears that on Windows the TCP connections over localhost
have an very large or infinite sized buffer. Change the test to simply
set the deadline to the current time before triggering the write.
The first 5 fake non-blocking reads are limited to 1 byte. This should
ensure that there is a measurement of a read where bytes are already
waiting in Go or the OS's read buffer.
The reason for a high max wait time was to ensure that reads aren't
cancelled when there is data waiting for it in Go or the OS's receive
buffer. Unfortunately, there is no way to know ahead of time how long
this should take.
This new code uses 2x the fastest successful read time as the max read
time. This allows the code to adapt to whatever host it is running on.
https://github.com/jackc/pgx/issues/1481
Jack Christensen
maksymnevajdev
Ville Skyttä
Nicola Murino
Dmitry K
Alexey Palazhchenko
Mark Chambers