GOMODULES
/
ksql
mirror of https://github.com/VinGarcia/ksql.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Update README by moving some specialized sections to the Wiki

pull/23/head
Vinícius Garcia 2022-07-03 23:43:58 -03:00
parent be9f686a9a
commit cf93f7441a
3 changed files with 41 additions and 394 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

417
README.md
View File

@ -7,13 +7,13 @@
KissSQL or the "Keep it Simple" SQL package was created to offer an
actually simple and satisfactory tool for interacting with SQL Databases.
The core idea on `ksql` is to offer an easy to use interface,
The core idea on KSQL is to offer an easy to use interface,
the actual comunication with the database is decoupled so we can use
`ksql` on top of `pgx`, `database/sql` and possibly other tools.
You can even create you own backend adapter for `ksql` which is
KSQL on top of `pgx`, `database/sql` and possibly other tools.
You can even create you own backend adapter for KSQL which is
useful in some situations.
## Using `ksql`
## Using KSQL
This is a TLDR version of the more complete examples below.
@ -59,7 +59,7 @@ func main() {
fmt.Println("number of users by type:", count)
// For loading entities from the database `ksql` can build
// For loading entities from the database KSQL can build
// the SELECT part of the query for you if you omit it like this:
var users []User
err = db.Query(ctx, &users, "FROM users WHERE type = $1", "admin")
@ -81,97 +81,15 @@ but work on different databases, they are:
- `ksqlserver.New(ctx, os.Getenv("POSTGRES_URL"), ksql.Config{})` for SQLServer, it works on top of `database/sql`
- `ksqlite3.New(ctx, os.Getenv("POSTGRES_URL"), ksql.Config{})` for SQLite3, it works on top of `database/sql`
## Why `ksql`?
> Note: If you want numbers see our [Benchmark section](https://github.com/vingarcia/ksql#benchmark-comparison) below
ksql is meant to improve on the existing ecosystem by providing
a well-designed database package that has:
1. A small number of easy-to-use helper functions for common use cases
2. Support for more complicated use-cases by allowing
the user to write SQL directly
This strategy allows the API to be
- Very simple with a small number of functions
- Harness all the power of SQL, by just allowing the user to type SQL
- Less opportunities for making mistakes, which makes code reviews easier
- A succinct and idiomatic Go idiom reducing the cognitive complexity of your code
- Easy ways of mocking your database when you need to
- Support for all common relational database: `mysql`, `sqlite`, `sqlserver` and `postgres`
Some special use-cases also have some special support:
- The `QueryChunks()` method helps you in the few situations when you might
need to load in a single query more data than would fit in memory.
- For saving you time when you are selecting all fields from a struct you
can omit the `SELECT ...` part of the query which causes ksql to write
this part for you saving a lot of work when working with big structs/tables.
- The Nested Structs feature will help you reuse existing structs/models when working with JOINs.
**Supported Drivers:**
ksql is well decoupled from its backend implementation which makes
it easy to change the actual technology used, currently we already
support the following options:
- Using the `database/sql` as the backend we support the following drivers:
- `"postgres"`
- `"sqlite3"`
- `"mysql"`
- `"sqlserver"`
- We also support `pgx` (actually `pgxpool`) as the backend which
is a lot faster for Postgres databases.
If you need a new `database/sql` driver or backend adapter included
please open an issue or make your own implementation
and submit it as a Pull Request.
## Comparing `ksql` with other tools
`ksql` was created because of a few insatisfactions
with the existing packages for interacting with
relational databases in Go. To mention a few:
**Low Level Tools:**
Tools like `database/sql`, `sqlx` and even `pgx` will usually
require you to check errors several times for the same query and
also when iterating over several rows you end up with a `for rows.Next() {}`
loop which is often more cognitive complex than desirable.
**High Level Tools such as ORMs:**
More high level tools such as `gorm` and `bun` will often force you
and your team to interact with a complicated DSL which requires
time to learn it and then ending up still being a little bit harder
to read than a regular SQL query would be.
**Code Generation tools:**
Tools like `sqlc` and `sqlboiler` that rely on code generation
are good options if performance is your main goal, but they also
have some issues that might bother you:
- There is some learning curve that goes beyond just reading a GoDoc as with most packages.
- You will often need to copy to and from custom generated structs instead of using your own.
- Sometimes the generated function will not be as flexible as you'd prefer forcing you to make
some tricks with SQL (e.g. that happens with `sqlc` for partial updates for example).
- And it does add an extra step on your building process.
And finally you might just prefer to avoid codegen when possible,
in which case ksql is also for you.
## Kiss Interface
The current interface is as follows and we plan on keeping
it with as little functions as possible, so don't expect many additions:
```go
// Provider describes the ksql public behavior
// Provider describes the KSQL public behavior
//
// The Insert, Patch, Delete and QueryOne functions return ksql.ErrRecordNotFound
// The Insert, Patch, Delete and QueryOne functions return `ksql.ErrRecordNotFound`
// if no record was found or no rows were changed during the operation.
type Provider interface {
Insert(ctx context.Context, table Table, record interface{}) error
@ -187,9 +105,25 @@ type Provider interface {
}
```
## Usage examples
## Using KSQL
This example is also available [here](./examples/crud/crud.go)
In the example below we'll cover all the most common use-cases such as:
1. Inserting records
2. Updating records
3. Deleting records
4. Querying one or many records
5. Making transactions
More advanced use cases are illustrated on their own pages on [our Wiki](https://github.com/VinGarcia/ksql/wiki):
- [Querying in Chunks for Big Queries](https://github.com/VinGarcia/ksql/wiki/Querying-in-Chunks-for-Big-Queries)
- [Avoiding Code Duplication with the Select Builder](https://github.com/VinGarcia/ksql/wiki/Avoiding-Code-Duplication-with-the-Select-Builder)
- [Reusing Existing Structs on Queries with JOINs](https://github.com/VinGarcia/ksql/wiki/Reusing-Existing-Structs-on-Queries-with-JOINs)
- [Testing Tools and `ksql.Mock`](https://github.com/VinGarcia/ksql/wiki/Testing-Tools-and-ksql.Mock)
For the more common use-cases please read the example below,
which is also available [here](./examples/crud/crud.go)
if you want to compile it yourself.
```Go
@ -228,7 +162,7 @@ type Address struct {
City string `json:"city"`
}
// UsersTable informs ksql the name of the table and that it can
// UsersTable informs KSQL the name of the table and that it can
// use the default value for the primary key column name: "id"
var UsersTable = ksql.NewTable("users")
@ -297,7 +231,7 @@ func main() {
}
// Retrieving Cristina, note that if you omit the SELECT part of the query
// ksql will build it for you (efficiently) based on the fields from the struct:
// KSQL will build it for you (efficiently) based on the fields from the struct:
var cris User
err = db.QueryOne(ctx, &cris, "FROM users WHERE name = ? ORDER BY id", "Cristina")
if err != nil {
@ -323,7 +257,7 @@ func main() {
// Partial update technique 2:
err = db.Patch(ctx, UsersTable, PartialUpdateUser{
ID: cris.ID,
Age: nullable.Int(28),
Age: nullable.Int(28), // (just a pointer to an int, if null it won't be updated)
})
if err != nil {
panic(err.Error())
@ -374,305 +308,18 @@ func main() {
}
```
## Query Chunks Feature
It's very unsual for us to need to load a number of records from the
database that might be too big for fitting in memory, e.g. load all the
users and send them somewhere. But it might happen.
For these cases it's best to load chunks of data at a time so
that we can work on a substantial amount of data at a time and never
overload our memory capacity. For this use case we have a specific
function called `QueryChunks`:
```golang
err = db.QueryChunks(ctx, ksql.ChunkParser{
Query: "SELECT * FROM users WHERE type = ?",
Params: []interface{}{usersType},
ChunkSize: 100,
ForEachChunk: func(users []User) error {
err := sendUsersSomewhere(users)
if err != nil {
// This will abort the QueryChunks loop and return this error
return err
}
return nil
},
})
if err != nil {
panic(err.Error())
}
```
It's signature is more complicated than the other two Query\* methods,
thus, it is adivisible to always prefer using the other two when possible
reserving this one for the rare use-case where you are actually
loading big sections of the database into memory.
## Select Generator Feature
There are good reasons not to use `SELECT *` queries the most important
of them is that you might end up loading more information than you are actually
going to use putting more pressure in your database for no good reason.
To prevent that `ksql` has a feature specifically for building the `SELECT`
part of the query using the tags from the input struct.
Using it is very simple and it works with all the 3 Query\* functions:
Querying a single user:
```golang
var user User
err = db.QueryOne(ctx, &user, "FROM users WHERE id = ?", userID)
if err != nil {
panic(err.Error())
}
```
Querying a page of users:
```golang
var users []User
err = db.Query(ctx, &users, "FROM users WHERE type = ? ORDER BY id LIMIT ? OFFSET ?", "Cristina", limit, offset)
if err != nil {
panic(err.Error())
}
```
Querying all the users, or any potentially big number of users, from the database (not usual, but supported):
```golang
err = db.QueryChunks(ctx, ksql.ChunkParser{
Query: "FROM users WHERE type = ?",
Params: []interface{}{usersType},
ChunkSize: 100,
ForEachChunk: func(users []User) error {
err := sendUsersSomewhere(users)
if err != nil {
// This will abort the QueryChunks loop and return this error
return err
}
return nil
},
})
if err != nil {
panic(err.Error())
}
```
The implementation of this feature is actually simple internally.
First we check if the query is starting with the word `FROM`,
if it is then we just get the `ksql` tags from the struct and
then use it for building the `SELECT` statement.
The `SELECT` statement is then cached so we don't have to build it again
the next time in order to keep the library efficient even when
using this feature.
## Select Generation with Joins
So there is one use-case that was not covered by `ksql` so far:
What if you want to JOIN multiple tables for which you already have
structs defined? Would you need to create a new struct to represent
the joined columns of the two tables? no, we actually have this covered as well.
`ksql` has a special feature for allowing the reuse of existing
structs by using composition in an anonymous struct, and then
generating the `SELECT` part of the query accordingly:
Querying a single joined row:
```golang
var row struct{
User User `tablename:"u"` // (here the tablename must match the aliased tablename in the query)
Post Post `tablename:"posts"` // (if no alias is used you should use the actual name of the table)
}
err = db.QueryOne(ctx, &row, "FROM users as u JOIN posts ON u.id = posts.user_id WHERE u.id = ?", userID)
if err != nil {
panic(err.Error())
}
```
Querying a page of joined rows:
```golang
var rows []struct{
User User `tablename:"u"`
Post Post `tablename:"p"`
}
err = db.Query(ctx, &rows,
"FROM users as u JOIN posts as p ON u.id = p.user_id WHERE name = ? LIMIT ? OFFSET ?",
"Cristina", limit, offset,
)
if err != nil {
panic(err.Error())
}
```
Querying all the users, or any potentially big number of users, from the database (not usual, but supported):
```golang
err = db.QueryChunks(ctx, ksql.ChunkParser{
Query: "FROM users as u JOIN posts as p ON u.id = p.user_id WHERE type = ?",
Params: []interface{}{usersType},
ChunkSize: 100,
ForEachChunk: func(rows []struct{
User User `tablename:"u"`
Post Post `tablename:"p"`
}) error {
err := sendRowsSomewhere(rows)
if err != nil {
// This will abort the QueryChunks loop and return this error
return err
}
return nil
},
})
if err != nil {
panic(err.Error())
}
```
As advanced as this feature might seem we don't do any parsing of the query,
and all the work is done only once and then cached.
What actually happens is that we use the "tablename" tag to build the `SELECT`
part of the query like this:
- `SELECT u.id, u.name, u.age, p.id, p.title `
This is then cached, and when we need it again we concatenate it with the rest
of the query.
This feature has two important limitations:
1. It is not possible to use `tablename` tags together with normal `ksql` tags.
Doing so will cause the `tablename` tags to be ignored in favor of the `ksql` ones.
2. It is not possible to use it without omitting the `SELECT` part of the query.
While in normal queries we match the selected field with the attribute by name,
in queries joining multiple tables we can't use this strategy because
different tables might have columns with the same name, and we don't
really have access to the full name of these columns making, for example,
it impossible to differentiate between `u.id` and `p.id` except by the
order in which these fields were passed. Thus, it is necessary that
the library itself writes the `SELECT` part of the query when using
this technique so that we can control the order or the selected fields.
Ok, but what if I don't want to use this feature?
You are not forced to, and there are a few use-cases where you would prefer not to, e.g.:
```golang
var rows []struct{
UserName string `ksql:"name"`
PostTitle string `ksql:"title"`
}
err := db.Query(ctx, &rows, "SELECT u.name, p.title FROM users u JOIN posts p ON u.id = p.user_id LIMIT 10")
if err != nil {
panic(err.Error())
}
```
In the example above, since we are only interested in a couple of columns it
is far simpler and more efficient for the database to only select the columns
that we actually care about, so it's better not to use composite structs.
## Testing Examples & ksql.Mock
`ksql.Mock` is a simple mock that is available out of the box for the `ksql.Provider` interface.
For each of the methods available on the interface this Mock has a function attribute with the same
function signature and with the same name but with an `Fn` in the end of the name.
For instantiating this mock and at the same time mocking one of the functions all you need to do is
this:
```golang
var capturedRecord interface{}
var capturedQuery string
var capturedParams []interface{}
mockDB := ksql.Mock{
QueryOneFn: func(ctx context.Context, record interface{}, query string, params ...interface{}) error {
capturedRecord = record
capturedQuery = query
capturedParams = params
// For simulating an error you would do this:
return fmt.Errorf("some fake error")
},
}
var user User
err := GetUser(db, &user, otherArgs)
assert.NotNil(t, err)
assert.Equal(t, user, capturedRecord)
assert.Equal(t, `SELECT * FROM user WHERE other_args=$1`, capturedQuery)
assert.Equal(t, []interface{}{otherArgs}, capturedParams)
```
For different types of functions you might need do some tricks with structs to make it easier
to write the tests such as:
Converting a struct to something that is more easy to assert like a `map[string]interface{}`
or filling a struct with fake data from the database, for these situations the library provides
the following functions:
- `ksqltest.FillStructWith(struct interface{}, dbRow map[string]interface{}) error`
- `ksqltest.FillSliceWith(structSlice interface{}, dbRows []map[string]interface{}) error`
- `ksqltest.StructToMap(struct interface{}) (map[string]interface{}, error)`
For example:
```golang
createdAt := time.Now().Add(10*time.Hour)
mockDB := ksql.Mock{
QueryOneFn: func(ctx context.Context, record interface{}, query string, params ...interface{}) error {
// For simulating a succesful scenario you can just fillup the struct:
return ksqltest.FillStructWith(record, map[string]interface{}{
"id": 42,
"name": "fake-name",
"age": 32,
"created_at": createdAt,
})
},
}
var user User
err := GetUser(db, &user, otherArgs)
assert.Nil(t, err)
assert.Equal(t, user, User{
ID: 42,
Name: "fake-name",
Age: 32,
CreatedAt: createdAt,
})
```
And for running tests on the `QueryChunks` function which is a particularly complex function
we also have this test helper:
- `ksqltest.CallFunctionWithRows(fn interface{}, rows []map[string]interface{}) (map[string]interface{}, error)`
If you want to see examples (we have examples for all the public functions) just
read the example tests available on our [example service](./examples/example_service/).
Please note that that in the example service above we have two sets
of tests in two different files, exemplifying how to use `gomock` and
then how to use `ksql.Mock{}`.
## Benchmark Comparison
The results of the benchmark are good:
they show that ksql is in practical terms,
they show that KSQL is in practical terms,
as fast as sqlx which was our goal from the start.
To understand the benchmark below you must know
that all tests are performed using Postgres 12.1 and
that we are comparing the following tools:
- ksql using the adapter that wraps `database/sql`
- ksql using the adapter that wraps `pgx`
- KSQL using the adapter that wraps `database/sql`
- KSQL using the adapter that wraps `pgx`
- `database/sql`
- `sqlx`
- `pgx` (with `pgxpool`)
@ -732,7 +379,7 @@ Benchmark executed at: 2022-05-31
Benchmark executed on commit: ed0327babe06a657b2348d2e9d5e5ea824a71fc0
```
## Running the ksql tests (for contributors)
## Running the KSQL tests (for contributors)
The tests use `docker-test` for setting up all the supported databases,
which means that:

2
examples/crud/crud.go
View File

@ -128,7 +128,7 @@ func main() {
// Partial update technique 2:
err = db.Patch(ctx, UsersTable, PartialUpdateUser{
ID: cris.ID,
Age: nullable.Int(28),
Age: nullable.Int(28), // (just a pointer to an int, if null it won't be updated)
})
if err != nil {
panic(err.Error())

16
ksql.go
View File

@ -25,9 +25,9 @@ func initializeQueryCache() map[string]map[reflect.Type]string {
return cache
}
// DB represents the ksql client responsible for
// DB represents the KSQL client responsible for
// interfacing with the "database/sql" package implementing
// the KissSQL interface `ksql.Provider`.
// the KSQL interface `ksql.Provider`.
type DB struct {
driver string
dialect Dialect
@ -36,9 +36,9 @@ type DB struct {
// DBAdapter is minimalistic interface to decouple our implementation
// from database/sql, i.e. if any struct implements the functions below
// with the exact same semantic as the sql package it will work with ksql.
// with the exact same semantic as the sql package it will work with KSQL.
//
// To create a new client using this adapter use ksql.NewWithAdapter()
// To create a new client using this adapter use `ksql.NewWithAdapter()`
type DBAdapter interface {
ExecContext(ctx context.Context, query string, args ...interface{}) (Result, error)
QueryContext(ctx context.Context, query string, args ...interface{}) (Rows, error)
@ -74,7 +74,7 @@ type Tx interface {
}
// Config describes the optional arguments accepted
// by the ksql.New() function.
// by the `ksql.New()` function.
type Config struct {
// MaxOpenCons defaults to 1 if not set
MaxOpenConns int
@ -528,13 +528,13 @@ func assertStructPtr(t reflect.Type) error {
}
// Delete deletes one record from the database using the ID or IDs
// defined on the ksql.Table passed as second argument.
// defined on the `ksql.Table` passed as second argument.
//
// For tables with a single ID column you can pass the record
// to be deleted as a struct, as a map or just pass the ID itself.
//
// For tables with composite keys you must pass the record
// as a struct or a map so that ksql can read all the composite keys
// as a struct or a map so that KSQL can read all the composite keys
// from it.
//
// The examples below should work for both types of tables:
@ -944,7 +944,7 @@ func scanRowsFromType(
if info.IsNestedStruct {
// This version is positional meaning that it expect the arguments
// to follow an specific order. It's ok because we don't allow the
// user to type the "SELECT" part of the query for nested ksqltest.
// user to type the "SELECT" part of the query for nested structs.
scanArgs, err = getScanArgsForNestedStructs(dialect, rows, t, v, info)
if err != nil {
return err