diff --git a/README.md b/README.md index 4627cfe..e6c6214 100644 --- a/README.md +++ b/README.md @@ -7,13 +7,13 @@ KissSQL or the "Keep it Simple" SQL package was created to offer an actually simple and satisfactory tool for interacting with SQL Databases. -The core idea on `ksql` is to offer an easy to use interface, +The core idea on KSQL is to offer an easy to use interface, the actual comunication with the database is decoupled so we can use -`ksql` on top of `pgx`, `database/sql` and possibly other tools. -You can even create you own backend adapter for `ksql` which is +KSQL on top of `pgx`, `database/sql` and possibly other tools. +You can even create you own backend adapter for KSQL which is useful in some situations. -## Using `ksql` +## Using KSQL This is a TLDR version of the more complete examples below. @@ -59,7 +59,7 @@ func main() { fmt.Println("number of users by type:", count) - // For loading entities from the database `ksql` can build + // For loading entities from the database KSQL can build // the SELECT part of the query for you if you omit it like this: var users []User err = db.Query(ctx, &users, "FROM users WHERE type = $1", "admin") @@ -81,97 +81,15 @@ but work on different databases, they are: - `ksqlserver.New(ctx, os.Getenv("POSTGRES_URL"), ksql.Config{})` for SQLServer, it works on top of `database/sql` - `ksqlite3.New(ctx, os.Getenv("POSTGRES_URL"), ksql.Config{})` for SQLite3, it works on top of `database/sql` -## Why `ksql`? - -> Note: If you want numbers see our [Benchmark section](https://github.com/vingarcia/ksql#benchmark-comparison) below - -ksql is meant to improve on the existing ecosystem by providing -a well-designed database package that has: - -1. A small number of easy-to-use helper functions for common use cases -2. Support for more complicated use-cases by allowing - the user to write SQL directly - -This strategy allows the API to be - -- Very simple with a small number of functions -- Harness all the power of SQL, by just allowing the user to type SQL -- Less opportunities for making mistakes, which makes code reviews easier -- A succinct and idiomatic Go idiom reducing the cognitive complexity of your code -- Easy ways of mocking your database when you need to -- Support for all common relational database: `mysql`, `sqlite`, `sqlserver` and `postgres` - -Some special use-cases also have some special support: - -- The `QueryChunks()` method helps you in the few situations when you might - need to load in a single query more data than would fit in memory. -- For saving you time when you are selecting all fields from a struct you - can omit the `SELECT ...` part of the query which causes ksql to write - this part for you saving a lot of work when working with big structs/tables. -- The Nested Structs feature will help you reuse existing structs/models when working with JOINs. - -**Supported Drivers:** - -ksql is well decoupled from its backend implementation which makes -it easy to change the actual technology used, currently we already -support the following options: - -- Using the `database/sql` as the backend we support the following drivers: - - `"postgres"` - - `"sqlite3"` - - `"mysql"` - - `"sqlserver"` -- We also support `pgx` (actually `pgxpool`) as the backend which - is a lot faster for Postgres databases. - -If you need a new `database/sql` driver or backend adapter included -please open an issue or make your own implementation -and submit it as a Pull Request. - -## Comparing `ksql` with other tools - -`ksql` was created because of a few insatisfactions -with the existing packages for interacting with -relational databases in Go. To mention a few: - -**Low Level Tools:** - -Tools like `database/sql`, `sqlx` and even `pgx` will usually -require you to check errors several times for the same query and -also when iterating over several rows you end up with a `for rows.Next() {}` -loop which is often more cognitive complex than desirable. - -**High Level Tools such as ORMs:** - -More high level tools such as `gorm` and `bun` will often force you -and your team to interact with a complicated DSL which requires -time to learn it and then ending up still being a little bit harder -to read than a regular SQL query would be. - -**Code Generation tools:** - -Tools like `sqlc` and `sqlboiler` that rely on code generation -are good options if performance is your main goal, but they also -have some issues that might bother you: - -- There is some learning curve that goes beyond just reading a GoDoc as with most packages. -- You will often need to copy to and from custom generated structs instead of using your own. -- Sometimes the generated function will not be as flexible as you'd prefer forcing you to make - some tricks with SQL (e.g. that happens with `sqlc` for partial updates for example). -- And it does add an extra step on your building process. - -And finally you might just prefer to avoid codegen when possible, -in which case ksql is also for you. - ## Kiss Interface The current interface is as follows and we plan on keeping it with as little functions as possible, so don't expect many additions: ```go -// Provider describes the ksql public behavior +// Provider describes the KSQL public behavior // -// The Insert, Patch, Delete and QueryOne functions return ksql.ErrRecordNotFound +// The Insert, Patch, Delete and QueryOne functions return `ksql.ErrRecordNotFound` // if no record was found or no rows were changed during the operation. type Provider interface { Insert(ctx context.Context, table Table, record interface{}) error @@ -187,9 +105,25 @@ type Provider interface { } ``` -## Usage examples +## Using KSQL -This example is also available [here](./examples/crud/crud.go) +In the example below we'll cover all the most common use-cases such as: + +1. Inserting records +2. Updating records +3. Deleting records +4. Querying one or many records +5. Making transactions + +More advanced use cases are illustrated on their own pages on [our Wiki](https://github.com/VinGarcia/ksql/wiki): + +- [Querying in Chunks for Big Queries](https://github.com/VinGarcia/ksql/wiki/Querying-in-Chunks-for-Big-Queries) +- [Avoiding Code Duplication with the Select Builder](https://github.com/VinGarcia/ksql/wiki/Avoiding-Code-Duplication-with-the-Select-Builder) +- [Reusing Existing Structs on Queries with JOINs](https://github.com/VinGarcia/ksql/wiki/Reusing-Existing-Structs-on-Queries-with-JOINs) +- [Testing Tools and `ksql.Mock`](https://github.com/VinGarcia/ksql/wiki/Testing-Tools-and-ksql.Mock) + +For the more common use-cases please read the example below, +which is also available [here](./examples/crud/crud.go) if you want to compile it yourself. ```Go @@ -228,7 +162,7 @@ type Address struct { City string `json:"city"` } -// UsersTable informs ksql the name of the table and that it can +// UsersTable informs KSQL the name of the table and that it can // use the default value for the primary key column name: "id" var UsersTable = ksql.NewTable("users") @@ -297,7 +231,7 @@ func main() { } // Retrieving Cristina, note that if you omit the SELECT part of the query - // ksql will build it for you (efficiently) based on the fields from the struct: + // KSQL will build it for you (efficiently) based on the fields from the struct: var cris User err = db.QueryOne(ctx, &cris, "FROM users WHERE name = ? ORDER BY id", "Cristina") if err != nil { @@ -323,7 +257,7 @@ func main() { // Partial update technique 2: err = db.Patch(ctx, UsersTable, PartialUpdateUser{ ID: cris.ID, - Age: nullable.Int(28), + Age: nullable.Int(28), // (just a pointer to an int, if null it won't be updated) }) if err != nil { panic(err.Error()) @@ -374,305 +308,18 @@ func main() { } ``` -## Query Chunks Feature - -It's very unsual for us to need to load a number of records from the -database that might be too big for fitting in memory, e.g. load all the -users and send them somewhere. But it might happen. - -For these cases it's best to load chunks of data at a time so -that we can work on a substantial amount of data at a time and never -overload our memory capacity. For this use case we have a specific -function called `QueryChunks`: - -```golang -err = db.QueryChunks(ctx, ksql.ChunkParser{ - Query: "SELECT * FROM users WHERE type = ?", - Params: []interface{}{usersType}, - ChunkSize: 100, - ForEachChunk: func(users []User) error { - err := sendUsersSomewhere(users) - if err != nil { - // This will abort the QueryChunks loop and return this error - return err - } - return nil - }, -}) -if err != nil { - panic(err.Error()) -} -``` - -It's signature is more complicated than the other two Query\* methods, -thus, it is adivisible to always prefer using the other two when possible -reserving this one for the rare use-case where you are actually -loading big sections of the database into memory. - -## Select Generator Feature - -There are good reasons not to use `SELECT *` queries the most important -of them is that you might end up loading more information than you are actually -going to use putting more pressure in your database for no good reason. - -To prevent that `ksql` has a feature specifically for building the `SELECT` -part of the query using the tags from the input struct. -Using it is very simple and it works with all the 3 Query\* functions: - -Querying a single user: - -```golang -var user User -err = db.QueryOne(ctx, &user, "FROM users WHERE id = ?", userID) -if err != nil { - panic(err.Error()) -} -``` - -Querying a page of users: - -```golang -var users []User -err = db.Query(ctx, &users, "FROM users WHERE type = ? ORDER BY id LIMIT ? OFFSET ?", "Cristina", limit, offset) -if err != nil { - panic(err.Error()) -} -``` - -Querying all the users, or any potentially big number of users, from the database (not usual, but supported): - -```golang -err = db.QueryChunks(ctx, ksql.ChunkParser{ - Query: "FROM users WHERE type = ?", - Params: []interface{}{usersType}, - ChunkSize: 100, - ForEachChunk: func(users []User) error { - err := sendUsersSomewhere(users) - if err != nil { - // This will abort the QueryChunks loop and return this error - return err - } - return nil - }, -}) -if err != nil { - panic(err.Error()) -} -``` - -The implementation of this feature is actually simple internally. -First we check if the query is starting with the word `FROM`, -if it is then we just get the `ksql` tags from the struct and -then use it for building the `SELECT` statement. - -The `SELECT` statement is then cached so we don't have to build it again -the next time in order to keep the library efficient even when -using this feature. - -## Select Generation with Joins - -So there is one use-case that was not covered by `ksql` so far: - -What if you want to JOIN multiple tables for which you already have -structs defined? Would you need to create a new struct to represent -the joined columns of the two tables? no, we actually have this covered as well. - -`ksql` has a special feature for allowing the reuse of existing -structs by using composition in an anonymous struct, and then -generating the `SELECT` part of the query accordingly: - -Querying a single joined row: - -```golang -var row struct{ - User User `tablename:"u"` // (here the tablename must match the aliased tablename in the query) - Post Post `tablename:"posts"` // (if no alias is used you should use the actual name of the table) -} -err = db.QueryOne(ctx, &row, "FROM users as u JOIN posts ON u.id = posts.user_id WHERE u.id = ?", userID) -if err != nil { - panic(err.Error()) -} -``` - -Querying a page of joined rows: - -```golang -var rows []struct{ - User User `tablename:"u"` - Post Post `tablename:"p"` -} -err = db.Query(ctx, &rows, - "FROM users as u JOIN posts as p ON u.id = p.user_id WHERE name = ? LIMIT ? OFFSET ?", - "Cristina", limit, offset, -) -if err != nil { - panic(err.Error()) -} -``` - -Querying all the users, or any potentially big number of users, from the database (not usual, but supported): - -```golang -err = db.QueryChunks(ctx, ksql.ChunkParser{ - Query: "FROM users as u JOIN posts as p ON u.id = p.user_id WHERE type = ?", - Params: []interface{}{usersType}, - ChunkSize: 100, - ForEachChunk: func(rows []struct{ - User User `tablename:"u"` - Post Post `tablename:"p"` - }) error { - err := sendRowsSomewhere(rows) - if err != nil { - // This will abort the QueryChunks loop and return this error - return err - } - return nil - }, -}) -if err != nil { - panic(err.Error()) -} -``` - -As advanced as this feature might seem we don't do any parsing of the query, -and all the work is done only once and then cached. - -What actually happens is that we use the "tablename" tag to build the `SELECT` -part of the query like this: - -- `SELECT u.id, u.name, u.age, p.id, p.title ` - -This is then cached, and when we need it again we concatenate it with the rest -of the query. - -This feature has two important limitations: - -1. It is not possible to use `tablename` tags together with normal `ksql` tags. - Doing so will cause the `tablename` tags to be ignored in favor of the `ksql` ones. -2. It is not possible to use it without omitting the `SELECT` part of the query. - While in normal queries we match the selected field with the attribute by name, - in queries joining multiple tables we can't use this strategy because - different tables might have columns with the same name, and we don't - really have access to the full name of these columns making, for example, - it impossible to differentiate between `u.id` and `p.id` except by the - order in which these fields were passed. Thus, it is necessary that - the library itself writes the `SELECT` part of the query when using - this technique so that we can control the order or the selected fields. - -Ok, but what if I don't want to use this feature? - -You are not forced to, and there are a few use-cases where you would prefer not to, e.g.: - -```golang -var rows []struct{ - UserName string `ksql:"name"` - PostTitle string `ksql:"title"` -} -err := db.Query(ctx, &rows, "SELECT u.name, p.title FROM users u JOIN posts p ON u.id = p.user_id LIMIT 10") -if err != nil { - panic(err.Error()) -} -``` - -In the example above, since we are only interested in a couple of columns it -is far simpler and more efficient for the database to only select the columns -that we actually care about, so it's better not to use composite structs. - -## Testing Examples & ksql.Mock - -`ksql.Mock` is a simple mock that is available out of the box for the `ksql.Provider` interface. - -For each of the methods available on the interface this Mock has a function attribute with the same -function signature and with the same name but with an `Fn` in the end of the name. - -For instantiating this mock and at the same time mocking one of the functions all you need to do is -this: - -```golang -var capturedRecord interface{} -var capturedQuery string -var capturedParams []interface{} -mockDB := ksql.Mock{ - QueryOneFn: func(ctx context.Context, record interface{}, query string, params ...interface{}) error { - capturedRecord = record - capturedQuery = query - capturedParams = params - - // For simulating an error you would do this: - return fmt.Errorf("some fake error") - }, -} - -var user User -err := GetUser(db, &user, otherArgs) -assert.NotNil(t, err) -assert.Equal(t, user, capturedRecord) -assert.Equal(t, `SELECT * FROM user WHERE other_args=$1`, capturedQuery) -assert.Equal(t, []interface{}{otherArgs}, capturedParams) -``` - -For different types of functions you might need do some tricks with structs to make it easier -to write the tests such as: - -Converting a struct to something that is more easy to assert like a `map[string]interface{}` -or filling a struct with fake data from the database, for these situations the library provides -the following functions: - -- `ksqltest.FillStructWith(struct interface{}, dbRow map[string]interface{}) error` -- `ksqltest.FillSliceWith(structSlice interface{}, dbRows []map[string]interface{}) error` -- `ksqltest.StructToMap(struct interface{}) (map[string]interface{}, error)` - -For example: - -```golang -createdAt := time.Now().Add(10*time.Hour) -mockDB := ksql.Mock{ - QueryOneFn: func(ctx context.Context, record interface{}, query string, params ...interface{}) error { - // For simulating a succesful scenario you can just fillup the struct: - return ksqltest.FillStructWith(record, map[string]interface{}{ - "id": 42, - "name": "fake-name", - "age": 32, - "created_at": createdAt, - }) - }, -} - -var user User -err := GetUser(db, &user, otherArgs) -assert.Nil(t, err) -assert.Equal(t, user, User{ - ID: 42, - Name: "fake-name", - Age: 32, - CreatedAt: createdAt, -}) -``` - -And for running tests on the `QueryChunks` function which is a particularly complex function -we also have this test helper: - -- `ksqltest.CallFunctionWithRows(fn interface{}, rows []map[string]interface{}) (map[string]interface{}, error)` - -If you want to see examples (we have examples for all the public functions) just -read the example tests available on our [example service](./examples/example_service/). - -Please note that that in the example service above we have two sets -of tests in two different files, exemplifying how to use `gomock` and -then how to use `ksql.Mock{}`. - ## Benchmark Comparison The results of the benchmark are good: -they show that ksql is in practical terms, +they show that KSQL is in practical terms, as fast as sqlx which was our goal from the start. To understand the benchmark below you must know that all tests are performed using Postgres 12.1 and that we are comparing the following tools: -- ksql using the adapter that wraps `database/sql` -- ksql using the adapter that wraps `pgx` +- KSQL using the adapter that wraps `database/sql` +- KSQL using the adapter that wraps `pgx` - `database/sql` - `sqlx` - `pgx` (with `pgxpool`) @@ -732,7 +379,7 @@ Benchmark executed at: 2022-05-31 Benchmark executed on commit: ed0327babe06a657b2348d2e9d5e5ea824a71fc0 ``` -## Running the ksql tests (for contributors) +## Running the KSQL tests (for contributors) The tests use `docker-test` for setting up all the supported databases, which means that: diff --git a/examples/crud/crud.go b/examples/crud/crud.go index 91624e7..d6bd271 100644 --- a/examples/crud/crud.go +++ b/examples/crud/crud.go @@ -128,7 +128,7 @@ func main() { // Partial update technique 2: err = db.Patch(ctx, UsersTable, PartialUpdateUser{ ID: cris.ID, - Age: nullable.Int(28), + Age: nullable.Int(28), // (just a pointer to an int, if null it won't be updated) }) if err != nil { panic(err.Error()) diff --git a/ksql.go b/ksql.go index f5f53c6..a447eae 100644 --- a/ksql.go +++ b/ksql.go @@ -25,9 +25,9 @@ func initializeQueryCache() map[string]map[reflect.Type]string { return cache } -// DB represents the ksql client responsible for +// DB represents the KSQL client responsible for // interfacing with the "database/sql" package implementing -// the KissSQL interface `ksql.Provider`. +// the KSQL interface `ksql.Provider`. type DB struct { driver string dialect Dialect @@ -36,9 +36,9 @@ type DB struct { // DBAdapter is minimalistic interface to decouple our implementation // from database/sql, i.e. if any struct implements the functions below -// with the exact same semantic as the sql package it will work with ksql. +// with the exact same semantic as the sql package it will work with KSQL. // -// To create a new client using this adapter use ksql.NewWithAdapter() +// To create a new client using this adapter use `ksql.NewWithAdapter()` type DBAdapter interface { ExecContext(ctx context.Context, query string, args ...interface{}) (Result, error) QueryContext(ctx context.Context, query string, args ...interface{}) (Rows, error) @@ -74,7 +74,7 @@ type Tx interface { } // Config describes the optional arguments accepted -// by the ksql.New() function. +// by the `ksql.New()` function. type Config struct { // MaxOpenCons defaults to 1 if not set MaxOpenConns int @@ -528,13 +528,13 @@ func assertStructPtr(t reflect.Type) error { } // Delete deletes one record from the database using the ID or IDs -// defined on the ksql.Table passed as second argument. +// defined on the `ksql.Table` passed as second argument. // // For tables with a single ID column you can pass the record // to be deleted as a struct, as a map or just pass the ID itself. // // For tables with composite keys you must pass the record -// as a struct or a map so that ksql can read all the composite keys +// as a struct or a map so that KSQL can read all the composite keys // from it. // // The examples below should work for both types of tables: @@ -944,7 +944,7 @@ func scanRowsFromType( if info.IsNestedStruct { // This version is positional meaning that it expect the arguments // to follow an specific order. It's ok because we don't allow the - // user to type the "SELECT" part of the query for nested ksqltest. + // user to type the "SELECT" part of the query for nested structs. scanArgs, err = getScanArgsForNestedStructs(dialect, rows, t, v, info) if err != nil { return err