Commit Graph

1077 Commits (93380c59e855e430f29d209c3f13d5e660e37e28)

Author SHA1 Message Date
Piotr Tabor 93380c59e8 testing: group (reorder) imports.
Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-28 15:18:46 +01:00
Piotr Tabor 26cc5e3e8b Tests: Migrate from test DB to btesting.DB
Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-28 15:18:46 +01:00
Piotr Tabor f3e164d8dd Example: How to use the separated 'btesting' library.
Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-28 15:18:46 +01:00
Piotr Tabor 46cdbf5b6c Factor out (bolt)testing library.
Thanks to putting it outside the bbolt_test package it can be used to test other packages.
The replacement of the bbolt_test variant inlined in db_test.go is to be performed in a follow up PR.

Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-28 15:18:46 +01:00
Benjamin Wang 9830ab2f14
Merge pull request #365 from ahrtr/bolt_cmd_readonly_20221227
Open db file in readonly mode for commands which shouldn't update db file
2022-12-28 15:51:29 +08:00
Benjamin Wang 5dbd2ec7f3
Merge pull request #366 from ahrtr/workflow_windows_20221227
Support running workflow on Windows-latest
2022-12-28 15:49:41 +08:00
Benjamin Wang 7cc5901171 Extended the test timeout to 30m
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-27 14:51:47 +08:00
Benjamin Wang 99fe21f5e0 Disabled linter on field 'dataref'
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-27 14:51:10 +08:00
Benjamin Wang d04dc0c2d0 Support running workflow on Windows-latest
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-27 14:49:58 +08:00
Benjamin Wang f8dc40c033 Removed the use of log and reorder the import items
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-27 13:05:12 +08:00
Benjamin Wang 329eba7818 Open db file in readonly mode for commands which shouldn't update db file
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-27 12:58:32 +08:00
Piotr Tabor d4831e6217
Merge pull request #358 from ptabor/20221218-defensive-pageread
Validate page being fetched at possition 'p' self identifies as page 'p'
2022-12-23 11:07:33 +01:00
Piotr Tabor 86ce028a52 Defensive: Expect page that node writes onto to be zeroed.
I've seen data corruption that seen like a random bit-flip
or application on already allocated page.

Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-23 09:43:03 +01:00
Piotr Tabor 3cbd9c9a44 Detect pages that have multiple flags set.
Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-23 09:43:03 +01:00
Piotr Tabor d1aa8034d4 Validate page being fetched at possition 'p' self identifies as page 'p'.
It's the easiest verification whether the page is actually written, or its 'random' garbage in the block.

Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-23 09:37:41 +01:00
Piotr Tabor 109f510a5e
Merge pull request #359 from ptabor/20221219-bbolt-page-all
bbolt page supports --all and --value-format=redacted formats.
2022-12-22 10:05:21 +01:00
Piotr Tabor 577a4d99b9 bbolt page supports --all and --value-format=redacted formats.
--all - prints all pages (only skips pages that were considered successful overflow pages)
--value-format=redacted just prints a length of the value, to reduce length and protect privacy.

The auto & redacted formats are supported in all tools that accept '--format'.

Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-22 09:56:16 +01:00
Piotr Tabor 4b8c53388b
Merge pull request #250 from x1ddos/sort-order-docs
bucket,cursor: clarify sort order and seek algorithm
2022-12-21 22:07:36 +01:00
Piotr Tabor 1231aef645
Merge pull request #241 from adream307/master
remove unreachable code
2022-12-21 22:05:57 +01:00
Piotr Tabor a3749e4c36
Merge pull request #287 from dramirez-qb/patch-1
Update README.md fixed typo
2022-12-21 21:39:15 +01:00
Piotr Tabor 696c85ce3d
Merge pull request #356 from ptabor/20221215-flags
Add internal iterator to Bucket that goes over buckets.
2022-12-21 12:58:50 +01:00
Piotr Tabor ebca452da7 Add internal iterator to Bucket that goes over buckets.
So far the code was frequently traversing all the keys (ignoring flag whether key is a bucket)
and trying to open each of the keys as bucket (seeking the same entry from the scratch).

In this proposal, we iterate only through bucket keys.

Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-21 12:16:50 +01:00
Piotr Tabor fa80ceeb19
Merge pull request #357 from ptabor/20221218-stack-to-error
Pages stacktrace printed for initialization errors.
2022-12-21 11:57:31 +01:00
Benjamin Wang f0ccf4e887
Merge pull request #363 from ahrtr/seek_assert_20221221
only perform assert check on exported method
2022-12-21 18:38:20 +08:00
Benjamin Wang ed0438722e only perform assert on exported method
Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-12-21 16:39:30 +08:00
Benjamin Wang dd0ab6da68
Merge pull request #307 from fyrchik/fix-windows-readonly
Fix readonly file mapping on windows
2022-12-21 09:12:26 +08:00
Piotr Tabor 038b2b4d5a
Merge pull request #355 from ptabor/20221216-test-on-16KB-pages
Fix bbolt tests for 16KB block page (e.g. Mac with M1 CPU).
2022-12-20 13:59:00 +01:00
Piotr Tabor cc771afa1e Move page command to a separate file. It's already a huge file of tools.
Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-19 00:42:54 +01:00
Piotr Tabor bad964e850
Merge pull request #354 from ptabor/20221214-fix-bbolt-safety
Add safeguards to bbolt CLI tool
2022-12-18 13:17:08 +01:00
Piotr Tabor 7190a8fe57 Pages stacktrace should be provided for initialization errors.
It makes it easy to find which page actually looks to be corrupted.
```
go build ./cmd/bbolt/ && ./bbolt check ~/Downloads/db
panic: freepages: failed to get all reachable pages (page 8314893338927566090: out of bounds: 6258 (stack: [4517 395 821]))

goroutine 18 [running]:
go.etcd.io/bbolt.(*DB).freepages.func2()
        /Users/ptab/gits/bbolt/db.go:1056 +0x8c
created by go.etcd.io/bbolt.(*DB).freepages
        /Users/ptab/gits/bbolt/db.go:1054 +0x138
```

Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-17 15:32:54 +01:00
Piotr Tabor c34493c3d1 Safety check to 'bbolt page' command.
Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-17 14:43:12 +01:00
Piotr Tabor 1d5a2b0e24
Merge pull request #306 from ptabor/20211220-bbolt_cmd_format
Add support for --format flag for 'keys'
2022-12-16 20:41:01 +01:00
Piotr Tabor b09984ea16 Fix bbolt test for 16KB block page (e.g. Mac with M1 CPU).
The tests were failing when doing development on M1 mac. This was due to page-size difference (4KB assumed in tests) vs. the
OS defaults.

Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-16 19:14:51 +01:00
Piotr Tabor 4acf9d93a5 Fixing golangci warnings.
Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-16 16:08:56 +01:00
Piotr Tabor 505c3888a1
Merge pull request #328 from xpetit/patch-1
Comment fix.
2022-12-16 15:50:57 +01:00
Benjamin Wang 307b638af3
Merge pull request #350 from ahrtr/update_cache_20221124
change freelist.cache from map[pgid]bool to map[pgid]struct{}
2022-12-16 06:13:34 +08:00
Piotr Tabor 5495c633ad bbolt get support --parseFormat and --format flags.
Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-07 22:36:11 +01:00
Piotr Tabor ed3e67c69b writelnBytes method has cleaner contract
Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-07 22:36:11 +01:00
Piotr Tabor b9199f476d Add support for --format flag for 'bbolt keys' command.
This allows to print keys as 'hex', especially useful in context of etcd revisions:
  ./bbolt keys --format=hex ./foo/db key

Signed-off-by: Piotr Tabor <ptab@google.com>
2022-12-07 22:36:11 +01:00
Xavier Petit 8fe68b8f4d
Fix comment
Signed-off-by: xpetit <32063953+xpetit@users.noreply.github.com>
2022-12-01 11:44:53 +01:00
Benjamin Wang 020684ea1e
Merge pull request #315 from missinglink/enable-main-test
cmd/bbolt: re-enable "bbolt" testing
2022-11-30 18:40:10 +08:00
missinglink e7e1f0c05a cmd/bbolt: re-enable "bbolt" testing
Signed-off-by: missinglink <insomnia@rcpt.at>
2022-11-30 11:20:27 +01:00
Benjamin Wang 89c1e714f1
Merge pull request #341 from dchaofei/fix_cursor_last
fix(cursor): `Last` method needs skip empty pages
2022-11-30 10:13:14 +08:00
Evgenii Stratonikov 4504feba82 Close `CreateFileMapping` handle after `MapViewOfFile` failure
mmap-go does this, see 82d537b921

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-11-26 13:25:01 +03:00
Evgenii Stratonikov 52d028ffe9 Fix readonly file mapping on windows
`CreateFileMapping` tries to extend the file to the size of mapping
which leads to failure if database was opened in readonly and calculated
mmap size is bigger than the file size.
Providing 0 to `MapViewOfFile` will create a view which has size of
mapping, i.e. file-size in read-only mode and full size if file was
truncated.

Also, swap `sizehi` and `sizelo` names to reflect windows API docs.
This was changed in 1c97a490d for seemingly no reason.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
2022-11-26 13:15:13 +03:00
Benjamin Wang 70e7654959 change freelist.cache from map[pgid]bool to map[pgid]struct{}
We just need to cache a list of freepage ID, and don't dare what's
the value (true or false) at all, so changed the map's value from
bool to struct{}.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-24 16:06:38 +08:00
dchaofei f1918806b2 fix(cursor): Last needs skip empty pages
Signed-off-by: dchaofei <dchaofei@163.com>
2022-11-18 14:25:48 +08:00
Marek Siarkowicz eedea6cb26
Merge pull request #347 from ahrtr/add_changelog
add changelog for boltDB
2022-11-14 12:41:33 +01:00
Benjamin Wang f4160e70b0 add changelog for boltDB
Added a changelog file for 1.3.x, and planning to release
1.3.7 with a detailed changelog for users reference.

Signed-off-by: Benjamin Wang <wachao@vmware.com>
2022-11-14 18:42:44 +08:00
Marek Siarkowicz c5901d2879
Merge pull request #346 from ahrtr/bump_sys_20221113
bump golang.org/x/sys to v0.2.0
2022-11-13 12:21:35 +01:00