Fixes an issue where failing to open a non-existent database in ReadOnly
mode would make you unable to properly initialize it in ReadWrite mode
afterwards due to a hanging lock.
This allows enabling NoSync when you only have access to the Options
passed into Open() and not the returned DB (as is frequently the case
with libraries).
When the database has a lot of freepages, the cost to sync all
freepages down to disk is high. If the total database size is
small (<10GB), and the application can tolerate ~10 seconds
recovery time, then it is reasonable to simply not sync freelist
and rescan the db to rebuild freelist on recovery.
Read txns would lock pages allocated after the txn, keeping those pages
off the free list until closing the read txn. Instead, track allocating
txid to compute page lifetime, freeing pages if all txns between
page allocation and page free are closed.
The existing append-based implementation left a hanging reference to
the last tx.
For example, if db.txs was:
[]*Tx{0x1, 0x2, 0x3, 0x4, 0x5}
and we removed the second element, db.txs would now be:
[]*Tx{0x1, 0x3, 0x4, 0x5, 0x5}[:4]
The garbage collector cannot reclaim anything anywhere in a slice,
even pointers between its len and cap, because the len can always
be extended up to the cap.
This hanging reference to the Tx could last indefinitely,
and since the Tx has a reference to user-provided functions,
which could be closures, this bug could prevent arbitrary
amounts of user garbage from being collected.
Since db.txs is unordered anyway, switch to a simpler--and O(1) instead
of O(n)--implementation. Swap the last element into the spot to be
deleted, nil out the original last element, and shrink the slice.
The subtraction for `TxN` was previously transposed which caused
the result to be a negative number. This change alters the order
to return the correct (positive) result.
Bolt stores the two latest transactions' metadata, but previously did
not recover from validation failures in the latest by using the second
latest. Fix this by correctly handling validation failures in db.go, as
well as returning the metadata with highest txid which is also valid in
DB.meta().
Signed-off-by: Aleksa Sarai <asarai@suse.de>
This commit refactors the test suite to make it cleaner and to use the
standard testing library better. The `assert()`, `equals()`, and `ok()`
functions have been removed and some test names have been changed for
clarity.
No functionality has been changed.
InitialMmapSize is the initial mmap size of the database in bytes.
Read transaction won't block write transaction if InitialMmapSize
is large enough to handle mmap size.
Copied from https://github.com/boltdb/bolt/pull/432.
- ErrInvalid is returned when a data file is not a Bolt-formatted
database.
- ErrVersionMismatch is returned when the data file was created with a
different version of Bolt.
- ErrChecksum is returned when either meta page checksum does not match.
Also:
- Do not wrap errors from os.Stat, so that a caller could handle os.Stat
errors just like it can handle errors from os.Open that bolt.Open
might return.
- Name tests consistently, following the pattern "TestOpen_*".
- Remove deferred calls to `os.Remove(path)`.
The calls are not only unnecessary, but also in all cases `os.Remove`
returns an error that is ignored. All those calls are meant to remove
a file that was already removed by `tmpfile()`.
- Combine "bad path" tests and use filepath.Join to build the path.
This adds MmapFlags to DB.Options in case we need syscall.MAP_POPULATE
flag in Linux 2.6.23+ to do the sequential read-ahead, as discussed in [1].
---
[1]: https://github.com/coreos/etcd/issues/3786
Only grow the database size when the high watermark increases.
We also grows the database size a little bit aggressively to
save a few ftruncates.
I have tested this on various environments. The performance impact
is ignorable with 16MB over allocation. Without over allocation,
the performance might decrease 100% when each Tx.Commit needs a new
page on a very slow disk (seek time dominates the total write).
This commit adds documentation to clarify that read-only and read-write
transactions should not be mixed in the same goroutine as it can cause
deadlocks during remapping.
See: https://github.com/boltdb/bolt/issues/378
This commit adjusts the minimum mmap size from 1MB to 32KB. The
previous limit was arbitrary and causes wasted space for very small
databases.
Thanks to @mcuadros for submitting the original pull request:
https://github.com/boltdb/bolt/pull/351
This commit adds the DB.NoGrowSync flag to optionally revert mmap()
calls to how they were implemented before the ext3/ext4 fix. When
NoGrowSync is true, remapping the data file will not force the file
system to resize it immediately. This works for non-ext3/4 file
systems.
The default value of NoGrowSync is false so it is still safe for
ext3/ext4 file systems by default.
See also: https://github.com/boltdb/bolt/issues/284
DB.Batch makes it easy to make lots of small transactions with
significantly better performance. Batch combines multiple concurrent
Update calls into a single disk transaction, managing errors smartly.
This commit fixes a bug where only calling NextSequence() on a Bucket does not cause the Bucket to be
peristed. The simple fix is to simply materialize the root node so that the bucket is flushed out
during commit.
Thanks to Matthew Dawson (@MJDSys) for reporting.
https://github.com/boltdb/bolt/issues/296
This commit expands calls to _assert() that use variadic arguments. These calls require conversion to interface{} so there
was a large number of calls to Go's internal convT2E() function. In some profiling this was taking over 20% of total runtime.
I don't remember seeing this before Go 1.4 so perhaps something has changed.
This commit adds fixes suggested by @tv42 for the mmap step fix in 834b38e:
* Check max size before calculating the new the mmap size.
* Fix mmap step loop to go to 1GB instead of 512MB.
This commit fixes an issue where the database would grow whenever it was opened. This was caused by
a recent change that performed a truncation when the database grew. Now there are fixed growth sizes
for the database (1MB, 2MB, 4MB, 8MB, etc) up to 1GB and then the database will grow by 1GB when it
resizes.
See also: 6bb25854a1
The max mmap size was previous unchecked which resulted in a panic once
the maximum size was reached. This commit adds a check for the max size
when re-mapping and returns an error if the new map will exceed the size.
Thanks to Tamás Gulácsi for testing out the change on i386.
OpenBSD does not include a UBC kernel and writes must be synchronized
with the msync(2) syscall. In addition, the NoSync field of the DB
struct should be ignored on OpenBSD, since unlike other platforms,
missing msyncs will result in data corruption.
Depends on PR #258.
Fixes#257.
This commit moves the DB.FillPercent field to Bucket.FillPercent. This
allows the fill percentage to be specified per-bucket, per-tx. This
value is not persisted and should be set whenever using it.
This commit adds the DB.NoSync flag to skip fsync() calls on each commit. This should only
be used for bulk loading as it can corrupt your database in the event of a system failure.
Initial tests show it can provide a 2x speed up for sequential inserts.
This commit adds a defer handler to ensure that transactions are always closed out - even
in the event of a panic within user code. It's recommended that applications always fail
in the event of a panic but some packages such as net/http will automatically recover
which is a problem (IHMO).
This commit fixes an issue on Windows where the database was doubling
when it was re-opened. This occurred because Windows has to truncate the
file to the mmap size and the mmap resizing code was doubling the size
whenever the DB size was at the next threshold. This has been changed so
that the DB size will double only when the DB size is above the next
threshold.
This commit adds a cache to the freelist which combines the available free pages and pending free pages in
a single map. This was added to improve performance where freelist.isFree() was consuming 70% of CPU time
for large freelists.
This commit adds an explicit DefaultOptions variable for additional documentation.
Open() can still be passed a nil options which will cause options to be change to
the DefaultOptions variable. This change also allows options to be set globally for
an application if more than one database is being opened in a process.
This commit also moves all errors to errors.go so that the godoc groups them together.
This commit changes Open() to provide an additional Options argument. The options
argument currently only has a Timeout which will cause the Open() to return
ErrTimeout if a file lock cannot be obtained in time.
Fixes#207.
This commit reverts merge-split and fixes the node.split() to do a multi-page split. This issue
caused problems with bulk loading because it would split into a small page and a very large page.
The very large page, in turn, would be an arbitrary size so when it was freed later it would be
difficult to reuse and would cause serious fragmentation issues.
This commit adds Windows support to Bolt. Windows memory maps return an address instead of a byte
slice so the DB.data field had to be refactored to be a pointer to a large byte array.
This commit changes Tx.Check() to return a channel through which check errors are returned. This allows
errors to be found before checking the entire data file.
This commit removes the DB.Check() function and instead makes the user decide
whether a transaction should be writable or read-only. Tx.Check() is not safe
to use concurrently on a read-only transaction, however, it significantly
improves the performance of it.
This commit fixes the freelist so that it frees from the beginning of the data file
instead of the end. It also adds a fast path for pages which can be allocated from
the first free pages and it includes read transaction stats.
This commit adds the ability to adjust the fill percentage for splitting nodes. This
works by setting a threshold that is a percentage of a total page size. When that
threshold is crossed during a split then a new node is created.
This is primarily beneficial for append-only workloads.
Fixes#163.
This commit adds support for writing small buckets directly inline to their value in
their parent's leaf node. Previously, subbuckets would simply have a bucket header
stored in their parent bucket which pointed to the root page. This required that
every bucket use at least a single page. This has a high overhead for buckets with
only one or two small items.
Inline buckets checks subbuckets to see if they only have a small amount of data
(about 1kb) and no subbuckets. If these conditions are met then the bucket's root
node is written to a fake page which is simply a pointer to the end of the bucket's
header.
Fixes#124.
This commit consolidates some of the smaller files into some of the larger files.
The smaller files cluttered the file tree and made it harder to see the logical
groupings of structs.
Gyuho Lee
CJ DiMaggio
Rodrigo Coelho
Tommi Virtanen
Anthony Romano
lorneli
Raphael Geronimi
Tyler Treat
Sokolov Yura aka funny_falcon
Xiang Li
Josh Bleecher Snyder
Ben Johnson
Aleksa Sarai
Chris Hines
Alexander Morozov
Rodolfo Carvalho
sasha-s
funkygao
pmcgrath
Josh Rickmar
Martin Kobetic
Steven Normore