The main goal of this library is to provide fast and strong database to Dart developers.
This repository contains fork of Tarantool Database (currently 2.8.4 version, will be updated to 2.11 after release).
Currently, there are a lot of local database solutions for Dart (Isaar, Hive, ObjectBox) and a lot of connectors to commonly used databases (such as Redis, Postgres, Mongo, etc.) as well.
But also there is a need of predictable and controllable data storage solution with ability of data processing logic customization.
Dart Tarantool storage should satisfy these needs by combining Tarantool features and Dart language features.
- Introduction
- Idea and concepts
- Installation & Usage
- API
- Storage
- [async]
boot()
- Initializing and bootstraping Tarantool and binding library components mutable()
- Checking that Tarantool is not read onlyinitialized()
- Checking that Tarantool is read only- [async]
awaitInitialized()
- Waiting for Tarantool initialization - [async]
awaitImmutable()
- Waiting for Tarantool is available for reading - [async]
awaitMutable()
- Waiting for Tarantool is available for writing shutdown()
- Shutdowning Tarantool and bindingclose()
- Closing Dart portsloadModuleByPath()
- Loading Native module by full pathloadModuleByName()
- Loading Native module by namereload()
- Reloading Lua and Native modulesexecutor
- Provider for StorageExecutor
- [async]
- Executor - StorageExecutor
- [async]
next()
- Getting next element of the iterator - [async]
destroyIterator()
- Destroying Tarnatool iterator - [async]
begin()
- Tarnatool transaction begin - [async]
commit()
- Tarnatool transaction commit - [async]
rollback()
- Tarnatool transaction rollback - [async]
transactional()
- Executing function inside Tarantool transaction - [async]
hasTransaction()
- Checking transaction status
- [async]
- Schema - StorageSchema
- [async]
spaceById()
- Creating StorageSpace instance for space with id - [async]
spaceByName()
- Creating StorageSpace instance for space with name - [async]
spaceId()
- Getting space id by its name - [async]
spaceExists()
- Checking that space exists inside Tarantool - [async]
indexByName()
- Creating StorageIndex instance for space and index names - [async]
indexExists()
- Checking that index exists by space id and index name - [async]
indexById()
- Creating StorageIndex instance for space and index ids - [async]
indexId()
- Getting index id by its name and space id - [async]
createSpace()
- DDL operation for creating Tarantool space - [async]
alterSpace()
- DDL operation for altering Tarantool space - [async]
renameSpace()
- DDL operation for renaming Tarantool space - [async]
dropSpace()
- DDL operation for dropping Tarantool space - [async]
createIndex()
- DDL operation for creating Tarantool index - [async]
alterIndex()
- DDL operation for altering Tarantool index - [async]
dropIndex()
DDL operation for dropping Tarantool space - [async]
createUser()
- DDL operation for creating Tarantool user - [async]
dropUser()
- DDL operation for dropping Tarantool user - [async]
changePassword()
- Changing Tarnatool user password - [async]
grantUser()
- DDL operation for granting Tarantool user permissions see - [async]
revokeUser()
- DDL operation for revoking Tarantool user permissions see
- [async]
- Space - StorageSpace
- [async]
count()
- Counting tuples in the space - [async]
isEmpty()
- Checking that space is empty - [async]
isNotEmpty()
- Checking that space is not empty - [async]
length()
- Getting space length (faster than count) - [async]
iterator()
- Creating iterator for tuples (see pairs) - [async]
insert()
- Inserting tuple into the space - [async]
put()
- Putting tuple into the space (ignore existence) - [async]
get()
- Getting tuple from the space by key - [async]
delete()
- Deleting tuple from the space - [async]
min()
- Getting minimal tuple from the space - [async]
max()
- Getting maximal tuple from the space - [async]
truncate()
- Truncating space - [async]
update()
- Updating space tuple by key with operations - [async]
upsert()
- Updating with operations or inserting tuple in the space - [async]
select()
- Selecting tuples from the space - [async]
batch()
- Batch operations provider
- [async]
- Index - StorageIndex
- [async]
count()
- Counting tuples in the space - [async]
iterator()
- Creating iterator for tuples (see pairs) - [async]
delete()
- Deleting tuple from the space by index - [async]
min()
- Getting minimal tuple from the space - [async]
max()
- Getting maximal tuple from the space - [async]
update()
- Updating space tuple by key with operations - [async]
select()
- Selecting tuples from the space - [async]
batch()
- Batch operations provider
- [async]
- Iterator - StorageIterator
- Batch
- StorageBatchSpaceBuilder
insert()
- Adding insert operation to batchput()
- Adding put operation to batchdelete()
- Adding delete operation to batchupdate()
- Adding update operation to batchupsert()
- Adding update operation to batchinsertMany()
- Adding insert operations to batchputMany()
- Adding put operations to batchdeleteMany()
- Adding delete operations to batch
- StorageBatchIndexBuilder
- StorageBatchSpaceBuilder
- Lua - StorageLuaExecutor
- [async]
startBackup()
- Executing box.backup.start() see - [async]
stopBackup()
- Executing box.backup.stop() see - [async]
promote()
- Executing box.ctl.promote() see - [async]
configure()
- Executing box.cfg{} see - [async]
script()
- Evaluating Lua script - [async]
file()
- Evaluating Lua file - [async]
require()
- Calling Lua 'require' function - [async]
call()
- Calling Lua function
- [async]
- Native - StorageNativeExecutor
- Storage
- Configuration
- Perfomance
- Limitations
- Further work
- Contribution
- schemed or schemaless per space
- fast
- in-memory or disk engines per space
- asynchronous or synchronous replication modes with raft and automatic leader election
- fluent and predictable data processing: you write your data processing code in procedural style
- transactional
- (after 2.10) MVCC and interactive (stream) transactions - currently not supported in this library
- sharding - currently not supported in this library
Tarantool is used as shared library (.so) and running in separate single thread.
Ring buffer between Tarantool and Dart code is used to transport messages from Dart to Tarantool.
After execution of message Tarantool thread will notify DartVM with Dart_Post.
Message structure: {type,function,input,output,batch[{function,input,output,error}],error}
.
type
- type of messagefunction
- pointer to binding function which should be called in Tarantool thread and has access to Tarantool APIinput
- binding function argumentoutput
- holder for function resulterror
- holder for Tarantool error which could happen during function callingbatch
- array of structures similar to message (for bulk execution of functions)
- call - calling function on Tarantool thread with access to Tarantool API
- batch - mark that it is batch message and binding should handle batch processing
- begin - Tarantool transaction begin
- commit - Tarantool transaction commit
- rollback - Tarantool transaction rollback
- stop - used only inside native binding code, stops the binding message loop
- management request - operations for manage Tarantool, for example initialize, shutdown, etc.
- space request - data operations for space, usually contains space id
- index request - data operations for index, usually contains space id and index id
- iterator next request - get next element of iterator
- execution request - execute Lua or Native function on Tarantool thread with access to Tarantool API
- Create Dart project with pubspec.yaml
- Add this section to dependencies:
tarantool_storage:
git:
url: https://github.com/antonbashir/dart-tarantool-storage/
path: binding/dart
- Run
dart pub get
- Run
dart run tarantool_storage:setup
- Look at the API and Enjoy!
You can find simple example here
To initialize and boot library use Storage.boot()
.
You can provide bootstrap lua script, change configuration and also provide initial user which will be used for replication.
There are Lua files in lua
directory.
They are loading from root file storage.lua
.
You can write custom code in the end of that file or in module.lua
file.
All of Tarantool Lua APIs are available for usage.
For execution of custom Lua functions you can use StorageExecutor.lua
.
There are native header files in native
directory.
You can compile them using dart run tarantool_storage:compile
.
You can write custom definitions in module.h
file and create module.c
for implementations.
All of Tarantool Native APIs are available for usage in your functions.
For execution of custom Native functions you can use StorageExecutor.native
.
If activateReloader
in Storage.boot
function is specified, Tarantool will reload Native and Lua modules on receiving SIGHUP signal.
So you can change Lua scripts or Native files (and recompile them) and your changes will be applied after SIGHUP.
If you want to distribute your module, run dart run tarantool_storage:pack ${path to main dart file}
.
This command will recompile Native files and create ${directory name}.tar.gz
archive with executables, libraries and lua scripts.
After this you can transfer archive to whatever place you want, unarchive it and run module.exe
.
StorageBootstrapScript script
- Initial Lua script representationStorageMessageLoopConfiguration loop
- See- [optional]
StorageBootConfiguration boot
- See - [optional]
bool activateReloader
- Activates Tarantool reloading
- [return]
bool
- True if Tarantool is available for writing
- [return]
bool
- True if Tarantool is available for reading
String libraryPath
- Module path- [return]
StorageNativeModule
- Loaded module instance
String libraryName
- Module name- [return]
StorageNativeModule
- Loaded module instance
StorageIterator iterator
- Input iteratorint count
- Prefetch count. Binding can prefetch more tuples than 1 and combine it after return- [return]
List<List<dynamic>>?
- Tuples or null if iterator reached the end
StorageIterator iterator
- Input iterator
Function(StorageExecutor executor) function
- Function which should be executed in transaction
- [return]
bool
- True if inside transaction
int id
- Input space id- [return]
StorageSpace
- Created instance
String name
- Input space name- [return]
StorageSpace
- Created instance
String space
- Input space name- [return]
int
- Received space id
String space
- Input space name- [return]
bool
- True if space exists
String spaceName
- Input space nameString indexName
- Input index name- [return]
StorageIndex
- Created instance
int spaceId
- Input space idString indexName
- Input index name- [return]
bool
- True if exists
int spaceId
- Input space idint indexId
- Input index id- [return]
StorageIndex
- Created instance
int spaceId
- Input space idString index
- Input index name- [return]
int
- Received index id
String name
- Space name- [optional]
StorageEngine engine
- Space engine (memtx or vinyl) - [optional]
int fieldCount
- Space tuple fields count - [optional]
List<StorageSpaceField> format
- Space tuple format - [optional]
int id
- Space id - [optional]
bool ifNotExists
- If true then Tarantool will ignore space existence and update it - [optional]
bool local
- If true then space will not participate in replication - [optional]
bool synchronous
- If true then space operations will require synchronous commit - [optional]
bool temporary
- If true then space will be empty on every server restart - [optional]
String user
- Space owner
String name
- Space name- [optional]
int fieldCount
- Space tuple fields count - [optional]
List<StorageSpaceField> format
- Space tuple format - [optional]
bool synchronous
- If true then space operations will require synchronous commit - [optional]
bool temporary
- If true then space will be empty on every server restart - [optional]
String user
- Space owner
String from
- Current space nameString to
- Space name after rename
String name
- Space name
String spaceName
- Space name (owner of creating index)String indexName
- Index name- [optional]
StorageIndexType type
- Index type (could be hash, set, tree, bitset, rtree) - [optional]
int id
- Index id - [optional]
bool unique
- If true then index should not contains duplicates - [optional]
bool ifNotExists
- If true then Tarantool will ignore index existence and update it - [optional]
List<StorageIndexPart> parts
- Index parts. Every part includes field number and field type
String spaceName
- Space name (owner of altering index)String indexName
- Index name- [optional]
List<StorageIndexPart> parts
- Index parts. Every part includes field number and field type
String spaceName
- Space name (owner of dropping index)String indexName
- Index name
String name
- User nameString password
- User password- [optional]
bool ifNotExists
- If true then Tarantool will ignore user existence and update it
String name
- User name
-
String name
- User name -
String password
- New user password -
[async]
userExists()
- Checking that Tarantool user exists -
String name
- User name -
[return]
bool
- True if exists
[async] grantUser()
- DDL operation for granting Tarantool user permissions see
String name
- User nameString privileges
- User privileges- [optional]
String objectType
- Permission object type - [optional]
String objectName
- Permission object name - [optional]
String roleName
- User role - [optional]
bool ifNotExists
- If true then Tarantool will ignore grant existence
[async] revokeUser()
- DDL operation for revoking Tarantool user permissions see
String name
- User nameString privileges
- User privileges- [optional]
String objectType
- Permission object type - [optional]
String objectName
- Permission object name - [optional]
String roleName
- User role - [optional]
bool ifNotExists
- If true then Tarantool will ignore grant existence
- [optional]
List<dynamic> key
- Include in count only matched tuples by key - [optional]
StorageIteratorType iteratorType
- Tarantool iterator type see - [return]
int
- Count of tuples
- [return]
bool
- True if empty
- [return]
bool
- True if not empty
- [return]
int
- Space length
[async] iterator()
- Creating iterator for tuples (see pairs)
- [optional]
List<dynamic> key
- Include only matched tuples by key - [optional]
StorageIteratorType iteratorType
- Tarantool iterator type see - [return]
StorageIterator
- Created iterator instance
List<dynamic> data
- Tuple- [return]
List<dynamic>
- Inserted tuple
List<dynamic> data
- Tuple- [return]
List<dynamic>
- Putted tuple
List<dynamic> key
- Key to find tuple- [return]
List<dynamic>
- Received tuple
List<dynamic> key
- Key to find tuple for deletion- [return]
List<dynamic>
- Deleted tuple
- [optional]
List<dynamic> key
- Key to find tuple - [return]
List<dynamic>
- Received tuple
- [optional]
List<dynamic> key
- Key to find tuple - [return]
List<dynamic>
- Received tuple
List<dynamic> key
- Tuple keyList<StorageUpdateOperation> operations
- Update oeprations. See- [return]
List<dynamic>
- Updated tuple
List<dynamic> tuple
- TupleList<StorageUpdateOperation> operations
- Update oeprations. See- [return]
List<dynamic>
- Updated or inserted tuple
- [optional]
List<dynamic> key
- Selecting tuples only matched with key - [optional]
int offset
- Offset for selection - [optional]
int limit
- Limit for selection - [optional]
StorageIteratorType iteratorType
- Tarantool iterator type see - [return]
List<dynamic>
- Selected tuples
Function(StorageBatchSpaceBuilder builder) builder
- Batch operations builder- [return]
List<dynamic>
- Result tuple after batch execution
- [optional]
List<dynamic> key
- Include in count only matched tuples by key - [optional]
StorageIteratorType iteratorType
- Tarantool iterator type see - [return]
int
- Count of tuples
[async] iterator()
- Creating iterator for tuples (see pairs)
- [optional]
List<dynamic> key
- Include only matched tuples by key - [optional]
StorageIteratorType iteratorType
- Tarantool iterator type see - [return]
StorageIterator
- Created iterator instance
List<dynamic> key
- Key to find tuple for deletion- [return]
List<dynamic>
- Deleted tuple
- [optional]
List<dynamic> key
- Key to find tuple - [return]
List<dynamic>
- Received tuple
- [optional]
List<dynamic> key
- Key to find tuple - [return]
List<dynamic>
- Received tuple
List<dynamic> key
- Tuple keyList<StorageUpdateOperation> operations
- Update oeprations. See- [return]
List<dynamic>
- [optional]
List<dynamic
- Selecting tuples only matched with key - [optional]
int offset
- Offset for selection - [optional]
int limit
- Limit for selection - [optional]
StorageIteratorType iteratorType
- Tarantool iterator type see - [return]
List<dynamic>
- Selected tuples
Function(StorageBatchIndexBuilder builder) builder
- Batch operations builder- [return]
List<dynamic>
- Result tuple after batch execution
- [optional]
count
- Prefetch count. Binding can prefetch more tuples than 1 and combine it after return - [return]
List<List<dynamic>>?
- Tuples or null if iterator reached the end
- [optional]
bool Function(List<dynamic> value) filter
- Iterator filter - [optional]
dynamic Function(List<dynamic> value) map
- Iterator mapper - [optional]
int limit
- Iterator limit - [optional]
int offset
- Iterator offset - [optional]
int count
- Prefetch count. Binding can prefetch more tuples than 1 and combine it after return - [return]
List<dynamic>
- Collected tuples
void Function(dynamic element) action
- Action on tuple- [optional]
bool Function(List<dynamic> value) filter
- Iterator filter - [optional]
dynamic Function(List<dynamic> value) map
- Iterator mapper - [optional]
int limit
- Iterator limit - [optional]
int offset
- Iterator offset - [optional]
int count
- Prefetch count. Binding can prefetch more tuples than 1 and combine it after return
- [optional]
bool Function(List<dynamic> value) filter
- Iterator filter - [optional]
dynamic Function(List<dynamic> value) map
- Iterator mapper - [optional]
int limit
- Iterator limit - [optional]
int offset
- Iterator offset - [optional]
int count
- Prefetch count. Binding can prefetch more tuples than 1 and combine it after return - [return]
Stream<dynamic>
- Stream of tuples
List<dynamic> data
- Inserting tuple
List<dynamic> data
- Putting tupple
List<dynamic> data
- Deleting tuple
List<dynamic> key
- Tuple keyList<StorageUpdateOperation> operations
- Update oeprations. See
List<dynamic> tuple
- Tuple to insert or updateList<StorageUpdateOperation> operations
- Update oeprations. See
List<List<dynamic>> data
- Inserted tuples
List<List<dynamic>> data
- Putted tuples
List<List<dynamic>> data
- Deleted tuples
List<dynamic> key
- Tuple keyList<StorageUpdateOperation> operations
- Update oeprations. See
[async] startBackup()
- Executing box.backup.start() see
[async] stopBackup()
- Executing box.backup.stop() see
[async] promote()
- Executing box.ctl.promote() see
[async] configure()
- Executing box.cfg{} see
StorageConfiguration configuration
- Tarantool Configuration
String expression
- Script- [optional]
List<dynamic> arguments
- Script arguments - [return]
List<dynamic>
- Script result
File file
- Lua script file
String module
- Requiring module
String function
- Lua function- [optional]
List<dynamic> arguments
- Lua function arguments - [return]
List<dynamic>
- Lua function result
tarantool_function function
- Pointer to Native function- [optional]
tarantool_function_argument argument
- Pointer or Native value as function argument - [return]
Pointer<Void>
- Pointer to function result
This class fully identical to Tarantool configuration
int boxOutputBufferCapacity
- Internal buffer initial capacity for some returing tuplesdouble messageLoopMaxSleepSeconds
- Max count of seconds for binding message loop idle sleeping timeint messageLoopRingSize
- Internal ring buffer size for all messages between Dart and Tarantooldouble messageLoopRegularSleepSeconds
- Count of seconds for binding message loop idle sleeping time before count of empty cycles reached maximumint messageLoopMaxEmptyCycles
- Maximum of empty cyclesint messageLoopEmptyCyclesMultiplier
- Multiplier for empty cyclesint messageLoopInitialEmptyCycles
- Initial count for empty cyclesint messageLoopRingRetryMaxCount
- Maximum retry count during tries of returing message into queue (used in concurrent transactions)
String user
- Initial Tarantool user nameString password
- Initial Tarantool user passwordDuration delay
- Delay after boot (could be usable for delay between replica bootstraps)
String host
- Replica hostString port
- Replica port- [optional]
String user
- Replication user (should exists on all replicas) - [optional]
String password
- Replication user password
int port
- Replica port
String uri
- Replica uri
- [return]
String
- Lua table
TODO: Make benchmark on preferable machine
Latest benchmark results (count of entities - 1M, single Dart Isolate):
- Get RPS: 200k
- Lua function RPS: 200k
- Select time: 250 milliseconds
- Iterator time (with 1k prefetch count): 1 second
- Batch insert: 2.7 seconds
- Linux only
- Currently, Tarantool VShard is not supported, but you still can use it by writing lua code to
module.lua
- Currently, Tarantool MVCC is not tested, but you can enable it by configuration
- Currently, tested only on x86 architecture, ARM and others are not tested but could work
- Not production tested, current version is coded and tested by function unit tests, bugs are possible
- Process restart required to restart Tarantool, because it can't be fully shutdown programmatically and some stuff stays in memory
- Full size of library static build is 70mb which could be critical for embedded or mobile devices
- Benchmarks and optimization
- CI for building library and way to provide it to user modules (currently library included with sources, that is not good)
- Dart network transport based on io_uring
- Flutter UI for management and administration
- Upgrade to Tarantool 2.11
- Demo project
Currently maintainer hasn't resources on maintain pull requests but issues are welcome.
Every issue will be observed, discussed and applied or closed if this project does not need it.