] {Build
Status] {Dependency
Status]
]
/]
]
Moneta provides a standard interface for interacting with various kinds of key/value stores. Moneta supports the well-known NoSQL and document based stores.
A short overview of the features:
Supports a lot of backends with consistent behaviour (See below)
Allows a full configuration of the serialization -> compression -> adapter stack using proxies (Similar to Rack middlewares)
Configurable serialization via Moneta::Transformer
proxy
(Marshal/JSON/YAML and many more)
Configurable value compression via Moneta::Transformer
proxy
(Zlib, Snappy, LZMA, …)
Configurable key transformation via Moneta::Transformer
proxy
Expiration for all stores (Added via proxy Moneta::Expires
if
not supported natively)
Atomic operations
Atomic incrementation and decrementation for most stores (Method
#increment
and #decrement
)
Atomic creation of entries (Method #create
)
Shared/distributed database-wide synchronization primitives
Moneta::Mutex
and Moneta::Semaphore
Includes a simple pure-ruby key/value server (Moneta::Server
)
and client (Moneta::Adapters::Client
)
If you are not yet convinced, you might ask why? What are the goals of the project?
Get people started quickly with key/value stores! Therefore all the adapters are included in the gem and you are ready to go. Tilt does the same for template languages.
Make it easy to compare different key/value stores and benchmark them
To hide a lot of different and maybe complex APIs behind one well-designed and simple Moneta API
Give people a starting point or example code to start working with their favourite key/value store. Feel free to copy code, please mention Moneta then :)
Create a reusable piece of code, since similar things are solved over and over again (Rails brings its own cache stores, and many frameworks do the same…)
Moneta is tested thoroughly using Travis-CI.
Install Moneta via Rubygems
~~~ $ gem install moneta ~~~
or add it to your Gemfile
~~~ ruby gem 'moneta' ~~~
Now you are ready to go:
~~~ ruby require 'moneta'
store = Moneta.new(:File, dir: 'moneta')
store = 'value'
store.key?('key') # returns true store # returns 'value'
store.close ~~~
Source: github.com/minad/moneta
Tests and benchmarks: travis-ci.org/minad/moneta
API documentation:
Latest Gem: rubydoc.info/gems/moneta/frames
GitHub master: rubydoc.info/github/minad/moneta/master/frames
Out of the box, it supports the following backends. Use the backend name
symbol in the Moneta constructor (e.g.
Moneta.new(:Memory)
).
Memory:
In-memory store (:Memory
)
LRU hash - prefer this over :Memory! (:LRUHash
)
LocalMemCache
(:LocalMemCache
)
Memcached store
(:Memcached
, :MemcachedNative
and
:MemcachedDalli
)
Relational Databases:
DataMapper (:DataMapper
)
ActiveRecord
(:ActiveRecord
)
Sequel (:Sequel
)
Sqlite3 (:Sqlite
)
Filesystem:
Key/value databases:
Berkeley
DB using DBM interface or NDBM (Depends on Ruby environment)
(:DBM
)
Cassandra
(:Cassandra
)
Daybreak
(:Daybreak
)
GDBM
(:GDBM
)
HBase (:HBase
)
LevelDB
(:LevelDB
)
LMDB (:LMDB
)
Redis (:Redis
)
Riak (:Riak
)
SDBM
(:SDBM
)
KyotoCabinet
(:KyotoCabinet
)
TokyoCabinet
(:TokyoCabinet
)
TokyoTyrant
(:TokyoTyrant
)
Simple Samba database TDB
(:TDB
)
Document databases:
Moneta network protocols:
Other
Fog cloud storage which supports Amazon S3,
Rackspace, etc. (:Fog
)
Storage which doesn't store anything (:Null
)
Some of the backends are not exactly based on key/value stores, e.g. the relational ones. These are useful if you already use the corresponding backend in your application. You get a key/value store for free then without installing any additional services and you still have the possibility to upgrade to a real key/value store.
NOTE: <a name=“backend-matrix”></a>The backend matrix is much more readable on rubydoc.info than on github. Go there!
Adapter | Required gems | Multi-thread safe[1] | Multi-process safe[2] | Atomic increment[8] | Atomic create[9] | Native expires[3] | Persistent | Description |
---|---|---|---|---|---|---|---|---|
Persistent stores | ||||||||
Mongo | mongo or moped | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | MongoDB database |
MongoOfficial | mongo | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | MongoDB database |
MongoMoped | moped | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | MongoDB database |
Redis | redis | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | Redis database |
ActiveRecord | activerecord | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ActiveRecord ORM |
File | - | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | File store |
LMDB | lmdb | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | Symas Lightning Memory-Mapped Database (LMDB) |
Sequel | sequel | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | Sequel ORM |
TokyoTyrant | tokyotyrant or ruby-tokyotyrant | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ | TokyoTyrant database |
PStore | - | ✗ | ✓[10] | ✓ | ✓ | ✗ | ✓ | PStore store |
YAML | - | ✗ | ✓[10] | ✓ | ✓ | ✗ | ✓ | YAML store |
Sqlite | sqlite3 | ? | ✓[10] | ✓ | ✓ | ✗ | ✓ | Sqlite3 database |
Daybreak | daybreak | ✗ | (✓)[7] | ✓ | ✓ | ✗ | ✓ | Incredibly fast pure-ruby key/value store Daybreak |
DBM | - | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ | Berkeley DB using DBM interface or NDBM (Depends on Ruby environment) |
GDBM | - | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ | GDBM database |
LevelDB | leveldb | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ | LevelDB database |
SDBM | - | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ | SDBM database |
TDB | tdb | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ | TDB database |
KyotoCabinet | kyotocabinet | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ | KyotoCabinet database |
TokyoCabinet | tokyocabinet | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ | TokyoCabinet database |
DataMapper | dm-core, dm-migrations | ✓ | ✓ | ✗ | ✓ | ✗ | ✓ | DataMapper ORM |
Couch | faraday, multi_json | ✗ | ✓ | ✗ | ✓ | ✗ | ✓ | CouchDB database |
HBase | hbaserb | ? | ✓ | ✓ | ✗ | ✗ | ✓ | HBase database |
Cassandra | cassandra | ? | ✓ | ✗ | ✗ | ✓ | ✓ | Cassandra distributed database |
LocalMemCache | localmemcache | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ | LocalMemCache database |
Fog | fog | ? | ✓ | ✗ | ✗ | ✗ | ✓ | Fog cloud store |
Riak | riak-client | ✗ | ✓ | ✗ | ✗ | ✗ | ✓ | Riak database |
Non persistent stores | ||||||||
MemcachedDalli | dalli | ✓ | ✓ | ✓ | ✓ | ✓ | ✗[4] | Memcached database with Dalli library |
Memcached | dalli or memcached | ? | ✓ | ✓ | ✓ | ✓ | ✗[4] | Memcached database |
MemcachedNative | memcached | ✗ | ✓ | ✓ | ✓ | ✓ | ✗[4] | Memcached database with native library |
Cookie | - | ✗ | (✓)[6] | ✓ | ✓ | ✓ | ✗ | Cookie in memory store |
LRUHash | - | ✗ | (✓)[6] | ✓ | ✓ | ✗ | ✗ | LRU memory store |
Memory | - | ✗ | (✓)[6] | ✓ | ✓ | ✗ | ✗ | Memory store |
Null | - | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | No database |
Client | - | ✗ | ✓ | ?[5] | ?[5] | ?[5] | ?[5] | Moneta client adapter |
RestClient | faraday | ✗ | ✓ | ✗ | ✗ | ✗ | ?[5] | Moneta REST client adapter |
[1]: Make adapters thread-safe by using Moneta::Lock
or by
passing the option threadsafe: true
to
Moneta#new
. There is also Moneta::Pool
which can
be used to share a store between multiple threads if the store is
multi-process safe. I recommend to add the option :threadsafe
to ensure thread-safety since for example under JRuby and Rubinius even the
basic datastructures are not thread safe due to the lack of a global
interpreter lock (GIL). This differs from MRI where some adapters might
appear thread safe already but only due to the GIL.
[2]: Share a Moneta store between multiple
processes using Moneta::Shared
(See below).
[3]: Add expiration support by using Moneta::Expires
or by
passing the option expires: true
to Moneta#new
.
[4]: There are some servers which use the memcached protocol but which are persistent (e.g. MemcacheDB, Kai, IronCache, Roma, Flare and Kumofs)
[5]: Depends on server
[6]: Store is multi-process safe because it is an in-memory store, values are not shared between multiple processes
[7]: Store is multi-process safe, but not synchronized automatically between multiple processes
[8]: If a store provides atomic increment it can be used with
Moneta::Semaphore
. You can add weak #increment
support using the Moneta::WeakIncrement
proxy.
[9]: If a store provides atomic creation it can be used with
Moneta::Mutex
. You can add weak #create
support
using the Moneta::WeakCreate
proxy.
[10]: Sqlite/YAML/PStore are multiprocess safe, but the performance suffers badly since the whole database file must be locked for writing. Use a key/value server if you want multiprocess concurrency!
In addition it supports proxies (Similar to Rack middlewares) which add additional features to storage backends:
Moneta::Proxy
and Moneta::Wrapper
proxy base
classes
Moneta::Expires
to add expiration support to stores which
don't support it natively. Add it in the builder using use
:Expires
.
Moneta::Stack
to stack multiple stores (Read returns result
from first where the key is found, writes go to all stores). Add it in the
builder using use(:Stack) {}
.
Moneta::Transformer
transforms keys and values (Marshal, YAML,
JSON, Base64, MD5, …). Add it in the builder using use
:Transformer
.
Moneta::Cache
combine two stores, one as backend and one as
cache (e.g. Moneta::Adapters::File
+
Moneta::Adapters::LRUHash
). Add it in the builder using
use(:Cache) {}
.
Moneta::Lock
to make store thread safe. Add it in the builder
using use :Lock
.
Moneta::Pool
to create a pool of stores as a means of making
the store thread safe. Add it in the builder using use(:Pool)
{}
.
Moneta::Logger
to log database accesses. Add it in the builder
using use :Logger
.
Moneta::Shared
to share a store between multiple processes.
Add it in the builder using use(:Shared) {}
.
Moneta::WeakIncrement
and Moneta::WeakCreate
to
add #create
and #increment
support without
atomicity (weak) to stores which don't support it.
Moneta::Transformer
)¶ ↑Supported serializers:
BEncode (:bencode
)
BERT (:bert
)
BSON (:bson
)
JSON (:json
)
Marshal (:marshal
)
MessagePack (:msgpack
)
Ox (:ox
)
PHP (:php
)
TNetStrings (:tnet
)
YAML (:yaml
)
Supported value compressors:
Bzip2 (:bzip2
)
LZ4 (:lz4
)
LZMA (:lzma
)
LZO (:lzo
)
Snappy (:snappy
)
QuickLZ (:quicklz
)
Zlib (:zlib
)
Supported encoders:
Base64 (:base64
)
Url escape (:escape
)
Hexadecimal (:hex
)
QP (:qp
)
UUEncode (:uuencode
)
Special transformers:
Digests (MD5, Shas, CityHash, …)
Add prefix to keys (:prefix
)
HMAC to verify values (:hmac
, useful for
Rack::MonetaCookies
)
The Moneta API is purposely extremely similar to the Hash API with a few minor additions. Every method takes also a optional option hash. In order so support an identical API across stores, Moneta does not support iteration or partial matches.
~~~
block and return its return value.
keys set using []= will never expire.
which is not supported by all stores. Returns current value.
which is not supported by all stores. Returns current value. This is just syntactic sugar for incrementing with a negative value.
Returns true if the value was created.
~~~
There is a simple interface to create a store using
Moneta.new
. You will
get automatic key and value serialization
which is provided by Moneta::Transformer
.
This allows you to
store arbitrary Ruby objects. You can tune some options
when you call
Moneta.new
. However for very fine tuning use
Moneta.build
.
~~~ ruby store = Moneta.new(:Memcached, server: 'localhost:11211') store = 'value' store = {a: 1, b: 2} store = MarshallableRubyObject.new ~~~
If you want to have control over the proxies, you have to use
Moneta.build
:
~~~ ruby store = Moneta.build do # Adds expires proxy use :Expires
# Transform key using Marshal and Base64 and value using Marshal use :Transformer, key: [:marshal, :base64], value: :marshal
# IMPORTANT: adapter must be defined last for the builder to function properly.
# Memory backend adapter :Memory end ~~~
You can also directly access the underlying adapters if you don't want to use the Moneta stack.
~~~ ruby db = Moneta::Adapters::File.new(dir: 'directory') db = {a: 1, b: 2} # This will fail since you can only store Strings
db = Moneta::Adapters::Couch.new db = {a: 1, b: 2}
db = Moneta::Adapters::Mongo.new db = {a: 1, b: 2} ~~~
The Cassandra, Memcached, Redis and Mongo backends support expiration natively.
~~~ ruby cache = Moneta::Adapters::Memcached.new
cache = Moneta.build do adapter :Memcached end
cache.store(key, value, expires: 60)
cache.store(key, value, expires: 0) cache.store(key, value, expires: false)
cache.load(key, expires: 30) cache.key?(key, expires: 30)
cache.load(key, expires: false) cache.key?(key, expires: 0) ~~~
You can add the expires feature to other backends using the
Moneta::Expires
proxy. But be aware
that expired values are
not deleted automatically if they are not looked up.
~~~ ruby
cache = Moneta.new(:File, dir: '…', expires: true)
cache = Moneta::Expires.new(Moneta::Adapters::File.new(dir: '…'))
cache = Moneta.build do use :Expires adapter :File, dir: '...' end ~~~
The stores support the #increment
which allows atomic
increments of unsigned integer values. If you increment
a non existing
value, it will be created. If you increment a non integer value an
exception will be raised.
~~~ ruby store.increment('counter') # returns 1, counter created store.increment('counter') # returns 2 store.increment('counter', -1) # returns 1 store.increment('counter', 13) # returns 14 store.increment('counter', 0) # returns 14 store.decrement('counter') # returns 13 store = 'Moneta' store.increment('name') # raises an Exception ~~~
If you want to access the counter value you have to use raw access to the
datastore. This is only important
if you have a
Moneta::Transformer
somewhere in your proxy stack which
transforms the values e.g. with Marshal
.
~~~ ruby store.increment('counter') # returns 1, counter created store.load('counter', raw: true) # returns 1
store.store('counter', '10', raw: true) store.increment('counter') # returns 11 ~~~
Fortunately there is a nicer way to do this using some syntactic sugar!
~~~ ruby store.increment('counter') # returns 1, counter created store.raw # returns 1 store.raw.load('counter') # returns 1
store.raw = '10' store.increment('counter') # returns 11 ~~~
You can also keep the raw
store in a variable and use it like
this:
~~~ ruby counters = store.raw
counters.increment('counter') # returns 1, counter created counters # returns 1 counters.load('counter') # returns 1
counters = '10' counters.increment('counter') # returns 11 ~~~
The stores support the #create
which allows atomic creation of
entries. #create
returns true
if the value was created.
~~~ ruby store.create('key', 'value') # returns true store.create('key', 'other value') # returns false ~~~
Moneta provides shared/distributed synchronization primitives which are shared database-wide between all clients.
Moneta::Mutex
allows a single thread to enter a critical
section.
~~~ ruby mutex = Moneta::Mutex.new(store, 'mutex_key')
mutex.synchronize do mutex.locked? # returns true
# Synchronized access to counter store += 1 end
begin mutex.lock mutex.locked? # returns true # ... ensure mutex.unlock end ~~~
Moneta::Semaphore
allows max_concurrent
threads
to enter a critical section.
~~~ ruby semaphore = Moneta::Semaphore.new(store, 'semaphore_counter', max_concurrent)
semaphore.synchronize do semaphore.locked? # returns true # ... end
begin semaphore.enter semaphore.locked? # returns true # ... ensure semaphore.leave end ~~~
If an underlying adapter doesn't provide atomic #create
or
#increment
and #decrement
you can
use the proxies
Moneta::WeakIncrement
and Moneta::WeakCreate
to
add support without atomicity.
But then you have to ensure that the store is not shared by multiple
processes and thread-safety is
provided by Moneta::Lock
.
For raw data access as described before the class
Moneta::OptionMerger
is used. It works like this:
~~~ ruby
store.with(raw: true).load('key')
store.with(raw: true, only: :load).load('key') store.with(raw: true, except: [:key?, :increment]).load('key')
store.raw.load('key')
substore = store.prefix('sub') substore = 'value' store # returns nil store # returns 'value'
short_lived_store = store.expires(60) short_lived_store = 'value' ~~~
You can add proxies to an existing store. This is useful if you want to compress only a few values for example.
~~~ ruby compressed_store = store.with(prefix: 'compressed') do use :Transformer, value: :zlib end
store = 'this value will not be compressed' compressed_store = 'value will be compressed' ~~~
Inspired by redis-store there exist integration classes for Rails and Rack. You can also use all the Rack middlewares together with Rails and the Sinatra framework. There exist the following integration classes:
Rack::Session::Moneta
is a Rack
middleware to use Moneta for storing sessions
Rack::MonetaStore
is a Rack middleware
which places a Moneta store in the environment
and enables per-request caching
Rack::MonetaCookies
is a Rack
middleware which uses Moneta to store cookies
Rack::MonetaRest
is a Rack application
which exposes a Moneta store via REST/HTTP
Rack::Cache::Moneta
provides meta and entity stores for
Rack-Cache
Ramaze::Cache::Moneta
is integrated into the Ramaze project
and allows Ramaze to use
Moneta as caching store
Padrino adopted Moneta to replace their cache stores in padrino-cache.
You can use Moneta as a Rack session store. Use it in your
config.ru
like this:
~~~ ruby require 'rack/session/moneta'
use Rack::Session::Moneta, store: :Redis
use Rack::Session::Moneta, store: Moneta.new(:Memory, expires: true)
use Rack::Session::Moneta, key: 'rack.session', domain: 'foo.com', path: '/', expire_after: 2592000, store: Moneta.new(:Memory, expires: true)
use Rack::Session::Moneta do use :Expires adapter :Memory end ~~~
There is a simple middleware which places a Moneta store in the Rack
environment at env['rack.moneta_store']
. It supports
per-request
caching if you add the option cache: true
. Use it
in your config.ru
like this:
~~~ ruby
use Rack::MonetaStore, :Memory, cache: true
run lambda { |env| env # is a Moneta store with per-request caching }
use Rack::MonetaStore do use :Transformer, value: :zlib adapter :Cookie end
run lambda { |env| env # is a Moneta store without caching } ~~~
If you want to expose your Moneta key/value store
via HTTP, you can use the Rack/Moneta REST service. Use it in your
config.ru
like this:
~~~ ruby require 'rack/moneta_rest'
map '/moneta' do run Rack::MonetaRest.new(:Memory) end
run Rack::MonetaRest.new do use :Transformer, value: :zlib adapter :Memory end ~~~
You can use Moneta as a Rack-Cache store. Use it
in your config.ru
like this:
~~~ ruby require 'rack/cache/moneta'
use Rack::Cache, metastore: 'moneta://Memory?expires=true', entitystore: 'moneta://Memory?expires=true'
Rack::Cache::Moneta = Moneta.build do use :Expires adapter :Memory end use Rack::Cache, metastore: 'moneta://named_metastore', entity_store: 'moneta://named_entitystore' ~~~
Use Moneta to store cookies in Rack. It uses the
Moneta::Adapters::Cookie
. You might
wonder what the purpose of
this store or Rack middleware is: It makes it
possible
to use all the transformers on the cookies (e.g.
:prefix
, :marshal
and :hmac
for
value verification).
~~~ ruby require 'rack/moneta_cookies'
use Rack::MonetaCookies, domain: 'example.com', path: '/path' run lambda { |env| req = Rack::Request.new(env) req.cookies #=> is now a Moneta store! env #=> is now a Moneta store! req.cookies #=> retrieves 'key' req.cookies = 'value' #=> sets 'key' req.cookies.delete('key') #=> removes 'key' [200, {}, []] } ~~~
Add the session store in your application configuration
config/environments/*.rb
.
~~~ ruby require 'moneta'
config.cache_store :moneta_store, store: :Memory
config.cache_store :moneta_store, store: Moneta.new(:Memory)
config.cache_store :moneta_store, store: Moneta.build do use :Expires adapter :Memory end ~~~
Add the cache store in your application configuration
config/environments/*.rb
. Unfortunately the
Moneta cache store doesn't support matchers. If
you need these features use a different server-specific implementation.
~~~ ruby require 'moneta'
config.cache_store :moneta_store, store: :Memory
config.cache_store :moneta_store, store: Moneta.new(:Memory)
config.cache_store :moneta_store, store: Moneta.build do use :Expires adapter :Memory end ~~~
Padrino adopted Moneta to replace their cache stores in padrino-cache. You use it like this
~~~ ruby
Padrino.cache = Moneta.new(:Memory, expires: true)
set :cache, Moneta.new(:Memory, expires: true) ~~~
You can use Moneta to build your own key/value
server which is shared between
multiple processes. If you run the following
code in two different processes,
they will share the same data which will
also be persistet in the database shared.db
.
~~~ ruby require 'moneta'
store = Moneta.build do use :Transformer, key: :marshal, value: :marshal use :Shared do use :Cache do cache do adapter :LRUHash end backend do adapter :GDBM, file: 'shared.db' end end end end ~~~
If you want to go further, you might want to take a look at
Moneta::Server
and Moneta::Adapters::Client
which
are used by Moneta::Shared
and provide the networking
communication. But be aware that they are experimental
and subjected to
change. They provide an acceptable performance (for being ruby only), but
don't have a stable protocol yet.
You might wonder why I didn't use DRb to implement server and client - in fact my first versions used it, but with much worse performance and it was real fun to implement the networking directly :) There is still much room for improvement and experiments, try EventMachine, try Kgio, ...
If you want something more advanced to handle your objects and
relations,
use John Nunemaker's ToyStore which
works
together with Moneta. Assuming that
Person
is a ToyStore::Object
you can
add
persistence using Moneta as follows:
~~~ ruby
Person.adapter :memory, Moneta.new(:Redis) ~~~
Testing is done using Travis-CI. Currently we support Ruby >= 1.9.3.
Benchmarks for each store are done on Travis-CI for each build. Take a look there to compare the speed of the different key value stores for different key/value sizes and size distributions. Feel free to add your own configurations! The impact of Moneta should be minimal since it is only a thin layer on top of the different stores.
Always feel free to open an issue on github.com/minad/moneta/issues if something doesn't work as you expect it to work. Feedback is also very welcome!
My only request about patches is that you please try to test them before submitting.
If you want support for another adapter you can at first at it to the list of missing adapters at github.com/minad/moneta/issues/16
If you choose to implement an adapter please also add tests. Usually
you
only have to add a few lines to script/generate-specs
to
generate appropriate
tests for your adapter. Please check also if
travis.yml needs changes, for example
if you need to start additional
services.
Check if the default settings in Moneta#new are appropriate for your adapter. If not specify a better one.
Don't forget to edit the README.md and the CHANGES.
Horcrux: Used at github, supports batch operations but only Memcached backend
ActiveSupport::Cache::Store: The Rails cache store abstraction
ToyStore: ORM mapper for key/value stores
ToyStore Adapter: Adapter to key/value stores used by ToyStore, Moneta can be used directly with the ToyStore Memory adapter
Cache: Rubygem cache wraps Memcached and Redis
Ramaze::Cache: Cache stores of the Ramaze framework with support for LocalMemCache, Memcached, Sequel, Redis, …
Originally by Yehuda Katz and contributors