RavenDB - The best .NET Document Database

Faceted Search

Mon, 12 Sep 2011 16:43:09 GMT

When displaying a large amount of data, often paging is used to make viewing the data manageable. However it's also useful to give some context of the entire data-set and a easy way to drill-down into particular categories. The common approach to doing this is "faceted search", as shown in the image below. Note how the count of each category within the current search is across the top.

To achieve this in RavenDB, lets say you have a document like this:

{
    DateOfListing: "2000-09-01T00:00:00.0000000+01:00"
    Manufacturer: "Jessops"
    Model: "blah"
    Cost: 717.502206059872
    Zoom: 9
    Megapixels: 10.4508949012733
    ImageStabiliser: false
}

Step 1: You need to setup your facet definitions and store them in RavenDB as a document, like so:

_facets = new List<Facet>
                        {
                            new Facet {Name = "Manufacturer"},
                            new Facet
                                {
                                    Name = "Cost_Range",
                                    Mode = FacetMode.Ranges,
                                    Ranges =
                                        {
                                            "[NULL TO Dx200.0]",
                                            "[Dx200.0 TO Dx400.0]",
                                            "[Dx400.0 TO Dx600.0]",
                                            "[Dx600.0 TO Dx800.0]",
                                            "[Dx800.0 TO NULL]",
                                        }
                                },
                            new Facet
                                {
                                    Name = "Megapixels_Range",
                                    Mode = FacetMode.Ranges,
                                    Ranges =
                                        {
                                            "[NULL TO Dx3.0]",
                                            "[Dx3.0 TO Dx7.0]",
                                            "[Dx7.0 TO Dx10.0]",
                                            "[Dx10.0 TO NULL]",
                                        }
                                }
                        };

session.Store(new FacetSetup { Id = "facets/CameraFacets", Facets = _facets });

This tells Raven that you would like to get the following facets

For the Manufacturer field look at the documents and return a count for each unique Term found
For the Cost field, return the count of the following ranges:
- Cost <= 200.0
- 200.0 <= Cost <= 400.0
- 400.0 <= Cost <= 600.0
- 600.0 <= Cost <= 800.0
- Cost >= 800.0

Step 2: Next you need to create an index to work against, this can be setup like so:

store.DatabaseCommands.PutIndex("CameraCost",
                            new IndexDefinition
                            {
                                Map = @"from camera in docs 
                                    select new 
                                    { 
                                        camera.Manufacturer, 
                                        camera.Model, 
                                        camera.Cost,
                                        camera.DateOfListing,
                                        camera.Megapixels
                                    }"
                            });

Step 3: Finally you can write the following code and you get back the data below.

var facetResults = s.Query<Camera>("CameraCost")
.Where(x => x.Cost >= 100 && x.Cost <= 300 )
.ToFacets("facets/CameraFacets");

This is equivalent to hitting the following Url:

http://localhost:8080/facets/CameraCost?facetDoc=facets/CameraFacets&query=Cost_Range:[Dx100 TO Dx300.0]

Note: The data returned represents the count of the faceted data that satisfies the query

Where(x => x.Cost >= 100 && x.Cost <= 300 )

Raven Backup & Restore

Thu, 11 Aug 2011 12:01:53 GMT

Like all databases, Raven supports backup and restore. Backups can be performed while the database in online and accepting requests (including writes).

A Raven database can be backed up using:

Using your existing enterprise backup solution.
Raven supports VSS backups, which is how most backup solutions on Windows work. You can do that by configuring your backup solution to take backups of Raven's data directory.
Using Raven's own backup / restore system. You can ask Raven to perform a complete backup of its data to a specified directory at any time. During the backup procedure, the database remains online and can respond to read and write requests. However, it is the state of the database at the start of the backup.

Unlike backups, restores are offline operation. You cannot restore to a running database (indeed, the notion makes little sense).

Backward compatibility

Raven relies on OS services to manage data storage and backup. Those services are forward compatible (if you backup on Windows XP you can restore on Windows 7) but not backward compatible (if you backup on Windows 2008 you cannot restore on Windows 2003).

How to initiate a backup

When running in embedded mode, all you need is to call the DocumentDatabase.StartBackup, when running in server mode, a POST request to /admin/backup will initiate a backup, you must provide the backup location:

> curl -X POST http://localhost:8080/admin/backup -d "{ 'BackupLocation': 'C:\\Backups\\2010-05-06' }"

Only administrators can initiate a backup using this approach.

Either method is asynchronous, the backup process will start, and the request will complete before the backup process is completed. You can check the status of the backup by querying the document with the key: "Raven/Backup/Status". The backup is completed when the IsRunning field in the document is set to false. The backup current status can be tracked by querying the backup status document, this includes any errors that occur during the backup process.

Only one backup may run at any given time.

If running from IIS, make sure to enable Windows Authentication for RavenDB's IIS application.

How to restore a database

Restoring a database is an offline operation, it cannot operate on a running instance of Raven. In embedded mode, you can restore using DocumentDatabase.Restore, or through the command line:

> Raven.Server.exe -src [backup location] -dest [restore location] -restore

If the restore location doesn't exists, Raven will create it.

You cannot restore to an existing database data directory, the restore operation will fail if it detects that the restore operation will overwrite existing data. If you need to restore to an existing database data directory, shutdown the database instance and delete the data directory.

Unlike backups, restores are fully synchronous.

Raven - Triggers

Mon, 01 Aug 2011 04:37:29 GMT

Raven allows you to use triggers to add custom behavior for the database. Those triggers are wired together using MEF, by default, Raven will search for triggers in a directory called Plugins under the application base directory.

Raven supports the following triggers:

While not precisely a trigger, the document codec does fall into the same category:

This allows you to control the actual byte storage format on the disk. This is useful if you want to encrypt the data on the disk.

Enabling debug logging for Raven

Mon, 01 Aug 2011 04:36:22 GMT

Raven has extensive support for debug logging, enabling you to figure out exactly what is going on in the server. By default, logging is turned off but you can enable it at any time by creating a file called "NLog.config" in Raven's base directory with the following content:

<nlog xmlns="http://www.nlog-project.org/schemas/NLog.netfx35.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <targets>
        <target 
            xsi:type="AsyncWrapper"
            name="AsyncLog">
            
        <target xsi:type="OutputDebugString">
        <layout xsi:type="SimpleLayout"/>
      </target>        
            
        </target>
        
    </targets>
    <rules>
        <logger name="Raven.*" writeTo="AsyncLog"/>
    </rules>
</nlog>

You can then use a tool such as dbgview to view the logs.

Recommended Deployment

Tue, 26 Jul 2011 07:46:53 GMT

Minimum requirements for RavenDB to run in embedded mode:

512 MB of RAM
Pentium III-class, 700MHz
150 MB of disk space
32 bits OS

Those are the absolute minimum amounts required for it to run. Please note that this is strictly for embedded mode, usually for dedicated devices such as Point of Sales systems where the amount of data and the number of queries are known in advance and the system has been configured specifically to allow that.

Minimum requirements for RavenDB to run in server mode:

2 GB of RAM
1 GHz Dual core
1 GB of disk space
64 bits OS

Recommended configuration for RavenDB in server mode:

4 GB of RAM (more is better)
2 GHz Quad core (more is better)
50 GB of disk space (more is better)

Note that splitting the db across multiple physical HDs is also recommended

64 bits OS

If you have any questions regarding deployment, please contact us at support@hibernatingrhinos.com

Support options

Sun, 24 Jul 2011 06:51:39 GMT

RavenDB has multiple avenues for support:

The RavenDB mailing list is available at: http://groups.google.com/group/ravendb/
Commercial support (available for users who purchased RavenDB commercially) can be obtained by contacting support@hibernatingrhinos.com

Batching requests

Wed, 01 Jun 2011 11:18:10 GMT

RavenDB supports batching multiple operations into a single request, reducing the number of remote calls and allowing several operations to share the same transactions.

Request batching in RavenDB is handled using the '/bulk_docs' endpoint, which accepts an array of operations to execute. The format for the operations is:

method - PUT, PATCH or DELETE.
key - the document key that this operation pertains to.
etag - optional - the etag to check against the current document etag.
document - the JSON document to PUT.
@metadata - the metadata associated with the document to PUT.

Below you can see an example of the the operation format:

[
    {
        Method: "PUT",
        Document:
        {
            name: "BatchPut1_Name"
        },
        Metadata:
        {
            info: "BatchPut1_Info"
        },
        Key: "BatchPut1"
    },
    {
        Method: "PUT",
        Document:
       {
           name: "BatchPut2_Name"
       },
       Metadata:
       {
           info: "BatchPut2_Info"
       },
        Key: "BatchPut2"
    },
    {
        Method: "DELETE",
        Key: "BatchPut1"
    },
    {
        Method: "DELETE",
        Key: "NonExistent"
    }
]

This can be executed using curl with the following syntax:

> curl http://localhost:8080/bulk_docs -X POST -d "[ { Method:'PUT', Document:{  name:'BatchPut1_Name' }, Metadata:{  info:'BatchPut1_Info' },Key:'BatchPut1' }, 
                                                   { Method:'PUT', Document:{  name:'BatchPut2_Name' }, Metadata:{  }, Key:'BatchPut2' } ]"


[
 {
 "Etag":"4c06db4e-4c86-11df-8ec2-001fd08ec235",
 "Method":"PUT",
 "Key":"BatchPut1"
 },
 {
 "Etag":"4c06db4f-4c86-11df-8ec2-001fd08ec235",
 "Method":"PUT",
 "Key":"BatchPut2"
 }
]

Concurrency

If an etag is specified in the command, that etag is compared to the current etag on the document on the server. If the etags do no match, a 409 Conlict status code is returned. In such a case, the entire operation fails and non of the updates that were tried will succeed.

Transactions

All the operations in the batch will succeed or fail as a transaction. Other users will not be able to see any of the changes until the entire batch completes.

Using projections

Thu, 12 May 2011 19:03:39 GMT

Usually, RavenDB can tell what types you want to return based on the query type and the CLR type encoded in a document, but there are some cases where you want to query on one thing, but the result is completely different. This is usually the case when you are using live projections.

For example, let us take a look at the following index:

public class PurchaseHistoryIndex : AbstractIndexCreationTask<Order, Order>
{
    public PurchaseHistoryIndex()
    {
        Map = orders => from order in orders
                        from product in order.Items
                        select new
                                   {
                                       UserId = order.UserId,
                                       ProductId = product.Id
                                   };
        TransformResults = (database, orders) =>
                           from order in orders
                           from item in order.Items
                           let product = database.Load<Product>(item.Id)
                           where product != null
                           select new
                                      {
                                          ProductId = item.Id,
                                          ProductName = product.Name
                                      };

    }
}

Note that when we query this index, we can query based on UserId or ProductId, but the result that we get back aren't of the same type that we query on. For that reason, we have the As<T>() extension method. We can use it to change the result type of the query:

documentSession.Query<Shipment, PurchaseHistoryIndex>()
   .Where(x => x.UserId == userId)
   .As<PurchaseHistoryViewItem>()
   .ToArray(),

In the code above, we query the PurchaseHistoryIndex using Shipment as the entity type to search on, but we get the results as PurchaseHistoryViewItem.

Raven DB – Hello World

Wed, 11 May 2011 05:57:04 GMT

This will guide you throughout building your first Raven DB application.

We start by heading to download page and getting the latest build.
Unzip the downloaded file.
Go to the /Server directory and double click on the Raven.Server.exe
- This will start the RavenDB server listening on localhost:8080
- If you already have an application listening on 8080, you can change the port RavenDB will listen to by editing the "Raven/Port" appSetting value in the Raven.Server.exe.config file.
- RavenDB will automatically create a database in the /Data directory

In Visual Studio, create a new console application project named: Raven.HelloWorld
Add a reference to the Raven.Client.Lightweight.dll from the /Client directory
- If you are using Visual Studio 2008, add a reference to Raven.Client-3.5.dll from /Client-3.5 directory

Initialize the client:
- DocumentStore should be created once per application and stored as a singleton.
```
var store = new DocumentStore {Url = "http://localhost:8080"};
store.Initialize();
```

We define an entity model that we can work with:

Note that all the entities are POCO
The only requirement is that a root entity string Id property (configurable)

public class Product
{
    public string Id { get; set; }
    public string Name { get; set; }
    public decimal Cost { get; set; }
}

public class Order
{
    public string Id { get; set; }
    public string Customer { get; set; }
    public IList<OrderLine> OrderLines { get; set; }

    public Order()
    {
        OrderLines = new List<OrderLine>();
    }
}

public class OrderLine
{
    public string ProductId { get; set; }
    public int Quantity { get; set; }
}

Let us save some data:
- The client API works using the session / data context model.
- The client API implements the Unit of Work pattern.
- Entity ids are only set when the SaveChanges method is called.
```
using (var session = store.OpenSession())
{
    var product = new Product
    {
        Cost = 3.99m,
        Name = "Milk",
    };
    session.Store(product);
    session.SaveChanges();

    session.Store(new Order
    {
        Customer = "customers/ayende",
        OrderLines =
                      {
                          new OrderLine
                          {
                              ProductId = product.Id,
                              Quantity = 3
                          },
                      }
    });
    session.SaveChanges();
}
```
Execute the code.
- By default, RavenDB only allow authenticate users to modify documents and attachments.
- The client API attempts to authenticate using the current user credentials.
- You can set the credentials that the client API will use globally (set the DocumentStore.Credentials method) or per session (use the OpenSession(ICredentials) overload).
Go to http://localhost:8080 in your browser.
Click on the "documents" tab, it should look something like this:

The first two documents are the ones that we created, the third is part of the default documents that Ravens come with, including Raven's own documentation.

Looking more closely at those documents, you can see that we have two documents:
- Note that order lines are embedded inside the orders document. You can read more on this design decision.
- Note the document keys: "products/1", "orders/1". Those are generated by Raven, the intend is to make them RESTful and human readable.

And now let us get those documents back:
```
using (var session = store.OpenSession())
{
    var order = session.Load<Order>("orders/1");
    Console.WriteLine("Customer: {0}", order.Customer);
    foreach (var orderLine in order.OrderLines)
    {
        Console.WriteLine("Product: {0} x {1}", orderLine.ProductId, orderLine.Quantity);
    }
    session.SaveChanges();
}
```
Execute the code, which gives us:
- If you'll look in Fiddler, you'll note that only one request was made, to get the entire object model.

Now, for querying, we need to query all order that contains a particular product.

Raven requires you to define an index before you can query it.
Think about an index as a stored procedure.
You should only create an index once.

store.DatabaseCommands.PutIndex("OrdersContainingProduct", new IndexDefinitionBuilder<Order>
{
    Map = orders => from order in orders
                    from line in order.OrderLines
                    select new { line.ProductId }
});

With the index in existence, we can now query it:
- The query syntax in Raven is based on the Lucene syntax.
- We call WaitForNonStaleResults because we just updated the database and Raven updates indexes on the background.
  - Normally you shouldn't call this method!
```
 using (var session = store.OpenSession())
 {
     var orders = session.Advanced.LuceneQuery<Order>("OrdersContainingProduct")
         .Where("ProductId:products/1")
         .WaitForNonStaleResults()
         .ToArray();
     foreach (var order in orders)
     {
         Console.WriteLine("Id: {0}", order.Id);
         Console.WriteLine("Customer: {0}", order.Customer);
         foreach (var orderLine in order.OrderLines)
         {
             Console.WriteLine("Product: {0} x {1}", orderLine.ProductId, orderLine.Quantity);
         }
     }
 }
```
Which gives this output:

This is all, we have seen how we can read, write and query Raven. We worked with the client API and seen how we can persist .NET entities into Raven.

Raven - Deploying to IIS

Mon, 18 Apr 2011 14:47:56 GMT

Raven can be run as an IIS application.

Installing Raven in IIS

Extract the distribution zip.
In IIS Manager, create a new Web Site and point its physical path to the "/Web" folder in the extracted folder.
Set the Application Pool to "ASP.Net v4.0" (or create a new Application Pool set to .NET 4.0 Integrated Pipeline)
Set port and host, if you need them.
Make sure that the user you set for the site has write access to the physical database location.

Setting up on IIS 7.5

Remove the WebDAVModule from the installed Modules in your server
If you wish to run with authentication
- Ensure that "Windows Authentication" is installed support to IIS 7.5 (by default it is not)
- Enable "Windows Authentication" for the RavenDB website.
- In the web.config file, set the app settings value "Raven/AnonymousAccess" to Get or None
If you with to run with no authentication
- In the web.config file, set the app settings value "Raven/AnonymousAccess" to All

Setting up on IIS 6

On IIS 6, you need to modify the Web.config and remove the system.webServer element and add the following system.web element:

       <system.web>
               <httpHandlers>
                       <add path="*" verb="*" 
                                type="Raven.Web.ForwardToRavenRespondersFactory, Raven.Web"/>
               </httpHandlers>
       </system.web>

Web Configuration

You can set the following configuration options in the Web.config appSettings:

Raven/DataDir - The physical location for the Raven data directory.
Raven/AnonymousAccess - What access rights anonymous users have. The default is Get (anonymous users can only read data). The other options are None and All.
Raven/Port - The port that Raven will listen to. The default is 8080.
Raven/VirtualDirectory - The virtual directory that Raven will listen to. The default is empty.
Raven/PluginsDirectory - The plugin directory for extending Raven. The default is a directory named "Plugins" under Raven base directory.
Raven/MaxPageSize - The maximum number of results a Raven query can return (overrides any page size set by the client). The default is 1024.

Recommended IIS Configuration

Raven isn't a typical web site because it needs to be running at all times. In IIS 7.5, you can set this using the following configuration settings:

If you created a dedicated application pool for Raven, change the application pool configuration in the application host file (C:\Windows\System32\inetsrv\config\applicationHost.config) to:
```
<add name="RavenApplicationPool" managedRuntimeVersion="v4.0" startMode="AlwaysRunning" />
```

If Raven runs in an application pool with other sites, modify the application host file (C:\Windows\System32\inetsrv\config\applicationHost.config) to:
```
<application path="/Raven" serviceAutoStartEnabled="true" />
```

Configuration options

Sun, 17 Apr 2011 17:10:39 GMT

All the configuration options detailed below are defined in the app.config's <appSettings> as separate values.

Core settings:

Raven/BackgroundTasksPriority
What thread priority to give the various background tasks RavenDB uses (mostly for indexing)
Allowed values: Lowest, BelowNormal, Normal, AboveNormal, Highest
Default: Normal
Raven/MaxPageSize
The maximum allowed page size for queries.
Default: 1024
Minimum: 10

Index settings:

Raven/MaxNumberOfParallelIndexTasks
The maximum number of indexing tasks allowed to run in parallel
Default: the number of processors in the current machine
Raven/MaxNumberOfItemsToIndexInSingleBatch
Max number of items to take for indexing in a batch
Default: 2500
Minimum: 128
Raven/TempIndexPromotionMinimumQueryCount
How many times a temporary, auto-generated index has to be accessed before it can be promoted to be a permanent one
Default: 100
Raven/TempIndexPromotionThreshold
Time (in milliseconds) the index has to be queried at least once in order for it to become permanent
Default: 60000 (once per minute)
Raven/TempIndexCleanupPeriod
How often to run the temporary index cleanup process (in seconds)
Default: 600 (10 minutes)
Raven/TempIndexCleanupThreshold
How much time in seconds to wait after a temporary index has been used before removing it if no further calls were made to it during that time
Default: 1200 (20 minutes)
Raven/TempIndexInMemoryMaxMB
Temp indexes are kept in memory until they reach this integer value in MB
Default: 25 MB
Minimum: 1 MB

Data settings:

Raven/RunInMemory
Should RavenDB's storage be in-memory. If set to true, Munin would be used as the storage engine, regardless of what was specified for StorageTypeName
Allowed values: true/false
Default: false
Raven/DataDir
The directory for the RavenDB database.
You can use the ~\ prefix to refer to RavenDB's base directory.
Default: ~\Data
Raven/StorageTypeName
What storage type to use (see: RavenDB Storage engines)
Allowed values: esent, munin (at this point of time only Esent is fully supported by RavenDB)
Default: esent
Raven/TransactionMode
What sort of transaction mode to use.
Allowed values:
Lazy - faster, but can result in data loss in the case of server crash.
Safe - slower, but will never lose data
Default: Safe

Http settings:

Raven/HostName
The hostname to use when creating the http listener (null to accept any hostname or address)
Default: none, binds to all host names
Raven/Port
The port to use when creating the http listener.
Default: 8080
Raven/VirtualDirectory
The virtual directory to use when creating the http listener.
Default: /
Raven/HttpCompression
Whether to use http compression or not.
Allowed values: true/false;
Default: true
Raven/AccessControlAllowOrigin
Determine the value of the Access-Control-Allow-Origin header sent by the server.
Allowed values: null (don't send the header), *, http://example.org
Raven/AnonymousUserAccessMode
Defines which operations are allowed for anonymous users.
Allowed values: All, Get, None
Default: Get

Misc settings:

Raven/PluginsDirectory
Where to look for plugins for RavenDB.
Default: ~\Plugins
Raven/WebDir
The directory to search for RavenDB's WebUI.
This is usually only useful if you are debugging RavenDB's WebUI.
Default: ~/Raven/WebUI

Esent settings:

Raven/Esent/CacheSizeMax
The maximum size of the in memory cache that is used by the storage engine. The value is in megabytes.
Default: 1024
Raven/Esent/MaxVerPages
The maximum size of version store (in memory modified data) available. The value is in megabytes.
Default: 128
Raven/Esent/DbExtensionSize
The size that the database file will be enlarged with when the file is full. The value is in megabytes.
Lower values result in smaller file size, but slower performance.
Default: 16
Raven/Esent/LogFileSize
The size of the database log file. The value is in megabytes.
Default: 16
Raven/Esent/LogBuffers
The size of the in memory buffer for transaction log.
Default: 16
Raven/Esent/MaxCursors
The maximum number of cursors allowed concurrently.
Default: 2048

How the indexes work

Thu, 14 Apr 2011 08:50:35 GMT

In order to allow fast queries over your indexes, RavenDB processes them in the background, executing the queries against the stored documents and persisting the results to a Lucene index. Lucene is a full text search engine library (Raven uses the .NET version) which allows us to perform lightning fast full text searches.

The best way of thinking about RavenDB's indexes is to imagine them as a database's materialized views. RavenDB executes the indexing processes in the background, and the results are written to disk. That means that when we are querying, we have to do very little work. This is how RavenDB manages to achieve its near instantaneous replies for your queries, it doesn't have to think, all the processing has already been done.

By using Lucene as the indexing format, we can support some really fancy querying types: Range based, partial string matching, full text searches, etc. You can read more about queries supported by Lucene here.

If you are using the Raven Client API, you can simply use the Linq provider that comes with it and it will deal with most of those issues for you. If you want to query RavenDB externally, or if you want to understand how Raven manages those indexes and take advantage of advanced Lucene features, please read on.

When RavenDB needs to store the results of your queries in the Lucene index, it analyzes each value, and produce the following results:

If the value is null - create a single field with the unanalyzed value 'NULL_VALUE'.
If the value is string - create a single field with the supplied name and the value.
If the value is set unanalyzed - create a single field with the value set to not analyzed.
If the value is date, create a single field with millisecond precision with the supplied name.
If the value is numeric (int, long, double, decimal, or float) will create two fields:
- The first will be created with the supplied name, containing the numeric value as an unanalyzed string. This is useful if you want to query the by the exact value.
- The second will be create with the name: '[name]_Range', containing the numeric value in a form that allows range queries.
- Sample, if we try to index 'Age', 18, we will have the following fields:
  - Age:18
  - Age_Range: [18 in a binary format that is applicable for range searching]

Using this format, it is pretty easy to perform both exact queries and range queries, including when you need to detect nulls.

Using custom analyzers

Lucene uses Analyzers to split up text into the tokens that are then stored in the index. Normally the default analyzers is okay, but Raven lets you specify which built-in Lucene analyzer to use, in the case when the default analyzer isn't suitable. You can control the analyzer per-field like so:

store.DatabaseCommands.PutIndex("Movies",
        new IndexDefinition
        {
            Map = "from movie in docs.Movies select new { movie.Name, movie.Tagline }",
            Analyzers =
                {
                    {"Name", typeof(SimpleAnalyzer).FullName},
                    {"Tagline", typeof(StopAnalyzer).FullName},                                    
                }
        });

The output of the built-in analyzers are shown below, they are all tokenizing the following text:

The quick brown fox jumped over the lazy dog, bob@hotmail.com 123432.

Keyword Analyzer - tokenizes the entire stream as a single token.

[The quick brown fox jumped over the lazy dog, bob@hotmail.com 123432.]

Whitespace Analyzer - tokenizes on white space only (note the punctuation at the end of "dog")

[The] [quick] [brown] [fox] [jumped] [over] [the] [lazy] [dog,] [bob@hotmail.com] [123432.]

Stop Analyzer - strips out common English words (such as "and", "at" etc), tokenizes letters only and converts everything to lower case

[quick] [brown] [fox] [jumped] [over] [lazy] [dog] [bob] [hotmail] [com]

Simple Analyzer - only tokenizes letters and makes all tokens lower case

[the] [quick] [brown] [fox] [jumped] [over] [the] [lazy] [dog] [bob] [hotmail] [com]

Standard Analyzer - simple tokenizer that uses a stop list of common English works, also handles numbers and emails addresses correctly

[quick] [brown] [fox] [jumped] [over] [lazy] [dog] [bob@hotmail.com] [123432]

A simple way to see how text is tokenized by the Lucene analyzers is to use the tool available here.

You can also create your own custom analyzer, compile it to a dll and drop it in in directory called "Analyzers" under the RavenDB base directory. Afterward, you can then use the fully qualified type name of your custom analyzer as the analyzer for a particular field.

Tutorials

Thu, 14 Apr 2011 08:40:35 GMT

Videos & Presentations

From Raven DB launch event - Fly, Raven, Fly!
A presentation in Melbourne Users Group Hello RavenDB
RavenDB presentation for the European Virtual Alt.NET - RavenDB

On site tutorials

External tutorials

Fitzchak's RavenDB in practice series:

Part 1: An introduction to RavenDB

By Rob Ashton:

The image gallery project, a series of blog posts describing the creation of an image gallery website step-by-step. Source code is here.
Raven DB - An Introduction (outdated)

Using RavenDB with ASP.NET MVC

Sample applications

Raven comes with the following sample applications:

Shard Client - Sample of using Raven's Sharding and Master/Master replication
Failover - Sample of Raven's transparent failover
Replication - Sample of Raven's Master/Slave replication, conflict detection and recovery
Event Sourcing - Sample of Raven's event sourcing and compiled indexes capabilities.

All those samples are included in the Raven zip file, which you can download here.

Denormalized References

Wed, 13 Apr 2011 08:04:20 GMT

RavenDB doesn't allow references between documents in the sense that an RDBMS user will understand them. Moreover, while RavenDB offers the Includes feature, document isolation is still an important design feature when you build you data model.

Consider the following case:

Order has a reference to the customer. However, the above is the logical model, which ignores the persistence concerns. The model we use to store the data in RavenDB is by necessity different. We can't hold a direct reference to the customer, because RavenDB will serialize the entire Customer with our Order, which isn't what we want.

On the other hand, holding just the document id (common in RDBMS systems) is usually going to be insufficient. A document should contain all the data that is required to perform all routine operations on it. In the case of the Order, that means displaying the Customer's name.

From our experiences, it seems that the vast majority of cases requires just holding a name/id pair as a denormalized reference for a document. There are cases where we need more, certainly, but the method outline here works for those as well.

In other words, we need to hold in the Order class what we call a denormalized reference to the customer. That reference includes the document id, but it also include any properties that are important for the Order. In this case, the name.

What properties should be included in the denormalized reference?

Properties which rarely change (such as name, email, etc).

Properties that are required for common processing of the referencing document.

Properties whose value needs to be separate than the current value in the owner document (Product.Cost is a good example where we want to copy the value to the OrderLine.Cost, so we can keep track if things have changed).

The following shows how we can do this easily using code. First, the model:

public interface INamedDocument
{
    string Id { get; set; }
    string Name { get; set; }
}

public class Customer : INamedDocument
{
    public string Id { get; set; }
    public string Name { get; set; }
}

public class Order : INamedDocument
{
    public string Id { get; set; }
    public string Name { get; set; }
    public DenormalizedReference<Customer> Customer { get;set;}
}

Note that the Customer property on the Order is a DenormalizedReference of T. This class looks like this:

public class DenormalizedReference<T> where T : INamedDocument
{
    public string Id { get; set; }
    public string Name { get; set; }
    
    public static implicit operator DenormalizedReference<T> (T doc)
    {
        return new DenormalizedReference<T>
        {
            Id = doc.Id,
            Name = doc.Name
        }
    }
}

This is pretty simple, except that we use something that you might not have seen before, implicit operators. This is just a C# language feature that tells the compiler that whenever it sees a expression of type T passed where the expected type is DenormalizedReference<T>, the implicit operator will be called to convert it.

This allows the following code:

Customer customer = GetCurrentCustomer();
var order = new Order
{
   Customer = customer // implicit operator called here
}

The result is a more natural approach for working with the denormalized reference.

Replication

Tue, 22 Mar 2011 12:11:48 GMT

Raven replication can be enabled by dropping the Raven.Bundles.Replication.dll to Raven's Plugins directory.

You can read about potential deployment options for the replication bundle here: Mixing Replication and Sharding.

The replication module will effect the following changes:

Track the server the document was originally written on. The replication bundle uses this information to determine if a replicated document is conflicting with the existing document.
Documents that encountered a conflict would be marked appropriately and will require automated or user involvement to resolve.
Document deletes result in delete markers, which the replication bundle needs in order to be able to replicate deletes to sibling instances. This is an implementation detail and is not noticeable to clients.
Several new endpoints will begin responding, including, but not limited to:
- /replication/replicate
- /replication/lastEtag
The replication bundle will create several system documents, including, but not limited to:
- Raven/Replication/Destinations - List of servers we need to replicate to
- Raven/Replication/Sources/[server] - Information about the data replicated from a particular server
The replication bundle will not replicate any system documents (whose key starts in Raven/)

Format of Raven/Replication/Destinations

The destination document format is:

{
"Destinations": [
    {
      "Url": "http://raven_two:8080/"
    },
    {
      "Url": "http://raven_three:8080/"
    },
]
}

With an object containing a url per each instance to replicate to. Whenever this document is updated, replication kicks off and start replicating to the updates destination list.

How replication works?

On every transaction commit, Raven will look up the list of replication destination. For each of the destination, the replication bundle will:

Query the remote instance for the last document that we replicated to that instance.
Start sending batches of updates that happened since the last replication.

Replication happens in the background and in parallel.

What about failures?

Because of the way it is designed, a node can fail for a long period of time, and still come up and start accepting everything that it missed in the meanwhile. The Replication Bundle keeps track of only a single data item per each replicating server, the last etag seen from that server. This means that we don't have to worry about missing replication windows.

When the replication bundle encountered a failure when replicating to a server, it has an intelligent error handling strategy. It is meant to be hands off, so nodes can fail and come back up without any administrative intervention. The strategy is outlined below:

If this is the first failure encountered, immediately try to replicate the same information again. This is done under the assumption that most failures are transient in nature.
If the second attempt fails as well, we note the failure (in Raven/Replication/Destinations/[server url])
After ten consecutive failures, Raven will start replicating to this node less often
- Once every 10 replication cycles, until failure count reaches 100
- Once every 100 replication cycles, until failure count reaches 1,000
- Once every 1,000 replication cycles, when failure count is above 1,000
Any successful replication will reset the failure count, on the assumption that

Can I bring up a new node and start replicating to it?

Yes, you can. You can edit the Raven/Replication/Destinations document in the replicating instance to add the new node, and the Replication Bundle will immediately start replicating to that server.

What about timeouts?

By default, the replication bundle have a timeout of 500 ms. You can control that by specifying "Raven/Replication/ReplicationRequestTimeout" in the configuration file <appSettings/> section.

The default value assumes servers that are near one another, for replication over the WAN, you would likely want to add additional time. Please note that timeout failures for the replication bundle counts as failures, and the replication bundle will reduce the number of replication attempts against a node that fails often.

What happen if there is a conflict?

In a replicating system, it is possible that two writes to the same document will occur on two different servers, resulting in two independent versions of the same document. When replication occur between these two version, the Replication Bundle is faced with a problem. It has two authentic versions of the same thing, saying different things. At that point, the Replication Bundle will mark that document as conflicting, store all the conflicting documents in a safe place and set the document content to point to the conflicting documents.

Resolving a conflict is easy, you just need to PUT a new version of the document. On PUT, the Replication Bundle will consider the conflict resolved.

More details about conflicts are here: Dealing with replication conflicts.

Client integration

Raven's Client API will detect and respond appropriately whenever a server has the replication bundle installed. This includes:

Detecting that an instance is replicating to another set of instances.
When that instance is down, will automatically shift to the other instances.

The Raven Client API is quite intelligent in this regard, upon failure, it will:

Assume that the failure is transient, and retry the request.
If the second attempt fails as well, we record the failure and shift to a replicated node, if available.
After ten consecutive failures, Raven will start replicating to this node less often
- Once every 10 requests, until failure count reaches 100
- Once every 100 requests, until failure count reaches 1,000
- Once every 1,000 requests, when failure count is above 1,000
On the first successful request, the failure count is reset.

If the second replicated node fails, the same logic applies to it as well, and we move to the third replicated node, and so on. If all nodes fail, an appropriate exception is thrown.

At a lower level, those are the operations that support replication:

Get - single document and multi documents
Put
Delete
Query
Rollback
Commit

The following operation do not support replication in the Client API:

PutIndex
DeleteIndex

Who is using RavenDB?

Tue, 22 Mar 2011 07:37:31 GMT

Case Study - The First RavenDB Deployment

Case Study: The First RavenDB Deployment

Tue, 22 Mar 2011 07:36:43 GMT

1. Who are you? (Name, company, position)

Henning Christiansen, Senior Consultant at Webstep Fokus, Norway (www.webstep.no) I am working on a development team for a client in the financial sector.

2. In what kind of project / environment did you deploy RavenDB?

The client is, naturally, very focused on delivering solutions and value to the business at a high pace. The solutions we build are both for internal and external end users. On top of the result oriented environment, there is a heavy focus on building sustainable and maintainable solutions. It is crucial that modules can be changed, removed or added in the future.

In this particular project, we built a solution based on NServiceBus for communication and RavenDB for persistence. This project is part of a larger development effort, and integrates with both old and new systems. The project sounds unimpressive when described in short, but I'll give it a go:

The project's main purpose was to replace an existing system for distribution of financial analysis reports. Analysts/researchers work on reports, and submit them to a proprietary system which adds additional content such as tables and graphs of relevant financial data, and generates the final report as XML and PDF. One of the systems created during this project is notified when a report is submitted, pulls it out of the proprietary system, tags it with relevant metadata and stores it in a RavenDB instance before notifying subscribing systems that a new report is available. The reports are instantly available on the client's website for customers with research access.

One of the subscribing systems is the distribution system which distributes the reports by email or sms depending on the customer's preferences. The customers have a very fine-grained control over their subscriptions, and can filter them on things such as sector, company, and report type among other things. The user preferences are stored in RavenDB. When a user changes preferences, notifications are given to other systems so that other actions can be performed based on what the customers are interested in.

3. What made you select a NoSQL solution for your project?

We knew the data would be read a lot more than it would be written, and it needed to be read fast. A lot of the team members were heavily battle-scarred from struggling with ORMs in the past, and with a very tight deadline we weren't very interested in spending a lot of time maintaining schemas and mappings.

Most of what we would store in this project could be considered a read model (à la CQRS) or an aggregate root (DDD), so a NoSQL solution seemed like a perfect fit. Getting rid of the impedance mismatch couldn't hurt either. We had a lot of reasons that nudged us in the direction of NoSQL, so if it hadn't been RavenDB it would have been something else.

4. What made you select RavenDB as the NoSQL solution?

It was the new kid on the block when the project started, and had a few very compelling features such as a native .NET API which is maintained and shipped with the database itself. Another thing was the transaction support. A few of us had played a bit with RavenDB, and compared to other NoSQL solutions it seemed like the most hassle-free way to get up and running in a .NET environment. We were of course worried about RavenDB being in an early development stage and without reference projects, so we had a plan B in case we should hit a roadblock with RavenDB. Plan B was never put into action.

5. How did you discover RavenDB?

I subscribe to your blog (http://ayende.com/Blog/) :)

There was a lot of fuss about NoSQL at the time, and RavenDB received numerous positive comments on Twitter and in blog posts.

6. How long did it take to learn to use RavenDB?

I assume you're asking about the basics, as there's a lot to be learnt if you want to. What the team struggled with the most with was indexes. This was before dynamic indexes, so we had to define every index up front, and make sure they were in the database before querying. Breaking free from the RDBMS mindset and wrapping our heads around how indexes work, map/reduce, and how and when to apply the different analyzers took some time, and the documentation was quite sparse back then. The good thing is that you don't really need to know a lot about this stuff to be able to use RavenDB on a basic level.

The team members were differently exposed to RavenDB, so guessing at how long it took to learn is hard. But in general I think it's fair to say that indexes was the team's biggest hurdle when learning how to use RavenDB.

7. What are you doing with RavenDB?

On this particular project we weren't using a lot of features, as we were learning to use RavenDB while racing to meet a deadline.

Aside from storing documents, we use custom and dynamic indexes, projections, the client API, and transactions. We're also doing some hand-rolled Lucene queries.

On newer projects however, with more experience and confidence with RavenDB, and as features and bundles keep on coming, we're doing our best to keep up with the development and making the best use of RavenDB's features to solve our problems.

8. What was the experience, compared to other technologies?

For one thing, getting up and running with RavenDB is super-easy and only takes a couple of minutes. This is very different from the RDBMS+ORM experience, which in comparison seems like a huge hassle. Working with an immature and rapidly changing domain model was also a lot easier, as we didn't need to maintain mappings. Also, since everything is a document, which in turn easily maps to an object, you're sort of forced to always work through your aggregate roots. This requires you to think through your domain model perhaps a bit more carefully than you'd do with other technologies which might easier allow you to take shortcuts, and thus compromise your domain model.

9. What do you consider to be RavenDB strengths?

It's fast, easy to get started with, and it has a growing community of helpful and enthusiastic users. Our support experience has also been excellent, any issues we've had have usually been fixed within hours. The native .NET API is also a huge benefit if you're working in a .NET environment

10. What do you consider to be RavenDB weaknesses?

If we're comparing apples to apples, I can't think of any weaknesses compared to the other NoSQL solutions out there aside from the fact that it's new. Hence it's not as heavily tested in production environments as some of the older NoSQL alternatives might be. The relatively limited documentation, which admittedly has improved tremendously over the last few months, was also a challenge. The community is very helpful, so anything you can't find in the documentation can normally be answered by someone on the forum. There's also a lot of blog posts and example applications out there.

I find the current web admin UI a bit lacking in functionality, but hopefully the new Silverlight UI will take care of that.

11. Now that you are in production, do you think that choosing RavenDB was the right choice?

Yes, definitely. We've had a few pains and issues along the way, but that's the price you have to pay for being an early adopter. They were all quickly sorted out, and now everything's been ticking along like clockwork for months. I'm confident that choosing RavenDB over another persistence technology has allowed us to develop faster and spend more time on the problem at hand.

12. What would you tell other developers who are evaluating RavenDB?

I have little experience with other document databases, but obviously tested a bit and read blog posts when evaluating NoSQL solutions for this project. Since we decided to go with RavenDB there's been a tremendous amount of development done, and at this time none of the competitors are even close featurewise.

Patching documents via the client API

Sun, 20 Mar 2011 22:33:30 GMT

RavenDB supports document patching, a process which is described in full here. This feature allows us to modify a document without having to get and put the entire document. This can save bandwidth, as well as reduce the potential for concurrency conflicts.

For example, let us consider the scenario of wanting to add a comment to a blog post. Using the standard API, we would use:

var post = session.Load<Post>("posts/123");
post.Comments.Add(comment);
session.SaveChanges();

This load and saves the entire document, as a single unit. But we can also just instruct RavenDB to just add the new comment to the post, using the following code:

session.Advanced.DatabaseCommands.Batch(
    new[]
    {
        new PatchCommandData
        {
            Key = "posts/123",
            Patches = new[]
            {
                new PatchRequest
                {
                    Type = PatchCommandType.Add,
                    Name = "Comments",
                    Value = JObject.FromObject(comment)
                }, 
            }
        }
    });

This will perform the work on the server side. Note that we can pass several patch requests, to make several modifications to the same document all at once.

Working with System.Transactions

Wed, 02 Mar 2011 17:55:02 GMT

fully supports System.Transactions as a durable resource manager. That means that you can write the following code and it will be fully transactional:

using(var tx = new TransactionScope())
{
   using(var session = documentStore.OpenSession())
   {
           session.Store(new User { Name = "Ayende" } );
           session.SaveChanges();
   }
   using(var session = documentStore.OpenSession())
   {
           session.Store(new User { Name = "Rahien" } );
           session.SaveChanges();
   }
   tx.Complete();
}

This code will either save both documents, or neither of them.

That said, the way that System.Transactions is implemented leads to some interesting issues. Let us examine this code:

using(var tx = new TransactionScope())
{
   using(var session = documentStore.OpenSession())
   {
           var user = session.Load<User>("users/1");
           user.Name = "Ayende"; // old name is "Oren"
           session.SaveChanges();
   }
   tx.Complete();
}

using(var session = documentStore.OpenSession())
{
    var user = session.Load<User>("users/1");
    Console.WriteLine(user.Name);
}

What would you expect this code to produce? Probably you would expect this to output "Ayende". But as a matter of fact, "Oren" (the old value) will be outputted.

That is not a bug, actually. It is an implication of how System.Transactions work. When you dispose a completed transaction scope, the transaction doesn't actually commit. What happens is that the transaction commit process is started. Since this is a background process running in another thread, you are actually going to load the users/1 document before the commit is over, which means that you are going to read the committed value of "Oren" (vs. the uncommitted value "Ayende").

It may not be a bug, since everything works as it is designed to work, but it sure isn't clear what is going on. And it looks like a bug. RavenDB detects this situation and will inform you that the document that you have loaded has been modified by an uncommitted transaction (using 203 Non Authoritative Information as the HTTP response code and "Non-Authoritative-Information" metadata property), so you can decide what to do about it.

In practice, for most read-only scenarios, we can use the non authoritative value (since we don't want to show uncommitted data, we will show the committed data, even if it is currently being modified). But when we want to update the document, we can't really do that, since the document is being locked by another transaction. If we'll try, we will get a Conflict Exception (409 Conflict on the HTTP response code).

You can ask RavenDB to wait until the pending transaction fully commits, by setting AllowNonAuthoritativeInformation to false, like this:

using(var tx = new TransactionScope())
{
   using(var session = documentStore.OpenSession())
   {
           var user = session.Load<User>("users/1");
           user.Name = "Ayende"; // old name is "Oren"
           session.SaveChanges();
   }
   tx.Complete();
}

using(var session = documentStore.OpenSession())
{
    session.Advanced.AllowNonAuthoritativeInformation = false;
    var user = session.Load<User>("users/1");
    Console.WriteLine(user.Name);
}

This code will print "Ayende".

If the transaction doesn't commit within 15 seconds, an exception is thrown. You can control the timeout by setting the NonAuthoritativeInformationTimeout property on the session.

Licensing

Wed, 23 Feb 2011 10:13:11 GMT

Hibernating Rhinos offers both Open Source and commercial editions of RavenDB.

Commercial editions can be used in closed source environment and are available under a subscription or perpetual pricing model. Prices are per instance. As long as the subscription is valid, new releases are included in it automatically.

What does this means?

You can use Raven for free, if your project is Open Source. If you want to use Raven in to build commercial software, you must buy a commercial license.

Pricing

	Cost	Support
Subscriptions	25 USD / instance / month	2 incidents / instance /year	Buy now!
	300 USD / instance / year	2 incidents / instance /year	Buy now!
One time payment	599 USD / instance	2 incidents / instance for the first 6 months. Additional support: 249 USD instance / year	Buy now!
OEM (embedded only) Allow royalty free distribution of RavenDB	999 USD / developer / year Renewal: 549 USD / developer / year	2 incidents / developer / year	Buy now!

If you are a startup company, open source project, or would like to use RavenDB in a non-commercial closed source project, please contact us to request free licensing.

Would I be forced to release my OSS projects using Raven under the AGPL?

No, Raven's AGPL license contains an explicit exception for OSS projects. You can release your project under any OSI approved license. Note, however, that you can't change Raven's own licensing. Users of your project would still need to comply with Raven's licensing.