What would be the best way for storing and quering RDF data (triples or quads) using HBase?


I've just found this:

Unfortunately, the .tar.gz has been removed from the utdallas.edu website.

asked 12 Apr '10, 20:19

castagna's gravatar image

accept rate: 27%

edited 14 Dec '10, 16:37


Storing is easy, isn't the complex part the querying especially joins?

(12 Apr '10, 22:08) Ian Davis Ian%20Davis's gravatar image

Changed the question to: "for storing and queryng" :-)

But, storing would be the first step and it would have the advantage of being able to easily use MapReduce over it. For querying I would look at Pig and how to use HBase as input/output source/destination for PigLatin scripts (see: http://www.semanticoverflow.com/questions/715/translating-sparql-algebra-into-piglatin-script).

(15 Apr '10, 06:27) castagna castagna's gravatar image

The thesis linked in this blog post describes some options for making a HBase based triplestore.

(29 Oct '12, 07:03) Jerven ♦ Jerven's gravatar image

I can't comment specifically on HBase, but I have implemented RDF storage for Cassandra which has a very similar BigTable-inspired data model.

You basically have two options in how to store RDF data in wide-column databases like HBase and Cassandra: the resource-centric approach and the statement-centric approach.

In the statement-oriented approach, each RDF statement corresponds to a row key (for instance, a UUID) and contains subject, predicate and object columns. In Cassandra, each of these would be supercolumns that would then contain subcolumns such as type and value, to differentiate between RDF literals, blank nodes and URIs. If you needed to support named graphs, each row could also have a context column that would contain a list of the named graphs that the statement was part of.

The above is a relatively simple mapping to implement but suffers from some problems, notably the fact that preventing the creation of duplicate statements (an important RDF semantic) means having to do a read before doing a write, which at least in Cassandra quickly becomes a performance bottleneck as writes are much faster than reads.

There are ways to work around this problem, in particular by using content-addressable statement identifiers (e.g. the SHA-1 of the canonicalized N-Triples representation of each statement) as the row keys, but this in turn introduces other trade-offs such as no longer being able to version statement data: every time statement data changes, the old row needs to be deleted and a new one inserted with the new row key.

In view of the previous considerations, the resource-oriented approach is generally a better natural fit for storing RDF data in wide-column databases. In this approach, each RDF subject/resource corresponds to a row key, and each RDF predicate/property corresponds to a column or supercolumn. Keyspaces can be used to represent RDF repositories, and column families can be used to represent named graphs.

The main trade-off with the resource-based approach is that some statement-oriented operations become more complex and/or slower, notably those counting the total number of statements or querying for predicate or object terms without specifying a subject. To support performant basic graph pattern matching, additional POS, OPS, etc. indices may need to be created and maintained.

See RDF::Cassandra, my Cassandra storage adapter for RDF.rb, for a more detailed example of a resource-centric mapping from the RDF data model to a wide-column data model.

permanent link

answered 25 Apr '10, 19:13

Arto%20Bendiken's gravatar image

Arto Bendiken
accept rate: 20%

edited 25 Apr '10, 19:21

Thank you for your detailed answer. In a "resource-oriented" approach, what do you do when you have multiple objects with the same property?

(25 Apr '10, 20:20) castagna castagna's gravatar image

I take it you mean multiple object values for a given predicate on a particular subject/resource?

With Cassandra this is relatively easy since predicates are represented by supercolumns, and each predicate value can be stored in a subcolumn of the supercolumn.

With HBase, I suspect you'd have to look at making use of the column's timestamp value (which can be any arbitrary integer, apparently) to differentiate between multiple object term values. This means that the resource-centric storage approach may be less appropriate for HBase than it is for Cassandra...

(25 Apr '10, 21:43) Arto Bendiken Arto%20Bendiken's gravatar image

Are you using this just for storage of RDF or are you actively querying RDF with this? My concern with Cassandra is the need to create my own indexes if I then want to be able to do real querying over the data

(07 Oct '10, 13:04) Rob Vesse ♦ Rob%20Vesse's gravatar image

FWIW, I'm currently working on mrlin, which at its core addresses the RDF-in-HBase question.

permanent link

answered 28 Oct '12, 16:15

Michael%20Hausenblas's gravatar image

Michael Haus...
accept rate: 15%

Given that HBase is designed for row based storage of data the simplest thing would be to have a GSPO layout i.e.

Graph | Subject | Predicate | Object

In terms of actually reading and writing data you'll most likely need to look at whatever API you want to use to manipulate the RDF and implement the necessary interfaces/classes that allows your chosen API to get data in and out of HBase.

As for querying then you'll need to implement a SPARQL engine for your database which could be rather complex or use an API that does SPARQL in-memory but this will have the overhead of needing to read out some/all of the data from HBase first.

Personally I don't know enough about HBase to say whether such an approach is viable or sensible, if you have hardware on which you can install and run HBase then you would most likely be better off installing and running a Triple Store for your RDF data.

permanent link

answered 13 Apr '10, 09:41

Rob%20Vesse's gravatar image

Rob Vesse ♦
accept rate: 29%

Hey Arto or Vesse, Why not use Cassandra/HBase as a secondary storage mechanism and the Jena GraphMemFaster as a Write-Through Cache? In the Cassandra case you have to use twice as much memory but it would solve your performance problems. Alternatively you could implement a Hexastore in Cassandra if you can afford the memory costs.

permanent link

answered 26 Aug '10, 07:08

Alexi's gravatar image

accept rate: 0%

edited 26 Aug '10, 07:23

Such solutions don't scale to datasets larger than memory, nor beyond a single node, negating much of the point of using Cassandra or HBase. And, as you say, it would use at least twice the memory even for use cases where you could fit it all on a single node. As I outlined above, the resource-centric approach is simpler in many ways, at least for Cassandra.

(27 Aug '10, 00:51) Arto Bendiken Arto%20Bendiken's gravatar image

You're making the assumption here that everyone uses Jena. Main issue with widetable nosql stores is that you have to do value indexing yourself which means they may be very good for just/reading RDF but they aren't so good for querying

(07 Oct '10, 13:09) Rob Vesse ♦ Rob%20Vesse's gravatar image

I just discovered this paper discussing Rya. It uses Accumulo rather than HBase, and there is no source code available, but it does seem to address the querying issue, at least for the simpler semantic relationships such as rdfs:subclassOf.

permanent link

answered 28 Dec '12, 17:01

Kevin%20Pauli's gravatar image

Kevin Pauli
accept rate: 50%

Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here



Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:


question asked: 12 Apr '10, 20:19

question was seen: 66,017 times

last updated: 28 Dec '12, 17:01