Merge Triple Streams in Jena or Sesame?

A completely different approach is to simply concatenate the files and then run sort(1) on them. This assumes you don't care about comments and blank lines, and there is no leading whitespace before subjects but if the n-triples was produced by a program that's likely true.

Big advantage - you need to write a 2 line shell script to do it, not write-and-debug java code.

Bonus prize - strip out leading whitespace on subjects with sed(1).

I did an adapter for RIO using Guava's AbstractService that turned their result listener interface into an iterator using a blocking queue; pretty straightforward little piece of code to work out. You could do the same w/ Jena and then a sorted iterator over the two would be very trivial. I had the AbstractService handle starting the parsing and collecting the results, the only trick I recall was using using an object to poison the queue to signify the end of parsing so you don't end up waiting on a take.

Interesting question. As both files are already sorted its not so bad.

Take two queues one for each file. Take the head of both files. Compare the heads. Write the head which should go first. Take a new head Compare the heads Write the head which should go first. etc... till no heads/triples remain.

In sesame you can have two rio readers both in a thread each pushing statements into a BlockingQueue per thread. You have a third thread which does the comparison and the writing. That third thread needs to pulls the heads of the queues.

Make sense?

Jena does have a parser with a pull API for N-Triples and N-Quads [*]:

   public final class LangNTriples implements Iterator<Triple> { ... }
   // To use:
   LangNTriples parser = RiotReader.createParserNTriples(inputStream, null);

You may also want to look at org.openjena.atlas.data.SortedDataBag.SpillSortIterator which provides an implementation of a sort-merge iterator based on a PriorityQueue.

_{* For other RDF formats, you could try RiotTripleParsePuller, which was just added in SVN recently. This uses a thread and a BlockingQueue to create a pull API. Note this class is new and subject to change.}