tdb2.tdbquery batch processing

[Laura Morales]

Is tdb2.tdbquery suitable for batch-processing hundreds of queries (sequential, not parallel)? Or should I just use Fuseki/HTTP?

[ajs6f]

Can you explain your use case a bit more?

Avoiding network traffic would certainly be a gain for performance, but starting and stopping hundreds of system processes would have its own costs. Much would depend on how you use the tools, so more information would be helpful, including some idea of the data and queries.

It is very hard to give general advice about performance (especially concurrent performance). It much easier in a specific concrete context.

What can be said is that certainly the CLI tools can be scripted for batch work.

ajs6f

Post by Laura Morales
Is tdb2.tdbquery suitable for batch-processing hundreds of queries (sequential, not parallel)? Or should I just use Fuseki/HTTP?

[Laura Morales]

I just have a script that runs a few 100s queries one after the other in sequence. I was only wondering if it would be practical to use tdb2.tdbquery to query the database directly by skipping fuseki entirely.

 
 

Sent: Saturday, November 10, 2018 at 10:34 PM
From: ajs6f <***@apache.org>
To: ***@jena.apache.org
Subject: Re: tdb2.tdbquery batch processing
Can you explain your use case a bit more?

Avoiding network traffic would certainly be a gain for performance, but starting and stopping hundreds of system processes would have its own costs. Much would depend on how you use the tools, so more information would be helpful, including some idea of the data and queries.

It is very hard to give general advice about performance (especially concurrent performance). It much easier in a specific concrete context.

What can be said is that certainly the CLI tools can be scripted for batch work.

ajs6f

Post by Laura Morales
Is tdb2.tdbquery suitable for batch-processing hundreds of queries (sequential, not parallel)? Or should I just use Fuseki/HTTP?

 

[ajs6f]

Yes, certainly.

Depending on all kinds of things, it may or may not be more performant, but it should certainly work.

ajs6f

Post by Laura Morales
I just have a script that runs a few 100s queries one after the other in sequence. I was only wondering if it would be practical to use tdb2.tdbquery to query the database directly by skipping fuseki entirely.
Sent: Saturday, November 10, 2018 at 10:34 PM
Subject: Re: tdb2.tdbquery batch processing
Can you explain your use case a bit more?
Avoiding network traffic would certainly be a gain for performance, but starting and stopping hundreds of system processes would have its own costs. Much would depend on how you use the tools, so more information would be helpful, including some idea of the data and queries.
It is very hard to give general advice about performance (especially concurrent performance). It much easier in a specific concrete context.
What can be said is that certainly the CLI tools can be scripted for batch work.
ajs6f

Post by Laura Morales
Is tdb2.tdbquery suitable for batch-processing hundreds of queries (sequential, not parallel)? Or should I just use Fuseki/HTTP?