Queries as functions

What are the kinds of usages to which you are imagining these kind of types being put?

---
A. Soroka
The University of Virginia Library

Post by Martynas JuseviÄius
Hey,
has Jena considered taking advantage of the functional features in Java 8?
Construct extends Query implements Function<Dataset, Model>
Describe extends Query implements Function<Dataset, Model>
Select extends Query implements Function<Dataset, ResultSet>
Ask extends Query implements Function<Dataset, ResultSet>
Martynas
atomgraph.com

Martynas Jusevičius

2016-12-12 21:45:51 UTC

Well, this probably requires some generic method(s) in Dataset/Model
as well, something like:

T apply(Function<Dataset, T> f);

This would allow nice chaining of multiple queries, e.g DESCRIBE and SELECT:

ResultSet results = dataset.apply(describe).apply(select);

Seems more elegant to me than all the QueryExecution boilerplate.

Post by A. Soroka
What are the kinds of usages to which you are imagining these kind of types being put?
---
A. Soroka
The University of Virginia Library

Rob Vesse

2016-12-13 09:48:42 UTC

Well why don’t you put together and implementation and get it a try?

Jena has always supported multiple APIs e.g. Model and Graph for achieving the same goals. Contributions of alternative APIs would always be welcome

Rob

On 12/12/2016 21:45, "Martynas Jusevičius" <***@graphity.org> wrote:

Well, this probably requires some generic method(s) in Dataset/Model
as well, something like:

T apply(Function<Dataset, T> f);

This would allow nice chaining of multiple queries, e.g DESCRIBE and SELECT:

ResultSet results = dataset.apply(describe).apply(select);

Seems more elegant to me than all the QueryExecution boilerplate.

Post by A. Soroka
What are the kinds of usages to which you are imagining these kind of types being put?
---
A. Soroka
The University of Virginia Library

Martynas Jusevičius

2016-12-13 16:48:33 UTC

I will try some time during holidays.

Will a PR on https://github.com/apache/jena/tree/master/jena-core work for this?

Post by Rob Vesse
Well why don’t you put together and implementation and get it a try?
Jena has always supported multiple APIs e.g. Model and Graph for achieving the same goals. Contributions of alternative APIs would always be welcome
Rob
Well, this probably requires some generic method(s) in Dataset/Model
T apply(Function<Dataset, T> f);
ResultSet results = dataset.apply(describe).apply(select);
Seems more elegant to me than all the QueryExecution boilerplate.

Post by A. Soroka
What are the kinds of usages to which you are imagining these kind of types being put?
---
A. Soroka
The University of Virginia Library

A. Soroka

2016-12-13 17:16:07 UTC

No, you will have to work with ARQ (jena-arq). Some of the types under discussion are to be found there. Anything generally to do with SPARQL per se is from ARQ.

---
A. Soroka
The University of Virginia Library

Post by Martynas JuseviÄius
I will try some time during holidays.
Will a PR on https://github.com/apache/jena/tree/master/jena-core work for this?

Post by A. Soroka
What are the kinds of usages to which you are imagining these kind of types being put?
---
A. Soroka
The University of Virginia Library

Rob Vesse

2016-12-14 09:54:29 UTC

Yes pull requests on Git Hub are fine

This will probably need to rely upon ARQ because that is where the Dataset API lives. It might be useful initially to have it be a separate module under jena-extensions

Rob

On 13/12/2016 16:48, "Martynas Jusevičius" <***@graphity.org> wrote:

I will try some time during holidays.

Will a PR on https://github.com/apache/jena/tree/master/jena-core work for this?

Post by A. Soroka
What are the kinds of usages to which you are imagining these kind of types being put?
---
A. Soroka
The University of Virginia Library

Martynas Jusevičius

2016-12-14 11:24:33 UTC

Shouldn't Dataset be in rdf.model together with Model, now that RDF
1.1 standardized named graphs?

Post by Rob Vesse
Yes pull requests on Git Hub are fine
This will probably need to rely upon ARQ because that is where the Dataset API lives. It might be useful initially to have it be a separate module under jena-extensions
Rob
I will try some time during holidays.
Will a PR on https://github.com/apache/jena/tree/master/jena-core work for this?

Post by A. Soroka
What are the kinds of usages to which you are imagining these kind of types being put?
---
A. Soroka
The University of Virginia Library

Andy Seaborne

2016-12-14 17:09:20 UTC

Post by Martynas JuseviÄius
Shouldn't Dataset be in rdf.model together with Model, now that RDF
1.1 standardized named graphs?

Yes, and the class packaging of other things.

User code would have to change.

Andy

Post by A. Soroka
What are the kinds of usages to which you are imagining these kind of types being put?
---
A. Soroka
The University of Virginia Library

A. Soroka

2016-12-14 17:32:51 UTC

I've made https://issues.apache.org/jira/browse/JENA-1266 to remind ourselves about this.

---
A. Soroka
The University of Virginia Library

Post by Martynas JuseviÄius
Shouldn't Dataset be in rdf.model together with Model, now that RDF
1.1 standardized named graphs?

Yes, and the class packaging of other things.
User code would have to change.
Andy

Post by A. Soroka
What are the kinds of usages to which you are imagining these kind of types being put?
---
A. Soroka
The University of Virginia Library

Andy Seaborne

2016-12-14 18:21:14 UTC

You have to move a lot more because without it is just an interface (and
it has dependencies anyway).

Andy

Post by A. Soroka
I've made https://issues.apache.org/jira/browse/JENA-1266 to remind ourselves about this.
---
A. Soroka
The University of Virginia Library

Post by Martynas JuseviÄius
Shouldn't Dataset be in rdf.model together with Model, now that RDF
1.1 standardized named graphs?

Yes, and the class packaging of other things.
User code would have to change.
Andy

Post by A. Soroka
What are the kinds of usages to which you are imagining these kind of types being put?
---
A. Soroka
The University of Virginia Library

Andy Seaborne

2016-12-14 10:37:24 UTC

Post by Martynas JuseviÄius
Well, this probably requires some generic method(s) in Dataset/Model
T apply(Function<Dataset, T> f);
ResultSet results = dataset.apply(describe).apply(select);

No need to extend dataset and model and the rest to get experimenting:

static <T,X> T apply(X object, Function<X, T> f) {
return f.apply(object);
}
// BiFunction<X, Function<X, T>, T>

then

ResultSet results = apply(
apply(dataset, describe),
select);

The function f does have any access to the internals of a specific
dataset so it does not need to be a method of Dataset.

There is a style thing about how it looks if you are not used to reading
functional application (i.e. backwards!).

Andy

Post by Martynas JuseviÄius
Seems more elegant to me than all the QueryExecution boilerplate.

Post by A. Soroka
What are the kinds of usages to which you are imagining these kind of types being put?
---
A. Soroka
The University of Virginia Library

Martynas Jusevičius

2016-12-14 11:23:36 UTC

But that would still require the functional subclasses of Query?

Post by Martynas JuseviÄius
Well, this probably requires some generic method(s) in Dataset/Model
T apply(Function<Dataset, T> f);
ResultSet results = dataset.apply(describe).apply(select);

static <T,X> T apply(X object, Function<X, T> f) {
return f.apply(object);
}
// BiFunction<X, Function<X, T>, T>
then
ResultSet results = apply(
apply(dataset, describe),
select);
The function f does have any access to the internals of a specific dataset
so it does not need to be a method of Dataset.
There is a style thing about how it looks if you are not used to reading
functional application (i.e. backwards!).
Andy

Post by Martynas JuseviÄius
Seems more elegant to me than all the QueryExecution boilerplate.

Post by A. Soroka
What are the kinds of usages to which you are imagining these kind of types being put?
---
A. Soroka
The University of Virginia Library

Andy Seaborne

2016-12-14 12:52:36 UTC

Post by Martynas JuseviÄius
But that would still require the functional subclasses of Query?

Yes, but it required no changes to the jena code. There could be a
library of such utilities.

static <T,X> T apply(X object, Function<X, T> f) {
return f.apply(object);
}

static <X> void apply(X object, Consumer<X> c) {
c.accept(object);
}

Dataset dataset = ... ;
Select selectQuery = new Select("SELECT * { ?s ?p ?o}");
ResultSet rs = selectQuery.apply(dataset);
Consumer<ResultSet> rsp = (t)->ResultSetFormatter.out(t);
apply(apply(dataset, selectQuery), rsp);

(because Consumer isn't a Function<,Void> :-()

Andy

Post by Martynas JuseviÄius
Well, this probably requires some generic method(s) in Dataset/Model
T apply(Function<Dataset, T> f);
ResultSet results = dataset.apply(describe).apply(select);

Post by Martynas JuseviÄius
Seems more elegant to me than all the QueryExecution boilerplate.

Post by A. Soroka
What are the kinds of usages to which you are imagining these kind of
types being put?
---
A. Soroka
The University of Virginia Library

Andy Seaborne

2016-12-15 20:46:28 UTC

A more considered solution:

https://gist.github.com/afs/2b8773d10cbe4bc1161e9851de02b3eb

Andy

Post by Martynas JuseviÄius
But that would still require the functional subclasses of Query?

Yes, but it required no changes to the jena code. There could be a
library of such utilities.
static <T,X> T apply(X object, Function<X, T> f) {
return f.apply(object);
}
static <X> void apply(X object, Consumer<X> c) {
c.accept(object);
}
Dataset dataset = ... ;
Select selectQuery = new Select("SELECT * { ?s ?p ?o}");
ResultSet rs = selectQuery.apply(dataset);
Consumer<ResultSet> rsp = (t)->ResultSetFormatter.out(t);
apply(apply(dataset, selectQuery), rsp);
(because Consumer isn't a Function<,Void> :-()
Andy

Post by Martynas JuseviÄius
Well, this probably requires some generic method(s) in Dataset/Model
T apply(Function<Dataset, T> f);
ResultSet results = dataset.apply(describe).apply(select);

Post by Martynas JuseviÄius
Seems more elegant to me than all the QueryExecution boilerplate.

Post by A. Soroka
What are the kinds of usages to which you are imagining these kind of
types being put?
---
A. Soroka
The University of Virginia Library

On Dec 12, 2016, at 2:03 PM, Martynas Jusevičius
Hey,
has Jena considered taking advantage of the functional features in
Java
8?
Construct extends Query implements Function<Dataset, Model>
Describe extends Query implements Function<Dataset, Model>
Select extends Query implements Function<Dataset, ResultSet>
Ask extends Query implements Function<Dataset, ResultSet>
Martynas
atomgraph.com

A. Soroka

2016-12-16 15:43:57 UTC

It seems to me that these ideas begin to border on the Stream API, with something like Stream<Tuple> at work.

---
A. Soroka
The University of Virginia Library

Post by Andy Seaborne
https://gist.github.com/afs/2b8773d10cbe4bc1161e9851de02b3eb
Andy

Post by Martynas JuseviÄius
But that would still require the functional subclasses of Query?

Yes, but it required no changes to the jena code. There could be a
library of such utilities.
static <T,X> T apply(X object, Function<X, T> f) {
return f.apply(object);
}
static <X> void apply(X object, Consumer<X> c) {
c.accept(object);
}
Dataset dataset = ... ;
Select selectQuery = new Select("SELECT * { ?s ?p ?o}");
ResultSet rs = selectQuery.apply(dataset);
Consumer<ResultSet> rsp = (t)->ResultSetFormatter.out(t);
apply(apply(dataset, selectQuery), rsp);
(because Consumer isn't a Function<,Void> :-()
Andy

Post by Martynas JuseviÄius
Well, this probably requires some generic method(s) in Dataset/Model
T apply(Function<Dataset, T> f);
ResultSet results = dataset.apply(describe).apply(select);

Post by Martynas JuseviÄius
Seems more elegant to me than all the QueryExecution boilerplate.

Post by A. Soroka
What are the kinds of usages to which you are imagining these kind of
types being put?
---
A. Soroka
The University of Virginia Library

Andy Seaborne

2016-12-16 21:22:26 UTC

There are elements of that - see CommonsRDF - though here the operations
are whole objects (dataset - to query is as a Stream<Tuple> would to
collect the tuples).

It is also like building up a executable pipeline of operations but not
doing it until the final step which allows optimization of the pipeline.

c.f. Apache spark.

Post by A. Soroka
It seems to me that these ideas begin to border on the Stream API, with something like Stream<Tuple> at work.
---
A. Soroka
The University of Virginia Library

Post by Andy Seaborne
https://gist.github.com/afs/2b8773d10cbe4bc1161e9851de02b3eb
Andy

Post by Martynas JuseviÄius
But that would still require the functional subclasses of Query?

Yes, but it required no changes to the jena code. There could be a
library of such utilities.
static <T,X> T apply(X object, Function<X, T> f) {
return f.apply(object);
}
static <X> void apply(X object, Consumer<X> c) {
c.accept(object);
}
Dataset dataset = ... ;
Select selectQuery = new Select("SELECT * { ?s ?p ?o}");
ResultSet rs = selectQuery.apply(dataset);
Consumer<ResultSet> rsp = (t)->ResultSetFormatter.out(t);
apply(apply(dataset, selectQuery), rsp);
(because Consumer isn't a Function<,Void> :-()
Andy

Post by Martynas JuseviÄius
Well, this probably requires some generic method(s) in Dataset/Model
T apply(Function<Dataset, T> f);
ResultSet results = dataset.apply(describe).apply(select);

Post by Martynas JuseviÄius
Seems more elegant to me than all the QueryExecution boilerplate.

Post by A. Soroka
What are the kinds of usages to which you are imagining these kind of
types being put?
---
A. Soroka
The University of Virginia Library

Dick Murray

2016-12-17 19:08:13 UTC

Just posted a question regarding Spark because I'm heading down the
streaming route as we're aggregating multiple large datasets together and
our 1.5TB TDB was causing us some issues. We have many large graph writes
of between 1-4Mb triples which I currently write to a number of TDB's and
use a set of streaming utility methods to aggregate the TDB's for the find
methods. This lends itself to RDD filter calls.

On 16 Dec 2016 21:22, "Andy Seaborne" <***@apache.org> wrote:

There are elements of that - see CommonsRDF - though here the operations
are whole objects (dataset - to query is as a Stream<Tuple> would to
collect the tuples).

It is also like building up a executable pipeline of operations but not
doing it until the final step which allows optimization of the pipeline.

c.f. Apache spark.

Post by A. Soroka
It seems to me that these ideas begin to border on the Stream API, with
something like Stream<Tuple> at work.
---
A. Soroka
The University of Virginia Library

Post by Andy Seaborne
https://gist.github.com/afs/2b8773d10cbe4bc1161e9851de02b3eb
Andy

Post by Martynas JuseviÄius
But that would still require the functional subclasses of Query?

Yes, but it required no changes to the jena code. There could be a
library of such utilities.
static <T,X> T apply(X object, Function<X, T> f) {
return f.apply(object);
}
static <X> void apply(X object, Consumer<X> c) {
c.accept(object);
}
Dataset dataset = ... ;
Select selectQuery = new Select("SELECT * { ?s ?p ?o}");
ResultSet rs = selectQuery.apply(dataset);
Consumer<ResultSet> rsp = (t)->ResultSetFormatter.out(t);
apply(apply(dataset, selectQuery), rsp);
(because Consumer isn't a Function<,Void> :-()
Andy

Post by Martynas JuseviÄius
Well, this probably requires some generic method(s) in Dataset/Model
T apply(Function<Dataset, T> f);
ResultSet results = dataset.apply(describe).apply(select);

Post by Martynas JuseviÄius
Seems more elegant to me than all the QueryExecution boilerplate.

Post by A. Soroka
What are the kinds of usages to which you are imagining these kind of
types being put?
---
A. Soroka
The University of Virginia Library
On Dec 12, 2016, at 2:03 PM, Martynas JuseviÄius

Post by Martynas JuseviÄius
Hey,
has Jena considered taking advantage of the functional features in
Java
8?
Construct extends Query implements Function<Dataset, Model>
Describe extends Query implements Function<Dataset, Model>
Select extends Query implements Function<Dataset, ResultSet>
Ask extends Query implements Function<Dataset, ResultSet>
Martynas
atomgraph.com

A. Soroka

2016-12-17 23:46:05 UTC

Well, I was thinking of e.g. a DESCRIBE as a source of tuples over which you could map a CONSTRUCT to produce tuples (repeatedly) over which you could map a SELECT to produce tuples or that you could collect into an ASK, that kind of thing. And, yes, in a perfect world, you could use information from the SELECT to elide work in the CONSTRUCT, and you could use information in the CONSTRUCT to elide work in the DESCRIBE, etc.

But honestly, I'd need to think a good bit more about the use cases for "conversations in queries" like this to understand how to do it right. I'm looking forward to some sketches and examples to take the conversation forward.

---
A. Soroka
The University of Virginia Library

There are elements of that - see CommonsRDF - though here the operations are whole objects (dataset - to query is as a Stream<Tuple> would to collect the tuples).
It is also like building up a executable pipeline of operations but not doing it until the final step which allows optimization of the pipeline.
c.f. Apache spark.

Post by A. Soroka
It seems to me that these ideas begin to border on the Stream API, with something like Stream<Tuple> at work.
---
A. Soroka
The University of Virginia Library

Post by Andy Seaborne
https://gist.github.com/afs/2b8773d10cbe4bc1161e9851de02b3eb
Andy

Post by Martynas JuseviÄius
But that would still require the functional subclasses of Query?

Yes, but it required no changes to the jena code. There could be a
library of such utilities.
static <T,X> T apply(X object, Function<X, T> f) {
return f.apply(object);
}
static <X> void apply(X object, Consumer<X> c) {
c.accept(object);
}
Dataset dataset = ... ;
Select selectQuery = new Select("SELECT * { ?s ?p ?o}");
ResultSet rs = selectQuery.apply(dataset);
Consumer<ResultSet> rsp = (t)->ResultSetFormatter.out(t);
apply(apply(dataset, selectQuery), rsp);
(because Consumer isn't a Function<,Void> :-()
Andy