Streaming LOB support

• Key: OPENJPA-130
• Type: New Feature
• Status: Resolved
• Resolution: Fixed
• Assignee: Ignacio Andreu
• Reporter: Patrick Linskey
• Component/s: datacache, jdbc, jpa, kernel
• Fix Version/s: 1.1.0

Description

BLOB and CLOB fields can only be mapped in their entirety in OpenJPA. It would be nice to support fields of type java.io.InputStream (for BLOBs) and java.io.Reader (for CLOBs).

The usage pattern could look like so:

@Entity
public class Employee {
    ...
    private InputStream photoStream;

    public void setPhotoStream(InputStream in) {
        photoStream = in;
    }

    public InputStream getPhotoStream() {
        return photoStream;
    }
}

So, when the user wants to provide a stream, she will set the InputStream field, and when the user wants to obtain a stream, she will use the field.

The behavior of such an implementation would be a bit different than how other fields work, in that if the user set the stream and then consumed it within a single transaction, presumably no data would be written out to the database at commit time. But that is the nature of streams.

(FTR, I think that I stole this idea from an email Craig Russell sent out years ago.)

Comments

[Patrick Linskey]

Ignacio Andreu will be working on this issue for the Google Summer of Code project. It would seem that JIRA only allows me to assign issues to committers, so I'm assigning it to myself to keep it out of the unassigned queue.

[Patrick Linskey]

Assigning to Ignacio, who is working on this as part of the Google Summer of Code internship program.

[Ignacio Andreu]

Now is the time to decide what will be the best way to mark this fields. In my opinion we have two options, we can to use the @LOB annotation or to create a @Stream annotation. What do you think?

[Craig Russell]

I don't yet see the need to create another annotation.

If you annotate an InputStream with @LOB then OpenJPA should be smart enough to figure it out.
@Entity
public class Employee {
    ...
    @LOB @Column(name="PHOTO")
    private InputStream photoStream;
...
}

My only concern with Craig's suggestion is compatibility if the JPA spec team decides to go a different route in a future version. If we use our own annotation, then we can maintain compatibility for it even in the face of spec changes.

[Ignacio Andreu]

This patch was part of my Summer of Code work, all the tests work fine in MySQL, Oracle and SQL Server.

Streams are mapped using @Persistent annotation.

@Entity
public class Employee {
    ...
    @Persistent
    private InputStream photoStream;
...
}

[Patrick Linskey]

Great work, Ignacio! I checked in the patch (with a couple of minor whitespace tweaks).

I'm going to leave this issue open for right now; we need to document the new feature still, and this will help us track that work.

[Ignacio Andreu]

This patch corrects some bugs and add new test cases.

New test cases & fixed bugs:

    - testUpdateWithNull
    - testUpdateANullObjectWithoutNull

Modified:

    - blobBufferSize and clobBufferSize. I've increased the buffer value.
    
  

BTW, I'm trying to add Streaming support to Derby, my approach is not finished yet, It loads the results from the database when I use the load() method but it fails when I try to load with a Query. Next week i will put here my approach.

[Ignacio Andreu]

Sorry, my derby approach is success using the find() method

[Ignacio Andreu]

This patch adds support for Streams in PostgreSQL and documentation about this issue.

[Patrick Linskey]

Resolved with Ignacio's recent work. There is still an open issue regarding Postgres and database cleanup; this will be managed through a separate JIRA issue.