By default, Flux stores its information in an HSQL in-memory database. Flux is preconfigured with HSQL by default so that people can download Flux and get started with running simple examples within a few minutes. No need to configure your Oracle, DB2, or MySQL instance.

Over the years, a few problems have arisen from this approach:

1) HSQL's database transaction isolation level is READ_UNCOMMITTED (instead of the usual READ_COMMITTED), which places severe limitations on Flux's ability to run multiple jobs and workflows concurrently. In short, when using HSQL as its database, Flux can run at most one job or workflow at a time.

2) Flux requires up to three database connections for its internal use. Due to this requirement, not all of Flux's features are available when using HSQL. For one, Flux messaging doesn't work when using HSQL.

3) It would be useful if HSQL could store its information persistently. It can, actually, but it performs so poorly with sizable databases that Flux explicitly prevents HSQL from running in a persistent mode. For a Flux user to get a persistent database working with Flux, it would be nice if Flux's built-in database supported it. Not everyone wants to go to a full-fledged database such as Oracle, DB2, or MySQL.

Moving from HSQL to Derby would solve these problems.

Derby is well supported, would allow many jobs and workflows to run concurrently from the get-go, and would allow for an option to persistently store Flux data.

The plan for Flux 8.0 is to move from HSQL to Derby. For nearly all those using HSQL with Flux, the transition will be transparent and seamless.

Flux 8.0 development begins imminently. Flux 7.7 is very nearly ready to be released. Flux 8.0 starts afterwards.

I've already coded a proof-of-concept transition from HSQL to Derby in the Flux 7.7 code base. It was straightforward. I ran Derby-based Flux jobs and workflows.

It was cool.