JPPF Issue Tracker
Please log in to bookmark issues
OPEN  Task JPPF-559  -  Explore the possibilities for job streaming
Posted Nov 18, 2018 - updated Mar 24, 2019
lolo4j (lolocohen) has been working on this issue since November 20, 2018 (14:21)
Issue details
  • Type of issue
  • Status
  • Assigned to
  • Progress
  • Type of bug
    Not triaged
  • Likelihood
    Not triaged
  • Effect
    Not triaged
  • Posted by
  • Owned by
    Not owned by anyone
  • Estimated time
    Not estimated
  • Category
  • Resolution
    Not determined
  • Priority
  • Targetted for
    icon_milestones.png JPPF 6.2
Issue description
JPPF currently has a very basic API for job streaming. However, ths is wholly insufficient for real stream processing, as many things are missing which have to be implemented by the users in an ad-hoc manner:
  • ability to plug-in stream data sources (from files, databases, event-based systems, etc.)
  • stateful computations over streams of data (aggregates, analytics, etc.)
  • map/reduce capabilities
  • ... ?
We propose to explore what can be done in that area. Some ideas that popped into my (tiny) head:
  • make the job's data provider read-write instead of read-only, so as to provide state at the job level instead of only at the task level. We could then associate a merge strategy / reducer to each job. The data provider seems a good candidate, as opposed to the job metadata, because it is loaded the same class loader as the tasks.
  • provide a common API for various sources of straming data and plug them into a job sream