JPPF Issue Tracker
star_faded.png
Please log in to bookmark issues
task_small.png
OPEN  Task JPPF-559  -  Explore the possibilities for job streaming
Posted Nov 18, 2018 - updated Nov 20, 2018
lolo4j (lolocohen) has been working on this issue since November 20, 2018 (14:21)
Issue details
  • Type of issue
    Task
  • Status
     
    New
  • Assigned to
     lolo4j
  • Progress
       
  • Type of bug
    Not triaged
  • Likelihood
    Not triaged
  • Effect
    Not triaged
  • Posted by
     lolo4j
  • Owned by
    Not owned by anyone
  • Estimated time
    Not estimated
  • Category
    Core
  • Resolution
    Not determined
  • Priority
    Normal
  • Targetted for
    icon_milestones.png JPPF 6.1
Issue description
JPPF currently has a very basic API for job streaming. However, ths is wholly insufficient for real stream processing, as many things are missing which have to be implemented by the users in an ad-hoc manner:
  • ability to plug-in stream data sources (from files, databases, event-based systems, etc.)
  • stateful computations over streams of data (aggregates, analytics, etc.)
  • map/reduce capabilities
  • ... ?
We propose to explore what can be done in that area. Some ideas that popped into my (tiny) head:
  • make the job's data provider read-write instead of read-only, so as to provide state at the job level instead of only at the task level. We could then associate a merge strategy / reducer to each job. The data provider seems a good candidate, as opposed to the job metadata, because it is loaded the same class loader as the tasks.
  • provide a common API for various sources of straming data and plug them into a job sream