JPPF Issue Tracker
star_faded.png
Please log in to bookmark issues
feature_request_small.png
CLOSED  Feature request JPPF-288  -  Clients task sets node configuration
Posted Jun 30, 2014 - updated Apr 25, 2016
action_vote_minus_faded.png
0
Votes
action_vote_plus_faded.png
icon_info.png This issue has been closed with status "Closed" and resolution "RESOLVED".
Issue details
  • Type of issue
    Feature request
  • Status
     
    Closed
  • Assigned to
     lolo4j
  • Type of bug
    Not triaged
  • Likelihood
    Not triaged
  • Effect
    Not triaged
  • Posted by
     Roger Nell
  • Owned by
    Not owned by anyone
  • Category
    Server
  • Resolution
    RESOLVED
  • Priority
    Normal
  • Targetted for
    icon_milestones.png JPPF 5.2
Issue description
I see that there are feature requests to control the Node's JVM. I also saw that a client can remotely manage a node, execute a restart, even set the number of threads.

The problem I see is that my client does not know where its jobs are going to run and who else is using the grid. I also don’t want to restart the entire grid when my jobs may only need to run on a couple of nodes. e.g. I set the number of threads (and maybe one day, the JVM) using the current JMX API but another client with different jobs jumps in front of me and is now using my client’s node configuration.

I propose that the client can set a node SLA/configuration where nodes are restarted/reconfigured if the node configuration does not match the client’s requested node SLA/configuration prior to running my client’s job – this implies that if the node is restarting for a config change that the node is reserved for my job – I don’t want someone else’s job getting my node.

If the node matches my configuration, I’d still like the option to have it restart prior to starting my job anyway. For instance, I may know that other types of jobs in the grid cause problems – JVM in low memory state after a job, etc. and want to be sure the JVM is in a guaranteed state provided by the restart.

#1
Comment posted by
 lolo4j
Jul 01, 08:09
This is a very interesting feature request. I see that it will be very difficult to implement and I don't think it will fit in the schedule for v4.2 planned for July 15th. As a rule, we never provide new features in a maintenance release. We can sometimes cheat by disguising a feature as an enhancement, but this one is too disruptive of the existing code to fit in.

As a preliminary reflection, and as a base for discussion and investigation, here are my initial thoughts, in no particular order:
  • this will change the way job scheduling is done in the server
  • this feature introduces a potential semantic conflict with regards to the job SLA: the SLA's purpose is to filter out nodes that do not match it, whereas the present proposal is to make the nodes match the SLA. We'll need to find a way to reconcile the two, but we can imagine the magnitude of the design change this implies
  • a major difficulty is to "reserve" a node for a given job, ensuring no other job is scheduled for that node. A possibility could be to use an exclusion list in the server, where each node in the list is mapped to a job.
  • a possibility could be to do this using the node provisioning feature: instead of restarting the node, we deactivate it, provision a single slave node (eventually unprovisioning an already existing slave first), add as a configuration override to the slave a property like "job.uuid = job_identifier" and then add this as a criteria for the job scheduling, maybe as part of a custom execution policy or a scripted policy.


#4
Comment posted by
 Roger Nell
icon_reply.pngJul 01, 18:44, in reply to comment #1
Hi, Thanks for considering my enhancement. We are evaluating the possibility of using JPPF as a replacement for a commercial product.

The concept behind the request comes from this commercial product. A task has an node filter where it is possible to filter on a nodes properties (OS, Ram, Cores, custom, etc.) The JVM version and number of threads is *not* part of the node filter but rather the task's requested node configuration. So a task could specify a node filter (e.g. OS=Windows,Cores=4) and also a node configuration (e.g. JVM1.8, threads=8 ). The thread count doesn't trigger a restart, changing the JVM might trigger a restart if the correct version wasn't already running. Also a task can ask that the the JVM to always be restarted.

The reconfiguration of a node allows many different tasks to coexist on the same grid. Some tasks have there own threading and don't want the node to process many tasks at once, others are single threaded and it makes sense to have the node processes many jobs simultaneously.

I understand it might not follow your model - I expected that. I wanted to throw it out there to see what was possible today and make sure I wasn't missing anything.

I appreciate your reply and attention. You have an amazing product and it's great to see you actively supporting it.

Regards, Roger

#9
Comment posted by
 lolo4j
Mar 29, 06:45
Now that the ability to specify the JVM path has been implemented in Feature request JPPF-176 - Enable specifying the JVM location for the driver and node launcher, all that remains to do is:
  • add the appropriate attribute(s) to the job SLA to describe the desired node configuration and tell the server to restart the node with that configuration. We could try and compute a similarity score for each eligible node, and use the ndoe with the highest score
  • add the ability to reserve a node for a specific job - it could be a good idea to have a timeout on the reservation, to avoid having nodes waiting for a job that is no more
#11
Comment posted by
 lolo4j
Apr 25, 13:24
Implemented in trunk revision 4023