Following feature request JPPF-563, create a new sample in the samples pack which demonstrates how to start a driver, node and client programmatically, all embedded within the same JVM. The sample will show the following functionalities:
* embedded driver life cycle: create, start, stop
* embedded node life cycle: create, start, stop
* connecting a client and submitting a job
* programmatically creating the configuration for a driver, node and client
* using management and monitoring APIs for an embedded driver and node
We propose that the following features be either deprecated or dropped altogether:
'''1. .Net integration'''
This feature relies heavily on the [http://jni4net.com/ '''jni4net'''] framework, which hasn't seen a new version in 4 years. Following the switch to Java 8 (feature request JPPF-548), its .Net proxy generator is no longer fully working, as it doesn't handle new Java 8 constucts such as default methods in interfaces, It is currently not possible to build it with the current code, and I don't see any solution that can be mainained in the long term. I propose to drop this feature from JPPF 6.1 forward. We will still maintain it for prior versiosn.
'''2. Android integration'''
The switch to Java 8 requires a lot of changes to the Android port, including, but definitely not limited to, the min Android sdk version and build tools. I haven't evaluated the changes that need to be done to the code itself, and, given the lack of bandwith (I'm just 1 developer), I tend to think it should be dropped so we can focus on more modern features such as job streaming and big data. If anyone volunteers to take this feature on, I'll be happy to assist in any way, In any case, we'll keep maintaining it for versions up to 6.0.
'''3. Persistent data in NodeRunner'''
The [https://www.jppf.org/javadoc/6.1/index.html?org/jppf/node/NodeRunner.html '''NodeRunner'''] class provides 3 methods '''setPersistentData()''', '''getPersistentData()''' and '''removePersistentData()''', which were intended for tasks to be able to store, access and manage data on the nodes accross job executions. These methods are inherently dangerous, because they can cause the nodes to retain objects and classes from many different class loaders, resulting in class loader leaks and potential out-of-memory conditions. This feature isn't used anymore in the JPPF code, and I believe now's a good time to remove it.
'''4. Node security policy'''
I can no longer see the benefit of the [https://www.jppf.org/doc/6.0/index.php?title=Node_configuration#Security_policy '''security policy'''] in the node configuration. We haven't touched this code in years, the default node security policy in the node distribution is no longer close to being useful or even accurate, and this can be easily replaced with a standard SecurityManager and associated security policy file. This feature should definitely be removed.
Current the [https://www.jppf.org/javadoc/6.1/index.html?org/jppf/server/JPPFDriver.html '''JPPFDriver'''] is implemented as a singleton. This means there can be only one per JVM. The same is true for the nodes, whether local to the driver's JVM or remote.
We propose to change that and make it possible to have any number of drivers and/or nodes per JVM. They might have the possibility to share some common resources, for example the NIO thread pool.
Currently the [https://www.jppf.org/doc/6.1/index.php?title=Execution_Policy_Elements#Preference '''Preference'''] execution poliy is applied to each node individually and is identical to the [https://www.jppf.org/doc/6.1/index.php?title=Execution_Policy_Elements#OR '''OR'''] execution policy. To sum up: despite its name, it has nothing to do with a "preference".
We propose to make it live up to its name, which implies:
* it should define a real order of preference for a number of node execution policies, where a node that satisfies policy N in the list will have priority over a node that satisfies policy N + 1
* it should be applied globally to all the nodes available to the driver
* because of the previous point, it should be a separate attribute of the job SLA
* it should be applicable to the client-side as well, where it would define a driver preference rather than a node preference
* special care should be taken about perfomance, as the algorithm will be in O(nbNodes * nbJobs). Should we allow parallel (with regards to the nodes) evaluation of the policy for each job?
After seeing one failure of the TestJPPFUuid unit test (the only one I've seen in years), I performed some more rigorous testing of the uuids generated by [https://github.com/jppf-grid/JPPF/blob/master/common/src/java/org/jppf/utils/JPPFUuid.java '''JPPFUuid''']. I found that there is around 1 collision for every 40 million (40,000,000) uuids. Essentially, this is due to the use of a non cryptographically strong random number generator, namely [https://docs.oracle.com/javase/8/docs/api/index.html?java/util/Random.html '''java.util.Random''']. Instead, we should use [https://docs.oracle.com/javase/8/docs/api/index.html?java/security/SecureRandom.html '''java.security.SecureRandom'''].
JPPF currently has a very basic [https://www.jppf.org/doc/6.1/index.php?title=Submitting_multiple_jobs_concurrently#Job_streaming '''API for job streaming''']. However, ths is wholly insufficient for real stream processing, as many things are missing which have to be implemented by the users in an ad-hoc manner:
* ability to plug-in stream data sources (from files, databases, event-based systems, etc.)
* stateful computations over streams of data (aggregates, analytics, etc.)
* map/reduce capabilities
* ... ?
We propose to explore what can be done in that area. Some ideas that popped into my (tiny) head:
* make the job's [https://www.jppf.org/doc/6.1/index.php?title=Sharing_data_among_tasks_:_the_DataProvider_API '''data provider'''] read-write instead of read-only, so as to provide state at the job level instead of only at the task level. We could then associate a merge strategy / reducer to each job. The data provider seems a good candidate, as opposed to the job metadata, because it is loaded the same class loader as the tasks.
* provide a common API for various sources of straming data and plug them into a job sream
DefaultJPPFSerialization fails to serialize JPPF task execution notification, throws java.io.IOException: Can not set boolean field org.jppf.management.OffloadableNotification.userDataOffloaded to javax.management.ObjectName.
The problem can lead to unexpected task cancellations in case, when DefaultJPPFSerialization is used.
The constructors of the various execution policy rules perform no argument checking in their constructors. This is inconsistent with the execution policy XML schema and makes usage of the API a potential problem because default values are used for null or missing arguments.
Missing checks I have identified so far:
* null checks
* min/max number of arguments (e.g. AND rule should have at least 2 non-null arguments)
To simplify execution policy customization it would be nice if there were "AcceptAll" and "RejectAll" execution policies available.
Or simply "empty" but then it is non-obvious if if accepts or rejects everything.
These are useful for eg. disabling parts of rule "and(config, rejectAll)" / "or(config/ acceptAll)".
Currently, the communication between JPPF servers and client is based on a synchronous request/response model. As a consequence, a connection can only handle a single job at a time. This can impair the scalability of JPPF applications.
We propose to switch to an asynchronous model where each connection can handle multiple jobs, removing or mitigating the need for multiple connections.
JPPF 6.1 will stop supporting Java 7, whose end-of-life happened a long time ago.
The first step will be to make the compilation, build and tests pass without errors or warnings.
Usage of Java 8 new features and APIs will be either progressive, while writing new code or updating existing code, or systematic in some cases if we determine that the benefits are worth it.
The first difficulty will be to refactor or rewrite the custom doclet code we use for the @exclude tag used in the javadoc.
The SLA of each JPPF job has a classpath attribute which allows the job to transport a set of jar files to dynamically add to its classloader when it is executed on a node.
There are currently a number of problems with the classpath itself and with the way it is handled by the node:
* classpath elements refer to a "local" and a "remote" location. The semantics here is confusing and we propose to use "source" and "target" locations instead
* there is currently no way to avoid downloading jar files if they already exists on the file system. We should allow a flow to allow it, make it default to false to preserve the existing behavior as a default
* the notion of "force classloader reset" isn't clear ly explained and does not work as documented. This shouid be fixed
Many JPPF classes use JSR 223-compliant scripts to perform their functionality:
* scripted tasks
* scripted execution policies
* scripted node selectors
* scripted job selectors
* scripted validation function in ClassPathElementImpl
We propose to introduce a new class ScriptDefinition which groups the attributes needed to execute the scripts:
* script: the actual script content as text
* bindings: an optional set of variable bindings made available to the script
* reusableId: an optional identifier for the script which allows reusing the compiled form of the script so as to avoid uncessary recompilation of the script
One of the objectives is to simplify the code that uses scripts, in particular so as to hide the internal mechanism to cache and pool the scripting engines and reusable compiled scripts.
A longer-term objective (targetting JPPF v6.1) is to expose the ScriptDefinition class wherever scripts are used in the public APIs. This will simplify the exposed APIs while increasing flexibility and add consistency with regards to scripting.