JPPF Issue Tracker
star_faded.png
Please log in to bookmark issues
feature_request_small.png
CLOSED  Feature request JPPF-270  -  Execution policy for valid subnet bitmask
Posted May 19, 2014 - updated Jun 10, 2014
action_vote_minus_faded.png
0
Votes
 action_vote_plus_faded.png
icon_info.png This issue has been closed with status "Closed" and resolution "RESOLVED".
Issue details
  • Type of issue
    Feature request
  • Status
     
    		 Closed
  • Assigned to
    lolo4j
  • Type of bug
    Not triaged
  • Likelihood
    Not triaged
  • Effect
    Not triaged
  • Posted by
    Daniel Widdis
  • Owned by
    Not owned by anyone
  • Category
    Node
  • Resolution
    RESOLVED
  • Priority
    Normal
  • Targetted for
    icon_milestones.png JPPF 4.2
Issue description
Users may wish to restrict tasks to process on particular networks based on characteristics. For example, a network may consist of dedicated servers within one address space, suitable for tasks that need dedicated 100% CPU, other computers with a shared task load that may have less CPU available, and possibly even volunteer "offline mode" computers that have less reliable completion percentages due to idle time, possilbe network issues, etc., and are suitable for shorter tasks with an expiry on processing time.

For example, Rackspace private IP addresses for a cloud implementation use IP addresses in one of two ranges: 10.176.0.1 through 10.191.255.254 (10.176.0.0/12) and 10.208.0.1 through 10.233.255.254 (10.208.0.0/12).

Presently there is no easy way to easily denote subnets: the only existing methods that work well with IP addresses are the string matching in the Contains class. Creating a long string of Contains().or(Contains()) is a less than optimal way of restricting tasks to operating on those nodes, and not on nodes that volunteers may connect to the server from home, etc. Certainly a CustomRule could be used and is an adequate workaround for the lack of this feature; however, given the network nature of JPPF and node logic customization abilities already present in the config files, this seems like a good candidate for a built-in Execution Policy.

It would be relatively easy to create an execution policy that does IP address bitmasking. Apache Commons includes a SubnetUtils class that does all the heavy lifting.

#1
Comment posted by
Daniel Widdis
May 19, 06:52
A file was uploaded. Execution Policy for Subneticon_open_new.png This comment was attached:

A basic implementation of subnet matching. Does not handle the IllegalArgumentException that will be thrown if either of the strings provided is not in the valid format.
#3
Comment posted by
Daniel Widdis
May 19, 18:04
A file was uploaded. IPv4 Subnet Execution Policyicon_open_new.png This comment was attached:

Updated my implementation, actually compiling and testing it this time. Of note:
  • Since we're always checking the ipv4.addresses property, hardcoded that instead of forcing user to provide it
  • parses space-delimited list of IP|DOMAIN format of ipv4.addresses
  • Catches IllegalArgumentException and returns false if invalid subnet string


Also added commons-net-3.3.jar to classpath.
#4
Comment posted by
Daniel Widdis
May 19, 18:12
The Apache Commons Subnet implementation is limited to ipv4. The Spring Framework has an IPAddressMatcher class that will work with both IPv4 and IPv6 and would probably be a preferred implementation. I'll stop hacking around now and leave it to the official devs to discuss if, and how best to implement. My goal was proof-of-concept.
#5
Comment posted by
Daniel Widdis
May 20, 01:25
It looks like the guys at JIRA have written their own set of classes based on the Spring class linked above, with only dependencies on JRE classes. To avoid importing the atlassian-ip package, you could just use these classes. I'm implementing a CustomPolicy based on these classes and will upload my results when done and tested. https://docs.atlassian.com/atlassian-ip/3.1/xref/
#6
Comment posted by
lolo4j
icon_reply.pngMay 20, 07:35, in reply to comment #5
Dan, thanks a lot for researching into this topic. Ideally, a preferred solution would not introduce new dependencies, as we are trying to keep the driver/node installations to a very low footprint on disk (currently about 2 MB) as well as memory-wise.

Couldn't we use the classes in the org.jppf.net package, along with the IPFilter class as a starting point?
#7
Comment posted by
Daniel Widdis
May 22, 08:52
A file was uploaded. InSubnet Execution Policyicon_open_new.png This comment was attached:

I agree about keeping a small footprint and no dependencies. For my proof of concept I just copied over the files directly from the atlassian-ip package, so there are no additional dependencies, plus a lot of unused methods to get rid of. But as you mention, IPFilter is probably a great starting point. It's probably simple to write a method to convert the standard CIDR notation to the IP ranges you have defined and allow both notations in both places.

Attaching the current work in progress, with InSubnet extending the CustomPolicy. I'll work up a modification to IPFilter to handle both format cases.

Attached file tested to work with 
job.getSLA().setExecutionPolicy(new InSubnet("10.176.0.0/12", "10.208.0.0/12"));
#8
Comment posted by
Daniel Widdis
May 22, 22:57
Of note, the existing IPv6AddressPattern class does not handle ipV6 address formats including the "::" shorthand for consecutive zeroes: it requires the user to explicitly include all fields (or assumes that any omissions are trailing) fields.

This most notably affects the ability to use localhost "::1" which represents 0:0:0:0:0:0:0:1, but instead parses as follows: 
pattern 4 for source '::1' = '0-ffff:0-ffff:1:0-ffff:0-ffff:0-ffff:0-ffff:0-ffff'

Also the all-zeroes pattern "::" is perfectly legal in ipV6 but is disallowed in IPv6AddressPattern.

This is consistent with the way the patterns are documented in the javadoc, but inconsistent with the IPv6 standard. I expect the classes I'm working on will provide users an alternative consistent with the IPv6 shorthand.
#9
Comment posted by
Daniel Widdis
May 23, 00:10
A file was uploaded. Subnet Execution Policiesicon_open_new.png This comment was attached:

Laurent,

I think I have a very workable implementation for you.
  • I simply extended the IPv4AddressPattern class to an IPv4AddressNetmask class (and similarly for IPv6), parsing the Netmask input to be consistent with what the Pattern expects.
  • I created an IPv4Subnet class (and IPv6Subnet) to be used as an Execution Policy, extending the CustomPolicy, to make use of the netmasks. By placing the attached jar in the driver's classpath I've successfully tested the following:
job.getSLA().setExecutionPolicy(new IPv4Subnet("10.176.0.0/12", "10.208.0.0/12"));
job.getSLA().setExecutionPolicy(new IPv4Subnet("192.168.237.0/8"));
job.getSLA().setExecutionPolicy(new IPv6Subnet("::1"));
 Of note, the IPv4 version of the CustomPolicy will also work with the same format as inclusion/exclusion patterns because it just calls super() with an unmodified string if no netmask is included. This will also work for the IPv6 version, but only if all 8 fields are specified. (In the example cited, "::1" is treated as "::1/128".) I think I can also handle the IPv6 pattern without all 8 fields as long as there is no "::" included, and I'm working on that now.

I think this is mostly complete except for the non-:: pattern changes, and Javadoc documentation/more comments, which I'll work on (but wanted to let you take a look at it for review/comment).

I would suggest you also make the netmask version an option for inclusion/exclusion patterns in the config file(s).
#10
Comment posted by
Daniel Widdis
May 23, 01:20
A file was uploaded. Documented Subnet Execution Policiesicon_open_new.png This comment was attached:

Laurent,

I've finished up the Javadocs, changed the inheritance to what I had originally intended, and changed the IPv6 to also use patterns as long as there is no "::" included. Officially ceasing work on these and turning it over to you.

Dan
#11
Comment posted by
Daniel Widdis
May 23, 02:13
A file was uploaded. Documented Netmask Classesicon_open_new.png This comment was attached:

I think that uploading the file with the same name as an older version doesn't work. Here's the same file renamed... really, I'm quitting now!
#12
Comment posted by
lolo4j
icon_reply.pngMay 31, 10:44, in reply to comment #11
I apologize for my lack of recent answer. I was relying on notifications from the issue tracker to keep updated, but I didn't get notifications for your latest comments. First thing I want to say is thank you, I am very grateful for the work you've put into this great feature.

I've been through the code, and I believe it is great. There are a few changes I would like to make, as I think they really make sense:
  • I'd like to make IPv4AddressNetmask and IPv6AddressNetmask first-class execution policies, with their own name in the execution policy XML schema. This invloves making them extend ExecutionPolicy instead of CustomPolicy, and adding a toString() method that prints the proper XML representation. I think I'll also have to change the XML parsing code appropriately - my problem :)
  • rather than creating a new org.jppf.ip package, I believe it would make sense to dispatch IPv4Subnet and IPv6Subnet to the existing org.jppf.net package, and IPv4AddressNetmask and IPv6AddressNetmask into org.jppf.node.policy.
Please let me know if you see any issue with the above.
#13
Comment posted by
Daniel Widdis
icon_reply.pngJun 03, 01:33, in reply to comment #12
I completely agree with both of your comments.

I only created the classes in my own package as CustomPolicy implementations in order to make them work independently of modifying the JPPF source, as an example of how it could be done simply. I had assumed you would create "first class" versions with all the other stuff included... all that was "details" that I didn't care about (or required XML schema decisions above my paygrade), I just wanted to make something that worked.

Two quick comments:
  • I think it would be very useful if you could allow either bitmask OR your pattern notation in any appropriate places in the code where they would apply (not only the Execution Policies, but the parsing of the configuration files for inclusion/exclusion patterns). I realized while implementing it that you already did have most of the capability I needed, just with a different notation system than I had been looking for.
  • In light of the above, I think there are few of places in IPv*AddressNetmask's netmaskToRange() method where I throw an IllegalArgumentException, when I should probably just return the unmodified original string to be passed up to the pattern. The idea being, if the Netmask syntax works, it is the first choice; if not, it is (attempted to be) treated as a pattern (where it may fail as well). This would allow, for example, the :1:::::: code as a pattern, while it is currently treated as an (invalid) netmask.


#15
Comment posted by
lolo4j
icon_reply.pngJun 05, 07:27, in reply to comment #13
I committed a first implementation in trunk revision 3220

I made a few more changes to the original sources: renamed IPv*Subnet into IsInIPv*Subnet so the names match the predicate semantics, added a constructor that takes a Collection<String>, added toString() method that converts to XML representation.

What remains to do on this:
  • I spotted a few places where defensive code need to be added to avoid NPEs
  • implement and run test cases
  • document
Lastly, based on your latest comments, I registered the Feature request JPPF-283 - Allow use of either CIDR or IPv*AdressPattern notations
#16
Comment posted by
lolo4j
Jun 10, 20:41
Implemented, tested and documented in trunk revision 3224