In an admittedly poorly synchronized attempt to implement a home-grown version of my request for JPPF-303 (because I don't want to wait for JPPF 5!) I've stumbled on a sequence of events that result in duplicate host:port combinations in nodesInformation, with undesirable side effects.

If I shutdown() a slave node, but then employ node provisioning on its corresponding master node before it is removed from the driver's node list, I end up with duplicates that won't go away without rebooting the driver. Additionally, if the "dead" node was executing a task at the time it was shutdown, this task is never reported as incomplete.

Screenshot of bug Although the duplicate node is in "red" indicating it's shutdown, the presence of another node (the newly provisioned one) with the same host and port appears to prevent it from ever disappearing, and it remains in the Collection<JPPFManagementInfo> returned from the driver's nodesInformation() method. Some GUI features, such as the "JVM Health" get information for the node twice (the CPU% differs slightly but most of the other elements are identical between the dupes) and the number of Active Nodes listed is, of course inaccurate.

Further, even when the master node is eventually shut down, the "stray" shutdown nodes remain in the driver's list (possibly with unfinished tasks), and can only be removed by restarting the driver. Restarting the driver results in sending the incomplete tasks back to the client, which are then resubmitted to the driver when it starts up.

While I can obviously improve my sequencing to avoid causing this problem in the first place, and work around some of the the duplicate issues when iterating, I think these side effects bear looking into from a behind-the-scenes consistency standpoint. Clearly the GUI is able to detect that the node is shutdown, but never seems to do anything about it. It seems reasonable to assume that if there's another active node in the collection with the same host:port combination, that the "dead" node is unreachable, and it should be removed and any of its unfinished tasks resubmitted.

Of note, I do have jppf.recovery.enabled = true in my driver's configuration file, but I don't think the recovery pings are set up to go to slave nodes.