The clustering module provides the clustering feature for carbon kernel which adds support for High Availability, Scalabilty and Failover. The overall architecture of clustering implementation in a single node is given below.
Below are the description of each components of the clustering framework.
ClusterConfiguration – The Cluster Configuration which holds the static information of the cluster. This is will be build and populated using the cluster.xml
ClusterContext – The cluster context which holds the run-time information of the cluster such as members, membership listeners.
ClusteringAgent – The ClusteringAgent which manages the cluster node in a cluster. This will basically do the starting, joining and shutdown the node with cluster. It also provide the functionality to send cluster messages to the cluster, or w set of cluster members in the cluster. Any new clustering implementation that need to be plugged into carbon, should implement this and register it as an OSGi service with the service level property (Agent) to uniquely identify it at run-time.
ClusterService – The cluster service is provided by the cluster framework. It provides a set of functions such as sending cluster message to cluster or to a set of members in the cluster retrieve the current members in the cluster, etc.
MebershipScheme – A representation of a membership scheme such as “multicast based” or “well-known address (WKA) based” schemes. This is directly related to the membership discovery mechanism.
The clustering framework by default supports Multicast or WKA (Well Known Addressing) membership schemes.
- Multicast – membership is automatically discovered using multicasting
- WKA – Well-Known Address based membership scheme. Membership is discovered with the help of one or more nodes running at a Well-Known Address. New members joining a cluster will first connect to a well-known node, register with the well-known node and get the membership list from it. When new members join, one of the well-known nodes will notify the others in the group. When a member leaves the cluster or is deemed to have left the cluster, it will be detected by the Group Membership Service (GMS) using a TCP ping mechanism.
Clustering Agent
The main part of a clustering implementation is the clustering agent. By default carbon will ship “Hazelcast” based clustering agent implementation. If a new implementation need to be plugged into carbon, it should implement this and then should be registered as an OSGi service with the service level property (Agent) to uniquely identify it at run-time.
Below provides a step by step guide on plugging a new clustering agent implementation (For example : zookeeper coordination framework based clustering agent implementation).
1. Implement the clustering agent interface.
public interface ClusteringAgent {
/**
* Initialize the agent which will initialize this node, and join the cluster
*/
void init(ClusterContext clusterContext) throws ClusterInitializationException;
/**
* Shutdown the agent which will remove this node from cluster
*/
void shutdown();
/**
* Send a message to all members in the cluster
*/
void sendMessage(ClusterMessage msg) throws MessageFailedException;
/**
* Send a message to a set of specific members in the cluster
*/
void sendMessage(ClusterMessage msg, List<ClusterMember> members) throws MessageFailedException;
}
2. Register this as an OSGi service so that the cluster framework will discover this. But the service registration should also provide a service level parameter “Agent” with the meaningful value, where the cluster framework will compare that parameter with the value in cluster.xml to correctly identify the cluster agent implementation at run-time. Service registration can now be done via annotation based approach. In carbon the Apache Felix SCR Annotation plugin is used for this purpose. The above agent class implementation will look like the below after it has been annotated with required annotation.
@Component(
name = "ZookeeperClusteringAgentServiceComponent",
description = "The ClusteringAgent class which is based on Zookeeper",
immediate = true
)
@Service
@Property(name = "Agent", value = "zookeeper")
public class ZookeeperClusteringAgent implements ClusteringAgent {
@Override
public void init(ClusterContext clusterContext) throws ClusterInitializationException {
// Add the logic that should be executed for agent initialization
}
@Override
public void shutdown() {
// This will be called when the cluster agent bundle/component deactivates.
}
@Override
public void sendMessage(ClusterMessage msg) throws MessageFailedException {
// This method should implement the logic of sending a message to all members in the cluster.
}
@Override
public void sendMessage(ClusterMessage msg, List<ClusterMember> members)
throws MessageFailedException {
// This method should implement the logic of sending a message to specific members in the cluster.
}
}
Cluster Configration
The clustering can be configured via the cluster.xml for a carbon server. This file is located in $CARBON_HOME/repository/conf/cluster.xml. The following shows the default cluster configuration. To enable clustering, simply change the “Enable” element to “true”, which will enable clustering for that node.
<Cluster xsi:schemaLocation="http://wso2.com/schema/clustering/config cluster.xsd"
xmlns="http://wso2.com/schema/clustering/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<!--Indicates whether clustering should be enabled or disabled for this node-->
<Enable>false</Enable>
<!-- The agent implementation used with cluster framework-->
<Agent>hazelcast</Agent>
<!--
The clustering domain/group. Nodes in the same group will belong to the same multicast
domain. There will not be interference between nodes in different groups.
-->
<Domain>wso2.carbon.domain</Domain>
<!-- Properties specific to this member -->
<LocalMember>
<Host>127.0.0.1</Host>
<Port>4000</Port>
<Properties>
<Property name="backendServerUrl">https://${hostName}:${httpsPort}/services/</Property>
<Property name="mgtConsoleUrl">https://${hostName}:${httpsPort}/</Property>
<Property name="subDomain">worker</Property>
</Properties>
</LocalMember>
<!--
The membership scheme used in this setup. The only values supported at the moment are
"Multicast" and "WKA"
1. Multicast - membership is automatically discovered using multicasting
2. WKA - Well-Known Address based multicasting. Membership is discovered with the help
of one or more nodes running at a Well-Known Address. New members joining a
cluster will first connect to a well-known node, register with the well-known node
and get the membership list from it. When new members join, one of the well-known
nodes will notify the others in the group. When a member leaves the cluster or
is deemed to have left the cluster, it will be detected by the Group Membership
Service (GMS) using a TCP ping mechanism.
-->
<MembershipScheme>
<!-- multicast membership scheme related properties-->
<Multicast>
<Group>228.0.0.4</Group>
<Port>45564</Port>
<TTL>100</TTL>
<Timeout>60</Timeout>
</Multicast>
<!-- wka membership scheme related properties-->
<!--WKA>
<Members>
<Member>
<Host>127.0.0.1</Host>
<Port>4000</Port>
</Member>
</Members>
</WKA-->
</MembershipScheme>
</Cluster>
Cluster Service
The Cluster API which is given as an OSGi service to carbon platform will have the following methods. The users can consume this service using OSGi service reference/discovery mechanism.
public interface Cluster {
/**
* Send the given cluster message to the whole cluster
*
* @param clusterMessage the cluster message to be sent
* @throws MessageFailedException on error
*/
void sendMessage(ClusterMessage clusterMessage) throws MessageFailedException;
/**
* Send the given cluster message to a set of members in the cluster
*
* @param clusterMessage the cluster message to be sent
* @param members the list of members to send the cluster message
* @throws MessageFailedException on error
*/
void sendMessage(ClusterMessage clusterMessage, List<ClusterMember> members)
throws MessageFailedException;
/**
* Return the list of currently available members in the cluster
*
* @return the member list
*/
List<ClusterMember> getMembers();
}
Pingback: Carbon 5 Kernel Architecture | Kishanthan's Blog