Carbon 5.0 Clustering Framework – My Notes

The clustering module provides the clustering feature for carbon kernel which adds support for High Availability, Scalabilty and Failover. The overall architecture of clustering implementation in a single node is given below.

Below are the description of each components of the clustering framework.

ClusterConfiguration – The Cluster Configuration which holds the static information of the cluster. This is will be build and populated using the cluster.xml

ClusterContext – The cluster context which holds the run-time information of the cluster such as members, membership listeners.

ClusteringAgent – The ClusteringAgent which manages the cluster node in a cluster. This will basically do the starting, joining and shutdown the node with cluster. It also provide the functionality to send cluster messages to the cluster, or w set of cluster members in the cluster. Any new clustering implementation that need to be plugged into carbon, should implement this and register it as an OSGi service with the service level property (Agent) to uniquely identify it at run-time.

ClusterService – The cluster service is provided by the cluster framework. It provides a set of functions such as sending cluster message to cluster or to a set of members in the cluster retrieve the current members in the cluster, etc.

MebershipScheme – A representation of a membership scheme such as “multicast based” or “well-known address (WKA) based” schemes. This is directly related to the membership discovery mechanism.

The clustering framework by default supports Multicast or WKA (Well Known Addressing) membership schemes.

  1. Multicast – membership is automatically discovered using multicasting
  2. WKA – Well-Known Address based membership scheme. Membership is discovered with the help of one or more nodes running at a Well-Known Address. New members joining a cluster will first connect to a well-known node, register with the well-known node and get the membership list from it. When new members join, one of the well-known nodes will notify the others in the group. When a member leaves the cluster or is deemed to have left the cluster, it will be detected by the Group Membership Service (GMS) using a TCP ping mechanism.

Clustering Agent

The main part of a clustering implementation is the clustering agent. By default carbon will ship “Hazelcast” based clustering agent implementation. If a new implementation need to be plugged into carbon, it should implement this and then should be registered as an OSGi service with the service level property (Agent) to uniquely identify it at run-time.

Below provides a step by step guide on plugging a new clustering agent implementation (For example : zookeeper coordination framework based clustering agent implementation).

1. Implement the clustering agent interface.

public interface ClusteringAgent {
    /**
     * Initialize the agent which will initialize this node, and join the cluster
     */
    void init(ClusterContext clusterContext) throws ClusterInitializationException;
    /**
     * Shutdown the agent which will remove this node from cluster
     */
    void shutdown();
    /**
     * Send a message to all members in the cluster
     */
    void sendMessage(ClusterMessage msg) throws MessageFailedException;
    /**
     * Send a message to a set of specific members in the cluster
     */
    void sendMessage(ClusterMessage msg, List<ClusterMember> members) throws MessageFailedException;
}

2. Register this as an OSGi service so that the cluster framework will discover this. But the service registration should also provide a service level parameter “Agent” with the meaningful value, where the cluster framework will compare that parameter with the value in cluster.xml to correctly identify the cluster agent implementation at run-time. Service registration can now be done via annotation based approach. In carbon the Apache Felix SCR Annotation plugin is used for this purpose. The above agent class implementation will look like the below after it has been annotated with required annotation.

@Component(
        name = "ZookeeperClusteringAgentServiceComponent",
        description = "The ClusteringAgent class which is based on Zookeeper",
        immediate = true
)
@Service
@Property(name = "Agent", value = "zookeeper")
public class ZookeeperClusteringAgent implements ClusteringAgent {
    @Override
    public void init(ClusterContext clusterContext) throws ClusterInitializationException {
        // Add the logic that should be executed for agent initialization
    }
    @Override
    public void shutdown() {
        // This will be called when the cluster agent bundle/component deactivates.
    }
    @Override
    public void sendMessage(ClusterMessage msg) throws MessageFailedException {
        // This method should implement the logic of sending a message to all members in the cluster.
    }
    @Override
    public void sendMessage(ClusterMessage msg, List<ClusterMember> members)
            throws MessageFailedException {
        // This method should implement the logic of sending a message to specific members in the cluster.
    }
}

Cluster Configration

The clustering can be configured via the cluster.xml for a carbon server. This file is located in $CARBON_HOME/repository/conf/cluster.xml. The following shows the default cluster configuration. To enable clustering, simply change the “Enable” element to “true”, which will enable clustering for that node.

<Cluster xsi:schemaLocation="http://wso2.com/schema/clustering/config cluster.xsd"
         xmlns="http://wso2.com/schema/clustering/config"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <!--Indicates whether clustering should be enabled or disabled for this node-->
    <Enable>false</Enable>
    <!-- The agent implementation used with cluster framework-->
    <Agent>hazelcast</Agent>
    <!--
     The clustering domain/group. Nodes in the same group will belong to the same multicast
     domain. There will not be interference between nodes in different groups.
    -->
    <Domain>wso2.carbon.domain</Domain>
    <!-- Properties specific to this member -->
    <LocalMember>
        <Host>127.0.0.1</Host>
        <Port>4000</Port>
        <Properties>
            <Property name="backendServerUrl">https://${hostName}:${httpsPort}/services/</Property>
            <Property name="mgtConsoleUrl">https://${hostName}:${httpsPort}/</Property>
            <Property name="subDomain">worker</Property>
        </Properties>
    </LocalMember>
    <!--
        The membership scheme used in this setup. The only values supported at the moment are
        "Multicast" and "WKA"
        1. Multicast - membership is automatically discovered using multicasting
        2. WKA - Well-Known Address based multicasting. Membership is discovered with the help
            of one or more nodes running at a Well-Known Address. New members joining a
            cluster will first connect to a well-known node, register with the well-known node
            and get the membership list from it. When new members join, one of the well-known
            nodes will notify the others in the group. When a member leaves the cluster or
            is deemed to have left the cluster, it will be detected by the Group Membership
            Service (GMS) using a TCP ping mechanism.
    -->
    <MembershipScheme>
        <!-- multicast membership scheme related properties-->
        <Multicast>
            <Group>228.0.0.4</Group>
            <Port>45564</Port>
            <TTL>100</TTL>
            <Timeout>60</Timeout>
        </Multicast>
        <!-- wka membership scheme related properties-->
        <!--WKA>
            <Members>
                <Member>
                    <Host>127.0.0.1</Host>
                    <Port>4000</Port>
                </Member>
            </Members>
        </WKA-->
    </MembershipScheme>
</Cluster>

Cluster Service

The Cluster API which is given as an OSGi service to carbon platform will have the following methods. The users can consume this service using OSGi service reference/discovery mechanism.

public interface Cluster {
    /**
     * Send the given cluster message to the whole cluster
     *
     * @param clusterMessage the cluster message to be sent
     * @throws MessageFailedException on error
     */
    void sendMessage(ClusterMessage clusterMessage) throws MessageFailedException;
    /**
     * Send the given cluster message to a set of members in the cluster
     *
     * @param clusterMessage the cluster message to be sent
     * @param members        the list of members to send the cluster message
     * @throws MessageFailedException on error
     */
    void sendMessage(ClusterMessage clusterMessage, List<ClusterMember> members)
            throws MessageFailedException;
    /**
     * Return the list of currently available members in the cluster
     *
     * @return the member list
     */
    List<ClusterMember> getMembers();
}
Advertisements

About kishanthan

I’m currently working as a Software Engineer at WSO2, an open source software company. I hold an Engineering degree, majoring in Computer Science & Engineering field, from University of Moratuwa, Sri Lanka.
This entry was posted in Carbon5, WSO2 and tagged , , , . Bookmark the permalink.

One Response to Carbon 5.0 Clustering Framework – My Notes

  1. Pingback: Carbon 5 Kernel Architecture | Kishanthan's Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s