Creating a service registry with WildflyAS 8.2.0 and Zookeeper for microservices discovery.

Introduction

The last couple of months in my new job we are working in the definition of a rather complex system.Again all the hype words are there , like cloud , service registry/discovery, microservices, containers(docker/vagrant) etc. Among the tasks of our team, we had to evaluate registry service/discovery services solutions like Zookeeper,etcd,Consul and Netflix Eureka. The precondition was that all microservices should be deployed and discovered as rest endpoints. So in my setup I used Wildfly 8.2.0.Final to deploy my ReST JAX-RS 2.0 web services and then exploiting the power of EJB 3.1 specification to automate the registration of the endpoints to the Zookeeper. Also everything was implemented of course using JEE7 and in the source code I will try to show case all the new cool features like the new CDI Eventing mechanism and how it can be used in order to imitate an asynchronous light weight deployment system. Also the powerful @PostConstruct annotation.

It took me quite a while to reach to a working project, mostly due to the fact that Zookeeper doesn’t have that extended documentation. That included a lot of source code reading. In this blog entry I will try to introduce you to the concepts of why in cloud services registries are important and how they fit in to the big picture. From the tools I will try to give the pros and cons of each according to my needs and the decision taken. So let’s start,I hope you will enjoy it. The source code will be available in Github

Keywords and Tools

DevOps,WildflyAS 8.2.0, Apache Zookeeper,Apache Mesos,Marathon,Cloud,Elasticity,Scalability

Tools and Environment setup

• Wildfly 8.2.0. Final: Java EE7 certified application server by RedHat Inc.
• Apache ZooKeeper: A coordination and general purpose registry service  Apache Foundation.
• Apache Mesos : A cluster management solution for big datacenters,an Apache Foundation project.
• Marathon : A cluster-wide init and control system for services in cgroups or Docker containers by Mesosphere.
• JDK 1.7 : Java Development Kit 7 by Oracle Corp.
• IntelliJ||Netbeans||Eclipse: IDE for Java chose your own  flavour.
• Ubuntu server: Operating system with no X server by Canonical.
• VirtualBox : A virtualisation tool by Oracle Corp.
• Docker : A lxc creation and configuration tool by Docker Inc.

I will not delve into the details of each of these products cause there is a wealth of documentation on how to administer them and extend them.I will only compare them, cause that will help you to understand my final decisions.

For the necessary environment to work you need to setup a cluster of servers either on your favour cloud provider or using virtual box. You need to install zookeeper and apache mesos in case you want to play with full cloud solution emulation. The two fundamental parts of this tutorial are, WildflyAS 8.2.0 Final and Zookeeper. On how to setup them please read my previous blog post on how to setup a cluster. I will not dive into that I will make the assertion that you have the environment up and running.

Service Registries/Discovery

In the era of cloud and all this IoT(Internet of Things) era the service registration and service discovery is a key point for the successful implementation of these concepts. The tools to be used in order to implement these ideas are divided in two to major categories:

• General purpose registries,like Zookeeper.
• And tailor made solutions like Consul/etcd and Netflix Eureka.

The first one offers a rich programming API where the developers can create full functional registries and service discovery, providing also guarantees in the distributed world.The later are offering key value stores to store and query fast services registered there. Both Consul,etcd and Netflix Eureka offer more or less the same capabilities.The above are nice in case you don’t have really complex

In our system we opt for Zookeeper. At first glance it looks really difficult because it has an extensive API and the developer must take care of many things,but has full control and power over the tool. Is really low level and doesn’t offer any abstractions. We used Apache Curator, which is a project that simplifies the access to Zookeeper for new developers.

One of the most difficult part is that ZooKeeper can be run as part of an application server , since the architecture it has a rather complex and the software itself triggers threads. So in the Thread context of an application server that would create a hell for the developer to monitor correctly the threads.

Block Diagram

The solution will look like the following diagram:

The code will be available on Github really soon.Now to the details of diving in the code.

Registration of endpoints

In order to register the endpoints we will leverage the power of @PostConstruct annotation when applied in a rest endpoint.Like in the following class:
EventProducerBean

@Startup
@Singleton(name="eventProducerBean")
@Path("/discover-leader/{system}")
public class EventProducerBean implements Serializable{
    private static final long serialVersionUID = 5693753599350013603L;
    private static final Logger logger = LoggerFactory.getLogger(EventProducerBean.class);

    @Inject
    private ZookeeperService zookeeperService;

    @Inject
    @Any
    Event<ServiceInstance> registerEvent;

    @PostConstruct
    private void registerEndpoint(){
        for(Annotation annotation : this.getClass().getDeclaredAnnotations()){
            if(annotation.toString().contains("Path")){
                logger.info("[Action]->Found annotation:{}",annotation.toString());
                zookeeperService.registerService("/services" + annotation.toString().substring(annotation.toString().indexOf("/"), annotation.toString().lastIndexOf(")")), null);
            }
        }
    }


    @GET
    @Produces("application/json")
    public String createRegisterEvent(@PathParam(value = "system") String system) throws ClassNotFoundException,IOException{
        logger.info("Started creating the event.");
        ServiceInstance serviceInstance = new ServiceInstance();
        serviceInstance.setDescription("Register Event");
        serviceInstance.setVersion("1.0");
        serviceInstance.setEndpoint(system.equalsIgnoreCase("marathon")?"/marathon/leader":"/mesos/app/cdi-events");
        Deployer deployer = new Deployer();
        ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
        ObjectOutputStream outputStream = new ObjectOutputStream(byteArrayOutputStream);
        outputStream.writeObject(deployer);
        serviceInstance.setData(byteArrayOutputStream.toByteArray());
        registerEvent.fire(serviceInstance);
        logger.info("Event fired:{}",serviceInstance.toString());
        return "{action:done}";
    }
}

The above class contains 3 of the most awesome features of JEE7. What we are doing are the following:

1. Declaring a singleton bean and ask to be initialised in startup.
2. Using @PostConstruct we are registering the endpoint to the zookeeper.
3. We expose the path to be served from the @GET annotated method.
4. We do the registration by also using the CDI Event mechanism, where we fire and event and then there is a class responsible for consuming the event.

The above class also fires the event. There is a consumer class which is also an EJB. As you have already figure it out I am really big fun of these so cool new feature in the JEE6/7 world called CDI Events. The big benefit out of it is that you can create small integrated ESB’s that are far lightweight than a full blown JMS implementation but also allows the asynch model to work. The consumer class is rather simple:
EventConsumerBean

@Stateless
public class EventConsumerBean implements Serializable{
    private static final long serialVersionUID = 4775947387215201659L;
    private static final Logger logger = LoggerFactory.getLogger(EventConsumerBean.class);

    @Inject
    private ZookeeperService zookeeperService;

    /**
     * The observer method for registration requests.
     * @param serviceInstance : The instance data to save in Zookeeper.
     */
    public void onRegisterEvent(@Observes ServiceInstance serviceInstance) {
        try {
            ObjectInputStream objectInputStream = new ObjectInputStream(new ByteArrayInputStream(serviceInstance.getData()));
            Deployer deployer = (Deployer)objectInputStream.readObject();
            logger.info("Object read:{}",deployer.toString());
            zookeeperService.getMasterEnsemble(serviceInstance.getEndpoint());
        } catch (IOException | ClassNotFoundException e) {
            logger.error("Failed to do stuff:{}",e);
        }
    }
}

In the above class the @Observes implies that this method is listening for events of the specific type.In case you want to actually do something more cooler than that, you should use the @Qualifier annotation and create really fine grained events and producers/consumers for them. This goes out of the scope of this small project.

The above class will request from the Zookeeper master to fetch the master quorum of various registered/clustered services.

The zookeeper part is rather simple. You need max 2 class. A connection factory and a service class:

ZookeeperConnectionFactory

@Singleton
public class ZookeeperConnectionFactory implements Serializable{
    private static final Logger logger = LoggerFactory.getLogger(ZookeeperConnectionFactory.class);
    private static final long serialVersionUID = -5349403327084706445L;


    private String connectionString = "ip1:2181,ip2:2181,ip3:2181";

    private int sessionTimeout = 15000;

    private ZooKeeper zooKeeper = null;

    public ZookeeperConnectionFactory(){
        super();
    }

    public ZooKeeper getConnection(){
        try {
            zooKeeper = new ZooKeeper(connectionString,sessionTimeout,new CdiWatcher());
            Thread.sleep(5000);
        } catch (IOException | InterruptedException e) {
            logger.error("Error:",e);
        }
        return zooKeeper;
    }
}

ZooKeeperService

@Singleton
public class ZookeeperService implements Serializable{
    private static final long serialVersionUID = 3607178392179991956L;
    private static final Logger logger = LoggerFactory.getLogger(ZookeeperService.class);

    private ZooKeeper zooKeeper;

    @Inject
    private ZookeeperConnectionFactory zookeeperConnectionFactory;


    /**
     * A method for retrieving the masters registered for a path in a zookeeper ensemble
     * @param path : The path registered.
     * @return A <b>List<String></b> that contains strings of format ip:port
     */
    public List<String> getMasterEnsemble(String path){
        zooKeeper = zookeeperConnectionFactory.getConnection();
        List<String> masterEnsemble = new ArrayList<String>();
        try {
            List<String> children  = zooKeeper.getChildren(path,false);
            for (String child : children) {
                String readData = new String(zooKeeper.getData(path+"/"+child,false,new Stat()));
                logger.info("Data read:{}",readData);
                if(readData.contains("master@")){
                    masterEnsemble.add(readData.substring(readData.indexOf("@")+1,readData.indexOf("*")));
                }
            }
        } catch (KeeperException | InterruptedException e) {
            logger.error("Error error:",e.getCause());
        } finally {
            logger.info("[Action]->Closing the connection to zookeeper.");
            try {
                zooKeeper.close();
            } catch (InterruptedException e) {
                logger.error("Error:",e.getCause());
            }
        }
        return  masterEnsemble;
    }

    /**
     * Method for registering for a path.
     * @param path: The path of interest.
     * @return : a <b>boolean</b> indicating the success of the failure of the process
     */
    public boolean registerService(String path,byte[] nodeData){
        boolean succeed = false;
        try{
            zooKeeper = zookeeperConnectionFactory.getConnection();
            String messageCreate= zooKeeper.create(path, null, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
            logger.info("Registering result:{}",messageCreate);
            succeed=true;
        } catch (KeeperException | InterruptedException e) {
            logger.error("Error:{}",e);
        }finally{
            try{
                zooKeeper.close();
            }catch(InterruptedException e){
                logger.error("Error:",e.getCause());
            }
        }
        return succeed;
    }
}

Starting with these 4 classes you can experiment by playing further. Key points if you want to start implementing you own stuff are the following:
1)In order to register endpoints and enable the auto discovery of rest endpoints in WildflyAS you have to comply with the need of RestEasy. You have to extend and leave an empty class that extends Application and has an @ApplicationPath annotation. But be careful. For the scanner to work correctly the class must end with the suffix Application. So names like MyApp don’t work.Names like MyApplication, MyJAXRSApplication are legit and the scanner will scan the packages of your project for rest paths.

Also the @ApplicationPath annotation takes as a parameter, which is actual the root of your rest services. So if the application deploys under the appName and the @ApplicationPath(“services”) is how you have annotated your extended Application class then the to invoke an endpoint you must prefix in your favour tool like this: /appName/services/myEndpoint

Final thoughts

The purpose of this post, is only to show how much the JEE7 echosystem has matured and what tools gives to the developer. WildflyAS 8.2.0.Final is like a swiss army knife that can be used to achieve amazing software solutions. The sole purpose is also to understand that microservices are not something new to us that come from the JEE5/6/7 world. You can label them as you want but correctly architected solutions for “Cloud” architectures were always there in the arsenal of the JEE framework.

Also with the suggested solution and with the power of JEE7 you can forward the same eventing mechanism and events that happen in Zookeeper to your actual application. This allows a more easy integration between two different systems.

Currently I am in the process of cleaning up a bit the code but soon I will post all the source code for the project in my Github account. Dig up for cirix there. Within the next 2-3 days the code will also be available.I would like to thanks my virtual friends @reza_rahman @brunoborges @arungupta for inspiring most of my work. I hope you enjoy reading this stuff and keep javing to the bone.

p.s.: As promised code available in bitbucket: https://cirix@bitbucket.org/cirix/cdi-events.git

github: https://github.com/cirix/cdi-events