Invoking Celery Tasks from Java Application – Part #1

Invoking a celery task from java application is not hassle but not an easy one either.  This java celery integration was  implemented  with the help of a message broker/queue  and  what I chose for this was RabbitMQ.  There are many options out there  for message broker but I opted this as I had used it earlier with celery.

This article is based on the assumptions that  readers have a bit prior experience with celery and RabbitMQ. For a start, we can go through a very very brief introduction. (might write a few articles on celery, if time favours ..)

As we know, celery is task queue which absorbs messsags from message queues (iie here RabbitMQ ) and execute them in celery worker threads. Here, in order to trigger a celery task from java application, what we need to is, to create a rabbitMQ queue and push the messages to this queue from java application, in an appropriate format ie format that celery tasks messages adhere to (http://docs.celeryproject.org/en/latest/internals/protocol.html). Then We need to  define tasks and start  worker thread to execute these tasks as messages are available in this message queue, (as the java application pushes them).

Following are the steps I have done to make it working  :

1. Download java – rabbitMQ client library   from here                                                                                                                    Then extract the zip file and copy all the jar files into the root directory of the project.

2.  Now we need to write the java code to integrate RabbitMQ and push tasks messages to  specific RabbitMQ queues.          This needs to be placed in the root directory of the project where we have just copied jar files into, in the previous step.

Send.java

import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.AMQP;
public class Send {
public static void main(String[] argv) throws Exception {
String QUEUE_NAME = “celery”;
ConnectionFactory factory = new ConnectionFactory();
factory.setHost(“localhost”);
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.queueDeclare(QUEUE_NAME, true, false, false, null);
String message = “{\”id\”: \”4cc7438e-afd4-4f8f-a2f3-f46567e7ca77\”, \”task\”: \”tasks.add\”, \”args\”: [1,2], \”kwargs\”: {}, \”retries\”: 0, \”eta\”: \”2009-11-17T12:30:56.527191\”}”;
channel.basicPublish(“”, QUEUE_NAME, new AMQP.BasicProperties.Builder()
.contentType(“application/json”).contentEncoding(“utf-8”)
.build(), message.getBytes(“utf-8″));
System.out.println(” [x] Sent ‘” + message + “‘”);
channel.close();
connection.close();
}
}

In this, we are sending messages to a queue named “celery” . Also the format of the message is specified in the variable message and from which it is clear that name of the task is tasks.add and arguments are 1 and 2. This is equivalent to calling add.delay(1,2). Hence bow the task message for addition has been sent to the queue celery.

3.  Now we need to define the celery task.                                                                                                                                      In the root directory of the project add following python files.

tasks.py

 

from celery import Celery
app = Celery(‘tasks’)
app.config_from_object(‘celeryconfig’)

@app.task
def add(x, y):
# return x + y
print “haiii”
print x+y

celeryconfig.py

CELERY_IMPORTS = (“tasks”, )
CELERY_RESULT_BACKEND = “amqp”
BROKER_URL = “amqp://guest:guest@localhost:5672//”
CELERY_TASK_RESULT_EXPIRES = 300

4.  Now we can start celery worker to execute the tasks messages

From the root directory of the project open a terminal and run the following command :                                                           celery -A tasks worker –loglevel=info    

here -A stands for application name, since the argument passed in while creating celery instance is                                       tasks ( app =Celery(“tasks”) ), here our application name is tasks.

5. compile and run Send.java to push celery asks to rabbitMQ queue :

 javac -cp rabbitmq-client.jar Send.java                                                                                                                                      java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar Send

Here we use the parameter -cp to keep the required jar files in class path. With this we have successfully pushed the          tasks messages to queue.

6. Now open the terminal where celery worker is started, you would get the following result.

Screenshot from 2016-05-26 17:33:31

 

 

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s