Episode 10: Using the Task Queue Service

Welcome to Episode 10. In this episode, we shall cover the experimental Task Queue Service in Google App Engine. This is an experimental service, which means that it could undergo change in its core functionality from all respects like methods, package names, etc. In any case, we can safely expect that it shall get confirmed in some form or the other in a future release of the Google App Engine.

This episode is sort of a natural progression on our last episode, which covered the Cron Service. To reiterate, the Cron Service was used for background jobs that you wish to perform outside of a user request. They are not typically initiated by a user but instead configured by your application to perform certain periodic tasks like summary reports at end of the day, daily backups, etc. The Task Queue Service provides a facility to process tasks in the background which are :

  • Typically initiated by the user.
  • They can be created internally by the application also to break down a background job into smaller tasks.

Some of the examples of performing a Task in the background include:

1. A user signs up at the website and submits his/her information. On receiving the request, a simple message is displayed back to the user that his/her subscription request has been received and they will hear soon about the status. At the same time, a task can be created that can process this users request, perform some verification and then send out an email to the user if the subscription is setup successfully. So you could have done either of the following:

  • Receive the user details and create a Task. This single task will take care of creating the required records in the database and send out an email.
  • Receive the user details and create a Task. This task will simply create a record in the database and then create another task to send out an email.
  • And so on.

2. An online shopping site could accept the order from a buyer. A task is then created in the system that will process the order. As part of the order processing, once shipping is done, another task is told to update the shipping status. Similarly when the logistics provider gives updates on the shipment locations, a task could be launched to update the shipment route of the package.

To summarize, any background (asynchronous) task can be launched when a user initiates the event or when an application or business event occurs. All these tasks are put into one or more user defined or default queues and are executed by the Google App Engine infrastructure on behalf of your application.

What does a Task constitute? What is a Queue ? Who executes it ?

The official documentation of Tasks is excellent and I suggest reading that in detail. In this episode I will cover just about enough for you to get started on Tasks and then dig deeper into them depending on your needs. So let us first understand what is the basic information that I need let the Google App Engine know about my task. I will take the liberty here to describe all the key concepts via an example that we will build in this application.

We wish to implement the following flow in our application:

1. A user wishes to sign up for your newsletter by providing an email id in a web form provided at our site.
2. The user enters the email id and clicks on sign up (a button).
3. The request is sent to a Servlet that accepts the email id and creates a Task. The response is sent back to the user thanking them for their interest.
4. The Task is then executed independently by the Google App Engine infrastructure and our code inside the Task i.e. checking if the email id is used, etc is verified and then an email is sent.

So, as you can see we have decoupled the background task (in step 4) from the sign up process (step 1, step 2 & step3).

It should now be straightforward to define the key elements:

1. Task : This is the unit of work that we wish to perform. In fact, the actor that will be performing this task is Google App Engine. So when we create a Task, we need to provide a standard way for the Google App Engine to invoke tasks and pass them their payload. It should now be straightforward to see that when we create a task, all we need to tell GAEJ is the URL (where to invoke the task) and the parameterized payload (data). This can be done in a standard fashion that you know. The URL is nothing but the servlet URL that will invoke the servlet that implements the Task. And the parameterized data is nothing but request parameters passed into your servlet so that it can execute accordingly. The data in this case will be nothing but the email id of the user who wants to sign up for your newsletter.

2. Queue : All Tasks when they are created are placed in a queue. They are then executed by the Google App Engine. To help us manage and categorize our tasks, you can define your queues by giving them appropriate names. For e.g. myqueue, emailqueue, etc. What this helps you to do is to place your tasks in the appropriate queue. Few points to note about queues (refer to the documentation for finer details):

  • All queues in your application are defined in a file named queue.xml that is present in the WEB-INF folder of your application.
  • Each queue has a unique name and you can control the rate at which tasks are executed in this queue by the Google App Engine. If you do not specify a rate, then the default rate is 5 tasks/second.
  • There is a default queue for your application named ‘default’ and if you can chose to add your tasks to the default queue. Alternately, you can add them to your application defined queue.

I believe this should be sufficient for us to begin developing a simple flow that will demonstrate how to define a task queue, create a task and then see it getting executed by the Google App Engine.

Task Queue in Action

The diagram below shows what we will implement in the next few sections. The use case is that of an user that wishes to subscribe to our newsletter and how we shall break up the process into the request processing and then the task processing.

Let us break down the above flow into the steps given below and what we shall be developing at each step:

1. In Step 1, the user visits a web page and enters his/her email id to sign up for the newsletter. We shall not be developing a web page over here to keep things simple. Instead we shall be testing it out by directly invoking a servlet that accepts the email id of the user. This is the same thing that you would have normally done  by hooking up the action of the HTML form to that of this servlet.

2. In Step 2, we will be looking at a Servlet whose existence is to do some verification and then create a Task for background processing. This servlet (GAEJCreateTaskServlet) will create a Task in a queue called subscription-queue. As we covered earlier, to create a Task, we need to provide two pieces of information to the Google App Engine so that it can execute it. They are the URL and the Data. The URL (/gaejsignupsubscriber) is going to be that of a Servlet (GAEJSignupSubscriberServlet) that shall be doing the core task. And the data will be what the servlet needs to complete the task. In our case, the data is the emailid request parameter.

3. In Step 3, Google App Engine automatically scans the queues for any tasks that are queued up and picks them up for execution. In our case, it will find a Task instance and execute it by invoking the URL (/gaejsignupsubscriber) and passing it the relevant data i.e. emailid

4. Finally in Step 4, our Servlet (GAEJSignupSubscriberServlet) is invoked and it will complete its task. To keep things simple in our example, it will currently only print out a message. But it should be obvious that core logic associated with the task would have gone into the Servlet here. For our specific case, it would have involved checking if the user has not signed up already, creating a database record and then sending off a welcome email.

Implementing the above flow

To summarize the above steps in terms of what we have to code, here is the list:

1. Code the GAEJCreateTaskServlet that will accept the request parameter and create a Task instance in the subscription-queue.

2. Code the GAEJSignupSubscriberServlet that will be invoked by Google App Engine automatically. We will currently only print out a log statement because the intent is to demonstrate the whole sequence.

3. Configure our queue (subscription-queue) in a file named queue.xml. This file needs to be placed in the WEB-INF folder of your application.

4. Configure our GAEJCreateTaskServlet and GAEJSignupSubscriberServlet in the web.xml file.

Finally, we can execute our application and use the local development server to see the application in action. Users can optionally even deploy it to the Google App Engine cloud if they wish.

So let us get started.

GAEJCreateTaskServlet.java

This servlet accepts our request for subscription. We shall invoke it via the following url : http://appurl/gaejcreatetask?emailid=XYZ.

 

package com.gaejexperiments.taskqueue;

import java.io.IOException;

import javax.servlet.ServletException;
import javax.servlet.http.*;

import com.google.appengine.api.labs.taskqueue.Queue;
import com.google.appengine.api.labs.taskqueue.QueueFactory;
import com.google.appengine.api.labs.taskqueue.TaskOptions;

@SuppressWarnings("serial")
public class GAEJCreateTaskServlet extends HttpServlet {
public void doGet(HttpServletRequest req, HttpServletResponse resp)
throws IOException {

String strCallResult = "";
resp.setContentType("text/plain");
try {
//Extract out the To, Subject and Body of the Email to be sent
String strEmailId = req.getParameter("emailid");

//Do validations here. Only basic ones i.e. cannot be null/empty

if (strEmailId == null) throw new Exception("Email Id field cannot be empty.");

//Trim the stuff
strEmailId = strEmailId.trim();
if (strEmailId.length() == 0) throw new Exception("Email Id field cannot be empty.");
//Queue queue = QueueFactory.getDefaultQueue();
Queue queue = QueueFactory.getQueue("subscription-queue");
queue.add(TaskOptions.Builder.url("/gaejsignupsubscriber").param("emailid",strEmailId));
strCallResult = "Successfully created a Task in the Queue";
resp.getWriter().println(strCallResult);
}
catch (Exception ex) {
strCallResult = "Fail: " + ex.getMessage();
resp.getWriter().println(strCallResult);
}
}

@Override
public void doPost(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
doGet(req, resp);
}
}

 

The code listed below is straightforward to understand. It does the following:

1. Extracts out the request parameter (emailid) and does some basic validation on it.

2. It gets a handle to the subscription-queue through the following statement:

Queue queue = QueueFactory.getQueue("subscription-queue");

3. It adds a Task to the above queue by providing a Task URL (/gaejsignupsubscriber) and Data (emailid parameter). It uses a helper class TaskOptions.Builder to help create the instance of the task. As you can see it provides a url and then the param. The task is created by invoking the add method on the queue.

queue.add(TaskOptions.Builder.url("/gaejsignupsubscriber").param("emailid",strEmailId));

4. For the readers information, I have shown a commented out line

//Queue queue = QueueFactory.getDefaultQueue();

which shows how to get the handle to the default queue in case you wish to place your tasks in the default queue itself.

5. Do not that all the Task Queue classes are experimental and are present in the com.google.appengine.api.labs.taskqueue package. This could change in the future.

GAEJSignupSubscriberServlet.java

This servlet contains the core task logic. This will be invoked by Google App engine if it finds any tasks present in the subscription-queue. If any tasks are there, it will pick them up and invoke the URL mentioned in the Task and pass to it the data present in the Task instance.  The code shown below is straightforward, it simply logs a statement saying that it got invoked and it also logs the email id.

 

package com.gaejexperiments.taskqueue;

import java.io.IOException;
import java.util.logging.Logger;

import javax.servlet.ServletException;
import javax.servlet.http.*;

@SuppressWarnings("serial")
public class GAEJSignupSubscriberServlet extends HttpServlet {
private static final Logger _logger = Logger.getLogger(GAEJSignupSubscriberServlet.class.getName());
public void doGet(HttpServletRequest req, HttpServletResponse resp)
throws IOException {

String strCallResult = "";
resp.setContentType("text/plain");
try {
String strEmailId = req.getParameter("emailid");
_logger.info("Got a Signup Subscriber Request for Email ID : " + strEmailId);
//
// PUT YOUR TASK CODE HERE
//
strCallResult = "SUCCESS: Subscriber Signup";
_logger.info(strCallResult);
resp.getWriter().println(strCallResult);
}
catch (Exception ex) {
strCallResult = "FAIL: Subscriber Signup : " + ex.getMessage();
_logger.info(strCallResult);
resp.getWriter().println(strCallResult);
}
}

@Override
public void doPost(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
doGet(req, resp);
}

}

 

queue.xml

All queues are configured in a file named queue.xml. Google App Engine provides a default queue. This queue is aptly named “default“. But in case you need to define your own queues, which is what we are going to do, we need to define them in a file called queue.xml. This file is placed in the WEB-INF directory of your application. You can also override settings of the default queue by defining it in the file and providing your own values.

Take a look at the queue.xml shown below:

<?xml version="1.0" encoding="UTF-8"?>
<queue-entries>
<queue>
<name>default</name>
<rate>5/s</rate>
</queue>
<queue>
<name>subscription-queue</name>
<rate>5/s</rate>
</queue>
</queue-entries>

In the above configuration, you will find that we have defined our own queue named “subscription-queue”. There is also another element that we have defined for the <queue> called <rate>. This element determines the rate at which you tell Google App Engine to execute tasks. If you do not specify a rate, then the default execution rate is 5 tasks per second. In the above file, we have provided the expression as “5/s”, which reads as 5 per second. Other examples of <rate> expressions are 1000/d (One thousand per day), etc. I suggest to read up the documentation for more examples.

You will also find that we have defined the default queue and we can change the rate if we want. But I have left it as is.

Please make sure that the above file (queue.xml) is present in the WEB-INF folder at the time of deploying the application.

Configuring the Servlets (web.xml)

We need to add the <servlet/> and <servlet-mapping/> entry to the web.xml file. This file is present in the WEB-INF folder of the project. The necessary fragment to be added to your web.xml file are shown below. Please note that you can use your own namespace and servlet class. Just modify it accordingly if you do so. We are defining here both our servlets.

<servlet>
<servlet-name>GAEJCreateTaskServlet</servlet-name>
<servlet-class>com.gaejexperiments.taskqueue.GAEJCreateTaskServlet</servlet-class>
</servlet>
<servlet>
<servlet-name>GAEJSignupSubscriberServlet</servlet-name>
<servlet-class>com.gaejexperiments.taskqueue.GAEJSignupSubscriberServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>GAEJSignupSubscriberServlet</servlet-name>
<url-pattern>/gaejsignupsubscriber</url-pattern>
</servlet-mapping>
<servlet-mapping>
<servlet-name>GAEJCreateTaskServlet</servlet-name>
<url-pattern>/gaejcreatetask</url-pattern>
</servlet-mapping>

Task Execution in Action

I am assuming that you have already created a new Google Web Application Project and have created the above Servlets, web.xml and queue.xml respectively. For a change, we shall be running this episode within our local development server only.

So assuming that all is well, we will run our application, by right-clicking on the project and selecting Run As –> Web Application. Once you see the following message shown below, then the local server is ready to accept requests.

Nov 24, 2009 5:11:33 AM com.google.apphosting.utils.jetty.JettyLogger info
INFO: jetty-6.1.x
Nov 24, 2009 5:11:40 AM com.google.apphosting.utils.jetty.JettyLogger info
INFO: Started SelectChannelConnector@127.0.0.1:8080
The server is running at http://localhost:8080/

Follow the steps given below:

1. Launch the browser on your local machine and navigate to http://localhost:8080/_ah/admin. This is the administrative console of the local development server.

2. Click on Task Queues to view the current task queues that are configured in your application. You should see the screen shown below, which shows that we have configured two task queues: default and subscription-queue.

You will notice that both of the queues currently do not have any tasks since we have not created any.

3. The next step is to create a task in the subscription-queue. To do that, all we need to do is invoke the following url :

http://localhost:8080/gaejcreatetask?emailid=romin@rocketmail.com

This invokes our GAEJCreateTaskServlet that we have configured. It will create the sample task in the subscription-queue and we will see a message as shown below in the browser:

“Successfully created a Task in the Queue”

4. Click on the Task Queues link again, you will find that there will now be 1 task listed in the subscription-queue as shown below:

5. Click on the subscription-queue link. This will display the task instance as shown below:

6. Since this is the development server, no automatic execution of tasks (this will occur in Google App Engine cloud) will take place and you have to run them manually, by clicking on the Run button. Click that. It will display that there are no more tasks present if the task executes successfully.

7. To check that our task has executed successfully, we can visit the Console tab in Eclipse and you will see the success log as shown below:

Moving on

In this episode, we saw how to split up a process into individual tasks and assign them to the Google App Engine for execution. What we have demonstrated here is a simple flow and the specifics of the configuration. I encourage you to try it out in your applications and as an exercise deploy it to the Google App Engine and monitor it there via the Administration Console.

Do keep in mind that this API is experimental and is likely to change drastically. At the same time, if you have any feedback, it would be nice to pass it along to the Google App Engine team.

Till the next episode, Happy Multitasking!

Read more Episodes on App Engine Services

 

About these ads

About rominirani

Google Developer Expert Cloud 2014. Harnessing the power of software by learning, teaching and developing simple solutions. I love learning about new technologies and teaching it to others.
This entry was posted in Cloud Computing, Google App Engine. Bookmark the permalink.

22 Responses to Episode 10: Using the Task Queue Service

  1. Pingback: Tweets that mention Episode 10: Using the Task Queue Service « Google App Engine Java Experiments -- Topsy.com

  2. pegandstef says:

    Great tutorial!!!

    Thanks a lot!

    Steve

    http://www.sprosys.com

  3. edward says:

    Hi, I’ve tried several of your tutorial episodes … very helpful indeed; need your help though on this task queue service. I found that the task only shows count result on “Run in Last Minute” but the “Task in Queue” has always been empty. I suspect that something went wrong executing these lines:

    Queue queue = QueueFactory.getQueue(“subscription-queue”);
    queue.add(TaskOptions.Builder.url(“/gaejsignupsubscriber”).param(“emailid”,strEmailId));

    Appreciate your comment …

  4. rominirani says:

    Hi Edward,

    I verified the project code with the 1.3.0 release of the Google SDK. If my memory serves me right, with version 1.2.8 of the SDK, what has happened is that your scheduled tasks are run immediately by the Dev Server (local environment). So, there will be no need for you to run the tasks explicitly, they will get executed as soon as they are scheduled (ASAP).

    So I suggest that you do the following:
    1. Keep logger statements in your core Task code.
    2. Once the task is created successfully, the Task code should execute. Console tab will show the logger statements.

    I am also pasting an excerpt from the official documentation on the Task Queue API:

    http://code.google.com/appengine/docs/java/taskqueue/overview.html

    “When your app is running in the development server, tasks are automatically executed at the appropriate time just as in production. However, there are minor differences in behavior between the development server and production that you should be aware of. First, the development server does not respect the “rate” and “bucket-size” attributes of your queues. As a result, tasks will be executed as close to their scheduled execution times as possible, and setting a rate of 0 will not prevent tasks from being automatically executed. Second, the development server does not retry tasks. Finally, the development server does not preserve queue state across server restarts. We hope to implement support for these features in the development server in a future release.”

    To disable automatic execution of tasks, set the “task_queue.disable_auto_task_execution” jvm flag: –jvm_flag=-Dtask_queue.disable_auto_task_execution=true

    Hope this helps.

    Thanks
    Romin

  5. Prasath says:

    Hi,

    The Response time for any request in app engine is only 30 Seconds. If the processing time in GAEJSignupSubscriberServlet is more than 30 seconds, I am getting the below error :-

    Request was aborted after waiting too long to attempt to service your request. Most likely, this indicates that you have reached your simultaneous dynamic request limit. This is almost always due to excessively high latency in your app. Please see http://code.google.com/appengine/docs/quotas.html for more details.

    How to solve it ?

  6. saggy says:

    thanks a lot for these kind of tutorial which save many precious hours and help newbies like me to understand GAE apps better and thoroughly.Thanks once again.

  7. Yury says:

    Can you explain please, how can I define, that tast is ended.

    • rominirani says:

      Hi Yury,

      When a task is launched, it is your code that is executing. So if you finish your execution with the task timeout period — then you can consider that the task is complete. Maybe some log statements could help you.

      In case your task does not complete with the task time limit, then it will timeout, an exception will be thrown. So in that scenario, you can assume that your task has not completed since it was busy when it ran out of time.

      Thanks
      Romin

      • Yury says:

        Thanks for the answer. But I have a question. I will make a lot of changes in DB (10 Tasks for example) and then make caching. How can I define, that all tasks are ended and I can start caching?

      • rominirani says:

        Hi Yury,

        This will be completely dependent on your logic. For e.g. if you are going to run a long running process in the tasks and if it is likely that the task might timeout before you complete the work, then you should be doing something like this:
        1. Keep a track of your task progress. For e.g. if you are processing 100s of records, keep a record count.
        2. If you run out of time, an exception could get thrown. Catch it and save the count somewhere. Another alternative could be to relaunch another task with the recordid from where to start the work.

        Continue the above pattern till the work is all complete. I am sure there are other similar ways but what I have given above is a general way to go about planning your tasks to work around the time limit.

        Thanks
        Romin

  8. Yury says:

    Thanks for the answer and for the article.

  9. Sathish says:

    Hi thank you for your post. I struggled to use task queue concept in my application. With help of your sample program i understand the concept. Thank you very much

  10. Bruno says:

    With Google SDK 1.4.x, com.google.appengine.api.labs.taskqueue.Queue is deprecated.

    Example with 1.4 version:

    import com.google.appengine.api.taskqueue.Queue;
    import com.google.appengine.api.taskqueue.QueueFactory;
    import static com.google.appengine.api.taskqueue.TaskOptions.Builder.*;

    Queue queue = QueueFactory.getQueue(“subscription-queue”);
    queue.add(withUrl(“/gaejsignupsubscriber”).param(“emailid”,strEmailId));

  11. Stewie says:

    It looks like Task Execution is 10minutes per task:

    http://code.google.com/intl/en/appengine/docs/java/taskqueue/overview.html#Task_Execution

    Has this been upgraded from the old 30 seconds limit?

  12. sonphm says:

    It’s so Great! Thank you so much!!

    sonphm

  13. Hi rominirani,

    I’ve been having this message from App Engine (online) Log page:

    Request failed because URL requires user login. For requests invoked within App Engine (offline requests like Task Queue, or webhooks like XMPP and Incoming Mail), the URL must require admin login (or no login).

    Do you know why? If yes, could you please paste or write what i have to do with that?
    Thank you so much :-)

    • rominirani says:

      Hi Stefano,

      It is not clear from your message what is causing this error. Are you trying to access the web application or are you trying to directly access the URL endpoints for services like XMPP, Mail, Task Queue, etc ?

      This error is likely due to your security constraints in the web.xml ? Check those out and see if you have protected those URL patterns so that people can access the url directly without being authenticated. What this also means is that you will need to be logged into your particular Google Account for the App Engine application to be able to execute those URLs directly too.

      • I removed the web-constraints inside the web.xml, and now the Tasks can be executed by all users. However it’s strange because if i want to use secure URLs, the Task Queue doesn’t work with the above config as well:

        tasks
        /tasks/*

        *

      • I posted an xml script of web.xml and this website didn’t allow me to write that. I’m sorry. anyway, it doesn’t work either with role-name * instead of admin

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s