Job Queue problem

[jeff_diademi]

Hello,
Adding a job with createHttpJob function is not working properly. The error message is "Bad response from Job Queue server to a createHttpJob request. QLocalSocket: Remote closed."
$job->createHttpJob("http://esop-pre.europe.panorg.com/testing/ifx_box.php", $params, array("predecessor"=> $predecessor));

It seems that this is happening when $params is quite big, however I don't think that there is limit?

Here is the outpout of the job queue server:

[JobQueueExt 27.05.2010 11:48:00 DBG3] Creating a queue connection for binding unix://jobqueue.sock
[JobQueueExt 27.05.2010 11:48:00 DBG3] Creating connection for binding unix://jobqueue.sock
[JobQueueExt 27.05.2010 11:48:00 DBG3] Attempting to connect to binding unix://jobqueue.sock with timeout of 10 seconds
[JobQueueExt 27.05.2010 11:48:00 DBG3] Connection succeeded on /usr/local/zend/tmp/jobqueue.sock
[JobQueueExt 27.05.2010 11:48:00 DBG3] Connected to binding unix://jobqueue.sock
[JobQueueExt 27.05.2010 11:48:00 DBG3] Creating new HTTP job for http://esop-pre.europe.panorg.com/testi ... ipment.php
[JobQueueExt 27.05.2010 11:48:00 DBG3] Written 641:{"app_id":"HMS","cmd":"createHttpJob","script":"http://esop-pre.europe.panorg.com/testing/ifx_shipment.php","vars":{"packid":"LUX10PCH7728696","issued":"2010-05-19 17:21","totalcase":23,"cweight":289,"gweight":288.8,"ccode":"PA-MOWFS ","cname":"OOO FAKTOR SPROSA","caddress1":"BOLSHOY TOLMACHEVSKY PEREULOK, 5","caddress2":"BLD.4","caddress3":"","ccity":"MOSCOW","cpostcode":"119017 ","ccountry":"RU ","scode":"LC02 ","sname":"LGI HUENXE","saddress1":"WERNER-HEISENBERG-STR. 1","saddress2":"","saddress3":"","scity":"HUENXE","spostcode":"46569 ","scountry":"DE ","ordernumber":"0001073742"},"options":{}} over the jq connection.
[JobQueueExt 27.05.2010 11:48:00 DBG3] Read 3:426 from the jq connection.
[JobQueueExt 27.05.2010 11:48:00 DBG3] Created a new HTTP job for url http://esop-pre.europe.panorg.com/testi ... ipment.php with id 426
[JobQueueExt 27.05.2010 11:48:00 DBG3] Creating new HTTP job for http://esop-pre.europe.panorg.com/testing/ifx_box.php
[JobQueueExt 27.05.2010 11:48:00 DBG3] Written 1726:{"app_id":"HMS","cmd":"createHttpJob","script":"http://esop-pre.europe.panorg.com/testing/ifx_box.php","vars":{"SUMMARY":{"boxid":"SUMMARY","gweight":288.811,"quantity":0},"N820L4J1A":{"boxid":"N820L4J1A","gweight":12.557,"quantity":23},"N820L4J1B":{"boxid":"N820L4J1B","gweight":12.557,"quantity":23},"N820L4J1C":{"boxid":"N820L4J1C","gweight":12.557,"quantity":23},"N820L4J1D":{"boxid":"N820L4J1D","gweight":12.557,"quantity":23},"N820L4J1E":{"boxid":"N820L4J1E","gweight":12.557,"quantity":23},"N820L4J1F":{"boxid":"N820L4J1F","gweight":12.557,"quantity":23},"N820L4J1G":{"boxid":"N820L4J1G","gweight":12.557,"quantity":23},"N820L4J1H":{"boxid":"N820L4J1H","gweight":12.557,"quantity":23},"N820L4J1J":{"boxid":"N820L4J1J","gweight":12.557,"quantity":23},"N820L4J1K":{"boxid":"N820L4J1K","gweight":12.557,"quantity":23},"N820L4J1L":{"boxid":"N820L4J1L","gweight":12.557,"quantity":23},"N820L4J1M":{"boxid":"N820L4J1M","gweight":12.557,"quantity":23},"N820L4J1N":{"boxid":"N820L4J1N","gweight":12.557,"quantity":23},"N820L4J1P":{"boxid":"N820L4J1P","gweight":12.557,"quantity":23},"N820L4J1Q":{"boxid":"N820L4J1Q","gweight":12.557,"quantity":23},"N820L4J1R":{"boxid":"N820L4J1R","gweight":12.557,"quantity":23},"N820L4J1S":{"boxid":"N820L4J1S","gweight":12.557,"quantity":23},"N820L4J1T":{"boxid":"N820L4J1T","gweight":12.557,"quantity":23},"N820L4J1U":{"boxid":"N820L4J1U","gweight":12.557,"quantity":23},"N820L4J1V":{"boxid":"N820L4J1V","gweight":12.557,"quantity":23},"N820L4J1W":{"boxid":"N820L4J1W","gweight":12.557,"quantity":23},"N820L4J1X":{"boxid":"N820L4J1X","gweight":12.557,"quantity":23},"N820L4J1Y":{"boxid":"N820L4J1Y","gweight":12.557,"quantity":23},"fileref":"LUX10PCH7728696"},"options":{"predecessor":426}} over the jq connection.
[JobQueueExt 27.05.2010 11:48:00 DBG3] Read from the jq connection.
[JobQueueExt 27.05.2010 11:48:00 WARNING] Response from JQ server was empty. QLocalSocket: Remote closed
[JobQueueExt 27.05.2010 11:48:00 ERROR] Bad response from Job Queue server to a createHttpJob request. QLocalSocket: Remote closed.
[JobQueueExt 27.05.2010 11:48:00 DBG2] Disconnecting connection from jq server

If you have any idea, don't hesitate.
Any help would be appreciated,

Thanks.

[massimilianoc]

I have forwarded your post to the R&D team for analysis.

What is supposed to be the output of the script?

[jeff_diademi]

Well this is simple,
the below script is working with $limit = 25 but failed with $limit = 26

Obviously, there is a limitation of the size of the array...

<?php
try
{
$job = new ZendJobQueue("test");
$test = array();
$limit = 26;
for($i=0;$i<$limit;$i++)
{
$test["a$i"] = "abcdefghijklmnopqrstuvwxyz";
}
print_r($test);
$job->createHttpJob("http://esop-pre.europe.panorg.com/testing/log.php", $test);
echo "Success";

}
catch (Exception $e)
{ echo "Error";
}
?>

[terry_blackburn]

Is this a dead topic? I am encountering the same issue. It would be nice to know what the limits are. I have a situation where I am generating hundreds to thousands of pdf files in a batch. I have to pass the database ID (a 26 character timestamp) of each document to print in an array. If the array contains more than around 1,500 entries, the ->createHttpJob fails with the 'Remote Closed' error documented above. Below 1,500 the batch job starts, but terminates after processing fewer than 600 documents, and I can find no error to indicate why (I'm suspecting memory issues?). I finally took and chunked the array and submitted batches of 100, and it now works great, but I don't appreciate the trial and error nature of getting this up and running in production. A little information goes a long way.

[mkherlakian_zend]

Hi,
I am sorry to hear you had to resort to trial and error to resolve this problem. I just saw this post and did some testing:

For your first concern, about the limitation of the parameter size, it is set to 65535 bytes as a maximum value. Therefore the serialized length of the array you send to Job queue should be smaller that that threshold.

As for the second one, the JQ daemon only opens an http connection to the remote server that is to execute the job at the end. Therefore it is subject to the limitations set on that server - memory_limit, max_execution_time etc. You will most than likely find the answer in the logs of the worker server, i.e. the one you are submitting your jobs to.

Please let me know if I can be of any further assistance.

[terry_blackburn]

Thank you. That is very helpful. Is this information documented someplace? The only documentation I've run across is http://files.zend.com/help/Zend-Server/ ... queue.html, and while it's a good place to start, I find this style of documentation a bit cryptic.

Now, a question about the max_execution_time. My reason for starting a batch process is specifically for long-running functions. This particular application, the user selects a batch of database records to generate pdf files and email them. As I mentioned before, these can number in the thousands. The time limit I have set for interactive processing is not going to be sufficient to handle batch processes.

I have processes that run upwards of an hour that are submitted from the command line. I'm working on an i5 and frequently submit batch PHP jobs from CL by calling QP2SHELL and executing PHP-CLI (I've built a frontend command RUNPHP that accepts the script name and up to 50 parameters). These jobs don't seem to be subject to the execution time limit. I certainly understand the need to limit execution time in an interactive environment, but I can't have the payroll check generation process terminated after 30 seconds because it wouldn't have created my check by that time!

What are your thoughts? I could back-door the system and use my own batch submission tool, but I'd prefer to work within the standard framework where possible.

[mkherlakian_zend]

I quickly scanned the JQ doc, and I didn't see a mention of the limitation of size in the parameters.
Your use case for job queue is exactly what the component is intended for, long running tasks. However depending on the architecture used, you would either use the same server for processing interactive and queued jobs or you'd use separate servers. In the latter case, you'd use a different set of configurations for batch processing servers. In the i world however, you'd probably be processing both on the same server, and in order to accomplish that you have to override the default configured values for max_execution_time and memory_limit values locally, in your batch script.
So in the script that processes your pdf's and email, somewhere in the beginning, you could do:

Code: Select all

set_time_limit(0);


Which sets an unlimited amount of time for the script

and

Code: Select all

ini_set('memory_limit', '256M'); //sets max memory to 256M


This will control limits locally for that particular script only. Does that all make sense?

References:
[url]http://php.net/manual/en/function.set-time-limit.php
[/url]
http://www.php.net/manual/en/ini.core.php#ini.memory-limit

[terry_blackburn]

perfect! works great!. Now, next issue :(

The interactive session submits the requests to generate a couple of thousand PDF files in batches of 100. The last createHttpJob calls an i5 CL program to move those PDF files to the user's folder in QNTC. This program includes a 5 minute dealy (DLYJOB(300)) so it doesn't start trying to move things before they're all done being generated. Watching things execute in WRKACTJOB, I see the move job come up and go to a delay status. One minute later, another one pops up. A minute later another one pops up. Eventually there are a total of 6 jobs, all running the same request. The delay on the first job expires and it begins processing the move request. It completes, and one by one the other requests time out and disappear. However, they also each try to execute the move, but there's nothing to do so they quietly and quickly end.

The Zend Server Monitor only shows the one move job, it sits around and waits and eventually displays a status of 'failed', even though everything I expected to see completed normally (and more!) and eventually all queued instances complete without error.

Besides the obvious issues (if two of those jobs were to become active simultaneously, one or both of them would fail. Then the subsequent queued jobs would come active, contributing to the accumulating mayhem) is there something I am doing that is causing this duplication of jobs? Here's the submit code...

Code: Select all

$id = $q->createHttpJob("$pathOnServer/ePAmassMove.php",
                  array('from' => $arg2, 'to' => $arg3),
                  array('name' => 'massMove'));
echo "<br>Job $id submitted to move files to user folder...";


and the entire mass move

Code: Select all

<?php
   set_time_limit(0);
   $params = ZendJobQueue::getCurrentJobParams();

//   error_log(print_r($params,true));

   $cmdstr = "call cpy2qntc parm('{$params['from']}' '{$params['to']}')";
   error_log($cmdstr);
   if (!$i5connection = i5_connect("myi5","MyUSER","myPasswd")) die('i5 connection failed');
   $program = ('*LIBL/QCMDEXC');
     
   $plist = array (
   array ( 'name' => 'command', 'io' => I5_IN, 'type' => I5_TYPE_CHAR, 'length' => strlen($cmdstr) ),
   array ( 'name' => 'length', 'io' => I5_IN, 'type' => I5_TYPE_PACKED, 'length' => '15.5' )
   );
   
   $inParms = array ('command' => $cmdstr, 'length' => strlen($cmdstr));
   $outParms = array ('command' => 'command', 'length' => 'length');
      $prepare = i5_program_prepare ( $program, $plist );
   $ret = i5_program_call ( $prepare, $inParms, $outParms );
   if (!$ret):
      $error = i5_error($i5connection);
      $msg = i5_errormsg($i5connection);
      error_log(print_r($error,true).'='.$msg);
      error_log("<br>...mass move failure...<br>");
      error_log(print_r ($inParms,true));
   endif;
   i5_program_close ( $prepare );
?>

the call to QCMDEXC executes the cpy2qntc CL program that includes the DLYJOB(300) command. Yes, I suppose I could move the delay into the PHP program, but I'm trying to reuse a piece of legacy CL code. Also, if the system call takes more than a minimum amount of time to execute, is there still risk that additional attempts will be requeued by the server?

[mkherlakian_zend]

Hi,
Is it possible that the move job is being called more than once, inside a loop for instance? If not the other possibility would be that job queue is detecting a failure (timeout or else, and is not returning the correct status). It is always good practice to set a job's status at the end of execution, using:

Code: Select all

ZendJobQueue::setCurrentJobStatus(ZendJobQueue::OK);


However, more generally, the issue is that you have a dependency between jobs. The move job shouldn't execute before the last 100 pdf processing job has completed. Job queue actually allows setting dependencies for exactly that situation:

Code: Select all

//Job to create PDF
$jq = new ZendJobQueue();
$jobId = $jq->createHttpJob('$pathOnServer/createPdf.php', array('key' => 'value'));


And then the move process:

Code: Select all

$q = new ZendJobQueue();
$id = $q->createHttpJob("$pathOnServer/ePAmassMove.php",
                  array('from' => $arg2, 'to' => $arg3),
                  array(
                          'name' => 'massMove',
                          'predecessor' => $jobId
                  ));

So $jobId would be the id of the last job submitted.

[terry_blackburn]

OK, thanks, that's working better. I find the requeueing issue to be baffling. I do a lot of cross environment processing on the i5. I submit batch php jobs from CL, I schedule them off of the native job scheduler. I call CL functions from PHP. I have a CL command RUNPHP that allows me to execute PHP scripts and pass parameters from the i5 command line. The fact that a long running call to a native CL procedure in a queued job can result in a PHP timeout and subsequent requeueing of a job seems like a problem to me. It at least needs to be a configurable, controllable feature. In this particular case, if I use i5_program_call to execute a long running procedure and PHP takes it upon itself to requeue the job, I get multiple procedures now trying to access the same resources and stuff starts blowing up all over the place.

I'm slowly finding my way around the Zend Server documentation and finally found the detailed documentation of the CreateHTTPJob method. I have not, as of yet, found a function that allows me to tell the Job Queue Daemon to chill while this non-PHP function runs inline for a while. Suggestions? Am I missing some important concept that I should be taking advantage of, rather than identifying it as a problem? I'll be the first to admit that my 30+ years experience in the S/38-AS/400-iSeries-i5 family may have predisposed me to take a very narrow view of queued job processing!