Saturation / very low access of Zend Server

[m-poncelin]

Hello everybody (sorry for approximately english),

since the installation of our new Power7 - V7R1M0 (to replace a i5 V5R4M0 platform) and the installation of Zend Server 5.6 (to replace Zend Core) in CE edition 3 weeks ago, we have a recurrent problem of our web server.

Some times after start the Zend Server subsystem and the zendsvr http server, they appears a phenomena of saturation and very low time respond of the server. We MUST restart the subsystem and the server to retrieve a normal run. The problem doesn't appears every day, somedays they are no problem, or 1, or many times in one day (all the 45-60 minutes). When the problem appears, they are NOT message in QSYSOPR message queue and no high CPU using in ZENDSVR subsystem, QHTTPSVR subsystem or any subsystems. Also, they are not spooled files generated by Zendadmin profile who contains problems or explication to justify what happened.

We open a support cases for Zend and we made verification of configuration in fastcgi.conf (like the LANG change in fr_FR), level of PTF and they are no problem for us.

We have in our Zend Server a INTERNET web site (available in the web) with approximately 5000 connections per days and many web services (with 1500 requests per day). The server listen the 80 (http), 8000 (webservices) and 443 ports (https, SSL is active and check the good certificate). Every pages in the website or script in webservices do a connection to the Power7 and use the XMLToolikit to call programs or do SQL request. We NEED to have a constant and good availability.

We try to add configurations in httpd.conf like the configuration we have in the httpd.conf of our old Zend Core. We add MaxCGIJobs and StartCGIJobs to try to resolve the problem. After that, it apperas less but continue sometimes and with a high number in this two directives, no changes appears realy. We see configuration in fastcgi.conf like DynamicServers, is a track ? We see too configuration in the HTTPAdmin interface of IBM i (available by the 2001 port), try to changes configuration in CGI or requests properties but no effect.

Any ideas or suggestions to a special configuration of a internet-good availibility in the web ? Today we are in the wall because we can not to find the reason of the problem and also a solution to resolve that.

Than you very much for your help, we can send any configuration files or other informations if you need.

[aseiden]

First, I don't think the dynamic FastCGI setting would help. Static has worked best for me.

Can you tell if the slow response comes from Apache, PHP FastCGI, or from within your scripts, such as when accessing DB2, program calls, or IFS reads/writes?

To find out: If you have Zend Code Trace available, can you see a problem there? In other words, is the problem is occurring within PHP, and what might it be?

-- Alan Seiden

[m-poncelin]

Hi aseiden and thank you for your response.

For the time being, the problem doesn't appear since saturday afternoon but i monitor the webiste regularly.

When the problem appears next, i'll try to see logs, jobq, etc. to see it's a Apache problem, php or other. We do a simple php page and simple html page and we'll try to acess this when a phenomena of new saturation appears. Do you have an idea of a special place to see a problem (particulary jobq, subsystem or IFS logs's folders etc.)

Unfortunately, we have a simple CE version of Zend Server, so we don't have the Zend Code Trace. Maybe we buy a licence key to have this functionnality

[m-poncelin]

Hi aseiden,

i post that because the phenomena appears today in 11h AM.
When that happened, we search spool files, message queues, apache logs, error_log, jobs queues, CPU usage etc. but NO particulary problem. We just see in the logs of one job in QHTTPSVR error messages from a SQL request, do you think a problem with SQL can make this phenomena ?

We launch a php script with a simple echo 'Hello Wolrd' (no program calls, no SQL request) and a html script with a simple 'Hello World' too but we can't access to this two script so we think the problem is in Apache process.
The server doesn't respond when the phenomena run, we can see some (but no many) access in the access_logs apache, particulary a 502 error code.

With this information, do you think it's a Apache problem ? Do you think changing configuration in the HTTPAdmin IBM interface in Proxy, System Ressources or other can change the problem ?

Thank you very much for your response.

[aseiden]

Hello--

You mentioned the Proxy configuration. Are you using any sort of proxy now?

And to make sure, did you check /usr/local/zendsvr/var/log/php.log?

Alan

[m-poncelin]

Hello,

Yes we use Proxy directives (ProxyPass and ProxyPassReverse), see the attachment file (it's our httpd.conf).
We use that because a client send us in a webservice an incomplete request (no Content-Length) so we have a 411 Apache error and we open the 8000 port and redirect the request to that port to complete the request. When the phenomena comes, we see in the Apache access logs many 502 error codes concerning the URL and scripts redirect by the Proxy directives but we don't know if it's error causes the problem or it's a consequence of one another problem.

In the php error log, we see messages 'PHP Warning: PHP Startup: Unable to load dynamic library etc.' 12 February when the phenomena comes but we see also this messages in other hours and the server run correctly.

No problems comes Monday, one time Tuesday morning. Do you think it's a problem of high number of requests of simultaneous request ? We explored this track but our conclusion are the problems doesn't comes of high number requests because we don't have more request Tuesday morning than Monday entire day.

[aseiden]

Hi,

I understand that when this phenomenon occurs, even a "hello world" script will not work.

I'm curious, though, about the SQL error you mentioned: "We just see in the logs of one job in QHTTPSVR error messages from a SQL request."

What SQL error do you see?

Alan

[m-poncelin]

Hi and sorry for the very late response but we made changes in our configuration and wait many days to see the situation before respond

First of all, thank your very much for your help and for your active response.

The logs in the job in QHTTPSVR subsystem came from an older Net.Data pages (YES, we have alwalys Net.Data pages but just for redirection, we'll change this situation and made the redirection by Apache soon)

The main modification in our httpd.conf is the directive ThreadsPerChild. In fact, in the interface HTTP Admin of iSeries (available by the 2001 port), there is an option 'Real Time Server Statistics'.

There are two importants informations :
-Active threads
-Idle threads

The value default is 10. When the saturation phenomena appeared, the value of Active threads is 10 and no change. We high this value and the problem does'nt appears for 10 days, and we see sometimes there are more 10 active threads.

We don't very known what is realy threads, but there are important in the good run of our web applications. We see too when a call program by php generate an error (an iSeries error like CPFXXXX), this error solicits a thread. Do you know anything about that ?

[aseiden]

Do the IBM i CPFXXXX errors produce an inquiry message on DSPMSG QSYSOPR? If so, these can cause your application to hang. PHP will wait until the message is answered. It can affect other users, not only the person whose job caused the error.

I suggest that you consider adding a system reply list entry to automatically answer such CPF, MCH, etc. messages.

Let me know if this makes sense.

Alan