Character set issues - random garbage characters returned

General discussion on Zend Server for IBM System i

Character set issues - random garbage characters returned

Postby longnoz on Wed Dec 13, 2017 9:00 pm

Hi

We are working on an ibm i where the QCCSID value is 65535. And the database files have a CCSID of 500 (French). Our web pages that are served by the php engine, using the ibm i toolkit to access the database (db2_connect), are returning properly formatted data about 75% of the time. The other 25% of the time, if a database field has a character with a French accent, the field is returned to the browser with some garbage characters appended to it. The garbage characters are random, but sometimes contain data such as "text/html" or "charset=UTF-8". Even more frustrating is that a page can load perfectly, then you press F5 or CTRL-F5 refresh and the some of the fields are corrupted.

We have verified that all our html is presented to the browser with this header:
<META http-equiv="Content-Type" content="text/html; charset=UTF-8">

And we do this at the beginning of every file we prepare:
header('Content-Type: text/html; charset=utf-8');

AddCharset UTF-8 .htm .html .xml
DefaultFsCCSID 500

If I append .php to the first line, the page does not load
Note that if I remove the 2nd line from the apache config, I get toolkit errors (8 Dec 2017 20:13:36.081592 Toolkit request failed. Review the database code and message. Database code (if any): '57017'. Message: Conversion de caractères entre CCSID 1208 et CCSID 65535 incorrecte.)
I have tried different options for CGIJobCCSID, but they have not solved the problem

Here are some entries from php.ini
; By default, PHP will output a media type using the Content-Type header. To
; disable this, simply set it to be empty.
;
; PHP's built-in default media type is set to text/html.
; http://php.net/default-mimetype
default_mimetype="text/html"

; PHP's default character set is set to UTF-8.
; http://php.net/default-charset
default_charset="UTF-8"

; PHP internal character encoding is set to empty.
; If empty, default_charset is used.
; http://php.net/internal-encoding
;internal_encoding =

; PHP input character encoding is set to empty.
; If empty, default_charset is used.
; http://php.net/input-encoding
;input_encoding =

; PHP output character encoding is set to empty.
; If empty, default_charset is used.
; See also output_buffer.
; http://php.net/output-encoding
;output_encoding =

I am using version 1.6 of the toolkit, which seems to use a different charset:
Exec start: 2017-12-04 16:06:51
Version of toolkit front end: 1.6.0
IPC: ''. Control key: *cdata *here
Stmt: call ZENDPHP7.iPLUG32K(?,?,?,?) with transport: ibm_db2
Input XML: <?xml version="1.0" encoding="ISO-8859-1" ?>

We are using Zendserver 9 with php 7

Thanks in adcvance

Brad
longnoz
 
Posts: 30
Joined: Thu Feb 16, 2012 8:55 pm

Re: Character set issues - random garbage characters returne

Postby longnoz on Wed Dec 13, 2017 9:31 pm

I forgot to mention that fast cgi has this: SetEnv="CCSID=1208" SetEnv="LANG=C"

I have since changed the toolkit.ini file to use UTF-8. Still no improvement

I have set both these values to 37 as per another thread I found. Still no improvement
CGIJobCCSID
DefaultFsCCSID

I have read some many articles on the topic that I am dizzy! Not sure what to do next. It seems that moving away from CCSID 65535 is highly recommended, but I don't know what impact this will have on the millions of other tasks that run on this ibm
longnoz
 
Posts: 30
Joined: Thu Feb 16, 2012 8:55 pm

Re: Character set issues - random garbage characters returne

Postby longnoz on Fri Dec 15, 2017 5:46 pm

I was able to set up this same web site on a test server. I changed QCCSID to 37. It did not solve the problem. Help!

PS. The garbage characters change when I do CTRL-F5 vs F5 to refresh my page. Not sure if this is a clue
longnoz
 
Posts: 30
Joined: Thu Feb 16, 2012 8:55 pm

Re: Character set issues - random garbage characters returne

Postby aseiden on Sat Dec 16, 2017 5:28 am

Hi, Brad,

We've been seeing this issue recently. Our solution is to add a trim() in your SQL, such as
Code: Select all
select trim(myfield) as myfield
. Try it and let us know.

Alan Seiden
http://seidengroup.com
aseiden
 
Posts: 873
Joined: Thu Apr 09, 2009 5:45 pm

Re: Character set issues - random garbage characters returne

Postby longnoz on Mon Dec 18, 2017 9:13 pm

Thanks Alan. That did the trick.
longnoz
 
Posts: 30
Joined: Thu Feb 16, 2012 8:55 pm

Re: Character set issues - random garbage characters returne

Postby aseiden on Wed Dec 20, 2017 1:38 am

Longnoz,

Working with IBM, we've found a simpler solution.

In the file /usr/local/zendphp7/etc/conf.d/ibm_db2.ini (or equivalent for other versions of PHP), set this value:
Code: Select all
ibm_db2.i5_dbcs_alloc=1

and then restart the ZENDPHP7 (or equivalent) Apache instance.

Or you can set this value from the Zend Server admin GUI to make sure the value is preserved after any future GUI restarts.

What does ibm_db2.i5_dbcs_alloc=1 do? It provides a larger buffer to account for situations where "special" characters, converted from EBCDIC, take up multiple bytes in their UTF-8 representation.

Try it and see.

Best,
Alan Seiden
http://seidengroup.com
aseiden
 
Posts: 873
Joined: Thu Apr 09, 2009 5:45 pm

Re: Character set issues - random garbage characters returne

Postby longnoz on Fri Dec 22, 2017 9:06 pm

That's great! It solved the problem at the server level. Thanks!
longnoz
 
Posts: 30
Joined: Thu Feb 16, 2012 8:55 pm

Re: Character set issues - random garbage characters returne

Postby rose4807 on Wed Jan 10, 2018 8:49 am

Nice Post Thanks For Sharing!
rose4807
 
Posts: 1
Joined: Wed Jan 10, 2018 8:48 am

Re: Character set issues - random garbage characters returne

Postby rafa3602 on Sun Mar 11, 2018 12:47 am

nice post!!!
rafa3602
 
Posts: 13
Joined: Wed Mar 07, 2018 6:08 pm


Return to Zend Server for IBM i

Who is online

Users browsing this forum: No registered users and 4 guests