Working with multiple occurrence data structures

[aseiden]

Timo,

How about this idea:

Create a data structure parameter containing two arrays, with different names, whose element counts will add up to the maximum number of elements that could be returned.

Array 1: Will hold the non-blank input data.
Array 2: Don't put any data in this one. Use Count to set the number of elements to expand to, which will the the difference between the maximum number and the size of Array 1.

XMLSERVICE won't know or care how the elements are divided between arrays, so it should put output data in both arrays properly.

Then, on return, you could merge the two output arrays together.

Does this sound workable?

Alan

[timo_karvinen]

Well yes, I guess that could work, but that would mean I have to dynamically generate new program call description with every call made.

I may have omitted important information describing the problem initially.
The situation is as follows:
We have RPG programs that are used from web browser interface we are developing with PHP + ExtJS on the I5, here it's not so bad because we can control better what gets send to the program call.
But we also have Web Services created with PHP that get called from all kinds of different places, some call with 500 input arrays because they want big catalog of different items, some call the same Web Service with only one input array because they are focused on one particular item and some call it with like 10 input items and so on.

So if I understood correctly what you suggested, I'd need to create new description with every Web Service call, this doesn't sound a bit better than the initializing the empty arrays with '' on every call.

Let me put an example here of one of the descriptions if I'm explaining this bit unclearly.

Code: Select all

    private $rpgAllParms = array(
        array(
            'name' => 'Pb_Code',
            'io' => 3,
            'type' => 0,
            'length' => '1'
            ),
        array(
            'name' => 'Pi_Caller',
            'io' => 3,
            'type' => 0,
            'length' => '18'
            ),
        array(
            'name' => 'Pi_Pyks',
            'io' => 3,
            'type' => 0,
            'length' => '2'
            ),
        array(
            'name' => 'Pi_User',
            'io' => 3,
            'type' => 0,
            'length' => '10'
            ),
        array(
            'name' => 'Po_ErrNbr',
            'io' => 3,
            'type' => 7,
            'length' => '3.0'
            ),
        array(
            'DSName' => 'PO_Err',
            'count' => 10,
            'DSParm' => array(
                array(
                    'name' => 'Err_Id',
                    'io' => 3,
                    'type' => 0,
                    'length' => '7'
                    ),
                array(
                    'name' => 'Err_Msg',
                    'io' => 3,
                    'type' => 0,
                    'length' => '132'
                    ),
                array(
                    'name' => 'Err_Field',
                    'io' => 3,
                    'type' => 0,
                    'length' => '30'
                    ),
                array(
                    'name' => 'Err_Sev',
                    'io' => 3,
                    'type' => 7,
                    'length' => '1.0'
                    )
                )
            ),
        array(
            'name' => 'Pi_Ohtu',
            'io' => 3,
            'type' => 0,
            'length' => '4'
            ),
        array(
            'name' => 'Pi_Ohava',
            'io' => 3,
            'type' => 0,
            'length' => '18'
            ),
        array(
            'name' => 'Pb_OhNbr',
            'io' => 3,
            'type' => 7,
            'length' => '5.0'
            ),
        array(
            'DSName' => 'Pb_Oh',
            'count' => 500,
            'DSParm' => array(
                array(
                    'name' => 'Oh_Haku',
                    'io' => 3,
                    'type' => 0,
                    'length' => '1'
                    ),
                array(
                    'name' => 'Oh_Field',
                    'io' => 3,
                    'type' => 0,
                    'length' => '30'
                    ),
                array(
                    'name' => 'Oh_Ohtu',
                    'io' => 3,
                    'type' => 0,
                    'length' => '4'
                    ),
                array(
                    'name' => 'Oh_Ohava',
                    'io' => 3,
                    'type' => 0,
                    'length' => '18'
                    ),
                array(
                    'name' => 'Oh_Tieto',
                    'io' => 3,
                    'type' => 0,
                    'length' => '100'
                    )
                )
            )
        );


-Timo

[aseiden]

Timo, a solution is on the way. In a week or two, I think, I'll be releasing an update that provides, in the new API, efficient, initialized, large multiple-occurrence arrays of DSes and scalar values, and also the option to pass any number of records (up to the total) filled with data into those arrays.

Alan

[timo_karvinen]

I'm very glad to hear you are working on this. This is still a big problem for us with the new toolkit.

We have actually implemented your suggestion of initializing the arrays/DS's up to maximum size with empty strings ourselves, but the solution isn't working so great.
I was supposed to post more about this earlier, but haven't had time.
I'll just post some info in here anyway, though you are working on it already.
I ran some performance tests on our programs with old toolkit, and the new toolkit (with our own DS initializer), here are the results:

Zend Core, i5_toolkit:

Code: Select all

Call times:
1: 0.71s   2: 0.43s   3: 0.44s   4: 0.44s   5: 0.43s   6: 0.44s   7: 0.44s   8: 0.43s   9: 0.43s   10: 0.44s
10 calls total time: 4.6243221759796, average: 0,4624 seconds.


Zend Server new Toolkit / XMLSERVICE:

Code: Select all

Call times:
1: 2.35s   2: 2.19s   3: 2.23s   4: 2.21s   5: 2.20s   6: 2.21s   7: 2.21s   8: 2.22s   9: 2.23s   10: 2.21s
10 calls total time: 22.250317335129, average: 2,2250 seconds.


The problem obviously stems from the fact that the XML gets so damn large on these calls.
Here's example of the new toolkit debug log from one of the calls above (brace yourself for 4665 lines of XML):

Code: Select all

Exec start: 2012-10-30 17:21:08
IPC: '/tmp/xtoolkit/WEBSMERX10'. Control key: *cdata *sbmjob(ZENDSVR/ZSVR_JOBD/XTOOLKIT)
Stmt: call XMLSERVICE.iPLUG1M(?,?,?,?)
Input XML: <?xml version="1.0" encoding="ISO-8859-1" ?>
<script>
<pgm name='RMPOHHOH' lib='*LIBL'>
<parm comment='Pb_Code'><data var='Pb_Code' type='1a' /></parm>
<parm comment='Pi_Caller'><data var='Pi_Caller' type='18a' /></parm>
<parm comment='Pi_Pyks'><data var='Pi_Pyks' type='2a' /></parm>
<parm comment='Pi_User'><data var='Pi_User' type='10a' /></parm>
<parm comment='Po_ErrNbr'><data var='Po_ErrNbr' type='3s0'>0</data></parm>
<parm comment='PO_Err'><ds var='PO_Err' array='on'>
<ds var='PO_Err'>
<data var='Err_Id' type='7a' /><data var='Err_Msg' type='132a' /><data var='Err_Field' type='30a' /><data var='Err_Sev' type='1s0'>0</data></ds>
<ds var='PO_Err'>
<data var='Err_Id' type='7a' /><data var='Err_Msg' type='132a' /><data var='Err_Field' type='30a' /><data var='Err_Sev' type='1s0'>0</data></ds>
<ds var='PO_Err'>
<data var='Err_Id' type='7a' /><data var='Err_Msg' type='132a' /><data var='Err_Field' type='30a' /><data var='Err_Sev' type='1s0'>0</data></ds>
<ds var='PO_Err'>
<data var='Err_Id' type='7a' /><data var='Err_Msg' type='132a' /><data var='Err_Field' type='30a' /><data var='Err_Sev' type='1s0'>0</data></ds>
<ds var='PO_Err'>
<data var='Err_Id' type='7a' /><data var='Err_Msg' type='132a' /><data var='Err_Field' type='30a' /><data var='Err_Sev' type='1s0'>0</data></ds>
<ds var='PO_Err'>
<data var='Err_Id' type='7a' /><data var='Err_Msg' type='132a' /><data var='Err_Field' type='30a' /><data var='Err_Sev' type='1s0'>0</data></ds>
<ds var='PO_Err'>
<data var='Err_Id' type='7a' /><data var='Err_Msg' type='132a' /><data var='Err_Field' type='30a' /><data var='Err_Sev' type='1s0'>0</data></ds>
<ds var='PO_Err'>
<data var='Err_Id' type='7a' /><data var='Err_Msg' type='132a' /><data var='Err_Field' type='30a' /><data var='Err_Sev' type='1s0'>0</data></ds>
<ds var='PO_Err'>
<data var='Err_Id' type='7a' /><data var='Err_Msg' type='132a' /><data var='Err_Field' type='30a' /><data var='Err_Sev' type='1s0'>0</data></ds>
<ds var='PO_Err'>
<data var='Err_Id' type='7a' /><data var='Err_Msg' type='132a' /><data var='Err_Field' type='30a' /><data var='Err_Sev' type='1s0'>0</data></ds>
</ds>
</parm>
<parm comment='Pi_Ohtu'><data var='Pi_Ohtu' type='4a'>IFSK</data></parm>
<parm comment='Pi_Ohava'><data var='Pi_Ohava' type='18a' /></parm>
<parm comment='Pb_OhNbr'><data var='Pb_OhNbr' type='5s0'>0</data></parm>
<parm comment='Pb_Oh'><ds var='Pb_Oh' array='on'>
<ds var='Pb_Oh'>
...
**************SNIP***************
...
</parm>
</pgm>
</script>
Exec end: 2012-10-30 17:21:10. Seconds to execute: 1,6965289115906.


Ah, the forum wouldn't let me post so long XML, anyway you have take my word for it then it's one big a** XML.
(Your message contains 323347 characters. The maximum number of allowed characters is 60000.)

Well anyway, thanks for working on it, looking forward for the results.

-Timo

[rangercairns]

While Alan is dealing with hurricane Sandy, i am lurking on this forum now and again, so you can ...

XMLSERVICE already has a built in <overlay> ability, where you can init dim='n' elements to default data on input, BUT also <overlay> specific array locations with custom data (see below). I am not 100% sure it will handle all your sparse allocation needs, but we can probably mess with it a bit to make it do any sort of offset/overlay based array index setting. Because Toolkit and ibm_db2 can share connections (ibm_db2/odbc), you can play around with this technique today using the same technique described by another DS forum entry just answered ( viewtopic.php?f=113&t=80608 ).

BTW -- you can see your monster XML document could be greatly reduced using the <ds dim='n'> capability when Alan gets around to wrappering in PHP toolkit ... usually cuts big XML document down to just a few lines because "big data" is almost 95% output concern (rarely monster input) ... monster XML says Happy Halloween.

You can see current fancy XMLSRVICE capabilities at this page http://www.youngiprofessionals.com/wiki ... RVICEQuick, but Alan has not had time to wrapper up all these things into PHP code.

Code: Select all

<?xml version='1.0'?>
<script>
<pgm name='ZZVLAD3''>
<parm io='in' dim='20' comment='init all'>
<ds var='MyErrorDs' comment='first data'>
<data type='8a' var='ErrorId'/>
<data type='3u0' var='Severity'/>
<data type='80a' var='Description'/>
</ds>
</parm>
<overlay io='both' top='on' comment='set any number messages'>
<ds var='MyErrorDs' comment='first data'>
<data type='8a' var='ErrorId'>12345678</data>
<data type='3u0' var='Severity'>1</data>
<data type='80a' var='Description'>Toad wrangler</data>
</ds>
<ds var='MyErrorDs' comment='right behind first'>
<data type='8a' var='ErrorId'>87654321</data>
<data type='3u0' var='Severity'>3</data>
<data type='80a' var='Description'>Frog wrangler</data>
</ds>
</overlay>
</pgm>
</script>



[timo_karvinen]

Well for us "big data" is actually an input problem also, most of the time.
That monster XML mentioned above is one just such a program, it can be called with single input and get single (or multiple) output as a result, but it can also have 500 input occurrences which then will of course return 500 output occurrences.

And the above functionality isn't a design flaw (just put everything IN/OUT for the heck of it), it's actually just the inteted functionality that we need, and most of our php-enabled rpg programs use the same scheme.

-Timo

[rangercairns]

Mmm ... not matching my 95% output theory, so it would appear we will have to come up with a different performance scheme for completely unique "big data" input XMLSERVICE, not just "big data" output ... i have a few ideas, would you like to be a test subject for a few iterations of xmlservice "big data"???

No offense is meant here, but i want to double check your actual requirement, so we don't spend a bunch of time chasing artificial benchmark concerns, that is to say nothing wrong with artificial loop tests, etc., but real life stuff is more important to work on.

First, i would like to clarify ...
1) ... you must send equal size completely unique big XML data in/out (every parameter, every record, every field, unique data)??
2) ... you cannot send "change only" data (sparse records/fields, all others default)???

[rangercairns]

Timo and Alan,
My 5000 element big XML data test cut time in half 40 seconds to 15 seconds ...
... still pure RPG (still much slower than PASE), but pure RPG
... i am not completed with my RPG-ILE-must-work activities, so stay tuned
... anyway, if ILE is a bust i can always drop into PASE assembler (sub-second prototype)
... you can try this test version on your testing xmlservice-rpg-1.7.4-sg2.zip (below)


http://www.youngiprofessionals.com/wiki ... ICETesting

Active test versions
2012–11–02 -xmlservice-rpg-1.7.4-sg2.zip
... ADD — working on performance improvement for extreme large data sets (5000 elements <data>)
... current test improved for 40 seconds to 15 seconds, not done with work, but try if you would like
... small degradation on normal workloads, but will probably figure that out soon (very small)

[timo_karvinen]

Mmm ... not matching my 95% output theory, so it would appear we will have to come up with a different performance scheme for completely unique "big data" input XMLSERVICE, not just "big data" output ... i have a few ideas, would you like to be a test subject for a few iterations of xmlservice "big data"???

No offense is meant here, but i want to double check your actual requirement, so we don't spend a bunch of time chasing artificial benchmark concerns, that is to say nothing wrong with artificial loop tests, etc., but real life stuff is more important to work on.

First, i would like to clarify ...
1) ... you must send equal size completely unique big XML data in/out (every parameter, every record, every field, unique data)??
2) ... you cannot send "change only" data (sparse records/fields, all others default)???

Yes I can test new versions with big data, when my time allows.
No offence taken, you are right about that particular program, it's more of a benchmark tool than real program (well it is a real program too but not in use at our customers). But it's the one I'm running because it can be easily used as benchmark tool, and it reflects the real programs very well. Also it's the only one we can run at all at the moment, the real programs (that are in production use at our customers) take so long time they always timeout completely (they have even bigger data) with the new toolkit.

1) and 2) Equal size is not the concern, it's actually the variable size, since we can have anywhere from 0 to 200 (for example) occurrences of input parameters coming in.
But we also send same kind of things out, mind you not the _same things_, though they might have a variable or two in common, so we could probably optimize some (not _everything_ we have is all both input and output) I think we would still have performance problems because most of the data is still unique.

Here's example of one of the real programs entry parameters, this one is in production use with several of our customers:

Code: Select all

Param name     Len De    Description
------------  -------    ---------------------------------------------
Caller          18A      Kutsuva järjestelmä                         
Action           1A      Toimintokoodi                               
Pyks             2A      Yksikkö                                     
Vano            15A      Varasto                                     
Asno            15A      Asiakas                                     
Kiekdi           3A      Kielikoodi                                   
Myyja           15A      Myyjä                                       
Rows             5S 0    Tuotetaulun rivien lkm                       
ProdTable                Tuotetaulu DS (occurs 200)                   
  Pr_Tuno        20A      Tuotenumero                                 
  Pr_Toim        15A      Toimittaja (brändi)                         
  Pr_Mra          9S 2    Määrä                                       
  Pr_Myks         3A      Myyntiyksikkö                               
OutCode          1A      Toimintokoodi, palautuu '0'=kaikki ok       
ErrNbr           3S 0    Virheiden lukumäärä                         
Err                      Virheet DS (occurs 200)                     
  Err_Id          7A      Tunnus                                       
  Err_Msg       132A      Teksti                                       
  Err_Field      30A      Kenttä                                       
  Err_Sev         1S 0    Vakavuustaso                                   
  Err_Row         5S 0    Sisäänluetun tuotetaulun virheellinen rivi     
Valk             3A      Valuutta                                       
Rows             5S 0    Tuote/hintataulun rivien lkm                   
ProdTable                Tuote/hintataulu DS (occurs 200)               
  Pr_Error        1A      Virhe rivillä                                 
  Pr_NoPrice      1A      Koodi löytyikö hintaa - '0' löytyi             
  Pr_Tuno        20A      Tuotenumero                                   
  Pr_Vetn        20A      Vaihtoehtoinen tuotenumero                     
   Pr_Toim       15A      Toimittaja (brändi)                           
   Pr_TuSta       3A      Tuotestatus                                   
   Pr_Vapaa       9S 2    Vapaa saldo                                   
   Pr_TuDesc     40A      Kuvaus                                         
   Pr_Tvalm       3A      Toimitusvalmius                               
   PriceTable             Hintataulu DS (dim 5)                         
    Pc_Hity       3A      Hinnan tyyppi                                 
    Pc_Myhi      13A      Myyntihinta verollinen                  (11S 4)
    Pc_Vthi      13A      Myyntihinta veroton                     (11S 4)
    Pc_Ale1       7A      Riviale 1 % (vain jos bruttohinta)      ( 5S 2)
    Pc_Ale2       7A      Riviale 2 % (vain jos bruttohinta)      ( 5S 2)
    Pc_Ale3       7A      Riviale 3 % (vain jos bruttohinta)      ( 5S 2)
    Pc_Alvp       7A      Alv-%                                   ( 5S 2)
    Pc_Hikt       3A      Hinnan käsittelytapa                           
    Pc_MHalr     11A      Määrähinnan alaraja                     ( 9P 2)
    Pc_Myks       3A      Vyks (jos Myks välitetty)                     
    Pc_Vano      15A      Varasto                                       


-Timo

[rangercairns]

Cool. Thanks for the input and the volunteer when you have time.

Gee, I hope i do not sound geek programmer defensive, all this is simply fun programming code for me (xmlserivce, php, etc.) ... a bug is a bug, a design issue an opportunity, etc. ... actually i really enjoy IBM+Zend Open Source development, where we can nearly instantly turn around customer requests/problems for things to make your web/PHP life easy (including RPG xmlservice code).

I have yet another version on my way to helping you achieve "reasonable" performance ...
http://www.youngiprofessionals.com/wiki ... ICETesting
Active test versions
2012–11–02 -xmlservice-rpg-1.7.4-sg3.zip
ADD — working on performance improvement for extreme large data sets (5000 elements <data>)
current test improved for 40 seconds to 8 seconds, not done with work, but try if you would like
still using only RPG ILE XML parsing … PASE supplemented XML parsing option possible, but not implemented (yet)
small degradation on normal workloads, but will probably figure that out soon (very small)

Personal note:
I don't mind chasing better performance, we needed to do this anyway as Alan/I were just getting XMLSERVICE to run at all first go around.

However, hopefully we/you keep in mind actual route of any "real script" request usually runs all way from browser ... through IBM i Apache job/thread ... FastCGI php-cgfi+ibm_db2 job ... DB2 QSQSRVR help job ... XMLSERVICE job (private/stateless) ... and all the way back to browser, so even if we could run xmlservice one million call loops a second, Apache could only cycle out maybe 400-800 hits second practical web site.

Last resort (not yet):
I have NOT dropped into PASE assembler yet (i have a very fast prototype), so XMLSERVICE is still pure ILE RPG (Yahoo) that anyone can read ... mmm ... PLEASE be careful what you asked for in terms of performance, i am very good AIX/PASE developer, but RPG folks will NOT be able to read Open Source XMLSERVICE if i resort to this PASE method (i don't need PASE compiler can write binary/object code).