Working with multiple occurrence data structures

Re: Working with multiple occurrence data structures

Postby timo_karvinen on Thu Nov 15, 2012 4:52 pm

I understand the need to understand reasoning behind doing something very well. But I have to say I was much happier when we were talking about technical stuff (I'm just a programmer too), instead of theory/philosophy behind operation, we seem to have shifted to here. But I will try to address, best I can, some of these worries you have about our methods and reasoning.

kentatzend wrote:As Tony commented we had a set of priorities for the development of this new toolkit and initially performance was not our highest item. But now that we've address a lot of the basic problems we're looking more at performance than previously.

Yes, this is also why we're here now, asking about performance for our use cases. Now that much of everything else is more or less working, I'd imagine everyone would want to focus a bit on the performance side.

kentatzend wrote:But I'm a bit worried that this is really just an artificial test case that is not going to exist in very many real world test cases. Of course I'm not 100% sure that this "assessment" by me is correct so I'd like to ask a few questions.

Like both me and Jan have already mentioned several times, there's nothing artificial about this, this is how our programs/services work and have worked in production environments at several of our customers for couple of years now. I can't tell you either, if such cases are really popular out in the rest of the world, but I can tell you it's certainly the way we use most of our php-rpg programs/services. And the way that our ERP (rpg) is designed to work with php applications / interfaces.

kentatzend wrote:It seems like there is a mismatch between what Jan says about the test case and the example Timo is giving us.

There's indeed a mismatch, but it's not between what Jan says and the data I have send you. It's between what we are saying and how you are reading that information. You seem to understand our posts so that 200 products equals 200 (xml) elements. A product we are talking about here is actual physical product in the real world, it has many many properties in the databases. So instead of 200 product rows equaling 200 xml-elements (just one property per row), they actually have like 30 or maybe even 100 properties (like name, description, packaging size, price without tax, price with tax, warehouse availability in several different warehouses and so on and so on), so 200 rows * 100 properties becomes 20000 xml elements. Well in reality even more than that, because in xml even a product row with only one property would consist of several xml elements because of the wrapper elements.

kentatzend wrote:Quite frankly I'm wondering what application really ever needs 10K+ elements to be fetched at one time and have to do that on any sort of regular basis. I could see where you might fetch 10K element once or even periodically where the period is fairly long (15 minutes, hourly, etc). The case would be for something like a local cached "catalog" of items for performance, etc. I could even see where you would fetch 10K elements in bunches (as Jan describes). But I find it hard to imagine (and I have a pretty good imagination) a scenario where a user request would need to pull all 10K+ element and could display or process them as part of a single user request/response.

Again, our ERP solution that talks to a lot of other systems like CRM's, Web Shops, Finance systems etc. etc. So it's not necessarily "a user request" as you put it, it's likely an another system requesting/inputting the data for multitude of different reasons. And again Jan was talking about 10k products instead of elements, which actually could mean like 10k*100 xml elements or so forth. And you are exactly right there, the case Jan described is "local cached catalog of items for performance" at a Web Shop on different server, which sells items based on inventory and product information stored in our ERP.

kentatzend wrote:Am i missing something here or is this really just a good artificial test case that was designed to push the edge of the envelope?

And once more with a feeling :); no, definitely not an artificial test case, at least not for us here in Finland, these are real in-production-use programs/services and we have several of them. And I can't for the world of me imagine what on earth my/our motivation would be for making an artificial test case like that, to begin with.

-Timo
timo_karvinen
 
Posts: 74
Joined: Wed Aug 12, 2009 7:58 am
Location: Tampere, Finland

Re: Working with multiple occurrence data structures

Postby rangercairns on Fri Nov 16, 2012 7:29 pm

1st reminder ...
I am NEVER authorised public product to product compares, so any path leading this direction immediately voids my help (maybe permanently). Again, if you need/want only Easycom product, please seek business relationship with the good Aura company.

Well then ...
Really big production data BOTH input/output XML elements 10K, 20K, 200K, millions? (... well beyond my text editor cut/paste ability, Uff da!), to me seems a bit more task FTP zip files instead of "web service", but production code is production code ... so ...
1) Is current XMLSERVICE 15-17 seconds test example fast enough? ... ie. does giant data input operation only occur 10-100 times a day???
2) If not fast enough for practical use, do you wish to continue helping test/improve free XMLSERVICE/PHP toolkit???
3) Are you "married" to one script API technology (CW), or can we innovate something possibly work better for xmlservice "gigantic data"?

XMLSERVICE design guidelines (so you know where current design lines fall) ...
a) Very thin client side -- avoiding heavy client side "download" code (PECL extension, etc), which mostly leads to client driver install/management anarchy
b) Heavy lifting server side -- xmlservice should take arrows for performance by design (my toes not offended), keep performance problem one place on server where it belongs away from unpredictable clients
c) Protocol "strings" like pure XML/JSON -- avoiding proprietary "binary" protocols which never keep up with device and language proliferation (mobile, tablets, PHP/Ruby/Java, etc.)
d) Transport existing protocols (DRDA, ODBC, HTTP REST, etc.) -- avoiding proprietary "connection" protocols, which nearly always lead to additional security issues
e) Open Source code -- avoid source code technologies people can't understand ... RPG, PHP, etc. are ok
rangercairns
 
Posts: 215
Joined: Fri Jul 24, 2009 6:28 pm

Re: Working with multiple occurrence data structures

Postby kentatzend on Mon Nov 19, 2012 7:58 pm

Timo,

In you post you say:
Again, our ERP solution that talks to a lot of other systems like CRM's, Web Shops, Finance systems etc. etc. So it's not necessarily "a user request" as you put it, it's likely an another system requesting/inputting the data for multitude of different reasons. And again Jan was talking about 10k products instead of elements, which actually could mean like 10k*100 xml elements or so forth. And you are exactly right there, the case Jan described is "local cached catalog of items for performance" at a Web Shop on different server, which sells items based on inventory and product information stored in our ERP.


So does this mean that the operation that run here are not run in a end user request/response cycle? What I mena is the end user is not sitting at some web browser for 17 seconds while this is being done?

Kent
User avatar
kentatzend
 
Posts: 1778
Joined: Thu Dec 11, 2008 1:08 pm

Re: Working with multiple occurrence data structures

Postby rangercairns on Tue Nov 20, 2012 10:35 pm

Timo,
You ask ... we deliver 30,000 records less than 2 seconds ...

Alan will have to decide how best to wrapper the "big data" PHP Toolkit API (just raw XML).
I will post xmlservice-1.7.5 when Alan and I are closer to complete testing ...

Run it -> http://www.youngiprofessionals.com/wiki ... bigboy.php
View it -> http://www.youngiprofessionals.com/wiki ... oy.src.php

==========
Sample output
===========
Calling Big Boy PGM
30000 Records Input/Ouput

Welcome aboard xmlservice toolkit hacker crew, flying 30000 records round trip.

Your flight time today: Time (loop=1) total=1.96 sec (0.51 calls/sec) (1.955565 sec per call)
rangercairns
 
Posts: 215
Joined: Fri Jul 24, 2009 6:28 pm

Re: Working with multiple occurrence data structures

Postby rangercairns on Wed Nov 21, 2012 12:06 am

Looks like increasing number to 100,000 content records performance curve is fairly linear ... thanks to DB2 best product on IBM i.

Welcome aboard xmlservice toolkit hacker crew, flying 100000 records round trip.
Your flight time today: Time (loop=1) total=7.41 sec (0.13 calls/sec) (7.410856 sec per call)

100,000 records run it -> http://www.youngiprofessionals.com/wiki ... boy100.php
100,000 records view it -> http://www.youngiprofessionals.com/wiki ... 00.src.php
rangercairns
 
Posts: 215
Joined: Fri Jul 24, 2009 6:28 pm

Re: Working with multiple occurrence data structures

Postby rangercairns on Wed Nov 21, 2012 3:17 pm

Outstanding best of geek funny (my view), we put up a demo of new big data record XMLSERVICE ... and of course as is true public demos everywhere ... Yips machine ISP provider has been having issues, so sorry if you can not reach Yips machine today. I looked and machine isn't doing anything at all, but ISP connection can be slow (including telnet green screen).
rangercairns
 
Posts: 215
Joined: Fri Jul 24, 2009 6:28 pm

Re: Working with multiple occurrence data structures

Postby timo_karvinen on Wed Nov 21, 2012 5:52 pm

That sounds great, the improved performance of the toolkit that is, not the problems with the ISP.
I look forward to trying this on our environment when I have the time.
Thank you for the continued improvement of the XMLSERVICE.


Also to answer Kent's earlier question:
kentatzend wrote:So does this mean that the operation that run here are not run in a end user request/response cycle? What I mena is the end user is not sitting at some web browser for 17 seconds while this is being done?

Yes, it does mean that sometimes (usually the calls with full 200 products/occurrences populated) are background calls, like in the situation we earlier described in more detail. But those same programs/services are also called in situation where the user is sitting behind the screen and waiting for the response, typically though this would be less than the maximum of 200 occurrences, unless of course the user is ordering 200 different products or so on. This dual use of same program/service is why I was originally in this thread asking about how to get rid of those performance degrading empty occurrences in this kind of situation. But as I understand it, those empty occurrences should be handled very nicely already, as soon as the new version of PHP Toolkit wrapper comes around.

-Timo
timo_karvinen
 
Posts: 74
Joined: Wed Aug 12, 2009 7:58 am
Location: Tampere, Finland

Re: Working with multiple occurrence data structures

Postby rangercairns on Tue Nov 27, 2012 12:10 am

Hi Timo,

I built a test version with 8K data similar to empty data you sent me just to check complex data also performs well ... roughly 2000 products ... it did ok (links below)

try it -> http://www.youngiprofessionals.com/wiki ... ztable.php
view it -> http://www.youngiprofessionals.com/wiki ... le.src.php

I did not have your RPG/Cobol program, so i ran xmlservice $ctl=" *test" mode, which will push/pop all data as if your program was called.
rangercairns
 
Posts: 215
Joined: Fri Jul 24, 2009 6:28 pm

Re: Working with multiple occurrence data structures

Postby aseiden on Wed Dec 05, 2012 6:58 am

Timo,

There's now a beta version of the PHP toolkit 1.4.0 that handles empty array structures efficiently, as you referred to earlier. Let me know if you'd like to test it. If you have a snippet of PHP code that defines your parameters, I can adapt it to handle "empty" structures. I can do it here on the forum as an example.

Thanks,
Alan
aseiden
 
Posts: 804
Joined: Thu Apr 09, 2009 5:45 pm

Re: Working with multiple occurrence data structures

Postby timo_karvinen on Fri Dec 07, 2012 1:10 pm

Thanks guys, I'm looking forward to testing these improvements.
But I have been sick for a while and thus I'm very much behind with my work, so I am/will be very busy for some time.

About those parameters, there's one example I posted in page 2 of this thread at least. I can also post another if you want, and tell me in what kind of format you'd like it.

-Timo
timo_karvinen
 
Posts: 74
Joined: Wed Aug 12, 2009 7:58 am
Location: Tampere, Finland

PreviousNext

Return to New Toolkit

Who is online

Users browsing this forum: No registered users and 3 guests