PDA

View Full Version : PivotGrids and large datasets



BulletzBill
10 Oct 2011, 5:41 AM
I am attempting to use extjs' PivotGrid component in my application and have realized that it I could very easily encounter scaling issues if it is used on very large datasets. Since the pivot grid does the calculations to summarize the data on the client side, it needs the entire dataset to be loaded at once; this becomes a problem on the backend for example when the PHP server needs to retrieve and output 80,000 records from the mysql server, it runs out of memory. I could also see this being an issue on the client side, if the grid had to process a, say 100K+ record json result set?

Has anyone successfully used the pivotgrid with very large datasets like this, and if you have how did you go about implementing it on both frontend and backend? Or is there a "remote" calculation method available for the PivotGrid, where it would request the values for each cell by sending the top and left axis values to the server, or something simliar?

Thanks in advance.

el_chief
24 Oct 2011, 7:02 PM
what you are looking for is called OLAP.

basically, you would pre-calculate some aggregates (sums, counts, etc) on the database server, or data warehouse.

the OLAP server would then either return the pre-calculated values if available, or do aggregations on the fly.

then, hardly any data would have to flow between the OLAP client (pivot table) and the OLAP server.

there is an open source OLAP server called Mondrian, though I'm not sure exactly how it would interop with ExtJS

BulletzBill
25 Oct 2011, 2:16 AM
Yeah, I am more or less already doing that on a mysql server via grouped queries that return aggregate values. But like you said, I wasn't sure if there was any way to get ExtJS's pivot grid accept those pre-calculated values instead of trying to calculate them itself, i.e. have a "remote" option akin to a normal grid panel's remote sorting or paging options.

marlonlom
17 Mar 2012, 2:15 PM
greetings,

Currently i'm trying to use the PivotGrid component for rendering some data that was previously prepared by Mondrian (xml response).

i've created a java xml parser using SAX for preparing the data. the response, configs for the pivotgrid are sent and used posteriorly for the component.

is there any trick for use xml source or xmla service url for rendering properly in a pivotgrid component?

Im using extjs 3.4.0 gpl.