PDA

View Full Version : Millions of records grid



rozenman
23 Mar 2015, 4:20 AM
Hey guys,
I am writing a desktop app using Extjs and CEF (Chromium Embedded Framework). So, all my app is an Extjs style app, and the renderer is CEF.
My app is loading dynamically huge amount of records into a grid. I do not take the data from a database - it is getting the data dynamically (doesn't matter how).
The problem is - there is a huge amount of data - each second about 10,000 new records. I need to be able to 'display' them all (I use 'bufferedrenderer' of course). After some minutes, the store will include millions of records...
I need a way to load the data quickly into the store and grid - as with this amount of data, every store.add(data) - the screen freezes for couple of seconds (even with suspending the events and refreshing the view every 10 seconds).
I need the store to have all the records, as sometimes I add filters etc, and the user need to be able to view them all.
What's the best way to do so, and keep the app fluid without freezing?

Thanks!

yeghikyan
23 Mar 2015, 9:58 AM
how fast works json_encode for 10000 elements?I do not think there a human being which can see and analyse 10000 elements at once, so why to show him/her this huge amount of data?

suzuki1100nz
23 Mar 2015, 10:06 AM
I agree with yeghikyan why load so much data into the store?
Is there not anyway you could use a buffered store which dynamically load as the user scrolls?

Sounds like your trawling for data rather than having a server your're loading from?

If you're using at least a buffered renderer loading items into the store should have no effect to the dom after the first viewable page is loaded. Unless you allow and have column sorting on.
Batching your store updates is definitely needed.

Without a test case showing your issue its going to be difficult to help - can you post a fiddle test case?

rozenman
23 Mar 2015, 12:31 PM
Hey guys, thanks for the reply.
I know it's too much data to view, but these are the requirements.. I don't need the user to watch them all together, but I do need them all in the store - as the user may use filters to see relevant ones or search through all (no columns sort - it will take too much time, and the search is 'find next match').
My main problem is that there is no server side to store all the data, and send only relevant records.. I must use the app to store it.. I cannot control the data flow (about 10,000 records in 1-2 seconds), and once I pull the data - it is lost, and stored only in the app.
I am open to every idea. Also, as I use CEF, I can run native C++ functions - maybe I should store the data in the C++ part, and pull it from there? The problem is that I must use functions - can I make the store pull data from a function instead of URL? Anyways, as I use native functions, if I store the data in a C++ structure, when I call a native function, it blocks and the GUI might freeze too..
The most simple example I can show, is something like that:

setInterval(function() {
var data = get_new_data();
store.add(data);
}, 5000);
This is almost what's happening - I pull data using a native function every 5 seconds (in this case - get_new_data) which returns an array of records (about 50,000 records, using V8 JavaScript Engine for natives), and then I add the data to the store of the grid.
Also, even if you have a creative idea, do you have any suggestions how to make the current code work better (the app is only one big grid with this store, and some buttons for search/filters - nothing fancy)?

Many thanks to the helpers!!!!

scottmartin
27 Mar 2015, 6:18 AM
The fact that you are shoving thousands of records in to your grid on an short interval is going to cause delays, simple.

I am scratching my head as to why you chose ( or are forced to use ) this approach for your app to deal with all of this data.

Then you are adding a search option to ever changing data? Since it is not sorted, new is 'yet to be searched'?

I assume this is realtime data that is being fed to be analyzed?

Any thoughts to have a grid assigned to another store for view search results and leave the main store for incoming data?

You would have to move incoming results to grid store by creating a custom reader perhaps.

Scott.

rozenman
30 Mar 2015, 8:56 AM
Hey,
Thanks for reply!
This is a real time app, that shows real time data and can expand data for specific records if needed. I do not need to sort the grid. The data is already sorted (by time of create), and that is the order I need - so when loading the array of records into the store (at the end of it, appended to the already loaded data), it is already in the order I want it.
The search function is only 'find next' search - I am not filtering the grid, but just jumping to the first next relevant row.
But, the store can have filters (which can be changed by user) - and after applying them, it is OK for it to take time until it finishes (by the way - is there a way to cancel filtering while it is going on? As it may take several minutes, it would be great to have the option to cancel and stop the filtering in the middle, even if the data won't consistent after canceling).
So, I do not need another store for the search results, but - if this idea is good for the filtering, the search, and the loading times - it would be great. I don't really know how to have one huge store for incoming data, while another store is only for viewing (with filters, and using data from the main store) - would be great if you can explain a bit about it :)
Also, after having millions of records, even if I stop pulling data (so no more loading data into store), the app is a bit slow (although only rendering few records with BufferedRenderer). If the data isn't changing, and I am only viewing it - why is it slow (even with millions of records)?

Again, thanks for the help!

rozenman
8 Apr 2015, 1:28 PM
Bump?
I am open to any ideas... Maybe I should store all the data in a C++ structure, and use the store to load from it, like a remote store. But then, do I need to do all the filtering in the C++ code? Because that would be insane.. Is it possible to have a remote store, but with local filters - that dynamically shows only filtered records (but fills the grid.. I don't want it to show 0 records because the whole page is filtered)?
My main problem when storing the records in C++ is that you need to work really hard to get all the things you want - like filtering, or searching, or using regular expressions etc..
Any ideas..? :-/