PDA

View Full Version : Grid rendering very slow on adding data every second



rattler
12 Feb 2013, 8:08 AM
I am trying to make a live dashboard application with grid and charts on it. The data is supplied by rest service in JSON format.

I am using the following code to add the data to store. The grid and charts get automatically refreshed once the data is added to the store. I am using Ext.data.Store , Ext.grid.Panel and Ext.chart.Chart.

The problem is that the grid becomes unresponsive after just a few seconds and scrolling hangs. I have also tried to refresh it every 10 seconds, still it becomes unresponsive after 10-15 seconds. I am adding approximately 6 rows every second in the grid.


setInterval( function() { d3.json("http://localhost:8080/output-console/rest/getCSVList", function(jsondata) {
store.add(jsondata);
});
}, 1000 );

Can someone please help and advise if there is a better way to do it?

PS: I am using D3 just because I was using the same in my previous testing. If someone thinks, that can be the issue, i can replace it.

sdt6585
12 Feb 2013, 9:55 AM
What exactly is coming back in that json? If it's the entire store worth of data, I could see that causing problems. You will need a way to only add the new records rather than updating the entire thing. Maybe a webservice call that you send the timestamp of your last request and it sends back a json object that contains an add and remove object so you can add and remove as necessary. Or if it's reading from a data source that generally only gets new data, maybe just send the time stamp in and get back an object with all the new records.

rattler
12 Feb 2013, 10:25 AM
Yes, I am just sending the delta updates via that JSON call, i.e Only the new rows are being sent from the server. Then I am adding those rows to the data store using store.add().

sdt6585
12 Feb 2013, 10:37 AM
Well...without having it to play with I'm probably not much more help. If you're concerned that using the d3.json call has performance issue, you could certainly switch to Ext.Ajax.request() or configure a proxy on the store and just call store.sync(). If you get a chance and it's not confidential info in that request, you might stick a demo up on jsfiddle so someone can trace through the calls to find the performance culprit. Good luck.

scottmartin
13 Feb 2013, 10:52 AM
You will need to provide a test case for us to continue ...


I am adding approximately 6 rows every second in the grid

Are you checking for collision of your data? Once every second it a bit much .. at your 10 second interval, are you only returning 6 new records? if you stop your interval timer, does the grid become responsive?

Scott.

rattler
15 Feb 2013, 7:45 PM
I have posted the code , couple of times. But my post keeps getting lost (:|
---
I am doing a proof of concept actually powering Grid and charts using the same data source. (Still needs to figure out how to do grouping for a time series, whereas same data would be without grouping in the grid).


The actual dashboard I intend to built can have multiple charts and grids using same or different data sources. They will be based on the push model (may be comet). And its quite possible to have this amount of data flowing in that dashboard.


To answer your second question, its approximately 6 records per second and will be 60 records for 10 seconds. The page becomes completely unresponsive after some time and looks like it becomes CPU bound. The CPU utilisation goes upto 100% for one core from 15-20% without the chart (with just a grid).


Please see the code below:
HTML:




<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Infinite Scroll Grid Example</title>




<link rel="stylesheet" type="text/css" href="../extjs/resources/css/ext-all.css"/>
<link rel="stylesheet" type="text/css" href="../extjs/example.css" />




<!-- GC -->




<script type="text/javascript" src="../extjs/ext-all.js"></script>




<script type="text/javascript" src="infinite-scroll.js"></script>

<script src="../vendor/d3.min.js"></script>
<script src="../vendor/d3.layout.min.js"></script>
<script src='http://d3js.org/d3.v2.js'></script>

</head>
<body>
<h1>Infinite Scrolling</h1>

</body>
</html>



JavaScript File

// Set up a model to use in our Store
Ext.require('Ext.chart.*');
Ext.require(['Ext.Window', 'Ext.fx.target.Sprite', 'Ext.layout.container.Fit', 'Ext.window.MessageBox']);


Ext.onReady(function(){
Ext.define('Ext.app.GridPortlet', {
extend: 'Ext.data.Model',
fields: [
{name: 'name', type: 'string'},
{name: 'price', type: 'float'},
{name: 'time', type: 'double'},
]
});


var store = Ext.create('Ext.data.Store', {
model: 'Ext.app.GridPortlet',
autoLoad: true,
pageSize: 4,
// groupField: 'name',
proxy: {
type: 'ajax',
url : 'http://localhost:8080/output-console/rest/getCSVList',
reader: {
type: 'json',
root: 'stocks',
totalProperty: 'count'
}
}
});
var grid = Ext.create('Ext.grid.Panel', {
renderTo: Ext.getBody(),
store: store,
width: 400,
height: 200,
title: 'Application Users',
invalidateScrollerOnRefresh: false,
columns: [
{
text: 'name',
width: 100,
sortable: false,
hideable: false,
dataIndex: 'name'
},
{
text: 'price',
width: 150,
dataIndex: 'price',

},
{
text: 'time',
flex: 1,
dataIndex: 'time'
}
]
// ,features: [{ftype:'grouping'}]
});
var chart = Ext.create('Ext.chart.Chart', {
xtype: 'chart',
style: 'background:#fff',
animate: true,
width: 900,
height: 500,
store: store,
shadow: true,
theme: 'Category1',
renderTo: Ext.getBody(),
legend: {
position: 'right'
},
axes: [{
type: 'Numeric',
minimum: 0,
position: 'left',
fields: ['price'],
title: 'Number of Hits',
minorTickSteps: 1,
grid: {
odd: {
opacity: 1,
fill: '#ddd',
stroke: '#bbb',
'stroke-width': 0.5
}
}
}, {
type: 'Category',
position: 'bottom',
fields: ['name'],
title: 'Month of the Year'
}],
series: [{
type: 'line',
highlight: {
size: 7,
radius: 7
},
axis: 'left',
xField: 'time',
yField: 'price',
markerConfig: {
type: 'cross',
size: 4,
radius: 4,
'stroke-width': 0
}
}]
});
setInterval( function() {
d3.json("http://localhost:8080/output-console/rest/getCSVList", function(jsondata) {
var stocks = jsondata.stocks;
for(var i =0;i<stocks.length;i++){
store.add(stocks[i]);
}

});
// Ext.Ajax.request({
// url: 'http://localhost:8080/output-console/rest/getCSVList',
// success: function(response){
// var stock = response.responseText;
// for(var i =0;i<stock.stocks.length;i++){
// store.add(stocks[i]);
// }
//
// }
// });
}, 3000 );




});



Sample JSON Data:




{"stocks":[{"name":"IBM","price":105.0,"time":1.360986119E9},{"name":"Facebook","price":314.0,"time":1.360986119E9},{"name":"Google","price":529.0,"time":1.360986119E9},{"name":"IBM","price":105.0,"time":1.360986119E9},{"name":"Facebook","price":314.0,"time":1.360986119E9},{"name":"Google","price":529.0,"time":1.360986119E9},{"name":"IBM","price":105.0,"time":1.360986119E9},{"name":"Facebook","price":314.0,"time":1.360986119E9},{"name":"Google","price":529.0,"time":1.360986119E9}],"count":9}

evant
15 Feb 2013, 7:48 PM
The loading code doesn't make a whole lot of sense. You're looping over each item and calling loadData, why is that?

rattler
15 Feb 2013, 7:54 PM
The loading code doesn't make a whole lot of sense. You're looping over each item and calling loadData, why is that?

I am trying to build a real time dashboard. This code is continuously fetching delta updates from the backend. In this example, it is fetching stock price updates for 3 stocks from the backend (Tomcat based REST service) , every second.

I intend to display all the updates in a grid and showcase them on a series chart. Grid is working perfectly fine, but when i add a series chart, its not able to cope up with these updates and the browser tab keeps crashing. I am using chrome, but have also tried it on safari.

I expect it to crash after a while when the data becomes huge. I understand in a real world scenario, I will have to manage it differently. But my concern is that, even after 10-15 seconds of data, the page becomes unresponsive. I tried the same with D3/Rickshaw library earlier and the series use to work fine for 10-15 mins.

PS: My Bad, the code I pasted above is not correct. I am using Store.add actually. It was not taking my json object as such, so I had to loop through. I will edit the post above

UPDATE:
I have removed the loop and using this now, changed the update from 1 to 3 seconds or 10 seconds.

<CODE>
setInterval( function() {
d3.json("http://localhost:8080/output-console/rest/getCSVList", function(jsondata) {
var stocks = jsondata.stocks;
//for(var i =0;i<stocks.length;i++){
store.add(stocks);
//}
});

}, 3000 );
</CODE>
Still the page is becoming completely unresponsive. Do we have any examples of live dashboard being using EXTJS where a grid and chart is sharing a data source ?

evant
15 Feb 2013, 9:33 PM
You should call suspend/resumeEvents on the store, then refresh the view if you're bulk adding multiple records.

rattler
15 Feb 2013, 10:24 PM
I have added suspend/resume events as shown below. But the grid/chart is not refreshing automatically. Do i need to explicitly call refresh on the whole view or grid/chart ? Sorry, I am new to EXTJS.


Also I am not doing any bulkloading, this code is polling data from the server and adding the delta updates to datastore (approx 5-6 rows per second) .
<CODE>
setInterval( function() {
store.suspendEvents();
d3.json("http://localhost:8080/output-console/rest/getCSVList", function(jsondata) {
var stocks = jsondata.stocks;
//for(var i =0;i<stocks.length;i++){
store.add(stocks);
store.resumeEvents();
//}


});
</CODE>

You should call suspend/resumeEvents on the store, then refresh the view if you're bulk adding multiple records.

waqar
11 Mar 2013, 4:49 AM
rattler, did you got any success in performance ?

waqar
11 Mar 2013, 5:04 AM
I am also using long polling and Publish and subscribe, and grid is update per incoming publishing can be 1 message per second or so .. .. I have narrowed down to problem a little but unable to find a solution to this...

Grid I have has around 40 columns (only 12 visible) and can have 1000rows I am using a non paged grid/store and not buffering it as infinite grid.



voucherGrid.suspendEvents(false);
selectionlMod.suspendEvents(false);
gridView.suspendEvents(false);

updateTreeNode(node, incoming.data);

var scrollPositionbefore = clientGrid
.getEl().down('.x-grid-view').getScroll();
//node.commit(true); // this is dead slow but displays data but cant afford this slow
// voucherGrid.sync() // more slow

voucherGrid.getView().refresh(); // this one is still slow but atleast works

voucherGrid.getEl().down('.x-grid-view')
.scrollTo('top', scrollPositionbefore.top, false);
voucherGrid.getView().refresh();.resumeEvents();
selectionlMod.resumeEvents();
gridView.resumeEvents();




the whole bunch of code executes in a mili-second but only this line halts and as the grid store set of rows increses from intial few 100 records to more the grid continues to slow down on the line below:



voucherGrid.getView().refresh();


can any body guide me what can I do to optimise ? why I need to refesh the whole grid if I added only a single row ?



Help needed.

Thanks.

amackay11
26 May 2015, 5:41 PM
This helped for me:


Ext.suspendLayouts();
.
.
.
mygrid.getView().refresh();


Ext.resumeLayouts(false);