PDA

View Full Version : Chart and Large Set of Records



jmayer
26 Oct 2011, 6:13 PM
I have a database which records data sent from a *PLC and a record is created every 10 seconds. A few hours produced roughly 4500 records and the records are displayed in a chart. When I try to load the roughly 4500 into the chart, of course it takes a while to load.

*A PLC is a programmable device that receives signals from mechanical devices and sends signals to mechanical devices.

Is the chart built to handle large sets of data? (A days worth of records.)
Are there any improvements I can make on the load process?

The line chart displays a days worth of records and it only has three series.
The data store used by the chart uses an ajax request and it uses the json parser.

I run the load process on a Task Runner (runs every second) and the process works like this:
1. Build the request's parameters for an hour worth of records.
2. Send request.
3. On callback, merge records into main data store.
4. If hour 24, return false to kill the task else wait for task to run again.

Table which holds the data is formatted like:
ChartId, Timestamp, Value1, Value2, ... , Value10

Timestamp is used for the X-axis.
Value1-10 are used for the Y-axis.

Any help will be appreciated.

jmayer
3 Nov 2011, 10:01 AM
After looking into this futher, I found out the chart significantly loads faster in Chrome than in IE, it loads faster if I request all the data and if I remove the label object from the series that reduces load time by half.

The users have the ability to create comments on the data being displayed and the comments are divided into different types, for example New Batch or Divert Check. When the chart is generated, I override the renderer function on the label object and search if the data point has a comment and if so display the name of the comment type. The user only enters about 1 to 10 comments per day so a majority of the data points don't need labels.

Originally I thought my search ate up performance either because it took too long or it was the sheer number searches being made. But then I realized the test data I was looking at had no comments so the renderer function would just return an empty string each time it was called.

I assume the chart is drawing a label even if an empty string is returned. So I tried to return null or false to tell the chart not to bother drawing the label but that only caused the chart to draw a bunch of nulls or falses.

So does anyone know if there is a way to tell the chart not to bother drawing a label?

jmayer
6 May 2013, 6:33 AM
A very late update of what I did in 2011/2012.

I could never find a way to tell the chart not to draw a label for an empty string and I did not want to dig around source code so it remained disabled. For a while the chart ran fine until I had to add 4 more series to it. After banging my head against the wall a lot, I found out drawing markers for each data point was very, very expensive.

So short story, for large data sets, do not use labels and markers.

Also sometime in 2012 I had to dig through the Ext.chart.series.Line code because it was averaging my data when I needed it to show exact and I found an inherit runtime issue. Throughout the day, the chart requests a few more points every 1 minute. In the code, whenever data is added to the store, the chart calls the drawSeries function and it reprocesses all the data in the store to generate a new SVG draw path and then applies it to the line sprite. What this means, as the chart accumulates more and more points throughout the day, the drawSeries function slowly takes longer and longer to run and the CPU process time slowly increases as a result.

I am currently working on expanding the Ext.chart.Chart and Ext.chart.series.Line classes to use less overhead when the Ext.data.Store.add event is called. If I have any luck, I will post the code in a separate thread.

clifficious
11 Jun 2013, 6:34 AM
Hi jmayer,

thanks for posting your answer in this forum.
I recently ran into the same problem. I have a set of data, which had the size of approximately 2500 elements. Loading all the data took the line graph ages. So I successfully narrowed the size down to about 400 elements. They are stored in an array. When I fill the store by calling the 'loadData' function, the chart still takes about 5 seconds to load, which in my eyes is far to much.

Where you able to extend the chart class to reduce the overhead? Also I'm not sure why it takes so long to load the data. Aren't the charts made for this size of data?

Thanks for your answers!

jmayer
11 Jun 2013, 11:41 AM
I was able to reduce the draw time. Running the webpage on the server, an Intel Atom dual core 1.60 GHz, the draw time for 9624 records went from 20 seconds to 8.3 seconds. This was with chrome.

Attached is the modified Ext.chart.series.Line code and the example code for the line series I used to test the modifications with. The modified code was taken from extjs 4.2.0 but it works with 4.2.1. I am not sure if it would run with earlier versions.

I will admit I am not sure how I reduced the draw time. Originally all the draw code was in the drawSeries function and I just broke it up into smaller pieces. Also for some reason, the extjs developers like to define a bunch of unnecessary variables, for example 'var me = this, chart = me.chart, shadow = chart.shadow...'--I think it is for debug purposes--and I removed a lot of that. I have been meaning to post my modifications with test cases and see if anyone knows why it draw faster.

Edit: The numbers above came from a chart with 7 line series.

clifficious
13 Jun 2013, 3:10 AM
thanks for the files. I'll try them out later.
Yesterday I noticed, that I did not change the type of the axis to numeric, but instead just left it with category, just like the tutorial. After this change I noticed a tremendous speed increase as well.