Grid Performance for very large datasets (50000+ rows)

[ssenecal]

I am currently working on a project that requires several grids on various pages to be able to support 50K rows of data with many columns, and paging isn't an option. Does anyone know of any existing solutions to this problem? Or perhaps, is this something that is in the GXT roadmap? Currently I am using an in-house solution that involves paging the results via RPC as the user scrolls down the grid. This gives acceptable performance, but of course the problem is that once the grid contains a large number of rows it becomes unusable. I think the ideal solution would be to implement windowing on top of the RPC paging. Ie, only render, say, 300 rows within the Grid at any given point. Once the user has scrolled out of that window, re-render a new window of data. I think the current GXT Grid framework would allow for this, and if such a solution doesnt already exist I plan on rolling my own in my next iteration of development. However, why reinvent the wheel right? :)

Any pointers would be appreciated... tips, tricks, existing implementations..

[jpnet]

The ExtJS library has a solution for that, it's called LiveGrid. I believe it's a plugin developed by a third party. It's my understanding that LiveGrid has been a requested feature to the GXT team for some time. However, to my knowledge, I don't believe there is currently a LiveGrid solution for GXT. I may be wrong though.

-JP

[ssenecal]

I just checked out LiveGrid, and it looks like exactly what I was talking about! Too bad it doesn't have a GWT implementation yet. If anyone on the Ext GXT team knows about a similar feature in the pipeline, I would love to hear about it.

[kht]

hi darrell,

this would be a really great feature and i also need this - paging is in some cases not an option. i did such a widget in java/eclipse.
just remember height of each row - any unrendered row has an "average" height.
for non visible rows, add one div before first visible with sum of all row heights and one after last visible.
if rows come into view through scrolling or page up/down keys, these rows are rendered and shown. rows which get out of view are released in background....
for selection, just remember the state.
even the translation of beans into beanmodels should be done only on request as this is very expensive.
thats all....

thanks for all your efforts!!

[les]

+1 GXT version needs a LiveGrid !!!

[baskor]

I'm all in for live grid. Paging is not needed in these AJAX days!

[jpnet]

I'm not sure why everyone wants a live grid so bad. Can someone give me some rational insight as to why a live grid is necessary? It seems to me that if you want to display that much data to your users, you need to rethink your UI design. That's my 2 cents.

I would like to hear what others think.

-JP

[baskor]

@JP
How often do you see paging in desktop apps? Almost never, because it makes it difficult to browse through your data. ExtGWT is a RIA framework that let's us develop desktop like applications on the web, and therefore we need LiveGrid. Otherwise we can't develop applications like MS Outlook that lets users scroll through thousands of mails.

How do you suppose one should redesign the UI, so that users can quickly browse through thousands of mails?

[jpnet]

How often do you see paging in desktop apps? Almost never, because it makes it difficult to browse through your data. ExtGWT is a RIA framework that let's us develop desktop like applications on the web, and therefore we need LiveGrid. Otherwise we can't develop applications like MS Outlook that lets users scroll through thousands of mails.
I don't think it matters much whether we're referring to a desktop application or a web application. I think that any UI that wants to display thousands of records to a user at once needs to be redesigned.



How do you suppose one should redesign the UI, so that users can quickly browse through thousands of mails?
I think this statement is humorous. Please note, I am not intending to insult you either. But, asking your users to *quickly* browse through thousands of emails seems to be wasting users time. I believe that an application that contains that kind of data should contain functionality to allow users to quickly search or filter large data sets, so that users can find the data that is important to them. You could say I take on Google's mantra for Gmail, which is, "search don't sort."

Now, I'm not saying that applications that display large data sets are poorly designed, I just think in most cases they can be designed in a better manner. However, I would like to know of examples where it is absolutely necessary to display large data sets to users.

Thanks,

JP

[jratcliff]

@jpnet - I don't think the issue is displaying large data sets to users. Both paging and scrolling solutions for grids still only display a subset of the data. The mechanism to get to this subset is just different.

With scrolling you allow users more control of which records/rows are displayed in the grid. For example, if a user really wants to see rows 20 - 50 and a paging solution shows 30 rows per page the user would then be forced to look at the bottom of page 1 to see rows 20 - 30 and would then have to go to the next page to see rows 31-50. However, with a scrolling solution the user could scroll the grid until row 20 was at the top and then rows 31-50+ would just automatically come into view and thus all 20 - 50 rows would then be visible.

So again, it's not about displaying a large data set, its about giving users more control on which "subset" of data they wish to see.

[les]

I don't think it matters much whether we're referring to a desktop application or a web application. I think that any UI that wants to display thousands of records to a user at once needs to be redesigned.

I think this statement is humorous. Please note, I am not intending to insult you either. But, asking your users to *quickly* browse through thousands of emails seems to be wasting users time. I believe that an application that contains that kind of data should contain functionality to allow users to quickly search or filter large data sets, so that users can find the data that is important to them. You could say I take on Google's mantra for Gmail, which is, "search don't sort."

Now, I'm not saying that applications that display large data sets are poorly designed, I just think in most cases they can be designed in a better manner. However, I would like to know of examples where it is absolutely necessary to display large data sets to users.

Thanks,

JP


Philosophically, I agree with you. Based upon usability sessions I have been part of, users cannot comprehend large amounts of data. A couple 100 rows is about the max.

However, also from a user experience standpoint, when the user performs a search query and the result set is 100,000 rows, the UI should reflect this. It doesn't mean that the UI has to retrieve all the data, but it should be able to present it to the user in an accurate way. For example, the scroll bars should represent the amount of data, etc... LiveGrid does that and I think that is one aspect of what makes it nice.

Paging arbitrarily chunks data into groups that are not necessarily meaningful to the user. Again, LiveGrid does this nicely.

We live in a world with massive amounts of data. We need UI controls that can allow the users to easily visualize large amounts of data.

YMMV, just MHO.

[baskor]

I don't think it matters much whether we're referring to a desktop application or a web application. I think that any UI that wants to display thousands of records to a user at once needs to be redesigned.

I think this statement is humorous. Please note, I am not intending to insult you either. But, asking your users to *quickly* browse through thousands of emails seems to be wasting users time. I believe that an application that contains that kind of data should contain functionality to allow users to quickly search or filter large data sets, so that users can find the data that is important to them. You could say I take on Google's mantra for Gmail, which is, "search don't sort."

Now, I'm not saying that applications that display large data sets are poorly designed, I just think in most cases they can be designed in a better manner. However, I would like to know of examples where it is absolutely necessary to display large data sets to users.

Thanks,

JP

I was expecting that reply when posting :) and I agree with you to some extent. I know searching is the best way of finding data, and that filters can help. But I think it is important to visualize reality to users, as someone already said. If a user has a folder containing 500 items, the UI should visualize this. Many users, including my self, have no problem with comprehending large amount of data.

Lets say you are absolutely right, that an application is badly designed if it needs LiveGrid. Then its also badly designed if it uses paging. Lets just drop paging from ExtGWT and set an upper limit of, say 20 records in a Grid.

Well some of us don't mind displaying large amount of data. Our only option is to use paging, and I personally hate paging.

If you where forced to choose between the two evils (paging and LiveGrid) what would you then choose?

[jpnet]

OK, I can concede to the points made about choosing between a live grid and a paging grid. I can also agree that it is necessary for your UI to somehow reflect the amount of data in the application/system. I suppose, I should have been maybe a bit more clear; my intent was to address the title of this thread.



If you where forced to choose between the two evils (paging and LiveGrid) what would you then choose?


Part of me wants to say that I would choose a live grid. I can see how it makes it's more of a convenience for users to browse through the data. Another part of me thinks that user may come to rely on a live grid over using search functionality. If that's the case, one could argue that the search functionality isn't designed well enough then. To be honest, I'm not sure. But thinking about this, reminds me of a quote from the great book "GUI Bloopers 2.0." The quote is supposedly from a usability test participant stating "I was in a hurry, so I did it the long way." It makes me think of browsing through the large data sets... live grid or not.

Interesting topic indeed.

-JP

[kht]

@jpnet - I don't think the issue is displaying large data sets to users. Both paging and scrolling solutions for grids still only display a subset of the data. The mechanism to get to this subset is just different.
...
So again, it's not about displaying a large data set, its about giving users more control on which "subset" of data they wish to see.

i totally agree with jratcliff.

take a book as an example. my widget which i have done in java/eclipse renders books. a whole book at once (just taking milliseconds for the whole book with thousands of pages). like with any physical book you can just access any page immediatly. as fetching and rendering was so fast, that users see the pages during dragging the scrollbar thumb.

it is possible to show paragraph by paragraph - but it is not natural with paging. a paragraph will not fit into the current visible rectangle (so you need scrolling additionally or have large sections blank maybe).

on the other side some things are impossible with paging: try mark from word 3 in paragraph 2 to word 25 on paragraph 1000 and copy paste this text into your word processor..... try make this with paging...... same for search results where i show fragments of paragpraphs with marked search result. try copy paste parts of this with paging. simply not possble.

of course we can say: each book in the world should have only one page and you should carefully select in a library which book you take. at least books are much faster to read then... (just joking)

both mechanisms are clearly necessary.
looking forward to see a really fast live grid from gxt team ;)

regards, kht

[kht]

hi darrell,

will this find it's way into the 2.0 roadmap?
would be really great.

thanks!

regards, kht

[drz]

hi darrell,

will this find it's way into the 2.0 roadmap?
would be really great.

thanks!

regards, kht

Hi Gwt Team,

mid 2009?

[jpnet]

Hi Gwt Team,

mid 2009?


How about now? :D

http://extjs.com/forum/showthread.php?t=54582

[ssenecal]

Any word from the Ext GWT team on this? Will LiveGrid officially be included in the v2.0 release slated for "early 2009"? Will it only support flat tables, or will it support hierarchical/tree views as well?