[4.1 RC2] Infinite Grid skips pages when using grouping.

[karlsnyder0]

Don-

One last thing. This post should probably not be marked as "Success! Looks like we've fixed this one." and instead marked as "You found a bug!" since it is technically not fixed and still an issue in the latest release.

From a user perspective the known status of a issue is sometimes as important as getting the issue addressed.

Thanks,

Karl

[dongryphon]

As it stands you'll have the same problems with a standard grid and infinite grid which is, with grouping you'll potentially never have enough data to satisfy the group because you don't have all of the records (because the records are paged). The only difference in paging between the standard grid and infinite grid is that the number of records that you need to fetch over a time period is different.

I realize that I am oversimplifying the complexity of the challenge with the infinite however I am trying to point out that the GroupingFeature will cause issues with all grids, because the grouping count can never be completely accurate.

You are correct that in standard paging the actual number of records in the group in the result set is unknown. If the groupTpl chooses to render that information, it may be that it has some meaning in the partial results in view. Otherwise, it would be best to remove that from the text of the group header.

The problems with infinite grids are compounded not because of the count of records in each group but the number of distinct groups. This is just because each group header will occupy space in the view and the algorithm we use to virtualize the scrolling space cannot really take that into account. There are a few places where we "project" the rendered view into the virtual space where all other records are counted as having a nominal height and then where we project backwards.

We plan to make another go at conceptualizing this interplay, but at present, I expect this will create a discontinuous result in the math due to the presence of group headers.

Having said this the root problem with this ticket is this. The infinite grid fetches and remembers the pages of data that it has fetched. These pages are stored in a PageMap. When a "re-group" happens, that is a change in how grouping works by turning it on or grouping by a different field, the sort must change because the data must come in sorted first by the grouped column. The first row that was originally fetched by the prefetch is refetched if you are on the first page, however with Ext 4.1 any data pages in the PageMap aren't refetched... and data is skipped.

The workaround I provided simple negatives the PageMap and forces a complete refetch when a new grouping is determined. Upon looking at the solution I provided it appears that the "

me.pageMap.clear();

" may be in the wrong place and should probably be in the "newGroup" if block, and I'm going to test this. The approach has worked for us just fine in our grids... which can contain several hundred thousand records.

We can certainly include these patches since they seem to help in practice.

- The grouping count will never be absolutely accurate in a standard paged or infinite grid until the next group value is seen; a new group value marks the end of the last group and the start of the next group.

- Grouping in a grid (standard paged or infinite) can be dangerous from a user perspective because stating a count could lead the end user to believe that a count is accurate when in fact it is impossible to determine a complete count with a grid unless we prefetch the grouping information before hand.

Correct. So unless the server response can include the group count (in each record probably), the best bet is to not display this value in the groupTpl... unless the partial count is helpful.

- When grouping with an infinite grid one should reset the grid to page 1 on a regroup.

- When grouping with an infinite grid any prefetched pages, or already fetched pages, are useless on a regroup.

Agreed. I think we can make these adjustments.

- We've found that grouping with an infinite grid and preventing the groups from being closed provides a better user experience. The value the grid grouping provides is always visual only. A user isn't going to be able to track through a group of 1000 records and notice the group boundaries. The smaller groups will make more of a difference to the user.

Yes, that would be an important compromise for infinite grid certainly since it strives to make it appear as if the data is continuous.

Of course, this all changes if a grid performs a local grouping, where all of the data that is grouped is already contained in the client.

Indeed. If that were the case, in theory, sufficient logic could be crafted to make everything work as one might hope. At present there is no special awareness of this in the grouping feature or infinite scrolling logic - but it is theoretically possible.

[dongryphon]

Don-

One last thing. This post should probably not be marked as "Success! Looks like we've fixed this one." and instead marked as "You found a bug!" since it is technically not fixed and still an issue in the latest release.

From a user perspective the known status of a issue is sometimes as important as getting the issue addressed.

Thanks,

Karl

I've reopened the ticket. That should correct the status on the next sync.

[karlsnyder0]

The problems with infinite grids are compounded not because of the count of records in each group but the number of distinct groups. This is just because each group header will occupy space in the view and the algorithm we use to virtualize the scrolling space cannot really take that into account. There are a few places where we "project" the rendered view into the virtual space where all other records are counted as having a nominal height and then where we project backwards.

I want to respond to this but first let me noodle for a bit on this. Thank you for addressing this complex feature.

[nbourdeau]

Hi,

I have a similar use case where users want Grouping with potential large datasets, it quickly becomes a performance issue ...
I've read all this thread. The discussion is really interesting !

I think that for scenarios where the dataset is not too big, it could be interesting to have a solution where the backends returns all the grouping info with the subset of data (group value and total count). This way, all information will be available for doing right calculations.

I know this will cause overhead on the backend but I think it is better this than the alternative to have the client do all the work and download all the data.

Also, to simplify things, maybe we should explore using another kind of rendering/GUI where, having recieved all the groups information from the backend, the user could pick the group he wants to see and than use "standard" infinite scrolling for that group by only adding a filter on the store.

What do you think ?