Success! Looks like we've fixed this one. According to our records the fix was applied for EXTJS-5934 in a recent build.
  1. #1
    Sencha Premium Member karlsnyder0's Avatar
    Join Date
    Mar 2010
    Location
    Maryland, USA
    Posts
    82
    Vote Rating
    12
    karlsnyder0 is on a distinguished road

      0  

    Default [4.1 RC2] Infinite Grid skips pages when using grouping.

    [4.1 RC2] Infinite Grid skips pages when using grouping.


    REQUIRED INFORMATION

    Ext version tested:
    • Ext 4.1 rev RC2
    Browser versions tested against:
    • Chrome 17.0.963.83 (Mac)
    Description:
    • When using Infinite Grid and GroupingFeature together page fetches are skipped during scrolling through the grid results.
    Steps to reproduce the problem:
    • Use the /examples/grid/infinite-scroll.html example and modify the source to use remote grouping as described in the code below.
    • Run the modified example in a browser.
    • Group by a field by selecting the field's column menu and click "Group By This Field".
    • Scroll the grid down so that it fetches the second page. Note that the second page isn't fetched next and instead the 4th page is fetched.
    Code:
        var store = Ext.create('Ext.data.Store', {
            id: 'store',
            model: 'ForumThread',
            remoteGroup: true,
            // allow the grid to interact with the paging scroller by buffering
            buffered: true,
            pageSize: 100,
            proxy: {
                // load using script tags for cross domain, if the data in on the same domain as
                // this page, an HttpProxy would be better
                type: 'jsonp',
                url: 'http://www.sencha.com/forum/remote_topics/index.php',
                reader: {
                    root: 'topics',
                    totalProperty: 'totalCount'
                },
                // sends single sort as multi parameter
                simpleSortMode: true
            },
            sorters: [{
                property: 'threadid',
                direction: 'ASC'
            }],
            autoLoad: true
        });
    The result that was expected:
    • After "Group By This Field" is selected the page cache should reset and pages should be fetched in sequential order.
    The result that occurs instead:
    • After "Group By This Field" is selected pages are skipped during scroll.
    HELPFUL INFORMATION

    Possible fix:
    • No clue. I am STILL debugging this one. A suggestion would be appreciated.
    Additional CSS used:
    • Not applicable
    Operating System:
    • Mac OSX Lion

  2. #2
    Sencha Premium Member karlsnyder0's Avatar
    Join Date
    Mar 2010
    Location
    Maryland, USA
    Posts
    82
    Vote Rating
    12
    karlsnyder0 is on a distinguished road

      0  

    Default


    I have just confirmed that this bug also exists in 4.1 RC3. Any ideas on this?

  3. #3
    Sencha - Ext JS Dev Team Animal's Avatar
    Join Date
    Mar 2007
    Location
    Notts/Redwood City
    Posts
    30,508
    Vote Rating
    56
    Animal has a spectacular aura about Animal has a spectacular aura about Animal has a spectacular aura about

      0  

    Default


    How can this possibly work?

    The store only ever holds (viewSize + ~20) records, that's the whole point of buffered scrolling.

  4. #4
    Sencha Premium Member karlsnyder0's Avatar
    Join Date
    Mar 2010
    Location
    Maryland, USA
    Posts
    82
    Vote Rating
    12
    karlsnyder0 is on a distinguished road

      0  

    Default


    Quote Originally Posted by Animal View Post
    How can this possibly work?

    The store only ever holds (viewSize + ~20) records, that's the whole point of buffered scrolling.
    You are kidding, right?

    Responses like this are very frustrating. I spent many hours and a sleepless night trying to figure out this problem. I didn't just throw this together on a whim.

    Did you run the example with the changes I provided because if you did you would see that the store grabs the first 3 pages of data? (300 records). Also keep in mind that I didn't write ExtJS4 or the examples, you did. I simply made minor changes to expose a bug.

    If you do a little digging in the Store code you will see that it actually stores many pages of records for a buffered store, not viewSize + ~20 as you suggest.

    I did find a workaround for this problem. Keep in mind that the backend needs to honor the "group" parameter for the group sorting to work properly. The code fix below does invalidate the pageMap when grouping (as sorting and filtering already does).

    Code:
    Ext.override(Ext.data.Store, {
        group: function(groupers, direction) {
            var me = this,
                prefetchData = me.pageMap,
                hasNew = false,
                grouper,
                newGroupers;
    
    
            if (Ext.isArray(groupers)) {
                newGroupers = groupers;
            } else if (Ext.isObject(groupers)) {
                newGroupers = [groupers];
            } else if (Ext.isString(groupers)) {
                grouper = me.groupers.get(groupers);
    
    
                if (!grouper) {
                    grouper = {
                        property : groupers,
                        direction: direction
                    };
                    newGroupers = [grouper];
                } else if (direction === undefined) {
                    grouper.toggle();
                } else {
                    grouper.setDirection(direction);
                }
            }
    
    
            if (newGroupers && newGroupers.length) {
                hasNew = true;
                newGroupers = me.decodeGroupers(newGroupers);
                me.groupers.clear();
                me.groupers.addAll(newGroupers);
            }
    
    
            if (me.remoteGroup) {
                if (me.buffered) {
                    prefetchData.clear();
                }
                me.load({
                    scope: me,
                    callback: me.fireGroupChange
                });
            } else {
                // need to explicitly force a sort if we have groupers
                me.sort(null, null, null, hasNew);
                me.fireGroupChange();
            }
        },
    
    
        /**
         * Clear any groupers in the store
         */
        clearGrouping: function() {
            var me       = this,
                prefetchData = me.pageMap,
                groupers = me.groupers.items,
                gLen     = groupers.length,
                grouper, g;
    
    
            for (g = 0; g < gLen; g++) {
                grouper = groupers[g];
    
    
                me.sorters.remove(grouper);
            }
    
    
            me.groupers.clear();
            if (me.remoteGroup) {
                if (me.buffered) {
                    prefetchData.clear();
                }
                me.load({
                    scope: me,
                    callback: me.fireGroupChange
                });
            } else {
                me.sort();
                me.fireEvent('groupchange', me, me.groupers);
            }
        }
    });

  5. #5
    Sencha Premium Member karlsnyder0's Avatar
    Join Date
    Mar 2010
    Location
    Maryland, USA
    Posts
    82
    Vote Rating
    12
    karlsnyder0 is on a distinguished road

      0  

    Default


    This bug is NOT FIXED in 4.1.0 Sprint 13 (GA). An updated workaround for 4.1.0 Sprint 13 (GA) is as follows:

    Code:
    Ext.override(Ext.data.Store, {
        /**
         * Groups data inside the store.
         * @param {String/Object[]} groupers Either a string name of one of the fields in this Store's
         * configured {@link Ext.data.Model Model}, or an Array of grouper configurations.
         * @param {String} [direction="ASC"] The overall direction to group the data by.
         */
        group: function(groupers, direction) {
            var me = this,
                hasNew = false,
                grouper,
                newGroupers;
        
            if (Ext.isArray(groupers)) {
                newGroupers = groupers;
            } else if (Ext.isObject(groupers)) {
                newGroupers = [groupers];
            } else if (Ext.isString(groupers)) {
                grouper = me.groupers.get(groupers);
        
                if (!grouper) {
                    grouper = {
                        property : groupers,
                        direction: direction
                    };
                    newGroupers = [grouper];
                } else if (direction === undefined) {
                    grouper.toggle();
                } else {
                    grouper.setDirection(direction);
                }
            }
        
            if (newGroupers && newGroupers.length) {
                hasNew = true;
                newGroupers = me.decodeGroupers(newGroupers);
                me.groupers.clear();
                me.groupers.addAll(newGroupers);
            }
        
            if (me.remoteGroup) {
                if (me.buffered) {
                    me.pageMap.clear();
                }
                me.load({
                    scope: me,
                    callback: me.fireGroupChange
                });
            } else {
                // need to explicitly force a sort if we have groupers
                me.sort(null, null, null, hasNew);
                me.fireGroupChange();
            }
        }
    });

  6. #6
    Sencha Premium Member karlsnyder0's Avatar
    Join Date
    Mar 2010
    Location
    Maryland, USA
    Posts
    82
    Vote Rating
    12
    karlsnyder0 is on a distinguished road

      0  

    Default


    Would someone please reopen this bug? This is not fixed in 4.1 and appears nowhere in the "known issues" of the 4.1.1 nightly.

  7. #7
    Sencha - Ext JS Dev Team dongryphon's Avatar
    Join Date
    Jul 2009
    Posts
    1,367
    Vote Rating
    136
    dongryphon is a splendid one to behold dongryphon is a splendid one to behold dongryphon is a splendid one to behold dongryphon is a splendid one to behold dongryphon is a splendid one to behold dongryphon is a splendid one to behold

      0  

    Default


    As it stands, this is internally viewed as a feature request rather than a bug. So the bug was closed and then accidentally synchronized here as "fixed". Apologies for the confusion and frustration this has created.

    We will discuss this further, but there are many challenges with this combination of features. Following are just the highlights.

    The first problem is that the rendered data is quite small (view size +/- ~20 records) and the algorithm has to account for the extra rows numerically to provide a proper scroll range and position. This is done by assigning approximate row heights to the non-rendered rows. The key though is that we know the total number of rows from the server response.

    This is not true of grouping. In the worst-case scenario, there could be 1 group per record (where no records have common values in the group field). Even if we knew the number of groups in the data set, our current algorithm would also need to know the number of groups above and below the rendered section of the data in order to add appropriate scroll range for them and position the current section within that range.

    Even more complex: collapsing of the groups would require sequential fetching of pages until we finally got past the end of the group. Imagine a group of 10k records.

    The only way this could be made to work is if 100% of the data set were loaded into the cache. In this case, all we would be talking about is buffered rendering, not buffered access to the data. This may still be quite useful even with this restriction. If there is a clever way to avoid this restriction, it has not come up in our discussions so far.
    Don Griffin
    Ext JS Development Team Lead

    Check the docs. Learn how to (properly) report a framework issue and a Sencha Cmd issue

    "Use the source, Luke!"

  8. #8
    Sencha - Ext JS Dev Team dongryphon's Avatar
    Join Date
    Jul 2009
    Posts
    1,367
    Vote Rating
    136
    dongryphon is a splendid one to behold dongryphon is a splendid one to behold dongryphon is a splendid one to behold dongryphon is a splendid one to behold dongryphon is a splendid one to behold dongryphon is a splendid one to behold

      0  

    Default


    In a nutshell, I believe the problems come from trying to calculate the position in the data set from the position in the view and vise-verse, given the group headers appearing in the rendered data "unexpectedly" ... that is, unexpected by the paging scroller which only understands rows. The complexity of accounting for grouping is as I explained above.
    Don Griffin
    Ext JS Development Team Lead

    Check the docs. Learn how to (properly) report a framework issue and a Sencha Cmd issue

    "Use the source, Luke!"

  9. #9
    Sencha - Ext JS Dev Team dongryphon's Avatar
    Join Date
    Jul 2009
    Posts
    1,367
    Vote Rating
    136
    dongryphon is a splendid one to behold dongryphon is a splendid one to behold dongryphon is a splendid one to behold dongryphon is a splendid one to behold dongryphon is a splendid one to behold dongryphon is a splendid one to behold

      0  

    Default


    On the work around code you've posted, what behaviors does it correct and what problems do you still have with that correction in place? I believe that we will find some of the bugs you are encountering can be fixed, but the overall combination will still have problems as I've described.
    Don Griffin
    Ext JS Development Team Lead

    Check the docs. Learn how to (properly) report a framework issue and a Sencha Cmd issue

    "Use the source, Luke!"

  10. #10
    Sencha Premium Member karlsnyder0's Avatar
    Join Date
    Mar 2010
    Location
    Maryland, USA
    Posts
    82
    Vote Rating
    12
    karlsnyder0 is on a distinguished road

      0  

    Default


    Don-

    Thank you for explaining the pieces of Infinite Grid and GroupingFeature that make this a complex. I'm familiar with the variables that you've described when grouping and we weighed the potential of using the GroupingFeature with the Infinite Grid.

    As it stands you'll have the same problems with a standard grid and infinite grid which is, with grouping you'll potentially never have enough data to satisfy the group because you don't have all of the records (because the records are paged). The only difference in paging between the standard grid and infinite grid is that the number of records that you need to fetch over a time period is different.

    I realize that I am oversimplifying the complexity of the challenge with the infinite however I am trying to point out that the GroupingFeature will cause issues with all grids, because the grouping count can never be completely accurate.

    Having said this the root problem with this ticket is this. The infinite grid fetches and remembers the pages of data that it has fetched. These pages are stored in a PageMap. When a "re-group" happens, that is a change in how grouping works by turning it on or grouping by a different field, the sort must change because the data must come in sorted first by the grouped column. The first row that was originally fetched by the prefetch is refetched if you are on the first page, however with Ext 4.1 any data pages in the PageMap aren't refetched... and data is skipped.

    The workaround I provided simple negatives the PageMap and forces a complete refetch when a new grouping is determined. Upon looking at the solution I provided it appears that the "
    me.pageMap.clear();

    " may be in the wrong place and should probably be in the "newGroup" if block, and I'm going to test this. The approach has worked for us just fine in our grids... which can contain several hundred thousand records.

    Some facts that we've noticed with grid grouping.

    - The grouping count will never be absolutely accurate in a standard paged or infinite grid until the next group value is seen; a new group value marks the end of the last group and the start of the next group.

    - Grouping in a grid (standard paged or infinite) can be dangerous from a user perspective because stating a count could lead the end user to believe that a count is accurate when in fact it is impossible to determine a complete count with a grid unless we prefetch the grouping information before hand.

    - When grouping with an infinite grid one should reset the grid to page 1 on a regroup.

    - When grouping with an infinite grid any prefetched pages, or already fetched pages, are useless on a regroup.

    - We've found that grouping with an infinite grid and preventing the groups from being closed provides a better user experience. The value the grid grouping provides is always visual only. A user isn't going to be able to track through a group of 1000 records and notice the group boundaries. The smaller groups will make more of a difference to the user.

    Of course, this all changes if a grid performs a local grouping, where all of the data that is grouped is already contained in the client.

    I'm going to take another look at the solution I provided, trace through the prefetch code and determine if the pagemap clear is in the wrong place.

    -Karl