Support for Buffered Tree?

[jej2003]

Is there any plan for a buffered tree in the development plan? We currently have a requirement to support somewhere in the neighborhood of 3000 distinct nodes (from memory) but this obviously doesn't work well with the current implementation. Any thoughts would be appreciated.

Also if there are any tips on how to go about extending treepanel to support this I would be willing to take a stab at it.

[Greendrake]

I am very interested in this functionality as well and thinking whether I should go ahead writing my own implementation or wait for an official one. Would the team confirm or deny plans on buffered trees, please?

[SamuraiJack1]

In our products we support buffered+lockable tree panel: http://www.bryntum.com/examples/gant.../buffered.html

Required several overrides in TreeStore and Lockable mixin. I can share some code snippets if you want. In return, would be great if you could polish them and publish as an "ux".

[Greendrake]

Oh, this is cool!

I can share some code snippets if you want. In return, would be great if you could polish them and publish as an "ux".

I would love to contribute and publish. The only tiny thing that puzzles me is that according to this post, the way guaranteeRange works is going to be reviewed significantly in 4.1, so the overridings would have to be overhauled too. On the other hand, I need the solution now, so I look forward to polishing your code. Thanks in advance!

[xiami]

im interesting in it, it's super

[SamuraiJack1]

So, the 1st override is:

    // to enable the usage of the same NodeStore in both TreeViews
    // also to be able to configure the NodeStore being created
    Ext.tree.View.override({
        
        providedStore       : null, 
        storeConfig         : null,
    
        initComponent: function() {
            var me = this;
            
            if (me.initialConfig.animate === undefined) {
                me.animate = Ext.enableFx;
            }
            
            me.store = me.providedStore || Ext.create('Ext.data.NodeStore', Ext.apply({
                recursive: true,
                rootVisible: me.rootVisible,
                listeners: {
                    beforeexpand: me.onBeforeExpand,
                    expand: me.onExpand,
                    beforecollapse: me.onBeforeCollapse,
                    collapse: me.onCollapse,
                    scope: me
                }
            }, me.storeConfig || {}));
            
            if (me.node) {
                me.setRootNode(me.node);
            }
            me.animQueue = {};
            me.callParent(arguments);
        }
    });

See also: http://www.sencha.com/forum/showthre...-and-NodeStore.

The override will allow us to configure the underlaying "NodeStore" of the tree view with the "storeConfig" option (or provide our own store at all - "providedStore").


Then, the NodeStore of the view should be configured like this:

            lockedGrid.viewConfig.storeConfig = {
                buffered        : true,
                pageSize        : me.store.pageSize || 50,
                
                // never purge any data, we prefetch all up front
                purgePageCount  : 0,
                
                refreshFromTree : function () {
                    var eventsWereSuspended     = this.eventsSuspended;
                    
                    this.suspendEvents();
                    
                    this.removeAll();
                    
                    var root            = me.store.getRootNode(),
                        linearNodes     = [];
                    
                    var cascadeBy       = function (node, func) {
                        func(node);
                        
                        if (node.isExpanded()) {
                            var childNodes  = node.childNodes,
                                length      = childNodes.length;
                            
                            for (var k = 0; k < length; k++) {
                                cascadeBy(childNodes[ k ], func);
                            }
                        }
                    };
                    
                    cascadeBy(root, function (node) {
                        if (node != root) {
                            linearNodes.push(node);
                        }
                    });
                    
                    this.cacheRecords(linearNodes);
                    
                    if (!eventsWereSuspended) {
                        this.resumeEvents();
                    }
                }
            };

Note we added "buffered" option and a new method "refreshFromTree". This method will update the NodeStore from the tree.

Then there's a problem, that initial loading of the TreeStore is dead slow. Not just slow, but it will trigger the "script limit" in the IE7/8 for any big dataset. It is performed record-by-record using the bubbling events from the tree. And the tree view will try to respond to each event - all this makes the tree performance unacceptable on big datasets.

The solution will be:

    // much faster implementation of `fillNode` method for buffered case which uses `node.appendChild` with `suppressEvent` option
    // and bypasses all the events fireing/bubbling machinery, calling the `onNodeAdded` directly
    fillNode : function (node, records) {
        if (node.isRoot()) {
            this.fireEvent('root-fill-start', this, node, records);
        }
        
        var me = this,
            ln = records ? records.length : 0,
            i = 0, sortCollection;

        if (ln && me.sortOnLoad && !me.remoteSort && me.sorters && me.sorters.items) {
            sortCollection = Ext.create('Ext.util.MixedCollection');
            sortCollection.addAll(records);
            sortCollection.sort(me.sorters.items);
            records = sortCollection.items;
        }
        
        node.set('loaded', true);
        
        if (this.buffered) {
        
            for (; i < ln; i++) {
                // suppress the events -------|
                //                           \/            
                node.appendChild(records[i], true, true);
                
                // directly call 'onNodeAdded'
                this.onNodeAdded(null, records[i]);
                
                // register the node in tree (for `getNodeById` to work properly)
                this.tree.registerNode(records[i]);
            }
        } else {
            for (; i < ln; i++) {
                node.appendChild(records[i], false, true);
            }
        }
            
        if (node.isRoot()) {
            this.fireEvent('root-fill-end', this, node, records);
        }
        
        return records;
    },

Note, it will also fires the additional events "root-fill-start/end".

Then, somewhere in the tree view should be this code:

            treeStore.on('root-fill-start', function () {
                fillingRoot = true;
                
                nodeStore.suspendEvents();
                
                if (isBuffered) {
                    oldRootNode = nodeStore.node;
                    
                    // setting the root node of NodeStore to null - so we are now should update the NodeStore manually for all CRUD operations in tree
                    // with `refreshFromTree` call
                    nodeStore.setNode();
                }
            });
            
            treeStore.on('root-fill-end', function () {
                fillingRoot = false;
                
                if (isBuffered) {
                    nodeStore.refreshFromTree();

                    nodeStore.resumeEvents();
                    
                    nodeStore.guaranteeRange(0, (treeStore.pageSize || 50) - 1);
                    
                    me.refreshView();
                    
                } else {
                    nodeStore.resumeEvents();
                    
                    lockedView.refresh();
                    normalView.refresh();
                    
                }
            });
            
            if (isBuffered) {
                nodeStore.on('bufferchange', function () { me.refreshView(); });
                
                var updateNodeStore = function () {
                    if (fillingRoot) return;
                    
                    nodeStore.refreshFromTree();
                    
                    var rangeStart  = nodeStore.guaranteedStart,
                        rangeEnd    = nodeStore.guaranteedEnd;
                    
                    delete nodeStore.guaranteedStart;
                    delete nodeStore.guaranteedEnd;
                    
                    nodeStore.guaranteeRange(rangeStart, rangeEnd);
                };
                
                treeStore.on({
                    append      : updateNodeStore,
                    insert      : updateNodeStore,
                    remove      : updateNodeStore,
                    move        : updateNodeStore,
                    expand      : updateNodeStore,
                    collapse    : updateNodeStore,
                    sort        : updateNodeStore,
                    
                    buffer      : 1
                });
            }

With it, we ignore the initial loading, and doing a full manual refresh after it. Also doing a full refresh after any CRUD action.

And finally The 'bufferchange' event in the previous override goes from the customized scroller:

/**
 * A copy of Ext.grid.PagingScroller which always uses the own store reference (not from panel/view) 
 * 
 * @class Sch.scroller.Paging
 * @extends Ext.grid.Scroller
 *
 * @private
 */
Ext.define('Sch.scroller.Paging', {
    extend      : 'Ext.grid.Scroller',
    alias       : 'widget.schpagingscroller',
    
    //renderTpl: null,
    //tpl: [
    //    '<tpl for="pages">',
    //        '<div class="' + Ext.baseCSSPrefix + 'stretcher" style="width: {width}px;height: {height}px;"></div>',
    //    '</tpl>'
    //],
    
    /**
     * @cfg {Number} percentageFromEdge This is a number above 0 and less than 1 which specifies
     * at what percentage to begin fetching the next page. For example if the pageSize is 100
     * and the percentageFromEdge is the default of 0.35, the paging scroller will prefetch pages
     * when scrolling up between records 0 and 34 and when scrolling down between records 65 and 99.
     */
    percentageFromEdge: 0.35,

    /**
     * @cfg {Number} scrollToLoadBuffer This is the time in milliseconds to buffer load requests
     * when scrolling the PagingScrollbar.
     */
    scrollToLoadBuffer: 200,

    activePrefetch: true,

    chunkSize: 50,
    snapIncrement: 25,

    syncScroll: true,

    initComponent: function() {
        var me = this,
            ds = me.store;

        ds.on('guaranteedrange', this.onGuaranteedRange, this);
        this.callParent(arguments);
    },

    onGuaranteedRange: function(range, start, end) {
        var me = this,
            ds = me.store,
            rs;
        // this should never happen
        if (range.length && me.visibleStart < range[0].index) {
            return;
        }

        ds.loadRecords(range);
        
        // HACK: letting the others know that new portion of records were loaded
        ds.fireEvent('bufferchange', ds, range, start, end, this)
        // EOF HACK

        if (!me.firstLoad) {
            if (me.rendered) {
                me.invalidate();
            } else {
                me.on('afterrender', this.invalidate, this, {single: true});
            }
            me.firstLoad = true;
        } else {
            // adjust to visible
            me.syncTo();
        }
    },

    syncTo: function() {
        var me            = this,
            pnl           = me.getPanel(),
            store         = this.store,
            scrollerElDom = this.scrollEl.dom,
            rowOffset     = me.visibleStart - store.guaranteedStart,
            scrollBy      = rowOffset * me.rowHeight,
            scrollHeight  = scrollerElDom.scrollHeight,
            clientHeight  = scrollerElDom.clientHeight,
            scrollTop     = scrollerElDom.scrollTop,
            useMaximum;

        // BrowserBug: clientHeight reports 0 in IE9 StrictMode
        // Instead we are using offsetHeight and hardcoding borders
        if (Ext.isIE9 && Ext.isStrict) {
            clientHeight = scrollerElDom.offsetHeight + 2;
        }

        // This should always be zero or greater than zero but staying
        // safe and less than 0 we'll scroll to the bottom.
        useMaximum = (scrollHeight - clientHeight - scrollTop <= 0);
        this.setViewScrollTop(scrollBy, useMaximum);
    },

    getPageData : function(){
        var panel = this.getPanel(),
            store = this.store,
            totalCount = store.getTotalCount();

        return {
            total : totalCount,
            currentPage : store.currentPage,
            pageCount: Math.ceil(totalCount / store.pageSize),
            fromRecord: ((store.currentPage - 1) * store.pageSize) + 1,
            toRecord: Math.min(store.currentPage * store.pageSize, totalCount)
        };
    },

    onElScroll: function(e, t) {
        var me = this,
            panel = me.getPanel(),
            store = this.store,
            pageSize = store.pageSize,
            guaranteedStart = store.guaranteedStart,
            guaranteedEnd = store.guaranteedEnd,
            totalCount = store.getTotalCount(),
            numFromEdge = Math.ceil(me.percentageFromEdge * store.pageSize),
            position = t.scrollTop,
            visibleStart = Math.floor(position / me.rowHeight),
            view = panel.down('tableview'),
            viewEl = view.el,
            visibleHeight = viewEl.getHeight(),
            visibleAhead = Math.ceil(visibleHeight / me.rowHeight),
            visibleEnd = visibleStart + visibleAhead,
            prevPage = Math.floor(visibleStart / store.pageSize),
            nextPage = Math.floor(visibleEnd / store.pageSize) + 2,
            lastPage = Math.ceil(totalCount / store.pageSize),
            //requestStart = visibleStart,
            requestStart = Math.floor(visibleStart / me.snapIncrement) * me.snapIncrement,
            requestEnd = requestStart + pageSize - 1,
            activePrefetch = me.activePrefetch;

        me.visibleStart = visibleStart;
        me.visibleEnd = visibleEnd;

        me.syncScroll = true;
        if (totalCount >= pageSize) {
            // end of request was past what the total is, grab from the end back a pageSize
            if (requestEnd > totalCount - 1) {
                this.cancelLoad();
                if (store.rangeSatisfied(totalCount - pageSize, totalCount - 1)) {
                    me.syncScroll = true;
                }
                store.guaranteeRange(totalCount - pageSize, totalCount - 1);
            // Out of range, need to reset the current data set
            } else if (visibleStart < guaranteedStart || visibleEnd > guaranteedEnd) {
                if (store.rangeSatisfied(requestStart, requestEnd)) {
                    this.cancelLoad();
                    store.guaranteeRange(requestStart, requestEnd);
                } else {
                    store.mask();
                    me.attemptLoad(requestStart, requestEnd);
                }
                // dont sync the scroll view immediately, sync after the range has been guaranteed
                me.syncScroll = false;
            } else if (activePrefetch && visibleStart < (guaranteedStart + numFromEdge) && prevPage > 0) {
                me.syncScroll = true;
                store.prefetchPage(prevPage);
            } else if (activePrefetch && visibleEnd > (guaranteedEnd - numFromEdge) && nextPage < lastPage) {
                me.syncScroll = true;
                store.prefetchPage(nextPage);
            }
        }

        if (me.syncScroll) {
            me.syncTo();
        }
    },

    getSizeCalculation: function() {
        // Use the direct ownerCt here rather than the scrollerOwner
        // because we are calculating widths/heights.
        var owner = this.ownerGrid,
            view   = owner.getView(),
            store  = this.store,
            dock   = this.dock,
            elDom  = this.el.dom,
            width  = 1,
            height = 1;

        if (!this.rowHeight) {
            this.rowHeight = view.el.down(view.getItemSelector()).getHeight(false, true);
        }

        // If the Store is *locally* filtered, use the filtered count from getCount.
        height = store[(!store.remoteFilter && store.isFiltered()) ? 'getCount' : 'getTotalCount']() * this.rowHeight;

        if (isNaN(width)) {
            width = 1;
        }
        if (isNaN(height)) {
            height = 1;
        }
        return {
            width: width,
            height: height
        };
    },

    attemptLoad: function(start, end) {
        var me = this;
        if (!me.loadTask) {
            me.loadTask = Ext.create('Ext.util.DelayedTask', me.doAttemptLoad, me, []);
        }
        me.loadTask.delay(me.scrollToLoadBuffer, me.doAttemptLoad, me, [start, end]);
    },

    cancelLoad: function() {
        if (this.loadTask) {
            this.loadTask.cancel();
        }
    },

    doAttemptLoad:  function(start, end) {
        var store = this.store;
        store.guaranteeRange(start, end);
    },

    setViewScrollTop: function(scrollTop, useMax) {
        var owner = this.getPanel(),
            items = owner.query('tableview'),
            i = 0,
            len = items.length,
            center,
            centerEl,
            calcScrollTop,
            maxScrollTop,
            scrollerElDom = this.el.dom;
            
        var store   = this.store;

        owner.virtualScrollTop = scrollTop;

        center = items[1] || items[0];
        centerEl = center.el.dom;

        maxScrollTop = ((store.pageSize * this.rowHeight) - centerEl.clientHeight);
        calcScrollTop = (scrollTop % ((store.pageSize * this.rowHeight) + 1));
        if (useMax) {
            calcScrollTop = maxScrollTop;
        }
        if (calcScrollTop > maxScrollTop) {
            //Ext.Error.raise("Calculated scrollTop was larger than maxScrollTop");
            return;
            // calcScrollTop = maxScrollTop;
        }
        for (; i < len; i++) {
            items[i].el.dom.scrollTop = calcScrollTop;
        }
    }
});

See: http://www.sencha.com/forum/showthre...126#post623126

I just copied the scroller sources and fixed the usage of store and added a 'bufferchange' event.

After all tricks, the initial loading for the tree store triggers a script limit for datasets with > 1000.

Feel free to ask questions and good luck!

[Greendrake]

SamuraiJack1, thank you for the code excerpts. I am currently working on a solid/robust/reusable bundle of extensions based on your code. Could you please advise if you have tested it with the latest build (4.0.5), does it work fine on it? Thanks.

[SamuraiJack1]

Seems to work fine with it, yes.

[Dumbledore]

any working example? I don´t know where to put

treeStore.on('root-fill-start', function () {

...

hmm. i must explain more:

I have a MVC-Application. I put the storeConfig inside my tree like

viewConfig : {
    storeConfig : {
        [...]
    }
}

and the fillNode inside my store. So i hav no explicity treeview, where must i put the treeStore.on[...]?

[SamuraiJack1]

any working example? I don´t know where to put

treeStore.on('root-fill-start', function () {

...

hmm. i must explain more:

I have a MVC-Application. I put the storeConfig inside my tree like

viewConfig : {
    storeConfig : {
        [...]
    }
}

and the fillNode inside my store. So i hav no explicity treeview, where must i put the treeStore.on[...]?

But you still have the tree store instance available somewhere, right? It has to be the subclass of Ext.data.TreeStore with the override for "fillNode" method. Then you can subscribe to "root-fill-start/end" in any convenient place.