Success! Looks like we've fixed this one. According to our records the fix was applied for TOUCH-4074 in a recent build.
  1. #1
    Sencha User
    Join Date
    Dec 2008
    Location
    Mainz
    Posts
    241
    Vote Rating
    1
    crp_spaeth is on a distinguished road

      1  

    Default Performancetuning of Store by optimizing Ext.util.Collection

    Performancetuning of Store by optimizing Ext.util.Collection


    Hey Guys,

    I am currently running some performance test on the Ext.data.Store and syncing back and forth with different Proxies like Ext.data.proxy.SQL.

    I run into some bottlenecks which are kindly fatal with a big amount of records managed by The store.

    While profiling I located two bottlenecks in the replace method of the internal used Ext.util.Collection class.

    In 2.1 the Method looks like this:

    Code:
       replace: function(oldKey, item) {
            var me = this,
                sorted = me.sorted,
                filtered = me.filtered,
                filterable = me.mixins.filterable,
                items = me.items,
                keys = me.keys,
                all = me.all,
                map = me.map,
                returnItem = null,
                oldItemsLn = items.length,
                oldItem, index, newKey;
    
    
            if (arguments.length == 1) {
                item = oldKey;
                oldKey = newKey = me.getKey(item);
            } else {
                newKey = me.getKey(item);
            }
    
    
            oldItem = map[oldKey];
            if (typeof oldKey == 'undefined' || oldKey === null || typeof oldItem == 'undefined') {
                 return me.add(newKey, item);
            }
    
    
            me.map[newKey] = item;
            if (newKey !== oldKey) {
                delete me.map[oldKey];
            }
    
    
            if (sorted && me.getAutoSort()) {
                Ext.Array.remove(items, oldItem);
                Ext.Array.remove(keys, oldKey);
                Ext.Array.remove(all, oldItem);
    
    
                all.push(item);
    
    
                me.dirtyIndices = true;
    
    
                if (filtered && me.getAutoFilter()) {
                    // If the item is now filtered we check if it was not filtered
                    // before. If that is the case then we subtract from the length
                    if (filterable.isFiltered.call(me, item)) {
                        if (oldItemsLn !== items.length) {
                            me.length--;
                        }
                        return null;
                    }
                    // If the item was filtered, but now it is not anymore then we
                    // add to the length
                    else if (oldItemsLn === items.length) {
                        me.length++;
                        returnItem = item;
                    }
                }
    
    
                index = this.findInsertionIndex(items, item);
    
    
                Ext.Array.splice(keys, index, 0, newKey);
                Ext.Array.splice(items, index, 0, item);
            } else {
                if (filtered) {
                    if (me.getAutoFilter() && filterable.isFiltered.call(me, item)) {
                        if (items.indexOf(oldItem) !== -1) {
                            Ext.Array.remove(items, oldItem);
                            Ext.Array.remove(keys, oldKey);
                            me.length--;
                            me.dirtyIndices = true;
                        }
                        return null;
                    }
                    else if (items.indexOf(oldItem) === -1) {
                        items.push(item);
                        keys.push(newKey);
                        me.indices[newKey] = me.length;
                        me.length++;
                        return item;
                    }
                }
    
    
                index = me.items.indexOf(oldItem);
    
    
                keys[index] = newKey;
                items[index] = item;
                this.dirtyIndices = true;
            }
    
    
            return returnItem;
        },
    In this Implementation the relativly slow (O(n)) indexOf method of the item-Array is used to get the index of the given item. By using the Collections-Class "indexOf"-Method

    Code:
         indexOf: function(item) {
            if (this.dirtyIndices) {
                this.updateIndices();
            }
    
            var index = this.indices[this.getKey(item)];
            return (index === undefined) ? -1 : index;
        },
    which uses the member indices to internal hashmap the keys to the index (~O(1)) tweaks this method a lot.

    The other bottleneck is in the logic of setting the dirtyIndices flag to true. Setting this flag will intearnaly lead to rehashing all indices. As long as the keys didn't change this is not nessasery at all.
    By adding the following condition before the flagging the boost is tremendous and shouldn't cause side effects.
    Code:
    if(oldKey == newKey)
        this.dirtyIndices = true;
    
    I ended up with the following Implementation:
    Code:
    replace: function(oldKey, item) {
           var me = this,
               sorted = me.sorted,
               filtered = me.filtered,
               filterable = me.mixins.filterable,
               items = me.items,
               keys = me.keys,
               all = me.all,
               map = me.map,
               returnItem = null,
               oldItemsLn = items.length,
               oldItem, index, newKey;
    
    
    
    
           if (arguments.length == 1) {
               item = oldKey;
               oldKey = newKey = me.getKey(item);
           } else {
               newKey = me.getKey(item);
           }
    
    
    
    
           oldItem = map[oldKey];
           if (typeof oldKey == 'undefined' || oldKey === null || typeof oldItem == 'undefined') {
                return me.add(newKey, item);
           }
    
    
    
    
           me.map[newKey] = item;
           if (newKey !== oldKey) {
               delete me.map[oldKey];
           }
    
    
    
    
           if (sorted && me.getAutoSort()) {
               Ext.Array.remove(items, oldItem);
               Ext.Array.remove(keys, oldKey);
               Ext.Array.remove(all, oldItem);
    
    
    
    
               all.push(item);
    
    
    
    
               me.dirtyIndices = true;
    
    
    
    
               if (filtered && me.getAutoFilter()) {
    // If the item is now filtered we check if it was not filtered
    // before. If that is the case then we subtract from the length
                   if (filterable.isFiltered.call(me, item)) {
                       if (oldItemsLn !== items.length) {
                           me.length--;
                       }
                       return null;
                   }
    // If the item was filtered, but now it is not anymore then we
    // add to the length
                   else if (oldItemsLn === items.length) {
                       me.length++;
                       returnItem = item;
                   }
               }
    
    
    
    
               index = this.findInsertionIndex(items, item);
    
    
    
    
               Ext.Array.splice(keys, index, 0, newKey);
               Ext.Array.splice(items, index, 0, item);
           } else {
               if (filtered) {
                   if (me.getAutoFilter() && filterable.isFiltered.call(me, item)) {
                       if (me.indexOf(oldItem) !== -1) {
                           Ext.Array.remove(items, oldItem);
                           Ext.Array.remove(keys, oldKey);
                           me.length--;
                           me.dirtyIndices = true;
                       }
                       return null;
                   }
                   else if (me.indexOf(oldItem) === -1) {
                       items.push(item);
                       keys.push(newKey);
                       me.indices[newKey] = me.length;
                       me.length++;
                       return item;
                   }
               }
    
    
    
    
               index = me.indexOf(oldItem);
    
    
    
    
               keys[index] = newKey;
               items[index] = item;
               if (newKey !== oldKey) {
                   this.dirtyIndices = true;
               }
           }
    
    
    
    
           return returnItem;
       },
    
    For syncing a Store into a WebSql - Database with a big amount of phantom Records (> 10000) the impact is kind of big... When running the implementation of 2.1 it tooks Minutes on my developing maschine:

    before.jpg

    with the optimized version only seconds...

    after.jpg

    It would be nice to see some upcomming optimation in the new Release!

    Regards Martin

  2. #2
    Sencha - Senior Forum Manager mitchellsimoens's Avatar
    Join Date
    Mar 2007
    Location
    Gainesville, FL
    Posts
    37,647
    Vote Rating
    899
    mitchellsimoens has a reputation beyond repute mitchellsimoens has a reputation beyond repute mitchellsimoens has a reputation beyond repute mitchellsimoens has a reputation beyond repute mitchellsimoens has a reputation beyond repute mitchellsimoens has a reputation beyond repute mitchellsimoens has a reputation beyond repute mitchellsimoens has a reputation beyond repute mitchellsimoens has a reputation beyond repute mitchellsimoens has a reputation beyond repute mitchellsimoens has a reputation beyond repute

      1  

    Default


    We will take a look at this, thanks for digging into this!
    Mitchell Simoens @SenchaMitch
    Sencha Inc, Senior Forum Manager
    ________________
    Check out my GitHub, lots of nice things for Ext JS 4 and Sencha Touch 2
    https://github.com/mitchellsimoens

    Think my support is good? Get more personalized support via a support subscription. https://www.sencha.com/store/

    Need more help with your app? Hire Sencha Services services@sencha.com

    Want to learn Sencha Touch 2? Check out Sencha Touch in Action that is in print!

    When posting code, please use BBCode's CODE tags.

  3. #3
    Sencha User
    Join Date
    Mar 2007
    Location
    Haarlem, Netherlands
    Posts
    1,243
    Vote Rating
    10
    TommyMaintz will become famous soon enough TommyMaintz will become famous soon enough

      1  

    Default


    Hi Martin,

    I have applied your optimizations to our codebase for the next release. Thanks for digging into this! I think there might be even more room for optimization in regards of just updating the indices instead of setting the flag in certain places, but that change is more drastic and one I dont want to make right now.

    Best,
    Tommy

Thread Participants: 2