High Performance Large List component - UxBufferedList

[MahlerFreak]

Benjamin,

The latest bug is fixed. The file is attached in my original post - I discovered that if you edit posts under IE, of all things, you get the advanced editor and can change the attached files. All others reading this thread should get the fix from the original post.

Edit: My bad (at least partly). It turns out that if you click "edit post", then Go Advanced, you don't get the advanced editor. But if you double-click on edit post, it goes straight to advanced editor on all platforms.

Scott

[chrisfarms]

This is excellent, thanks.

I was disappointed to find that this method wasn't used in the standard List control, I hope this makes it into core, I'm sure they will need it if they intend on making some kind of grid control.

I did struggle a little at first as I did not have the "fullscreen: true" set, and have no idea why that was required .... guess I should learn about layouts next

Have you thought about using github/bitbucket as some others do to for their extensions, could make updates a bit easier than replacing a zip. Thanks again anyway sure this will come in handy.

[MahlerFreak]

Have you thought about using github/bitbucket as some others do to for their extensions, could make updates a bit easier than replacing a zip. Thanks again anyway sure this will come in handy.

Yes, I should set something up. It's been 10 years since I was an active coder, and the whole world of source code control and collaboration is radically different now

[Shale]

Scott,
This is great stuff! Thanks very much for you work. I only just found this and the List performance consolidation thread and it looks like exactly what I need. One thing is the current UxBufList014.zip attached to the first post of this thread doesn't contain albumdata.js. I found a copy in the BuffererdList4.zip from the other thread, but you might want to make a new UxBufList zip with everything in it.

thanks again!

[Shale]

Scott, there's one other thing I wanted to ask. In the other thread you said the following

I'm building a music control application, consisting of a dedicated LAN-only HTTP server, and Sencha-based clients. The nature of this application means that I must deal gracefully with lists of data potentially in the 10,000 to 100,000 item range, with a high degree of interactivity.

That's the exact use case I'm interested in. Does BufferedList meet those requirements for you?

I'm still getting up to speed with sencha touch, but it seems like Store doesn't natively support incrementally loading rows as needed and forgetting old ones. Also, I've been toying with different types of lists, both very large lists and dynamically changing lists, which would be difficult for my HTTP server to expose as a list of known size indexed 0 to N. Instead I'd like to support both lists that have no known total length and ordered lists that use javascript objects as keys instead of integers.

I want to be clear: this isn't a feature request aimed at you! Your BufferList is an eye-opening example of extending the library's List widget for me, and I'd really appreciate your thoughts on what I said above. Also, has anyone else tried anything to support different types of lists or anything similar?

thanks

[gcallaghan]

First off, this is soooo helpful. It saved me an unbelievable amount of time. Thank you so much for sharing.

for my particular need I have a list with groups but no indexBar. I was able to use this with headers and without an index bar by making the following changes.

...
initGroupIndexMap: function() {
        this.groupIndexMap = {};
        var i, 
            key,
            firstKey,
            store = this.store, 
            recmap = {},
            groupMap = this.groupIndexMap,
            prevGroup = '',
            sc = store.getCount();

        // build temporary map of group string to store index from store records
        for ( i = 0; i < sc; i++ ) {
            key = store.getGroupString(store.getAt(i)).toLowerCase();
            if ( recmap[key] === undefined ) {
                recmap[key] = { index: i, closest: key, prev: prevGroup } ;
                prevGroup = key;
            }
            if ( !firstKey ) {
                firstKey = key;
            }
        }

        // now make sure our saved map has entries for every index string
        // in our index bar.
        if (!!this.indexBar) {
            var barStore = this.indexBar.store, bc = barStore.getCount(), grpid, idx = 0, recobj;
            prevGroup = '', key = '';
            for (i = 0; i < bc; i++) {
                grpid = barStore.getAt(i).get('key').toLowerCase();
                recobj = recmap[grpid];
                if (recobj) {
                    idx = recobj.index;
                    key = recobj.closest;
                    prevGroup = recobj.prev;
                }
                else if (!key) {
                    key = firstKey;
                }
                groupMap[grpid] = {
                    index: idx,
                    closest: key,
                    prev: prevGroup
                };
            }
        }
        else {
            this.groupIndexMap = recmap;
        }
        
    },
...

[gcallaghan]

I also had a case wher I had special characters in my heading, namely &.

This caused issues with the groupIndexMap. I was able to workaround this with the following changes.

// @private - create a map of grouping strings to start index of the groups
    initGroupIndexMap: function() {
        this.groupIndexMap = {};
        var i, 
            key,
            firstKey,
            store = this.store, 
            recmap = {},
            groupMap = this.groupIndexMap,
            prevGroup = '',
            sc = store.getCount();

        // build temporary map of group string to store index from store records
        for ( i = 0; i < sc; i++ ) {
            key = escape(store.getGroupString(store.getAt(i)).toLowerCase());
            if ( recmap[key] === undefined ) {
                recmap[key] = { index: i, closest: key, prev: prevGroup } ;
                prevGroup = key;
            }
            if ( !firstKey ) {
                firstKey = key;
            }
        }

        // now make sure our saved map has entries for every index string
        // in our index bar.
        if (!!this.indexBar) {
            var barStore = this.indexBar.store, bc = barStore.getCount(), grpid, idx = 0, recobj;
            prevGroup = '', key = '';
            for (i = 0; i < bc; i++) {
                grpid = barStore.getAt(i).get('key').toLowerCase();
                recobj = recmap[grpid];
                if (recobj) {
                    idx = recobj.index;
                    key = recobj.closest;
                    prevGroup = recobj.prev;
                }
                else if (!key) {
                    key = firstKey;
                }
                groupMap[grpid] = {
                    index: idx,
                    closest: key,
                    prev: prevGroup
                };
            }
        }
        else {
            this.groupIndexMap = recmap;
        }
        
    },
  
    // @private - get an encoded version of the string for use as a key in the hash 
   getKeyFromId: function (groupId){
        return escape(groupId.toLowerCase());
    },
     // @private - get the group object corresponding to the given id
    getGroupObj:function (groupId){
        return this.groupIndexMap[this.getKeyFromId(groupId)];
    },
    
    // @private - get starting index of a group by group string
    groupStartIndex: function(groupId) {
        return this.getGroupObj(groupId).index;
    },
    
    
    // @private - get group preceding the one in groupId
    getPreviousGroup: function(groupId) {
        
        return this.getGroupObj(groupId).prev;
    },
    
    // @private - get closest non-empty group to specified groupId from indexBar
    getClosestGroupId: function(groupId) {
        return this.getGroupObj(groupId).closest;
    },

[MahlerFreak]

Scott,
This is great stuff! Thanks very much for you work. I only just found this and the List performance consolidation thread and it looks like exactly what I need. One thing is the current UxBufList014.zip attached to the first post of this thread doesn't contain albumdata.js. I found a copy in the BuffererdList4.zip from the other thread, but you might want to make a new UxBufList zip with everything in it.

thanks again!

Thanks for catching that! I've updated the zip file in the first thread post.

That's the exact use case I'm interested in. Does BufferedList meet those requirements for you?

Not entirely as it stands, because as you note you don't really want to download all 100K tracks in one lump. I'm experimenting on my end with some modifications/additions to dynamically load/unload data as part of the dynamic rendering process. These modifications do require that the server be able to report what the total number of items in a given list is, and to be able to return subsets of that data based on indexes (see the documentation on AjaxProxy for how this would be used). This also implies that I do not do any searching/sorting of the JSON data on the client - it's all done by the server, and the results of said searches/sorts are cached in the server against a unique id supplied by the client. So when the user performs some action which changes the current contents/order of a list, I first signal the server to perform the search/sort and cache the results, then make repeated queries against that cache to return subsets of the data as needed for rendering. Does that make sense? Because I've written my own server in C++, I can have it do exactly what is needed to support the client - you may not have that luxury.

Can you tell me anything about what you're working on? You can PM me if you want ...

[MahlerFreak]

First off, this is soooo helpful. It saved me an unbelievable amount of time. Thank you so much for sharing.

for my particular need I have a list with groups but no indexBar. I was able to use this with headers and without an index bar by making the following changes.

Thank you gcallahan for these enhancements, and the ones below. I think someone else earlier in the thread asked for exactly what you needed: groups without index bar. I'll put these additions in the next "release".

Glad you found this useful.

Scott

[norabora]

Thank you MichaelFrank for this awesome BufferedList.
and also thank you, gcallaghan.
your fix makes getGroupString possible to return more than a char.