Opinions on performance using large data sets
I am currently evaluating different approaches to build a mobile app.
We have a data set with ~ 20,000 records and need to
Currently only for iOS, but later maybe Android / Blackberry OS.
- filter a list of these records by user input based on one (maybe more than one) property (mabe even substring match)
- filter the list by an exact property and then filter again by user input
- group the list by a property
- be able to place calls / send e-mails out of the app (e.g. if some records contain those information)
- package the whole thing up as an app (should work offline, the records would be included in the app, no network connection should be required)
My questions would be:
I really appreciate every thought!
- if so, would you recommend Sencha Touch for that purpose (I have experience in ExtJS 2 / 4, so this was my first choice, but I read about certain bugs in combination with phonegap for example)?
- if not, what else?
- If you would recommend a native app, what do you think about the learning curve (esp. for iOS)?
I'm by no means an expert, and anyone please feel free to correct me if I'm wrong... I'd love to learn something new! I just finished using Sencha Touch for a school project where we had to connect up to an online store, download roughly 600 total records, populating various stores, and then move all those records to offline stores, which Sencha Touch calls "local storage, which is the HTML5 local storage utilized within the browser. Those records populated several lists, located on different tabs on a tab panel, with roughly 100-300 records apiece. I did notice that there was a significant delay in the views loading even after that transfer from online to offline had taken place.
As an example, I noticed that the views of my lists were taking much longer than usual to load (there was a noticeable delay, and you could actually see the list/view render starting at the top working its way down), and it only had about 200 records. To try and fix that, I looked into doing paging with Sencha Touch 2... only loading 10+/- records at a time, with a 'load more' button at the bottom of the list, but apparently that only works with online stores, so that wasn't an option for me... I ended up just having to live with the delay in the views for the time being. I did notice, however, that it seemed that once the views had been loaded that first time, they didn't take as long when switching to different tabs on my main view, as long as they hadn't been destroyed and recreated since visiting the list last.
Another thing to worry about is that I couldn't get offline associations in the models to work. And from what I read, they weren't meant to work with Sencha Touch 2. I had to manually create the associations, and at first I was doing it on the device but it ate up too much memory, so I had to pass in the JSON objects completely nested, etc. You can implement phone-gap (cordova)'s API to use a native sqlite db, but I don't have any experience with that to contrast with.
I hope this helps, I apologize if it doesn't make any sense, and good luck!
Re. Performance on large data sets
thanks for your comments.
I think I will not need localStorage, since the data is read only, so for me it makes no sense to let it decode the whole thing first.
The data would be a fixed Array.
I think I will give it a shot, and see if the performance is acceptable.
20.000 is way to much if you use localstorage or sencha touch objects.
20.000 records on one html page without paging can't be done on mobile (not with good performance), regardless of technology.
You need to have some sort of paging / filtering implementation. ST can be responsible for the view layer without issues if you restrict yourself to showing 100 to 200 records at once. Your issue is likely going to be with how to load the records into a store.
I'm pretty sure it can be done, but it may need some custom extensions to the data classes (e.g. a custom proxy to load pages of data from a separate non-ST data structure).
The thing is, the app we wanna port on iOS currently runs as Java MIDlet on a Blackberry and the filtering is instantaneous.
Originally Posted by joeri
The whole 20,000 are displayed (which does not make sense in my opinion, especially as you cannot flick on a BB
Yes, I also thought of the whole store mechanism being a problem.
Originally Posted by joeri
See above, but what do you mean by "tiny smartphone"?
Originally Posted by Möhre
The app would solely be targeted at iPhones.
Last edited by boedah; 25 Apr 2012 at 4:32 AM.
Just wanted to give a quick update on this topic.
These are my findings:
The whole thing was a POC for us and regarding the whole experience, it was really nice and easy to work with Sencha Touch.
- 20000 records in a list are ok on the iOS-Simulator
- 20000 records in a list are ok on the iPhone
- the desktop Chrome / Safari stalled for > 10 seconds upon loading the page - starting at 5000 records; scrolling performance was also worse (which I find really strange, since the desktop browsers should get more performance than the browser in the simulator...)
- filtering 20000 records did require around 3 seconds on the device (clearing the filter a little more)
What bugged me were the Sencha Architect / Sencha SDK Tools / Sencha Touch versions where in such an inconsistent state, along with the documentation. We also did not get native packaging to work.
Just read about Sencha Touch 2.1, where they addressed the List performance (I can not believe each list item was rendered before...), maybe I will give it a spin, especially as they are releasing a new Sencha command, maybe the whole stuff works together better then.
Cya so long!
Have used the new Infinite list feature and it works a treat - so Id say yes go for it.
We use the auto paging feature with list and keep the page size at around 100.
Put in and override so the page loading occurs before the user hits the bottom of the scroll.
Had to do this on scroll end rather than scroll as we didnt want to impact the scrolling, so the user can still hit the bottom with a big swipe.
With 20 000 records (we have more - General ledger accounts) you need to implement sorting server side and force a page load after the sort.
Having 20 000 object in the datastore may impact memory
thanks for your post.
We will give it a try!
Originally Posted by suzuki1100nz
We have not checked the memory consumption, but it works ok on iOS.
Originally Posted by suzuki1100nz
As the data is readOnly, and the whole dataset would be replaced for an update, the data currently simply is an ArrayStore with inline data...
Tags for this Thread