1. #1
    Sencha User
    Join Date
    Jun 2008
    Posts
    5
    Vote Rating
    0
    boedah is on a distinguished road

      0  

    Default Opinions on performance using large data sets

    Opinions on performance using large data sets


    Hi all,

    I am currently evaluating different approaches to build a mobile app.

    We have a data set with ~ 20,000 records and need to
    • filter a list of these records by user input based on one (maybe more than one) property (mabe even substring match)
    • filter the list by an exact property and then filter again by user input
    • group the list by a property
    • be able to place calls / send e-mails out of the app (e.g. if some records contain those information)
    • package the whole thing up as an app (should work offline, the records would be included in the app, no network connection should be required)
    Currently only for iOS, but later maybe Android / Blackberry OS.

    My questions would be:
    1. should I even start thinking about building that as a hybrid app using JavaScript?
    2. if so, would you recommend Sencha Touch for that purpose (I have experience in ExtJS 2 / 4, so this was my first choice, but I read about certain bugs in combination with phonegap for example)?
    3. if not, what else?
    4. if performance turns out to be a problem, is it possible to store the data in a correctly indexed SQLite db and use that as a data source for Javascript?
    5. If you would recommend a native app, what do you think about the learning curve (esp. for iOS)?
    I really appreciate every thought!

  2. #2
    Sencha User
    Join Date
    Dec 2011
    Posts
    6
    Vote Rating
    0
    gblotter is on a distinguished road

      0  

    Default


    @boedah -

    I'm by no means an expert, and anyone please feel free to correct me if I'm wrong... I'd love to learn something new! I just finished using Sencha Touch for a school project where we had to connect up to an online store, download roughly 600 total records, populating various stores, and then move all those records to offline stores, which Sencha Touch calls "local storage, which is the HTML5 local storage utilized within the browser. Those records populated several lists, located on different tabs on a tab panel, with roughly 100-300 records apiece. I did notice that there was a significant delay in the views loading even after that transfer from online to offline had taken place.

    As an example, I noticed that the views of my lists were taking much longer than usual to load (there was a noticeable delay, and you could actually see the list/view render starting at the top working its way down), and it only had about 200 records. To try and fix that, I looked into doing paging with Sencha Touch 2... only loading 10+/- records at a time, with a 'load more' button at the bottom of the list, but apparently that only works with online stores, so that wasn't an option for me... I ended up just having to live with the delay in the views for the time being. I did notice, however, that it seemed that once the views had been loaded that first time, they didn't take as long when switching to different tabs on my main view, as long as they hadn't been destroyed and recreated since visiting the list last.

    Another thing to worry about is that I couldn't get offline associations in the models to work. And from what I read, they weren't meant to work with Sencha Touch 2. I had to manually create the associations, and at first I was doing it on the device but it ate up too much memory, so I had to pass in the JSON objects completely nested, etc. You can implement phone-gap (cordova)'s API to use a native sqlite db, but I don't have any experience with that to contrast with.

    I guess my overal point is... it can be done, but its sometimes not the easiest of the most intuitive. I'm sure there are solutions, work arounds, and tweaks to get the performance up to par, but for me, the biggest motivator to use this framework was I already knew html & javascript, and therefore there wasn't too big of a learning curve to pick up Sencha Touch.

    I hope this helps, I apologize if it doesn't make any sense, and good luck!

  3. #3
    Sencha User
    Join Date
    Jun 2008
    Posts
    5
    Vote Rating
    0
    boedah is on a distinguished road

      0  

    Default Re. Performance on large data sets

    Re. Performance on large data sets


    Hi gblotter,

    thanks for your comments.

    I think I will not need localStorage, since the data is read only, so for me it makes no sense to let it decode the whole thing first.
    The data would be a fixed Array.

    I think I will give it a shot, and see if the performance is acceptable.

    Greetings!

  4. #4
    Sencha User
    Join Date
    Oct 2011
    Location
    Germany
    Posts
    146
    Vote Rating
    10
    Möhre will become famous soon enough

      0  

    Default


    20.000 is way to much if you use localstorage or sencha touch objects.

    Pure (basic) javascript with pure html might work on a tiny smartphone.

  5. #5
    Ext JS Premium Member
    Join Date
    Aug 2007
    Location
    Antwerp, Belgium
    Posts
    555
    Vote Rating
    27
    joeri has a spectacular aura about joeri has a spectacular aura about joeri has a spectacular aura about

      0  

    Default


    20.000 records on one html page without paging can't be done on mobile (not with good performance), regardless of technology.

    You need to have some sort of paging / filtering implementation. ST can be responsible for the view layer without issues if you restrict yourself to showing 100 to 200 records at once. Your issue is likely going to be with how to load the records into a store.

    I'm pretty sure it can be done, but it may need some custom extensions to the data classes (e.g. a custom proxy to load pages of data from a separate non-ST data structure).

  6. #6
    Sencha User
    Join Date
    Jun 2008
    Posts
    5
    Vote Rating
    0
    boedah is on a distinguished road

      0  

    Default


    Quote Originally Posted by joeri View Post
    20.000 records on one html page without paging can't be done on mobile (not with good performance), regardless of technology.
    The thing is, the app we wanna port on iOS currently runs as Java MIDlet on a Blackberry and the filtering is instantaneous.
    The whole 20,000 are displayed (which does not make sense in my opinion, especially as you cannot flick on a BB

    Quote Originally Posted by joeri View Post
    You need to have some sort of paging / filtering implementation. ST can be responsible for the view layer without issues if you restrict yourself to showing 100 to 200 records at once. Your issue is likely going to be with how to load the records into a store.

    I'm pretty sure it can be done, but it may need some custom extensions to the data classes (e.g. a custom proxy to load pages of data from a separate non-ST data structure).
    Yes, I also thought of the whole store mechanism being a problem.
    Simply filtering the list via Javascript would maybe work.

    Quote Originally Posted by Möhre View Post
    20.000 is way to much if you use localstorage or sencha touch objects.

    Pure (basic) javascript with pure html might work on a tiny smartphone.
    See above, but what do you mean by "tiny smartphone"?
    The app would solely be targeted at iPhones.

    Cya!
    Last edited by boedah; 25 Apr 2012 at 4:32 AM. Reason: typo

  7. #7
    Sencha User
    Join Date
    Jun 2008
    Posts
    5
    Vote Rating
    0
    boedah is on a distinguished road

      0  

    Default Quick update

    Quick update


    Just wanted to give a quick update on this topic.

    These are my findings:
    • 20000 records in a list are ok on the iOS-Simulator
    • 20000 records in a list are ok on the iPhone
    • the desktop Chrome / Safari stalled for > 10 seconds upon loading the page - starting at 5000 records; scrolling performance was also worse (which I find really strange, since the desktop browsers should get more performance than the browser in the simulator...)
    • filtering 20000 records did require around 3 seconds on the device (clearing the filter a little more)
    The whole thing was a POC for us and regarding the whole experience, it was really nice and easy to work with Sencha Touch.

    What bugged me were the Sencha Architect / Sencha SDK Tools / Sencha Touch versions where in such an inconsistent state, along with the documentation. We also did not get native packaging to work.

    Just read about Sencha Touch 2.1, where they addressed the List performance (I can not believe each list item was rendered before...), maybe I will give it a spin, especially as they are releasing a new Sencha command, maybe the whole stuff works together better then.

    Cya so long!

  8. #8
    Touch Premium Member suzuki1100nz's Avatar
    Join Date
    Apr 2011
    Location
    New Zealand
    Posts
    447
    Vote Rating
    62
    suzuki1100nz is a jewel in the rough suzuki1100nz is a jewel in the rough suzuki1100nz is a jewel in the rough suzuki1100nz is a jewel in the rough

      0  

    Default


    Hi,

    Have used the new Infinite list feature and it works a treat - so Id say yes go for it.
    We use the auto paging feature with list and keep the page size at around 100.
    Put in and override so the page loading occurs before the user hits the bottom of the scroll.
    Had to do this on scroll end rather than scroll as we didnt want to impact the scrolling, so the user can still hit the bottom with a big swipe.

    With 20 000 records (we have more - General ledger accounts) you need to implement sorting server side and force a page load after the sort.
    Having 20 000 object in the datastore may impact memory

  9. #9
    Sencha User
    Join Date
    Jun 2008
    Posts
    5
    Vote Rating
    0
    boedah is on a distinguished road

      0  

    Default


    Hi suzuki1100nz,

    thanks for your post.

    Quote Originally Posted by suzuki1100nz View Post
    Have used the new Infinite list feature and it works a treat - so Id say yes go for it.
    We will give it a try!

    Quote Originally Posted by suzuki1100nz View Post
    With 20 000 records (we have more - General ledger accounts) you need to implement sorting server side and force a page load after the sort.
    Having 20 000 object in the datastore may impact memory
    We have not checked the memory consumption, but it works ok on iOS.
    As the data is readOnly, and the whole dataset would be replaced for an update, the data currently simply is an ArrayStore with inline data...