This is a patch proposal for the data package of Sencha Platform.

The patch extends the Data package to be able to output a Model and its associations as denormalized data blobs, purely based on configuration of the Model.

I will try to present the general use case, and at some points exemplify with an implementation of the Portable Contacts (PoCo) draft specification found at: http://portablecontacts.net/draft-spec.html

Premise:
Denormalized databases, or document databases, have gained a lot of momentum in the last years. There are a lot of use cases where denormalized data has been shown to be a more effective way of storing and scaling large datasets. Thus it is an interesting use case for the Sencha Platform's Data package to implement some way of working with denormalized datasets.

The current Data package currently handles eager loading of related data through a nested data structure quite well. The model still assumes complete normalization on the backend though. Here are some of the things that would be nice to have when working with denormalized data:

- A way to define when an association should be treated as denormalized
I will refer to these relations as 'inner'.

- Optional properties
Most document databases allow for schemaless datasets, which don't require all properties to be set. For example the PoCo specification includes a lot of optional properties. The Sencha Model should be able to represent this by not serializing empty properties when saving data.

- Optional validation
If a property can be optional, then the model itself should be able to validate even when the optional property is not set.

- Deep validation
A Model that has inner relations should do a deep validation of all its inner relations, since upon serialization these are conceptually a part of the Model itself.

- Deep serialization
When a Model is serialized, all its inner relations should be serialized as well for remote storage.

And specific for ExtJS and Sencha Touch:
- A Record that is marked as dirty should mark its parent relation as dirty as well if it is an inner relation.

- Saving a modified record should delegate to the records parent record if it is an inner relation.

Architecture:
I have had to make some choices along the way to be able to differentiate these different types of properties and relations. I have checked in Sencha Touch 1.0.1a on Github and applied my changes on top of that, available here: https://github.com/One-com/Sencha-To...s/denormalized

I will run through each of the changes and try to explain my reasoning, and give implementation examples:

https://github.com/One-com/Sencha-Touch/commit/c64d
"Added innerOf property to inner associated models on instantiation"

This commit adds a tracking of whether a model is an inner association of another model by adding a 'innerOf' property on the instantiated Record.
A configuration property has been added, so you can set this up by adding an 'inner'-property on the configuration for the hasMany association. Example:
Code:
{ inner: true, model: 'Email', name: 'emails' }

https://github.com/One-com/Sencha-Touch/commit/41a7
"Added deep validation of inner associations"

This commit makes a Record accumulate errors in validation of inner associations.
It uses the 'inner'-property of the association configuration to determine whether this is relevant.

https://github.com/One-com/Sencha-Touch/commit/6714
"Added optional properties"

Sometimes you will have optional properties in schemaless databases, and you don't really want the ExtJS autoconversion to run with its default settings, converting absent properties to empty string and similar. This commit adds conversion functions for each of ExtJS datatypes, which allow a property to be optional.
I have chosen to use null as the internal value for absent properties, which will be used as markers to remove when serializing.
This is an example of an optional property configuration:
Code:
{ name: 'nickname', type: 'string', optional: true, defaultValue: null }
Currently I have to add a defaultValue of null for this to work. This shouldn't be needed for optional properties in the long run.


https://github.com/One-com/Sencha-Touch/commit/8667
"Added optional validations that accept null as a valid value"

If the premise of an optional property is accepted, this commit is obvious.
An optional property with the value of null needs to be valid, hence the validations need to be changed so they return true for optional properties with the value null.


https://github.com/One-com/Sencha-Touch/commit/87b5
"Added deep serialization of inner associations and removal of optional properties set to null."

Since the concept of denormalized data is that the parent node is the only entry in the database, we need to do a deep serialization when saving to the database.
Instead of just returning Record.data, Writer.getRecordData() now does this:
- Iterate through all properties and add them to the return object unless they are optional and the value is null.
- Iterate through all associated records that are marked as inner associations, serialize them and strip all foreign key properties.


https://github.com/One-com/Sencha-Touch/commit/1a45
"Added state update for inner records, so the parent record will be marked as dirty if an inner record is marked dirty. Now uses built-in setDirty method."

This is more of a convenience change for ExtJS. It makes sense that any Record that is marked as dirty should mark its parent as dirty, so that the outer store, which synchronizes with the backend, can see when any data has been updated.
Implementing this in application logic wouldn't make sense because you would have to do it over and over again.


https://github.com/One-com/Sencha-Touch/commit/07b3
"Made all inner relations delegate saving of records to their containing records"

Another convenience change for ExtJS. When calling the save method on a Record that is an inner association, it is assumed that the Model does not have a proxy configured, and hence will not be able to store the changes on the backend by itself.
Instead the changed inner Record should call the save method on the Record it is inside, which will eventually lead to a serialization of the complete tree that needs to be synced with the server.


I expect that I might have made some design choices that Sencha will not agree on. I am prepared to put in the needed work to fix these if the end result will be Sencha accepting the patch.
I haven't put in any work to port the patch to ExtJS4, since Ed Spencer mentioned in his talk about the data package in Ext4 that it will run on Sencha Platform as well. As I don't see this being the case yet, I will wait for a release that does, and put in the work then.

I hope that Sencha will take this patch into serious consideration.