The architecture of Store - open thoughts

[deitch]

In line with the persisting Ext.data.Store that I listed in the User Extensions forum, I have been thinking a lot about the Store in general. Basically, a Store right now is a subset representation. It gets its raw data from a proxy. The proxy may or may not give all of the data available on the server side, but that is irrelevant. What is relevant is that based on the root (JsonReader) or record (XmlReader), the Store itself actually only contains a subset of the data. This is sufficient for a read-only Store, but one that writes has several problems.

How does the updated server end know which section to update? E.g. if a Json object has two elements, a and b, and we use a as our root, then all our updates belong to a, but we no longer know that and transmit it. Fine, this is easy to solve, since we can send it as a parameter back on update(). The reader itself keeps it in this.meta.root anyways.
What do we do with holistic updates. Specifically, the above works if the server can handle sub-element changes, i.e. just the a section. But what if the server is all-or-none? Send me an entire replacement data set or send me nothing? This is particularly relevant with encrypted data (e.g. host-proof hosting). In that case, we need to keep the entire store of data, not just the a sub-element.


So here is my thought. Right now, Ext.data.Store represents just the sub-element that fits with the Reader. It would seem better to have some Ext.data.WholeStore (OK, awful nomenclature) that represents *all* of the data received by the proxy. When we want to use a reader to get some sub-element and have a store to work with, e.g. with a grid, we call Ext.data.WholeStore.getStore({reader: someReader});. Any changes to the "sub-store" we just got are pushed up to the WholeStore when we commitChanges. The WholeStore, on the other hand, handles all server-side interaction. Since it kept the entirety of data received, it knows exactly what subset each sub-store works with, it can transmit contextualized journal entries or the whole data set to the server, etc.

Any thoughts on this?
Avi

[JasonMichael]

Well, I'm not sure if what I'm about the say is in concurrence with the above, but I think it would be nice to be able to access all records received from a source, instead of only being able to assign one 'root' from a JSON formatted data source. I've been looking at the source code and apparently there are currently no methods that allow for this type of flexibility - to easily change the current 'root' of the cached JsonData and reference it.

[deitch]

I think that may be the other side of the coin. Basically, right now a Store is a read-only representation of a well-defined subset of data. In XmlReader, that subset is defined by the record; in JsonReader, the root.

Put in RDBMS terms, a Store is a read-only representation of a table. You are asking if it can be a representation of a database. I believe that the direction I proposed above moves that way, but not entirely. I think (but the Ext guys will have to correct me here) that the reason it developed this way was because Store was originally viewed as a way to store row-type data that would underlie combo boxes and grids. In other words, it is more like a view and less like a SQL store that can have CRUD operations on it.

The current paradigm makes it easy to work with, but hard to update. I hope to add something to change that.

[deitch]

The more I think about this, the more I realize that ext misses ORM. Thus, I think the superstore/masterstore model may be incorrect, and an ORM manager model may make more sense. You can then completely centralize connections - Ajax, Gears, whatever - into the manager.

The store remains what it was: model for grid, etc. that is loaded with a series of arbitrary objects. When you want to load something, you get it from the manager and, if desired, add it to Ext.data.Store; when you want to save an object, you save it via the manager. If you want to delineate a transaction, you do it through the manager. The manager looks at the records themselves, i.e. the objects.



var Person = Ext.data.Record.create([{...},{...}]);

// note that the ORMManager below does not have to go over the net
// the proxy could be to a Google Gears store, in memory store, anything
var mgr = new Ext.data.ORMManager({proxy: myProxy});

// get an object and modify it
var person = mgr.getById(Person,123);
person.name = "John Smith";
mgr.save(person);

// query for objects
var person2 = mgr.createQuery("from Person where Person.name = 'John Smith'");

// create an object
var person = new Person({...});
person.set("name","John Smith");
mgr.save(person);


So, are we talking about Hibernate for JS? Perhaps. Still, I feel that the basic read-only model for data, with no ORM, is insufficient to build full apps.

Thoughts?

[Collin Miller]

Here's some perspective from a RubyOnRails developer. (Ext makes Prototype and Scriptaculous feel wimpy.)

Rails 2.0 will ship with an ActiveResource class. The pattern is easily ported to client-side. Where ActiveRecord directly queries the database ActiveResource makes HTTP calls to highly conventionalized Rails apps.

I've ported the pattern to two EcmaScript platforms so far: HD-DVD and web browser.

Using it looks like this:



Resource("thing", "//my-domain/");
Thing.find("collins_thing", {
success: function(theThing) { /* ... */ }
});



Thing.on('destroy', function(destroyedThing) { /* ... */};
aThing.destroy()

And so on and so forth mirroring ActiveRecord's life cycle methods including build and save.

With Ext in mind I've started adding methods for it to act as a data store. It needs a few more long nights before it supports validations and such.

It has struck me that using the ActiveResource pattern to encapsulate storage in gears or even "comet"-style pub/sub would be fun too.

[deitch]

Collin,

Let me see if I get it. My understanding of Rails' ActiveRecord is that each CRUD action that you take is directly reflected in the underlying database. The ActiveResource does the same thing, but does it on a RESTful basis using HTTP instead of SQL to an RDBMS.

ext appears to me to have a different model. Ext.data.Store is a read-only representation of an Array of Ext.data.Record with some manipulation (well, read-write, except that the writes are transient). Store makes a lot of sense; it is the model part behind a grid or similar object, so you end up with some elements of MVC.

So.. are you saying that you end up with a hidden ORM? Rather than using an ORM manager to manage the objects (find, remove, save, etc.) you have the objects self-manage when you take appropriate actions? How does that tie into the store, which normally gets its objects from a proxy processed through a reader?

Also, how does all this play into TrimPath Junction? Its model seems incredibly close to RoR (and unabashedly says so).

[Collin Miller]

I reference Ruby on Rails a lot here, but there is no reason Rails has to be the back end for this.

Some of my code formatted poorly in this post, well, better to get the ball rolling than get it rolling straight, eh?

IT sound like we're on a similar page.

I'll start with the easy bit. I've no idea how this relates to TrimPath Junction, I've read over that lightly, but would rate myself a weak one out of ten on it.

I'll start with the simple REST transport I wrote. This is the Ext version, but I've got a plain old javascript version and an HD-DVD version too.



Ext.ux.Rest = {
transport: {
doConnection: function(path, options, callbackWrapper, content) {
console.log(path)
Ext.Ajax.request({
url: path,
params: "",
method: options.method || "GET",
xmlData: content,
/* I'd like to see this callback handling tie more strongly into
specific HTTP status codes.
success: function(response, params) {
if(callbackWrapper) {
callbackWrapper(response, options.success)
}
else {
options.success(response);
}
},
failure: function(response, params) {
console.error("XMLHTTPRequest failure");
console.log("Request: ", params);
console.log("Response: ", response);
}
});
},

/*
The different HTTP methods. Most of these are straightforward
with an understanding of HTTP. The trickiest bit is the callbackWrapper.
Some requests require a wrapper function to process the response
before passing it along to the user-defined callback.
*/
POST: function(path, connectionOptions, callbackWrapper, content) {
connectionOptions.method = "POST";
this.doConnection(path, connectionOptions, callbackWrapper, content);
},

PUT: function(path, connectionOptions, callbackWrapper, content) {
connectionOptions.method = "PUT";
this.doConnection(path, connectionOptions, null, content);
},

DELETE: function(path, connectionOptions) {
connectionOptions.method = "DELETE";
this.doConnection(path, connectionOptions);
}
}
};

// GET requests are default.
Ext.ux.Rest.transport.GET = Ext.ux.Rest.transport.doConnection;

The next bit; I'm sure something could be done to minimize the need for these functions, I added them because I was trying to follow the Rails ActiveResource implementation closely.


"whatUpHomeSlice".underscore() >> "what_up_home_slice"
"why-do-YouSuck".camelize() >> "WhyDoYouSuck"
"cat".pluralize() >> "cats"
"octopus".pluralize >> "octopi"
"hey".capitalize() >> "Hey"


Some of these were ripped from other js libs, don't remember which from what just now.

I don't have the time tonight to do a proper documentation/commenting of the internals of the Resource constructor, I'l just go over the important bits and get into how I'm integrating it with Ext.data.Store.

A function to specify resources.

Resource(name, config);

Here are current configuration options:

scope: The object where resources will be placed.
site: The base URL to make REST requests to.
transport: The transport to use. ( see Ext.ux.Rest.transport above )

Once you have specified your resources we get a lot a friendly methods.

if we were to


Resource("Person", {
site: "//my_rails_app.com/",
scope: window.models,
transport: Ext.ux.Rest.transport
});


We would have an object accessible through

models.Person

The main methods of Person we want to concern ourselves with now are:

Person#build(attributes, options)
Person#find(identifier, options)
Person#destroy(identifier, options)

build returns a non-persisted Person resource.

find does an HTTP request to to the site and you must specify callbacks to handle the response. Success returns a Person record.

destroy does an HTTP request to destroy a resource



Once we have a berson record from either build or find we have a host of methods to call upon it. These all have callbacks.

save: makes an HTTP PUT or POST depending on whether the record has aready been saved
updateAttributes: convenience to update attributes and HTTP PUT all in one go
destroy: HTTP DELETE

This object also allows direct access to attributes through:

theResource.attributes.someAttribute

Right now the library only supports XML, I'd like to get plugin support for other wire formats(JSON). This was a limitation of the HD DVD platform which has limitations making JSON not an option.

Well, I'm going to stop talking and show the Resource code. Please bear in mind this code was coded under the gun(just one of a half million things to do under a two week deadline for a demo) and as such has very very sparse comments/ lots of room for refactoring(I hope to get it more modular such that It can adapt to more REST strategies that the Rails way.

Oh crap, I forgot to talk about the work I did adding functions to let this act as a data.Store.

I'll write about that tomorrow morning.



function Resource(name, options) {
if (!options.site) {
throw "Missing option: site";
}

if (!options.transport) {
throw "Missing option: transport";
}

var name = name,
site = options.site,
transport = options.transport,
resource = name, slot;


if (name.constructor !== String) {
name = resource.attributes.name.camelize().capitalize();
}

function Model() {
if (resource.constructor !== String) {
//this.reflections = [];
for (slot in resource.attributes) {
if (resource.attributes[slot].constructor == (Array || Object)){
this[slot]=(resource.attributes[slot]);
//this.reflections[slot] = resource.attributes[slot].getModel();
}
}
}
this.name = name;
this.collectionName = name.pluralize();
this.options = options;
this.cache = new Ext.util.MixedCollection(false, function(resource){
return resource.attributes.id;
});
}

Ext.apply(Model.prototype,{
build: function(attributes, options) {
if (this.fireEvent("beforebuild", attributes)) {
var record = new this.Base(attributes, options, this.name);
this.fireEvent("afterbuild", record);
return record;
}
},

find: function(identifier, options) {
var path = this.pathWithAssumptions(identifier, options);
if (identifier && identifier.constructor != Number) {options = identifier};
if (this.cache.get(identifier)) {
return this.foundRecordResponder(this.cache.get(identifier), options);
}
else {
this.fireEvent("beforefind", identifier, options);
return transport.GET(path, options||{}, this.responseTextInstantiator.createDelegate(this));
}
},

destroy: function (identifier, options) {
var path = this.pathWithAssumptions(identifier, options);
if (identifier && identifier.constructor != Number) {options = identifier};
this.fireEvent("beforedestroy", identifier, options);
return transport.DELETE(path, options||{}, this.destructionWrapper.createDelegate(this));
},

addToCache: function(record) {
this.cache.add(record);
},

removeFromCache: function(record) {
this.cache.removeKey(reccord.attributes.id);
},

responseTextInstantiator: function(connection, callback) {
var attributes, findings, doc = connection.responseXML,
that = this;

records = this.recordFromDocument(doc);
// records = this.recordFromJson(Ext.util.JSON.decode(doc));

if (records.constructor !== Array) {
findings = [records];
}

Ext.each((findings || records),
function(record){
that.addToCache(record);
that.fireEvent("afterfind", record)
});

if (callback) {
callback(records, connection);
}
},

foundRecordResponder: function(record, options) {
if (options && options.success) {
options.success(record);
}
},

destructionWrapper: function(connection, callback) {
// this.removeFromCache();
this.fireEvent("afterdestroy", connection);
callback(connection);
},

// recordFromJson: function(json) {
// var slot;
// for(slot in json) {
//
// }
// },

documentIsCollection: function(doc) {
/*
var isCollection = false;
Ext.each(doc.childNodes,
function(childNode){
if (childNode.childNodes.length > 1) {
isCollection = true;
}
});
return isCollection;
*/
return doc.tagName == doc.tagName.pluralize();
},

// This one is a real WTF? Should probably just go with JSON
// This code makes me cry at night.
// But there are more important things to do.
// And I can refactor on a sunnier day.
documentIsSingular: function(doc) {
var isSingular = false
Ext.each(doc.childNodes,
function(childNode){
childNode = Ext.fly(childNode).clean(true).dom;
if (childNode.childNodes.length !== 1) {
}
else {
if (childNode.childNodes[0].nodeType !== 3) {// Text
isSingular = true;
}
}
});
return isSingular;
},

recordFromDocument: function(doc) {
var clean = new Ext.Element(doc).clean(),
attributes = {}, attribute, type,
attributeName, builder, model, that = this,
cleanChild;

Ext.each(clean.dom.childNodes,
function(childNode){
attribute = null;

if (childNode.attributes && (type = childNode.attributes.item("type"))) {
if (type.nodeValue == "integer") {
attribute = parseInt(childNode.firstChild.textContent, 10);
}
else if (type.nodeValue == "boolean") {
attribute = (childNode.firstChild.textContent == "true" ? true : false);
}
}
else {
if (that.documentIsCollection(childNode)) {
attribute = [];
Ext.each(Ext.fly(childNode).clean(true).dom.childNodes,
function(member){
attribute.push(that.recordFromDocument(member));
});
}
else if(that.documentIsSingular(childNode)) {
attribute = that.recordFromDocument(childNode);
}
else {
attribute = childNode.firstChild.textContent;
}
}

if (!attribute.inspect) {
attributeName = childNode.tagName.replace(/-/g, "_").camelize();
attributes[attributeName] = attribute;
}
else {
attributes = attribute;
}
});

if (clean.dom.tagName) {
if (model = options.scope[clean.dom.tagName.camelize().capitalize()]) {
builder = model.build.createDelegate(model);
return builder(attributes, {newRecord: false});
};
return attributes;
}
else if (attributes[attributeName]) {
return attributes[attributeName];
}
else {
return attributes
}
},

objectToPrefix: function (prefixes) {
var prefix, builtPrefix = "";

prefixes = prefixes || {};

for (prefix in prefixes) {
builtPrefix += "/";
builtPrefix += prefix;
if (prefixes[prefix] !== null) {
builtPrefix += "/" + prefixes[prefix];
}
}
return builtPrefix;
},

objectToQuery: function (query) {
var slot, builtQuery = "?";

query = query || {}

for (slot in query) {
if (query[slot].charAt!=="undefined") {
builtQuery += "&" + slot + "=" + query[slot];
}
else {
builtQuery += this.objectToQuery(query[slot]);
}
}
return builtQuery.length === 1 ? "" : builtQuery;
},

domainAndPrefix: function (prefixes) {
return site + this.objectToPrefix(prefixes) + "/" + this.collectionName.underscore();
},

fileTypeAndQuery: function (query) {
return ".xml" + this.objectToQuery(query);
},

elementPath: function (id, prefixes, query) {
return this.domainAndPrefix(prefixes) + "/" + id + this.fileTypeAndQuery(query);
},

collectionPath: function (prefixes, query) {
return this.domainAndPrefix(prefixes) + this.fileTypeAndQuery(query);
},

pathWithAssumptions: function (identifier, options){
options = options || {};
options.parents = options.parents || {};
options.query = options.query || {};

if (!identifier || identifier.constructor !== Number) {
return this.collectionPath(options.parents, options.query);
}
else {
return this.elementPath(identifier, options.parents, options.query);
}
}
});

Ext.apply(Model.prototype, new Ext.util.Observable());

var eventsToAdd = {};
"build find destroy save update".split(" ").forEach(
function(word){
eventsToAdd["before"+word] = true;
eventsToAdd["after"+word] = true;
});
Model.prototype.addEvents(eventsToAdd);

// Using opts to avoid namespace conflict
Model.prototype.Base = function(attributes, opts, modelName) {
var newRecord = false;
this.setAttributes(attributes);
this.newRecord = newRecord;
this.modelName = modelName;
this.editing = false;
this.dirty = false;

if (opts.newRecord == "undefined") {
newRecord = true;
}
else {
this.getModel().addToCache(this);
}
this.id = Ext.id();
};

Ext.apply(Model.prototype.Base.prototype,{
getModel: function(name) {
if (name) {
name = name.camelize().capitalize();
}
return options.scope[name||this.modelName];
},

save: function(options) {
var path = this.pathForSave(options),
method = this.putOrPost();

callbackWrapper = this.callbackWrappers[method].createDelegate(this);

this.fireEvent("beforesave");

return transport[method](path, options||{}, callbackWrapper, this.writeXML());
},

updateAttributes: function(attributes) {
this.setAttributes(attributes);
this.fireEvent("beforeupdate");
this.save();
},

destroy: function(options) {
var path = this.model.pathWithAssumptions(this.attributes.id, options)
this.fireEvent("beforedestroy");
return transport.DELETE(path, options||{},
function(connection, callback) {
Model.removeFromCache(this);
this.fireEvent("afterdestroy");
callback();
});
},

inspect: function() { /* STUB */},

get: function(attribute) {
return this.attributes[attribute];
},

beginEdit: function() {
this.editing = true;
this.modified = {};
},

cancelEdit: function () {
this.editing = false;
delete this.modified;
},

endEdit: function() {
this.editing = false;
if (this.dirty && this.store) {
this.store.afterEdit(this);
}
},

set: function(field, value) {
if (this.attributes[field] == value) { return; }
if (!this.modified) { this.modified = {}; }
if(typeof this.modified[field] == "undefined") {
this.modified[field] = this.attributes[field];
}
this.attributes[field] = value;
this.dirty = true;
},

commit: function() {
this.dirty = false;
delete this.modified;
if (this.store) { this.store.afterCommit(this) };
},

reject: function() {
var slot;
for (slot in this.modified) {
this.attributes[slot] = this.modified[slot];
}
this.dirty = false;
delete this.modified;
if (this.store) { this.store.afterReject(this) }
},

join: function(store) {
this.store = store;
},

setAttributes: function(attributes) {
if (!this.attributes) { this.attributes = {}; }
Ext.apply(this.attributes, attributes);
this.data = this.attributes;
},

pathForSave: function(options) {
if (this.newRecord) {
return this.model.pathWithAssumptions(options);
}
else {
return this.model.pathWithAssumptions(this.attributes.id, options);
}
},

putOrPost: function () {
return this.newRecord ? "POST" : "PUT";
},

callbackWrappers: {
PUT: function(connection, callback) {
this.fireEvent("afterupdate");
callback(connection);
},

POST: function(connection, callback) {
if (connection.statusCode == "400"){
//Not going to instantiate errors just yet
//that.errors = that.instantiateErrors(connection);
callback(connection);
}
else {
this.attributes.id = connection.getResponseHeader["Id"];
this.newRecord = false;
this.fireEvent("aftersave");
callback(connection);
}
}
},

fireEvent: function(eventName) {
this.model.fireEvent(eventName, this);
},

extractWritableAttributes: function () {
var slot, attribute, writable ={};
for (slot in this.attributes) {
attribute = this.attributes[slot];
if (!attribute.inspect && !attribute.push) {
writable[slot] = attribute;
}
}
return writable;
},

writableAttributes: function() {
var attributeWrapper = {};
attributeWrapper[this.model.name.toLowerCase()] = this.extractWritableAttributes();
return attributeWrapper;
},

writeXML: function () {
return XMLObjTree.hash_to_xml(null, this.writableAttributes());
}

});
//console.log(options.scope)
options.scope[name] = new Model();
}

[deitch]

Interesting. Very interesting. So you use the ActiveRecord/Resource paradigm to extend any objects so they can be managed/persisted. You don't have an ORM manager; you have the class (and instances) itself have the necessary functions. Kind of breaks OOP a little, but not awfully (I am sure someone will disagree with me here). It seems like it would be cleaner if we could just create instances, and then when ready hand them off to a different manager. When we load, we load from the manager. I guess it is the Hibernate ORM with POJO vs. Ruby on Rails with AR debate.

Are you releasing what you did as a library with docs?

Out of curiosity, what do you do for a day-job? Contact me offline avi [at] atomicinc [dot] com.

[wout]

I just found http://www.jesterjs.org, probably it can help...

Jester is a JavaScript client for REST APIs that use Rails conventions, and is inspired by Rails' own ActiveResource.

Simple, ActiveRecord-like syntax
Works with JSON or XML
Speaks with Rails scaffolds

[Collin Miller]

I really would like to release this work to the crowd with some documentation to get it into the whole open source ecosystem.

This is one of the reasons I started this thread: http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=84

I just need to take an evening aside to get it into googlecode or something.
Then another evening to restructure it for Ext documentation.

[Collin Miller]

[QUOTE=wout;78149]I just found http://www.jesterjs.org, probably it can help...

Jester is a JavaScript client for REST APIs that use Rails conventions, and is inspired by Rails' own ActiveResource.

Simple, ActiveRecord-like syntax
Works with JSON or XML
Speaks with Rails scaffolds

[deitch]

Still a bit strange to me. I like the idea of the functions being on a separate manager for cleanliness in separation of concerns. You are going down the "assign a data source for a class, then dress up the class to do all its functions." I like the "assign a data source for a class, then use a dressed up manager to manage the class." I think my primary interests here are in knowing that logic code can use the objects and only the objects, and thus change persistence model in the future. It also makes it easier to change the manager's functionality in one place.

What I like about what you have done, though, is that each class has its own data source. No reason a manager cannot do that, too, though, even though traditional ORMs have not.

[deitch]

Oh, and Collin, where is that Ext.data.Store tie-in you offered?
Avi

[Collin Miller]

Avi,

Again, an awkward way to do it, so lets just call it a proof of concept.

Store exposes a handful of methods:

get, set, commit, reject, etc.

I just made versions of those methods and stuck them right into the code already shown.

[deitch]

Ah, so you let your class act like a Store. It is a bit of an issue, because the Store has filter, etc. items on it.

[Collin Miller]

Yeah, I like the idea of having a centralized resource manager sitting in front of gears, xmlhttp, in-memory objects but I think what I've been working on went off on the wrong track.

[deitch]

Different track. Lots of people like it your way. The hard part about my way is that JavaScript is awful for introspection to determine the class name.