Announcing SilkJS

[jay@moduscreate.com]

Mike,

Congratulations!!

[mschwartz]

Thanks Jay!

I'd like to also announce that I wrote up a nice ExtJS 4.1 beta demo using SilkJS. It can be viewed and downloaded at http://github.com/mschwartz/SilkJS-extjs4.

To whet peoples' appetite, here's the server side code:

SQL = new MySQL();
SQL.connect();


Schema.add({
    name: 'Users',
    fields: [
        { name: 'userId', type: 'int', autoIncrement: true, defaultValue: 0 },
        { name: 'username', type: 'varchar', size: 64, header: 'User Name', width: 128, autoExpand: true, editable: true },
        { name: 'password', type: 'varchar', size: 64, header: 'Password', serverOnly: true, editable: true },
        { name: 'email', type: 'varchar', size: 128, header: 'E-Mail Address', width: 128, editable: true },
        { name: 'phone', type: 'varchar', size: 128, header: 'Phone Number', width: 128, editable: true },
        { name: 'created', type: 'int', defaultValue: function() { return parseInt(new Date().getTime() / 1000, 10); }, header: 'Created', width: 150, format: 'DateTime', editable: false }
    ],
    primaryKey: 'userId',
    indexes: [
        'username',
        'email'
    ]
});


mysql.close();
delete SQL;


function Server_action() {
    switch (req.data.method) {
        case 'listUsers':
            Json.success(Schema.list('Users', {}, function(o) {
                o = Schema.clean('Users', o);
            }));
        case 'editUser':
            var example = Json.decode(req.data.example);
            example.userId = example.userId || 0;
            Schema.putOne('Users', example);
            Json.success();
        case 'deleteUsers':
            var examples = Json.decode(req.data.examples);
            forEach(examples, function(example) {
                Schema.remove('Users',example);
            });
            Json.success();
    }
}

The code implements a "schema" which is a noSQL-like or Ext Model-like database table definition. The Server_action() method implements the entire CRUD operations for Web Service requests via Ajax from the client. The listUsers "method" generates JSON suitable for use by Ext's data stores and paging toolbars.

Json.success() takes an object, adds "success: true" to it, and sends it JSON encoded to the client. Schema.list() generates an object of the form:

{
  count: n,
  list: items
}

The store on the client side would have totalProperty: 'count', root: 'list'.

The Schema.list() method takes three arguments:
* Name of the Schema
* An "example" object
* An optional callback that is called for each record of the result

In the code above, the callback calls Schema.clean(o) which removes any sensitive fields from the record, particularly the password field (which we don't want to send over the network!); that field being marked "serverOnly: true" in the schema.

The "example" object is a partial (example) record describing the records that should be returned from the database by Schema.list(). In this case, the empty object {} tells Schema.list() to return all records. If the example were { username: '%jay%' }, Schema.list() would return records with usernames LIKE '%jay%' (containing "jay"). The example can be as simple or complex as your logic demands. For example, { username: '%jay%', email: 'jay@moduscreate.com' } would return all records with username LIKE '%jay%' and with Jay's email address.

Many of the Schema class methods take an example object. Note the Schema.putOne() takes the name of the schema (table) and an example. If the example object is partial (contains only some of the table's fields), the remaining fields will be filled in with defaultValue from the schema definition. Schema.putOne() will add a new record to the database if userId (primary key, auto increment) is 0, or overwrite an existing record if userId is non-zero. Thus the one Server_action() method does both create and update of the CRUD operations.

One more thing I love to rave about the Schema implementation is that it is SQL free and dynamic. By SQL free, I mean that the SQL queries are dynamically generated under the hood, based upon the example objects passed in (and the schema definition). By dynamic I mean that if you edit the schema definition (passed to Schema.add), all the "ALTER TABLE" queries required to make the database table match the schema definition are automatically generated and executed. So if you add a field, the field is added to the table in the database; if you delete a field, it's deleted from the database; if you rename a field, it's renamed in the database, etc.

How slick is it? Let's see what it takes to implement a "change password" method:

var o = Schema.findOne({ username: 'jay' });
if (o) {
  o.password = 'newpassword';
  Schema.putOne(o);
}

Schema.findOne() just happens to return us a record which also happens to be a (fulL) example!

How about logging in a user?

if (Schema.findOne({ username: req.data.username, password: req.data.password })) {
  // log him in!
}

[mschwartz]

Just thought I'd share a screen shot of a SilkJS based application I've spent about 20 hours on (so far).

[mitchellsimoens]

Very nice! I see you are in JSmag again!

[mschwartz]

Very nice! I see you are in JSmag again!

Yeah, I've been regularly writing for JSMag for the past 1.5+ years now. Next month, I'll be writing about this IDE project ;-)

[mschwartz]

I implemented Modules 1.1 over the weekend.

As well as SSH2.

XMLHttpRequest was implemented last weekend.

The current specification in practice ( http://www.commonjs.org/specs/ )

Modules 1.1
Packages 1.0
System 1.0

The async transport definitions of modules & packages have the most traction in the browser world and would probably make the most sense.

I'm confused as to what is nebulous about the CommonJS effort...? We don't want to have to modify our code to work on the server and the browser. The ability to package up code and use it in multiple runtimes / runtimes.

in particular
define( ); require(); and the concepts of exports and module.

How a person goes about implementing that may be nebulous, but the movement itself is pretty straight forward. Personally I don't think the world needs another "flavor" of javascript with it's own set of quirks like .jst

[jakeonthenet]

Mike,

Very interesting project. Do you have anything that describes how silk compares to node?

Thanks,
-Jake

[mschwartz]

For starters, there's not a single asynchronous type mechanism in SilkJS or its libraries. For HTTP serving, the server is pre-fork model, not evented.

I call it a "swiss army knife" because you can basically use it as a command shell for just about any purpose aside from HTTP serving. It is something like PHP or Perl, an interpreter you run from the command line, but the language is obviously JavaScript. Like those other languages, SilkJS provides a rich set of functions to interface to the operating system (file system, processes, sockets, etc.) and to other interesting libraries (MySQL client, SSH2 client, GD2 graphics library, and so on).

You can write a WWW server in PHP or Perl and run it from the command line, if you like. The HTTP server for SilkJS is just that - a WWW server written in JavaScript. The beauty of it is the lightweight design - there's very little to have to learn about the internal workings of the compiled C++ program (silkjs binary).

You invoke it something like this:

silkjs path/to/somefile.js

SilkJS simply loads somefile.js compiles it, runs it, and calls a main() function in the script. The run phase is where your require() or include() and other global object initialization takes place. Any additional arguments on the silkjs command line are passed to main() as arguments; the WWW server main() function just includes any additional files on the command line, allowing you to extend the WWW server.

The require() function, BTW, was implemented 100% in JavaScript calling the existing SilkJS native methods.

The WWW server is blazingly fast. It's roughly as fast as Apache at serving static content, much faster than Apache + PHP (like 30x faster), faster than lighthttp, and many times faster than NodeJS for WWW applications. It's even faster than NodeJS for running console type applications - a fellow installed both NodeJS and SilkJS, ran some program he wrote for NodeJS in a loop 1000 times, and SilkJS was about 25% faster.

The HTTP server was designed especially for RIA implementations. The number of clients that can be accessing your server at one time is limited to the number of requests/second the server can do. Working with PHP, I find many simple Ajax type requests (like to send JSON for a grid/store) can take 50ms to 100ms if you use some PHP framework like Zend. Working with SilkJS, those requests are 1ms to 5ms, depending on how complex your server-side business logic is

Unlike CGI, mod_php, etc., the SilkJS WWW server is a true application server. When the server starts up, it (and your extensions) include() or require() whatever API/framework you want available, and it's callable from your JavaScript code executing each request.

The HTTP server serves static files, like HTML, CSS, JavaScripts, images, etc. If the file has a .jst extension, the file is loaded, compiled, and executed (JST is a JSP-like syntax). JST can be used to generate RSS, XML, dynamic CSS, HTML, or whatever your creativity allows. If the file has a .md extension, then the file is loaded and run through Showdown (a markdown interpreter) and output. Extending the server to handle additional file types is trivial (markdown took me about 10 minutes to add).

As I mentioned, the HTTP server is meant for RIA. The idea is you hit some .jst file that generates the <link> and <script> tags and HTML markup for your page, delivers the images and other static content, then handles all the Ajax requests for the app as well.

If you have specific questions, feel free to ask.

Mike,

Very interesting project. Do you have anything that describes how silk compares to node?

Thanks,
-Jake

[mschwartz]

For those of you interested in CoffeeScript, SilkJS now fully supports it.

You can make CoffeeScript pages. These are files with .coffee file extensions in the server's documentRoot. The server loads, compiles, and caches these upon request. If the file is changed on disk, it is recached.

You can also extend the server with CoffeeScript. Both include() and require() will load and compile a file/module if one exists with either .js or .coffee file extensions.

More details here:

https://github.com/mschwartz/SilkJS/wiki/CoffeeScript

[mschwartz]

SilkJS now compiles, caches, and serves LessCSS files (files with .less extension).

It also features UglifyJS for JavaScript minification.

More like these to come.