PDA

View Full Version : Announcing SilkJS



mschwartz
6 Dec 2011, 6:54 AM
I am pleased to announce SilkJS, a V8 based Swiss Army Knife (and HTTP server).

SilkJS currently compiles for Linux and Mac OSX, but should port trivially to any flavor of Unix. It was originally written for Ubuntu Linux and ported to OSX (a FreeBSD flavored Unix) in under 2 hours.

So what is it?

SilkJS is a C++ program that links with the Google V8 JavaScript engine and several other libraries. It provides as simple and direct as it can, methods that make calling operating system and those other libraries' functions from JavaScript context.

Running SilkJS is as simple as:


path_to/SilkJS some_javascript_file.js


SilkJS simply loads the specified JavaScript file, compiles it, runs it, then calls a main() function provided by the script. Any additional arguments on the command line are passed to the main() function.

SilkJS is not NodeJS. It merely provides the "glue" to the operating system and other library functions to make them callable from your JavaScript program.

I wrote an HTTP server almost entirely in JavaScript using SilkJS that's included with SilkJS. The HTTP server is a pre-fork model server, and it's very fast. I was able to run ab (apache benchmark) against it with 20,000 concurrent requests and it served a 13.3K file 50,000 times in under 3 seconds with no errors. It is several times faster than NodeJS at serving NodeJS' sample "hello, world" demo on the nodejs.org home page. It is 30x or more faster than Apache and PHP 5.3 at running real world server-side business logic.

I also wrote a console program in JavaScript using ncurses that technically minded webmasters will find quite useful. It refreshes the screen once per second, displaying in real-time the CPU usage, memory usage, disk activity, and virtual memory. With this program, you can tell if a busy WWW server is CPU, memory, and/or disk bound. Or if it's swapping heavily.

SilkJS comes with a number of library interfaces built-in. These include console I/O, file system I/O, the gd2 graphics library, MySQL, ncurses, networking, and process management (fork, wait, etc.).

nodejs.org is served by ngnix, not by nodejs. SilkJS can be viewed at h (http://server5.sportstwo.com:9090/)ttp://silkjs.org/ and it is entirely served by SilkJS itself.

SilkJS may be downloaded and used freely from github at http://github.com/mschwartz/SilkJS.

Please see the Wiki http://github.com/mschwartz/SilkJS/wiki, it contains more details about building SilkJS, benchmarks, example programs, etc. Of particular interest to us ExtJS developers is the ORM, which fully supports ExtJS DataStore.

I've started a Google Group for supporting SilkJS at http://groups.google.com/group/silkjs

Check it out!

mitchellsimoens
6 Dec 2011, 7:28 AM
Dude... this is awesome! I have just converted my stuff from Apache to Node but will play around with this. When I do get time and any updates... expect some forking going on!

mschwartz
6 Dec 2011, 7:49 AM
To add to my previous post...

The idea of compiling and running the script on the command line, then calling its main() function is so you can load up your library or API, then actually run your main business logic.

So:



include('lib/Util.js');
// include more stuff...

function main() {
// call stuff in lib/Util.js
}


SilkJS comes with an installer (for Linux):


./SilkJS Install.js


It creates a /usr/local/bin/SilkJS binary, and /usr/share/SilkJS where it copies lib/ and appropriate other files.

If you pass include() an absolute path, it looks for the file there, otherwise it looks in the current directory then /usr/share/SilkJS.

It also supports shebang:



#!/usr/local/bin/SilkJS

// file is ./hello.js
function main() {
println('hello, world');
}


chmod 755 ./hello.js

then you can run hello.js:
./hello.js

If you don't have access to Ubuntu, you can install it in VirtualBox (http://virtualbox.org). Download the ISO from http://ubuntu.com (I use the server version, not the desktop one).

The ORM works on top of MySQL. It's documented here: http://github.com/mschwartz/SilkJS/wiki/Schema.js.

The ORM is designed to work with ExtJS and DataStores, though it's flexible enough to do anything.

mschwartz
6 Dec 2011, 7:58 AM
Not on the wiki yet is Jst.

The HTTP server will statically serve .html, .css, .js, .jpg, etc., files. If it sees a .jst extension on a file, it loads it and runs it.

What is Jst? It's a lot like JSP (or PHP), except the language is JavaScript AND you can call any function of any class you've done include() on in your main program.



<% var title = 'Hello from Jst'; %>
<html>
<head>
<title><%= title %></title>
</head>
<body>
<h1>Users</h1>
<ul>

<% forEach(SQL.getDataRows('SELECT * FROM Users'), function(user) { %>
<li><%= user.username %></li>
<% } %>

</ul>
</body>
</html>


<%
// any JavaScript can go here
%>

<%= JavaScript expression %>
(emits result of the expression)

tobiu
6 Dec 2011, 8:03 AM
sounds promising, i will check it out when i find some free time!

mschwartz
8 Dec 2011, 5:19 AM
28 people watching the repository on github after 2 days.

tobiu
8 Dec 2011, 5:23 AM
it's a start :)

hendricd
8 Dec 2011, 6:10 AM
Sweet Mike!

mschwartz
8 Dec 2011, 6:40 AM
26 of those 28 yesterday alone. 13 overnight.

I haven't even been doing any "marketing" of it yet. I'm hoping to have silkjs.org WWW site up this week and then I have people who run JS blog sites wanting to write about it.

For my next JSMag article, I will write about how to integrate SilkJS' ORM with ExtJS grids in 250 lines of code or less.

ZeusTheTrueGod
9 Dec 2011, 4:28 AM
I am mostly interested in applications for this project for developing / testing extjs app .
Can we use that for unit testing, for intellisense or for anything useful?

mschwartz
9 Dec 2011, 5:21 AM
I am mostly interested in applications for this project for developing / testing extjs app .
Can we use that for unit testing, for intellisense or for anything useful?

The typical use for it is to have it serve your HTTP requests for your application. It can serve your html, css, images, scripts, etc. It's ideally suited for quickly responding to Ajax requests that Sencha Apps make back to the server. Like to get records for a DataStore, Ext.Direct, and that sort of thing.

It could be used to implement a pure server-side MVC Web 1.x style site.

However, it's also a command line processor. So you can load up all your libraries that you use in your WWW server application and do some long running operation, like importing a million records into your database, add watermarks to all the images in a directory structure, etc.'

It's perfectly suited to run something like JSDoc toolkit, JSLint, and so on. In theory, any Unix command line program, including screen-oriented ones like VIM, could be written in JavaScript using SilkJS.

This has nothing to do with code running in the browser, although you could share entire .js files between server and client. I could see having a Validator class that could validate form input on both sides of the connection, for example.

There are JavaScript libraries that implement the DOM for server-side programs. For example, there's env.js. In theory, you could use SilkJS + env.js and use that environment for unit testing browser-side code.

esatterwhite
9 Dec 2011, 8:06 AM
CommonJS compliant?

mschwartz
9 Dec 2011, 8:17 AM
I will be implementing require() this weekend.

CommonJS compliant is a nebulous term. There's quite a few standards.

Which are you thinking of?

esatterwhite
9 Dec 2011, 8:47 AM
The current specification in practice ( http://www.commonjs.org/specs/ )

Modules 1.1
Packages 1.0
System 1.0

The async transport definitions of modules & packages have the most traction in the browser world and would probably make the most sense.

I'm confused as to what is nebulous about the CommonJS effort...? We don't want to have to modify our code to work on the server and the browser. The ability to package up code and use it in multiple runtimes / runtimes.

in particular
define( ); require(); and the concepts of exports and module.

How a person goes about implementing that may be nebulous, but the movement itself is pretty straight forward. Personally I don't think the world needs another "flavor" of javascript with it's own set of quirks like .jst

mschwartz
9 Dec 2011, 8:53 AM
If I implement those things, but not Binary/B, is it CommonJS compliant?

That's what I meant by nebulous...

mschwartz
10 Dec 2011, 5:27 AM
28 people watching the repository on github after 2 days.

77 watching it on github.

chrixian
10 Dec 2011, 9:00 PM
SilkJS is a C++ program that links with the Google V8 JavaScript engine and several other libraries. It provides as simple and direct as it can, methods that make calling operating system and those other libraries' functions from JavaScript context.

Maybe I'm missing something but isn't that what Node already does?

mschwartz
11 Dec 2011, 8:12 AM
Maybe I'm missing something but isn't that what Node already does?

There's quite a difference between NodeJS and SilkJS. There's not a built-in (C++/native) function or any in the .js library code that has a single callback in SilkJS. It's entirely synchronous and blocking style. You write code for it in the same style you would for most other server-side languages (PHP, ASP, Java, etc.).

One of the guys who installed SilkJS on his system sent me this:



Request per second results...

silkjs = 26759 rps
lighttpd = 17183 rps (v1.4.29)
nodejs = 2212 (v0.6.5 - single core, simple static server)

CPU = i7-2630QM CPU @ 2.00GHz

mschwartz
11 Dec 2011, 11:53 AM
The official site for SilkJS is now up.

http://silkjs.org

I fixed the opening post to include the URL.

mschwartz
12 Dec 2011, 7:22 AM
http://www.sencha.com/forum/showthread.php?161210-Using-SilkJS-ORM-and-ExtJS-Grids-Together&p=689650#post689650

I (http://www.sencha.com/forum/showthread.php?161210-Using-SilkJS-ORM-and-ExtJS-Grids-Together&p=689650#post689650) created a demo Ext-3.4.0 app using SilkJS and its ORM. The server-side code, apart from defining the database tables and the HTML for the application's page consists of under 20 lines to list, create, edit, and delete records from the ExtJS GUI.

jay@moduscreate.com
28 Dec 2011, 8:07 AM
Mike,

Congratulations!!

mschwartz
28 Dec 2011, 8:44 AM
Thanks Jay!

I'd like to also announce that I wrote up a nice ExtJS 4.1 beta demo using SilkJS. It can be viewed and downloaded at http://github.com/mschwartz/SilkJS-extjs4.

To whet peoples' appetite, here's the server side code:



SQL = new MySQL();
SQL.connect();


Schema.add({
name: 'Users',
fields: [
{ name: 'userId', type: 'int', autoIncrement: true, defaultValue: 0 },
{ name: 'username', type: 'varchar', size: 64, header: 'User Name', width: 128, autoExpand: true, editable: true },
{ name: 'password', type: 'varchar', size: 64, header: 'Password', serverOnly: true, editable: true },
{ name: 'email', type: 'varchar', size: 128, header: 'E-Mail Address', width: 128, editable: true },
{ name: 'phone', type: 'varchar', size: 128, header: 'Phone Number', width: 128, editable: true },
{ name: 'created', type: 'int', defaultValue: function() { return parseInt(new Date().getTime() / 1000, 10); }, header: 'Created', width: 150, format: 'DateTime', editable: false }
],
primaryKey: 'userId',
indexes: [
'username',
'email'
]
});


mysql.close();
delete SQL;


function Server_action() {
switch (req.data.method) {
case 'listUsers':
Json.success(Schema.list('Users', {}, function(o) {
o = Schema.clean('Users', o);
}));
case 'editUser':
var example = Json.decode(req.data.example);
example.userId = example.userId || 0;
Schema.putOne('Users', example);
Json.success();
case 'deleteUsers':
var examples = Json.decode(req.data.examples);
forEach(examples, function(example) {
Schema.remove('Users',example);
});
Json.success();
}
}


The code implements a "schema" which is a noSQL-like or Ext Model-like database table definition. The Server_action() method implements the entire CRUD operations for Web Service requests via Ajax from the client. The listUsers "method" generates JSON suitable for use by Ext's data stores and paging toolbars.

Json.success() takes an object, adds "success: true" to it, and sends it JSON encoded to the client. Schema.list() generates an object of the form:


{
count: n,
list: items
}

The store on the client side would have totalProperty: 'count', root: 'list'.

The Schema.list() method takes three arguments:
* Name of the Schema
* An "example" object
* An optional callback that is called for each record of the result

In the code above, the callback calls Schema.clean(o) which removes any sensitive fields from the record, particularly the password field (which we don't want to send over the network!); that field being marked "serverOnly: true" in the schema.

The "example" object is a partial (example) record describing the records that should be returned from the database by Schema.list(). In this case, the empty object {} tells Schema.list() to return all records. If the example were { username: '%jay%' }, Schema.list() would return records with usernames LIKE '%jay%' (containing "jay"). The example can be as simple or complex as your logic demands. For example, { username: '%jay%', email: 'jay@moduscreate.com' } would return all records with username LIKE '%jay%' and with Jay's email address.

Many of the Schema class methods take an example object. Note the Schema.putOne() takes the name of the schema (table) and an example. If the example object is partial (contains only some of the table's fields), the remaining fields will be filled in with defaultValue from the schema definition. Schema.putOne() will add a new record to the database if userId (primary key, auto increment) is 0, or overwrite an existing record if userId is non-zero. Thus the one Server_action() method does both create and update of the CRUD operations.

One more thing I love to rave about the Schema implementation is that it is SQL free and dynamic. By SQL free, I mean that the SQL queries are dynamically generated under the hood, based upon the example objects passed in (and the schema definition). By dynamic I mean that if you edit the schema definition (passed to Schema.add), all the "ALTER TABLE" queries required to make the database table match the schema definition are automatically generated and executed. So if you add a field, the field is added to the table in the database; if you delete a field, it's deleted from the database; if you rename a field, it's renamed in the database, etc.

How slick is it? Let's see what it takes to implement a "change password" method:



var o = Schema.findOne({ username: 'jay' });
if (o) {
o.password = 'newpassword';
Schema.putOne(o);
}


Schema.findOne() just happens to return us a record which also happens to be a (fulL) example!

How about logging in a user?



if (Schema.findOne({ username: req.data.username, password: req.data.password })) {
// log him in!
}

mschwartz
9 Jan 2012, 7:01 AM
Just thought I'd share a screen shot of a SilkJS based application I've spent about 20 hours on (so far).

mitchellsimoens
9 Jan 2012, 7:07 AM
Very nice! I see you are in JSmag again!

mschwartz
9 Jan 2012, 7:22 AM
Very nice! I see you are in JSmag again!

Yeah, I've been regularly writing for JSMag for the past 1.5+ years now. Next month, I'll be writing about this IDE project ;-)

mschwartz
16 Jan 2012, 10:34 AM
I implemented Modules 1.1 over the weekend.

As well as SSH2.

XMLHttpRequest was implemented last weekend.


The current specification in practice ( http://www.commonjs.org/specs/ )

Modules 1.1
Packages 1.0
System 1.0

The async transport definitions of modules & packages have the most traction in the browser world and would probably make the most sense.

I'm confused as to what is nebulous about the CommonJS effort...? We don't want to have to modify our code to work on the server and the browser. The ability to package up code and use it in multiple runtimes / runtimes.

in particular
define( ); require(); and the concepts of exports and module.

How a person goes about implementing that may be nebulous, but the movement itself is pretty straight forward. Personally I don't think the world needs another "flavor" of javascript with it's own set of quirks like .jst

jakeonthenet
16 Jan 2012, 11:33 AM
Mike,

Very interesting project. Do you have anything that describes how silk compares to node?

Thanks,
-Jake

mschwartz
16 Jan 2012, 1:26 PM
For starters, there's not a single asynchronous type mechanism in SilkJS or its libraries. For HTTP serving, the server is pre-fork model, not evented.

I call it a "swiss army knife" because you can basically use it as a command shell for just about any purpose aside from HTTP serving. It is something like PHP or Perl, an interpreter you run from the command line, but the language is obviously JavaScript. Like those other languages, SilkJS provides a rich set of functions to interface to the operating system (file system, processes, sockets, etc.) and to other interesting libraries (MySQL client, SSH2 client, GD2 graphics library, and so on).

You can write a WWW server in PHP or Perl and run it from the command line, if you like. The HTTP server for SilkJS is just that - a WWW server written in JavaScript. The beauty of it is the lightweight design - there's very little to have to learn about the internal workings of the compiled C++ program (silkjs binary).

You invoke it something like this:

silkjs path/to/somefile.js

SilkJS simply loads somefile.js compiles it, runs it, and calls a main() function in the script. The run phase is where your require() or include() and other global object initialization takes place. Any additional arguments on the silkjs command line are passed to main() as arguments; the WWW server main() function just includes any additional files on the command line, allowing you to extend the WWW server.

The require() function, BTW, was implemented 100% in JavaScript calling the existing SilkJS native methods.

The WWW server is blazingly fast. It's roughly as fast as Apache at serving static content, much faster than Apache + PHP (like 30x faster), faster than lighthttp, and many times faster than NodeJS for WWW applications. It's even faster than NodeJS for running console type applications - a fellow installed both NodeJS and SilkJS, ran some program he wrote for NodeJS in a loop 1000 times, and SilkJS was about 25% faster.

The HTTP server was designed especially for RIA implementations. The number of clients that can be accessing your server at one time is limited to the number of requests/second the server can do. Working with PHP, I find many simple Ajax type requests (like to send JSON for a grid/store) can take 50ms to 100ms if you use some PHP framework like Zend. Working with SilkJS, those requests are 1ms to 5ms, depending on how complex your server-side business logic is

Unlike CGI, mod_php, etc., the SilkJS WWW server is a true application server. When the server starts up, it (and your extensions) include() or require() whatever API/framework you want available, and it's callable from your JavaScript code executing each request.

The HTTP server serves static files, like HTML, CSS, JavaScripts, images, etc. If the file has a .jst extension, the file is loaded, compiled, and executed (JST is a JSP-like syntax). JST can be used to generate RSS, XML, dynamic CSS, HTML, or whatever your creativity allows. If the file has a .md extension, then the file is loaded and run through Showdown (a markdown interpreter) and output. Extending the server to handle additional file types is trivial (markdown took me about 10 minutes to add).

As I mentioned, the HTTP server is meant for RIA. The idea is you hit some .jst file that generates the <link> and <script> tags and HTML markup for your page, delivers the images and other static content, then handles all the Ajax requests for the app as well.

If you have specific questions, feel free to ask.


Mike,

Very interesting project. Do you have anything that describes how silk compares to node?

Thanks,
-Jake

mschwartz
11 Feb 2012, 5:26 PM
For those of you interested in CoffeeScript, SilkJS now fully supports it.

You can make CoffeeScript pages. These are files with .coffee file extensions in the server's documentRoot. The server loads, compiles, and caches these upon request. If the file is changed on disk, it is recached.

You can also extend the server with CoffeeScript. Both include() and require() will load and compile a file/module if one exists with either .js or .coffee file extensions.

More details here:

https://github.com/mschwartz/SilkJS/wiki/CoffeeScript

(https://github.com/mschwartz/SilkJS/wiki/CoffeeScript)

mschwartz
21 Feb 2012, 6:29 AM
SilkJS now compiles, caches, and serves LessCSS files (files with .less extension).

It also features UglifyJS for JavaScript minification.

More like these to come.

jay@moduscreate.com
2 Apr 2012, 6:26 PM
Silk now has MemcacheD :D. Looking forward to integrating this soon.

mschwartz
3 Apr 2012, 4:09 AM
New and improved silkjs.org WWW site at http://silkjs.org. Powered by SilkJS, of course!

mschwartz
19 Oct 2012, 10:31 AM
Just an update. The site is now hosted at http://silkjs.net/