10 Mar 2014 12:27 PM #1
Best practices for developing shared packages under source control
Best practices for developing shared packages under source control
We are in the early stages of a relatively large development effort that will use Ext JS for its client user interface. The size of the team is not completely clear at the moment, but suffice it to say that there will be several developers working concurrently on building pieces of the UI.
The plan at the moment is to build many of the interface components following recommendations about using Sencha 'packages' and using Sencha Cmd to compile component packages into application pages. We also intend to setup Sonatype Nexus as a remote repository for compiled packages.
Where we are seeking guidance is in best practices on how to handle/setup development with a combination of multiple developers, source control and a remote package repository.
From other posts I've seen here, I gather that from a source control perspective people generally commit the workspace folder to source control (with the exception of the build folder and perhaps the Ext library itself). That means that source control has a workspace folder and the workspace folder contains a 'packages' folder that contains the source for each of the packages under development. This all makes sense to me.
Where I'm struggling is with the fact that the same 'packages' folder in the workspace is used for 2 things: it contains the source for packages that are under development and it is also the location where the source for shared packages is expanded. What you can end up with is a loop of sorts like this
- Code is edited in workspace\packages folder
- Developer commits change to source control
- Continuous integration or build server builds package and adds to remote repository
- Package is pulled from remote repository into local repository
- Package is expanded from the local repository into the workspace\packages folder (the same place where it started).
It looks to me like the app refresh command checks the version in the workspace\packages folder and compares to the desired version specified in app.json. If it's the same version, then the package is left alone. If the versions do not match, then the app refresh command generates a warning indicating that workspace\packages includes what may be another version. The warning is then followed by a NullPointerException which terminates the app refresh command. It will also terminate an 'app build' command as well - which could lead to a whole other set of issues.
I suppose in a way that's all good and provides at least some safety against losing changes. The developer can still overwrite their changes, though, by using 'sencha app refresh -packages' or 'sencha package extract'. The developer should also be able to get the build to complete by tweaking the versions in their local app.json file.
Q: Are other development teams just taking this as an acceptable risk (because it takes a little extra effort by the developer to make it happen) and relying on some proper instruction on what to do and what not to do? There is still the problem of actually building the application, though, which leads to my related question further below.
Q: Would it not make sense to have a way to distinguish which folder contains packages under development vs. those that were expanded from the package repository? I suppose one could take it upon themselves to move packages to a development folder (something like dev_packages) after using the 'sencha generate package' command and add that other folder to the application's classpath. I haven't tried that, though, so it'd not clear if the build process or generation of bootstrap.js would work properly.
Q: Is there a clear and clean way to separate development packages from those that come from the package repository?
A related question to this has do to with the build process. If you want to build the application with precise control over the versions of packages that the application uses, it seems like you need to make sure that the application only uses the expanded packages from the package repository. For example, suppose I'm building a new production release that uses version 184.108.40.206 of some package. At the same time, another developer has committed changes to the source repository that are targeted for version 220.127.116.11 of that package. I want to make sure that the application doesn't accidentally use the new code from source control - it should use the package from the package repository.
Q: How do people handle this (assuring that the app uses the proper versions of packages)? I think one way would be to have a second workspace in source control where only the application code is stored. Since pulling that workspace from source control would not pull any package source it would have to get all package source from the package repository. I'm also guessing that if one were to go this route for application builds, then there could be an application in the main/package workspace that does not specify package versions, causing it to just use whatever versions it gets from source control. Developers could then use that application for package development. Alternatively, I suppose one could include some other scripting in your build that removes the packages folder before building the application, thereby forcing the app build to download packages from the package repository.
Any specific advise as to how to manage developing packages and applications using source control and a package repository would be appreciated.
10 Mar 2014 3:33 PM #2
+1. We have similar questions and concerns regarding source management on a larger team.
12 Mar 2014 5:28 AM #3
- Join Date
- Mar 2007
- Gainesville, FL
- Vote Rating
I have a question on your workflow. So you commit a change into source control, your CI then pushes what it builds to another repository? Why not use a single repository. You commit a change, others can then just pull it down? Why have a CI involved for building a package? When you want to deploy then the CI can do an app build and the packages will get built into the app.Mitchell Simoens @SenchaMitch
Sencha Inc, Senior Software Engineer
Check out my GitHub, lots of nice things for Ext JS 4 and Sencha Touch 2
Think my support is good? Get more personalized support via a support subscription. https://www.sencha.com/store/
Need more help with your app? Hire Sencha Services email@example.com
Want to learn Sencha Touch 2? Check out Sencha Touch in Action that is in print!
When posting code, please use BBCode's CODE tags.
12 Mar 2014 9:16 AM #4
We were thinking that we would model this similar to how one does Java development - in particular when using something like Maven for resolving dependencies.
In the Java world, your source files are Java files and your artifacts are Jar files. Developers pull Java files out of source control and commit back to source control. The CI tool builds Jar files, but those Jars do not go back into source control. The CI tool pushes the artifacts into the dependency repository - in our case Sonatype Nexus. One reason the Jars don't go back into source control is that they aren't truly source. This also allows one to use dependency management to build things that simply reference the compiled Jars that get pulled out of the dependency repository. Other Java projects would normally use something like Maven to get the proper compiled version from the dependency repository (and not use the direct source). If you are working on multiple Java projects at the same time, you can get the source (from source control) from multiple projects and set your classpath so that source takes precedence over Jars that a project includes via its dependencies.
We were thinking in the Ext JS / Sencha Cmd world, we might be able to do something similar using the following analogies:
Jar files => Package Files (the zip that gets created by building a package)
Maven + Sonatype Nexus => Sencha Cmd + Sonatype Nexus (for dependency management)
Here's a use case to consider. Suppose Developer A is using some previously released package. Developer B is making a bunch of changes to a that same package and committing them to source control. The package is in a constant state of flux as he works out the new features. I could imagine Developer A wanting to just use the released version of the package until the new version is more stable. It seems this would be somewhat challenging to manage given the structure of the source repository and the fact that the package source directory is re-purposed for the expanded source from the package repository. Now you should be able to manage that by finding the right version in source control and pulling that source, but it seems more intuitive to use the dependency management features to get the right version for you. I think the way you'd need to do this would be to clear out what got pulled from source control and force Sencha Cmd to fill that source folder with the old version's packaged source. If he does that, the developer then needs to be extremely careful about what he commits so he doesn't inadvertently roll back the whole package (because the old extracted source is in the folder that normally includes real source so it looks like pending changes for source control).
Perhaps part of the trick to this is managing multiple applications and multiple app.json files to handle building things different ways. For example, developers might want to mostly deal with the latest versions of things, but the CI should perhaps use specific versions.
Perhaps some description of how teams normally setup their applications (including whether their source is in the same workspace) and how teams manage dependencies for different types of builds. That's what we're really looking for here. If the whole workspace is committed to source control and CI pushes packages into Nexus for dependency management, then how to people setup the application and development environment? Do developers just not use the remote Sencha repository at all? I'm not sure how well that works on large, modular efforts where you may really want developers to use a remote repository for certain things.
20 Mar 2014 5:31 AM #5
Bumping this thread.
Can anyone supply some guidelines on how they have developers work with Cmd workspaces, source control and a shared package repository?
I suppose most things are reasonably handled by committing the workspace (and what I'd call development packages) to source control and just having developers work off of code from source control.
How does one manage builds, though - particularly if you want to assure that a build uses specific versions of packages. To do that, do people use a different workspace for that sort of build? For example, suppose a support team needs to patch on older version of an application. They'd want to build that patch using the correct versions of packages from that old release.
Using a different workspace for certain types of builds, though, would seem to lead down a path of having different copies of the application folder itself, which could be error prone. I suppose you could use a folder junction of some source control 'magic' to share an application across multiple workspaces.
Please share your thoughts and experiences.
11 Apr 2014 4:27 AM #6
For posterity, this article was published recently:
@kpiland had a few comments (as "Kent") to the article which relate to these questions, so be sure to check that out. I believe he may update this thread at some later date with findings specific to his use case.
11 Apr 2014 7:29 AM #7
After some back and forth on the blog posting noted by Arthur, I've finally come to terms with how we should handle this.
There are really two purposes of putting code into packages: to share code between applications (pages) in the same workspace, and to share code between applications in separate workspaces. The way packages should be used is slightly different for those two cases. Generally speaking, I'd consider applications in a single workspace to be pages in a single project/web application, whereas applications in separate workspaces are likely for separate projects/web applications.
By virtue of working in the same workspace, there is an assumption that the source is all under that project's control - and the source will generally be stored in the same place under source control. In fact, there is a setting in the 'package.json' file for such a package that designates it as a 'local' package so Sencha Cmd 'understands' not to overwrite its source with source from a remote package repository. Because the source is all under that project's control, builds of applications in that workspace should just use the package source (from source control). One thing that means is that when an application lists a required local package (i.e. one whose source is in the workspace) in the application's 'app.json' file, the reference should NOT specify a version for the package. I initially questioned how to be precise about exactly what is included in this type of build, but because the source is all in the same place in source control, then source control is the master of exactly what is included in the build.
In contrast, when a package is used by an application whose source is in a different workspace from where the package is managed, then the application should be treated as a consumer of something over which it has no control - much like using some Java utility library. In this scenario, because the source for the package is stored in a separate place, the application needs to gain access to the package via a local or remote package repository. In our case, we will setup Nexus to act as a remote repository. It is not required, but when accessing packages via a local or remote package repository, an application should probably specify package versions in order to be precise about exactly what is being included.
- When an application references a package whose source is under source control in the same workspace as the application, the application should not use a version reference when it requires the package. The build for that application will use the source from source control.
- When an application references a package whose source is under source control in a different workspace, the application should (but doesn't really have to) use a version reference when it requires the package. The package is then loaded from the local or remote package repository for inclusion in a build.
You can execute a command like this:
sencha config -prop workspace.packages.dir=<path to build temp>/buildpackages then sencha app refresh then sencha app build testing
The first part specifies a different location for the packages directory. The second part causes the app to refresh the referenced packages in the package directory. Since this is presumably not where the source resides, the packages will get pulled from the local/remote repository - so this does mean that ALL packages would need to be available that way unless you further customized something. The third part does the actual build. Even though the build does some form of refresh, I found it necessary to formally do a refresh in order to force a refresh of the packages directory if it already contained different versions of the packages. Without doing so, the build would fail if the version in the packages directory is different from what the application requires.
I should note that while this worked for a relatively simple case, I'm not sure how well it works for more complicated scenarios.
And finally - I've decided that this is probably NOT the way to do things, but I thought I'd post these findings anyway in case someone finds it useful.