Font size:

Package import via a Builder

Overview

This document describes how to store packages in the ControlTier repository using a Builder. The Build workflow defined in the Builder checks out source code, invokes build tool, and then subsequently imports the produced build artifacts into the repository.

The last step in the Build workflow is carried out by repoImport, a command that looks in the target directory for matching files and loads them into the repository. Part of the import process is to not only upload the file to the repository but to also register it along with pertinent metadata.

The next sections describe how buildstamps, package types and repoImport options are used to identify and register package files to the repository.

Buildstamps

A buildstamp is a user-defined build identifier. Depending on local convention a buildstamp might be defined using:

Date-time stamp: YYYYMMDD (20080630), YYYMMDDhhmm (20080630), YYYYMMDD.# (20060630.1)
Revision id: Revision id maintained by the source code repository.
Product version: This might be a major.minor number.
Rolling build number: A counter incremented for every build.

From the perspective of a Builder, these are just aribitrary strings but most of the Builder commands accept the -buildstamp option.

Package types

Normally, one Builder uploads one or more package files of the same type.

ControlTier provides a set of package control module types, any of which can be used to manage the creation and installation life cycle for your build artifacts: bin, ear, jar, pkg, rpm, tgz, war, zip. Each of these package control module types corresponds to files with the same file extension. For example, the a file named "simple.war" has the "war" extension and is manageable using the "war" control module.

When the need arises, you can create subtypes of these package modules, as well. This might be necessary if you want to override standard behavior or prescribe your own dependency or property constraints.

repoImport command

The basic strategy used by the repoImport command is to recursively look for matching files starting from the specified target directory, specified by the -targetdir option.

Because repoImport must support a wide range of conventions, the command has a many arguments. For the most part though the following options drive file matching:

-filebase: This is the filename minus the buildstamp and extension. For example, given the filename simple-123.war, "simple" is the filebase.
-separator: This string separates the filebase from the buildstamp. By default the dash character ("-") is used but your local convention might use something else. The "-" between "simple" and "123" is the separator in our example.
-buildstamp: This is the build identifer and is the string following the separator. For example, "123" is the buildstamp for simple-123.war. The buildstamp often corresponds to the package version.
-extension: This is the suffix of the filename. For the filename simple-123.war, "war" is the extension.

Each of these parameters can accept Java-style regular expressions though in normal practice, using a regex makes sense mostly to specify the -filebase argument.

Besides these options, the -installroot option is also often necessary. This option specifies the directory where the package should be extracted.

Here is a complete listing of the repoImport command options:

repoImport [-- command-options]
-buildstamp               build identifier
-requirebuildstamp        true/false. if true buildstamp option required
-separator                string separating basename from buildstamp
-extension                file extension
-filebase                 file basename minus the buildstamp and extension
-type                     the Package module type name
-installroot              directory where package will be installed
-installrank              relative rankinging order
-version                  package version identifier
-propfile                 package metadata properties file
-targetdir                directory where import will begin looking for files
-min                      minimum package files to import
-max                      maximun package files to import

Builder configuration

Because it is cumbersome to type in so many repoImport command options, you can configure a Builder to default these options.

The following XML example shows how to configure a Builder with settings that default the repoImport options.

File listing: builder-metadata.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE project PUBLIC "-//ControlTier Software Inc.//DTD Project Document 1.0//EN" 
    "project.dtd">
<project>
  <!--
      **
      ** Defines Builder repoImport defaulting settings
      **
  -->

  <!-- 
    ** defaults option: -separator  
    -->
  <setting type="BuilderPackageSeparator" name="builder1" 
           description="String separating package base from version or buildstamp.
	   Dash is default" 
           settingValue="string"/>
  <!-- 
    ** defaults option: -extension  
    -->
  <setting type="BuilderPackageExtension" name="builder1" 
           description="File extension. Regex '.*' is default" 
           settingValue="string"/>
  <!-- 
    ** defaults option: -packagetype  
    -->
  <setting type="BuilderPackageType" name="builder1" 
           description="An existing Package subtype module. Package is default" 
           settingValue="type"/>
  <!-- 
    ** defaults option: -min  
    -->
  <setting type="BuilderImportMin" name="builder1" 
           description="Minimum packages to import. 1 is default" 
           settingValue="integer"/>
  <!-- 
    ** defaults option: -max  
    -->
  <setting type="BuilderImportMax" name="builder1" 
           description="Maximum packages to import. 1 is default" 
           settingValue="integer"/>
  <!-- 
    ** defaults option: -installroot  
    -->
  <setting type="BuilderPackageInstallroot" name="builder1" 
           description="Installation root directory" 
           settingValue="path"/>
  <!-- 
    ** defaults option: -filebase  
    -->
  <setting type="BuilderPackageFilebase" name="builder1" 
           description="The file base name. Regex '.*?' is default" 
           settingValue="string"/>

  <!--
      **
      ** Defines Builder resource and its settings
      **
  -->
  <deployment 
      type="Builder"
      name="builder1" 
      description="A builder object." 
      installRoot="target_directory" 
      basedir="source_directory">

   <resources>
      <resource type="BuilderPackageSeparator"   name="builder1"/>
      <resource type="BuilderPackageExtension"   name="builder1"/>
      <resource type="BuilderPackageType"        name="builder1"/>
      <resource type="BuilderImportMin"          name="builder1"/>
      <resource type="BuilderImportMax"          name="builder1"/>
      <resource type="BuilderPackageInstallroot" name="builder1"/>
      <resource type="BuilderPackageFilebase"    name="builder1"/>
   </resources>
   <referrers>
      <resource type="Node" name="localhost"/>
   </referrers>

  </deployment>
</project>

To load this XML data use the load-objects command:

ctl -m ProjectBuilder -c load-objects -- -filename builder-metadata.xml

To then deploy the builder object the first time to the localhost node, be sure to run the ctl-depot command:

ctl-depot -p default -a install

The output will show the new object being installed:

        "Install" command running for object: (Builder) moduleA

The subsequent sections provide examples for using repoImport to load your build artifacts into the repository.

Import single file with a buildstamp

This section describes how to import a single file of the given buildstamp. The examples below assume a buildstamp convention representing a date/time stamp.

Directory layout

The command examples that follow correspond to a file directory layout described here:

top_dir
  |
  |--- afile.xml          // a file
  |
  +--- src/               // source directory
  |
  `--- target/            // contains build artifacts
        |
        `--- moduleA      // moduleA artifacts
	       |
	       `-- moduleA-20080630.war   // file to be imported

The top_dir represents some abitrary base directory where various files and subdirectories exist. The "target" subdirectory contains "moduleA" which in turn contains a build artifact that is to be uploaded to the repository.

Command usage

To import the "moduleA-20080630.war" file run repoImport like so:

ctl -m Builder -c repoImport -- -targetdir top_dir/target \
    -filebase moduleA -buildstamp 20080630 -extension war  \
    -type war -installroot /path/to/install/dir

After the repoImport command completes the file will have been uploaded to the repository and a new object defined with the specified metadata.

After the package has been registered you can query the repository and now see it listed:

	$ ctl -p default -m Builder -c repoFind -- -packagetype war
	|
	|--(war) moduleA-20080630.war

Builder configuration

To default the repoImport options to those shown above define a Builder like so:

File listing: moduleA-Builder.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE project PUBLIC "-//ControlTier Software Inc.//DTD Project Document 1.0//EN" 
    "project.dtd">
<project>
  <!--
      **
      ** Defines Builder repoImport defaulting settings
      **
  -->

  <!-- 
    ** defaults option: -extension  
    -->
  <setting type="BuilderPackageExtension" name="moduleA" 
           description="File extension" 
           settingValue="war"/>
  <!-- 
    ** defaults option: -packagetype  
    -->
  <setting type="BuilderPackageType" name="moduleA" 
           description="Package control module" 
           settingValue="war"/>
  <!-- 
    ** defaults option: -installroot  
    -->
  <setting type="BuilderPackageInstallroot" name="moduleA" 
           description="Installation root directory" 
           settingValue="/path/to/installroot"/>
  <!-- 
    ** defaults option: -filebase  
    -->
  <setting type="BuilderPackageFilebase" name="moduleA" 
           description="The file base name" 
           settingValue="moduleA"/>

  <!--
      **
      ** Defines Builder resource and its settings
      **
  -->
  <deployment 
      type="Builder"
      name="moduleA" 
      description="A builder object." 
      installRoot="top_dir/target" 
      basedir="top_dir">

   <resources>
      <resource type="BuilderPackageExtension"   name="moduleA"/>
      <resource type="BuilderPackageType"        name="moduleA"/>
      <resource type="BuilderPackageInstallroot" name="moduleA"/>
      <resource type="BuilderPackageFilebase"    name="moduleA"/>
   </resources>
   <referrers>
      <resource type="Node" name="localhost"/>
   </referrers>

  </deployment>
</project>

To load this XML data use the load-objects command:

ctl -m ProjectBuilder -c load-objects -- -filename moduleA-Builder.xml

Command usage

After the moduleA Builder is defined, the command line options will have your defaults. Now just run repoImport with buildstamp:

ctl -t Builder -o moduleA -c repoImport -- -buildstamp 20080630

Import single file without a buildstamp

Not everyone uses a buildstamp convention. This example shows how to upload a file that does not contain a buildstamp in the filename.

Directory layout

top_dir
  |
  +--- src/               // source directory
  |
  `--- target/            // contains build artifacts
        |
        `--- moduleA      // war artifacts
	       |
	       `-- moduleA.war  // file to import

You can see the "moduleA.war" is the file to be imported.

Command usage

Import filename matching "moduleA.war" in directory top_dir/target:

ctl -m Builder -c repoImport -- -targetdir top_dir/target \
    -filebase moduleA -requirebuildstamp false -extension war  \
    -type war -installroot /path/to/install/dir

Note, the -requirebuildstamp option was specified to repoImport to not expect a buildstamp in the filename. This changes the file matching pattern to not expect a "${separator}${buildstamp}" component in the file name.

Builder configuration

To default the repoImport options to those shown above define a Builder like so:

File listing: moduleA-nobuildstamp.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE project PUBLIC "-//ControlTier Software Inc.//DTD Project Document 1.0//EN" 
    "project.dtd">
<project>
  <!--
      **
      ** Defines Builder repoImport defaulting settings
      **
  -->

  <!-- 
    ** defaults option: -extension  
    -->
  <setting type="BuilderPackageExtension" name="moduleA" 
           description="File extension" 
           settingValue="war"/>
  <!-- 
    ** defaults option: -packagetype  
    -->
  <setting type="BuilderPackageType" name="moduleA" 
           description="Package control module" 
           settingValue="war"/>
  <!-- 
    ** defaults option: -installroot  
    -->
  <setting type="BuilderPackageInstallroot" name="moduleA" 
           description="Installation root directory" 
           settingValue="/path/to/installroot"/>
  <!-- 
    ** defaults option: -filebase  
    -->
  <setting type="BuilderPackageFilebase" name="moduleA" 
           description="The file base name" 
           settingValue="moduleA"/>
  <!-- 
    ** defaults option: -requirebuildstamp  
    -->
  <setting type="BuilderPackageRequireBuildstamp" name="moduleA" 
           description="Should package require a buildstamp in the filename" 
           settingValue="false"/>

  <!--
      **
      ** Defines Builder resource and its settings
      **
  -->
  <deployment 
      type="Builder"
      name="moduleA" 
      description="A builder object." 
      installRoot="top_dir/target" 
      basedir="top_dir">

   <resources>
      <resource type="BuilderPackageExtension"   name="moduleA"/>
      <resource type="BuilderPackageType"        name="moduleA"/>
      <resource type="BuilderPackageInstallroot" name="moduleA"/>
      <resource type="BuilderPackageFilebase"    name="moduleA"/>
      <resource type="BuilderPackageRequireBuildstamp" name="moduleA"/>
   </resources>
   <referrers>
      <resource type="Node" name="localhost"/>
   </referrers>

  </deployment>
</project>

To load this XML data use the load-objects command:

ctl -m ProjectBuilder -c load-objects -- -filename moduleA-nobuildstamp.xml

Command usage

After the moduleA-nobuildstamp Builder is defined, the command line options will have your defaults. Now just run repoImport with no args:

ctl -t Builder -o moduleA -c repoImport

Import multiple files of the same type and buildstamp

Some builds generate more than one artifact of the same type. This section describes how to import multiple package files that share the same buildstamp.

Directory layout

top_dir
  |
  |--- afile.xml            
  |
  +--- src/                 // source directory
  |
  `--- target/              // contains build artifacts
        |
        |--- moduleA        // moduleA artifacts
	|      |
	|      `-- moduleA-20080630.war   // file to import
	|
        `--- moduleB         // moduleB artifacts
	       |
	       `-- moduleB-20080630.war   // file to import

The top_dir represents some abitrary base directory where various files and subdirectories exist. The "target" subdirectory contains two subdirectories of its own - moduleA and moduleB - each of which contain a build artifact.

Command usage

Specify the -filebase argument with a regex, "module.*", and a -max argument to import more packages than the default max of 1:

ctl -m Builder -c repoImport -- -targetdir top_dir/target \
    -filebase "module.*" -buildstamp 20080630 -extension war  \
    -type war -installroot /path/to/install/dir -max 2

The regex looks for filebase patterns that begin with "module" and zero or more other characters. Other regex that would have worked:

module(A|B)
module[AB]

Specify regexes that you believe are sufficiently open or constricting as your needs dictate.

Builder configuration

To default the repoImport options to those shown above define a Builder like so:

File listing: moduleAandB-Builder.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE project PUBLIC "-//ControlTier Software Inc.//DTD Project Document 1.0//EN" 
    "project.dtd">
<project>
  <!--
      **
      ** Defines Builder settings
      **
  -->
  <setting type="BuilderPackageExtension" name="moduleAandB" 
           description="File extension" 
           settingValue="war"/>
  <setting type="BuilderPackageType" name="moduleAandB" 
           description="Package control module" 
           settingValue="war"/>
  <setting type="BuilderImportMin" name="moduleAandB" 
           description="Minimum packages to import" 
           settingValue="1" settingType="script"/>
  <setting type="BuilderImportMax" name="moduleAandB" 
           description="Maximum packages to import" 
           settingValue="2"/>
  <setting type="BuilderPackageInstallroot" name="moduleAandB" 
           description="Installation root directory" 
           settingValue="/path/to/installroot"/>
  <setting type="BuilderPackageFilebase" name="moduleAandB" 
           description="The file base name" 
           settingValue="module.*"/>
  <!--
      **
      ** Defines Builder resource
      **
  -->
  <deployment 
      type="Builder"
      name="moduleAandB" 
      description="A builder object." 
      installRoot="top_dir/target" 
      basedir="top_dir/src">
  <!--
      **
      ** Reference the settings
      **
   -->
   <resources>
      <resource type="BuilderPackageExtension" name="moduleAandB"/>
      <resource type="BuilderPackageType" name="moduleAandB"/>
      <resource type="BuilderImportMin" name="moduleAandB"/>
      <resource type="BuilderImportMax" name="moduleAandB"/>
      <resource type="BuilderPackageInstallroot" name="moduleAandB"/>
      <resource type="BuilderPackageFilebase" name="moduleAandB"/>
   </resources>
   <referrers>
      <resource type="Node" name="localhost"/>
   </referrers>

  </deployment>
</project>

To load this XML data use the load-objects command:

ctl -m ProjectBuilder -c load-objects -- -filename moduleAandB-Builder.xml

To then deploy the builder object the first time to the localhost node, be sure to run the ctl-depot command:

ctl-depot -p default -a install

The output will show all the objects being installed:

        "Install" command running for object: (Builder) moduleA
        "Install" command running for object: (Builder) moduleAandB

Command usage

After the "moduleAandB" Builder is defined you just need to specify the buildstamp:

ctl -t Builder -o moduleAandB -c repoImport -- -buildstamp 20080630

Import multiple files with different package types

Some builds produce more than one artifact. Such a Builder might control a top level buildscript that kicks off other buildscripts each responsible for a different type of artifact. When this is the case, a single invocation of repoImport cannot be specified in such a way to both identify all the files to be imported as well as include their individualized metadata needed during the import process.

The repoImport command includes a strategy to support this use case but depends on an external configuration file that will contain the metadata and identify the directory as one that contains a build artifact.

Directory layout

top_dir
  |
  +--- src/                 // source directory
  |     |
  |     |--- moduleA        // moduleA source
  |     |
  |     `--- moduleB        // moduleB source
  |
  `--- target/              // contains build artifacts
        |
        |--- moduleA        // moduleA artifacts
	|      |
	|      |-- builder.properties     // controls repoImport
	|      `-- moduleA-20080630.war   // file to import
	|
        `--- moduleB         // moduleB artifacts
	       |
	       |-- builder.properties     // controls repoImport
	       `-- moduleB-20080630.jar   // file to import

This directory structure shows there are two artifacts to be uploaded - moduleA-20080630.war and moduleB-20080630.jar. You'll also notice the "builder.properties" file inside each target directory. This file contains metadata the repoImport command will use to register the package during the import process.

Note

Who and how the builder.properties file is prepared, generated or copied to the target directories is up to the developer, build engineer or whomever is responsible for importing the packages.

The repoImport properties file: "builder.properties"

The "builder.properties" file contains various bits of metadata about build artifacts. When the repoImport command finds a directory with a builder.properties file in it, and there is also matching file (based on the repoImport options), the filename is tokenized and the information is used to expand tokens in the property file.

The table below lists these tokens:

token	description
pFilename	matching package file name as matched by repoImport
pBuildtime	package buildstamp. tokenized from filename
pExtension	package file extension. tokenized from filename
pBase	package base name. tokenized from filename
pDepot	package depot name. tokenized from filename

The repoImport command then expands the properties file replacing tokens with their bound values and then loads the properties file. The expanded properties file should now have all the needed package metadata to register the file as a Package object.

Here is an example builder.properties file. Notice the tokens by their surrounding "@" characters:

# builder.properties template
entity.name=@pFilename@
entity.description=A description of Packagetype package file
entity.classname=Packagetype
entity.order=Package
entity.depot=@pDepot@
entity.package-arch = noarch
entity.package-buildtime = @pBuildtime@
entity.package-vendor = 
entity.package-base = package_base
entity.package-filename= @pFilename@
entity.package-filetype = @pExtension@
entity.package-release =
entity.package-version = @pBuildtime@
entity.package-restart = false
entity.package-install-root = /path/to/installroot
entity.package-repo-url=${framework.pkgRepo.url}/@pDepot@/Packagetype/@pExtension@s/@pFilename@

You can also see example import templates by going to Workbench, navigate to a Package subtype and click the "View Package Import Template" link.

Command usage

After the builder.properties files have been prepared and copied to the target directories, you can run repoImport like so, specifying a -max 2 option to import multiple files, and a -extension "(war|jar)" option to match only .war or .jar files. (Be sure to quote your regex if it interferes with your shell.):

ctl -m Builder -c repoImport -- -targetdir top_dir/target -buildstamp 20080630 -max 2 -extension "(war|jar)"

Builder configuration

The builder configuration is now simpler. Fewer settings are required as all the package metadata is being provided by the builder.properties file. Only settings to specify matching logic are required.

File listing: bulkImporter-Builder.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE project PUBLIC "-//ControlTier Software Inc.//DTD Project Document 1.0//EN" 
    "project.dtd">
<project>
  <!--
      **
      ** Defines Builder resource
      **
  -->
    <setting type="BuilderImportMax" name="bulkImporter" 
           description="Maximum packages to import" 
           settingValue="2"/>
    <setting type="BuilderPackageExtension" name="bulkImporter" 
           description="File extension" 
           settingValue="(war|jar)"/>


  <deployment 
      type="Builder"
      name="bulkImporter" 
      description="A builder that depends on builder.properties for bulk importing." 
      installRoot="top_dir/target" 
      basedir="top_dir/src">
      <resources>
        <resource type="BuilderImportMax" name="bulkImporter"/>
        <resource type="BuilderPackageExtension" name="bulkImporter"/>
      </resources>
      <referrers>
         <resource type="Node" name="localhost"/>
      </referrers>
  </deployment>
</project>

To load this XML data use the load-objects command:

ctl -m ProjectBuilder -c load-objects -- -filename bulkImporter-Builder.xml

Command usage

After the Builder is configured run the repoImport command like so:

ctl -t Builder -o bulkImporter -c repoImport -- -buildstamp 20080630

Possible anti-pattern

When one finds they are forced to use "builder.properties" files to drive the file import process, chances are they are overloading a Builder to manage multiple builds.

A preferred solution is to define "import only" Builders, one for each type of Package to be imported.