Quantcast
Channel: How-To’s : TeamCity | The JetBrains Blog
Viewing all 103 articles
Browse latest View live

Kotlin Configuration Scripts: Creating Configuration Scripts Dynamically

$
0
0

This is part three of the five-part series on working with Kotlin to create Configuration Scripts for TeamCity.

  1. An Introduction to Configuration Scripts
  2. Working with Configuration Scripts
  3. Creating Configuration Scripts dynamically
  4. Extending the TeamCity DSL
  5. Testing Configuration Scripts

We saw in the previous post how we could leverage some of Kotlin’s language features to reuse code. In this part, we’re going to take advantage of the fact that we are in fact dealing with a full programming language and not just a limited DSL, to create a dynamic build configuration.

Build Configurations based on parameters

The scenario is the following: we have an HTTP server that we need to test on different platforms with a range of concurrent connections on each one. This generates potentially a lot of different build configurations that we’d need to create and maintain.

Instead of doing this manually, what we can do is write some code to have our Kotlin DSL Configuration Script generate all the different build configurations for us.

Let’s say that we have a list of Operating Systems and a range of concurrent connections for each one. The first thing to do is to create a data class that represents this information

data class BuildParameters(val name: String, val operatingSystem: String, val connections: Int)

which in essence is like a regular class but provides a series of benefits such as a nicer string representation, equality and copy operations amongst other things.

Now let’s imagine we have the different platforms we want to test on represented as a list of BuildParameters

val operatingSystems = listOf(
            BuildParameters("Windows Build", "windows", 30),
            BuildParameters("MacOS Build", "osx", 20),
            BuildParameters("Linux Build", "ubuntu", 20)
    )

what we’d like to do, is iterate over this list and create a new build type for each combination. In essence, if our standard Project is

object Project : Project({
    uuid = "9179636a-39a3-4c2c-aa7e-6e2ea7cfbc5b"
    extId = "Wasabi"
    parentId = "_Root"
    name = "Wasabi"
    vcsRoot(Wasabi_HomeGitWasabiGitRefsHeadsMaster)
    buildType(WasabiBuild)
})

what we want to do is create a new buildType for each entry in the list

object Project : Project({
    uuid = "9179636a-39a3-4c2c-aa7e-6e2ea7cfbc5b"
    extId = "Wasabi"
    parentId = "_Root"
    name = "Wasabi"

    vcsRoot(Wasabi_HomeGitWasabiGitRefsHeadsMaster)

    val operatingSystems = listOf(
            BuildParameters("Windows Build", "windows", 30),
            BuildParameters("MacOS Build", "osx", 20),
            BuildParameters("Linux Build", "ubuntu", 20)
    )

    operatingSystems.forEach {
        buildType(OperatingSystemBuildType(it))
    }
})

Creating a base Build Type

Any parameter passed into buildType needs to be of the type BuildType. This means we’d need to inherit from this class and at the same time provide some parameters to it (BuildParameters)

class OperatingSystemBuildType(buildParameters: BuildParameters) : BuildType() {
    init {
        val paramToId = buildParameters.name.toExtId()
        uuid = "53f3d94a-20c6-43a2-b056-baa19e55fd40-$paramToId"
        extId = "BuildType$paramToId"
        name = "Build for ${buildParameters.name}"

        vcs {
            root(Wasabi.vcsRoots.Wasabi_HomeGitWasabiGitRefsHeadsMaster)

        }
        steps {

            gradle {
                tasks = "clean build"
                useGradleWrapper = true
                gradleWrapperPath = ""
                gradleParams = "-Dconnections=${buildParameters.connections}"
            }
        }
        triggers {
            vcs {
            }
        }

        requirements {
            contains("teamcity.agent.jvm.os.name", buildParameters.operatingSystem)
        }
    }

}

The code is actually a pretty standard Configuration Script that defines a Gradle build step, has a VCS definition and defines a VCS trigger. What we’ve done is just enhance it somewhat.

To begin with,  we’re using the BuildParameters name property to suffix it to the uuid, extId and name. This guarantees a unique ID per build configuration as well as provides us a name to identify it easily. To make sure the values are properly formatted, we use the TeamCity DSL defined extension function to String, named toExtId() which takes care of removing any forbidden characters.

In the steps definition, we pass in certain parameters to Gradle, which in our case is the number of concurrent connections we want to test with. Obviously, this is just a sample, and the actual data being passed in can be anything and used anywhere in the script.

Finally, we also use the BuildParameters operatingSystem property to define Agent requirements.

Summary

The above is just a sample of what can be done when creating dynamic build scripts. In this case, we created multiple build configurations, but we could just as well have created multiple steps, certain VCS triggers, or whatever else could come in useful. The important thing to understand is that at the end of the day, Kotlin Configuration Script isn’t just merely a DSL but a fully fledged programming language.

In the next part of this series, we’ll see how we can extend the Kotlin Configuration Scripts (hint – it’s going to involve extension functions).

 

 


Kotlin Configuration Scripts: Extending the TeamCity DSL

$
0
0

This is part four of the five-part series on working with Kotlin to create Configuration Scripts for TeamCity.

  1. An Introduction to Configuration Scripts
  2. Working with Configuration Scripts
  3. Creating Configuration Scripts dynamically
  4. Extending the TeamCity DSL
  5. Testing Configuration Scripts

TeamCity allows us to create build configurations that are dependent on each other, with the dependency being either snapshots or artifacts. The configuration for defining dependencies is done at the build configuration level. For instance, assuming that we have a build type Publish that has a snapshot dependency on Prepare Artifacts, we would define this in the build type Publish in the following way

class Publish : BuildType({
    uuid = "53f3d94a-20c6-43a2-b056-baa19e55fd40"
    extId = "Test"
    name = "Build"
    vcs {
        . . .
    }
    steps {
        . . .
    }
    dependencies {
        dependency(Prepare_Artifacts) {
            snapshot {
                onDependencyFailure = FailureAction.FAIL_TO_START
            }
        }
    }
})

and in turn if Prepare Artifacts had dependencies on previous build configurations, we’d define these in the dependencies segment of its build configuration.

TeamCity then allows us to visually see this using the Build Chains tab in the user interface

Build Chains

Defining the pipeline in code

Pipeline is a sequence of phases, phase is a set of buildTypes, each buildType in a phase depends on all buildTypes from the previous phase. This can handle simple but common kind of build chains where some build produces an artifact, several builds test it in parallel and the final build deploys the result if all its dependencies are successful.

It would often be beneficial to be able to define this build chain (or build pipeline) in code, so that we could describe what’s going on from a single location. In essence, it would be nice to be able to define the above using the Kotlin DSL as so

object Project : Project ({
   . . .
   pipeline {
        phase("Compile all clients") {
            + HttpClient
            + FluentHc
            + HttpClientCache
            + HttpMime
        }
        phase("Prepare artifacts") {
            + PrepareArtifacts
        }
        phase("Deploy and publish to CDN") {
            + Publish
        }
   }
})

Defining this at the Project.kt level in our DSL, would give us a good oversight of what’s taking place.

The issue is though that currently the TeamCity DSL does not provide this functionality. But that’s where Kotlin’s extensibility proves quite valuable as we’ll see.

Creating our own Pipeline definition

Kotlin allows us to create extension functions and properties, which are a means to extend a specific type with new functionality, without having to inherit from these. When passing extension functions as arguments to other functions (i.e. higher-order functions), we get what we call in Kotlin Lambdas with Receivers, something we saw in this series already when Generalising feature wrappers in the second part of this series. We apply the same concept here to create our pipeline DSL

class Pipeline {
    val phases = arrayListOf<Phase>()

    fun phase(description: String = "", init: Phase.() -> Unit = {}) {
        val newPhase = Phase()
        newPhase.init()
        phases.lastOrNull()?.let { prevPhase ->
            newPhase.buildTypes.forEach {
                it.dependencies {
                    for (dependency in prevPhase.buildTypes) {
                        snapshot(dependency)
                    }
                }
            }
        }
        phases.add(newPhase)
    }
}

class Phase {
    val buildTypes = hashSetOf<BuildType>()

    operator fun BuildType.unaryPlus() {
        buildTypes.add(this)
    }
}

fun Project.pipeline(init: Pipeline.() -> Unit = {}) {
    val pipeline = Pipeline()
    pipeline.init()
    //register all builds in pipeline
    pipeline.phases.forEach { phase ->
        phase.buildTypes.forEach { bt ->
            this.buildType(bt)
        }
    }
}

What the code above is doing is define a series of new constructs, namely pipeline and phase.

The code then takes the contents of what’s passed into each of these and defines, under the covers, the dependencies. In essence, it’s doing the same thing we would do in each of the different build configurations, but from a global perspective defined at the Project level.

In order to pass in the configuration to the pipeline as we saw earlier, we merely reference the specific build type (HttpClient, Publish, etc.), assigning it to a variable

val HttpClient = BuildType(....)

Flexibility of defining our own pipeline constructs

It’s important to understand that this is just one of many ways in which we can define pipelines. We’ve used the terms pipeline and phase. We could just as well have used the term stage to refer to each phase, or buildchain to refer to the pipeline itself (and thus align it with the UI). In addition to how we’ve named the constructs, and more importantly, is how the definition is actually constructed. In our case, we were interested in having this present in the Project itself. But we could just as well define a different syntax that is used at the build type level. The ability to easily extend the TeamCity DSL with our own constructs, provides us with this flexibility.

 

 

 

TeamCity as Debian Package Repository

$
0
0

Recently we’ve been experimenting around using TeamCity as a Debian repository and we’d like to share some tips and tricks we’ve come up with in the process.

We used the TeamCity tcDebRepository plugin to build *.deb packages and serve package updates to Debian/Ubuntu servers. The plugin, the special prize winner of the 2016 TeamCity plugin contest, works like a charm, but we’ve encountered a few difficulties with the software used to build and package .deb files, hence this blog post.

It’s important to note that tcDebRepository is not capable of serving packages to recent Ubuntu versions due to the lack of package signing. This is planned to be resolved in version 1.1 of the plugin (the plugin compatibility details).

We do not intend to cover the basics of TeamCity and Debian GNU/Linux infrastructure. To get acquainted with TeamCity, this video might be helpful. Building Debian packages is concisely described in the Debian New Maintainers’ Guide.

Everything below applies to any other Debian GNU/Linux distribution, e.g. Astra Linux.

Prerequisites and Project Setup

We arbitrarily chose 4 packages to experiment with:

Our setup uses:

  1. The free Professional version of TeamCity 10 with 3 agents managed by Debian 8.0 (Jessie).
  2. The tcDebRepository plugin installed as usual.
  3. Besides the Teamcity agent software, each of the agents required installing:
    • the build-essential package as a common prerequisite
    • the dependencies (from the 01-build-depends build-depends and 02-build-depends-indep build-depends-indep categories) required for individual packages.

Next we created a project in TeamCity with four build configurations (one per package):
teamcity-build-configs

Configuring VCS Roots

When configuring VCS Roots in TeamCity, types of Debian GNU/Linux packages had to be taken into account.

Debian packages can be native and non-native (details).
The code of native packages (e.g. debhelper, dpkg), developed within the Debian project, contains all the meta-information required for building (the debian/ directory in the source code tree).
The source code of non-native packages (e.g. bash) is not related to Debian, and an additional tree of their source code and meta-information with patches (contained in the debian/ directory) has to be maintained.

Therefore, for native packages we configured one VCS Root (pointing to Debian); whereas non-native packages needed two roots.

Next we configured checkout rules: the difference from the usual workflow is that in our case the artifacts (Debian packages) will be built outside the source code, one directory higher. This means that the checkout rules in TeamCity should be configured so that the source code of the dpkg package is checked out not into the current working directory, but into a subdirectory with the same name as the package, i.e. pkgname/. This is done by adding the following line to the checkout rules:
+:.=>pkgname
Finally, a VCS trigger was added to to every configuration.

Artifact paths

In terms of TeamCity, the binary packages we are about to build are artifacts, specified using artifact paths in the General Settings of every build configuration:

03-teamcity-debian-general-artifact-paths

For native packages, the pkgname.orig.tar.{gz,bz2,xz,lzma} and pkgname.debian.tar.{gz,bz2,xz,lzma} will not be created.

Build Step1: Integration with Bazaar

Building a package with Bazaar-hosted source code (bash in our case) requires integration with the Bazaar Version Control System not supported by TeamCity out-of-the box. There is an a 3rd party TeamCity plugin to support Bazaar, but it has at least two problems:

  1. agent-side checkout is not supported
  2. the plugin is based on bzr xmlls and bzr xmllog (the bzr commands invoked by the plugin), but these commands are provided by external modules not necessarily available in the default Bazaar installation

The TeamCity server we used is running Windows, so we decided not to opt for installing Bazaar on the server side. Instead, to build the bash package, we added a build step using the Command Line Runner and a shell script for Bazaar integration:

#!/bin/bash

export LANG=C
export LC_ALL=C

set -e

rm -rf bash/debian
bzr branch http://bazaar.launchpad.net/~doko/+junk/pkg-bash-debian bash/debian
major_minor=$(head -n1 bash/debian/changelog | awk '{print $2}' | tr -d '[()]' | cut -d- -f1)
echo "Package version from debian/changelog: ${major_minor}"
tar_archive=bash_${major_minor}.orig.tar
rm -f ${tar_archive} ${tar_archive}.bz2
# +:.=>bash checkout rule should be set for the main VCS root in TeamCity
tar cf ${tar_archive} bash
tar --delete -f ${tar_archive} bash/debian
# Required by dpkg-buildpackage
bzip2 -9 ${tar_archive}

This approach will not allow us to see the changes in one of the two trees of the source codes and run build automatically on changes, but it is ok as the first try.

Build Step 2. Initial configuration

We used the Command Line Runner to invoke dpkg-buildpackage. The -uc and -us keys in the image below indicate that digital signatures are currently out of scope for our packages. If you wish to sign the packages, you’ll have to load the corresponding pair of GnuPG keys on each of the agents.

Note that the “Working directory” is not set to the current working directory, but to the subdirectory with the same name as the package (bash in this case) where the source code tree will be checked out and dpkg-buildpackage will be executed. If the VCS Root is configured, the “Working directory” field can be filled out using the TeamCity directory picker, without entering the directory name manually:

07-teamcity-debian-dpkg-buildpackage-step

After the first build was run, we encountered some problems.

Build problems resolution

Code quality

For some packages (bash in our case), two code trees were not synchronized, i.e. they corresponded to slightly different minor versions, which forced us to use tags rather than branches from the main code tree. Luckily, TeamCity allows building on a tag and the “Enable to use tags in the branch specification” option can be set in such case:

teamcity-vcs-tags-in-branch-specs-small

Dependencies

Here is another problem we faced. When running the first build, dpkg-buildpackage exits with code 3 even though all the required dependencies were installed (see the prereqs section above).

There are a few related nuances:

  • The most common case for continuous integration is building software from Debian Unstable or Debian Experimental. So, to meet all the dependencies required for the build, TeamCity agents themselves need to run under Debian Unstable or Debian Experimental (otherwise dpkg-buildpackage will be reporting that your stable dependencies versions are too old). To resolve the error, we added the -d parameter:
    dpkg-buildpackage -uc -us -d
  • A special case of old dependencies is the configure script, created by the GNU Autotools that are newer than those currently installed on your system. dpkg-buildpackage is unable to detect this, so if the build log shows messages about the missing m4 macros, you need to to re-generate the configure script using the of GNU Autotools currently installed on the agent. This was done by adding the following command as the build step executed before dpkg-buildpackage:
    autoreconf -i

Broken unit tests

When working with Debian Unstable or Debian Experimental, unit test may fail (as it happened in our case). We choose to ignore the failing test and still build the package by running dpkg-buildpackage in a modified environment:
DEB_BUILD_OPTIONS=nocheck dpkg-buildpackage -uc -us

Other build profiles are described here.

Final Stage

After tuning all of the build configurations, our builds finally yielded expected artifacts (the dpkg package is used as an example):
11-teamcity-debian-artifacts

If you do not want each incremental package version to increase the number of artifacts of a single build (giving you dpkg_1.18.16_i386.deb, dpkg_1.18.17_i386.deb, etc.), the working directory should be selectively cleaned up before starting a new build. This was done by adding the TeamCity Swabra build feature to our build configurations.

Tuning up Debian repository

The tcDebRepository plugin adds the “Debian Repositories” tab in the project settings screen, which allows you to create a Debian repository. Once the repository is created, artifact filters need to be configured: at the moment each filter is added manually. Filters can be edited, deleted or copied.
teamcity-adding-an-artifact-filter

The existing artifacts will not be indexed, therefore after the configuration of the Debian repository, at least one build must be run in every build configuration. Then we can view the available indexed packages:
16-teamcity-debian-debrepo-page-2

When adding the repository to the /etc/apt/sources.list, all these packages are visible on the client side. Note that the packages are not digitally signed, which prevents packages from being available on recent versions of Ubuntu. Package signing is planned for tcDebRepository version 1.1.

17-teamcity-debian-aptitude-1

(!) If you are compiling for several architectures (i386, x32, amd64, arm*), it is worth having several build configurations corresponding to the same package with different agent requirements, or, in addition to the VCS Trigger, using the Schedule Trigger with the “Trigger build on all enabled and compatible agents” option enabled.

20-teamcity-debian-change-log

Happy building!

* We’d like to thank netwolfuk , the author of the tcDebRepository  plugin, for contributing to this publication.
This article, based on the original post in Russian, was translated by Julia Alexandrova.

Kotlin Configuration Scripts: Testing Configuration Scripts

$
0
0

This is part five of the five-part series on working with Kotlin to create Configuration Scripts for TeamCity.

  1. An Introduction to Configuration Scripts
  2. Working with Configuration Scripts
  3. Creating Configuration Scripts dynamically
  4. Extending the TeamCity DSL
  5. Testing Configuration Scripts

In this last part of the series we’re going to cover one aspect that using Kotlin Configurations Scripts enables, which is testing.

Given that the script is a programming language, we can simply add a dependency to a testing framework of our choice, set a few parameters and start writing tests for different aspects of our builds.

In our case, we’re going to use JUnit. For this, we add the JUnit dependency to the pom.xml file

<dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.12</version>
</dependency>

We also need to define the test directory

<sourceDirectory>Wasabi</sourceDirectory>
<testSourceDirectory>tests</testSourceDirectory>

In our case, this corresponds to the following directory layout

Directory Structure

Once we have this in place, we can write unit tests like in any other Kotlin or Java project, accessing the different components of our project, build types, etc. In our case we’re going to write a simple test that checks to see all build types start with a clean checkout

@Test
fun buildsHaveCleanCheckOut() {
    val project = Project
    project.buildTypes.forEach { bt ->
        assertTrue(bt.vcs.cleanCheckout ?: false, "BuildType '${bt.extId}' doesn't use clean checkout")
    }
}

 

That additional layer of certainty

When we make changes to the Kotlin Configuration Script and check it in to source control, TeamCity synchronises the changes and it will report of an errors it encounters, which usually are related to compilation. The ability to now add tests allow us to add another extra layer of checks to make sure that beyond our build script containing any scripting errors, certain things are validated such as the correct VCS checkout as we’ve seen above, whether the appropriate number of build steps are being defined, etc. The possibilities are essentially endless given that once again, we’re just using a regular programming language.

What’s New in TeamCity 2017.1, May 3rd Webinar

$
0
0

Join us Wednesday, May 3rd, 17:00 – 18:00 CEST (11:00 AM – 12:00 PM EDT) for our free live webinar, What’s New in TeamCity 2017.1 with Wes Higbee.

join

In this webinar, Wes Higbee will go over some of the major features of the latest TeamCity release. He will demonstrate the configuration of agent cloud profiles, which was moved to the project level; new integration with Visual Studio Team Services; disabling builds in the default branch; and the new secure values storage setting.

He will also show how easy it is to upgrade from v10.0 to v2017.1 in Docker, and showcase the updated parts of the TeamCity UI.

Space is limited, please register now. There will be an opportunity to ask questions during the webinar.

About the presenter:

Wes McClureWes Higbee is passionate about helping companies achieve remarkable results with technology and software. He’s had extensive experience developing software and working with teams to improve how software is developed to meet business objectives. Wes launched Full City Tech to leverage his expertise to help companies rapidly deliver high quality software to delight customers. He has a strong background in using Continuous Integration with TeamCity to bring quality to the table.

Test .NET Core projects with TeamCity

$
0
0

Greetings, everyone!

As many of you already know, TeamCity has a plugin to build .Net Core projects, which is basically a wrapper for the dotnet command. But if you try running dotnet test, you won’t see test results in TeamCity at the moment. As the test time can be significant, it is preferable to have fast feedback, so in this post we’ll explain how to integrate the dotnet test command with TeamCity and configure on-the-fly test reporting.

In short, you’ll need to perform the following:

  1. To run tests locally using the dotnet test command,  add references to the following packages to your test project: to Microsoft.NET.Test.Sdk, to the test framework, and the test adapter.
  2. To configure on-the-fly reporting, add a package reference to the TeamCity VSTest Adapter.
  3. To configure a build in TeamCity, use the TeamCity plugin for .NET Core projects or some other build runner.

Let’s go over these points in detail.

Run tests locally using dotnet test command

The approach suggested by Microsoft works fine for any target framework as well as for multiple frameworks at the same time, provided the test engine has a test adapter, e.g. MS tests, xunit tests, or some other test engine, for example.

That means that a test project needs to have at least three additional package references:

  • Microsoft.NET.Test.Sdk, which contains the MSBuild targets and properties for test projects and marks a project as a test project.
  • The selected Test Framework, e.g.  MSTest, or XUnit, or other
  • Test Adapter to discover and execute test framework based tests, e.g. the MSTest adapter, or the XUnit adapter, or other.

Configure on-the-fly reporting in TeamCity

Ideally, you should not need to do any additional preparation steps if you choose to run dotnet tests on TeamCity: TeamCity should pick up these tests automatically. For now this is not the case, so here is how you still can achieve this goal: use the TeamCity.VSTest.TestAdapter NuGet package containing a special logger which integrates with the test platform and sends service messages to a TeamCity server.

To turn on the integration, you’ll have to add a package reference to TeamCity VSTest Adapter. Note that this package does not impact the tests run locally, but only those run under TeamCity.

Here is an example of a project:

Visual Studio NuGet Dependencies

The project file for an MS test project can look as follows:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFrameworks>net45;netcoreapp1.0</TargetFrameworks>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.0.0" />
    <PackageReference Include="MSTest.TestAdapter" Version="1.1.14" />
    <PackageReference Include="MSTest.TestFramework" Version="1.1.14" />
    <PackageReference Include="TeamCity.VSTest.TestAdapter" Version="1.0.0-rc" />
  </ItemGroup>

  <ItemGroup>
    <Service Include="{82a7f48d-3b50-4b1e-b82e-3ada8210c358}" />
  </ItemGroup>

</Project>

Configure build in TeamCity

To configure a build project in TeamCity, we used the TeamCity plugin for .NET Core projects, described here in detail.

You can see an example on the public TeamCity server, where a test project contains one class, UnitTests:

namespace MS.Tests
{
    using System;
    using Microsoft.VisualStudio.TestTools.UnitTesting;

    [TestClass]
    public class UnitTests
    {
        [TestMethod, Ignore]
        public void TestIgnored()
        {
        }

        [TestMethod]
        public void TestPassed()
        {
            Console.WriteLine("some text");
        }
    }
}

Here is an example of the TeamCity tests step:

.

After a build is run, TeamCity shows the test results:

This .Net project has two target platforms, net45 and netcoreapp1.0, that is why each test is run for both framework and you can see several results per test.

The proposed approach does require some extra effort, however, it is quite viable and useful for testing .Net Core projects. Currently we are working on simplifying running tests and configuring on-the-fly test reporting from TeamCity, which would not require modifying your project files, so stay tuned for the latest news.

As usual, feel free to share your feedback.

Happy building and testing!

TeamCity to Access Private GitHub Repositories Securely

$
0
0

Introduction

One of the great features TeamCity has is its integration with GitHub. Authentication from TeamCity to GitHub should be configured for the integration to work. We consider TeamCity secure enough; however, stealing authentication data is a threat we all live with and the consequences of it should be clearly understood.

In this blog post we’ll go over the existing ways of configuring authentication in TeamCity VCS roots pointing to private GitHub repositories, discussing the advantages and disadvantages of each of them. We hope that TeamCity Server administrators and Project administrators will find this information useful when deciding on the approach to authentication from TeamCity to GitHub.

Username and Password Authentication

It is often the case when access to a repository from TeamCity is configured via a user’s GitHub username/password. All server administrators and administrators of the projects where the root is configured have access to this information.

If this data is stolen, it gives the intruder full access and the possibility to maliciously modify your GitHub repositories, including your GitHub profile and all your settings.

We do not recommend it as it seems to be the least secure approach.

Username and Generated OAuth Token

Not long ago TeamCity started to support OAuth for GitHub. In this case you use the username and a generated OAuth token.

If someone gains access to them, then all your repositories in all organizations where you have read rights will be at this user’s disposal as TeamCity uses the ‘repo’ token scope.
However, this authentication option gives access to repositories only and not your GitHub profile and settings; besides, it is easy to revoke this token. Besides, repositories cannot not be deleted, and although force-push may still be performed, using the protected branches feature of GitHub can help in this case. All in all, this way can be considered an improvement in comparison with the previous approach.

SSH Key

As an administrator, you can create an SSH key for your TeamCity server, with the public part of the SSH key uploaded to GitHub and the private part uploaded to the TeamCity server.

The advantages of this approach are the same as those of the one above.
As to the risks, it can be highly dangerous if you use this key in different servers and applications. This can be mitigated using the special SSH key added to the GitHub profile and TeamCity, which will be used for this integration exclusively.

‘Deploy Key’ GitHub Feature

For every GitHub repository that Teamcity has access to it is possible to generate an SSH key with the private part on the TeamCity and the public part added to the deploy keys of your repository using the repository settings page. GitHub administrator rights for this repository are required. The key can have either read-only or read-write permissions for the repository.

This seems to be the most secure approach, because an individual key can be added to each repository, which would make access revocation extremely easy in case of data loss.

Creating a key for every repository might be a nuisance for Windows users, this seems to be the main disadvantage (not affecting Linux or macOS users though).

Conclusion

From the approaches outlined above, using OAuth token authentication and Deploy keys are considered secure enough by the majority of people, with Deploy keys being more secure and therefore recommended by us. In the end, adopting this or that way is up to you.

Run TeamCity CI builds in Google Cloud

$
0
0

teamcity-google-cloudWe are happy to announce that now TeamCity provides a tight integration with Google Cloud services. The Google Cloud Agents plugin allows using Google Compute Engine to start cloud instances on demand to scale the pool of cloud build agents and also supports using cost-efficient preemptible virtual machines. The Google Artifact Storage plugin provides the ability to use Google Cloud Storage to keep external build artifacts.

Installation

To enable these TeamCity integrations with the Google Cloud, you need to download the Google Cloud Agents and Google Artifact Storage from the plugins gallery and install them as additional TeamCity plugins. Remember to restart the TeamCity server.

Google Cloud Agents Configuration

Prepare Google Cloud Image

Now you need to create cloud images to start new cloud build agents from. To do that, create a new cloud instance from the available public boot disks and uncheck “Delete boot disk when instance is deleted” in the “Management, disk, networking, SSH keys” section. Then, after the start of the cloud instance, connect to it via SSH or RDP depending on the selected image and install a TeamCity build agent. You need to point the build agent to the TeamCity server to update plugins and set the agent to start automatically.

Create Google Cloud Instance

Then install required build tools, remove temporary files, remove the cloud instance and create a new custom image from the cloud instance boot disk.

Create Custom Google Image

Create Service Account and Key

Now we need to create a new service account with the Compute Engine Instance Admin role and a new JSON private key for that account.

google-cloud-credentials

Create Cloud Agent Profile

To create a Google agent cloud profile, navigate to the project where you want to set up the profile and select the Cloud Profiles link. Then click the “Create new profile” button and select “Google Compute” as the cloud type. Specify the profile name and the JSON private key value in the corresponding field.

To add a new image to the profile, press the “Add image” button, select the recently created cloud image, and fill in other properties. Next you need to save the image and then the profile settings. That’s all.

teamcity-google-profile

Google Artifact Storage Configuration

Create Service Account and Key

To access Google Cloud Storage, the plugin requires a service account with the Compute Engine Instance Admin role and a new JSON private key for that account.

teamcity-google-profile

Configure Artifact Storage

Navigate to the Artifacts Storage Project tab and click the Add new storage button. Select Google Storage type and fill in the name, JSON private key, bucket name and press Save. After that, activate this artifact storage in the TeamCity project by clicking the Make Active link.

teamcity-google-profile

That’s it! Your new builds in this project will store artifacts in the Google Cloud Storage.

Feedback

You are welcome to try these plugins for TeamCity, your feedback will be really appreciated: please share your thoughts in the comments to this blog post or in our issue tracker.

Happy building in the Google Cloud!


Build React Apps with TeamCity

$
0
0

Greetings, everyone!

In this blog post we will talk about building a React application with TeamCity using one of the newest TeamCity features –  Docker integration, introduced in  TeamCity 2017.2 EAP1. (At the moment the TeamCity-Docker support can run on Mac/Linux build agents).

We’ve created an example on GitHub based on Create React App and Storybook, and set up a demo project on the public TeamCity server with the VCS Root pointing to the project on GitHub.

We used the Command Line build step for our build configuration.

Command Line Build Step

We are going to execute the following script:

#!/bin/bash
set -e -x

yarn
yarn test-ci
yarn build
yarn build-storybook

Here yarn is used for installing dependencies and running scripts, but it is possible to use npm as well.

Our command line will run in a docker container – in TeamCity 2017.2 EAP the Command Line runner settings have the Docker settings section where you can specify the image for the container:

TeamCity will start the container and run this build step in it. node:latest in the Docker settings section stands for Docker’s official node image):
buildStep

Report tabs

test-ci in our script runs tests with the jest-teamcity-reporter plugin (see this commit).
In the line react-scripts test --env=jsdom --coverage --testResultsProcessor=jest-teamcity-reporter, the --coverage option turns on generating a code coverage report, which will be displayed in a dedicated build tab. TeamCity has the report tabs feature, and on our server the Root project – the parent of the demo project – has a predefined Code Coverage tab which expects the report to be at coverage.zip!index.html:
rootCodeCoverage

All we have to is specify the artifact path coverage/lcov-report => coverage.zip in the General Settings section of our build configuration:
artPaths
We also added the artifact path for the build directory (build => build), so that it is available in the artifacts and can be deployed to some server later, which is out of scope for this post.

After running the build, the Code Coverage report is available on the build results page.
Screen Shot 2017-08-23 at 04.22.29

In our build the build-storybook script generates a living style guide for your component base using Storybook. It will be handy to use another build tab to display it, so the storybook-static => stories artifact path has been added. Then we set up a custom tab in the “React App” project settings using the Report tabs section:
reportTab

Note that this Stories tab available on the build results page is generated per build, so you can see how your components change over time and, for example, detect at which point something started looking and/or behaving in a strange way.
Screen Shot 2017-08-23 at 04.17.55

Your feedback is always welcome in our forum and tracker!

Happy building!

Introducing TeamCity RunAs Plugin

$
0
0

When building, testing, and deploying an application with TeamCity, you may need to run a build under a specific user account, different from the one used to run the build agent.

TeamCity RunAs Plugin Overview

The teamcity-runas plugin can be used to run TeamCity build steps under a specified user account on Windows or Linux.

The plugin is compatible with TeamCity 10.0 and later.

Installing Plugin

System Administrators can download the runAs.zip from here and install it on the TeamCity server as usual.

Configuring TeamCity Agents

On Windows

The teamcity-runas plugin requires the TeamCity agent’s account to have certain permissions. Depending on the way you run the agent, granting additional privileges may be required.

  • If you are running the TeamCity agent as a Windows service under the Local System account, it has all required permissions and the teamcity-runas plugin works out-of-the-box.
  • If you are starting the agent as a Windows service using a specified Windows account (for example AgentUser), it must have administrative privileges, must have the ability to replace a process level token (SeAssignPrimaryTokenPrivilege), and act as a part of the operating system (SeTcbPrivilege) or needs to work under the Local Service account. These privileges can be added via a security policy using Control Panel | Administrative Tools | Local Security Policy | Local Policies | User Rights Assignment | Replace a process level token. See Microsoft documentation for details.
    You can also do it using the provided scripts, e.g.: runGrantAccess.cmd BuildUser.
    The changes will not take effect until the next login, so you’ll have to restart the agent.
  • If you are starting the TeamCity agent as a Windows console application, your Windows user account must have administrative privileges.

On Linux

  • With Debian/Ubuntu/Linux, the recommended setup is to run the TeamCity agent as ROOT, as in this case the account has all the required privileges and the teamcity-runas plugin works out-of-the-box (see the related section below).
  • If you run the TeamCity agent as a user different from ROOT, you need to install socat using the following command:
    sudo apt-get update && sudo apt-get install socat

Using Plugin

After the server restart, the plugin provides the Run As build feature and you can add it to your build configuration specifying the username and password.

For Windows the following username formats are acceptable:〈username〉or〈username@domain〉or〈domain\username〉.

For Windows-based build agents it is possible to specify the Windows Integrity Level, the logging level, and additional arguments.

runAS

Once the build is run, the plugin will run all the commands under the specified user account.

Checking Account Used to Run Build

You can enable the teamcity.runAs.log.enabled= true parameter of a build configuration (set to “false” by default) and TeamCity will print the required information into the build log.

runASvalidCreds

If the credentials are invalid, the build fails with “Authentication failure” error:

runASinvalidCreds

Alternatively, to make sure the teamcity-runas plugin is configured properly, you can get the user whose account was used to run the build by checking the environment variables: USERNAME (for Windows) / USER, LOGNAME (for Linux).

For example, you can add a command line build step with the set USERNAME command on Windows / echo "$USER" echo “$LOGNAME” on Linux. This will print the user whose account was used to run the build into the log.

Accessing Files created by RunAs User

It is important to note that when running a build under a specified user account, all processes run in the environment specific for this user.

It means that all files created during the build (e.g. compiled classes, object files, executables, dlls, Gradle/Maven cache files, TeamCity artifacts, some test statistics and so on) will have the teamcity-runas user as the file owner.

The TeamCity agent must have the full control over the files created in the build checkout directory and ideally, over files in other locations, e.g. m2 or .gradle directories as well. Depending on the way you configured the agent (see Configuring TeamCity Agents above), the access level will differ and granting additional permissions may be required.

For Windows, the account with administrative permissions used to run the TeamCity build agent will have sufficient rights to manage the files.

For Linux, if the TeamCity agent is not running under ROOT, it may encounter problems accessing artifacts in the build checkout directory. In this case the necessary permissions can be granted by running the following command:
chmod -R a+rwX %teamcity.build.checkoutDir%.
It can be added as the last step of the build executed under the ‘Run As’ user.

Install the plugin on your TeamCity server, give it a try, and let us know what you think about it.

Happy building!

The TeamCity Team

TeamCity Plugin for HashiCorp Vault

$
0
0

When performing integration tests and deployments, build scripts need credentials to access external servers and services. Traditionally passwords are stored on the TeamCity server as secure parameters. But although secrets are masked in the UI, encrypted at rest, and protected from being exposed in the build log as plain text, this often does not provide a high enough level of security.

HashiCorp Vault offers a unified approach to managing secrets and credentials, allows auditing access, and helps with password rotation.

Integration with Vault

Today we are presenting a new plugin to help build scripts interact with Vault and obtain credentials dynamically.

The plugin allows connecting TeamCity to Vault, requesting new credentials when a build starts, passing them to the build script, and revoking them immediately when the build finishes.

Usage

We expect you have installed and configured:

  • a Vault server. Refer to the Getting Started guide to learn how to launch a testing Vault instance.
  • A TeamCity server. Refer to the Getting Started guide for details.

Perform the following:

  • Download the plugin from the plugin repository, and install it to TeamCity server.
  • Configure secret backends in Vault to obtain secrets, for example AWS credentials, or generic secrets.
  • Create an AppRole in Vault for the TeamCity server to access these backends.
  • Create a connection to Vault in your TeamCity project:1
  • Next in your build configuration declare new build parameters which refer to the Vault secrets to be used in this build:

2

The parameter value has a special syntax:%vault:PATH!KEY%where PATH is the secret name, and KEY is the specific value inside.
At the moment these values can be used in the build parameter declaration only, and cannot be specified in build steps.

  • Run a build step. Deployment scripts can read credentials from environment variables, and do not need to perform any Vault-related actions:3

Besides AWS, any other Vault backends can be used. In this case:

  • Declare a configuration parameter with a value from generic backend:4
  • In a build step refer to this  parameter explicitly:5

How It Works

  • The TeamCity server stores the AppRole ID and secret, and keeps the secret encrypted at rest. When required, it can be rotated manually in the UI, or via the REST API.
  • When a new build is started, the TeamCity server requests a one-time response wrappingtoken in Vault, and sends it to a build agent.
    The AppRole secret never leaves the TeamCity server machine.
  • The build agent requests the actual secrets from Vault.
    The actual credentials are obtained from Vault by a build agent machine directly, and never transferred to the TeamCity server.
  • The build parameters are resolved and the secrets are passed to environment variables and system properties.
  • The build steps are started, and they refer to these environment variables and system properties.
  • If secret values appear in a build log, they are automatically replaced by asterisks.
  • Right after the build finish, the credentials are explicitly revoked in Vault.

Security Considerations

It is advised that you limit access to TeamCity project settings. If a person can modify build configuration settings in a TeamCity project, they can get access to the secrets. Dynamic credentials are revoked after a build, but for generic secrets this may cause more risks. To minimize the risks, split build configurations for CI and deployments into separate projects with different permissions.

Make sure build agent machines are connected to the TeamCity server via HTTPS. Only one-time tokens are transferred over network, and if they are stolen, credentials will be immediately revoked. However, it is a good practice to encrypt all traffic between the server and agents.

Feedback Wanted

This is an initial release, and we continue to improve features and user experience, and encourage you to give it a try and tell us what do you think!

The plugin is open-sourced, and published on GitHub.

If you are attending HashiConf in Austin, TX, USA next week, visit the JetBrains sponsor booth for the demonstration of this plugin and our other plugins for integrating Packer and Terraform with TeamCity and IntelliJ-based IDEs.

The official TeamCity CloudFormation template

$
0
0

As you might have noticed, there was recently an option added to the Get TeamCity page of our website: AWS. This lets you run TeamCity in AWS using the official CloudFormation template.

get-teamcity-aws

In this post, we will go over what’s under the hood of the template, and why it may save you some time and effort.

Usually, installing TeamCity on top of AWS is quite a time-consuming task.
It requires the following steps:

  • Setting up an external database,
  • Configuring the EC2 instance to run a TeamCity server,
  • Configuring it to then connect to the database,
  • Installing the TeamCity server,
  • Installing a TeamCity agent.

And then making the whole installation secure requires even more effort.

We have tried to ease this process and created an official CloudFormation template to run the TeamCity stack in AWS. Using this template lets you run all the above steps with just a single click. And should you decide to destroy the stack, CloudFormation also provides a super simple way to do it with just one click.

The template is located in the S3 bucket. The stack can be launched via the ‘Run on AWS’ button available on the TeamCity site.

The template provides several parameters:

aws-1
It takes about 15 minutes for the template to deploy the whole stack, the most time-consuming task being the RDS Database instance roll up. Once the deployment is ready, you will see the TeamCity server endpoint in the Output section which points you to your TeamCity installation.

aws-2

Just generate the root account and it’s ready to use.

So what is under the hood?

The TeamCity server runs on an EC2 instance with CoreOS Container Linux. The default agent runs as a separate container on the same instance. The external database is provided by an RDS MySQL instance. We decided not to introduce a custom AMI with TeamCity. Instead, we use the official Docker images with the TeamCity server and build agent.

The server and the database are placed in their own VPC which is completely secure. The DB allows only internal connections within the VPC. It’s only possible to connect to the Server via HTTP(s) or SSH.

How the server is running

There are several systemd services that prepare the LVM on the EBS volume to persist your data, create the file system, and run the latest official TeamCity Server and TeamCity Build Agent from the DockerHub images. Those services are linked to each other and roll the whole system back after an instance reboot or failure.

To connect to the server’s console, you need to use your instance key:
ssh -i instance-key.pem core@[server IP]
To see the logs, just run the docker logs command for the desired container.

Once you have TeamCity up and running, there are a few more steps to consider:

Happy building with TeamCity on AWS!

TeamCity Kubernetes Support Plugin

$
0
0

Kubernetes nowadays is quite a popular way to run Docker containers. A number of teams and organizations already have a Kubernetes cluster configured and used in production.

Now with the help of the TeamCity Kubernetes Support plugin, it is possible to use the same infrastructure to run TeamCity build agents.

The plugin is compatible with TeamCity 2017.1.x and later.

First, download the plugin and install it on the TeamCity server as usual.

Then you can start by configuring a cloud profile in a project:

kub-1-new

Specify the URL of the Kubernetes API server (aka Kubernetes master), select the appropriate namespace.

Select one of the Kubernetes API authentication options:

kub-2

The next step after connecting to the Kubernetes API is creating a cloud image.

kub-3

There are two options available:

  • Simply run single container: good choice for those who don’t know about all the powerful Kubernetes features but want to simply run its container;
  • Use pod template from deployment: will handle all the possible advanced scenarios of configuring workload to a Kubernetes cluster like multi-container pods, node/pod affinity and toleration, etc.

When the “Simply run single container” mode is selected, users can specify the name of the Docker image with the build agent they want to use.

In our setup, we are using the official TeamCity Build Agent image which is supported by the plugin. You can also create your own image.

Other options like Docker command, Docker arguments, and image pull policy, will be useful as well.

kub-4

Another cloud image option is ‘Use pod template from deployment’.

Here you simply specify a deployment name: remember to check that the deployment belongs to the same namespace you’ve provided in the cloud profile. You can either use the official TeamCity Build Agent image in your deployment like in the example below, or your own image.

kub-5

There is a small trick here. When given a deployment name, the plugin will not actually use it as deployment, but it will extract the PodTemplateSpec section from its definition and use it to create the plugin’s own pods. The plugin’s own pods means that the pods will not be connected to deployment anyhow. The Kubernetes deployments feature will not be used to manage the pods’ lifecycle. The TeamCity server will take care of those pods on its own. Deployment will be used as a container of PodTemplateSpec which can be referenced by name.

After a cloud profile is created and saved, you will be able to start TeamCity agents running in containers in the scope of the pods on the Kubernetes cluster. TeamCity will mark every started pod with a set of specific labels.

kub-6

Using those labeled pods, you can always determine which TeamCity server started a particular pod, which cloud profile and cloud image are affected.

Feel free to download the plugin, try it, and share your feedback with us!

Or you can TestDrive it in the cloud.

Happy building with TeamCity!

Branch specific settings in TeamCity

$
0
0

We’re often asked how to run different build steps in different branches. In fact, this has already been possible in TeamCity for quite some time (since version 9.1), but it seems we need to do a better job explaining how to use this feature.

Let’s assume that you have a build configuration where building of feature branches is enabled.

First, enable Versioned settings for the project where this build configuration resides. TeamCity will ask for a VCS root where the settings are to be stored. For simplicity’s sake, let’s store the settings in the same VCS repository where other project files are located.

For the Versioned settings, make sure the option When build starts is set to the use settings from VCS value. TeamCity will then take the settings from a branch used by the build instead of the current settings.

versioned_settings

Once versioned settings are enabled, TeamCity will commit the .teamcity directory with the current project settings to the default branch of the selected VCS root. To start customizing the settings in our feature branch, we need to merge the .teamcity directory from the default branch.

But before making any changes in the .teamcity directory in your feature branch, bear in mind that not all the changes there will be applied. For instance, if you add a new project or a build configuration, such changes will be ignored. This happens because TeamCity does not support different sets of projects and build configurations in feature branches.

Besides, there is a chicken and egg problem. Currently TeamCity loads settings from .teamcity while the build sits in the queue. But to load the settings from a repository, TeamCity needs to know this repository settings (VCS roots). Thus VCS roots are always taken from the current project settings.

The same story is with build triggers and snapshot dependencies. TeamCity needs triggers to decide when to place builds into the queue. And TeamCity needs snapshot dependencies to create a build chain (graph of builds) before the builds are going to the queue. Hence the changes in .teamcity affecting these settings are ignored too.

But there are plenty of other settings whose changes will affect the build behavior:

  • build number pattern
  • build steps
  • system properties and environment variables
  • requirements
  • build features (except Automatic merge and VCS labeling)
  • failure conditions
  • artifact publishing rules
  • artifact dependencies

By default, the settings are stored in the XML format. Since there is no publicly available documentation for this format, it is hard to guess how to change build steps correctly, or how to add a build feature. This is what a command line build step looks like in XML:

<build-runners>
 <runner id="RUNNER_76" name="" type="simpleRunner">
   <parameters>
     <param name="script.content" value="echo &quot;Hello world!&quot;" />
     <param name="teamcity.step.mode" value="default" />
     <param name="use.custom.script" value="true" />
   </parameters>
 </runner>
</build-runners>

What can help here is seeing how TeamCity generates these settings. For instance, get a sandbox project on your TeamCity server for such experiments, and browse the audit page for settings difference after each change:

diff

Another option is to use Kotlin DSL (see the Settings format option on the Versioned settings page). In this case, instead of .xml files, .kt files will be stored under the .teamcity folder.

A Kotlin DSL project can be opened in IntelliJ IDEA Community (free and open-source IDE from JetBrains). Kotlin DSL files look much more concise, and IDE auto-completion makes it much easier to work with them, especially in the recent TeamCity versions. This is what the same command line build step looks like in Kotlin DSL:

steps {
    script {
        scriptContent = """echo "Hello world!""""
    }
}

Some additional hints:

  • Since TeamCity 2017.2 you can browse Kotlin DSL documentation using the following URL: <TeamCity server URL>/app/dsl-documentation/index.html
  • If a build loads settings from the VCS repository, you should see something like this in the build log:
    buildlog
  • Actual settings used by the build are stored in the buildSettings.xml file under the hidden build artifacts. This file can be helpful if something does not work as expected.
    buildSettings
  • If you’re using remote runs instead of feature branches, then all the same abilities are also available, provided that the .teamcity directory is  present in your repository and versioned settings are configured as described above.

Finally, one more thing to keep in mind. If changes under .teamcity in your feature branch were temporary, you should not forget to revert the settings during the merge with the default branch. For instance, if you removed some long running steps or switched off some tests for convenience.

Happy building!

TeamCity on Docker Hub – it’s official now!

$
0
0

As the popularity of Docker is growing exponentially, Docker Hub has become a huge software distribution platform. Users publish their own images, companies start their official repositories, and everyone is happy.

Up until now JetBrains wasn’t part of this. Then we noticed over 300 images related to TeamCity server, created by enthusiastic Docker Hub users. We decided to join the club and make support for this platform official.

We’ve published the latest version of our software as ready-to-use images and we are going to keep them updated as soon as new versions appear.

There are currently three official TeamCity images in Docker Hub:
1. jetbrains/teamcity-server
Use this image to run a fully functional, ready-to-use TeamCity server. There are multiple options and additional parameters available, which are summarized in the image description on its Docker Hub page.

Once the server is running, you will definitely need some build agents to do the actual work (building) for you. We’ve prepared two separate images for that.

2. jetbrains/teamcity-agent
This image is recommended for general Java development. Build agents created from this image will support agent-side checkout for Git, Mercurial, and SVN, any Java-related activity (as we provide OpenJDK installed), and a full set of features related to Java development.

3. jetbrains/teamcity-minimal-agent
As the requirements for build agents vary considerably, we have also created this minimalistic agent image which you can use for simple builds or as a base for your own images.

Take a look at the official JetBrains Docker Hub page for more details about our Docker images. Give them a try and let us know what you think!

banner_blog@2x


Installing GitHub Webhooks from TeamCity

$
0
0

Greetings, everyone!

Since TeamCity 10.0 it has become possible to use commit hooks with a TeamCity server. Now, when a VCS change is detected via a commit hook, TeamCity automatically increases the VCS repository polling interval, reducing the load on both the TeamCity server and VCS repository. And obviously the presence of the hook greatly decreases the time needed to detect a change.

Unfortunately, installation and configuration of commit hooks is not an easy task and for on-premises VCS repositories it requires administration skills. At the same time, popular VCS hostings such as GitHub, support installation of commit hooks via their REST API, and installation of commit hooks for them can be a lot simpler, provided that we use such an API from TeamCity…

So today, we’d like to announce one more TeamCity plugin whose task is to install and maintain GitHub commit hooks (do not worry, we plan to add support for other VCS hosting services too).

The plugin supports both GitHub.com and GitHub Enterprise. The plugin does not install a webhook automatically for GitHub.com because a webhook requires a connection from GitHub.com to the TeamCity server, and in the majority of cases, when TeamCity is installed in the intranet, such a connection is blocked by a firewall. In case of GitHub Enterprise, the plugin will install a webhook automatically for any TeamCity project created from a URL or via GitHub integration.

The plugin works with the GitHub REST API and has to make API calls to GitHub on behalf of the current user, so it requires a GitHub connection configured in the project or its parent.

The plugin is quite simple, basically it does three things:

  • It shows a suggestion to install a GitHub webhook if it finds a GitHub repository in a project without such a webhook:webhook_suggestion
  • It provides a new action in the project actions menu for webhook installation enabling you to install or reinstall a webhook at any time: project_actions
  • It checks the status of all of the installed webhooks and raises a warning via the health report if some problem is detected:webhook_problem

The plugin is open source, distributed under the Apache 2.0 license. Most of the code is written in Kotlin. The source code is published on GitHub.

The plugin relies on the new TeamCity API and will only work with TeamCity 10.0 and later. Download the plugin, install it on your TeamCity server and try the plugin. We’ll appreciate your feedback!

Happy building!

banner_blog@2x

TeamCity 10 GitHub-Related Improvements

$
0
0

Configuring TeamCity to use GitHub as the source code repository has always been easy, especially since the feature ‘create from URL‘ was first introduced. TeamCity 10 has brought a number of improvements related to integration with GitHub, which are worth a special mention.

Project-level GitHub Connection in TeamCity

We’ve been gradually improving the integration, and in TeamCity 10 you can now set up a connection to GitHub on the project level:

connectionproject

Once the connection is set up, you can create a project or a build configuration pointing to a GitHub repository:connectionprojectlist

Besides, with the connection configured, a small GitHub icon becomes active in several places where a repository URL can be specified: create project from URL, create Git VCS root, GitHub issue tracker, create create VCS root from URL:

createvcsroot

On clicking the icon, TeamCity will list GitHub repositories available to the user.

Configuring a GitHub connection is useful for the organization administrator who can create a parent project and configure a connection to GitHub there once; thus, all the TeamCity users of the organization will see a list of GitHub repositories URLs in the TeamCity Web UI. It makes setting up a subproject, a Git VCS root, or a GitHub issue tracker extremely easy.

GitHub Issue Tracker Integration in TeamCity

Built-in integration with GitHub issue tracker was also introduced in TeamCity 10. It is configured on the dedicated page of the project settings. If a project level connection to GitHub is configured, you can simply select the repository URL from the list available on clicking the GitHub icon:

tracker

If no connection is configured, the URL can be specified manually.

The rest is easy – select the type of authentication and provide the required information, tell TeamCity which strings should be recognized as references to issues: for GitHub, the regex syntax is used, e.g. #(\d+). TeamCity will resolve the issue number mentioned in a VCS comment and will display a link to this issue in the Web UI (e.g. on the Changes page, Issues tab of the build results page).

TeamCity Build Status in GitHub

Developers find it handy to view the status of their commits right in the repository. Earlier there were several external plugins allowing you to publish the TeamCity build status on GitHub, and in TeamCity 10 we delivered a bundled build feature, Commit Status Publisher, which automatically attaches the build status to your GitHub pull requests:

commitstatuspub

Besides, using the TeamCity REST API, you can publish the status icon of your TeamCity build in the README for your repository:

statuspic

TeamCity Commit Hooks

All the above-mentioned improvements are bundled with TeamCity 10, but we should also mention the TeamCity Commit Hooks Plugin, not bundled with TeamCity yet. This plugin is compatible with TeamCity 10 or later. Its task is to install webhooks for GitHub repositories specified in TeamCity VCS roots.
Installed GitHub webhooks greatly decrease the time required for TeamCity to detect a change. As an additional benefit, webhooks reduce the load on both the TeamCity server and the GitHub server.

 

Take the advantage of the TeamCity-GitHub improved integration and let these features make your experience of Continuous Integration & Delivery with GitHub and TeamCity nice and smooth.

Kotlin Configuration Scripts: An Introduction

$
0
0

This is a five part series on working with Kotlin to create Configuration Scripts for TeamCity.

  1. An Introduction to Configuration Scripts
  2. Working with Configuration Scripts
  3. Creating Configuration Scripts dynamically
  4. Extending the TeamCity DSL
  5. Testing Configuration Scripts

With TeamCity 10, we brought the ability to define configurations using Kotlin as opposed to using XML or the User Interface.

This provides several benefits amongst which are

  • A human-readable and editable configuration script, not relying on the user interface
  • Ability to do diffs on changes to a configuration script
  • Ability to check in your configuration scripts alongside your source code1
  • Ability to dynamically create configuration scripts, as well as test them

Why Kotlin?

You might ask why Kotlin and not some other language? For those not familiar with Kotlin, it is a statically-typed language developed by us at JetBrains and open sourced under the Apache 2 license. It targets the JVM, JavaScript (and we also have native in the works). It’s designed to be a pragmatic language that cuts down boilerplate code, remains expressive, and facilitates tooling. Kotlin provides a series of features that allow the creation of DSL’s (Domain Specific Languages) and TeamCity is a perfect fit for this purpose. Kotlin enables the benefits we’ve outlined and we’ll see.

Why not something more common such as YAML? Based on our experience, we believe that while YAML is great for the simpler setups, at some point it does fall short, and can lose clarity when it comes to defining more complex configuration scripts. We wanted to provide a solution that works for the simplest to the most demanding of scenarios, and that’s why we’ve gone with Kotlin. It’s important to understand though that we’re not providing just a DSL for creating configuration scripts. Kotlin is a programming language and as such we have the ability to write any kind of code in our configuration scripts (which, of course, like anything can be also abused). This enables many scenarios such as those we’ll see when creating dynamic configurations.

What is needed

Given Kotlin is a proper language, you might be wondering what tooling is required to edit configuration scripts. In principle any editor will do. Kotlin is built so that it can be used with any editor or IDE. In fact we ship support for not only IntelliJ IDEA (both Ultimate and the free OSS community editions), but also Eclipse and NetBeans. If Vim or another editor is your thing, you can also use that along with a command line compiler.

For the purpose of this blog post series, we’ll be using IntelliJ IDEA.

Creating a first script

While we can start with a completely blank script, the easiest way to create a Kotlin build script is to take an existing TeamCity project and either

  • Download it as a Kotlin Script
  • Enable Versioned Settings

The first is a great way to start playing with Kotlin configuration scripts and get a feel for them. The second option not only provides us with the configuration scripts checked in to source control, but it’s actually a required step for us to use Kotlin build scripts in production.

The recommended approach is to use the second option once we’re ready to enable Kotlin scripting in our projects. The first option, as mentioned, is great for discovering how things work and getting familiar with Kotlin configuration scripts.

Download Settings as a Kotlin Script

In order to see what a build configuration looks like in Kotlin, simply Edit Project Settings on the selected project and click on the Actions menu, picking the entry Download Configuration Script in Kotlin format

download-kotlin-settings

This downloads a zip file that contains a Maven project which can be obtained in IntelliJ IDEA. Note that the folder it contains is prefixed with a dot, which indicates it’s hidden under MacOS. The idea is that this folder can essentially be placed in version control at some point (TeamCity marks its configuration files as hidden).

Enable Versioned Settings

The second option, which is required for us to actually use Kotlin configuration scripts, is to enable Versioned Settings. This is done under Versioned Settings and selecting Enable, and Kotlin as the file format.

versioned-settings

 

As soon as we activate this, the TeamCity UI will no longer allow any changes and all the configuration will be stored in version control and modifications have to take place via the build script.

These script files will have the same folder layout as that of the downloaded zip file (first option). For instance, the following are the files checked in to version control for the Joda-Time project. We can see that once again it’s a Maven project containing a series of Kotlin files (.kt). 

Opening the script in IntelliJ IDEA

Once we have the configuration script (be it as a zip file or checked in to version control on enabled versioned settings), we can open it in IntelliJ IDEA, and Maven will pull down the correct dependencies. Our project layout should look like the one below

project-structure

Don’t feel overwhelmed by the number of files in there. In fact, TeamCity really only needs one file, which is settings.kts. Let’s examine each file in more detail.

settings.kts

kts is a Kotlin Script file, different from a Kotlin file (.kt) in that it can be run as a script. As mentioned above, all the code relevant to the TeamCity configuration could be stored in this single file, but it is divided into several files to provide a better separation of concerns.

This file contains usually two lines

settings

version indicates the TeamCity version, and project() is the main entry point to the configuration script. It represents a function call, which takes as parameter a Project, representing the entire TeamCity project.

The parameter GitExtensions.Project is an object in Kotlin. Think of objects in Kotlin as you would in JavaScript. Or to compare it to Java, they’d be a Singleton, a single instance of a Kotlin class. While Kotlin script could work with singletons, having a first-class support for objects, makes things much easier.

In this case, Project is the name of the actual object, and GitExtensions is the package name.

Project.kt

This file contains the GitExtensions.Project object, and is where all the magic happens. If you look at the script layout, it’s basically a code representation of the build steps we’re accustomed to seeing the TeamCity user interface

project

which would correspond to the following entries in the UI, in addition to VCS roots and Build Types.

settings-ui

GitExtensions_HttpsGithubComJetbrainsGitextensions.kt

This object defines the VCS root configuration. It’s important to note that we could have just placed all this information directly in the code above, as opposed to making the call vcsRoot(GitExtensions_…).  However, the advantage to doing this, as we’ll see, is not only to make the code cleaner and separate concerns, but also to provide  reusability.

vcs-settings-root

GitExtensions_Main.kt

Finally, we have the actual meat of where the build happens. This object defines the build steps, failure conditions, and everything else you’d come to expect for a build configuration in TeamCity

build-type

Summary

In this first part we’ve seen the components of a TeamCity configuration script. In the next part we’ll dive a little deeper into the DSL, modify the script, and see some of the benefits that Kotlin and IntelliJ IDEA already start providing us in terms of guidance via code assistants.

[1] While enabling version control storage for settings has been available since version 9, it was only available in XML format. TeamCity 10 brings Kotlin to this.

banner_blog@2x

Kotlin Configuration Scripts: Working with Configuration Scripts

$
0
0

This is part two of the five-part series on working with Kotlin to create Configuration Scripts for TeamCity.

  1. An Introduction to Configuration Scripts
  2. Working with Configuration Scripts
  3. Creating Configuration Scripts dynamically
  4. Extending the TeamCity DSL
  5. Testing Configuration Scripts

In the first part we saw the basics of Configuration Scripts and how to get started. Now we’ll dive a little deeper into TeamCity’s DSL and see what it provides us in terms of building configuration scripts.

Examining the DSL

The DSL comes in a series of packages with the main one being the actual DSL which is contained in the configs-dsl-kotlin-{version}.jar file. Examining it, we see that it has a series of classes that describe pretty much the entire TeamCity user interface.

main-dls-package

At any time you can navigate to a definition using for instance IntelliJ IDEA, and it will prompt you to download sources or decompile.  The sources are available from the actual TeamCity server your pom.xml file points to, and they’re a great way to see all the potential of the DSL.

The entry point to a project, as we saw in the previous post, is the Project object. From here we can define all the settings such as the basic project properties, the VCS roots, build steps, etc. There are a few basic parameters that should be set:

  • Uuid: it’s the internal ID TeamCity maintains. It is unique across the server. It’s not recommended that this value be changed as TeamCity uses it internally to associate data with the project.  
  • extId: this is the user-friendly ID used in URLs, in the UI, etc. and can be changed if required.
  • parentId: represents the extId of a project where this project belongs, defaulting to the value _Root for top-level projects.
  • name: the name of the project

and optionally

  • description: description for the project

Beyond the above, everything else is pretty much optional. Of course, if that’s the only thing we define, then not much is going to happen during the build process.

Modifying the build process

The code below is an excerpt from the configuration script for the Spek project (parts of the configuration are omitted for brevity). This particular build compiles the code and runs some tests using Gradle.

steps {
   gradle {
       name = "Snapshot Build"
       tasks = "clean jar test"
       jdkHome = "%env.JDK_18_x64%"
   }
}


triggers {
   vcs {
       branchFilter = "+:<default>"
       perCheckinTriggering = true
       groupCheckinsByCommitter = true
   }
}

Let’s extend this build to add a feature to that TeamCity has, Build Files Cleaner also known as Swabra. This build features makes sure that files left by the previous build are removed before running new builds.

We can add it using the features function. As we start to type, we can see that the IDE provides us with completion:

features-completion

The features function takes in turn a series of feature functions, each of which adds a particular feature. In our case, the code we’re looking for is

features {
   feature {
       type = "swabra"
   }
}

The resulting code should look like

steps {
   gradle {
       name = "Snapshot Build"
       tasks = "clean jar test"
       useGradleWrapper = true
       enableStacktrace = true
       jdkHome = "%env.JDK_18_x64%"
   }
}

triggers {
   vcs {
       branchFilter = "+:<default>"
       perCheckinTriggering = true
       groupCheckinsByCommitter = true
   }
}

features {
   feature {
       type = "swabra"
   }
}

And this works well. The problem is that, if we want to have this feature for every build configuration, we’re going to end up repeating the code. Let’s refactor it to a better solution.

Refactoring the DSL

What we’d ideally like is to have every build configuration automatically have the Build Files Cleaner feature without having to manually add it. In order to do this, we could introduce a function that wraps every build type with this feature. In essence, instead of having the Project call

buildType(Spek_Documentation)
buildType(Spek_Publish)
buildType(Spek_BuildAndTests)

we would have it call

buildType(cleanFiles(Spek_Documentation))
buildType(cleanFiles(Spek_Publish))
buildType(cleanFiles(Spek_BuildAndTests))

For this to work, we’d need to create the following function

fun cleanFiles(buildType: BuildType): BuildType {
   buildType.features {
       feature {
           type = "swabra"
       }
   }
   return buildType
}

Which essentially takes a build type, adds a feature to it, and returns the build type. Given that Kotlin allows top-level functions (i.e. no objects or classes are required to host a function), we can place it anywhere or create a specific file to hold it.

Now let’s extend this function to allow us to pass parameters to our feature, such as the rules to use when cleaning files.

fun cleanFiles(buildType: BuildType, rules: List<String>): BuildType {
   buildType.features {
       feature {
           type = "swabra"
           param("swabra.rules", rules.joinToString("\n"))
       }
   }
   return buildType
}

We pass in a list of files which are then passed in as parameter to the feature. The joinToString function allows us to concatenate a list of strings using a specific separator, in our case the carriage return.   

We can improve the code a little so that it only adds the feature if it doesn’t already exist:

fun cleanFiles(buildType: BuildType, rules: List<String>): BuildType {
   if (buildType.features.items.find { it.type == "swabra" } == null) {
       buildType.features {
           feature {
               type = "swabra"
               param("swabra.rules", rules.joinToString("\n"))
           }
       }
   }
   return buildType
}

Generalizing feature wrappers

The above function is great in that it allows us to add a specific feature to all build configurations. What if we wanted to generalize this so that we could define the feature ourselves? We can do that by passing a block of code to our cleanFiles function, which we’ll also rename to something more generic.

fun wrapWithFeature(buildType: BuildType, featureBlock: BuildFeatures.() -> Unit): BuildType {
   buildType.features {
       featureBlock()
   }
   return buildType
}

What we’re doing here is creating what’s known as a higher-order function, a function that takes another function as  a function. In fact this is exactly what features, feature, and many of the other TeamCity DSL’s are.

One particular thing about this function, however, is that it’s taking a special function as a parameter, which is an extension function in Kotlin. When passing in this type of parameter, we refer to it as Lambdas with Receivers (i.e. there is a receiver object that the function is applied on).

This then allows us to make the calls to this function in a nice way referencing  feature directly

buildType(wrapWithFeature(Spek_BuildAndTests, {
   feature {
       type = "swabra"
   }
}))

Summary

In this post we’ve seen how we can modify TeamCity configuration scripts using the extensive  Kotlin-based DSL.  What we really have in our hands is a full programming language along with all the features and power that it provides. We can encapsulate functionality in functions to re-use, we can use higher-order functions  as well as other things that open up many possibilities.

In the next part we’ll see how to use some of this to create scripts dynamically.

 

 

Kotlin Configuration Scripts: Creating Configuration Scripts Dynamically

$
0
0

This is part three of the five-part series on working with Kotlin to create Configuration Scripts for TeamCity.

  1. An Introduction to Configuration Scripts
  2. Working with Configuration Scripts
  3. Creating Configuration Scripts dynamically
  4. Extending the TeamCity DSL
  5. Testing Configuration Scripts

We saw in the previous post how we could leverage some of Kotlin’s language features to reuse code. In this part, we’re going to take advantage of the fact that we are in fact dealing with a full programming language and not just a limited DSL, to create a dynamic build configuration.

Build Configurations based on parameters

The scenario is the following: we have an HTTP server that we need to test on different platforms with a range of concurrent connections on each one. This generates potentially a lot of different build configurations that we’d need to create and maintain.

Instead of doing this manually, what we can do is write some code to have our Kotlin DSL Configuration Script generate all the different build configurations for us.

Let’s say that we have a list of Operating Systems and a range of concurrent connections for each one. The first thing to do is to create a data class that represents this information

data class BuildParameters(val name: String, val operatingSystem: String, val connections: Int)

which in essence is like a regular class but provides a series of benefits such as a nicer string representation, equality and copy operations amongst other things.

Now let’s imagine we have the different platforms we want to test on represented as a list of BuildParameters

val operatingSystems = listOf(
            BuildParameters("Windows Build", "windows", 30),
            BuildParameters("MacOS Build", "osx", 20),
            BuildParameters("Linux Build", "ubuntu", 20)
    )

what we’d like to do, is iterate over this list and create a new build type for each combination. In essence, if our standard Project is

object Project : Project({
    uuid = "9179636a-39a3-4c2c-aa7e-6e2ea7cfbc5b"
    extId = "Wasabi"
    parentId = "_Root"
    name = "Wasabi"
    vcsRoot(Wasabi_HomeGitWasabiGitRefsHeadsMaster)
    buildType(WasabiBuild)
})

what we want to do is create a new buildType for each entry in the list

object Project : Project({
    uuid = "9179636a-39a3-4c2c-aa7e-6e2ea7cfbc5b"
    extId = "Wasabi"
    parentId = "_Root"
    name = "Wasabi"

    vcsRoot(Wasabi_HomeGitWasabiGitRefsHeadsMaster)

    val operatingSystems = listOf(
            BuildParameters("Windows Build", "windows", 30),
            BuildParameters("MacOS Build", "osx", 20),
            BuildParameters("Linux Build", "ubuntu", 20)
    )

    operatingSystems.forEach {
        buildType(OperatingSystemBuildType(it))
    }
})

Creating a base Build Type

Any parameter passed into buildType needs to be of the type BuildType. This means we’d need to inherit from this class and at the same time provide some parameters to it (BuildParameters)

class OperatingSystemBuildType(buildParameters: BuildParameters) : BuildType() {
    init {
        val paramToId = buildParameters.name.toExtId()
        uuid = "53f3d94a-20c6-43a2-b056-baa19e55fd40-$paramToId"
        extId = "BuildType$paramToId"
        name = "Build for ${buildParameters.name}"

        vcs {
            root(Wasabi.vcsRoots.Wasabi_HomeGitWasabiGitRefsHeadsMaster)

        }
        steps {

            gradle {
                tasks = "clean build"
                useGradleWrapper = true
                gradleWrapperPath = ""
                gradleParams = "-Dconnections=${buildParameters.connections}"
            }
        }
        triggers {
            vcs {
            }
        }

        requirements {
            contains("teamcity.agent.jvm.os.name", buildParameters.operatingSystem)
        }
    }

}

The code is actually a pretty standard Configuration Script that defines a Gradle build step, has a VCS definition and defines a VCS trigger. What we’ve done is just enhance it somewhat.

To begin with,  we’re using the BuildParameters name property to suffix it to the uuid, extId and name. This guarantees a unique ID per build configuration as well as provides us a name to identify it easily. To make sure the values are properly formatted, we use the TeamCity DSL defined extension function to String, named toExtId() which takes care of removing any forbidden characters.

In the steps definition, we pass in certain parameters to Gradle, which in our case is the number of concurrent connections we want to test with. Obviously, this is just a sample, and the actual data being passed in can be anything and used anywhere in the script.

Finally, we also use the BuildParameters operatingSystem property to define Agent requirements.

Summary

The above is just a sample of what can be done when creating dynamic build scripts. In this case, we created multiple build configurations, but we could just as well have created multiple steps, certain VCS triggers, or whatever else could come in useful. The important thing to understand is that at the end of the day, Kotlin Configuration Script isn’t just merely a DSL but a fully fledged programming language.

In the next part of this series, we’ll see how we can extend the Kotlin Configuration Scripts (hint – it’s going to involve extension functions).

 

 

Viewing all 103 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>