Merge matrix feature branch into master.

Features included in this merge:
- Added script to build and upload docker image for matrix test.
- Added script to create test cases and created go__master testcases based on it.
- Created dictionary for runtimes and gRPC releases for supported languages.
- Added go 1.7 and 1.8 Dockerfile/templates.

See tools/interop_matrix/README.md for details.

templates/tools/dockerfile/go_build_interop.sh.include

+#!/bin/bash
+# Copyright 2015, Google Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+# Builds Go interop server and client in a base image.
+set -e
+
+# Clone just the grpc-go source code without any dependencies.
+# We are cloning from a local git repo that contains the right revision
+# to test instead of using "go get" to download from Github directly.
+git clone --recursive /var/local/jenkins/grpc-go src/google.golang.org/grpc
+
+# Get all gRPC Go dependencies
+(cd src/google.golang.org/grpc && make deps && make testdeps)
+
+# copy service account keys if available
+cp -r /var/local/jenkins/service_account $HOME || true
+
+# Build the interop client and server
+(cd src/google.golang.org/grpc/interop/client && go install)
+(cd src/google.golang.org/grpc/interop/server && go install)

templates/tools/dockerfile/interoptest/grpc_interop_go1.7/build_interop.sh.template

+%YAML 1.2
+--- |
+  <%include file="../../go_build_interop.sh.include"/>  

templates/tools/dockerfile/interoptest/grpc_interop_go1.8/build_interop.sh.template

+%YAML 1.2
+--- |
+  <%include file="../../go_build_interop.sh.include"/>  

protobuf @ 593e917c

-Subproject commit a6189acd18b00611c1dc7042299ad75486f08a1a
+Subproject commit 593e917c176b5bc5aafa57bf9f6030d749d91cd5

tools/README.md

@@ -13,6 +13,8 @@ container engine, BigQuery etc)
 
 internal_ci: Support for running tests on an internal CI platform.
 
+interop_matrix: Scripts to build, upload, and run gRPC clients in docker with various language/runtimes. 
+
 jenkins: Support for running tests on Jenkins.
 
 run_tests: Scripts to run gRPC tests in parallel.

tools/dockerfile/interoptest/grpc_interop_go1.7/build_interop.sh

+#!/bin/bash
+# Copyright 2015, Google Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+# Builds Go interop server and client in a base image.
+set -e
+
+# Clone just the grpc-go source code without any dependencies.
+# We are cloning from a local git repo that contains the right revision
+# to test instead of using "go get" to download from Github directly.
+git clone --recursive /var/local/jenkins/grpc-go src/google.golang.org/grpc
+
+# Get all gRPC Go dependencies
+(cd src/google.golang.org/grpc && make deps && make testdeps)
+
+# copy service account keys if available
+cp -r /var/local/jenkins/service_account $HOME || true
+
+# Build the interop client and server
+(cd src/google.golang.org/grpc/interop/client && go install)
+(cd src/google.golang.org/grpc/interop/server && go install)
+  

tools/dockerfile/interoptest/grpc_interop_go1.8/build_interop.sh

+#!/bin/bash
+# Copyright 2015, Google Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+# Builds Go interop server and client in a base image.
+set -e
+
+# Clone just the grpc-go source code without any dependencies.
+# We are cloning from a local git repo that contains the right revision
+# to test instead of using "go get" to download from Github directly.
+git clone --recursive /var/local/jenkins/grpc-go src/google.golang.org/grpc
+
+# Get all gRPC Go dependencies
+(cd src/google.golang.org/grpc && make deps && make testdeps)
+
+# copy service account keys if available
+cp -r /var/local/jenkins/service_account $HOME || true
+
+# Build the interop client and server
+(cd src/google.golang.org/grpc/interop/client && go install)
+(cd src/google.golang.org/grpc/interop/server && go install)
+  

tools/interop_matrix/README.md

+# Overview
+
+This directory contains scripts that facilitate building and running gRPC tests for combinations of language/runtimes (known as matrix).
+
+The setup builds gRPC docker images for each language/runtime and upload it to Google Container Registry (GCR). These images, encapsulating gRPC stack
+from specific releases/tag, are used to test version compatiblity between gRPC release versions.
+
+## Instructions for creating GCR images
+- Edit  `./client_matrix.py` to include desired gRPC release.
+- Run `tools/interop_matrix/create_matrix_images.py`.  Useful options:
+  - `--git_checkout` enables git checkout grpc release branch/tag.
+  - `--release` specifies a git release tag.  Make sure it is a valid tag in the grpc github rep.
+  - `--language` specifies a language.
+  For examle, To build all languages for all gRPC releases across all runtimes, do `tools/interop_matrix/create_matrix_images.py --git_checkout --release=all`.
+- Verify the newly created docker images are uploaded to GCR.  For example:
+  - `gcloud beta container images list --repository gcr.io/grpc-testing` shows image repos.
+  - `gcloud beta container images list-tags gcr.io/grpc-testing/grpc_interop_go1.7` show tags for a image repo.
+
+## Instructions for adding new language/runtimes*
+- Create new `Dockerfile.template`, `build_interop.sh.template` for the language/runtime under `template/tools/dockerfile/`.
+- Run `tools/buildgen/generate_projects.sh` to create corresponding files under `tools/dockerfile/`.
+- Add language/runtimes to `client_matrix.py` following existing language/runtimes examples.
+- Run `tools/interop_matrix/create_matrix_images.py` which will build and upload images to GCR.  Unless you are also building images for a gRPC release, make sure not to set `--gcr_tag` (the default tag 'master' is used for testing).
+
+*: Please delete your docker images at https://pantheon.corp.google.com/gcr/images/grpc-testing?project=grpc-testing afterwards.  Permissions to access GrpcTesting project is required for this step.
+
+## Instructions for creating new test cases
+- Create test cases by running `LANG=<lang> [RELEASE=<release>] ./create_testcases.sh`.  For example,
+  - `LANG=go ./create_testcases.sh` will generate `./testcases/go__master`, which is also a functional bash script.
+  - `LANG=go KEEP_IMAGE=1 ./create_testcases.sh` will generate `./testcases/go__master` and keep the local docker image so it can be invoked simply via `./testcases/go__master`.  Note: remove local docker images manually afterwards with `docker rmi <image_id>`.
+- Stage and commit the generated test case file `./testcases/<lang>__<release>`.
+
+## Instructions for running test cases against a GCR image
+- Run test cases by specifying `docker_image` variable inline with the test case script created above.
+For example:
+  - `docker_image=gcr.io/grpc-testing/grpc_interop_go1.7:master ./testcases/go__master` will run go__master test cases against `go1.7` with gRPC release `master` docker image in GCR.
+
+
+Note:
+- File path starting with `tools/` or `template/` are relative to the grpc repo root dir.  File path starting with `./` are relative to current directory (`tools/interop_matrix`).

tools/gcp/utils/gcr_upload.py → tools/interop_matrix/client_matrix.py

@@ -28,92 +28,21 @@
 # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
-"""Upload docker images to Google Container Registry."""
-
-from __future__ import print_function
-
-import argparse
-import atexit
-import os
-import shutil
-import subprocess
-import tempfile
-
-argp = argparse.ArgumentParser(description='Run interop tests.')
-argp.add_argument('--gcr_path',
-                  default='gcr.io/grpc-testing',
-                  help='Path of docker images in Google Container Registry')
-
-argp.add_argument('--gcr_tag',
-                  default='latest',
-                  help='the tag string for the images to upload')
-
-argp.add_argument('--with_files',
-                  default=[],
-                  nargs='+',
-                  help='additional files to include in the docker image')
-
-argp.add_argument('--with_file_dest',
-                  default='/var/local/image_info',
-                  help='Destination directory for with_files inside docker image')
-
-argp.add_argument('--images',
-                  default=[],
-                  nargs='+',
-                  help='local docker images in the form of repo:tag ' +
-                  '(i.e. grpc_interop_java:26328ad8) to upload')
-
-argp.add_argument('--keep',
-                  action='store_true',
-                  help='keep the created local images after uploading to GCR')
-
-
-args = argp.parse_args()
-
-def upload_to_gcr(image):
-  """Tags and Pushes a docker image in Google Containger Registry.
-
-  image: docker image name, i.e. grpc_interop_java:26328ad8
-
-  A docker image image_foo:tag_old will be uploaded as
-     <gcr_path>/image_foo:<gcr_tag>
-  after inserting extra with_files under with_file_dest in the image.  The
-  original image name will be stored as label original_name:"image_foo:tag_old".
-  """
-  tag_idx = image.find(':')
-  if tag_idx == -1:
-    print('Failed to parse docker image name %s' % image)
-    return False
-  new_tag = '%s/%s:%s' % (args.gcr_path, image[:tag_idx], args.gcr_tag)
-
-  lines = ['FROM ' + image]
-  lines.append('LABEL original_name="%s"' % image)
-
-  temp_dir = tempfile.mkdtemp()
-  atexit.register(lambda: subprocess.call(['rm', '-rf', temp_dir]))
-
-  # Copy with_files inside the tmp directory, which will be the docker build
-  # context.
-  for f in args.with_files:
-    shutil.copy(f, temp_dir)
-    lines.append('COPY %s %s/' % (os.path.basename(f), args.with_file_dest))
-
-  # Create a Dockerfile.
-  with open(os.path.join(temp_dir, 'Dockerfile'), 'w') as f:
-     f.write('\n'.join(lines))
-
-  build_cmd = ['docker', 'build', '--rm', '--tag', new_tag, temp_dir]
-  subprocess.check_output(build_cmd)
-
-  if not args.keep:
-    atexit.register(lambda: subprocess.call(['docker', 'rmi', new_tag]))
-
-  # Upload to GCR.
-  if args.gcr_path:
-    subprocess.call(['gcloud', 'docker', '--', 'push', new_tag])
-
-  return True
-
-
-for image in args.images:
-  upload_to_gcr(image)
+# Dictionaries used for client matrix testing.
+
+def get_github_repo(lang):
+  return {
+      'go': 'git@github.com:grpc/grpc-go.git',
+      'java': 'git@github.com:grpc/grpc-java.git',
+  }.get(lang, 'git@github.com:grpc/grpc.git')
+
+# Dictionary of runtimes per language
+LANG_RUNTIME_MATRIX = {
+    'go': ['go1.7', 'go1.8'],
+}
+
+# Dictionary of releases per language.  For each language, we need to provide
+# a tuple of release tag (used as the tag for the GCR image) and also github hash.
+LANG_RELEASE_MATRIX = {
+    'go': ['v1.0.1-GA', 'v1.3.0'],
+}

tools/interop_matrix/create_matrix_images.py

+#!/usr/bin/env python2.7
+# Copyright 2017, Google Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+"""Build and upload docker images to Google Container Registry per matrix."""
+
+from __future__ import print_function
+
+import argparse
+import atexit
+import multiprocessing
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+
+# Langauage Runtime Matrix
+import client_matrix
+
+python_util_dir = os.path.abspath(os.path.join(
+    os.path.dirname(__file__), '../run_tests/python_utils'))
+sys.path.append(python_util_dir)
+import dockerjob
+import jobset
+
+_IMAGE_BUILDER = 'tools/run_tests/dockerize/build_interop_image.sh'
+_LANGUAGES = client_matrix.LANG_RUNTIME_MATRIX.keys()
+# All gRPC release tags, flattened, deduped and sorted.
+_RELEASES = sorted(list(set(
+    i for l in client_matrix.LANG_RELEASE_MATRIX.values() for i in l)))
+
+# Destination directory inside docker image to keep extra info from build time.
+_BUILD_INFO = '/var/local/build_info'
+
+argp = argparse.ArgumentParser(description='Run interop tests.')
+argp.add_argument('--gcr_path',
+                  default='gcr.io/grpc-testing',
+                  help='Path of docker images in Google Container Registry')
+
+argp.add_argument('--release',
+                  default='master',
+                  choices=['all', 'master'] + _RELEASES,
+                  help='github commit tag to checkout.  When building all '
+                  'releases defined in client_matrix.py, use "all". Valid only '
+                  'with --git_checkout.')
+
+argp.add_argument('-l', '--language',
+                  choices=['all'] + sorted(_LANGUAGES),
+                  nargs='+',
+                  default=['all'],
+                  help='Test languages to build docker images for.')
+
+argp.add_argument('--git_checkout',
+                  action='store_true',
+                  help='Use a separate git clone tree for building grpc stack. '
+                  'Required when using --release flag.  By default, current'
+                  'tree and the sibling will be used for building grpc stack.')
+
+argp.add_argument('--git_checkout_root',
+                  default='/export/hda3/tmp/grpc_matrix',
+                  help='Directory under which grpc-go/java/main repo will be '
+                  'cloned.  Valid only with --git_checkout.')
+
+argp.add_argument('--keep',
+                  action='store_true',
+                  help='keep the created local images after uploading to GCR')
+
+
+args = argp.parse_args()
+
+def add_files_to_image(image, with_files, label=None):
+  """Add files to a docker image.
+
+  image: docker image name, i.e. grpc_interop_java:26328ad8
+  with_files: additional files to include in the docker image.
+  label: label string to attach to the image.
+  """
+  tag_idx = image.find(':')
+  if tag_idx == -1:
+    jobset.message('FAILED', 'invalid docker image %s' % image, do_newline=True)
+    sys.exit(1)
+  orig_tag = '%s_' % image
+  subprocess.check_output(['docker', 'tag', image, orig_tag])
+
+  lines = ['FROM ' + orig_tag]
+  if label:
+    lines.append('LABEL %s' % label)
+
+  temp_dir = tempfile.mkdtemp()
+  atexit.register(lambda: subprocess.call(['rm', '-rf', temp_dir]))
+
+  # Copy with_files inside the tmp directory, which will be the docker build
+  # context.
+  for f in with_files:
+    shutil.copy(f, temp_dir)
+    lines.append('COPY %s %s/' % (os.path.basename(f), _BUILD_INFO))
+
+  # Create a Dockerfile.
+  with open(os.path.join(temp_dir, 'Dockerfile'), 'w') as f:
+    f.write('\n'.join(lines))
+
+  jobset.message('START', 'Repackaging %s' % image, do_newline=True)
+  build_cmd = ['docker', 'build', '--rm', '--tag', image, temp_dir]
+  subprocess.check_output(build_cmd)
+  dockerjob.remove_image(orig_tag, skip_nonexistent=True)
+
+def build_image_jobspec(runtime, env, gcr_tag):
+  """Build interop docker image for a language with runtime.
+
+  runtime: a <lang><version> string, for example go1.8.
+  env:     dictionary of env to passed to the build script.
+  gcr_tag: the tag for the docker image (i.e. v1.3.0).
+  """
+  basename = 'grpc_interop_%s' % runtime
+  tag = '%s/%s:%s' % (args.gcr_path, basename, gcr_tag)
+  build_env = {
+      'INTEROP_IMAGE': tag,
+      'BASE_NAME': basename,
+      'TTY_FLAG': '-t'
+  }
+  build_env.update(env)
+  build_job = jobset.JobSpec(
+          cmdline=[_IMAGE_BUILDER],
+          environ=build_env,
+          shortname='build_docker_%s' % runtime,
+          timeout_seconds=30*60)
+  build_job.tag = tag
+  return build_job
+
+def build_all_images_for_lang(lang):
+  """Build all docker images for a language across releases and runtimes."""
+  if not args.git_checkout:
+    if args.release != 'master':
+      print('WARNING: --release is set but will be ignored\n')
+    releases = ['master']
+  else:
+    if args.release == 'all':
+      releases = client_matrix.LANG_RELEASE_MATRIX[lang]
+    else:
+      # Build a particular release.
+      if args.release not in ['master'] + client_matrix.LANG_RELEASE_MATRIX[lang]:
+        jobset.message('SKIPPED',
+                       '%s for %s is not defined' % (args.release, lang),
+                       do_newline=True)
+        return []
+      releases = [args.release]
+
+  images = []
+  for release in releases:
+    images += build_all_images_for_release(lang, release)
+  jobset.message('SUCCESS',
+                 'All docker images built for %s at %s.' % (lang, releases),
+                 do_newline=True)
+  return images
+
+def build_all_images_for_release(lang, release):
+  """Build all docker images for a release across all runtimes."""
+  docker_images = []
+  build_jobs = []
+
+  env = {}
+  # If we not using current tree or the sibling for grpc stack, do checkout.
+  if args.git_checkout:
+    stack_base = checkout_grpc_stack(lang, release)
+    var ={'go': 'GRPC_GO_ROOT', 'java': 'GRPC_JAVA_ROOT'}.get(lang, 'GRPC_ROOT')
+    env[var] = stack_base
+
+  for runtime in client_matrix.LANG_RUNTIME_MATRIX[lang]:
+    job = build_image_jobspec(runtime, env, release)
+    docker_images.append(job.tag)
+    build_jobs.append(job)
+
+  jobset.message('START', 'Building interop docker images.', do_newline=True)
+  print('Jobs to run: \n%s\n' % '\n'.join(str(j) for j in build_jobs))
+
+  num_failures, _ = jobset.run(
+      build_jobs, newline_on_success=True, maxjobs=multiprocessing.cpu_count())
+  if num_failures:
+    jobset.message('FAILED', 'Failed to build interop docker images.',
+                   do_newline=True)
+    docker_images_cleanup.extend(docker_images)
+    sys.exit(1)
+
+  jobset.message('SUCCESS',
+                 'All docker images built for %s at %s.' % (lang, release),
+                 do_newline=True)
+
+  if release != 'master':
+    commit_log = os.path.join(stack_base, 'commit_log')
+    if os.path.exists(commit_log):
+      for image in docker_images:
+        add_files_to_image(image, [commit_log], 'release=%s' % release)
+  return docker_images
+
+def cleanup():
+  if not args.keep:
+    for image in docker_images_cleanup:
+      dockerjob.remove_image(image, skip_nonexistent=True)
+
+docker_images_cleanup = []
+atexit.register(cleanup)
+
+def checkout_grpc_stack(lang, release):
+  """Invokes 'git check' for the lang/release and returns directory created."""
+  assert args.git_checkout and args.git_checkout_root
+
+  if not os.path.exists(args.git_checkout_root):
+    os.makedirs(args.git_checkout_root)
+
+  repo = client_matrix.get_github_repo(lang)
+  # Get the subdir name part of repo
+  # For example, 'git@github.com:grpc/grpc-go.git' should use 'grpc-go'.
+  repo_dir = os.path.splitext(os.path.basename(repo))[0]
+  stack_base = os.path.join(args.git_checkout_root, repo_dir)
+
+  # Assume the directory is reusable for git checkout.
+  if not os.path.exists(stack_base):
+    subprocess.check_call(['git', 'clone', '--recursive', repo],
+                          cwd=os.path.dirname(stack_base))
+
+  # git checkout.
+  jobset.message('START', 'git checkout %s from %s' % (release, stack_base),
+                 do_newline=True)
+  # We should NEVER do checkout on current tree !!!
+  assert not os.path.dirname(__file__).startswith(stack_base)
+  output = subprocess.check_output(
+      ['git', 'checkout', release], cwd=stack_base, stderr=subprocess.STDOUT)
+  commit_log = subprocess.check_output(['git', 'log', '-1'], cwd=stack_base)
+  jobset.message('SUCCESS', 'git checkout', output + commit_log, do_newline=True)
+
+  # Write git log to commit_log so it can be packaged with the docker image.
+  with open(os.path.join(stack_base, 'commit_log'), 'w') as f:
+    f.write(commit_log)
+  return stack_base
+
+languages = args.language if args.language != ['all'] else _LANGUAGES
+for lang in languages:
+  docker_images = build_all_images_for_lang(lang)
+  for image in docker_images:
+    jobset.message('START', 'Uploading %s' % image, do_newline=True)
+    # docker image name must be in the format <gcr_path>/<image>:<gcr_tag>
+    assert image.startswith(args.gcr_path) and image.find(':') != -1
+
+    # subprocess.call(['gcloud', 'docker', '--', 'push', image])
+    subprocess.call(['gcloud', 'docker', '--', 'push', image])

tools/interop_matrix/create_testcases.sh

+%#!/bin/bash
+#% Copyright 2017, Google Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+# Creates test cases for a language by running run_interop_test in manual mode
+# and save the generated output under ./testcases/<lang>__<release>.
+#
+# Params:
+#   LANG - The language.
+#   SKIP_TEST - If set, skip running the test cases for sanity.
+#   RELEASE - Create testcase for specific release, defautl to 'master'.
+#   KEEP_IMAGE - Do not clean local docker image created for the test cases.
+
+set -e
+
+cd $(dirname $0)/../..
+GRPC_ROOT=$(pwd)
+CMDS_SH="${GRPC_ROOT}/interop_client_cmds.sh"
+TESTCASES_DIR=${GRPC_ROOT}/tools/interop_matrix/testcases
+
+echo "Create '$LANG' test cases for gRPC release '${RELEASE:=master}'"
+
+# Invoke run_interop_test in manual mode.
+${GRPC_ROOT}/tools/run_tests/run_interop_tests.py -l $LANG --use_docker \
+  --cloud_to_prod --manual_run
+
+# Clean up
+function cleanup {
+  [ -z "$testcase" ] && testcase=$CMDS_SH
+  echo "testcase: $testcase"
+  if [ -e $testcase ]; then
+    # The script should generate a line with "${docker_image:=...}".
+    eval docker_image=$(grep -oe '${docker_image:=.*}' $testcase)
+    if [ -z "$KEEP_IMAGE" ]; then
+      echo "Clean up docker_image $docker_image"
+      docker rmi -f $docker_image
+    else
+      echo "Kept docker_image $docker_image"
+    fi
+  fi
+  [ -e "$CMDS_SH" ] && rm $CMDS_SH
+}
+trap cleanup EXIT
+# Running the testcases as sanity unless we are asked to skip.
+[ -z "$SKIP_TEST" ] && (echo "Running test cases: $CMDS_SH"; sh $CMDS_SH)
+
+mkdir -p $TESTCASES_DIR
+testcase=$TESTCASES_DIR/${LANG}__$RELEASE
+if [ -e $testcase ]; then
+  echo "Updating: $testcase"
+  diff $testcase $CMDS_SH || true
+fi
+mv $CMDS_SH $testcase
+chmod a+x $testcase
+echo "Test cases created: $testcase"

tools/interop_matrix/testcases/go__master

+#!/bin/bash
+echo "Testing ${docker_image:=grpc_interop_go:41fffd01-a6c8-41b6-8136-c0aaa1ec2437}"
+docker run -i --rm=true -w /go/src/google.golang.org/grpc/interop/client --net=host $docker_image bash -c "go run client.go --server_host=216.239.32.254 --server_host_override=grpc-test.sandbox.googleapis.com --server_port=443 --use_tls=true --test_case=large_unary"
+docker run -i --rm=true -w /go/src/google.golang.org/grpc/interop/client --net=host $docker_image bash -c "go run client.go --server_host=216.239.32.254 --server_host_override=grpc-test.sandbox.googleapis.com --server_port=443 --use_tls=true --test_case=empty_unary"
+docker run -i --rm=true -w /go/src/google.golang.org/grpc/interop/client --net=host $docker_image bash -c "go run client.go --server_host=216.239.32.254 --server_host_override=grpc-test.sandbox.googleapis.com --server_port=443 --use_tls=true --test_case=ping_pong"
+docker run -i --rm=true -w /go/src/google.golang.org/grpc/interop/client --net=host $docker_image bash -c "go run client.go --server_host=216.239.32.254 --server_host_override=grpc-test.sandbox.googleapis.com --server_port=443 --use_tls=true --test_case=empty_stream"
+docker run -i --rm=true -w /go/src/google.golang.org/grpc/interop/client --net=host $docker_image bash -c "go run client.go --server_host=216.239.32.254 --server_host_override=grpc-test.sandbox.googleapis.com --server_port=443 --use_tls=true --test_case=client_streaming"
+docker run -i --rm=true -w /go/src/google.golang.org/grpc/interop/client --net=host $docker_image bash -c "go run client.go --server_host=216.239.32.254 --server_host_override=grpc-test.sandbox.googleapis.com --server_port=443 --use_tls=true --test_case=server_streaming"
+docker run -i --rm=true -w /go/src/google.golang.org/grpc/interop/client --net=host $docker_image bash -c "go run client.go --server_host=216.239.32.254 --server_host_override=grpc-test.sandbox.googleapis.com --server_port=443 --use_tls=true --test_case=cancel_after_begin"
+docker run -i --rm=true -w /go/src/google.golang.org/grpc/interop/client --net=host $docker_image bash -c "go run client.go --server_host=216.239.32.254 --server_host_override=grpc-test.sandbox.googleapis.com --server_port=443 --use_tls=true --test_case=cancel_after_first_response"
+docker run -i --rm=true -w /go/src/google.golang.org/grpc/interop/client --net=host $docker_image bash -c "go run client.go --server_host=216.239.32.254 --server_host_override=grpc-test.sandbox.googleapis.com --server_port=443 --use_tls=true --test_case=timeout_on_sleeping_server"

tools/run_tests/dockerize/build_interop_image.sh

@@ -39,21 +39,24 @@ set -x
 #  TTY_FLAG - optional -t flag to make docker allocate tty
 #  BUILD_INTEROP_DOCKER_EXTRA_ARGS - optional args to be passed to the
 #    docker run command
+#  GRPC_ROOT - grpc base directory, default to top of this tree.
+#  GRPC_GO_ROOT - grpc-go base directory, default to '$GRPC_ROOT/../grpc-go'
+#  GRPC_JAVA_ROOT - grpc-java base directory, default to '$GRPC_ROOT/../grpc-java'
 
 cd `dirname $0`/../../..
-GRPC_ROOT=`pwd`
+echo "GRPC_ROOT: ${GRPC_ROOT:=$(pwd)}"
 MOUNT_ARGS="-v $GRPC_ROOT:/var/local/jenkins/grpc:ro"
 
-GRPC_JAVA_ROOT=`cd ../grpc-java && pwd`
-if [ "$GRPC_JAVA_ROOT" != "" ]
+echo "GRPC_JAVA_ROOT: ${GRPC_JAVA_ROOT:=$(cd ../grpc-java && pwd)}"
+if [ -n "$GRPC_JAVA_ROOT" ]
 then
   MOUNT_ARGS+=" -v $GRPC_JAVA_ROOT:/var/local/jenkins/grpc-java:ro"
 else
   echo "WARNING: grpc-java not found, it won't be mounted to the docker container."
 fi
 
-GRPC_GO_ROOT=`cd ../grpc-go && pwd`
-if [ "$GRPC_GO_ROOT" != "" ]
+echo "GRPC_GO_ROOT: ${GRPC_GO_ROOT:=$(cd ../grpc-go && pwd)}"
+if [ -n "$GRPC_GO_ROOT" ]
 then
   MOUNT_ARGS+=" -v $GRPC_GO_ROOT:/var/local/jenkins/grpc-go:ro"
 else

tools/run_tests/run_interop_tests.py

@@ -544,12 +544,14 @@ def docker_run_cmdline(cmdline, image, docker_args=[], cwd=None, environ=None):
   return docker_cmdline
 
 
-def manual_cmdline(docker_cmdline):
+def manual_cmdline(docker_cmdline, docker_image):
   """Returns docker cmdline adjusted for manual invocation."""
   print_cmdline = []
   for item in docker_cmdline:
     if item.startswith('--name='):
       continue
+    if item == docker_image:
+      item = "$docker_image"
     # add quotes when necessary
     if any(character.isspace() for character in item):
       item = "\"%s\"" % item
@@ -644,7 +646,9 @@ def cloud_to_prod_jobspec(language, test_case, server_host_name,
                                  docker_args=['--net=host',
                                               '--name=%s' % container_name])
     if manual_cmd_log is not None:
-      manual_cmd_log.append(manual_cmdline(cmdline))
+      if manual_cmd_log == []:
+        manual_cmd_log.append('echo "Testing ${docker_image:=%s}"' % docker_image)
+      manual_cmd_log.append(manual_cmdline(cmdline, docker_image))
     cwd = None
     environ = None
 
@@ -710,7 +714,9 @@ def cloud_to_cloud_jobspec(language, test_case, server_name, server_host,
                                  docker_args=['--net=host',
                                               '--name=%s' % container_name])
     if manual_cmd_log is not None:
-      manual_cmd_log.append(manual_cmdline(cmdline))
+      if manual_cmd_log == []:
+        manual_cmd_log.append('echo "Testing ${docker_image:=%s}"' % docker_image)
+      manual_cmd_log.append(manual_cmdline(cmdline, docker_iamge))
     cwd = None
 
   test_job = jobset.JobSpec(
@@ -770,7 +776,9 @@ def server_jobspec(language, docker_image, insecure=False, manual_cmd_log=None):
                                       environ=environ,
                                       docker_args=docker_args)
   if manual_cmd_log is not None:
-      manual_cmd_log.append(manual_cmdline(docker_cmdline))
+      if manual_cmd_log == []:
+        manual_cmd_log.append('echo "Testing ${docker_image:=%s}"' % docker_image)
+      manual_cmd_log.append(manual_cmdline(docker_cmdline, docker_iamge))
   server_job = jobset.JobSpec(
           cmdline=docker_cmdline,
           environ=environ,