Add GSoC report (Naresh)

summerofcode/2018/naresh.md

+# Project overview
+
+## Title
+
+Enable Building of gRPC Python with Bazel
+
+## Overview
+
+gRPC Python currently has a constellation of scripts written to build the
+project, but it has a lot of limitations in terms of speed and maintainability.
+[Bazel](https://bazel.build/) is the open-sourced variant of Google's internal
+system, Blaze, which is an ideal replacement for building such projects in a
+fast and declarative fashion. But Bazel in itself is still in active
+development, especially in terms of Python (amongst a few other languages).
+
+The project aimed to fill this gap and build gRPC Python with Bazel.
+
+[Project page](https://summerofcode.withgoogle.com/projects/#6482576244473856)
+
+[Link to proposal](https://storage.googleapis.com/summerofcode-prod.appspot.com/gsoc/core_project/doc/5316764725411840_1522049732_Naresh_Ramesh_-_GSoC_proposal.pdf)
+
+## Thoughts and challenges
+
+### State of Bazel for Python
+
+Although previously speculated, the project didn't require any contributions
+directly to [bazelbuild/bazel](https://github.com/bazelbuild/bazel). The Bazel
+rules for Python are currently being separated out into their own repo at
+[bazelbuild/rules_python](https://github.com/bazelbuild/rules_python/).
+
+Bazel is [still very much in active development for
+Python](https://groups.google.com/forum/#!topic/bazel-sig-python/iQjV9sfSufw)
+though. There's still challenges when it comes to building for Python 2 vs 3.
+Using pip packages is still in experimental. Bazel Python support is currently
+distributed across these two repositories and is yet to begin migration to one
+place (which will be
+[bazelbuild/rules_python](https://github.com/bazelbuild/rules_python/)).
+
+Bazel's roadmap for Python is publicly available [here as a Google
+doc](https://docs.google.com/document/d/1A6J3j3y1SQ0HliS86_mZBnB5UeBe7vExWL2Ryd_EONI/edit).
+
+### Cross collaboration between projects
+
+Cross contribution surprisingly came up because of building protobuf sources
+for Python, which is still not natively supported by Bazel. An existing
+repository, [pubref/rules_protobuf](https://github.com/pubref/rules_protobuf),
+which was maintained by an independent maintainer (i.e. not a part of Bazel)
+helped solve this problem, but had [one major blocking
+issue](https://github.com/pubref/rules_protobuf/issues/233) and could not be
+resolved at the source. But [a solution to the
+issue](https://github.com/pubref/rules_protobuf/pull/196) was proposed by user
+dududko, which was not merged because of failing golang tests but worked well
+for Python. Hence, a fork of this repo was made and is to be used with gRPC
+until the solution can be merged back at the source.
+
+### Building Cython code
+
+Building Cython code is still not supported by Bazel, but the team at
+[cython/cython](https://github.com/cython/cython) have added support for Bazel
+on their side. The way it works is by including Cython as a third-party Bazel
+dependency and using custom Bazel rules for building our Cython code using the
+binary within the dependency.
+
+### Packaging Python code using Bazel
+
+pip and PyPI still remain the de-facto standard for distributing Python
+packages. Although Bazel is pretty versatile and is amazing for it's
+reproducible and incremental build capabilities, these can only be still used
+by the contributors and developers for building and testing the gRPC code. But
+there's no way yet to build Python packages for distribution.
+
+### Building gRPC Python with Bazel on Kokoro (internal CI)
+
+Integration with the internal CI was one of the areas that highlighted how
+simple Bazel can be to use. gRPC was already using a dockerized Bazel setup to
+build some of it's core code (but not as the primary build setup). Adding a new
+job on the internal CI ended up being as simple as creating a new shell script
+to install the required dependencies (which were python-dev and Bazel) and a
+new configuration file which pointed to the subdirectiory (src/python) under
+which to look for targets and run the tests accordingly.
+
+### Handling imports in Python code
+
+When writing Python packages, imports in nested modules are typically made
+relative to the package root. But because of the way Bazel works, these paths
+wouldn't make sense from the Workspace root. So, the folks at Bazel have added
+a nifty `imports` parameter to all the Python rules which lets us specify for
+each target, which path to consider as the root. This parameter allows for
+relative paths like `imports = ["../",]`.
+
+### Fetching Python headers for Cython code to use
+
+Cython code makes use of `Python.h`, which pulls in the Python API for C
+extension modules to use, but it's location depending on the Python version and
+operating system the code is building on. To make this easier, the folks at
+Tensorflow wrote [repository rules for Python
+autoconfiguration](https://github.com/tensorflow/tensorflow/tree/e447ae4759317156d31a9421290716f0ffbffcd8/third_party/py).
+This has been [adapted with some some
+modifications](https://github.com/grpc/grpc/pull/15992) for use in gRPC Python
+as well.
+
+## How to use
+
+All the Bazel tests for gRPC Python can be run using a single command:
+
+```bash
+bazel test --spawn_strategy=standalone --genrule_strategy=standalone //src/python/...
+```
+
+If any specific test is to be run, like say `LoggingPoolTest` (which is present
+in
+`src/python/grpcio_tests/tests/unit/framework/foundation/_logging_pool_test.py`),
+the command to run would be:
+
+```bash
+bazel test --spawn_strategy=standalone --genrule_strategy=standalone //src/python/grpcio_tests/tests/unit/framework/foundation:logging_pool_test
+```
+
+where, `logging_pool_test` is the name of the Bazel target for this test.
+
+Similarly, to run a particular method, use:
+
+```bash
+bazel test --spawn_strategy=standalone --genrule_strategy=standalone //src/python/grpcio_tests/tests/unit/_rpc_test --test_arg=RPCTest.testUnrecognizedMethod
+```
+
+## Useful Bazel flags
+
+- Use `bazel build` with a `-s` flag to see the logs being printed out to
+    standard output while building. 
+- Similarly, use `bazel test` with a `--test_output=streamed` to see the the
+    test logs while testing. Something to know while using this flag is that all
+    tests will be run locally, without sharding, one at a time.
+
+## Contributions
+
+### Related to the project
+
+- [435c6f8](https://github.com/grpc/grpc/commit/435c6f8d1e53783ec049b3482445813afd8bc514)
+    Update grpc_gevent cython files to include .pxi
+- [74426fd](https://github.com/grpc/grpc/commit/74426fd2164c51d6754732ebe372133c19ba718c)
+    Add gevent_util.h to grpc_base_c Bazel target
+- [b6518af](https://github.com/grpc/grpc/commit/b6518afdd610f0115b42aee1ffc71520c6b0d6b1)
+    Upgrade Bazel to 0.15.0
+- [ebcf04d](https://github.com/grpc/grpc/commit/ebcf04d075333c42979536c5dd2091d363f67e5a)
+    Kokoro setup for building gRPC Python with Bazel
+- [3af1aaa](https://github.com/grpc/grpc/commit/3af1aaadabf49bc6274711a11f81627c0f351a9a)
+    Basic setup to build gRPC Python with Bazel
+- [11f199e](https://github.com/grpc/grpc/commit/11f199e34dc416a2bd8b56391b242a867bedade4)
+    Workspace changes to build gRPC Python with Bazel
+- [848fd9d](https://github.com/grpc/grpc/commit/848fd9d75f6df10f00e8328ff052c0237b3002ab)
+    Minimal Bazel BUILD files for grpcio Python
+
+### Other contibutions
+
+- [89ce16b](https://github.com/grpc/grpc/commit/89ce16b6daaad4caeb1c9ba670c6c4b62ea1a93c)
+    Update Dockerfiles for python artifacts to use latest git version
+- [32f7c48](https://github.com/grpc/grpc/commit/32f7c48dad71cac7af652bf994ab1dde3ddb0607)
+    Revert removals from python artifact dockerfiles
+- [712eb9f](https://github.com/grpc/grpc/commit/712eb9ff91cde66af94e8381ec01ad512ed6d03c)
+    Make logging after success in jobset more apparent
+- [c6e4372](https://github.com/grpc/grpc/commit/c6e4372f8a93bb0eb996b5f202465785422290f2)
+    Create README for gRPC Python reflection package
+- [2e113ca](https://github.com/grpc/grpc/commit/2e113ca6b2cc31aa8a9687d40ee1bd759381654f)
+    Update logging in Python to use module-level logger
+
+### Pending PRs
+
+- BUILD files for all tests in
+    [tests.json](https://github.com/ghostwriternr/grpc/blob/70c8a58b2918a5369905e5a203d7ce7897b6207e/src/python/grpcio_tests/tests/tests.json).
+- BUILD files for gRPC testing, gRPC health checking, gRPC reflection.
+- (Yet to complete) BUILD files for grpcio_tools. One test depends on this.
+
+## Known issues
+
+- [grpc/grpc #16336](https://github.com/grpc/grpc/issues/16336) RuntimeError
+    for `_reconnect_test` Python unit test with Bazel
+- Some tests in Bazel pass despite throwing an exception. Example:
+    `testAbortedStreamStream` in
+    `src/python/grpcio_tests/tests/unit/_metadata_code_details_test.py`.
+- [#14557](https://github.com/grpc/grpc/pull/14557) introduced a minor bug
+    where the module level loggers don't initialize a default logging handler.
+- Sanity test doesn't make sense in the context of Bazel, and thus fails.
+- There are some issues with Python2 vs Python3. Specifically,
+  - On some machines, “cygrpc.so: undefined symbol: _Py_FalseStruct” error
+    shows up. This is because of incorrect Python version being used to build
+    Cython.
+  - Some external packages like enum34 throw errors when used with Python 3 and
+    some extra packages are currently installed as Python version in current
+    build scripts. For now, the extra packages are added to a
+    `requirements.bazel.txt` file in the repository root.