diff --git a/README.md b/README.md
index f15a08ccffde160e639d94068a3981833e1a8c84..43c44a4f42f36318af9c20bd82ac0a9147c7e52a 100644
--- a/README.md
+++ b/README.md
@@ -4,347 +4,24 @@ Gitlab-workhorse is a smart reverse proxy for GitLab. It handles
"large" HTTP requests such as file downloads, file uploads, Git
push/pull and Git archive downloads.
-## Features that rely on Workhorse
+Workhorse itself is not a feature, but there are [several features in
+GitLab](doc/gitlab_features.md) that would not work efficiently without Workhorse.
-Workhorse itself is not a feature, but there are several features in
-GitLab that would not work efficiently without Workhorse.
+## Documentation
-To put the efficiency benefit in context, consider that in 2020Q3 on GitLab.com [we see][thanos] Rails application threads using on average about 200MB of RSS vs about 200KB for Workhorse goroutines.
+Workhorse documentation is available in the [`doc` folder of this repostitory](doc/).
-[thanos]: https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sum(ruby_process_resident_memory_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%20%2F%20sum(puma_max_threads%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g0.tab=1&g1.range_input=1h&g1.max_source_resolution=0s&g1.expr=sum(go_memstats_sys_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%2Fsum(go_goroutines%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g1.tab=1
+* Architectural overview
+ * [GitLab features that rely on Workhorse](doc/architecture/gitlab_features.md)
+ * [Websocket channel support](doc/architecture/channel.md)
+* Operating workhorse
+ * [Source installation](doc/operations/install.md)
+ * [Workhorse configuration](doc/operations/configuration.md)
+* [Contributing](CONTRIBUTING.md)
+ * [Testing your code](doc/development/tests.md)
-Examples of features that rely on Workhorse:
-
-### 1. `git clone` and `git push` over HTTP
-
-Git clone, pull and push are slow because they transfer large amounts
-of data and because each is CPU intensive on the GitLab side. Without
-workhorse, HTTP access to Git repositories would compete with regular
-web access to the application, requiring us to run way more Rails
-application servers.
-
-### 2. CI runner long polling
-
-GitLab CI runners fetch new CI jobs by polling the GitLab server.
-Workhorse acts as a kind of "waiting room" where CI runners can sit
-and wait for new CI jobs. Because of Go's efficiency we can fit a lot
-of runners in the waiting room at little cost. Without this waiting
-room mechanism we would have to add a lot more Rails server capacity.
-
-### 3. File uploads and downloads
-
-File uploads and downloads may be slow either because the file is
-large or because the user's connection is slow. Workhorse can handle
-the slow part for Rails. This improves the efficiency of features such
-as CI artifacts, package repositories, LFS objects, etc.
-
-### 4. Websocket proxying
-
-Features such as the web terminal require a long lived connection
-between the user's web browser and a container inside GitLab that is
-not directly accessible from the internet. Dedicating a Rails
-application thread to proxying such a connection would cost much more
-memory than it costs to have Workhorse look after it.
-
-## Quick facts (how does Workhorse work)
-
-- Workhorse can handle some requests without involving Rails at all:
- for example, JavaScript files and CSS files are served straight
- from disk.
-- Workhorse can modify responses sent by Rails: for example if you use
- `send_file` in Rails then gitlab-workhorse will open the file on
- disk and send its contents as the response body to the client.
-- Workhorse can take over requests after asking permission from Rails.
- Example: handling `git clone`.
-- Workhorse can modify requests before passing them to Rails. Example:
- when handling a Git LFS upload Workhorse first asks permission from
- Rails, then it stores the request body in a tempfile, then it sends
- a modified request containing the tempfile path to Rails.
-- Workhorse can manage long-lived WebSocket connections for Rails.
- Example: handling the terminal websocket for environments.
-- Workhorse does not connect to Postgres, only to Rails and (optionally) Redis.
-- We assume that all requests that reach Workhorse pass through an
- upstream proxy such as NGINX or Apache first.
-- Workhorse does not accept HTTPS connections.
-- Workhorse does not clean up idle client connections.
-- We assume that all requests to Rails pass through Workhorse.
-
-For more information see ['A brief history of
-gitlab-workhorse'][brief-history-blog].
-
-## Usage
-
-```
- gitlab-workhorse [OPTIONS]
-
-Options:
- -apiCiLongPollingDuration duration
- Long polling duration for job requesting for runners (default 50s - enabled) (default 50ns)
- -apiLimit uint
- Number of API requests allowed at single time
- -apiQueueDuration duration
- Maximum queueing duration of requests (default 30s)
- -apiQueueLimit uint
- Number of API requests allowed to be queued
- -authBackend string
- Authentication/authorization backend (default "http://localhost:8080")
- -authSocket string
- Optional: Unix domain socket to dial authBackend at
- -cableBackend string
- Optional: ActionCable backend (default authBackend)
- -cableSocket string
- Optional: Unix domain socket to dial cableBackend at (default authSocket)
- -config string
- TOML file to load config from
- -developmentMode
- Allow the assets to be served from Rails app
- -documentRoot string
- Path to static files content (default "public")
- -listenAddr string
- Listen address for HTTP server (default "localhost:8181")
- -listenNetwork string
- Listen 'network' (tcp, tcp4, tcp6, unix) (default "tcp")
- -listenUmask int
- Umask for Unix socket
- -logFile string
- Log file location
- -logFormat string
- Log format to use defaults to text (text, json, structured, none) (default "text")
- -pprofListenAddr string
- pprof listening address, e.g. 'localhost:6060'
- -prometheusListenAddr string
- Prometheus listening address, e.g. 'localhost:9229'
- -proxyHeadersTimeout duration
- How long to wait for response headers when proxying the request (default 5m0s)
- -secretPath string
- File with secret key to authenticate with authBackend (default "./.gitlab_workhorse_secret")
- -version
- Print version and exit
-```
-
-The 'auth backend' refers to the GitLab Rails application. The name is
-a holdover from when gitlab-workhorse only handled Git push/pull over
-HTTP.
-
-Gitlab-workhorse can listen on either a TCP or a Unix domain socket. It
-can also open a second listening TCP listening socket with the Go
-[net/http/pprof profiler server](http://golang.org/pkg/net/http/pprof/).
-
-Gitlab-workhorse can listen on redis events (currently only builds/register
-for runners). This requires you to pass a valid TOML config file via
-`-config` flag.
-For regular setups it only requires the following (replacing the string
-with the actual socket)
-
-### Redis
-
-Gitlab-workhorse integrates with Redis to do long polling for CI build
-requests. This is configured via two things:
-
-- Redis settings in the TOML config file
-- The `-apiCiLongPollingDuration` command line flag to control polling
- behavior for CI build requests
-
-It is OK to enable Redis in the config file but to leave CI polling
-disabled; this just results in an idle Redis pubsub connection. The
-opposite is not possible: CI long polling requires a correct Redis
-configuration.
-
-Below we discuss the options for the `[redis]` section in the config
-file.
-
-```
-[redis]
-URL = "unix:///var/run/gitlab/redis.sock"
-Password = "my_awesome_password"
-Sentinel = [ "tcp://sentinel1:23456", "tcp://sentinel2:23456" ]
-SentinelMaster = "mymaster"
-```
-
-- `URL` takes a string in the format `unix://path/to/redis.sock` or
-`tcp://host:port`.
-- `Password` is only required if your redis instance is password-protected
-- `Sentinel` is used if you are using Sentinel.
- *NOTE* that if both `Sentinel` and `URL` are given, only `Sentinel` will be used
-
-Optional fields are as follows:
-```
-[redis]
-DB = 0
-ReadTimeout = "1s"
-KeepAlivePeriod = "5m"
-MaxIdle = 1
-MaxActive = 1
-```
-
-- `DB` is the Database to connect to. Defaults to `0`
-- `ReadTimeout` is how long a redis read-command can take. Defaults to `1s`
-- `KeepAlivePeriod` is how long the redis connection is to be kept alive without anything flowing through it. Defaults to `5m`
-- `MaxIdle` is how many idle connections can be in the redis-pool at once. Defaults to 1
-- `MaxActive` is how many connections the pool can keep. Defaults to 1
-
-### Relative URL support
-
-If you are mounting GitLab at a relative URL, e.g.
-`example.com/gitlab`, then you should also use this relative URL in
-the `authBackend` setting:
-
-```
-gitlab-workhorse -authBackend http://localhost:8080/gitlab
-```
-
-### Interaction of authBackend and authSocket
-
-The interaction between `authBackend` and `authSocket` can be a bit
-confusing. It comes down to: if `authSocket` is set it overrides the
-_host_ part of `authBackend` but not the relative path.
-
-In table form:
-
-|authBackend|authSocket|Workhorse connects to?|Rails relative URL|
-|---|---|---|---|
-|unset|unset|`localhost:8080`|`/`|
-|`http://localhost:3000`|unset|`localhost:3000`|`/`|
-|`http://localhost:3000/gitlab`|unset|`localhost:3000`|`/gitlab`|
-|unset|`/path/to/socket`|`/path/to/socket`|`/`|
-|`http://localhost:3000`|`/path/to/socket`|`/path/to/socket`|`/`|
-|`http://localhost:3000/gitlab`|`/path/to/socket`|`/path/to/socket`|`/gitlab`|
-
-The same applies to `cableBackend` and `cableSocket`.
-
-## Installation
-
-To install gitlab-workhorse you need [Go 1.13 or
-newer](https://golang.org/dl) and [GNU
-Make](https://www.gnu.org/software/make/).
-
-To install into `/usr/local/bin` run `make install`.
-
-```
-make install
-```
-
-To install into `/foo/bin` set the PREFIX variable.
-
-```
-make install PREFIX=/foo
-```
-
-On some operating systems, such as FreeBSD, you may have to use
-`gmake` instead of `make`.
-
-## Dependencies
-
-### Exiftool
-
-Workhorse uses [exiftool](https://www.sno.phy.queensu.ca/~phil/exiftool/) for
-removing EXIF data (which may contain sensitive information) from uploaded
-images. If you installed GitLab:
-
-- Using the Omnibus package, you're all set.
- *NOTE* that if you are using CentOS Minimal, you may need to install `perl`
- package: `yum install perl`
-- From source, make sure `exiftool` is installed:
-
- ```sh
- # Debian/Ubuntu
- sudo apt-get install libimage-exiftool-perl
-
- # RHEL/CentOS
- sudo yum install perl-Image-ExifTool
- ```
-
-## Error tracking
-
-GitLab-Workhorse supports remote error tracking with
-[Sentry](https://sentry.io). To enable this feature set the
-`GITLAB_WORKHORSE_SENTRY_DSN` environment variable.
-You can also set the `GITLAB_WORKHORSE_SENTRY_ENVIRONMENT` environment variable to
-use the Sentry environment functionality to separate staging, production and
-development.
-
-Omnibus (`/etc/gitlab/gitlab.rb`):
-
-```
-gitlab_workhorse['env'] = {
- 'GITLAB_WORKHORSE_SENTRY_DSN' => 'https://foobar'
- 'GITLAB_WORKHORSE_SENTRY_ENVIRONMENT' => 'production'
-}
-```
-
-Source installations (`/etc/default/gitlab`):
-
-```
-export GITLAB_WORKHORSE_SENTRY_DSN='https://foobar'
-export GITLAB_WORKHORSE_SENTRY_ENVIRONMENT='production'
-```
-
-## Tests
-
-Run the tests with:
-
-```
-make clean test
-```
-
-### Coverage / what to test
-
-Each feature in gitlab-workhorse should have an integration test that
-verifies that the feature 'kicks in' on the right requests and leaves
-other requests unaffected. It is better to also have package-level tests
-for specific behavior but the high-level integration tests should have
-the first priority during development.
-
-It is OK if a feature is only covered by integration tests.
-
-## Distributed Tracing
-
-Workhorse supports distributed tracing through [LabKit][] using [OpenTracing APIs](https://opentracing.io).
-
-By default, no tracing implementation is linked into the binary, but different OpenTracing providers can be linked in using [build tags][build-tags]/[build constraints][build-tags]. This can be done by setting the `BUILD_TAGS` make variable.
-
-For more details of the supported providers, see LabKit, but as an example, for Jaeger tracing support, include the tags: `BUILD_TAGS="tracer_static tracer_static_jaeger"`.
-
-```shell
-make BUILD_TAGS="tracer_static tracer_static_jaeger"
-```
-
-Once Workhorse is compiled with an opentracing provider, the tracing configuration is configured via the `GITLAB_TRACING` environment variable.
-
-For example:
-
-```shell
-GITLAB_TRACING=opentracing://jaeger ./gitlab-workhorse
-```
-
-## Continuous Profiling
-
-Workhorse supports continuous profiling through [LabKit][] using [Stackdriver Profiler](https://cloud.google.com/profiler).
-
-By default, the Stackdriver Profiler implementation is linked in the binary using [build tags][build-tags], though it's not
-required and can be skipped.
-
-For example:
-
-```shell
-make BUILD_TAGS=""
-```
-
-Once Workhorse is compiled with Continuous Profiling, the profiler configuration can be set via `GITLAB_CONTINUOUS_PROFILING`
-environment variable.
-
-For example:
-
-```shell
-GITLAB_CONTINUOUS_PROFILING="stackdriver?service=workhorse&service_version=1.0.1&project_id=test-123 ./gitlab-workhorse"
-```
-
-More information about see the [LabKit monitoring docs](https://gitlab.com/gitlab-org/labkit/-/blob/master/monitoring/doc.go).
## License
This code is distributed under the MIT license, see the [LICENSE](LICENSE) file.
-[brief-history-blog]: https://about.gitlab.com/2016/04/12/a-brief-history-of-gitlab-workhorse/
-[LabKit]: https://gitlab.com/gitlab-org/labkit/
-[build-tags]: https://golang.org/pkg/go/build/#hdr-Build_Constraints
diff --git a/doc/architecture/channel.md b/doc/architecture/channel.md
new file mode 100644
index 0000000000000000000000000000000000000000..8423405a9cff1802e88e461555e77041cd7db7d7
--- /dev/null
+++ b/doc/architecture/channel.md
@@ -0,0 +1,194 @@
+# Websocket channel support
+
+In some cases, GitLab can provide in-browser terminal access to an
+environment (which is a running server or container, onto which a
+project has been deployed), or even access to services running in CI
+through a WebSocket. Workhorse manages the WebSocket upgrade and
+long-lived connection to the websocket connection, which frees
+up GitLab to process other requests.
+
+This document outlines the architecture of these connections.
+
+## Introduction to WebSockets
+
+A websocket is an "upgraded" HTTP/1.1 request. Their purpose is to
+permit bidirectional communication between a client and a server.
+**Websockets are not HTTP**. Clients can send messages (known as
+frames) to the server at any time, and vice-versa. Client messages
+are not necessarily requests, and server messages are not necessarily
+responses. WebSocket URLs have schemes like `ws://` (unencrypted) or
+`wss://` (TLS-secured).
+
+When requesting an upgrade to WebSocket, the browser sends a HTTP/1.1
+request that looks like this:
+
+```
+GET /path.ws HTTP/1.1
+Connection: upgrade
+Upgrade: websocket
+Sec-WebSocket-Protocol: terminal.gitlab.com
+# More headers, including security measures
+```
+
+At this point, the connection is still HTTP, so this is a request and
+the server can send a normal HTTP response, including `404 Not Found`,
+`500 Internal Server Error`, etc.
+
+If the server decides to permit the upgrade, it will send a HTTP
+`101 Switching Protocols` response. From this point, the connection
+is no longer HTTP. It is a WebSocket and frames, not HTTP requests,
+will flow over it. The connection will persist until the client or
+server closes the connection.
+
+In addition to the subprotocol, individual websocket frames may
+also specify a message type - examples include `BinaryMessage`,
+`TextMessage`, `Ping`, `Pong` or `Close`. Only binary frames can
+contain arbitrary data - other frames are expected to be valid
+UTF-8 strings, in addition to any subprotocol expectations.
+
+## Browser to Workhorse
+
+Using the terminal as an example, GitLab serves a JavaScript terminal
+emulator to the browser on a URL like
+`https://gitlab.com/group/project/-/environments/1/terminal`.
+This opens a websocket connection to, e.g.,
+`wss://gitlab.com/group/project/-/environments/1/terminal.ws`,
+This endpoint doesn't exist in GitLab - only in Workhorse.
+
+When receiving the connection, Workhorse first checks that the
+client is authorized to access the requested terminal. It does
+this by performing a "preauthentication" request to GitLab.
+
+If the client has the appropriate permissions and the terminal
+exists, GitLab responds with a successful response that includes
+details of the terminal that the client should be connected to.
+Otherwise, it returns an appropriate HTTP error response.
+
+Errors are passed back to the client as HTTP responses, but if
+GitLab returns valid terminal details to Workhorse, it will
+connect to the specified terminal, upgrade the browser to a
+WebSocket, and proxy between the two connections for as long
+as the browser's credentials are valid. Workhorse will also
+send regular `PingMessage` control frames to the browser, to
+keep intervening proxies from terminating the connection
+while the browser is present.
+
+The browser must request an upgrade with a specific subprotocol:
+
+### `terminal.gitlab.com`
+
+This subprotocol considers `TextMessage` frames to be invalid.
+Control frames, such as `PingMessage` or `CloseMessage`, have
+their usual meanings.
+
+`BinaryMessage` frames sent from the browser to the server are
+arbitrary text input.
+
+`BinaryMessage` frames sent from the server to the browser are
+arbitrary text output.
+
+These frames are expected to contain ANSI text control codes
+and may be in any encoding.
+
+### `base64.terminal.gitlab.com`
+
+This subprotocol considers `BinaryMessage` frames to be invalid.
+Control frames, such as `PingMessage` or `CloseMessage`, have
+their usual meanings.
+
+`TextMessage` frames sent from the browser to the server are
+base64-encoded arbitrary text input (so the server must
+base64-decode them before inputting them).
+
+`TextMessage` frames sent from the server to the browser are
+base64-encoded arbitrary text output (so the browser must
+base64-decode them before outputting them).
+
+In their base64-encoded form, these frames are expected to
+contain ANSI terminal control codes, and may be in any encoding.
+
+## Workhorse to GitLab
+
+Using again the terminal as an example, before upgrading the browser,
+Workhorse sends a normal HTTP request to GitLab on a URL like
+`https://gitlab.com/group/project/environments/1/terminal.ws/authorize`.
+This returns a JSON response containing details of where the
+terminal can be found, and how to connect it. In particular,
+the following details are returned in case of success:
+
+* WebSocket URL to **connect** to, e.g.: `wss://example.com/terminals/1.ws?tty=1`
+* WebSocket subprotocols to support, e.g.: `["channel.k8s.io"]`
+* Headers to send, e.g.: `Authorization: Token xxyyz..`
+* Certificate authority to verify `wss` connections with (optional)
+
+Workhorse periodically re-checks this endpoint, and if it gets an
+error response, or the details of the terminal change, it will
+terminate the websocket session.
+
+## Workhorse to the WebSocket server
+
+In GitLab, environments or CI jobs may have a deployment service (e.g.,
+`KubernetesService`) associated with them. This service knows
+where the terminals or the service for an environment may be found, and these
+details are returned to Workhorse by GitLab.
+
+These URLs are *also* WebSocket URLs, and GitLab tells Workhorse
+which subprotocols to speak over the connection, along with any
+authentication details required by the remote end.
+
+Before upgrading the browser's connection to a websocket,
+Workhorse opens a HTTP client connection, according to the
+details given to it by Workhorse, and attempts to upgrade
+that connection to a websocket. If it fails, an error
+response is sent to the browser; otherwise, the browser is
+also upgraded.
+
+Workhorse now has two websocket connections, albeit with
+differing subprotocols. It decodes incoming frames from the
+browser, re-encodes them to the the channel's subprotocol, and
+sends them to the channel. Similarly, it decodes incoming
+frames from the channel, re-encodes them to the browser's
+subprotocol, and sends them to the browser.
+
+When either connection closes or enters an error state,
+Workhorse detects the error and closes the other connection,
+terminating the channel session. If the browser is the
+connection that has disconnected, Workhorse will send an ANSI
+`End of Transmission` control code (the `0x04` byte) to the
+channel, encoded according to the appropriate subprotocol.
+Workhorse will automatically reply to any websocket ping frame
+sent by the channel, to avoid being disconnected.
+
+Currently, Workhorse only supports the following subprotocols.
+Supporting new deployment services will require new subprotocols
+to be supported:
+
+### `channel.k8s.io`
+
+Used by Kubernetes, this subprotocol defines a simple multiplexed
+channel.
+
+Control frames have their usual meanings. `TextMessage` frames are
+invalid. `BinaryMessage` frames represent I/O to a specific file
+descriptor.
+
+The first byte of each `BinaryMessage` frame represents the file
+descriptor (fd) number, as a `uint8` (so the value `0x00` corresponds
+to fd 0, `STDIN`, while `0x01` corresponds to fd 1, `STDOUT`).
+
+The remaining bytes represent arbitrary data. For frames received
+from the server, they are bytes that have been received from that
+fd. For frames sent to the server, they are bytes that should be
+written to that fd.
+
+### `base64.channel.k8s.io`
+
+Also used by Kubernetes, this subprotocol defines a similar multiplexed
+channel to `channel.k8s.io`. The main differences are:
+
+* `TextMessage` frames are valid, rather than `BinaryMessage` frames.
+* The first byte of each `TextMessage` frame represents the file
+ descriptor as a numeric UTF-8 character, so the character `U+0030`,
+ or "0", is fd 0, STDIN).
+* The remaining bytes represent base64-encoded arbitrary data.
+
diff --git a/doc/architecture/gitlab_features.md b/doc/architecture/gitlab_features.md
new file mode 100644
index 0000000000000000000000000000000000000000..051d00a50618ebd9eedb58087d2d381167476d59
--- /dev/null
+++ b/doc/architecture/gitlab_features.md
@@ -0,0 +1,69 @@
+# Features that rely on Workhorse
+
+Workhorse itself is not a feature, but there are several features in
+GitLab that would not work efficiently without Workhorse.
+
+To put the efficiency benefit in context, consider that in 2020Q3 on
+GitLab.com [we see][thanos] Rails application threads using on average
+about 200MB of RSS vs about 200KB for Workhorse goroutines.
+
+Examples of features that rely on Workhorse:
+
+## 1. `git clone` and `git push` over HTTP
+
+Git clone, pull and push are slow because they transfer large amounts
+of data and because each is CPU intensive on the GitLab side. Without
+workhorse, HTTP access to Git repositories would compete with regular
+web access to the application, requiring us to run way more Rails
+application servers.
+
+## 2. CI runner long polling
+
+GitLab CI runners fetch new CI jobs by polling the GitLab server.
+Workhorse acts as a kind of "waiting room" where CI runners can sit
+and wait for new CI jobs. Because of Go's efficiency we can fit a lot
+of runners in the waiting room at little cost. Without this waiting
+room mechanism we would have to add a lot more Rails server capacity.
+
+## 3. File uploads and downloads
+
+File uploads and downloads may be slow either because the file is
+large or because the user's connection is slow. Workhorse can handle
+the slow part for Rails. This improves the efficiency of features such
+as CI artifacts, package repositories, LFS objects, etc.
+
+## 4. Websocket proxying
+
+Features such as the web terminal require a long lived connection
+between the user's web browser and a container inside GitLab that is
+not directly accessible from the internet. Dedicating a Rails
+application thread to proxying such a connection would cost much more
+memory than it costs to have Workhorse look after it.
+
+## Quick facts (how does Workhorse work)
+
+- Workhorse can handle some requests without involving Rails at all:
+ for example, JavaScript files and CSS files are served straight
+ from disk.
+- Workhorse can modify responses sent by Rails: for example if you use
+ `send_file` in Rails then gitlab-workhorse will open the file on
+ disk and send its contents as the response body to the client.
+- Workhorse can take over requests after asking permission from Rails.
+ Example: handling `git clone`.
+- Workhorse can modify requests before passing them to Rails. Example:
+ when handling a Git LFS upload Workhorse first asks permission from
+ Rails, then it stores the request body in a tempfile, then it sends
+ a modified request containing the tempfile path to Rails.
+- Workhorse can manage long-lived WebSocket connections for Rails.
+ Example: handling the terminal websocket for environments.
+- Workhorse does not connect to Postgres, only to Rails and (optionally) Redis.
+- We assume that all requests that reach Workhorse pass through an
+ upstream proxy such as NGINX or Apache first.
+- Workhorse does not accept HTTPS connections.
+- Workhorse does not clean up idle client connections.
+- We assume that all requests to Rails pass through Workhorse.
+
+For more information see ['A brief history of gitlab-workhorse'][brief-history-blog].
+
+[thanos]: https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sum(ruby_process_resident_memory_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%20%2F%20sum(puma_max_threads%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g0.tab=1&g1.range_input=1h&g1.max_source_resolution=0s&g1.expr=sum(go_memstats_sys_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%2Fsum(go_goroutines%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g1.tab=1
+[brief-history-blog]: https://about.gitlab.com/2016/04/12/a-brief-history-of-gitlab-workhorse/
diff --git a/doc/channel.md b/doc/channel.md
index 31b57e9c4155d14eb210265c1a7a7482da1456cc..68553170775b97e4652e12ef87d3a3c440e5e214 100644
--- a/doc/channel.md
+++ b/doc/channel.md
@@ -1,193 +1 @@
-# Websocket channel support
-
-In some cases, GitLab can provide in-browser terminal access to an
-environment (which is a running server or container, onto which a
-project has been deployed), or even access to services running in CI
-through a WebSocket. Workhorse manages the WebSocket upgrade and
-long-lived connection to the websocket connection, which frees
-up GitLab to process other requests.
-
-This document outlines the architecture of these connections.
-
-## Introduction to WebSockets
-
-A websocket is an "upgraded" HTTP/1.1 request. Their purpose is to
-permit bidirectional communication between a client and a server.
-**Websockets are not HTTP**. Clients can send messages (known as
-frames) to the server at any time, and vice-versa. Client messages
-are not necessarily requests, and server messages are not necessarily
-responses. WebSocket URLs have schemes like `ws://` (unencrypted) or
-`wss://` (TLS-secured).
-
-When requesting an upgrade to WebSocket, the browser sends a HTTP/1.1
-request that looks like this:
-
-```
-GET /path.ws HTTP/1.1
-Connection: upgrade
-Upgrade: websocket
-Sec-WebSocket-Protocol: terminal.gitlab.com
-# More headers, including security measures
-```
-
-At this point, the connection is still HTTP, so this is a request and
-the server can send a normal HTTP response, including `404 Not Found`,
-`500 Internal Server Error`, etc.
-
-If the server decides to permit the upgrade, it will send a HTTP
-`101 Switching Protocols` response. From this point, the connection
-is no longer HTTP. It is a WebSocket and frames, not HTTP requests,
-will flow over it. The connection will persist until the client or
-server closes the connection.
-
-In addition to the subprotocol, individual websocket frames may
-also specify a message type - examples include `BinaryMessage`,
-`TextMessage`, `Ping`, `Pong` or `Close`. Only binary frames can
-contain arbitrary data - other frames are expected to be valid
-UTF-8 strings, in addition to any subprotocol expectations.
-
-## Browser to Workhorse
-
-Using the terminal as an example, GitLab serves a JavaScript terminal
-emulator to the browser on a URL like
-`https://gitlab.com/group/project/-/environments/1/terminal`.
-This opens a websocket connection to, e.g.,
-`wss://gitlab.com/group/project/-/environments/1/terminal.ws`,
-This endpoint doesn't exist in GitLab - only in Workhorse.
-
-When receiving the connection, Workhorse first checks that the
-client is authorized to access the requested terminal. It does
-this by performing a "preauthentication" request to GitLab.
-
-If the client has the appropriate permissions and the terminal
-exists, GitLab responds with a successful response that includes
-details of the terminal that the client should be connected to.
-Otherwise, it returns an appropriate HTTP error response.
-
-Errors are passed back to the client as HTTP responses, but if
-GitLab returns valid terminal details to Workhorse, it will
-connect to the specified terminal, upgrade the browser to a
-WebSocket, and proxy between the two connections for as long
-as the browser's credentials are valid. Workhorse will also
-send regular `PingMessage` control frames to the browser, to
-keep intervening proxies from terminating the connection
-while the browser is present.
-
-The browser must request an upgrade with a specific subprotocol:
-
-### `terminal.gitlab.com`
-
-This subprotocol considers `TextMessage` frames to be invalid.
-Control frames, such as `PingMessage` or `CloseMessage`, have
-their usual meanings.
-
-`BinaryMessage` frames sent from the browser to the server are
-arbitrary text input.
-
-`BinaryMessage` frames sent from the server to the browser are
-arbitrary text output.
-
-These frames are expected to contain ANSI text control codes
-and may be in any encoding.
-
-### `base64.terminal.gitlab.com`
-
-This subprotocol considers `BinaryMessage` frames to be invalid.
-Control frames, such as `PingMessage` or `CloseMessage`, have
-their usual meanings.
-
-`TextMessage` frames sent from the browser to the server are
-base64-encoded arbitrary text input (so the server must
-base64-decode them before inputting them).
-
-`TextMessage` frames sent from the server to the browser are
-base64-encoded arbitrary text output (so the browser must
-base64-decode them before outputting them).
-
-In their base64-encoded form, these frames are expected to
-contain ANSI terminal control codes, and may be in any encoding.
-
-## Workhorse to GitLab
-
-Using again the terminal as an example, before upgrading the browser,
-Workhorse sends a normal HTTP request to GitLab on a URL like
-`https://gitlab.com/group/project/environments/1/terminal.ws/authorize`.
-This returns a JSON response containing details of where the
-terminal can be found, and how to connect it. In particular,
-the following details are returned in case of success:
-
-* WebSocket URL to **connect** to, e.g.: `wss://example.com/terminals/1.ws?tty=1`
-* WebSocket subprotocols to support, e.g.: `["channel.k8s.io"]`
-* Headers to send, e.g.: `Authorization: Token xxyyz..`
-* Certificate authority to verify `wss` connections with (optional)
-
-Workhorse periodically re-checks this endpoint, and if it gets an
-error response, or the details of the terminal change, it will
-terminate the websocket session.
-
-## Workhorse to the WebSocket server
-
-In GitLab, environments or CI jobs may have a deployment service (e.g.,
-`KubernetesService`) associated with them. This service knows
-where the terminals or the service for an environment may be found, and these
-details are returned to Workhorse by GitLab.
-
-These URLs are *also* WebSocket URLs, and GitLab tells Workhorse
-which subprotocols to speak over the connection, along with any
-authentication details required by the remote end.
-
-Before upgrading the browser's connection to a websocket,
-Workhorse opens a HTTP client connection, according to the
-details given to it by Workhorse, and attempts to upgrade
-that connection to a websocket. If it fails, an error
-response is sent to the browser; otherwise, the browser is
-also upgraded.
-
-Workhorse now has two websocket connections, albeit with
-differing subprotocols. It decodes incoming frames from the
-browser, re-encodes them to the the channel's subprotocol, and
-sends them to the channel. Similarly, it decodes incoming
-frames from the channel, re-encodes them to the browser's
-subprotocol, and sends them to the browser.
-
-When either connection closes or enters an error state,
-Workhorse detects the error and closes the other connection,
-terminating the channel session. If the browser is the
-connection that has disconnected, Workhorse will send an ANSI
-`End of Transmission` control code (the `0x04` byte) to the
-channel, encoded according to the appropriate subprotocol.
-Workhorse will automatically reply to any websocket ping frame
-sent by the channel, to avoid being disconnected.
-
-Currently, Workhorse only supports the following subprotocols.
-Supporting new deployment services will require new subprotocols
-to be supported:
-
-### `channel.k8s.io`
-
-Used by Kubernetes, this subprotocol defines a simple multiplexed
-channel.
-
-Control frames have their usual meanings. `TextMessage` frames are
-invalid. `BinaryMessage` frames represent I/O to a specific file
-descriptor.
-
-The first byte of each `BinaryMessage` frame represents the file
-descriptor (fd) number, as a `uint8` (so the value `0x00` corresponds
-to fd 0, `STDIN`, while `0x01` corresponds to fd 1, `STDOUT`).
-
-The remaining bytes represent arbitrary data. For frames received
-from the server, they are bytes that have been received from that
-fd. For frames sent to the server, they are bytes that should be
-written to that fd.
-
-### `base64.channel.k8s.io`
-
-Also used by Kubernetes, this subprotocol defines a similar multiplexed
-channel to `channel.k8s.io`. The main differences are:
-
-* `TextMessage` frames are valid, rather than `BinaryMessage` frames.
-* The first byte of each `TextMessage` frame represents the file
- descriptor as a numeric UTF-8 character, so the character `U+0030`,
- or "0", is fd 0, STDIN).
-* The remaining bytes represent base64-encoded arbitrary data.
+This file was moved to [`architecture/channel.md`](doc/architecture/channel.md).
diff --git a/doc/development/tests.md b/doc/development/tests.md
new file mode 100644
index 0000000000000000000000000000000000000000..33d69954b5afa9e81fb5125df5870ce905886aa0
--- /dev/null
+++ b/doc/development/tests.md
@@ -0,0 +1,17 @@
+# Testing your code
+
+Run the tests with:
+
+```
+make clean test
+```
+
+## Coverage / what to test
+
+Each feature in gitlab-workhorse should have an integration test that
+verifies that the feature 'kicks in' on the right requests and leaves
+other requests unaffected. It is better to also have package-level tests
+for specific behavior but the high-level integration tests should have
+the first priority during development.
+
+It is OK if a feature is only covered by integration tests.
diff --git a/doc/operations/configuration.md b/doc/operations/configuration.md
new file mode 100644
index 0000000000000000000000000000000000000000..70c594dbc09931e64f335f3e9897ecadd853f3bf
--- /dev/null
+++ b/doc/operations/configuration.md
@@ -0,0 +1,217 @@
+# Workhorse configuration
+
+For historical reasons Workhorse uses both command line flags, a configuration file and environment variables.
+
+All new configuration options that get added to Workhorse should go into the configuration file.
+
+## CLI options
+
+```
+ gitlab-workhorse [OPTIONS]
+
+Options:
+ -apiCiLongPollingDuration duration
+ Long polling duration for job requesting for runners (default 50s - enabled) (default 50ns)
+ -apiLimit uint
+ Number of API requests allowed at single time
+ -apiQueueDuration duration
+ Maximum queueing duration of requests (default 30s)
+ -apiQueueLimit uint
+ Number of API requests allowed to be queued
+ -authBackend string
+ Authentication/authorization backend (default "http://localhost:8080")
+ -authSocket string
+ Optional: Unix domain socket to dial authBackend at
+ -cableBackend string
+ Optional: ActionCable backend (default authBackend)
+ -cableSocket string
+ Optional: Unix domain socket to dial cableBackend at (default authSocket)
+ -config string
+ TOML file to load config from
+ -developmentMode
+ Allow the assets to be served from Rails app
+ -documentRoot string
+ Path to static files content (default "public")
+ -listenAddr string
+ Listen address for HTTP server (default "localhost:8181")
+ -listenNetwork string
+ Listen 'network' (tcp, tcp4, tcp6, unix) (default "tcp")
+ -listenUmask int
+ Umask for Unix socket
+ -logFile string
+ Log file location
+ -logFormat string
+ Log format to use defaults to text (text, json, structured, none) (default "text")
+ -pprofListenAddr string
+ pprof listening address, e.g. 'localhost:6060'
+ -prometheusListenAddr string
+ Prometheus listening address, e.g. 'localhost:9229'
+ -proxyHeadersTimeout duration
+ How long to wait for response headers when proxying the request (default 5m0s)
+ -secretPath string
+ File with secret key to authenticate with authBackend (default "./.gitlab_workhorse_secret")
+ -version
+ Print version and exit
+```
+
+The 'auth backend' refers to the GitLab Rails application. The name is
+a holdover from when gitlab-workhorse only handled Git push/pull over
+HTTP.
+
+Gitlab-workhorse can listen on either a TCP or a Unix domain socket. It
+can also open a second listening TCP listening socket with the Go
+[net/http/pprof profiler server](http://golang.org/pkg/net/http/pprof/).
+
+Gitlab-workhorse can listen on redis events (currently only builds/register
+for runners). This requires you to pass a valid TOML config file via
+`-config` flag.
+For regular setups it only requires the following (replacing the string
+with the actual socket)
+
+## Redis
+
+Gitlab-workhorse integrates with Redis to do long polling for CI build
+requests. This is configured via two things:
+
+- Redis settings in the TOML config file
+- The `-apiCiLongPollingDuration` command line flag to control polling
+ behavior for CI build requests
+
+It is OK to enable Redis in the config file but to leave CI polling
+disabled; this just results in an idle Redis pubsub connection. The
+opposite is not possible: CI long polling requires a correct Redis
+configuration.
+
+Below we discuss the options for the `[redis]` section in the config
+file.
+
+```
+[redis]
+URL = "unix:///var/run/gitlab/redis.sock"
+Password = "my_awesome_password"
+Sentinel = [ "tcp://sentinel1:23456", "tcp://sentinel2:23456" ]
+SentinelMaster = "mymaster"
+```
+
+- `URL` takes a string in the format `unix://path/to/redis.sock` or
+`tcp://host:port`.
+- `Password` is only required if your redis instance is password-protected
+- `Sentinel` is used if you are using Sentinel.
+ *NOTE* that if both `Sentinel` and `URL` are given, only `Sentinel` will be used
+
+Optional fields are as follows:
+```
+[redis]
+DB = 0
+ReadTimeout = "1s"
+KeepAlivePeriod = "5m"
+MaxIdle = 1
+MaxActive = 1
+```
+
+- `DB` is the Database to connect to. Defaults to `0`
+- `ReadTimeout` is how long a redis read-command can take. Defaults to `1s`
+- `KeepAlivePeriod` is how long the redis connection is to be kept alive without anything flowing through it. Defaults to `5m`
+- `MaxIdle` is how many idle connections can be in the redis-pool at once. Defaults to 1
+- `MaxActive` is how many connections the pool can keep. Defaults to 1
+
+## Relative URL support
+
+If you are mounting GitLab at a relative URL, e.g.
+`example.com/gitlab`, then you should also use this relative URL in
+the `authBackend` setting:
+
+```
+gitlab-workhorse -authBackend http://localhost:8080/gitlab
+```
+
+## Interaction of authBackend and authSocket
+
+The interaction between `authBackend` and `authSocket` can be a bit
+confusing. It comes down to: if `authSocket` is set it overrides the
+_host_ part of `authBackend` but not the relative path.
+
+In table form:
+
+|authBackend|authSocket|Workhorse connects to?|Rails relative URL|
+|---|---|---|---|
+|unset|unset|`localhost:8080`|`/`|
+|`http://localhost:3000`|unset|`localhost:3000`|`/`|
+|`http://localhost:3000/gitlab`|unset|`localhost:3000`|`/gitlab`|
+|unset|`/path/to/socket`|`/path/to/socket`|`/`|
+|`http://localhost:3000`|`/path/to/socket`|`/path/to/socket`|`/`|
+|`http://localhost:3000/gitlab`|`/path/to/socket`|`/path/to/socket`|`/gitlab`|
+
+The same applies to `cableBackend` and `cableSocket`.
+
+## Error tracking
+
+GitLab-Workhorse supports remote error tracking with
+[Sentry](https://sentry.io). To enable this feature set the
+`GITLAB_WORKHORSE_SENTRY_DSN` environment variable.
+You can also set the `GITLAB_WORKHORSE_SENTRY_ENVIRONMENT` environment variable to
+use the Sentry environment functionality to separate staging, production and
+development.
+
+Omnibus (`/etc/gitlab/gitlab.rb`):
+
+```
+gitlab_workhorse['env'] = {
+ 'GITLAB_WORKHORSE_SENTRY_DSN' => 'https://foobar'
+ 'GITLAB_WORKHORSE_SENTRY_ENVIRONMENT' => 'production'
+}
+```
+
+Source installations (`/etc/default/gitlab`):
+
+```
+export GITLAB_WORKHORSE_SENTRY_DSN='https://foobar'
+export GITLAB_WORKHORSE_SENTRY_ENVIRONMENT='production'
+```
+
+## Distributed Tracing
+
+Workhorse supports distributed tracing through [LabKit][] using [OpenTracing APIs](https://opentracing.io).
+
+By default, no tracing implementation is linked into the binary, but different OpenTracing providers can be linked in using [build tags][build-tags]/[build constraints][build-tags]. This can be done by setting the `BUILD_TAGS` make variable.
+
+For more details of the supported providers, see LabKit, but as an example, for Jaeger tracing support, include the tags: `BUILD_TAGS="tracer_static tracer_static_jaeger"`.
+
+```shell
+make BUILD_TAGS="tracer_static tracer_static_jaeger"
+```
+
+Once Workhorse is compiled with an opentracing provider, the tracing configuration is configured via the `GITLAB_TRACING` environment variable.
+
+For example:
+
+```shell
+GITLAB_TRACING=opentracing://jaeger ./gitlab-workhorse
+```
+
+## Continuous Profiling
+
+Workhorse supports continuous profiling through [LabKit][] using [Stackdriver Profiler](https://cloud.google.com/profiler).
+
+By default, the Stackdriver Profiler implementation is linked in the binary using [build tags][build-tags], though it's not
+required and can be skipped.
+
+For example:
+
+```shell
+make BUILD_TAGS=""
+```
+
+Once Workhorse is compiled with Continuous Profiling, the profiler configuration can be set via `GITLAB_CONTINUOUS_PROFILING`
+environment variable.
+
+For example:
+
+```shell
+GITLAB_CONTINUOUS_PROFILING="stackdriver?service=workhorse&service_version=1.0.1&project_id=test-123 ./gitlab-workhorse"
+```
+
+More information about see the [LabKit monitoring docs](https://gitlab.com/gitlab-org/labkit/-/blob/master/monitoring/doc.go).
+
+[LabKit]: https://gitlab.com/gitlab-org/labkit/
+[build-tags]: https://golang.org/pkg/go/build/#hdr-Build_Constraints
diff --git a/doc/operations/install.md b/doc/operations/install.md
new file mode 100644
index 0000000000000000000000000000000000000000..a16124cf397a3e7aee4d16892d0bf01286ed48c0
--- /dev/null
+++ b/doc/operations/install.md
@@ -0,0 +1,44 @@
+# Installation
+
+To install gitlab-workhorse you need [Go 1.13 or
+newer](https://golang.org/dl) and [GNU
+Make](https://www.gnu.org/software/make/).
+
+To install into `/usr/local/bin` run `make install`.
+
+```
+make install
+```
+
+To install into `/foo/bin` set the PREFIX variable.
+
+```
+make install PREFIX=/foo
+```
+
+On some operating systems, such as FreeBSD, you may have to use
+`gmake` instead of `make`.
+
+*NOTE*: Some features depends on build tags, make sure to check
+[Workhorse configuration](doc/operations/configuration.md) to enable them.
+
+## Run time dependencies
+
+### Exiftool
+
+Workhorse uses [exiftool](https://www.sno.phy.queensu.ca/~phil/exiftool/) for
+removing EXIF data (which may contain sensitive information) from uploaded
+images. If you installed GitLab:
+
+- Using the Omnibus package, you're all set.
+ *NOTE* that if you are using CentOS Minimal, you may need to install `perl`
+ package: `yum install perl`
+- From source, make sure `exiftool` is installed:
+
+ ```sh
+ # Debian/Ubuntu
+ sudo apt-get install libimage-exiftool-perl
+
+ # RHEL/CentOS
+ sudo yum install perl-Image-ExifTool
+ ```