internal/sshd/README.md
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Overview
This package is responsible for handling SSH connections.
Since GitLab [13.9](https://gitlab.com/groups/gitlab-org/-/milestones/56), `gitlab-shell` provides an option to either use [OpenSSH](https://github.com/openssh/openssh-portable) to handle connections or utilize an internal implementation. For more information on the internal implementation, see the [Why we implemented our own SSHD solution](https://about.gitlab.com/blog/2022/08/17/why-we-have-implemented-our-own-sshd-solution-on-gitlab-sass/) blog post.
The package contains multiple public functions:
- [func NewServer](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L43) for initializing [Server](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L33).
- [func (s \*Server) ListenAndServe](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L52) for listening to new connections and serving them.
- [func (s \*Server) Shutdown](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L63) for shutting down the server.
- [func (s \*Server) MonitoringServeMux](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L73) that provides monitoring endpoints.
All these functions are used by the [`main`](https://gitlab.com/gitlab-org/gitlab-shell/-/blob/822e49b34afbc2092ae189091d693ae7867a8e5a/cmd/gitlab-sshd/main.go) function to initialize and shutdown the server. When a client tries to connect to the server, the following sequence of steps happens:
- When a new connection is [accepted](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L118) by the listener, it's [asynchronously handled](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L129).
- A new connection is [created](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L190) with [a customized server config](https://gitlab.com/gitlab-org/gitlab-shell/blob/1177e2675c15ed55d46528c17408fd98d797cdec/internal/sshd/server_config.go#L148) that allows the server to perform [the handshake](https://gitlab.com/gitlab-org/gitlab-shell/blob/1177e2675c15ed55d46528c17408fd98d797cdec/internal/sshd/connection.go#L77) that [verifies](https://gitlab.com/gitlab-org/gitlab-shell/blob/1177e2675c15ed55d46528c17408fd98d797cdec/internal/sshd/server_config.go#L151) the authorized keys via [a request](https://gitlab.com/gitlab-org/gitlab-shell/blob/1177e2675c15ed55d46528c17408fd98d797cdec/internal/sshd/server_config.go#L140) to Gitlab Rails. The server is ready to [handle sessions](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/connection.go#L65) via this connection. The reason why we [handle the sessions in a loop](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/connection.go#L98) is the [persisting connections](https://man.openbsd.org/ssh_config.5#ControlPersist): a connection remains open and reused for the future sessions.
- A new session is [handled](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/connection.go#L135) by the [channelHandler](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/connection.go#L39) function passed as a [parameter](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L199). This function [processes](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/session.go#L56) requests from the client.
- The client may send [env requests](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/session.go#L67) in order to set an environment variable but we [restrict it](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/session.go#L113) to only setting Git protocol version. The client eventually [executes a command](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/session.go#L68-74) either via `shell` or `exec` requests.
- When the command is [executed](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/session.go#L143), we [create a command](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/session.go#L165) of a [particular type](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/cmd/gitlab-shell/command/command.go#L57). More information on how the commands work can be found [here](https://gitlab.com/gitlab-org/gitlab-shell/-/tree/main/internal/command).
- When a command is executed, the [session is closed](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/session.go#L204) and the error about unsuccessful execution, if such took place, is [tracked](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/connection.go#L137).
- If a connection has been prematurely closed we [block execution](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/connection.go#L146) until all concurrent sessions are released.
## Proxy protocol
The package supports creating a server with PROXY protocol. The [`go-proxyproto`](https://github.com/pires/go-proxyproto) package is used to [wrap](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L98) the basic listener into the one that supports PROXY protocol. PROXY protocol enables us to implement [Group IP address restriction via SSH](https://gitlab.com/gitlab-org/gitlab/-/issues/271673). The policies are [configurable](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L203).
## Configurable OpenSSH alternatives
- [LoginGraceTime](https://man7.org/linux/man-pages/man5/sshd_config.5.html) is [implemented](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/connection.go#L73) via TCP deadlines.
- [ClientAliveInterval](https://man7.org/linux/man-pages/man5/sshd_config.5.html) is [implemented](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/connection.go#L151) via sending [keep-alive message](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/connection.go#L25) periodically.
## State machine
The server [maintains a state machine](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L26-31) to implement:
- **Graceful shutdown.** When a termination signal [has been detected](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/cmd/gitlab-sshd/main.go#L96), then a service [is being shut down](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/cmd/gitlab-sshd/main.go#L105). The status is [changed](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L68) accordingly and no new connections [are accepted](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L120). A configurable [grace period is given](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/cmd/gitlab-sshd/main.go#L107) in order to allow the ongoing connections to complete. When the period expires, then the top-level context is [canceled](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/cmd/gitlab-sshd/main.go#L109). That means that all the ongoing HTTP and SSH connections are [closed](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L173).
- **Liveness and readiness probes** that help Kubernetes to evaluate the state of the server. If a state [is any other than ready](https://gitlab.com/gitlab-org/gitlab-shell/blob/2094323308f8c6cb1d935af1b3331df29edcc9b6/internal/sshd/sshd.go#L80) (for example, during graceful shutdown), then 502 is returned.