Enterprise Server 3.0 release notes

Packages have been updated to the latest security versions. In these updates, Log4j has been updated to version 2.17.1. Note: previous mitigations released in 3.3.1, 3.2.6, 3.1.14, and 3.0.22 are sufficient to address the impact of CVE-2021-44228, CVE-2021-45046, CVE-2021-45105, and CVE-2021-44832 in these versions of GitHub Enterprise Server.

Sanitize more secrets in the generated support bundles

Packages have been updated to the latest security versions.

Running ghe-config-apply could sometimes fail because of permission issues in /data/user/tmp/pages.

The save button in management console was unreachable by scrolling in lower resolution browsers.

IOPS and Storage Traffic monitoring graphs were not updating after collectd version upgrade.

Some webhook related jobs could generated large amount of logs.

CRITICAL: A remote code execution vulnerability in the Log4j library, identified as CVE-2021-44228, affected all versions of GitHub Enterprise Server prior to 3.3.1. The Log4j library is used in an open source service running on the GitHub Enterprise Server instance. This vulnerability was fixed in GitHub Enterprise Server versions 3.0.22, 3.1.14, 3.2.6, and 3.3.1. For more information, please see this post on the GitHub Blog.

December 17, 2021 update: The fixes in place for this release also mitigate CVE-2021-45046, which was published after this release. No additional upgrade for GitHub Enterprise Server is required to mitigate both CVE-2021-44228 and CVE-2021-45046.

Support bundles could include sensitive files if they met a specific set of conditions.

A UI misrepresentation vulnerability was identified in GitHub Enterprise Server that allowed more permissions to be granted during a GitHub App's user-authorization web flow than was displayed to the user during approval. This vulnerability affected all versions of GitHub Enterprise Server prior to 3.3 and was fixed in versions 3.2.5, 3.1.13, 3.0.21. This vulnerability was reported via the GitHub Bug Bounty program and has been assigned CVE-2021-41598.

Running ghe-config-apply could sometimes fail because of permission issues in /data/user/tmp/pages.

A misconfiguration in the Management Console caused scheduling errors.

Docker would hold log files open after a log rotation.

GraphQL requests did not set the GITHUB_USER_IP variable in pre-receive hook environments.

Clarifies explanation of Actions path-style in documentation.

Updates support contact URLs to use the current support site, support.github.com.

Packages have been updated to the latest security versions.

Pre-receive hooks would fail due to undefined PATH.

Running ghe-repl-setup would return an error: cannot create directory /data/user/elasticsearch: File exists if the instance had previously been configured as a replica.

In large cluster environments, the authentication backend could be unavailable on a subset of frontend nodes.

Some critical services may not have been available on backend nodes in GHES Cluster.

An additional outer layer of gzip compression when creating a cluster support bundle with ghe-cluster-suport-bundle is now turned off by default. This outer compression can optionally be applied with the ghe-cluster-suport-bundle -c command line option.

We have added extra text to the admin console to remind users about the mobile apps' data collection for experience improvement purposes.

The GitHub Connect data connection record now includes a list of enabled GitHub Connect features. [Updated 2021-12-09]

A path traversal vulnerability was identified in GitHub Pages builds on GitHub Enterprise Server that could allow an attacker to read system files. To exploit this vulnerability, an attacker needed permission to create and build a GitHub Pages site on the GitHub Enterprise Server instance. This vulnerability affected all versions of GitHub Enterprise Server prior to 3.3, and was fixed in versions 3.0.19, 3.1.11, and 3.2.3. This vulnerability was reported through the GitHub Bug Bounty program and has been assigned CVE-2021-22870.

Packages have been updated to the latest security versions.

Some Git operations failed after upgrading a GitHub Enterprise Server 3.x cluster because of the HAProxy configuration.

Unicorn worker counts might have been set incorrectly in clustering mode.

Resqued worker counts might have been set incorrectly in clustering mode.

If Ubuntu's Uncomplicated Firewall (UFW) status was inactive, a client could not clearly see it in the logs.

Some pages and Git-related background jobs might not run in cluster mode with certain cluster configurations.

The enterprise audit log page would not display audit events for secret scanning.

Users were not warned about potentially dangerous bidirectional unicode characters when viewing files. For more information, see "Warning about bidirectional Unicode text" in the GitHub Blog.

Hookshot Go sent distribution type metrics that Collectd could not handle, which caused a ballooning of parsing errors.

Public repositories displayed unexpected results from secret scanning with a type of Unknown Token.

Several known weak SSH public keys have been added to the deny list and can no longer be registered. In addition, versions of GitKraken known to generate weak SSH keys (7.6.x, 7.7.x and 8.0.0) have been blocked from registering new public keys.

Packages have been updated to the latest security versions.

Several parts of the application were unusable for users who are owners of many organizations.

Packages have been updated to the latest security versions.

Custom pre-receive hooks could have failed due to too restrictive virtual memory or CPU time limits.

Attempting to wipe all existing configuration settings with ghe-cleanup-settings failed to restart the Management Console service.

During replication teardown via ghe-repl-teardown Memcached failed to be restarted.

During periods of high load, users would receive HTTP 503 status codes when upstream services failed internal healthchecks.

Pre-receive hook environments were forbidden from calling the cat command via BusyBox on Alpine.

The external database password was logged in plaintext.

An erroneous jq error message may have been displayed when running ghe-config-apply.

Failing over from a primary Cluster datacenter to a secondary Cluster datacenter succeeds, but then failing back over to the original primary Cluster datacenter failed to promote Elasticsearch indicies.

The Site Admin page for repository self-hosted runners returned an HTTP 500.

In some cases, GitHub Enterprise Administrators attempting to view the Dormant users page received 502 Bad Gateway or 504 Gateway Timeout response.

More effectively delete Webhook logs that fall out of the Webhook log retention window.

HIGH: A path traversal vulnerability was identified in GitHub Enterprise Server that could be exploited when building a GitHub Pages site. User-controlled configuration options used by GitHub Pages were not sufficiently restricted and made it possible to read files on the GitHub Enterprise Server instance. To exploit this vulnerability, an attacker would need permission to create and build a GitHub Pages site on the GitHub Enterprise Server instance. This vulnerability affected all versions of GitHub Enterprise Server prior to 3.1.8 and was fixed in 3.1.8, 3.0.16, and 2.22.22. This is the result of an incomplete fix for CVE-2021-22867. This vulnerability was reported via the GitHub Bug Bounty program and has been assigned CVE-2021-22868.

MEDIUM: An improper access control vulnerability in GitHub Enterprise Server allowed a workflow job to execute in a self-hosted runner group it should not have had access to. This affects customers using self-hosted runner groups for access control. A repository with access to one enterprise runner group could access all of the enterprise runner groups within the organization because of improper authentication checks during the request. This could cause code to be run unintentionally by the incorrect runner group. This vulnerability affected GitHub Enterprise Server versions from 3.0.0 to 3.0.15 and 3.1.0 to 3.1.7 and was fixed in 3.0.16 and 3.1.8 releases. It has been assigned CVE-2021-22869.

Resque worker counts were displayed incorrectly during maintenance mode.

Allocated memcached memory could be zero in clustering mode.

Fixes GitHub Pages builds so they take into account the NO_PROXY setting of the appliance. This is relevant to appliances configured with an HTTP proxy only. (update 2021-09-30)

The GitHub Connect configuration of the source instance was always restored to new instances even when the --config option for ghe-restore was not used. This would lead to a conflict with the GitHub Connect connection and license synchronization if both the source and destination instances were online at the same time. The fix also requires updating backup-utils to 3.2.0 or higher. [updated: 2021-11-18]

Packages have been updated to the latest security versions.

Attempting to tear down a newly-added replica node by specifying its UUID with ghe-repl-teardown would fail without reporting an error if replication was not started.

GitHub Pages builds were being passed through an external proxy if there was one configured.

Custom pre-receive hooks that created sub-processes would lack a PATH variable in their environment, resulting in "No such file or directory" errors.

MySQL could failover during an upgrade if mysql-auto-failover was enabled.

Packages have been updated to the latest security versions.

Attaching very large images or animated GIFs to images or pull requests would fail.

Journald messages related to automatic updates (Adding h/m/s random time.) were logged to syslog.

Custom pre-receive hooks that used a bash subshell would return an error: No such file or directory.

Custom pre-receive hooks that created named pipes (FIFOs) would crash or hang, resulting in a timeout error.

Adding filters to the audit log advanced search page did not populate the query text box in real-time with the correct facet prefix and value.

Git hooks to the internal API that result in failing requests returned the exception undefined method body for "success":String (NoMethodError) instead of returning an explicit nil.

When an integration was removed, it was possible for an unrelated OAuth application or integration to also be removed.

When a mandatory message containing an emoji character was added, attempting to view or change the message would return a 500 Internal Server Error.

When GitHub Actions is enabled without running regular scheduled backups the MSSQL Transaction Log could grow unbounded and can consume all available space on the appliance's Data Disk causing a possible outage.

Audit log entries for changes made to "Repository creation" organization settings were inaccurate.

Excessive logging of ActionController::UnknownFormat exceptions caused unnecessary disk usage.

LDAP group_dn values longer than 255 characters would result in errors being logged: Data truncated for column 'group_dn' at row 1.

Abuse rate limits are now called Secondary rate limits, since the behavior they limit is not always abusive.

Packages have been updated to the latest security versions.

Custom pre-receive hooks could lead to an error like error: object directory /data/user/repositories/0/nw/12/34/56/7890/network.git/objects does not exist; check .git/objects/info/alternates.

Unauthenticated HTTP proxy for the pages containers build was not supported for any users that use HTTP proxies.

A significant number of 503 errors were logged every time a user visited a repository''s /settings page if the dependency graph was not enabled.

Internal repositories were only returned when a user had affiliations with the repository through a team or through collaborator status, or queried with the ?type=internal parameter.

Failed background jobs had unlimited retries which could cause large queue depths.

A significant number of 503 errors were being created if the scheduled job to sync vulnerabilities with GitHub.com attempted to run when dependency graph was not enabled and content analysis was enabled.

The logs for babeld now include a cmd field for HTTP ref advertisement requests instead of only including it during the negotiation requests.

HIGH: A path traversal vulnerability was identified in GitHub Enterprise Server that could be exploited when building a GitHub Pages site. User-controlled configuration options used by GitHub Pages were not sufficiently restricted and made it possible to read files on the GitHub Enterprise Server instance. To exploit this vulnerability, an attacker would need permission to create and build a GitHub Pages site on the GitHub Enterprise Server instance. This vulnerability affected all versions of GitHub Enterprise Server prior to 3.1.3 and has been assigned CVE-2021-22867. This vulnerability was reported via the GitHub Bug Bounty program.

Packages have been updated to the latest security versions.

SAML expiration date variable was not configurable.

Application services would fail their health checks during config apply before they could enter a healthy state.

ghe-cluster-config-node-init would fail during cluster setup if HTTP proxy is enabled.

Pre-receive hooks could encounter an error Failed to resolve full path of the current executable due to /proc not being mounted on the container.

Collectd would not resolve the forwarding destination hostname after the initial startup.

The job that purged stale deleted repositories could fail to make progress if some of those repositories were protected from deletion by legal holds.

Running git nw-gc --pristine would result in an error.

Background jobs were being queued to the spam queue which were not being processed.

The preferred merge method would be reset when retrying after a failed PR merge.

Git pushes could result in a 500 Internal Server Error during the user reconciliation process on instances using LDAP authentication mode.

Improved the efficiency of config apply by skipping IP allow firewall rules that had not changed, which saved significant time on large clusters.

Packages have been updated to the latest security versions.

A large number of gauge-dependency-graph-api-dispatch_dispatch metrics could accumulate in the Management Console.

The sshd service would sometimes fail to start on instances running on Google Cloud Platform.

Old upgrade files would persist on the user disk, sometimes resulting in out of space conditions.

Log rotation could sometimes interrupt background jobs.

gh-migrator displayed an incorrect path to its log output.

An export archive would fail to import if it contained review requests from teams not present in the archive.

Packages have been updated to the latest security versions.

The upgrade process could fail while upgrading Actions if the instance could not make self-requests using its configured hostname.

SVN 1.7 and older clients showed an error when using the svn co and svn export commands.

Accessing a repository through the administrative shell using ghe-repo <owner>/<reponame> would hang.

After upgrading, users experienced reduced availability during heavy usage, because services restarted too frequently. This would occur due to timeout mismatches between the nomad configuration and that of the internal services.

In some instances, running ghe-repl-status after setting up GitHub Actions would produce an error and ghe-actions-teardown would fail.

ghe-dbconsole would return errors under some circumstances.

Import failures of organizations or repositories from non-GitHub sources could produce an undefined method '[]' for nil:NilClass error.

GitHub profile names might have changed unintentionally when using SAML authentication, if the GitHub profile name did not match the value of the attribute mapped to the Full name field in the Management Console.

The firstPatchedVersion field is now available on SecurityVulnerability objects in the GraphQL API.

Users of the GraphQL API can query the public field closingIssuesReferences on the PullRequest object. This field retrieves issues that will be automatically closed when the related pull request is merged. This approach will also allow this data to be migrated in future, as part of a higher fidelity migration process.

MEDIUM: Under certain circumstances, users who were removed from a team or organization could retain write access to branches they had existing pull requests opened for.

Packages have been updated to the latest security versions.

On the "Configure Actions and Packages" page of the initial installation process, when an admin clicked the "Test domain settings" button the test did not complete.

Running ghe-btop failed with error cannot find a 'babeld' container.

Users were experiencing service unavailability after upgrading due to a mismatch of internal and external timeout values.

Normal replication delays in MSSQL generated warnings.

Link for GitHub Enterprise Clustering Guide on management console was incorrect.

An IP address added by an admin using the "Create Whitelist Entry" button could still be locked out.

References to the "Dependency graph" and "Dependabot alerts" features were shown on repositories where they were not enabled.

HTTP POST requests to the /hooks endpoint could fail with a 401 response due to the hookID being set incorrectly.

The build-server process failed to clean up processes leaving them in the defunct state.

spokesd created excessive log entries including the phrase "fixing placement skipped".

Check annotations older than 4 months will be archived.

HIGH: A UI misrepresentation vulnerability was identified in GitHub Enterprise Server that allowed more permissions to be granted during a GitHub App's user-authorization web flow than was displayed to the user during approval. To exploit this vulnerability, an attacker would need to create a GitHub App on the instance and have a user authorize the application through the web authentication flow. All permissions being granted would properly be shown during the first authorization, but in certain circumstances, if the user revisits the authorization flow after the GitHub App has configured additional user-level permissions, those additional permissions may not be shown, leading to more permissions being granted than the user potentially intended. This vulnerability affected GitHub Enterprise Server 3.0.x prior to 3.0.7 and 2.22.x prior to 2.22.13. It was fixed in versions 3.0.7 and 2.22.13. This vulnerability has been assigned CVE-2021-22866 and was reported via the GitHub Bug Bounty Program.

Packages have been updated to the latest security versions.

Quotes included in Actions or Packages storage configuration could cause errors.

Custom pre-receive hooks could fail due to too restrictive file size or number of open file limits.

Orchestrator auto failover could be enabled during the phase of config apply.

Users with maintainer permissions to a repository were shown an e-mail verification warning instead of a successful page build on the repository Pages settings page.

The code owner of a wildcard rule would be incorrectly added to the list of owners for the code owners badge even if a later rule took precedence for that path.

OpenAPI documentation referred to an invalid header.

When creating or editing a pre-receive hook, a race condition in the user interface meant that after selecting a repository, files within the repository were sometimes not populated in files drop