Enterprise Server 3.1 release notes

Packages have been updated to the latest security versions.

An internal script to validate hostnames in the GitHub Enterprise Server configuration file would return an error if the hostname string started with a "." (period character).

In HA configurations where the primary node's hostname was longer than 60 characters, MySQL would fail to be configured.

The calculation of "maximum committers across entire instance" reported in the site admin dashboard was incorrect.

An incorrect database entry for repository replicas caused database corruption when performing a restore using GitHub Enterprise Server Backup Utilities.

In HA configurations where Elasticsearch reported a valid yellow status, changes introduced in a previous fix would block the ghe-repl-stop command and not allow replication to be stopped. Using ghe-repo-stop --force will now force Elasticsearch to stop when the service is in a normal or valid yellow status.

MEDIUM: A security issue in nginx resolver was identified, where an attacker who could forge UDP packets from the DNS server could cause 1-byte memory overwrite, resulting in worker process crashes or other potentially damaging impacts. The vulnerability has been assigned CVE-2021-23017.

Updated the actions/checkout@v2 and actions/checkout@v3 actions to address new vulnerabilities announced in the Git security enforcement blog post.

Packages have been updated to the latest security versions.

In some cluster topologies, the ghe-cluster-status command left behind empty directories in /tmp.

SNMP incorrectly logged a high number of Cannot statfs error messages to syslog.

For instances configured with SAML authentication and built-in fallback enabled, built-in users would get stuck in a “login” loop when attempting to sign in from the page generated after logging out.

When using SAML encrypted assertions, some assertions were not correctly marking SSH keys as verified.

The Releases page would return a 500 error when the repository has tags that contain non-ASCII characters. [Updated: 2022-06-10]

In high availability configurations, clarify that the replication overview page in the Management Console only displays the current replication configuration, not the current replication status.

When enabling GitHub Packages, clarify that using a Shared Access Signature (SAS) token as connection string is not supported.

Support bundles now include the row count of tables stored in MySQL.

Packages have been updated to the latest security versions.

Upgrading the nodes in a high availability pair with an upgrade package could cause Elasticsearch to enter an inconsistent state in some cases.

In some cluster topologies, the command line utilities ghe-spokesctl and ghe-btop failed to run.

Elasticsearch indices could be duplicated during a package upgrade, due to an elasticsearch-upgrade service running multiple times in parallel.

The maint_host_low job queues were not processed, resulting in some maintenance tasks failing to run.

When converting a user account to an organization, if the user account was an owner of the GitHub Enterprise Server enterprise account, the converted organization would incorrectly appear in the enterprise owner list.

Creating an impersonation OAuth token using the Enterprise Administration REST API resulted in an error when an integration matching the OAuth Application ID already existed.

When attempting to cache a value larger than the maximum allowed in Memcached, an error was raised however the key was not reported.

MEDIUM: A path traversal vulnerability was identified in GitHub Enterprise Server Management Console that allowed the bypass of CSRF protections. This vulnerability affected all versions of GitHub Enterprise Server prior to 3.5 and was fixed in versions 3.1.19, 3.2.11, 3.3.6, 3.4.1. This vulnerability was reported via the GitHub Bug Bounty program and has been assigned CVE-2022-23732.

MEDIUM: An integer overflow vulnerability was identified in the 1.x branch and the 2.x branch of yajil which leads to subsequent heap memory corruption when dealing with large (~2GB) inputs. This vulnerability was reported internally and has been assigned CVE-2022-24795.

Support bundles could include sensitive files if GitHub Actions was enabled.

Packages have been updated to the latest security versions.

The options to enable TLS 1.0 and TLS 1.1 in the Privacy settings of the Management Console were shown, although removal of those protocol versions occurred in an earlier release.

In a HA environment, configuring MSSQL replication could require additional manual steps after enabling GitHub Actions for the first time.

A subset of internal configuration files are more reliably updated after a hotpatch.

The ghe-run-migrations script would sometimes fail to generate temporary certificate names correctly.

In a cluster environment, Git LFS operations could fail with failed internal API calls that crossed multiple web nodes.

Pre-receive hooks that used gpg --import timed out due to insufficient syscall privileges.

In some cluster topologies, webhook delivery information was not available.

In HA configurations, tearing down a replica would fail if GitHub Actions had previously been enabled.

Elasticsearch health checks would not allow a yellow cluster status when running migrations.

Organizations created as a result of a user transforming their user account into an organization were not added to the global enterprise account.

When using ghe-migrator or exporting from GitHub.com, a long-running export would fail when data was deleted mid-export.

Links to inaccessible pages were removed.

Adding a team as a reviewer to a pull request would sometimes show the incorrect number of members on that team.

A large number of dormant users could cause a GitHub Connect configuration to fail.

The "Feature & beta enrollments" page in the Site admin web UI was incorrectly available.

The "Site admin mode" link in the site footer did not change state when clicked.

Memcached connection limits were increased to better accommodate large cluster topologies.

More effectively identify and delete webhook logs that are outside of the webhook log retention window.

The Dependency Graph API previously ran with a statically defined port.

The default shard counts for cluster-related Elasticsearch shard settings have been updated.

The “Triage” and “Maintain” team roles are preserved during repository migrations.

NotProcessedError exceptions were occurring unnecessarily.

Performance has been improved for web requests made by enterprise owners.

HIGH: An integer overflow vulnerability was identified in GitHub's markdown parser that could potentially lead to information leaks and RCE. This vulnerability was reported through the GitHub Bug Bounty program by Felix Wilhelm of Google's Project Zero and has been assigned CVE-2022-24724.

Upgrades could sometimes fail if a high-availability replica's clock was out of sync with the primary.

OAuth Applications created after September 1st, 2020 were not able to use the Check an Authorization API endpoint.

Packages have been updated to the latest security versions.

Fixes SystemStackError (stack too deep) when getting more than 2^{^16} keys from memcached.

Packages have been updated to the latest security versions.

Pages would become unavailable following a MySQL secret rotation until nginx was manually restarted.

When setting the maintenance schedule with a ISO 8601 date, the actual scheduled time wouldn't match due to the timezone not being transformed to UTC.

Spurious error messages concerning the cloud-config.service would be output to the console.

The version number would not be correctly updated after a installing a hotpatch using ghe-cluster-each.

Webhook table cleanup jobs could run simultaneously, causing resource contention and increasing job run time.

When using CAS authentication and the "Reactivate suspended users" option was enabled, suspended users were not automatically reactivated.

The ability to limit email-based notifications to users with emails on a verified or approved domain did not work correctly.

Several documentation links resulted in a 404 Not Found error.

The GitHub Connect data connection record now includes a count of the number of active and dormant users and the configured dormancy period.

Packages have been updated to the latest security versions. In these updates, Log4j has been updated to version 2.17.1. Note: previous mitigations released in 3.3.1, 3.2.6, 3.1.14, and 3.0.22 are sufficient to address the impact of CVE-2021-44228, CVE-2021-45046, CVE-2021-45105, and CVE-2021-44832 in these versions of GitHub Enterprise Server.

Sanitize more secrets in the generated support bundles

Packages have been updated to the latest security versions.

Running ghe-config-apply could sometimes fail because of permission issues in /data/user/tmp/pages.

The save button in management console was unreachable by scrolling in lower resolution browsers.

IOPS and Storage Traffic monitoring graphs were not updating after collectd version upgrade.

Some webhook related jobs could generated large amount of logs.

The repository permissions to the user returned by the /repos API would not return the full list.

Critical: A remote code execution vulnerability in the Log4j library, identified as CVE-2021-44228, affected all versions of GitHub Enterprise Server prior to 3.3.1. The Log4j library is used in an open source service running on the GitHub Enterprise Server instance. This vulnerability was fixed in GitHub Enterprise Server versions 3.0.22, 3.1.14, 3.2.6, and 3.3.1. For more information, please see this post on the GitHub Blog.

December 17, 2021 update: The fixes in place for this release also mitigate CVE-2021-45046, which was published after this release. No additional upgrade for GitHub Enterprise Server is required to mitigate both CVE-2021-44228 and CVE-2021-45046.

Support bundles could include sensitive files if they met a specific set of conditions.

A UI misrepresentation vulnerability was identified in GitHub Enterprise Server that allowed more permissions to be granted during a GitHub App's user-authorization web flow than was displayed to the user during approval. This vulnerability affected all versions of GitHub Enterprise Server prior to 3.3 and was fixed in versions 3.2.5, 3.1.13, 3.0.21. This vulnerability was reported via the GitHub Bug Bounty program and has been assigned CVE-2021-41598.

A remote code execution vulnerability was identified in GitHub Enterprise Server that could be exploited when building a GitHub Pages site. This vulnerability affected all versions of GitHub Enterprise Server prior to 3.3 and was fixed in versions 3.0.21, 3.1.13, 3.2.5. This vulnerability was reported via the GitHub Bug Bounty program and has been assigned CVE-2021-41599. Updated February 17, 2022

Running ghe-config-apply could sometimes fail because of permission issues in /data/user/tmp/pages.

A misconfiguration in the Management Console caused scheduling errors.

Docker would hold log files open after a log rotation.

GraphQL requests did not set the GITHUB_USER_IP variable in pre-receive hook environments.

Clarifies explanation of Actions path-style in documentation.

Updates support contact URLs to use the current support site, support.github.com.

Packages have been updated to the latest security versions.

Running ghe-repl-start or ghe-repl-status would sometimes return errors connecting to the database when GitHub Actions was enabled.

Pre-receive hooks would fail due to undefined PATH.

Running ghe-repl-setup would return an error: cannot create directory /data/user/elasticsearch: File exists if the instance had previously been configured as a replica.

After setting up a high availability replica, ghe-repl-status included an error in the output: unexpected unclosed action in command.

In large cluster environments, the authentication backend could be unavailable on a subset of frontend nodes.

Some critical services may not have been available on backend nodes in GHES Cluster.

An additional outer layer of gzip compression when creating a cluster support bundle with ghe-cluster-suport-bundle is now turned off by default. This outer compression can optionally be applied with the ghe-cluster-suport-bundle -c command line option.

We have added extra text to the admin console to remind users about the mobile apps' data collection for experience improvement purposes.

The GitHub Connect data connection record now includes a list of enabled GitHub Connect features. [Updated 2021-12-09]

A path traversal vulnerability was identified in GitHub Pages builds on GitHub Enterprise Server that could allow an attacker to read system files. To exploit this vulnerability, an attacker needed permission to create and build a GitHub Pages site on the GitHub Enterprise Server instance. This vulnerability affected all versions of GitHub Enterprise Server prior to 3.3, and was fixed in versions 3.0.19, 3.1.11, and 3.2.3. This vulnerability was reported through the GitHub Bug Bounty program and has been assigned CVE-2021-22870.

Packages have been updated to the latest security versions.

Some Git operations failed after upgrading a GitHub Enterprise Server 3.x cluster because of the HAProxy configuration.

Unicorn worker counts might have been set incorrectly in clustering mode.

Resqued worker counts might have been set incorrectly in clustering mode.

If Ubuntu's Uncomplicated Firewall (UFW) status was inactive, a client could not clearly see it in the logs.

Upgrading from GitHub Enterprise Server 2.x to 3.x failed when there were UTF8 characters in an LDAP configuration.

Some pages and Git-related background jobs might not run in cluster mode with certain cluster configurations.

When a new tag was created, the push webhook payload did not display a correct head_commit object. Now, when a new tag is created, the push webhook payload now always includes a head_commit object that contains the data of the commit that the new tag points to. As a result, the head_commit object will always contain the commit data of the payload's after commit.

The enterprise audit log page would not display audit events for secret scanning.

There was an insufficient job timeout for replica repairs.

Users were not warned about potentially dangerous bidirectional unicode characters when viewing files. For more information, see "Warning about bidirectional Unicode text" in the GitHub Blog.

Hookshot Go sent distribution type metrics that Collectd could not handle, which caused a ballooning of parsing errors.

Kafka configuration improvements have been added. When deleting repositories, package files are now immediately deleted from storage account to free up space. DestroyDeletedPackageVersionsJob now deletes package files from storage account for stale packages along with metadata records.

It was possible for cleartext passwords to end up in certain log files.

Several known weak SSH public keys have been added to the deny list and can no longer be registered. In addition, versions of GitKraken known to generate weak SSH keys (7.6.x, 7.7.x and 8.0.0) have been blocked from registering new public keys.

Packages have been updated to the latest security versions.

Restore might fail for enterprise server in clustering mode if orchestrator isnt healthily.

Several parts of the application were unusable for users who are owners of many organizations.

Fixed a link to https://docs.github.com.

Browsing and job performance optimizations for repositories with many refs.

Packages have been updated to the latest security versions.

Custom pre-receive hooks could have failed due to t