Skip to content

feat: add general experimentation infrastructure and telemetry attribution - #1873

Open
Stella Huang (StellaHuang95) wants to merge 4 commits into
microsoft:mainfrom
StellaHuang95:experimentation-infrastructure
Open

Stella Huang (StellaHuang95) wants to merge 4 commits into
microsoft:mainfrom
StellaHuang95:experimentation-infrastructure

Conversation

@StellaHuang95

Copy link
Copy Markdown
Contributor

Summary

Prepare reusable experimentation infrastructure for the Python Environments extension without enabling an experiment or changing feature settings.

  • Add vscode-tas-client with one activation-owned service, validated publisher configuration, isolated globalState caching, defaulted treatment queries, bounded HTTPS requests, and consent-aware cancellation and disposal.
  • Reuse a lifecycle-owned telemetry reporter and attach abexp.assignmentcontext to ordinary events, explicitly reported errors, and automatically captured exceptions. Preserve consent, data cleaning, capture-time attribution, and existing event names without duplicate automatic-error reporting.
  • Add classified SDK diagnostics that distinguish cache initialization from successful assignment fetching, deterministic lifecycle and transport coverage, and a contract test using the real SDK with a fake transport.
  • Document the configuration contract, onboarding prerequisites, and existing measurements and gaps for a baseline experimentation scorecard.

Scope and rollout

The publisher-owned experimentation block is intentionally absent. This build makes no extension TAS requests; existing consented telemetry continues, with a notConfigured initialization diagnostic. No feature setting is exposed or enabled, and no A/B test is launched.

Update TypeScript to support the current TAS SDK declarations and @vscode/extension-telemetry to 1.5.2 for its public automatic-error logger options. Generated native tooling and local verification artifacts are excluded.

Validation

Re-run for this PR:

  • npm run lint
  • npm run compile-tests
  • Full unit suite: 2,505 passing, 6 pending existing platform/symlink cases
  • npm run compile
  • git diff --cached --check

The implementation session also exercised Windows smoke/integration/discovery workflows, an isolated environment/package lifecycle, installed release VSIX behavior, and real-SDK/telemetry paths with controlled responses. Those broader checks were not repeated during this PR handoff.

Follow-up and limits

Live rollout still requires an approved endpoint, randomization identity and targeting contract, integration review, metrics/scorecard setup, and A/A verification. MachineId is not assumed to be DevDeviceId.

Production TAS authentication/proxies, non-Windows and remote hosts, and long-duration polling remain unverified. The earlier session reproduced independent multi-root selection and shutdown issues on the unchanged base commit; unrelated production behavior is deliberately outside this change.

…ution

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Comment thread src/common/experimentation/storage.ts
Prevent cached assignments from being reused when resolved audience parameters such as the VS Code language change.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@eleanorjboyd

Copy link
Copy Markdown
Member

Follow-up on the resolved cache-key thread: I think 14450bdb covers the caller-configured assignment parameters, but not all targeting values added by vscode-tas-client itself.

In vscode-tas-client@0.3.3, getExperimentationServiceFromConfig() automatically adds:

  • VSCodeAssignmentsFilterProvider, which sends vscode.version as ApplicationVersion and vscode.env.appName as Build to the assignments endpoint.
  • VSCodeFilterProvider, which includes VS Code version, app name, and language for the legacy endpoint.

Those responses share this Memento and are merged into the same cached SDK snapshot. However, the namespace in storage.ts contains only the configured/resolved assignmentParameters (plus the other explicit configuration fields). Therefore:

  1. Start with only the required machine-ID assignment parameter configured and cache a treatment in English.
  2. Change the VS Code display language and reload.
  3. The namespace is unchanged, so the English snapshot can be restored before the new request completes—or remain in use while offline.

The same mismatch applies to vscode.version and env.appName, which are always SDK-provided but are not part of resolvedAssignmentParameters in service.ts.

Could the cache identity include the SDK-provided built-in targeting dimensions as well? A regression test with only machineId configured, then changing language/version/app name, would cover the currently missed path.

Invalidate cached assignments when VS Code's built-in TAS targeting values change, including language, editor version, and application name.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@StellaHuang95

Copy link
Copy Markdown
Contributor Author

Eleanor Boyd (@eleanorjboyd) Thanks for catching this. Addressed in 436f6c72.

The cache namespace now includes the SDK-provided built-in targeting context—normalized VS Code application version, app/build name, legacy MachineId, and language—in addition to the caller-configured resolved assignment parameters. I also added the requested regression coverage using an identity-only configuration and verified that language, VS Code version, and app-name changes each produce a cache miss.

Validation passed with lint, TypeScript compilation, 37 focused experimentation tests, and the full unit suite (2,638 passing, 6 pending).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-changelog no-merge PR is currently blocked from merging

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants