# How to Add Python Duplicate-Code Detection to GitHub Actions Duplicate code is easiest to deal with before it becomes part of the codebase. That makes duplicate-code detection a natural CI check: run the same rules on every pull request, give developers a way to reproduce the result locally, and catch new duplication before it quietly becomes another piece of technical debt. The GitHub Actions part is straightforward. If a duplicate-code checker has a CLI, meaningful exit codes, and predictable output, GitHub Actions can run it. The more important decisions are what you scan, what constitutes a failure, and what to do when an existing repository already contains duplication. Those decisions determine whether the check becomes a useful engineering constraint or just another red CI job that everyone learns to ignore. ## CI Should Enforce a Policy, Not Just Run a Command The simplest duplicate-code policy is also the strictest: if any duplicate code is found, fail the build. That works well for a new project or a repository that is already clean, but it can be a bad adoption strategy for a mature codebase. Suppose you add duplicate detection to a five-year-old project and the first run finds 150 existing duplicate groups. The detector has done exactly what you asked, but CI can no longer pass until someone fixes five years of accumulated debt. Teams faced with that situation usually end up choosing between stopping feature work, accepting a permanently failing check, or disabling the check again. None of those outcomes are especially useful. Before adding the workflow, decide what you actually want CI to enforce. You may want zero duplication, you may initially want visibility without enforcement, or you may want to accept existing duplication while preventing new duplication from being introduced. Those are three different policies even though they can all use the same detector. ## A Minimal GitHub Actions Workflow GitHub Actions does not need any special understanding of duplicate-code detection. At its simplest, the job checks out the repository, runs the detector over the Python source, and interprets the result. For the examples here I am using [Arid](https://github.com/sponge-b0b/arid), the Python duplicate-code checker I built to provide focused, Python-aware duplicate detection without requiring a general-purpose linter. Arid has an official GitHub Action, so a minimal workflow looks like this: ```yaml name: Duplicate code on: pull_request: push: branches: [main] jobs: duplicate-code: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Check for duplicate Python code uses: sponge-b0b/arid@v2.2.3 with: paths: . ``` That is enough for a basic enforcing check. The Action handles the Python setup and installation needed to run Arid, scans the repository, and fails when active duplicate findings are present. For a new or already-clean project, this may be all you need. ## Keep Detection Policy in the Project One of the easiest ways to make CI irritating is to hide important tool configuration inside the workflow file. Developers then run one thing locally while CI quietly runs something else. A better separation of responsibilities is to let GitHub Actions decide **when** the tool runs and let the project configuration decide **how** it runs. For example: ```toml [tool.arid] min-lines = 6 exclude = [ "generated/**", ] ``` The workflow itself can remain unchanged: ```yaml - name: Check for duplicate Python code uses: sponge-b0b/arid@v2.2.3 with: paths: . ``` A developer can reproduce the same policy locally with: ```bash arid . ``` That property matters more than it may appear. CI failures are much easier to work with when the developer can reproduce the exact same check before pushing another commit. The workflow should automate the repository's quality policy, not define a second, hidden version of it. ## Decide Whether Findings Should Fail CI Enforcement does not have to begin on day one. In an existing codebase, it can be useful to run duplicate detection for a while before making findings blocking. That gives you a chance to understand what the detector reports, adjust exclusions or thresholds, and decide what level of duplication the team actually wants to enforce. With the Arid Action, that can be done by disabling failure on findings: ```yaml - name: Report duplicate Python code uses: sponge-b0b/arid@v2.2.3 with: paths: . fail-on-findings: "false" ``` The important distinction is that this changes the policy for **duplicate findings**, not the meaning of every possible failure. A completed scan that finds duplication is different from a scan that cannot complete because of invalid configuration, unreadable source, or another operational problem. That distinction is worth preserving in any CI tool. "We found something and chose not to block on it" is a policy decision. "The analysis did not complete" is a different state entirely. For many mature repositories, a reasonable adoption path is to begin in reporting mode, tune the check, and only then turn it into a required gate. There is little value in introducing a quality check so aggressively that the first lesson developers learn is how to bypass it. ## Brownfield Repositories Need a Different Strategy Reporting-only mode is useful during evaluation, but eventually you may want to enforce duplicate-code policy without first eliminating every duplicate that already exists. That is where baselining becomes useful. A baseline records the findings that already exist and treats them as acknowledged debt. Future scans can then distinguish between duplication the repository already had and duplication introduced later. The CI policy becomes "do not add new duplicate code" rather than "repair the entire history of the repository before the next pull request can merge." That is often a much more practical constraint for a brownfield project. Existing debt remains visible and can be reduced over time, but it does not prevent the team from enforcing a better rule going forward. There are enough subtleties around baseline creation, maintenance, stale findings, and debt reduction that baselines deserve a separate article. The important point here is that CI enforcement does not have to be binary. You do not have to choose between fixing everything immediately and ignoring duplication forever. ## Duplicate Detection Is a Corpus-Level Problem There is another CI mistake that is less obvious: scanning only the files changed by the pull request. That optimization makes sense for many tools. A formatter can format one file. A linter can inspect one file. A type checker may be able to limit part of its work to a dependency boundary. Duplicate detection is different because a duplicate is a relationship between two or more regions of code. Suppose a pull request adds this: ```python def normalize_items(items): result = [] for item in items: if item is None: continue result.append(str(item).strip()) return result ``` The duplicate may already exist in `src/legacy/importer.py`. If CI analyzes only the newly changed file, there is nothing to compare it with. The file appears unique because the rest of the corpus has been removed from the analysis. This is one of those cases where a performance optimization can quietly change the question being asked. "Does this changed file contain something that duplicates code elsewhere in the repository?" cannot be answered by looking only at the changed file. Arid separates those concerns with its `focus` option. The full corpus can still be analyzed while reporting is restricted to findings that touch the area you care about: ```yaml - name: Check package for new duplicate code uses: sponge-b0b/arid@v2.2.3 with: paths: . focus: src/package ``` The distinction is important: **focus the report, not the detector**. You can reduce noise without throwing away the context required to detect duplication correctly. ## CI Results Should Be Useful to Both Humans and Automation A command returning success or failure is enough to build a gate, but CI usually benefits from richer output. The Arid Action writes a Markdown report to the GitHub job summary by default, which means developers do not have to dig through raw workflow logs just to see what was found. It also exposes structured outputs such as duplicate-group counts, duplicated lines, duplication percentage, affected files, occurrence counts, scan completeness, and the underlying scan exit code. For example: ```yaml - name: Check for duplicate Python code id: arid uses: sponge-b0b/arid@v2.2.3 with: paths: . - name: Show duplicate metrics run: | echo "Duplicate groups: ${{ steps.arid.outputs.duplicate-groups }}" echo "Duplicate lines: ${{ steps.arid.outputs.duplicate-lines }}" ``` You may never need those outputs, and that is fine. Their value is that another piece of automation does not have to parse terminal text intended for a human. Human-readable output and machine-readable output serve different purposes. Trying to make one format do both usually produces output that is unpleasant for humans and fragile for machines. ## Add SARIF Only If It Improves the Workflow GitHub code scanning understands SARIF, so Arid can also publish duplicate findings through that interface. If your team already uses GitHub code scanning for static-analysis results, putting duplicate findings there can make sense. A workflow enabling SARIF looks like this: ```yaml name: Duplicate code on: pull_request: push: branches: [main] permissions: contents: read security-events: write jobs: duplicate-code: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Check for duplicate Python code uses: sponge-b0b/arid@v2.2.3 with: paths: . sarif: "true" ``` I would not enable SARIF merely because it exists. If developers already get everything they need from the job summary, adding another reporting surface may not improve anything. The useful architectural property is that reporting remains separate from analysis. Arid performs the scan once and can derive the job summary, Action outputs, and optional SARIF result from that same analysis. You should not need to run the detector three times because three consumers want the result in three different formats. ## A Reasonable Adoption Sequence For a new or clean repository, I would start with the simple enforcing workflow: ```yaml name: Duplicate code on: pull_request: push: branches: [main] jobs: duplicate-code: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Check for duplicate Python code uses: sponge-b0b/arid@v2.2.3 with: paths: . ``` For an established repository where you do not yet know what the check will uncover, I would begin with the same workflow but set `fail-on-findings` to `false`. Once the configuration is producing useful results, decide whether the repository should enforce zero duplication or whether existing findings should be baselined so that only new duplication fails CI. That progression is intentionally boring. Good CI policy usually is. The important part is that the detector does not dictate the adoption model. A greenfield project, a project evaluating the tool, and a brownfield project with years of historical duplication can all use the same underlying analysis while enforcing different policies. ## Is Arid the Right Tool for This? Not necessarily. If Pylint `R0801` already performs well enough for your repository and fits the rest of your tooling, there may be no reason to replace it. If you have a multi-language monorepo and want one clone detector for everything, a Python-specific tool is probably the wrong abstraction. If your real requirement is semantic clone detection—finding code that behaves similarly even though it is written differently—you need a detector designed for that problem. Arid deliberately has a narrower scope. It detects exact duplicated Python source after configurable Python-aware normalization, and it is designed to run alongside tools such as Ruff rather than become another general-purpose linter. That narrowness is why I built it. I had already moved most of my Python linting workflow to Ruff but still wanted corpus-level duplicate detection. Carrying a general-purpose linter primarily for one expensive check felt like the wrong dependency boundary, so I built a tool whose responsibility begins and ends with duplicate-code detection. The local workflow is intentionally unremarkable: ```bash ruff check . arid . ``` The GitHub Action simply makes the second command part of the repository's normal engineering process. ## The YAML Is the Easy Part Adding duplicate-code detection to GitHub Actions takes only a few lines. Deciding what those lines should mean is the more interesting part. A good CI check answers a clear policy question. Are duplicate findings always forbidden? Are you still evaluating the signal? Are existing findings accepted while new ones are blocked? Are you analyzing the whole repository even when you only want to report findings in a particular area? Those choices determine whether duplicate detection becomes useful or merely becomes present. The goal is not another green checkmark in a pull request. The goal is a constraint developers understand, can reproduce locally, and can keep green without working around it. Once the policy is clear, GitHub Actions is just the mechanism that makes it consistent. * * * **Arid** is an open-source Python duplicate-code checker written in Rust, designed as a focused replacement for Pylint `R0801` and to complement Ruff. GitHub: https://github.com/sponge-b0b/arid GitHub Marketplace: https://github.com/marketplace/actions/arid-duplicate-code-check ### About the Author Bob Taylor is a software engineer and architect who builds developer tools and AI systems. He is currently developing [Arid](https://github.com/sponge-b0b/arid) and [Polaris](https://github.com/sponge-b0b/Polaris).