How to set up/scan C & C++ project via command line?

[davewichers]

I started with the following:

• codeql database create path/to/DBNAME --language=cpp --command="MYBUILDCOMMAND"
• codeql database analyze path/to/DBNAME codeql/cpp-queries --format=sarifv2.1.0 --output=FILENAME.sarif

And when the analyze completes, it says: "CodeQL scanned 2 out of 1562 C/C++ files in this invocation. Typically CodeQL is configured to analyze a single CodeQL language per invocation, so check other invocations to determine overall coverage information."

I looked at the documentation, and apparently you can set the language to multiple languages, e.g., --language=cpp,c

But when I do that, it returned an error during database create that I had to also specify --db-cluster. So, I added that, so now my first command is:

• codeql database create path/to/DBNAME --language=cpp,c --db-cluster --command="MYBUILDCOMMAND"

But then, when I run the database analyze step, I now get this error:

"A fatal error has occurred: path\to\DBNAME is not a recognized CodeQL database."

How do I fix that?

[jketema]

c and cpp are just aliases, so there's no reason to specify both of them, and your initial

codeql database create path/to/DBNAME --language=cpp --command="MYBUILDCOMMAND"

was correct.

If only 2 files were scanned, then we detected only 2 C/C++ compiler invocations during database creation. The most common reason for this is that build artifacts were not cleaned before creating the database, and subsequently build steps were skipped. Another common reason is the use of caching, e.g., with ccache, which is not supported. It is hard to tell though what is going on without further details.

[davewichers]

That didn't help. I deleted all the .obj files in the directory where the compile command is run, and I can see the compilation of every file happening. For each source file I'm seeing: [DATE/TIME] [build-stdout] FILENAME.c

The parameters to the cl invocation are: cl /I "path\to\supportclasses" /W3 /MT /GS /RTC1 /bigobj /EHsc /nologo /MP /c main.cpp STANDARD_PREFIX*.c STANDARD_PREFIX*.cpp (Note: /I is slash capital I, not L)

An .obj file is created for each source file during this process.

I'm using 'cl' as the compiler if that makes any difference. A single invocation of cl with wildcards for all the files to compile is used. i.e., STANDARD_PREFIX*.c STANDARD_PREFIX*.cpp along with paths to the supporting classes in other paths.

[jketema]

I think you hit a limitation in our tooling: we don't handle wildcard expansion on Windows. You should be able to confirm this by looking at the build-tracer.log file that occurs somewhere in the database directory. I'd expect you'll see something along the following lines:

Catastrophic error: cannot open source file "STANDARD_PREFIX*.c": Invalid argument

[davewichers]

Yes, I'm seeing those errors in the logs. Do you have an open bug report and are working on a fix? If so, any ETA for the fix?

In the meantime, is there a workaround on Windows that does work? I.e., can I use a bash shell, or Windows Subsystem for Linux shell, or some other shell that does work?

If you have no other suggestions, then I guess I can try it on Rocky 9 instead (or some Linux flavor)?

[jketema]

We have an internal issue open, but fixing this has not been prioritized, because this is pretty much an edge case (build systems generally specify all files explicitly). The workaround would be to specify all files explicitly. I would assume you can do this by using a Bash shell on Windows, but I have not tested that.

[jketema]

Fixed in CodeQL 2.26.1.

[davewichers]

The problem with that is that CL, which is the Windows compiler installed and made accessible in the Visual Studio Command prompt when you install Visual studio is NOT accessible via bash or WSL, so you can't run CL outside of a Visual Studio Command prompt. So, wildcard expansion in Windows shell really should be fixed to make this work.

[jketema]

The problem with that is that CL, which is the Windows compiler installed and made accessible in the Visual Studio Command prompt when you install Visual studio is NOT accessible via bash or WSL, so you can't run CL outside of a Visual Studio Command prompt.

That's not true. You can make the compiler accessible from any prompt. It's not tied to the command prompts that Visual Studio provides. I can understand that you don't want to go that route that, but then the question is: why can't specify all files explicitly?

[davewichers]

It's actually very complicated to make cl work in a non-VS command prompt as you have to run vcvarsall.bat to set up the proper environment for cl AND cl.exe in the same command line. So, while it is technically possible, it is very complicated/hard to get work. And when you have 1500+ source files in each folder, and I have over one hundred folders, listing each source file is a pain. So, it would be massively simpler for the user if you could get wildcard expansion to work in Windows. I'm trying to get the vcvarsall.bat plus cl.exe setup to work but no luck. Way too complicated.

[jketema]

I'm just trying to offer workarounds here, as this is not something we can fix instantaneously. Especially since this is a bit of an edge case from our perspective, as I mentioned already.

And when you have 1500+ source files in each folder, and I have over one hundred folders, listing each source file is a pain.

Why list them by hand? The CL supports response files, so you could just write a script that generates the appropriate response files.

[davewichers]

OK. I tried that but ran into an error. I created two .rsp files, the 1st is the standard parameters to CL, and the 2nd is the list of all the source files to compile. So, I can run this just fine: cl @standardParams.rsp @listOfAllSourceFilesToCompile.rsp

listOfAllSourceFilesToCompile.rsp lists1776 source files, 1 per line.

This works no problem, but when I pass that cl command as the --command parameter value to: 'codeql database create', it responds with the error: "(eventual cause: IOException "CreateProcess error=206, The filename or extension is too long")"

So, it doesn't look like you can compile all of the source files in a single invocation when creating the database.

When you invoke codeql database create, the is shows this output:

Initializing database at: path\to\DBNAME
Running build command: [cl, @standardParams.rsp, @listOfAllSourceFilesToCompile.rsp]
Running command in DIRNAME: [cl, then each parameter from the first rsp file followed by a COMMA and SPACE, then each filename from the 2nd rsp file also followed by COMMA and SPACE, and then the error:
(eventual cause: IOException "CreateProcess error=206, The filename or extension is too long")