There are two parts to my answer.
Firstly, if you're using dotnet pack or msbuild -t:pack, without a nuspec file, then project references automatically become nuget dependencies. I verified this using the following commands:
dotnet new classlib -n ProjectA
dotnet new classlib -n ProjectB
dotnet add ProjectB/ProjectB.csproj reference ProjectA/ProjectA.csproj
dotnet pack ProjectB/ProjectB.csproj
ProjectB.1.0.0.nupkg has a nuget dependency on ProjectA v1.0.0. This works "out of the box" with SDK style projects, but if you have an "old style" project (that imports Microsoft.CSharp.targets), you'll need to add a package reference to NuGet.Build.Tasks.Pack. If you use PackageReference, you may need to edit your csproj and set this dependency to set PrivateAssets to all to avoid this package becoming a nuget dependency. I don't know what the equivalent is if you're using packages.config, or if it's even possible. If you're using msbuild or dotnet pack, but you also have your own nuspec file, I recommend against it. Anything you want to do in the nuspec should be possible by setting the right properties or item attributes in your csproj, and let the pack targets generate the nuspec automatically. When you have your own nuspec, parts of nuget's auto-generation gets turned off, so you may be making things harder for yourself.
My memory of nuget pack on old style csproj files is that nuget will automatically make a project reference a nuget dependency as long as the referenced project has a <project name>.nuspec file in the same directory as the <project name>.csproj file (I'm almost certain that it's documented somewhere on learn.microsoft.com/nuget). However, if your dependant project is a SDK style project (like is needed for .NET Core or .NET Standard projects), then I already advised against using a nuspec file with those projects, so you should really consider using the pack targets and stop using nuget pack.
The second part of my answer will directly answer your question, which is about how to make a project reference a package reference on certain builds. Firstly, you need to use pack targets, either by using a SDK style project or by referencing the NuGet.Build.Tasks.Pack package. This way, you don't use a nuspec file, and everything is defined in your csproj, which is just a MSBuild file. MSBuild has conditions, so hand edit your csproj and make your project reference have a certain condition and have the package reference have the opposite condition. I suggest use a variable like $(CI), so you can test the pack using msbuild -t:pack -p:CI=true, and on your CI machine just set CI as an environment variable to true. Since NuGet already has functionality built-in for making project references into nuget dependencies, I recommend using that, and not switching between package and project references, so I'm not going to give a copy-paste example on how to do this. But if this isn't a case of a XY-problem and you really need to do this, then I've already given enough pointers for you to be able to figure out how to do the rest.
