7

Problem: I'm writing a script that performs several HTTP requests with curl and I want to add the headers to a variable, CURL_HEADERS, so that I don't have to type them out constantly. When I echo the CURL_HEADERS variable in the curl command, single quotations appear where I don't want them. How can I prevent this? (The code below is simplified for the sake of clarity)

Code

#!/usr/bin/env bash
AUTH_KEY='1234'
set -x
CURL_HEADERS='-H "Authorization: Basic '${AUTH_KEY}'" -H "Content-Type: application/json"'
echo "${CURL_HEADERS}"
curl -s $(echo "${CURL_HEADERS}") 'http://www.example.org' > /dev/null
set +x

Expected Output:

+ CURL_HEADERS='-H "Authorization: Basic 1234" -H "Content-Type: application/json"'
+ echo '-H "Authorization: Basic 1234" -H "Content-Type: application/json"'
-H "Authorization: Basic 1234" -H "Content-Type: application/json"
++ echo '-H "Authorization: Basic 1234" -H "Content-Type: application/json"'
+ curl -s -H "Authorization: Basic 1234" -H "Content-Type: application/json" http://www.example.org
+ set +x

Actual Output

+ CURL_HEADERS='-H "Authorization: Basic 1234" -H "Content-Type: application/json"'
+ echo '-H "Authorization: Basic 1234" -H "Content-Type: application/json"'
-H "Authorization: Basic 1234" -H "Content-Type: application/json"
++ echo '-H "Authorization: Basic 1234" -H "Content-Type: application/json"'
+ curl -s -H '"Authorization:' Basic '1234"' -H '"Content-Type:' 'application/json"' http://www.example.org
+ set +x
AJ Schmidt
  • 93
  • 2
  • 7
  • Well formatted question, thank you from a reviewer of first post and low quality post queues :) – Will Barnwell Sep 23 '17 at 00:13
  • 1
    The answer to your question is here: [I'm trying to put a command in a variable, but the complex cases always fail!](http://mywiki.wooledge.org/BashFAQ/050) – John1024 Sep 23 '17 at 00:23
  • You realize that lines starting with `+ ` are printing the command after shell processing? The single quotes will not appear in the output. The manual is not very elaborate about it but I suppose it's the shell's way of saying "this is a verbatim string and will not be subjected to further expansions or word splitting". And that that is so is visible in the presence of the double quotes (which are part of the string, not string delimiters (not to the shell, that is)). – Peter - Reinstate Monica Sep 23 '17 at 00:24
  • @peter-a-schneider Please be respectful, I'd assume OP knows what `set -x` does. Furthermore he correctly identified the cause of his failing curl command and then specifically asked how to prevent it. Just run OP's code to reproduce. Those single quotes may not be visible to the shell but they do show how the shell is reading them. – Will Barnwell Sep 23 '17 at 00:36
  • @WillBarnwell No disrespect intended. The OP didn't say that his command failed -- does it? As far as I can see (tired as I am) the output is exactly as expected, `"-H "Authorization: Basic 1234" -H "Content-Type: application/json"`. How the shell splits the words is ... oh. You mean the shell passes `"Authorization:` as the first argument, instead of `Authorization: Basic 1234`?? Then there is a problem ;-). I supposed the OP wanted to echo the string somewhere. – Peter - Reinstate Monica Sep 23 '17 at 00:41
  • It may be worth noting that command substitution with parentheses creates a new "quoting context" which makes life so much easier, here perhaps as well (although, as Will correctly observed in his post, echoing the variable is redundant and can simply be left out). So one can write `"$(echo "${CURL_HEADERS}")"`, quoting the argument inside the command substitution *as well as the overall result*. The double quotes are paired correctly, and the outer double quotes (instead of single quotes which are always allowed around double quotes) still make the argument variable subject to expansion. – Peter - Reinstate Monica Sep 23 '17 at 00:50
  • 1
    @PeterA.Schneider: `"$(echo "${CURL_HEADERS}")"` is exactly the same as `"${CURL_HEADERS}"` (aside from being a lot more work for bash), and it suffers from exactly the same problem: it produces a single argument; the OP wants there to be four arguments. Command substitution does not magically reinterpret the string as though it were part of a bash command. The internal quotes are still just ordinary characters. – rici Sep 23 '17 at 01:16
  • @rici You are right. – Peter - Reinstate Monica Sep 23 '17 at 03:20
  • Here is a possible dup: https://stackoverflow.com/q/25777802 – rici Sep 23 '17 at 03:44
  • This comes up a lot, but there are not any great dupes for it. @rici, do you want to write a quintessential "I'm trying to put quotes in a variable" question that we can add to the bash tag wiki? – that other guy Sep 23 '17 at 04:52

2 Answers2

11

A reasonably straightforward solution is to use a bash array to store the four arguments you will want to pass:

CURL_HEADERS=(
             '-H' "Authorization: Basic ${AUTH_KEY}"
             '-H' 'Content-Type: application/json'
)

curl -s "${CURL_HEADERS[@]}" 'http://www.example.org' > /dev/null

Unlike scalar variables, which are just ordinary strings of ordinary characters no matter how many quotes they might contain, arrays are lists of strings, each one distinguished from each other one. In this sense, bash is just like almost every other programming language.

This problem, and the solution I suggest as well as several other ones, is well-described in the Bash FAQ entry 50 (I'm trying to put a command in a variable, but the complex cases always fail!), which is worth reading in detail. (Link taken from a comment by @John1024.)

rici
  • 234,347
  • 28
  • 237
  • 341
0

The proposed array is a good design approach, but curl still adds single quotes to the array items that already contain double quotes around them, thus making invalid headers.

Kerry Classen
  • 1
  • 1
    No it doesn't. The single quotes are part of bash's `-x` option, which is trying to show you the elements unambiguously. They're not in the data itself. – rici Mar 11 '20 at 17:18