0

I am running this piece of code:

annotate-output $((sed -E 's/^[ ]+//;'  <<____COMMAND

sshfs 
foo_user@fooserver.com:/sftp_folder 
  /var/sshfs.sandbox/server.com 
  -o 
    user=foo_user
    ,reconnect
    ,ServerAliveInterval=15
    ,ServerAliveCountMax=3

____COMMAND
) | sed -E -e ':a;N;$!ba;s/\n//g')

It's my (perhaps clumsy) attempt at having a generic way of achieving multiline in bash. Note that there is a trailing space whenever it is needed by the command I'm trying to split into multple lines:

sshfs 
foo_user@fooserver.com:/sftp_folder 
  /var/sshfs.sandbox/server.com 
  -o

And no trailing spaces for options that have to be concatenated without an intervening blank space:

user=foo_user
,reconnect
,ServerAliveInterval=15
,ServerAliveCountMax=3

It works ok if I don't have quotes or double quotes anywhere. If I run this:

annotate-output $((sed -E 's/^[ ]+//;'  <<____COMMAND

sshfs 
foo_user@fooserver.com:/sftp_folder 
  "/var/sshfs.sandbox/server.com"
  -o 
    user=foo_user
    ,reconnect
    ,ServerAliveInterval=15
    ,ServerAliveCountMax=3

____COMMAND
) | sed -E -e ':a;N;$!ba;s/\n//g')

I get:

fuse: bad mount point `"/var/sshfs.sandbox/server.com"': No such file or directory

It looks like the quotes are passed to sshfs as part of the "argument". Why?

Edit 1 (20170418 0739) [note this is now a separate question]:

I understand now why the quotes are preserved (ty Charles Duffy - see his answer below)

I modelled my solution after this answer. I need to apply sed twice, once for the leading space , and another one for the EOL, and that's why I end up using those comprehensions. PS: I have leading spaces, not leading tabs, hence the <<- marker that removes leading tabs is not useful

My original problem is that I want a boilerplate header/footer that I can paste in any script, and then just type the command enclosed in this header footer as if I didn't have to worry about indentation. I know I can solve the problem by assigning to a variable and then using the results later on, but I don't want that. I would like to be able to enter the newlined, indented command only in one place. Basically. with these characteristics:

  • End of line spaces before the actual EOL preserved
  • End of lines removed
  • New line leading spaces removed

Is this possible?

Edit 2 (20170421 0818): The original question has been answered. Please follow the evolution of the related problem I'm trying to solve here

Community
  • 1
  • 1
lucid_dreamer
  • 362
  • 4
  • 9
  • 1
    Covered in detail in [BashFAQ #50](http://mywiki.wooledge.org/BashFAQ/050). – Charles Duffy Apr 17 '17 at 20:37
  • BTW -- would you consider re-titling this "How can I split a command containing quotes across multiple lines without backslashes?" an appropriate change? – Charles Duffy Apr 17 '17 at 20:45
  • Two notes: (1) -- if the question you originally asked has an answer, and you now have a new/different question, that new/different question should be asked as a separate question -- that way edits to the original question don't invalidate that answer. (2) -- More description of what you want and why you want it would be helpful. "Achieving multiline in bash" may mean something useful to you, but I promise you, it doesn't mean that same thing to everyone. – Charles Duffy Apr 18 '17 at 15:57
  • Err, make it three notes: Also, when you say you "don't want" a solution based on string manipulation -- string manipulation is exactly what your original `sed` approach *is*, so it's very unclear what that statement should be interpreted to mean. – Charles Duffy Apr 18 '17 at 15:57
  • I have edited it. Pay attention to "I would like to be able to enter the newlined, indented command only in one place.". That "only in one place" is the key. Your solution splits the command. – lucid_dreamer Apr 20 '17 at 03:26
  • Regarding (1): I have accepted your answer. I will repost another question. Regarding (2). There is enough description I believe. I don't understand you quoting only "Achieving multiline in bash" . and leaving out all the rest e.g. "in one go" in the title, and the in the bullet points: " I would like to be able to enter the newlined, indented command only in one place. Basically. with these characteristics: End of line spaces before the actual EOL preserved End of lines removed New line leading spaces removed " – lucid_dreamer Apr 20 '17 at 03:29
  • I want to enter it in one place only. Exactly like the sed solution lets me do. I don't want to enter it in two places like your proposed solution. I can't be any more clear than this. – lucid_dreamer Apr 20 '17 at 03:30
  • Where *is* that separate question? Your profile (at http://stackoverflow.com/users/1712447/user1712447) only has this single question posted. – Charles Duffy Apr 20 '17 at 15:54
  • I had not submitted it yet. Updated and included. This answer is good as it is. i have referenced your second answer from there. Perhaps you can answer including excerpts / copy pasting [your answer below](http://stackoverflow.com/a/43524520/1712447) - and I'll accept there if/when appropriate. – lucid_dreamer Apr 21 '17 at 07:23

2 Answers2

1

An approach that actually works

IFS=,  # cause "${array[*]}" to combine all items with commas

options=(
  user=foo_user           # can put a comment here if you like
  reconnect               # or here
  ServerAliveInterval=15  # or so forth
  ServerAliveCountMax=3
)

sshfs_cmd=(
  sshfs 
  foo_user@fooserver.com:/sftp_folder 
    /var/sshfs.sandbox/server.com
      -o "${options[*]}"
)

"${sshfs_cmd[@]}"

Explaining why the other approach doesn't

An unquoted expansion goes through the following parsing steps, and only the following parsing steps:

  • String-splitting (splitting into multiple distinct words based on characters in IFS)
  • Glob expansion (replacement of any word which looks like a glob expression with a list of names matching that expression).

That's it. No quoting, no quote removal, no parameter substitution, etc. Without quotes being parsed as syntax, variables they would cause to be parsed as a single word aren't parsed in that manner; and without quote removal, those quotes are present as literal data in the commands being invoked.

See BashFAQ #50 for a detailed discussion.

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
  • When referring to "An unquoted expansion", do you mean an unquoted here document? Or are you referring to the expansion of the command substitution? – Fred Apr 17 '17 at 21:11
  • That would be the latter. – Charles Duffy Apr 17 '17 at 21:35
  • OK, thanks for the explanation, it makes sense. But I have a problem with your solution: it splits up the command. I know I can achieve what I want with string manipulation - but I don't want that. I want a generic header/footer that I can paste in any script and just type the command I would type if I didn't have to worry about indentation. Isn't there a way to make this work that way? – lucid_dreamer Apr 18 '17 at 06:13
  • The only reason it's split up here is because you have the comma-separated options that you want to be magically joined into a single string, instead of separated into distinct arguments as is the case for everything else. Otherwise, there'd be no need for `options` to be distinct, or for `IFS` to be modified. – Charles Duffy Apr 18 '17 at 13:47
  • Well, I understand that, but certainly is not magic, considering the sed solution I posted does exactly that (it has a problem with quotes because of the unquoted expansion issues you mentioned; but that's a separate problem, I can use it if I don't need to quote commands). You just include space, whenever you need it, AT THE END of the line. – lucid_dreamer Apr 20 '17 at 03:21
  • ...waitaminute. "Space, when you need it, at the end of the line" -- so you're depending on invisible contents that someone reading your code can't tell is there to determine how it parses? That's a **horrible** idea -- look at how much trouble Makefiles cause with the spaces-vs-tabs distinction. Now, you're looking at making *trailing* whitespace semantically meaningful -- keep in mind that many modern versions of git actually warn whenever someone checks in code with trailing whitespace, and many editors offer to remove it automatically! – Charles Duffy Apr 20 '17 at 15:51
  • ...at least someone can tell that *leading* whitespace is there (because content after it is indented!) -- trailing whitespace is as invisible as a character gets without becoming nonprinting entirely. – Charles Duffy Apr 20 '17 at 16:07
  • I have replied to your question below (basically, I can see the wisdom of what you are saying).I understand your "magically" comment a bit better now. Because at one point you have to tell the shell that it shouldn't join those few lines in the way it's joining the rest. that is probably doable by modifying the "stripping" sed (or whatever) expression (i.e. modify first (1, k) lines by substituting leading whitespace with one namespace.Then for (k+1, n) lines remove ALL leading whitespace), but I can't really copy paste this header and footers around, as k might change, or not be there at all. – lucid_dreamer Apr 21 '17 at 06:51
  • Basically there is no way of having all three: (1) keeping the statement all together (2) not having trailing whitespace (3) making the "header" and "footer" fixed and copy pastable in other scripts without modifications. – lucid_dreamer Apr 21 '17 at 06:53
  • Also, would you mind expanding a bit on "${sshfs_cmd[@]}" ? – lucid_dreamer Apr 21 '17 at 06:55
0

Using eval

The below does what you want -- removing leading but not trailing whitespace, and combining heredoc contents into a single command. (There's no need to go to lengths to preserve a trailing newline at the end of a command: eval "true" works perfectly well even without true ending in a newline literal).

eval "$(
  { sed -E -e 's/^[ ]+//;' -e ':a;N;$!ba;s/\n//g' | tr -d '\n'; } <<'____COMMAND'
sshfs 
foo_user@fooserver.com:/sftp_folder 
  /var/sshfs.sandbox/server.com 
  -o 
    user=foo_user
    ,reconnect
    ,ServerAliveInterval=15
    ,ServerAliveCountMax=3
____COMMAND
)"

Some items to note:

  • The sigil in the heredoc (____COMMAND) is quoted. This ensures that expansions only happen later, after eval is called, vs earlier (when the heredoc's contents themselves are being expanded).
  • Multiple commands can be passed to a single sed invocation using an invocation of -e per each.

But Don't. Seriously.

Giving semantic meaning to trailing whitespace is an easy way to confuse anyone else reading your code. Just as tabs-vs-space questions have concerned generations of developers working with Makefiles or with Python (before PEP-8 standardized on spaces only), having trailing-space vs no-trailing-space be a semantically meaningful distinction is going to seriously confuse anyone reading your code, and will lead to conflicts with automated format validation (such as functionality included in current releases of git that emit warnings when any line contains trailing whitespace) in the future.

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
  • Thank you for the insight. But how would a reader be confused? What's the alternative interpretation of the command above? None. someone collapsting that command in a line would have to collapse the leading whitespace manually anyway, at which point the issue of whether there was a trailing whitespace becomes irrelevant. PS: I reieterate: I do appreciate your comments, but I want to understand what is the difference in practice. I get the point on git warnings - and that alone is a good reason. But is that all? In practice what is the difference? – lucid_dreamer Apr 21 '17 at 06:31
  • But I can now see why the first solution you proposed uses a different "method", defined in a separate statement, for joining the arguments: there is no way that this can be entered in one go without making trailing whitespace significant. And applying the removal of leading whitespace only on some lines is going to end up being more kludgey than just definining and combining the array in the way you've done. – lucid_dreamer Apr 21 '17 at 06:41
  • Another idea. Can I use a trailing backslash, to make the trailing whitespace readable? I do that in python already and quite used to it. And how would that affect your sed / tr command? – lucid_dreamer Apr 21 '17 at 07:00
  • This doesn't work. The leading white space is convered to a single whitespace like this: `,reconnect ,ServerAliveInterval=15 ,ServerAliveCountMax=31` – lucid_dreamer May 09 '17 at 21:02
  • Quoting from `set -x` logs from running the command on my local system copied-and-pasted direct from the question: `++ sshfs foo_user@fooserver.com:/sftp_folder /var/sshfs.sandbox/server.com -o user=foo_user,reconnect,ServerAliveInterval=15,ServerAliveCountMax=3`. There are *not* any spaces preceding the commas. – Charles Duffy May 09 '17 at 22:49