Merge certain fields from one set of YAML files to another

Question

I have a set of config files in YAML in one folder and an almost identical set of files in another folder. I'm trying to migrate a site and I need to copy the "uuid" field from the source files into the dest ones. The "uuid" is different for each file. I couldn't find a script to do this, so have attempted with this:

SRC=$ROOT/in
DEST=$ROOT/out

cd $SRC
for f in *
do
  echo "Processing $f file..."
  # Find the original UUID
  repl = "$(cat $f |grep uuid)"
  echo "Found UUID: $repl"
  # Replace line into same file
  sed -i "s/^uuid:.*$/$repl/" $DEST/$f
done

The $repl variable is never populated, although I know 95% of files have that line and running cat FILENAME |grep uuid matches.

Is there a better way to do this? And/or what am I doing wrong?

Update: Thanks @sjsam I have updated. However, output still looks like:

Processing views.view.who_s_online.yml file...
./merge-uuids.sh: line 11: repl: command not found
Found UUID: 
sed: 1: "/Users/nic/git/cn-d8/co ...": extra characters at the end of n command

Update 2: @Anthon In my case the uuid: ... line is always the first line (and it's always a first order key), but not all files actually have this line. In any case shouldn't the sed regex match the uuid: prefix wherever it is in the file?

Update 3: Thanks @Anton, After fixing the spaces around repl I am actually reading the values, but sed still unhappy. Output now:

Processing views.view.who_s_online.yml file...
Found UUID: uuid: 07cc0b9d-feda-42cf-9584-cc3f640cbb41
sed: 1: "/Users/nic/git/cn-d8/co ...": extra characters at the end of n command

Do a need flag for regex line matching or something?

Update 4: Looks like I'm hitting some FreeBSD/OSX problem (see sed extra characters at end of l command). I've tried changing the sed line to:

  sed -i .orig "s!^uuid:.*$!${repl}!;" "${DEST}/${f}"

and now the errors are like:

Processing blah.yml file...
Found UUID: uuid: 5940516b-c188-4d78-8e85-cff8f746ba67
sed: 1: "s!^uuid:.*uuid: 5940516 ...": unterminated substitute in regular expression

Update 5: I changed the line to echo "Found UUID: '${repl}'" and output is like Found UUID: 'uuid: eff29d5a-a9fe-4430-a7e9-2a6ecb944ce0' - i.e. characters are just colon, alphanumeric and spaces.

Use double quotes `sed -i "s/^uuid:.*$/$repl/" "$DEST/$f"`. Within single quotes shell variables are not expanded. — sjsam, Sep 05 '16 at 10:23
Also `cd $SRC` can be problematic if `$SRC` contains spaces Do `cd "$SRC"`. Worth trying [\[ shellcheck \]](http://shellcheck.net). — sjsam, Sep 05 '16 at 10:26
There is no guarantee in YAML that the value for the `uuid` key is on the same line. You should use a real parser (e.g. in Python, Perl, or Ruby) — Anthon, Sep 05 '16 at 10:27
You should at least remove the spaces around `=` in `repl = $(...)` — Anthon, Sep 05 '16 at 10:34
`sed` is unhappy coz you have `/` in filenames which gets mixed with `s` commands `/`. You're free to use any delimiter. — sjsam, Sep 05 '16 at 10:56
I'm now trying `sed -i "s#^uuid:.*$#${repl}#" "$DEST/$f"` but getting the same `extra characters at the end of n command` error — Nic Cottrell, Sep 05 '16 at 11:38

Мона_Сах · Answer 1 · 2016-09-05T10:54:21.167

1

Multiple issues with this one:

cd "$SRC" # double quote to prevent word splitting
.
.
repl="$(grep '^uuid' "$f")" # no spaces around =, avoid useless use of cat
.
.
sed -i "s#^uuid:.*$#${repl}#" "${DEST}/${f}" 
#Double quote and use `${}` to easily detect errors
#Use a character other than '/' as the substitution delim, I used '#'

edited Sep 05 '16 at 10:54

answered Sep 05 '16 at 10:43

Мона_Сах

322
3
12

It seems that the `$` after `.*` caused the final problem. By changing to ` sed -i .orig "s!^uuid:.*!${repl}!;" "${DEST}/${f}"` everything worked. Maybe that `$` needed escaping? – Nic Cottrell Sep 06 '16 at 08:47

Merge certain fields from one set of YAML files to another

1 Answers1