Replace substring in bash command output with return value of another command (e.g. turn timestamp into date)

Question

I have a command that outputs a Unix timestamp followed by some text:

> mycommand
1484169138 happened XYZ
1484169139 happened XZY
...

I want to turn the timestamps into human readable strings but do not seem to be able to figure out how.

I have tried various tools (like sed and awk). The following answer on stackoverflow seems quite close; but I am on FreeBSD which does not seem to have the GNU extension to use e:

$ mycommand |sed -E 's#([0-9]{10})#date -r \1#e'
sed: 1: "s#([0-9]{10})#date -r \1#e": bad flag in substitute command: 'e'

As an aside, you seem to be running `date -r XX`, but `-r` does not seem to be doing what you are looking for. What exact `date` command do you want to run? — fedorqui, Jan 19 '17 at 14:12

ghoti · Accepted Answer · 2017-01-20T15:33:27.673

1

You can reformat dates using the date command in a while loop in your POSIX shell (which I should point out is probably not your default shell in FreeBSD).

$ while read d t; do printf "[%s] %s\n" "$(date -r "$d" '+%F %T')" "$t"; done < x.txt
[2017-01-11 16:12:18] happened XYZ
[2017-01-11 16:12:19] happened XZY

The while loop can be the end of a pipe, too:

$ mycommand | while read d t; do ...; done

edited Jan 20 '17 at 15:33

answered Jan 19 '17 at 14:14

ghoti

45,319
8
65
104

Are you sure it is OK to set IFS to null? I did a quick test and had to set it to a space for read to separate the first field from the rest of the line in the two variables. – Fred Jan 19 '17 at 14:30
Please note that this seems to rely on a behavior of "read" which I did not know of before (and deduced by testing), that if there are more fields than variables, the last variable will be loaded with the whole remainder of the line being read from. – Fred Jan 19 '17 at 14:31
1

@fedorqui, I've rolled back your edit. `IFS=` breaks the division of the line so that everything goes into `$d`, and since we don't know whether the OP's `mycommand` generates backslash-joined multiline output, we don't know whether `-r` is appropriate. Please verify intent and test the impact before making significant changes like this. – ghoti Jan 19 '17 at 18:44
Shouldn't it be `date -d@` rather than `date -r`? I am getting this error: `date: 1484169138: No such file or directory [] happened XYZ date: 1484169139: No such file or directory [] happened XZY` – codeforester Jan 19 '17 at 20:21
ghoti: ok sorry if my edit caused inconveniences. It is strange, though, specially with the sample file you are showing. As a reminder, I was using the described method in [How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?](http://mywiki.wooledge.org/BashFAQ/001), which is accepted as the canonical. – fedorqui Jan 19 '17 at 21:54
Edit: I had to switch on my computer because I kept thinking on this while washing the wishes :D Yes, you are right, `IFS=` breaks it completely and the approach I was following is to read all the line into a variable. Sorry! – fedorqui Jan 19 '17 at 22:25
1

@codeforester - your edit to my answer breaks its functionality in FreeBSD, which does not have a `-d` option for the `date` command. Please consider the tags and the rest of the content before making edits like this -- you made a significant change to the code in my answer which converted it into an incorrect answer. – ghoti Jan 20 '17 at 15:33
(okay, to be pedantic, FreeBSD's `date` *does* have a `-d` option, but it does something completely [different](https://www.freebsd.org/cgi/man.cgi?query=date).) – ghoti Jan 20 '17 at 15:57
Sorry @ghoti. Point noted. – codeforester Jan 20 '17 at 16:03

mklement0 · Answer 2 · 2017-01-20T02:47:25.773

BSD sed doesn't support the GNU-sed-specific /e option that allows using a shell command to provide the replacement string in an s function call, as you've discovered.

However, awk can provide similar functionality:

$ awk '{ ("date -r " $1) | getline $1 } 1' <<'EOF'
1484169138 happened XYZ
1484169139 happened XZY
EOF
Wed Jan 11 16:12:18 EST 2017 happened XYZ
Wed Jan 11 16:12:19 EST 2017 happened XZY

("date -r " $1) constructs a shell command from the 1st field, containing the Unix epoch time in seconds, e.g., date -r 1484169138, which on a system with en-US locale in the Eastern time zone yields Wed Jan 11 16:12:18 EST 2017 (note that GNU date's -r option has different semantics altogether).
... | getline $1 executes the shell command (via sh -c) and captures the command's stdout output in variable $1; i.e., it replaces the 1st field with the command's output.
1 is a common shorthand for simply printing the (modified) record at hand.

Replace substring in bash command output with return value of another command (e.g. turn timestamp into date)

2 Answers2