Bug in parsing format string %T in function strptime in Miller 6?

[derekmahar]

Is the following a bug in the parsing of the %T format string in function strptime in Miller 6.0.0?

$ printf "timestamp\n2015-08-28T13:33:21Z\n" | mlr --csv put '$seconds = strptime($timestamp, "%Y-%m-%dT%TZ");'
timestamp,seconds
2015-08-28T13:33:21Z,1440720000

Compare the former result to the results of the following commands:

$ printf "timestamp\n2015-08-28T13:33:21Z\n" | mlr --csv put '$seconds = strptime($timestamp, "%Y-%m-%dT%H:%M:%SZ");'
timestamp,seconds
2015-08-28T13:33:21Z,1440768801
$ date --date=2015-08-28T13:33:21Z +%s
1440768801

Do functions strftime and strptime in Miller 6.0.0 still use the format strings in function strftime in the C library?

[derekmahar]

Function strptime also cannot parse time zone name or abbreviations:

$ printf "timestamp\n2015-08-28T13:33:21Z\n" | mlr --csv put '$seconds = strptime($timestamp, "%Y-%m-%dT%H:%M:%S%Z");'
timestamp,seconds
2015-08-28T13:33:21Z,(error)

[johnkerl]

@derekmahar thanks!

Neither Miller 5 (using the C library's strptime) nor Miller 6 (using a fork of pbnjay/strptime which wraps the Go library's time.Parse) accepts Z in the input with format specifier %Z.

However they both do match Z in the input with Z in the format string:

$ printf "timestamp\n2015-08-28T13:33:21Z\n" | mlr5 --csv put '$seconds = strptime($timestamp, "%Y-%m-%dT%H:%M:%SZ");'
timestamp,seconds
2015-08-28T13:33:21Z,1440768801.000000

$ printf "timestamp\n2015-08-28T13:33:21Z\n" | mlr --csv put '$seconds = strptime($timestamp, "%Y-%m-%dT%H:%M:%SZ");'
timestamp,seconds
2015-08-28T13:33:21Z,1440768801

[Poshi]

It looks like a similar issue I found with number formatting:

$ mlr -n put 'begin {print(fmtnum(3, "%i"))}'
%!i(string=3)
$ mlr -n put 'begin {print(fmtnum(3, "%d"))}'
3

According to the manual page of the C printf method, %i and %d should be equivalent.

[johnkerl]

@derekmahar @Poshi indeed.

In the C implementation (Miller 5 and below) fmtnum and strptime used the C library -- the same thing you see with man 3 printf etc. In the Go implementation (Miller 6 and above) they use Go libraries -- fmtnum using whatever Go's fmt.Sprintf supports, and strptime using https://github.com/johnkerl/miller/tree/main/internal/pkg/pbnjay-strptime which is a fork of the original (which I've modified a little and can modify more).

In both C and Go I happily/glibly thought Hey cool there's this library I can use and gosh they thought of ever so many things! -- and I never made a catalog of all possible syntaxes. It all seemed like more choices than I would ever want or need. Which isn't entirely wrong, but, as you're both seeing, there's a regression when the catalog-of-all-supported-options changes from C to Go, and when one of the things in the old catalog (%i or %T) doesn't exist in the new one. :(

(Aside: %!i(string=3) is simply what the Go fmt.Sprintf does with format-strings it can't handle.)

What we can do here:

• For %i, since man 3 printf says %i is the same as %d, have Miller do a search-and-replace before calling fmt.Sprintf.
• For %T, I've forked pbnjay/strptime already & could probably tweak it to get %T working
• Bigger picture: we should have a catalog of all supported formats for both functions -- even if it's just a hyperlink to Go docs somewhere.

[derekmahar]

Does pbnjay strptime document all of the format strings that strptime in Miller 6 can parse? As you mentioned, %T isn't there and nor is %F as an abbreviation for %Y-%m-%d. Unlike the C implementation of strptime, the pbnjay and Miller 6 (Go) implementations support %f for microseconds and should support time zone Z using %Z, but the Miller 6 implementation does not, though it does support numeric time zones, without the colon separator between hour and minute, and alpha time zone names. I think the inability of strptime in Miller 6 (and I presume in pbnjay) to parse time zone Z is a serious omission since this abbreviation for UTC is very common in ISO 8601 dates and times.

Examples of Miller 6.0.0 strptime using %Z and %z:

$ printf "timestamp\n2015-08-28T13:33:21Z\n" | mlr --csv put '$seconds = strptime($timestamp, "%Y-%m-%dT%H:%M:%S%Z");'
timestamp,seconds
2015-08-28T13:33:21Z,(error)
$ printf "timestamp\n2015-08-28T13:33:21 EST\n" | mlr --csv put '$seconds = strptime($timestamp, "%Y-%m-%dT%H:%M:%S %Z");'
timestamp,seconds
2015-08-28T13:33:21 EST,1440768801
$ printf "timestamp\n2015-08-28T13:33:21+0000\n" | mlr --csv put '$seconds = strptime($timestamp, "%Y-%m-%dT%H:%M:%S%z");'
timestamp,seconds
2015-08-28T13:33:21+0000,1440768801
$ printf "timestamp\n2015-08-28T13:33:21-0500\n" | mlr --csv put '$seconds = strptime($timestamp, "%Y-%m-%dT%H:%M:%S%z");'
timestamp,seconds
2015-08-28T13:33:21-0500,1440786801
$ printf "timestamp\n2015-08-28T13:33:21-05:00\n" | mlr --csv put '$seconds = strptime($timestamp, "%Y-%m-%dT%H:%M:%S%z");'
timestamp,seconds
2015-08-28T13:33:21-05:00,(error)

Examples of Miller 6.0.0 strptime using %f:

$ printf "timestamp\n2015-08-28T13:33:21.123456\n" | mlr --csv put '$seconds = strptime($timestamp, "%Y-%m-%dT%H:%M:%S.%f");'
timestamp,seconds
2015-08-28T13:33:21.123456,1440768801.123456
$ printf "timestamp\n2015-08-28T13:33:21.123\n" | mlr --csv put '$seconds = strptime($timestamp, "%Y-%m-%dT%H:%M:%S.%f");'
timestamp,seconds
2015-08-28T13:33:21.123,1440768801.1230001

Note that for 123 ms, Miller strptime mysteriously displays seven decimal places including a trailing 1 digit instead of exactly six digits including three trailing zeros.

[Poshi]

Maybe this is just a documentation issue: the help for fmtnum says: Convert int/float/bool to string using printf-style format string.
Traditionally, printf-style refers to the printf in the C library. Even the shell command of the same name.
If you are relying in Go formatting abilities, explicitly state that and add a link to the Go documentation with the formats. Current documentation is ambiguous and leads naturally to the wrong (C) library.

[johnkerl]

@Poshi that would work! :)

[johnkerl]

@Poshi #895 re fmtnum

[johnkerl]

@derekmahar going through a checklist now -- this isn't answers, just a few for now:

I think the inability of strptime in Miller 6 (and I presume in pbnjay) to parse time zone Z is a serious omission since this abbreviation for UTC is very common in ISO 8601 dates and times.

I've always just used Z:

$ printf "timestamp\n2015-08-28T13:33:21Z\n" | mlr --csv put '$seconds = strptime($timestamp, "%Y-%m-%dT%H:%M:%S%Z");'
timestamp,seconds
2015-08-28T13:33:21Z,(error)

$ printf "timestamp\n2015-08-28T13:33:21Z\n" | mlr --csv put '$seconds = strptime($timestamp, "%Y-%m-%dT%H:%M:%SZ");'
timestamp,seconds
2015-08-28T13:33:21Z,1440768801

I may be able to map %Z to Z.

But there's something else going on here.

The timezone-related testing I did which was rather extensive during the Miller 6 implementation (https://github.com/johnkerl/miller/tree/main/test/cases/dsl-local-date-time-functions) was involving the TZ environment variable; less so using %Z or %z.

👀

[johnkerl]

@derekmahar Something very disturbing about Go's time.Parse (which pbnjay/strptime simply falls victim to):

https://pkg.go.dev/time#Parse

...

When parsing a time with a zone offset like -0700, if the offset corresponds to a time zone used by the current location (Local), then Parse uses that location and zone in the returned time. Otherwise it records the time as being in a fabricated location with time fixed at the given zone offset.

When parsing a time with a zone abbreviation like MST, if the zone abbreviation has a defined offset in the current location, then that offset is used. The zone abbreviation "UTC" is recognized as UTC regardless of location. If the zone abbreviation is unknown, Parse records the time as being in a fabricated location with the given zone abbreviation and a zero offset. This choice means that such a time can be parsed and reformatted with the same layout losslessly, but the exact instant used in the representation will differ by the actual zone offset. To avoid such problems, prefer time layouts that use a numeric zone offset, or use ParseInLocation.
...

To wit:

package main

import (
	"fmt"
	"time"
)

func tryParse(format, value string) {
	t, e := time.Parse(format, value)
	if e == nil {
		fmt.Printf("%s %s -> %v %d\n", format, value, t, t.Unix())
	} else {
		fmt.Println(e)
	}
}

func main() {
	// Arg 1 is the time.Parse "template" -- what pbnjay/strptime does is map %Y et al. into these.
	// In this test we're going straight to time.Parse to discover that that's where the
	// undesired behavior is.
	tryParse("2006 01 02 15 04 05 MST", "1970 01 01 00 00 00 EST")
	tryParse("2006 01 02 15 04 05 MST", "1970 01 01 00 00 00 CST")
	tryParse("2006 01 02 15 04 05 MST", "1970 01 01 00 00 00 MST")
	tryParse("2006 01 02 15 04 05 MST", "1970 01 01 00 00 00 PST")
	tryParse("2006 01 02 15 04 05 MST", "1970 01 01 00 00 00 GMT")
	tryParse("2006 01 02 15 04 05 MST", "1970 01 01 00 00 00 XYZ")
	tryParse("2006 01 02 15 04 05 MST", "1970 01 01 00 00 00 NOO")
	tryParse("2006 01 02 15 04 05 MST", "1970 01 01 00 00 00 YUP")
}

prints

2006 01 02 15 04 05 MST 1970 01 01 00 00 00 EST -> 1970-01-01 00:00:00 -0500 EST 18000
2006 01 02 15 04 05 MST 1970 01 01 00 00 00 CST -> 1970-01-01 00:00:00 +0000 CST 0
2006 01 02 15 04 05 MST 1970 01 01 00 00 00 MST -> 1970-01-01 00:00:00 +0000 MST 0
2006 01 02 15 04 05 MST 1970 01 01 00 00 00 PST -> 1970-01-01 00:00:00 +0000 PST 0
2006 01 02 15 04 05 MST 1970 01 01 00 00 00 GMT -> 1970-01-01 00:00:00 +0000 GMT 0
2006 01 02 15 04 05 MST 1970 01 01 00 00 00 XYZ -> 1970-01-01 00:00:00 +0000 XYZ 0
2006 01 02 15 04 05 MST 1970 01 01 00 00 00 NOO -> 1970-01-01 00:00:00 +0000 NOO 0
2006 01 02 15 04 05 MST 1970 01 01 00 00 00 YUP -> 1970-01-01 00:00:00 +0000 YUP 0

Note my location is EST -- Go's time.Parse is ignoring everything else and using what it calls a "fabricated location". :(

I think I can handle this inside Miller's fork of pbnjay/strptime. Also I'll let pbnjay know.

[derekmahar]

I think the inability of strptime in Miller 6 (and I presume in pbnjay) to parse time zone Z is a serious omission since this abbreviation for UTC is very common in ISO 8601 dates and times.

I've always just used Z:

According to the man page for strptime(), your approach is more standard than the one I had assumed. Unlike the Glibc implementation of strptime() which aims to preserve symmetry with strftime(), POSIX implementations of strptime() don't support format specifiers %Z or %z:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>

#define MAX_RESULT_SIZE 100

void test_strptime1(void) {
  const char input_format[] = "%Y-%m-%dT%H:%M:%S%Z";
  const char input_timestamp[] = "2015-08-28T13:33:21Z";
  struct tm output_time;
  char result[MAX_RESULT_SIZE];

  memset(&output_time, 0, sizeof(struct tm));

  if (strptime(input_timestamp, input_format, &output_time) == NULL) {
    strcpy(result, "strptime failed");
  }
  else {
    strftime(result, sizeof(result), "%s", &output_time);
  }

  printf("input_timestamp=%s input_format=%s result=%s\n", input_timestamp, input_format, result);
}

void test_strptime2(void) {
  const char input_format[] = "%Y-%m-%dT%H:%M:%SZ";
  const char input_timestamp[] = "2015-08-28T13:33:21Z";
  struct tm output_time;
  char result[MAX_RESULT_SIZE];

  memset(&output_time, 0, sizeof(struct tm));

  if (strptime(input_timestamp, input_format, &output_time) == NULL) {
    strcpy(result, "strptime failed");
  }
  else {
    strftime(result, sizeof(result), "%s", &output_time);
  }

  printf("input_timestamp=%s input_format=%s result=%s\n", input_timestamp, input_format, result);
}

int main(void) {
  test_strptime1();
  test_strptime2();
  return 0;
}

$ # Install musl-tools to avoid using GCC and Glibc.
$ sudo apt install musl-tools
$ # Use musl-gcc wrapper around GCC. See https://www.musl-libc.org/how.html.
$ musl-gcc -o test_strptime test_strptime.c
$ ./test_strptime
input_timestamp=2015-08-28T13:33:21Z input_format=%Y-%m-%dT%H:%M:%S%Z result=strptime failed
input_timestamp=2015-08-28T13:33:21Z input_format=%Y-%m-%dT%H:%M:%SZ result=1440768801
$ date --date=2015-08-28T13:33:21Z +%s
1440768801

Note that the first test of standard strptime() failed because it used format specifier %Z, but the second one succeeded because it used a plain Z.

(Writing this C program was a challenge because I haven't written a line of C in many years. It also reminded me why I long ago abandoned C!)

[derekmahar]

@derekmahar Something very disturbing about Go's time.Parse (which pbnjay/strptime simply falls victim to):

For what it's worth, the Linux date command is smart enough to parse these time zone abbreviations:

$ for timezone in EST CST MST PST GMT XYZ NOO YUP
> do
>   date --date="1970-01-01T00:00:00 $timezone" +'%Y-%d-%mT%H:%M:%S %z %Z %s'
> done
1970-01-01T00:00:00 -0500 EST 18000
1970-01-01T01:00:00 -0500 EST 21600
1970-01-01T02:00:00 -0500 EST 25200
1970-01-01T03:00:00 -0500 EST 28800
1969-31-12T19:00:00 -0500 EST 0
date: invalid date ‘1970-01-01T00:00:00 XYZ’
date: invalid date ‘1970-01-01T00:00:00 NOO’
date: invalid date ‘1970-01-01T00:00:00 YUP’

Given that standard strptime() doesn't recognise format specifiers %Z or %z, Linux date must use the Glibc implementation of strptime() or implement the parsing for these format specifiers itself. Unfortunately, after spending so much energy today writing tests for strptime(), I have little left to find and read the source code of date to see how it performs its magic. Does anyone else feel ambitious enough to investigate?

[johnkerl]

(Writing this C program was a challenge because I haven't written a line of C in many years. It also reminded me why I long ago abandoned C!)

I get that! Honestly I don't miss C ... strange to say ...

According to the man page for strptime(), your approach is more standard than the one I had assumed

I learned this along the way here as well! Glad we're figuring it out together. :)

[johnkerl]

@derekmahar

(Writing this C program was a challenge because I haven't written a line of C in many years. It also reminded me why I long ago abandoned C!)

I hear that! I don't miss C ... strange to hear myself say that after all these years but there it is ...

According to the man page for strptime(), your approach is more standard than the one I had assumed.

Sounds like we both learned this -- thanks for going through this with me! :D

[johnkerl]

Re %T and %F these are easy search-and-replace -- I can have Miller strptime automatically turn %F into %Y-%m-%d before handing off to strptime and likewise %T to %H:%M:%S

[johnkerl]

Note that for 123 ms, Miller strptime mysteriously displays seven decimal places including a trailing 1 digit instead of exactly six digits including three trailing zeros.

This one has a straightforward explanation: the result of strptime is a floating-point number, being printed with default formatting.

If we use 6-decimal output formatting we get

$ echo 'timestamp=2015-08-28T13:33:21.123,format=%Y-%m-%dT%H:%M:%S.%f' | $mlr --d2p --ofmt '%.6f' put '$seconds = strptime($timestamp, $format)'
timestamp               format               seconds
2015-08-28T13:33:21.123 %Y-%m-%dT%H:%M:%S.%f 1440768801.123000

$ echo 'timestamp=2015-08-28T13:33:21.123,format=%Y-%m-%dT%H:%M:%S.%f' | $mlr --d2p --ofmt '%.3f' put '$seconds = strptime($timestamp, $format)'
timestamp               format               seconds
2015-08-28T13:33:21.123 %Y-%m-%dT%H:%M:%S.%f 1440768801.123

[johnkerl]

@derekmahar in summary so far:

• %F and %T are easy to do. Already working on my laptop.
• .1230001 is floating-point formatting.
• pbnjay/strptime maps %z to Go time.Parse -0700 rather than -07:00 -- I can modify Miller's fork of pbnjay/strptime to try both. So, this is fixable.
• %Z isn't matching the final Z because pbjnay/strptime is looking for something three characters wide to match the template MST. I can special-case this ... on further thought, as noted above, Z in input doesn't match %Z in the format string for the C libary's strptime so I don't see a good reason to support it here. (You can replace %Z with Z in the format string you pass to strptime.)
• Bad news: Go's time.Parse will never match CST unless you're in it. Reason is that these three-letter abbreviations are non-unique globally: CST can mean Central Standard Time, Cuba Standard Time, China Standard Time.
• There may be a way to get %Z to accept globally unique timezone names like Asia/Istanbul. (Miller already does fine with these when they're the $TZ; only trick is to pull them out of strptime format strings. Here again the catch is that Go's time.Parse is looking for three-letter codes.)
• Things like UTC-8, Go's time.LoadLocation doesn't handle but maybe these can be regexed to things like -08:00 which it does.

[johnkerl]

@derekmahar re your original ask:

Is the following a bug in the parsing of the %T format string in function strptime in Miller 6.0.0?

$ printf "timestamp\n2015-08-28T13:33:21Z\n" | mlr --csv put '$seconds = strptime($timestamp, "%Y-%m-%dT%TZ");'
timestamp,seconds
2015-08-28T13:33:21Z,1440720000

Compare the former result to the results of the following commands:

$ printf "timestamp\n2015-08-28T13:33:21Z\n" | mlr --csv put '$seconds = strptime($timestamp, "%Y-%m-%dT%H:%M:%SZ");'
timestamp,seconds
2015-08-28T13:33:21Z,1440768801
$ date --date=2015-08-28T13:33:21Z +%s
1440768801

This was fixed by #944.

There were two issues: one, %F and %T not being recognized format codes (fixed in #944); two, I was using (erroneously) the pbnjay/strptime flag that said if there is an unrecognized format code in the format string, ignore the remaining format codes (also fixed in #944).

[johnkerl]

@derekmahar #944 and #951 contain what I think resolves all issues raised in this discussion.

See also:

• https://miller.readthedocs.io/en/latest/reference-dsl-time/#custom-formats-strptime-and-strftime
• https://miller.readthedocs.io/en/latest/reference-dsl-time/#a-note-on-timezones
• https://miller.readthedocs.io/en/latest/reference-dsl-time/#fractional-seconds

Please let me know if this covers all concerns, or if I've missed anything.

[johnkerl]

@all-contributors please add @derekmahar for ideas

[johnkerl]

@all-contributors please add @derekmahar for documentation

[johnkerl]

@derekmahar can you mark this as answered (unless I missed something) -- ?

[derekmahar]

How do I mark it as answered?

[johnkerl]

Ah, never mind @derekmahar , this is of type 'General' not 'Q&A' ...

[derekmahar]

Should I change the discussion to type to "Q&A"?

[johnkerl]

No, we're good -- sorry for the false alarm :)

[oPromessa]

Hello team. Great work with Miller! Thanks.

Is the combination of %f and %z in strptime possible?

With Miller 6.13:

• with both %f%z is giving me an error!.
• solely with %f works fine
• solely with %z works fine

Applying --ofmt '%.6f' does not make a difference.

 mlr -n put 'end {
print strptime("2012-06-15 11:38:33.160001+0100", "%Y-%m-%d %H:%M:%S.%f%z") ;
print strptime("2012-06-15 11:38:33+0100",        "%Y-%m-%d %H:%M:%S%z") ;
print strptime("2012-06-15 11:38:33.160001",      "%Y-%m-%d %H:%M:%S.%f")
} '
(error)
1339756713
1339760313.160001

[johnkerl]

Looks like a bug to me! #1702