Support more locales in R-CMD-check-occasional

[MichaelChirico]

As surfaced during review, lv_LV is not quite enough to catch all possible collation issues.

Since this CI is only run "rarely", combinatorial explosion of settings is not so big a concern.

Besides simply adding the locales, there are a few fixes to the suite to get it green. Mostly, it's because on R 3.5, we install from an ancient snapshot of CRAN:

• Workaroung upstream bug where {R.oo} doesn't install in certain locales
• Ancient (<0.3.0) {nanotime} lacked as.nanotime(); AFAICT, nanotime() is identical
• Ancient (<0.9-8, I think) {bit64} didn't give NA for out-of-bouds indexing
• {rcmdcheck} didn't always have env=; it turns out we can just Sys.setenv() for all cases anyway.

[github-actions]

• HEAD=occasional-more-locales much slower for transform improved in #5493
• HEAD=occasional-more-locales slower P<0.001 for DT[by] max regression fixed in #7480
  

Generated via commit ef77a1e

Download link for the artifact containing the test results: ↓ atime-results.zip

Task	Duration
R setup and installing dependencies	6 minutes and 26 seconds
Installing different package versions	12 minutes and 38 seconds
Running and plotting the test cases	5 minutes and 33 seconds

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.01%. Comparing base (e5916a1) to head (ab2482f).

Additional details and impacted files

@@             Coverage Diff              @@
##           tests-azeri    #7869   +/-   ##
============================================
  Coverage        99.01%   99.01%           
============================================
  Files               88       88           
  Lines            17292    17292           
============================================
  Hits             17122    17122           
  Misses             170      170           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[MichaelChirico]

At least partly this is blocked by an upstream error that {R.oo} can't install in az_AZ:

HenrikBengtsson/R.oo#31

[aitap]

Interesting. I've tried test.data.table() in all locales again (setting LC_ALL but also LANGUAGE=en; otherwise the failures are inconvenient to parse and many of those are probably due to R being translated while data.table is not) and the only remaining failures are tests 168.[1-3] in locales bo_CN, bo_IN, dz_BT, km_KH, shn_MM, tok, vi_VN: this is due to R bug 19117. Are the tests supposed to be still failing in CI?

[MichaelChirico]

Are the tests supposed to be still failing in CI?

Green CI is always good. A behavioral test against R itself would be good of the form

if (strptime(...)) {
  cat("Tests 168.* skipped as affected by R bug #19117\n")
} else {
  ...
}

[MichaelChirico]

Surprising that the same date works in all the problem locales!

passes = function() format(as.Date(strptime(strftime(as.Date('2000-11-01'), '%Y %d %b'), '%Y %d %b'))) == '2000-11-01'

for (ll in c("bo_CN", "bo_IN", "dz_BT", "km_KH", "shn_MM", "tok", "vi_VN"))
  print(withr::with_locale(list(LC_TIME=ll), passes()))
# all FALSE

# .utf8 suffix needed for https://bugs.r-project.org/show_bug.cgi?id=19117#c1
for (ll in c("en_US", "zh_CN.utf8", "lv_LV", "es_ES", "ru_RU.utf8", "pl_PL", "th_TH.utf8"))
  print(withr::with_locale(list(LC_TIME=ll), passes()))
# all TRUE

[MichaelChirico]

Looking at the actual 168 series, it looks like the strptime() behavior is pretty irrelevant?

NOV = format(strptime("2000-11-01", "%Y-%m-%d"), "%b")
x = c("09:29:16","10:42:40","23:47:12","01:06:01","11:35:34","11:51:09")
datetimes = paste0("2011 ", NOV, c(18,18,18,19,19,19), " ", x)
DT = IDateTime(strptime(datetimes,"%Y %b%d %H:%M:%S"))

AFAICT {data.table} only gets involved with the POSIXlt object received from strptime() -- why wouldn't we just use 2011-11-01 ... strings directly? The comment doesn't make it clearer to me:

# IDateTime conversion methods that ggplot2 uses (it calls as.data.frame method)
# Since %b is e.g. "nov." in LC_TIME=fr_FR.UTF-8 locale, we need to
# have the target/y value in these tests depend on the locale as well, #3450.

#4719 added that, it just took the step of making NOV more LC_TIME friendly, but I don't see any discussion of why not move to ISO inputs (I'm guessing it was just done note to risk messing with back-compatibility of tests). cc @tdhock