tex, string, and separator options for units

[Alex-Jordan]

I'm revising the description for this in the post following this post. Leaving the original here for reference.

This implements one of the suggestions from @dpvc in #1437, and replaces that pull request.

• Units defined in Units.pm can have string, tex, string_separator, and tex_separator properties.
• All aliases for a unit will carry over these properties. To demonstrate what to do when this is unwanted, I moved micron to be a named unit instead of an alias.
• The string property is used when the unit or its aliases are printed as a string.
• The string property needs to use a string that can actually be printed in hardcopy. There is a comment about this in the file above where the big units hash is defined.
• Since a unit may be interpolated as part of an argument to Compute(), make sure that any all string properties are also unit names.
• The tex property is used when the unit or its aliases are printed as tex.
• The tex_separator property are only used when the unit is not part of a larger fractional unit, and the unit is the first unit following a number. Otherwise a space is used, as has been the case. For example, 180° and 180° s, but 180 s ° (except as tex).
• The string_separator property is similar, except it is also used even when there is a larger fractional unit. For example, 180°/s.
• Only deg is using tex_separator and string_separator, with each set to the empty string. I considered doing this with degF and degC, but Google tells me there should be a space for those units for scientific publications.
• contextUnits.pl was adding aliases for L (liter, liters, litre, and litres). It was cleaner to move those over to Units.pm. If there was a reason they were not already in Units.pm, I could move them back.

One thing that could still be improved, is that for string production, the spaces that are used in between numbers and units (when string_separator is not set), or between units and units should be nonbreaking spaces. I tried things that I though should work, but did not work for one reason or another. So that can be addressed in some future PR.

Try the problem from #1437 for some basic testing. Although you may want to also try with some more complicated units that have unit products and quotients.

[drgrice1]

This satisfies my initial request to make the properties centralized and defined in Units.pm, but will not work for what @dpvc asked. That is so that custom units can also take advantage of the new features. For this the addUnit method will need some more modification. Also the copies of the %context::Units::Context::DISPLAY hash made on line 958 for the context::Units::Unit package and line 1442 for the context::Units::NumberWithUnit package are going to be an issue for that as well. If the addUnit method modifies the %context::Units::Context::DISPLAY hash, then the copies will not be modified appropriately. Making those copies is also not particularly efficient. Those should be references instead.

[drgrice1]

Note that something is also needed at line 1856 in the checkMultDiv method of the context::Units::BOP package in order for the string_separator and tex_separator to work for student answers. Perhaps @dpvc can offer suggestions here.

[Alex-Jordan]

I'll put a pin in this (mark it as draft) until I come back from travel on July 7 (or later). I'll rework it. I can say that I did try at first to create a new %UNITS hash in Units.pm, which later in that same file was boiled down to %known_units to preserve backwards compatibility with things like parserNumberWithUnits.pl. And then I tried modifying contextUnits.pl accordingly to use the more structured %UNITS from Units.pm. That way the data for the new keys is really where it should be the whole time. I just couldn't manage to track all the added complications from the additional structure that this imposed on contextUnits.pl. But I'll try again.

One thing to think about in the meantime. I am leaning to not including the separator keys with this. This was really only included because of the degree symbol when the units are an angle. Not for anything else at present. So, like 60° instead of 60 °. That much would be fine. But what about 60°/s versus 60 °/s? What about if there is a product involving degrees, like 60° s versus 60 ° s? The current attempt makes these come out as 60°/s and 60° s, but more and more I am not sure that is good. So unless the team convinces me otherwise, when I redo this, I will leave out the separator features. If it's preferred, I could leave deg without having tex and string keys, so that deg stays as deg rather than automatically converting to °. Meanwhile authors who directly use ° will continue to see things like 60 °, so it will be a status quo thing, not a regression.

[drgrice1]

I think that the removal of the space for the degree symbol is more important to me than any of the other changes. But do what you can.

I think that 60°/s is correct, not 60 °/s. I am not sure what 60° s is, or even how exactly that would come about.

[dpvc]

I can say that I did try at first to create a new %UNITS hash in Units.pm, ... I just couldn't manage to track all the added complications from the additional structure that this imposed on contextUnits.pl.

Yeah, it's complicated. I've made a version of these two files that seem to do the trick. I'm attaching them here. One of the differences is that the %known_units has has the aliases as top-level keys, while%UNITS hash doesn't, so the aliases have to be handled a bit more carefully. I've refactored some of the code that deals with that in the units context.

units.tgz
contextUnits.tgz

See what you think.

[Alex-Jordan]

I've been modifying @dpvc's files, getting close to pushing a new version for all of this.

I've run into something that is an issue even before the changes happening here. The old Units.pm has % as a unit that is not usable with contextUnits.pl. I'm seeing at least three issues.

• Try to make Compute('1 %'); and you get Unexpected character '%'; see position 3 of formula at line 49 of [PG]/macros/core/Parser.pl. Something does not like trying to even parse %.
• There is no category in contextUnits for dimensionless units like percent. I guess we might want a new category for that. Unless there is something better, I propose `dimensionless'.
• Once % is parsed, I'll need to look closely at its string and TeX output.

I believe I can handle the latter two issues. But @dpvc, do you see how to address the first one?

Or maybe this is not going to work unless % becomes a new fundamental unit. I'm wondering if contexUnits.pl machinery can even work as expected if the unit is dimensionless.

[Alex-Jordan]

This is largely changes made by @dpvc in the files posted earlier in this thread. The purpose is that entries in Units.pm are now structured more. For example, what used to be:

ohm => {
		factor  =>  1,
		kg      =>  1,
		m       =>  2,
		amp     => -2,
		s       => -3,
		aliases => ["\x{2126}"]
	}

is now

ohm => {
		factor => 1,
		units  => {
			kg =>  1,
			m  =>  2,
			A  => -2,
			s  => -3,
		},
		prefixes => [qw(k M)],
		string   => "\x{2126}",
		TeX      => "\x{2126}",
		aliases  => ["\x{2126}"]
	}

• By structuring in the allowed prefixes, we no longer need separate units like kohm in the list.
• There are explicit string and TeX properties where helpful.
• The actual fundamental unit dimensions are a level deeper, to cleanly separate them from other properties.

All of this has been done in a way to preserve backward compatibility with parserNumberWithUnits.pl and parserFormulaWithUnits.pl. Basically the new %UNITS hash is converted to a hash that mostly matches the old hash. Then, contextUnits.pl has been updated to handle the new structure of %UNITS.

There are a few other changes.

• I did not like that amp was the fundamental unit and A was its alias. It was inconsistent with how other SI fundamental units are declared. So I switched those.
• microN was a declared unit, and I removed it. I could bring it back if we are worried that will break some problem somewhere. It seems unlikely.
• kWh was a declared unit, and I removed it. Because kW and h are already units.
• Added d for days.
• Added l for liters, along with prefixed cousins dl.
• Added liter, liters, litre, and litres.
• Added mL (since L gets the m prefix)
• Added mamp (since A is a fundamental unit, and amp is an alias, and since A gets the m prefix, and since all aliases that are 3 characters or less get the same prefixes)
• Added msec, nsec, usec for similar reasons.
• Added pc as the primary unit for parsec. Moved parsec to be an alias. Added parsecs as an alias. This was motivated by kpc and Mpc already being units, but not pc.
• Everywhere that u is used as a prefix, there will be an automatic alias with \x{00B5} as a prefix too.

There is also the noSeparator option. This applies to the degree symbol. I did not apply it to degF or degC (or degK) because Google suggests it is more professional to use spacing with those. Meanwhile noSeparator should also apply to %, but there are some issues to work out with % as described in my previous post.

When noSeparator is in effect, there will be no space following the number (magnitude) but only in certain conditions.

• string method, when there is no denominator unit or when there is only one numerator unit. So there is still a space for something weird like 5 (° m)/kg.
• TeX method, when there is no denominator unit. So while the string method might produce 5°/kg, the TeX method still has a space: 5\,\frac{°}{kg}

[Alex-Jordan]

A list of things I think could still be improved:

• The situation described earlier with %.
• The space between a magnitude and unit should really be nonbreaking space. But getting that to be the right thing for HTML versus PTX versus TeX output seems tricky.
• I suspect we could now have a shift or affine property that could be leveraged to allow comparisons between K quantities and degC quantities, etc. At least, for contextUnits,pl.
• As things are, the TeX method always wraps the TeX property in \text{...}. That happens to work for all of the TeX properties that currently exist. But if a TeX property used a math mode LaTeX macro, then I think that would fail. For example, \text{\mu{A}} would fail. (We are instead using \text{µA}.) But something using \text{...} or \mathrm{...} is needed for the units to come out upright.
• I think that 'B' (for byte) and 'b' (for bit) are worth consideration to be added as fundamental units. When prefixes are applied, there should be some thought put into whether a kB is officially 1000 bytes or 1024 bytes.
• We don't have a mechanism to deal with conflicts nicely. For example, c is already here as the speed of light. But c could be better used as the abbreviation for cup. There is also rad for radian, but also one rad means something else in the context of radiation dosages.
• lbf should mean a pound of force. In other words, a pound. But it is coded as if it stood for "foot pound", which is a unit of torque or energy. I am in favor of correcting this here, and let the problems that relied on this incorrect coding be corrected downstream.

[dpvc]

These look like great new features that you have added. Thanks for working on that.

I make some suggested changes below, and discuss an issue with modifying the %UNITS hash, and how that is problematic for the units context, that I think needs to be addressed.

[dpvc]

Is there a preference for qw() rather than quoted strings? It seems inconsistent to use qw() here but not in the aliases lists.

[Alex-Jordan]

I have developed a preference for it when there is the potential for lots of quotation marks and commas. Here with the m unit, we have qw(f p n u m c k), which would be ('f', 'p', 'n', 'u', 'm', 'c', 'k') with quotes and commas. The quotes and commas make my eyes blur and wonder if every quote is paired properly. Once I switched that one to use qw, I switched them all for a different kind of consistency.

[dpvc]

For the longer words, I'm wondering if there should be aliases using the full prefix name, like kilometer and nanosecond. You could use a map from the single-letter prefixes to their full versions and use the correct one depending on the length of the alias. Just a thought.

[Alex-Jordan]

We could do that. I had an earlier version of this that did that. Actually it was doing a lot, too much I think. Each unit was tagged with a boolean for whether or not SI prefixes were appropriate. And if so, all of the SI prefixes were applied. The abbreviated prefix when the base unit was 3 characters or less. And the full prefix if longer. But that was adding all kinds of "new" things like "femtosecond". In the interest of keeping this PR from changing too much regarding what are or aren't valid units to use, I moved away from that. It leaves a few units that have always spelled out the SI prefixes: nanomol, micromol, and maybe more. Also there is the odd microN. I'm content to leave this out for now.

[dpvc]

I don't really like the idea of modifying the %UNITS hash, as it mixes the defining data with the processed data. More important, however, is that this will confuse the units context, which already handles aliases. My feeling is that you should be creating/mmodifying the %known_units has here, as that is the processed units hash.

Since the units context is already handling aliases, let it do that. But you would have to add the processing of the prefixes to the units context, as it is not currently doing that.

One reason to do that is that the units context allows addUnits() to accept new units using the format of the %UNITS hash (that probably means the documentation in the units context needs updating; there are only examples of the old-style units, so it would be good to do one for the new style; note that the old is converted to new automatically in the addUnit() method). So new units could be added that have not just aliases but also prefixes. So the units content needs to be able to handle these anyway, and it doesn't really save anything to have that processed into the %UNITS hash here. Indeed, it means that the units context will have to look through more units to find the ones that match a given fundamental unit pattern (and have to look through all the aliases for those that don't match), so there is an argument that this makes matters worse.

My suggestion does mean some duplication of code, however. Perhaps this file could provide a utility function that generates a list of alias hashes for a given unit, and then use that here and in the units context to get the aliases to be added to %known_units or into the units context. That would reduce the redundancy, and could simplify some code in the units context as well.

[dpvc]

I don't see where these aliases are added to the %UNITS or %known_units hashes (though they will be added in the units context, since it processes aliases itself; usually, that will be redundant given that you have added most aliases already, but not for these prefixed units).

Same for the ones at line 676.

[Alex-Jordan]

Well, this whole modification of %UNITS will probably be scrapped, per your previous comment. But what is happening here is that the original declaration of %UNITS had (for example) s => ... with aliases => ['sec'] and prefixes => ['n']. And this block of code adds an entirely new unit to %UNITS: ns => .... Which has aliases => ['nsec'].

At this point contextUnits.pl will do its thing with both s and ns as declared units, each with their own alias.

And also for the legacy code, aliases will be extracted for the %known_units hash.

[dpvc]

Try to make Compute('1 %'); and you get Unexpected character '%'; see position 3 of formula at line 49 of [PG]/macros/core/Parser.pl. Something does not like trying to even parse %.

Have you added the % unit to the context? The commands

loadMacros("contextUnits.pl");
Context("Units")->addUnits("%");
Compute("1 %");

works for me, both in the original contextUnits.pl, and the modified one.

It turns out, however, that you can't mix percentages with reals, so that

Compute('1 %) == .01

is false, and

`` perl
Compute('1 % + .01');

throws the error `Both operands of '+' must have units if one does; see position 5 of formula`,

while 

``` perl
Compute('1 %') + .01

throws Can't locate object method "classMatch" via package "0.01" (perhaps you forgot to load "0.01"?) at macros/contexts/contextUnits.pl line 1600.

That last error needs to be fixed, and I suspect the first two want to be fixed as well. There may be situations where you do want to force the use of '%' (like "What is .01 as a percentage?"), so there may need to be a flat to control that. It looks like the sameUnits flag might be reasonable for that.

But there are a number of other things that might need to be adjusted for percentages that I didn't think of earlier. For example 2% * 4% ends up as 8 %^2, and 10 / (2%) is 5/%. So there may need to be special rules for handling '%' (or "dimensionless" units in general). I'm not sure what to do with pure units involving %; currently, % % is %^2, for example.

I will think about how to address these issues, and let you know if I come up with anything.

[dpvc]

• The situation described earlier with %.

I'm working on that and think I have things pretty well worked out. I will post some updates as soon as they are ready.

• The space between a magnitude and unit should really be nonbreaking space. But getting that to be the right thing for HTML versus PTX versus TeX output seems tricky.

A \kern is non-breaking in TeX, so perhaps that could be used rather than \, and any of the spaces U+2000 to U+200B are non-breaking in HTML, I believe. Putting braces around the number, space, and unit would prevent inline line-breaks in TeX and MathJax. More aggressive would be to use \hbox{$...$} around the whole thing. Using U+00A0 in place of a space in the string output should prevent that from breaking.

• I suspect we could now have a shift or affine property that could be leveraged to allow comparisons between K quantities and degC quantities, etc. At least, for contextUnits.pl.

I was thinking the same thing.

• As things are, the TeX method always wraps the TeX property in \text{...}. That happens to work for all of the TeX properties that currently exist. But if a TeX property used a math mode LaTeX macro, then I think that would fail. For example, \text{\mu{A}} would fail. (We are instead using \text{µA}.) But something using\text{...}or \mathrm{...} is needed for the units to come out upright.

Well, one option would be to use TeX => '$\mu{A}$' so that you get \text{$\mu{A}$}, which would render properly. A bit awkward, but perhaps better than having to use \text{...} explicitly for everything else.

[Alex-Jordan]

First, a note that I pushed changes from your more minor comments. I agree with your more significant comments, just haven't gotten to addressing them.

Have you added the % unit to the context?

I can't reproduce what I thought was happening earlier, and my best guess is that I had a typo in "dimensionless" when I tried to load that new category of units.

About percents in general, if the exercise only involves percentages, then contextPercent.pl is great. The situations where treating % as a unit are good are when the answer is something like 2 %/s. Maybe that is all, in practice: relative growth rate questions.

I don't feel that adding a Real to a unit (even if it is %) needs to be supported. If you'd like to do that though, of course please do. On the other hand, if I see something like 2 + 1% in English, maybe the right interpretation for that is "2 + 1% (of 2)". So maybe the result would be 2.02, as opposed to 2.01. So I guess I lean toward having a controlled error message for this instead of it producing a result.

If we are in Units context, and we have Compute("2% * 2%"), I actually thing that 4 %^2 may be appropriate. The alternatives are (a) the Real number 0.0004, but now we are producing Reals form Units, which feels wrong. Or (b) the Unit 0.04 %. But now we have done something extra special with one of the 2% factors, converting it to a Real and then multiplying that by the other 2%. I don't know that the extra attention is worth it.

Question: radians are similar, in that they are arguably dimensionless. A radian is the ratio of a certain length (an arc length in meters, say) to a radius length (also in meters). But at one point long ago radian was blessed (not in the perl sense) as one of the fundamental units. Maybe % should be handled similarly.

Are there other dimensionless units? permille? ppm? Which units can you multiply by 1 m (for example) and get back something that should still be measured in meters? (note that would include radians.) Maybe all such units should be handled uniformly.