Skip to content

fix(train): write a sourceable sm_training.env for quotes and non-identifier names - #6439

Open
AkshayShah03 wants to merge 1 commit into
aws:masterfrom
AkshayShah03:fix/training-env-file-quoting
Open

AkshayShah03 wants to merge 1 commit into
aws:masterfrom
AkshayShah03:fix/training-env-file-quoting

Conversation

@AkshayShah03

Copy link
Copy Markdown

Issue #, if available: N/A

Description of changes:

The training container's script runs set -e with an ERR trap and sources /opt/ml/input/sm_training.env. set_env writes every variable as export NAME='value', which breaks in two cases:

  • A single quote in any value breaks the file's quoting. A hyperparameter like it's also ends up in SM_HPS and SM_TRAINING_ENV:
    sm_training.env: line 26: unexpected EOF while looking for matching `''
    
  • A channel name or hyperparameter key containing - or ., both allowed by the API (e.g. channel train-data, key model.lr), produces an invalid identifier:
    export: `SM_CHANNEL_TRAIN-DATA=/opt/ml/input/data/train-data': not a valid identifier
    

Either one kills the job before the entry script runs.

This change maps non-identifier characters in names to _; SM_HP_* already did this for -. It also escapes single quotes in values with the standard '"'"' sequence. The output for ordinary values is unchanged; the existing byte-for-byte test_set_env still passes. The fix is applied to both the sagemaker-train and the sagemaker-core copies of environment.py.

Testing:

  • A new test in each package writes the file with a train-data channel, a model.lr key and an it's a test value, then sources it with bash -c 'set -e; trap ... ERR; source ...' and checks the values. It fails before this change.
  • container_drivers tests: 33 passed (train) and 79 passed (core).

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

…ntifier names

The training container's bash script sources /opt/ml/input/sm_training.env
under 'set -e' with an ERR trap. set_env wrote every variable as
export NAME='value', so:

- a single quote in any value (e.g. a hyperparameter 'it's', which also
  appears in SM_HPS and SM_TRAINING_ENV) broke the file's quoting, and
- a channel name or hyperparameter key with '-' or '.' (allowed by the API,
  e.g. channel 'train-data' or key 'model.lr') produced an invalid
  identifier such as SM_CHANNEL_TRAIN-DATA, making 'export' fail,

either of which killed the job before the entry script ran.

Map non-identifier characters in names to '_' (as SM_HP_* already does for
'-') and escape single quotes in values. Output for ordinary values is
unchanged. Applied to the sagemaker-train and sagemaker-core copies.

This branch is waiting to be deployed

1 waiting deployment
manual-approval — 132f9853 Waiting Oct 12, 2026 by AkshayShah03 via wait-for-approval #623
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant