Support GlotLID language and script identification - #2281
Conversation
Signed-off-by: ccyyy1023 <2637034749@qq.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| label, score = model.predict([pp], k=1) | ||
| score = score[0][0].item() | ||
| lang_code = label[0][0][-2:].upper() | ||
| lang_code = label[0][0].removeprefix("__label__") |
There was a problem hiding this comment.
When a pipeline using lid.176 persists the language score, preserving the raw model label now stores en instead of the previously documented EN, causing existing downstream comparisons against uppercase language codes to stop matching.
Knowledge Base Used: Text Curation Stage Library
There was a problem hiding this comment.
Yes, this is fine and being documented.
Greptile SummaryThe PR adds GlotLID language-script support while retaining standard FastText compatibility.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains; the previously discussed casing change is explicitly documented as intended behavior. Important Files Changed
Reviews (4): Last reviewed commit: "Merge branch 'main' into fix/glotlid-lan..." | Re-trigger Greptile |
Signed-off-by: ccyyy1023 <2637034749@qq.com>
svcnvidia-nemo-ci
added
the
waiting-on-maintainers
| raise ValueError(msg) | ||
| self._model_path = model_path | ||
| self._lang_code = lang.upper() if lang else None | ||
| self._lang_code = lang.casefold() if lang else None |
There was a problem hiding this comment.
| self._lang_code = lang.casefold() if lang else None | |
| self._lang_code = lang if lang else None |
how about we change the behavior slightly. When we are doing the comparison we can use casefold but the lang_code that is returned in the dataframe does not need the casefold.
| lang_code = label[0][0][-2:].upper() | ||
| lang_code = label[0][0].removeprefix("__label__") | ||
| if "_" not in lang_code: | ||
| lang_code = lang_code.upper() |
There was a problem hiding this comment.
Similar comment as above, instead of using the underscore to decide about case sensitivity we can just not do it at all. And just use casefold for the comparison. Make sense?
There was a problem hiding this comment.
We already have mocked fasttext tests in https://github.com/NVIDIA-NeMo/Curator/blob/main/tests/stages/text/modules/test_filters.py . I think it makes sense to move them to this new file, can you do that?
svcnvidia-nemo-ci
added
waiting-on-customer
Signed-off-by: ccyyy1023 <2637034749@qq.com>
|
Hi @sarahyurick, I’ve addressed all of your review comments in fa849db:
The targeted tests pass (87 passed, 1 skipped), and I also ran the full non-GPU text test suite, with the environment-related cases rerun successfully. Could you please take another look when you have a chance? |
svcnvidia-nemo-ci
removed
the
waiting-on-customer
|
/ok to test 90c2ed8 |
|
🌿 Preview your docs: https://nvidia-preview-fix-glotlid-language-script.docs.buildwithfern.com/nemo/curator Here are the markdown pages you've updated: |
3 participants
Summary
__label__prefix, including GlotLID language-script labels such aseng_Latneng) or exact language-script pair (eng_Latn)Why
FastTextLangIdpreviously extracted only the final two characters of a model label. For a GlotLID prediction such as__label__eng_Latn, this producedTNand discarded both the language and script information.The updated parsing preserves the model label and lets users choose between language-level and language-script-level filtering while retaining the existing confidence cutoff behavior.
Validation
pytest -q tests/stages/text/filters/fasttext/test_fasttext_filters.py— 10 passedpytest -q tests/stages/text/modules/test_filters.py tests/stages/text/filters/fasttext/test_fasttext_filters.py— 85 passed, 1 skippedpytest -m 'not gpu' -q tests/stages/text— 780 passed, 5 skipped, 53 deselected, 1 xfailedruff checkandruff format --checkon the changed Python filesFixes #2276