Upstreaming from 5.3.7 to dev - #1483
Merged
Merged
Conversation
… แม่กน with vowel า (previously mistaken as แม่เกย)
In the check_marttra function, for สระ "า", letter ล is incorrectly assigned to the แม่เกย marttra instead of แม่กน. This cause the problem of word like พาล to be classified as แม่เกย and has a cascading problem to is_sumpus function.
print(kv.check_marttra("พาล")) --> เกย
print(kv.is_sumpus("บ้าน", "พาล")) --> False
Remove the incorrect elif word[-1] in ["ล"]: return "เกย"
and add "ล" back in elif word[-1] in ["ญ", "ณ", "น", "ร", "ฬ"]: return "กน" would solve this problem.
When both word meet the condition (sara == "อำ" and marttra == "กม") or (sara == "อำ" and marttra == "กม"), the elif part only change the first word sara and mattra and left the second word unchanged.
This make all of this incorrectly return False
print(kv.is_sumpus("ชัย", "วัย"))
print(kv.is_sumpus("วัย", "วัย"))
print(kv.is_sumpus("จำ", "ทำ"))
print(kv.is_sumpus("ทำ", "ทำ"))
change the elif to if would solve this edge case.
… ริ, รึ ฤ not only have "รึ" sound like ฤดู ฤทัย คฤหาสน์ but also can have "เรอ" sound from ฤกษ์ (เริก) and "ริ" sound like ฤทธิ์, อังกฤษ, ตฤณ This commit add additional check for ฤ sounds.
Add exception for 'อึ' sound in specific พฤทธิธรรม words and delete the old duplicated logic.
finetune the ฤ logic and patching the edge cases.
Edit handling of silent sounds (handle_karun_sound_silence function) to be able to handle more flexible Karun like พันธุ์, สิทธิ์, ฤทธิ์, จันทร์, พระลักษมณ์, กษัตริย์, ภาพยนตร์ instead of the old return word[:-2] which wouldn't correctly strip all the characters that should be silenced. Refactor check_sara function merging of vowel characters เอ เอ to แอ. Remove the previous implementation which mutating the list while iterating over the list.
Change 'แ' to 'แอ' in merging logic
Fix the discrepancy of อรรม - อัม - อำ. Normalize 'อะ' and 'เกย' to 'ไอ' and 'กา', and 'อำ' and 'อะ' to 'อำ' and 'กา'.
Remove Trailing whitespace in line 693 so the auto code reviewer wouldn't flag as an issue.
…ด Standalone words in check_mattra - In check_sara, use the new and improved word = self.handle_karun_sound_silence(word) instead of the old การันต์ Karun silenct word implementation word = word[:-2] - Refactor the check_sara code to accommodate สระประผม Transformed vowels (อัว, เอะ, แอะ, เออ, โอะ, เอีย, เอือ) especially with ไม้ไต่คู้ (-็) - Handle คำโดด Standalone words in check_mattra so it is now correctly classify as แม่ ก กา.
The previous implementation didn't accounted for tone marks at the end of the word (e.g., ใกล้), Silenced word ตัวการันต์, silent ย in ไ-ย, ใ-ย, สระประสม (เ-ีย), คำควบกล้ำ, and อักษรนำ. This new implementation take into account all of the above making the checker more robust. Limitation: This implementation haven't accounts for คำควบกล้ำ and อักษรนำ for letter "ร" and "ว" (โปร, แปร, ไกว) as this is not present in the original implementation. This would cause the these word to still be misclassified in check_mattra. The full rewrite of _has_true_final_yl that will account for ย, ล, ร, ว will coming soon.
Final revision. Extensively tested the integration against numerous edge cases and complex test cases as best as I could. This implementation improves the handling of: - `check_sara` - `check_marttra` - `is_sumpus` - `handle_karun_sound_silence` - Internal function `_is_true_final` Note: The `check_karu_lahu` and `check_aek_too` functions are left untouched. The `check_klon` function also remains unmodified, but it should perform better due to the underlying improvements in `is_sumpus`, which now correctly classifies previously failing edge cases.
Removed commented-out code and trialing whitespace. Due to the complexity of Thai language, to reduce the complexity of `_is_true_final` function from 27 down into 15, we will likely need to split the internal function into smaller sub-function. These could be done in the future as the functionality and correctness of the code is the main priority right now. I have include the comment that should sufficiently enough to guild any maintainer in the future that are going to continue working on this code.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Add original_word in `check_sara` Word like ฤทธิ์ now properly accounted for. handle comment properly according to PEP8 and fix the typo in the comment `ไกว`. The test case edit will need to be done later.
…eck_sara` and `check_mattra` Added handling for silent terminal vowels in Pali/Sanskrit words.
change ตริ to ตรี to match correctly with the สระ "อี" assertion in the test case.
…ng against "ว" and add unit tests for edge cases
…ied as แม่กม and fix check sara with สระอัว (ตัว ครัว บัว)
…n dealing with อักษรนำ, คำควบกล้ำ, สระอำ, and many sound of ฤ
…al require word before tonemark get stripped to evaluate some tone-dependent structures.
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.37.1 to 4.37.3. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v4.37.1...v4.37.3) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.37.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
…ions/checkout-7.0.1 Bump actions/checkout from 7.0.0 to 7.0.1
…idAnson/markdownlint-cli2-action-24.1.0 Bump DavidAnson/markdownlint-cli2-action from 24.0.0 to 24.1.0
…hub/codeql-action-4.37.3 Bump github/codeql-action from 4.37.1 to 4.37.3
romanize(..., engine="royin") raised IndexError on words containing ฤ (U+0E24), e.g. ฤดู, ฤก, ฤทธิ์. _romanize() built its consonant list with _RE_CONSONANT, which matches only thai_consonants and so excludes ฤ, while _replace_consonants() treats every key of _CONSONANTS - including ฤ - as a consonant and advances its index for each one. The two went out of sync and the index ran past the end of the list. The consonant list is now built with the same membership test the loop uses, so ฤ is romanized through its existing _CONSONANTS entry. Adds RU_TESTS and test_romanize_royin_ru to tests/core/test_transliterate.py. Closes #1444
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.37.3 to 4.37.4. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v4.37.3...v4.37.4) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.37.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [actions/stale](https://github.com/actions/stale) from 10.4.0 to 11.0.0. - [Release notes](https://github.com/actions/stale/releases) - [Changelog](https://github.com/actions/stale/blob/main/CHANGELOG.md) - [Commits](actions/stale@v10.4.0...v11.0.0) --- updated-dependencies: - dependency-name: actions/stale dependency-version: 11.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
…hub/codeql-action-4.37.4 Bump github/codeql-action from 4.37.3 to 4.37.4
…ions/stale-11.0.0 Bump actions/stale from 10.4.0 to 11.0.0
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.37.4 to 4.37.6. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v4.37.4...v4.37.6) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.37.6 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [DavidAnson/markdownlint-cli2-action](https://github.com/davidanson/markdownlint-cli2-action) from 24.1.0 to 24.2.0. - [Release notes](https://github.com/davidanson/markdownlint-cli2-action/releases) - [Commits](DavidAnson/markdownlint-cli2-action@v24.1.0...v24.2.0) --- updated-dependencies: - dependency-name: DavidAnson/markdownlint-cli2-action dependency-version: 24.2.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
…idAnson/markdownlint-cli2-action-24.2.0 Bump DavidAnson/markdownlint-cli2-action from 24.1.0 to 24.2.0
…hub/codeql-action-4.37.6 Bump github/codeql-action from 4.37.4 to 4.37.6
Add num_to_thaiword_float and fix เอ็ด rule in num_to_thaiword
It was fixed the release problem.
Added new version 5.3.7 to CHANGELOG with fixes and improvements.
Migrated `pythainlp.tokenize.deepcut` to ONNX, removing TensorFlow dependency. Updated tests and maintained backward compatibility for `custom_dict` parameter.
Upstreaming from 5.3 (v5.3.7)
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upstreaming from 5.3.7 to dev