Add scripts to allow addons from personal repos to be synchronized with Crowdin#1
Add scripts to allow addons from personal repos to be synchronized with Crowdin#1nvdaes wants to merge 146 commits into
Conversation
… addedwith dev role to Crowdin if they use a project not owned by them to upload source files)
…flow to upload/update files in Crowdin
…k pass creating a PR at nvdaes/translateNvdaaddonsWithCrowdin repo
PurposeAdd-on authors may wish to help translators use Crowdin, the same framework where they translate NVDA. to translate messages and documentation for maintained add-ons: Other details
Development approachA workflow (GitHub Actions), and several scripts (Python and Powershell), as well as a json file with language mappings have been added. |
|
I've tested that all check pass using this pyproject.toml file on this PR: nvdaes/translateNvdaAddonsWithCrowdin#11 I use precommit, CodeQL and a workflow to check that all translatable messages have comments for translators. I'll try to use the cache action to cache some add-on metadata like its id, and also hashfiles from l10nSources (taking the value of buildVars.py), and the hasf¡hfile of the readme.md, to determine if pot and xliff files should be updated. |
|
Export translations to Crowdin running the workflow with update=False works properly: https://github.com/nvdaes/translateNvdaAddonsWithCrowdin/actions/runs/19802210157 |
|
This time, updatexLiff is failing. Seems that adding blank lines to readme may cause problems: https://github.com/nvdaes/translateNvdaAddonsWithCrowdin/actions/runs/19802391926/job/56731562709 |
|
If someone can help with this issue when update xliff, I'll be grateful. |
|
It might be easier to avoid xliff and just translate the markdown files directly. This won't support diffs very well but worth experimenting with |
|
@seanbudd wrote:
OK. |
|
@CyrilleB79, you were interested in this framework. If you want, feel free to see how the translateNvdaAddonsWithCrowdin.md can be translated in the project. Using xliff files is causing problems, as mentioned, and we are experimenting uploading md files instead. |
|
Thanks @nvdaes! Glad we're aligned on this approach. I’m a bit busy with work at the moment, but I won’t hesitate to jump in and collaborate on the And yes, in the meantime I can keep an eye on this PR and continue contributing where I can. Thanks again — really appreciate the collaborative work. |
|
I strongly recommend removing the code that handles Markdown translation. I have reviewed the changes to the Simplified Chinese documentation of hkatic/clock@157726d. After uploading the local Markdown document to Crowdin and downloading it again, some translations were lost and the document format had changed. Additionally, having both XLIFF and Markdown format document files in Crowdin at the same time can cause confusion for translators. |
|
Please @abdel792 , when you can, look at @wmhn1872265132 comment about markdown. I'm not a translator and I don't know how to proceed about this. |
|
Hi @wmhn1872265132, Thank you for your feedback. I believe the issue is not related to the fact that both Markdown and XLIFF formats are available in Crowdin. When working with Crowdin, translations for documentation files—whether they are Markdown or XLIFF—must strictly preserve the structure and string order defined in the source language file. If even a single string becomes inconsistent with the original English source file, Crowdin may reject that translation segment during synchronization or export. For this reason, I strongly recommend that translators always verify the structure of the English source files before starting their work. For translators working directly in the Crowdin web interface, it is equally important to ensure that translated strings remain consistent with the source strings and document structure. Using Crowdin as our translation platform requires us to be aware of this constraint. In my opinion, the presence of both Markdown and XLIFF formats is not the cause of the issue. On the contrary, it provides greater flexibility for translators, allowing them to choose the format they are most comfortable working with. The Therefore, my recommendation would be to focus on validating the source file structure and translation consistency rather than removing support for one of the documentation formats. |
No, that is not the case for Markdown files. Even if the structure of your uploaded file matches the English source file exactly, issues still occur. I have run tests on this. XLIFF files do not have this problem, but Markdown files do.
The web interface may not have this problem because its processing mechanism differs from directly uploading Markdown files.
No, having two files with identical content but different formats actually creates confusion for translators: which file should I translate? If I translate the XLIFF but someone else translates the Markdown file, it results in duplicated work.
Only Markdown files need to be considered regarding structural issues. For XLIFF files, there are two scenarios:
By the way, there is another issue with Crowdin's Markdown support: it changes certain Markdown formatting markers to its own preferred style. For example, the list marker in the original text is |
|
Hi @wmhn1872265132, Thank you for your detailed feedback and for taking the time to investigate these issues. I respect your point of view, especially given your experience as a translator and the testing you have performed regarding Crowdin's Markdown handling. Your explanations about the Markdown-specific issues and the potential confusion caused by maintaining both Markdown and XLIFF translations are valuable. After reviewing the current workflow, it appears that moving to an XLIFF-only approach would require changes to the I'm not sure what @nvdaes thinks about this yet, but based on your findings, it may be worth discussing whether we should simplify the workflow and rely exclusively on XLIFF for documentation translations. As for me, I'm currently quite busy with work and have less time than usual to contribute to the project, which is why I haven't been participating as actively in this discussion recently. However, I wanted to acknowledge your feedback and the effort you put into analyzing the issue. Thanks again for sharing your observations and test results. |
|
Hi all I do not follow very well this PR / discussion. |
|
Hi @nvdaes, Following the recent comments from @wmhn1872265132 and @CyrilleB79, I took another look at the current implementation of Both comments seem to point in the same direction: simplifying the translation workflow by relying on a single documentation translation format rather than maintaining both Markdown and XLIFF translations in parallel. Based on the current implementation, one possible approach would be to use XLIFF as the only format exchanged with Crowdin, while continuing to store documentation in Markdown format within the repository. The main changes in
This would simplify the workflow to: instead of maintaining both Markdown and XLIFF translation files simultaneously. I have prepared an updated version of Of course, before making any changes to this PR, I would like to hear your opinion on this direction. Since this would change the current translation workflow, I think it is important to confirm that we all agree with the proposed approach before proceeding. As a side note, I have been quite busy with work recently, so I may not be able to react as quickly as usual, but I wanted to share this analysis and the proposed update for discussion. Thanks. |
|
@abdel792 and @nvdaes thanks for all the work. Would it be possible for translators to migrate legacy .md to xliff, as we did for NVDA? If not possible, translators will have to retranslate, but some add-ons have quite big documentation files... |
|
Hi @CyrilleB79, Thank you for your very relevant and helpful comments. As far as I remember, I read a message from @seanbudd on the NVDA translations mailing list indicating that, unfortunately, migrating to XLIFF effectively resets the translation state and requires translators to start over with their existing documentation translations. The Of course, I may be mistaken, and I would be happy to be corrected if someone has additional information or experience with migrating existing translated Markdown files to XLIFF. Thanks again for raising this important point. |
You can convert a translated Markdown file to an XLIFF file by running the following command in the root directory of the NVDA or nvdaL10n repository:
However, please note the following issues:
After that, upload the converted XLIFF file to Crowdin to complete the translation. |
As far as I know, when downloading files from Crowdin, even the fuzzy string in .po files seems to be lost, requiring a complete retranslation unless a web interface is used. |
|
Thanks @abdel792 and @CyrilleB79 . @abdel792 , in general, seems that your crowdinSync attached file is right. Aside note: My job requires working in a non regular scheduled time. This past night I was working until 1 AM, and this weekend I'll work today in the evening/night, and tomorrow in the morning. I'm a captioner working at requested events which require some previous work (for example, this weekend church pope will stay here in Spain, so my work will finish in the night and will start tomorrow in the morning). So I'll need to sleep, in addition to personal life. |
|
Hi @nvdaes, Thank you for your feedback, and thank you @wmhn1872265132 for the additional information regarding Markdown-to-XLIFF migration and PO file handling. @wmhn1872265132, your explanation about the @nvdaes, after reviewing your comments regarding the PO translation checks, I have prepared an updated version of In addition to the previously proposed XLIFF-only documentation workflow, this version updates the PO processing logic so that both PO and XLIFF files consistently use the translation percentage threshold ( This should make the validation process more consistent across all translation file types and better align with the current behavior of As before, the attached file has been renamed from Please have a look when you have time and let me know whether you think this updated version addresses the remaining concerns before I commit any changes to the Also, thank you for the explanation regarding your work schedule. No worries at all about response times; real life and work always come first. Thanks again to everyone for the constructive discussion. |
|
@abdel792 , seems that $scoreXliff = 0.0 is used, but it's not the same when evaluating the po score. I don't know if this is intentional. |
|
Hi @nvdaes, Thank you again for your review and feedback. Following your latest comments, I have updated both attached files:
In In and clarified that legacy Markdown files on Crowdin are ignored by the synchronization process. I also added documentation regarding the use of the Regarding the recent comment from @wmhn1872265132, it was interesting to learn that some of the functionality previously provided by Since Before any commits are made, I would appreciate it if you could review both attached files and verify that the proposed changes are consistent with the direction you have in mind for this PR. Please feel free to suggest any additional corrections or improvements before anything is merged. Thanks again for your time and for all your work on this. |
|
@abdel792 , for me, the crowdin sync script seems to be right. I think that in the readme we can add a note about the possibility of using a variable to set the min percentage translated, which can be added to GitHub Actions - variables. |
|
@abdel792 , sorry, abd also what happens if someone writes 100 in the min percentage translated variable, since I think that we are checking if the score is greater than the percentage, not greater or equal. |
|
Hi @nvdaes, While testing the workflow and reviewing the threshold logic, I found what appears to be an issue in In However, in the return progress / 100As a result, a translation progress of 93% becomes which always evaluate to I tested changing line 103 from: return progress / 100to: return progressand the workflow behaved as expected afterwards. Test run: I think this change is necessary so that the score returned by Could you please have a look and let me know what you think? Thanks. |
- Prefer XLIFF documentation translations. - Unify translation threshold validation. - Fix score comparison logic. - Return raw percentages from getScoreFromApi.
Document XLIFF-based documentation synchronization, translation threshold configuration, GitHub Actions variables, and related Crowdin workflow changes.
|
Hi @nvdaes and everyone, I have just pushed two new commits for review. The first commit updates the Crowdin synchronization logic to address the concerns raised by @wmhn1872265132 regarding the coexistence of Markdown and XLIFF documentation files in Crowdin. The main changes are:
Following @nvdaes' review, I also revisited the translation threshold logic and found an issue while testing the workflow. In MIN_PERCENTAGE_TRANSLATED: ${{ vars.MIN_PERCENTAGE_TRANSLATED || 50 }}This indicates that the workflow expects percentage values between 0 and 100. However, the return progress / 100As a result, a translation progress of 93% became To keep both values on the same scale, I changed the function to return the percentage directly: return progressI tested this modification in a workflow run and the threshold validation behaved correctly afterwards: https://github.com/abdel792/dayOfTheWeek/actions/runs/27070359050/job/79898355233 I also updated the comparison operator from This means that translations reaching exactly the configured threshold are now accepted. For example, if a repository maintainer sets: a file translated at exactly 100% will be synchronized, which seems more intuitive than requiring a value strictly greater than the configured threshold. The second commit updates the README documentation accordingly. The documentation now:
I also took into account the recent comments from @wmhn1872265132 regarding the migration path from Markdown to XLIFF and the current state of the tooling. Please review the changes when you have time and let me know if you see anything that should be adjusted before merging. One additional note regarding the README updates. I did not document the AddonTemplate update procedure for now. I understand the workflow that was suggested: git remote add addonTemplate https://github.com/nvaccess/AddonTemplate.git
git fetch addonTemplate
git branch -u addonTemplate/master
git pull --allow-unrelated-historiesHowever, in my own tests, especially with add-ons based on older versions of AddonTemplate, the My concern is not only the conflicts themselves, but also the possibility of accidentally overwriting add-on-specific metadata while resolving them. For example, files such as:
typically contain information that is specific to the local add-on repository and should not be replaced by the template version. These files frequently contain add-on-specific information such as metadata, version history, build configuration, documentation, and release notes that cannot always be merged automatically. When using Because of that, I was not fully comfortable documenting this procedure as a general recommendation in the README without additional guidance, warnings, or testing. I think this topic may deserve a dedicated section in the future explaining the expected conflicts and the best practices for safely merging AddonTemplate updates into existing add-ons, particularly those created from much older template versions. Thanks everyone for the feedback, testing, and suggestions. |
|
@abdel792 , I've reviewed diffs: And for me this is ready to request a review. Note: In the documentation, it's recommended to avoiding to run the workflow at the same time if various add-ons are managed. Anyway, as requested by Sean, now the workflow includes a procedure to start with a random delay: Anyway, we have also the possibility of collisions even with the random delay. I have seen your testing workflow. Let's request a review from NV Access. |
|
Hi @nvdaes, Thanks for your review and feedback. I have just pushed an additional documentation update. Regarding your comment about workflow scheduling, I noticed that the README was still recommending the use of different cron schedules without mentioning the random startup delay that was later added to the workflow. I have therefore updated the documentation to clarify that the workflow already includes a random delay before scheduled executions, while noting that collisions may still occur in some situations. I also revisited the Previously, the documentation explained how to create the The section has therefore been reorganized so that the process for creating repository variables is introduced before listing the supported variables, making the overall structure more consistent and easier to follow. Please let me know if you think any further adjustments are needed before requesting the NV Access review. Thanks again for all your work on this PR. |
|
@abdel792 , I've reviewed your last commit, and for me it's right. I requested a review from NV Access, and in fact I think that this PR is ready for review. |
This pull request introduces a complete, automated workflow for synchronizing translations with Crowdin, including new scripts, a scheduled GitHub Actions workflow, and supporting documentation. The main changes add Python and PowerShell scripts for translation status checking and synchronization, a workflow for scheduled and manual Crowdin sync, language mapping configuration, and documentation updates. These improvements enable seamless translation management for add-on projects.
Specifically: