At BeLazy we gave up on XLIFF as the interim localization format, and we are adopting a new, open, JSON-based localization format. This article explains the reasons behind this decision.
The TL;DR
- XLIFF has quietly stopped being a universal interoperability format. It was built so desktop CAT tools could pass bilingual files back and forth, but as cloud and developer-oriented TMSes took over, the information that actually matters (comments, segment locks, match rates, statuses) migrated into APIs.
- TMSes still send single segments to LLMs. This works for MT, but LLMs work better with the full document context, which is stored in XLIFFs. This resulted in LLMs changing the focus from segments to full documents, as they work better on a full document basis.
- LLM workflows need more than XLIFF can generally carry, and its verbosity now costs real money. Automated translation today depends on context, quality-estimation match rates, selective locking and QA comments, none of which generally live in XLIFF alone.
- BeLazy is replacing XLIFF with an open, JSON-based format built for the AI age. It bundles the full picture of a bilingual document —the XLIFF content plus everything only reachable via API— into a single file you can send straight to an LLM.
Good start, bad progress
XLIFF is an XML flavour that was born from the need to store segment pairs in documents in a bilingual format, to allow interoperability between translation tools. Interoperability is a noble cause, and I am all for it, however, the 1.x standard did leave many points unaddressed.
Portability
XLIFF as a standard was mostly influenced by localization buyers. While the standard does describe how to represent formatting tags and segment status, formatting tags are not harmonized among tools, and segment statuses are quite rudimentary. Until around 2010, desktop translation tools like Trados, Deja Vu, Wordfast and memoQ were the dominant CAT tools rather than server- or cloud-based TMSes, and these required that translators be able to share the non-completed documents with reviewers and project managers in a file-based exchange format. Tool providers soon realized that XLIFF offers this capability, however, the specification did not have enough support for everything a translation environment tool needed to transfer from one person to another: locking, comment handling, match rate handling, QA information were all missing from the specification.
Trados Studio was the first tool that gave up on the binary formats for document portability and standardized on XLIFF, at a time when XML based formats were getting prominence with Microsoft’s decision to move to the OpenDocument format (docx, xlsx, pptx). XLIFF is extensible, so starting with Trados all tools that implemented the file-based interoperability had a need to add their own namespace to this XML format. Tools implemented XLIFF differently: in some tools, an XLIFF carried all the information, but the original files were required to export the finalized document back into the original format, whereas in other tools the XLIFF carried the so-called skeleton, the information required for the back-conversion. XLIFF 1.2 is widely supported by any translation tool, however, in reality, SDLXLIFF, MQXLIFF, MXLIFF (interesting that out of these three only MQXLIFF is still related to the current name of the tool, right?), TXML all represent the same or similar information in different ways. Because filters, segmentation, and tool capabilities are different among CAT tools and TMSes, XLIFF-based interoperability where you import a file, transfer it in XLIFF, edit it somewhere else and export the translation in the original file format was only ever possible between various instances of the same tool.
XLIFF 2.x was later released, but as translation tool providers already addressed all their issues using extensions to 1.x, and 2.x still did not solve between-tools interoperability, enthusiasm for this new version was curbed. I remember my days at memoQ where Microsoft and other large players were trying to convince us to support 2.x, but we kept asking why. It may have benefitted them – but they had no interest in buying our tool -, but this version change had zero incentives for translation tool providers. A new standard was designed without specifying what challenge the new standard was supposed to solve.
XLIFF, cloud and REST APIs
With the advent of cloud-based tools, multiple people could work on the same document online. In the 2010s, the focus has shifted from the desktop tools towards the online editing environments with a web interface. Web-based tools also got somewhat simpler for the translators and reviewers to use, and unlike many desktop tools, did not require translators to buy a license for themselves. While the older, document-based TMSes retained compatibility with their desktop versions (do you remember that Memsource, what is today Phrase TMS, used to have a desktop editor as well?), going into the cloud and the appearance of development-oriented TMSes like Crowdin, Lokalise or Transifex changed the focus of XLIFF. XLIFF’s main use shifted: it was no longer a tool to start working on a document in one tool and move to another, but it became the main tool for text manipulation and comparison. Machine translation became mainstream, and XLIFF offered a great simple story for them to integrate with the mainstream human workflows: it was enough to replace the target segment and possibly a segment status. XLIFF also allowed metrics like edit distance to be easily calculated, regardless of the tool used for importing.
Functionality that was used to represent information essential for translators, such as commenting, locking, match rate representation gradually shifted from the XLIFF into the APIs, especially in development-oriented TMSes. Tools like Crowdin and Lokalise for example treat XLIFF as yet another document format, rather than a special format to carry translation between tools. And this takes us to the core message of my article:
XLIFF has quietly stopped being a universal interoperability format. It was built so desktop CAT tools could pass bilingual files back and forth, but as cloud and developer-oriented TMSes took over, the information that actually matters (comments, segment locks, match rates, statuses) migrated into APIs.
So what is this JSON format and why not a harmonized XLIFF?
In short: XML was very popular in the 2000s, whereas JSON became popular in the 2010s. By default, all LLMs create JSON representations for any task, unless you specifically ask them to create XML. JSON is widely used, easy to handle, and finally, and this was the decisive factor, more compact.
The choice of the interoperability format that you send to an LLM has a significant direct impact on the costs of your LLM-operated automated translation programme. LLMs require only limited information – things like previews, segment history, internal segment IDs, internal segment contexts do not add any value for them, but burn tokens. Long descriptions of segments with long attributes relentlessly burn tokens. Simply the embedding of angle brackets at the beginning and the end of each attribute extends the length of the representation. XML representations like XLIFF were consistently forwarded in ZIPs (Trados and memoQ packages are zipped file formats), which shortens such long representations, and with the cost of hard drives, this is fine. However, passing the same information over to an LLM is expensive.
LLMs can also learn from instructions. Skills appeared to explain LLMs how to solve problems. Skills are the ideal place to leave legend and instructions to a compact file format. It is the place where you can explain what segment statuses the single-letter abbreviations are representing, and how you want it to give back processed information.
We found that the JSON representation takes ca. 10% of the original XLIFF representation of the same information in token usage. If you just shorten the representation of tags, only passing the source and target segments over to the LLM, it can still save you 20-30% of the token cost. This is the main reason why we left XML behind, and moved to JSON – to save money for our customers.
What does this JSON format do?
Since 2023, BeLazy has traditionally been able to extract any XLIFF file at any workflow step from any of the supported TMSes, and then import back the XLIFF file later at the end of the workflow step. This enables workflows with human and automated steps.
We found that this XLIFF in itself is not enough to work with LLMs. Therefore, from now on, we are gradually adding support to extract not only the XLIFF, but also information that is only available about the document’s segments via public or private API calls. We will retain the XLIFF for those who wish to work with it, however, this XLIFF will not contain all the information BeLazy fetched, only the XLIFF that was fetched from the TMS.
The full information content about a bilingual document will be stored and passed over in a JSON file, and this very JSON format will be the one that you can also use to import the changes. Just the way BeLazy works with XLIFF export and API calls to fetch the information, it will also import the XLIFF back and perform the necessary API operations to update the information based on the contents of the JSON.
Therefore, the new JSON format is your one-stop solution to interoperability in the LLM age. You can send this directly to LLMs, or use it in your new application. The format is simple. The specification of the format is kept in an LLM skill (such as a Claude skill), which explains the format both to LLMs and to human users. Therefore this new format is as open and clearly documented as XLIFF is.
Eulogy
XLIFF, you’ve been around long enough. I found you to be the most useful standard in the translation industry. But times are changing, and we did not foresee that verbosity will one day become a cost factor. In the old days we used to pay more for long telephone calls. Today we pay more for long LLM requests. It is time to take a slow farewell. Developers, please feel free to use our format. It is truly meant to be interoperable.
