Evaluate Language Translation Models quicker using Tiyaro Experiment

Evaluate Language Translation Models quicker using Tiyaro Experiment

We evaluate the translation capabilities of our SaaS and PaaS models over Japanese to English translation.

Sharvil Mainkar's photo
Sharvil Mainkar
·Jul 11, 2022·

3 min read

Play this article

Language Translation is the task of converting one natural language to another without changing the meaning and the intent in the original language. Translation has a variety of use cases from the consumer, and retail to defense. Most common is analyzing large text documents from one language to another. There are numerous translation SaaS vendors and open-sourced machine learning models available, but selecting one model for any given use case is a time-consuming task. Finding and using such models / SaaS APIs take considerable effort. You can use Tiyaro to reduce the effort to merely minutes.

1s.png Search for translation models in Tiyaro Explore

2s.png Search results displaying various models.

Accelerating the rate of model evaluation using: Tiyaro Experiments

Tiyaro Experiments allows us a quicker way to compare different pre-trained machine learning models along with the SaaS Vendor APIs helping us accelerate the evaluation process. One such translation experiment is Japanese text translation which we conduct to translate different blocks of text (articles, short stories, news headlines, etc ) from Japanese to English. For that we utilize the following State of the Art pre-trained machine learning models / SaaS API :

Once we have completed searching and using demos for different models we can head on over to create experiments. There are various ways to create experiments but we are simply going over to the experiments tab. Where we start a new experiment.

3s.png

After that, we can select our experiment class type, which in our case would be Translation.

4s.png

Once completed we can select the models to train the experiments on, the model selection provides us with various filters.

5s.png

After selecting the models / SaaS API vendor solution you can upload your custom dataset, there are various formats to upload the translation dataset on Tiyaro for the experiment. Easiest is zipping the CSV file containing one column as the input.

6s.png

After running the experiment we can see the results tab show up, here you would also be able to see the latency of the models as well as the results of the experiment, for our use case we are able to see the table containing our input and the respective model predictions. We can also download the result in a zip file.

7s.png

The result below is shown in the table, the first column containing the actual raw input and the subsequent columns being the translation result.

8s.png

Comparing one such translated news headline:

Original News Headline in Japanese:

FIFA前会長が元副会長へ不正支払い 2人に無罪 スイスの裁判所

Translation by IBM Watson JP-EN:

Former FIFA president acquitted ex-vice president on fraud payments

Translation by Helsinki JP-EN model:

Pre-President FIFFA had an illegal payment to former vice presidents, two innocents, and a Swiss court.

As seen from the above results both the models / SaaS API service gives us different translations. The evaluation of such translation experiments requires human evaluation for the accuracy of the result.

You can share the experiment with your coworkers and on social media. Also, you can make a copy of the given experiment to enhance or modify the particular experiment.

Wish to create one? Head on over to Tiyaro!

 
Share this