Automate Your Translations

istock_000019221284small_translatecomputerbutton_reducedcopy-1[1] Taking a break from my posts on PowerShell this week, I would like to retouch a topic that I talked about before in my post Localization With Angular-GetText. In that post I talk about how to localize your angular application using PoEdit to add your translations. If you have a large number of people developing your application or a lot of languages to update this can cause friction.  I got to thinking that there had to be a better way of doing this.  There just had to be a way to automate this to reduce the amount of friction during the development process.

Old Way

You have to first annotate your code with the Translate attribute. Then you must perform the Grunt extract method to update your translation catalog file (*.pot). Next, you must open each translation file (*.po) inside PoEdit, update from catalog, and clear out the guesses that PoEdit does.
Note: PoEdit will perform this guesswork even if you have Translation Memory turned off!

Finally, you must enter the translation into every translation file and save it.
Once you have completed entering the translation for every word that needed to be translated, you must perform the grunt compile operation to update your translations.js file that your application is referencing. Let’s look at the steps:

  1. Annotate Application
  2. Run ‘Grunt Extract’
  3. Open first translation file in PoEdit
  4. Update translation file from Catalog file
  5. Remove the unwanted or fuzzy translation from the translation file
  6. Enter the correct translation for each word
  7. Save the translation file
  8. Repeat steps 3 – 7 for each translation file
  9. Run ‘Grunt Compile’ to build the translations.js file
  10. Done

If you plan on supporting the basic four languages in your application (English, Spanish, French, German), then you will end up performing a minimum of 23 steps just because you adding only 1 more item to be translated.  The more translations you have to update in each file, the longer it will take.  I have found in practice that this takes a lot of time away from the developer.  If he or she is spending their time on performing this repetitive action, they tend to resist this and finally not doing it.  This leads to translations getting missed and your application will show a lot of “MISSING” tags being shown when the debug mode is turned on.

angular.module('myApp').run(function(gettextCatalog) {
   gettextCatalog.debug = true;
});

Automated Approach

There had to be a better way, so I came up with a more automated approach. I wanted to be able to have the computer perform all of these operations for me, because that’s what computers are good at. The only part of the process that really required human intervention was placing the actual translation into to the individual translation files. While I made no illusion that I could replace a human being for this part of the process, I was able to get a basic and approximate translation from an on-line translation service such as Bing Translator, Google Translate, BabelFish, etc.  The problem was, how can I automate the calls to one of these services to solve my problem. A while back I created a Language Translation tool that would help me update my Microsoft String Resource files.  That project was created to allow Visual Studio developers to globalize their software, and can be found here.
Here is a sample of the code that I’m using to connect to Google’s translation service to get the basic translation from.

using System;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Net;
using System.Web;
using System.Web.Script.Serialization;
using AutomatedTranslation.Infos;

namespace AutomatedTranslation.Engines
{
    public interface IGoogleTranslateEngine
    {
        string FromCulture { get; set; }
        string ToCulture { get; set; }
        string TranslateWordOrPhrase(string wordOrPhraseToTranslate);
    }

    public class GoogleTranslateEngine : IGoogleTranslateEngine
    {
        private const string englishCulture = "en";
        private const string googleUrlFormat = "http://translate.google.com/translate_a/t?client=webapp&sl={0}&tl={1}&hl=en&q={2}sc=1";

        private readonly JavaScriptSerializer javaScriptSerializer;

        public string FromCulture { get; set; }
        public string ToCulture { get; set; }

        public GoogleTranslateEngine()
        {
            FromCulture = englishCulture;
            ToCulture = englishCulture;

            javaScriptSerializer = new JavaScriptSerializer();
        }

        public string TranslateWordOrPhrase(string wordOrPhraseToTranslate)
        {
            var translatedValue = wordOrPhraseToTranslate;

            try
            {
                var url = String.Format(googleUrlFormat, FromCulture, ToCulture, HttpUtility.UrlEncode(wordOrPhraseToTranslate));
                var webReq = CreateTranslationRequest(url);
                using (var webResponse = webReq.GetResponse())
                {
                    using (var responseStream = webResponse.GetResponseStream())
                    {
                        if (responseStream == null)
                            throw new Exception("No response stream found for the given url");

                        var streamReader = new StreamReader(responseStream, System.Text.Encoding.UTF8);
                        var responseData = streamReader.ReadToEnd();

                        translatedValue = GetTranslatedValueFromJson(responseData);
                    }
                }
            }
            catch (Exception ex)
            {
                Trace.WriteLine("Unable to translate due to the follow error.");
                Trace.WriteLine(ex);
                if (Debugger.IsAttached) Debugger.Break();
            }

            return translatedValue;
        }

        private HttpWebRequest CreateTranslationRequest(string url)
        {
            var webReq = (HttpWebRequest)WebRequest.Create(url);
            webReq.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
            webReq.ContentType = "application/json";
            webReq.UserAgent = "Opera/12.02 (Android 4.1; Linux; Opera Mobi/ADR-1111101157; U; en-US) Presto/2.9.201 Version/12.02";
            webReq.Referer = "http://translate.google.com/m/translate";

            return webReq;
        }

        private string GetTranslatedValueFromJson(string page)
        {
            var json = javaScriptSerializer.Deserialize<GoogleTranslationResult>(page);

            return json.sentences.First().trans;
        }
    }
}

Next I took this and wrote a utility around it that would read an updated catalog file and then get the translation utilizing the engine above. Next is open the existing translation files and then update the existing blank translations as well as create the new ones needed. Once all of the translation files are updated then it would need compiled back into the system.
This is good, but not quite good enough for me. So I decided to incorporated my new tool into my Visual Studio build process.

Here is a snippet of the project file XML to show what it looks like inside the project file.

<PropertyGroup>
  <PreBuildEvent>grunt extract --no-color --gruntfile $(ProjectDir)Gruntfile.js</PreBuildEvent>
  <PostBuildEvent>(SolutionDir)tools\Translation\bin\$(ConfigurationName)\AutomatedTranslation.exe /languagePath=$(ProjectDir)lang
grunt compile --no-color --gruntfile $(ProjectDir)Gruntfile.js</PostBuildEvent>
</PropertyGroup>

Now let’s review the steps and see what we have gained by doing all of this work. Let’s review the steps needed:

  1. Annotate Application
  2. Build The Solution
  3. Done

That is a substantial improvement in productivity!

Conclusion

Is this the only way to perform your translations?
No, you can feel free to come up with your own approach, I did this and it worked at my company.
Should you rely on the automated translations?
Not at all, automated translations WILL be wrong from time to time or use the wrong word for the context. You should have a professional translation service review your application and your translations to make sure that you can give your customer the best possible customer experience.
If you would like to see the complete source code to the Automated Translation project, then visit me on GitHub at Dacke/AutomatedTranslation. Take a look at the source code and maybe you can help me improve it!

Until next time; Keep Calm and Automate

UPDATE

OK, placing the “grunt extract” in my pre-build step might not have been the best solution. It was designed to demonstrate an example of the process rather than the full process itself. I’ve since changed this up a little bit as I found that running the grunt task a bit tedious when I didn’t need to do it. I’ve created a PowerShell script that will do the job for me.

Build

I need to build the solution and I can do that using MSBuild.exe from the command line. I needed the PowerShell script to run this process for all of the solutions that make up my application. My solutions use NuGet packages to manage the references for various third party components. The build script will need to restore those before the build process is called.

function RestoreNuGetPackages {
	Param([string]$solutionName)

	$solutionPath = "src\$solutionName";

	Write-Host ('`tRestoring NuGet packages for the ' + $solutionName + ' solution') -ForegroundColor Yellow
	Invoke-Expression ($sourceFolder + '\.nuget\NuGet.exe restore ' + $solutionPath) | Write-Host -ForegroundColor DarkGreen
}

function BuildSolutions {
	$solutions = New-Object 'system.collections.generic.dictionary[string,string]'
	$solutions.Add('Company.Solution1.sln', $null)
	$solutions.Add('Company.Solution2.sln', $null)
	$solutions.Add('Company.Solution3.sln', $null)

	foreach($sln in $solutions.GetEnumerator()) {
		BuildSolution $sln.Key $sln.Value
	}
}

Now that the NuGet packages are restored, we can move on to the rest of the script that will call the build process for the solutions.

[string] $msBuildLocation = 'C:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe'

function BuildSolution{
	Param([string]$solutionName, [string]$parallel = '/m')

	RestoreNuGetPackages $solutionName

	Write-Host ("`tBuilding Solution '" + $solutionName + "'") -ForegroundColor Yellow

	$solutionPath = 'src\$solutionName'

	if ([System.IO.File]::Exists($msBuildLocation) -eq $false) {
		Write-Error 'Unable to find the Microsoft Build executable for the 4.0 runtime.  Please verify this is installed before you proceed.'
		return 772
	}

	C:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe $solutionPath /t:Build /p:Configuration=Debug /verbosity:m /nodeReuse:False $parallel | Write-Host
	if($lastexitcode -gt 0)
	{
		[string] $erroredSolution = ($solutionName + ' failed.')
		$error.Add($erroredSolution)
		Write-Error $erroredSolution
		Throw $erroredSolution
	}

	Write-Host ("`t--> Successfully Built Solution '" + $solutionName + "'") -ForegroundColor Green

	return $lastexitcode
}

Execute Grunt Task

Remember that we have two different grunt tasks that we want to call; extract and compile. I’ve created a function that will allow us to do both operations depending on the parameters passed in.

function ExecuteGruntTask {
    Param([Parameter(HelpMessage='The operation that you want to perform. [extract, compile].`n')]
          [ValidateSet('extract', 'compile')]
          [string] $operation,
          [Parameter(HelpMessage='The path to the GruntFile.js (or equivalent).`n')]
          [string] $gruntFile)

    [System.Diagnostics.Process] $process = New-Object System.Diagnostics.Process
    $process.StartInfo.FileName = [System.IO.Path]::Combine([System.Environment]::GetFolderPath([System.Environment+SpecialFolder]::ApplicationData), 'npm\grunt.cmd')
    $process.StartInfo.Arguments = ($operation + ' --no-color --gruntfile ' + $gruntFile)
    $process.StartInfo.UseShellExecute = $false
    $process.Start()

    $process.WaitForExit();
    if ($process.ExitCode -ne 0) {
        throw ('Unable to perform the ' + $operation + ' correctly.  Please correct the error and try again.')
    }
}

Putting All Together

Now that we have all of our functions in place, it’s now time to put it all together.

Param([Parameter(HelpMessage='Pass in $false if you to skip the translation step.`n')]
      $doNotTranslate = $false)

BuildSolutions

if ($doNotTranslate -eq $false) {
    Write-Host 'Performing translation steps.'
    Write-Host 'Extracting strings to be translated...'
    ExecuteGruntTask -operation extract -gruntFile $sourceFolder\Company.Project1\Gruntfile.js

    Write-Host 'Getting the automated translations...'
    ExecuteAutomatedTranslationTask -sourceFolder ([System.IO.Path]::Combine($sourceFolder, 'Company.Project1')) -translationExecutable ([System.IO.Path]::Combine($sourceFolder, 'tools\TranslationTool\bin\Debug\AutomatedTranslation.exe'))

    Write-Host 'Compiling the translated strings...'
    ExecuteGruntTask -operation compile -gruntFile $sourceFolder\Company.Project1\Gruntfile.js
}

Note: The ExecuteAutomatedTranslationTask function is basically a duplicate of the ExecuteGruntTask method with the exception that it calls the Automated Translation program we built in the first portion of our post.

You can also include things like Unit Tests and Karma Testing in your PowerShell script as well.  This can ensure that your code is the best it can be before you check your code into the software repository.  I hope that this update helps make your development environment more productive.  Until next time, keep up the good work!

Important News

I’ve taken this idea forward and created an Automated Translation Tool that I have shared with all of you.  The package can be found on Microsoft’s NuGet server here.  I’ve also written about it here.  If you are looking for the source code you can look at my repo on GitHub.  Take a look at it, heck even fork the repo and help me make the tool better.  I look forward to your pull requests!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s