Localization and Internationalization

Introduction

Torque 3D's localization and internationalization (l10n and i18n) support is slightly broken. In this article we'll show you how to fix it up a little. It's actually remarkably simple once you see it laid out.

If you are not familiar with compiling C/C++ programs, go get familiar with it before attempting any engine-side changes. This is a simple code change, but if you don't know how to use a compiler then you will spend countless frustrated hours on this.

Engine-side Changes


First, open source/i18n/lang.h. After this line: #include "core/util/tVector.h" just near the start, add:

//lang_ localization
#include "core/fileObject.h"
#include "core/util/str.h"
#include "core/strings/unicode.h"


Next, in source/i18n/lang.cpp, replace the whole method bool LangFile::load(Stream *s) {...} with:

bool LangFile::load(Stream *s)
{
	freeTable();
	
	while(s->getStatus() == Stream::Ok)
	{
		char buf[2048];
		s->readLongString(2048,buf);
		if (s->getStatus() == Stream::Ok)
			addString((const UTF8*)buf);
	}
	return true;
}


Then replace the bool LangFile::save(Stream *s) {...} method with:

bool LangFile::save(Stream *s)
{
	if(!isLoaded())
		return false;
	
	U32 i;
	for(i = 0;i < mStringTable.size();i++)
	{
		s->writeLongString(2048, (char*)mStringTable[i]); //irei1as_ lang
	}
	return true;
}


Finally, at the end of source/i18n/lang.cpp add:

//lang_ localization
bool compiledFileNeedsUpdate(UTF8* filename)
{
	Torque::Path filePath = Torque::Path(filename);
	Torque::FS::FileNodeRef sourceFile = Torque::FS::GetFileNode(filePath);
	Torque::Path compiledPath = Torque::Path(filePath);
	compiledPath.setExtension("lso");
	Torque::FS::FileNodeRef compiledFile = Torque::FS::GetFileNode(compiledPath);

	Torque::Time sourceModifiedTime, compiledModifiedTime;

	if (sourceFile != NULL)
		sourceModifiedTime = sourceFile->getModifiedTime();

	if (compiledFile != NULL)
		compiledModifiedTime = compiledFile->getModifiedTime();

	if (sourceModifiedTime > compiledModifiedTime)
		return true;
	return false;
}

ConsoleFunction( CompileLanguage, void, 2, 3, "(string inputFile, [bool createMap]) Compiles a LSO language file."
                                              " if createIndex is true, will also create languageMap.cs with"
                                              " the global variables for each string index."
                                              " The input file must follow this example layout:"
                                              " TXT_HELLO_WORLD = Hello world in english!" )
{
	UTF8 scriptFilenameBuffer[1024];
	Con::expandScriptFilename((char*)scriptFilenameBuffer, sizeof(scriptFilenameBuffer), argv[1]);

	if (!Torque::FS::IsFile(scriptFilenameBuffer))
	{
		Con::errorf("CompileLanguage - file %s not found", scriptFilenameBuffer);
		return;
	}

	FileObject file;
	if (!file.readMemory(scriptFilenameBuffer))
	{
		Con::errorf("CompileLanguage - couldn't read file %s", scriptFilenameBuffer);
		return;
	}

	if (compiledFileNeedsUpdate(scriptFilenameBuffer))
	{
		bool createMap = argc > 2 ? dAtob(argv[2]) : false;
		FileStream *mapStream = NULL;
		if (createMap)
		{
			Torque::Path mapPath = scriptFilenameBuffer;
			mapPath.setFileName("languageMap");
			mapPath.setExtension("cs");
			if ((mapStream = FileStream::createAndOpen(mapPath, Torque::FS::File::Write)) == NULL)
				Con::errorf("CompileLanguage - failed creating languageMap.cs");
		}

		LangFile langFile;
		const U8* inLine = NULL;
		const char* separatorStr = " = ";
		S32 stringId = 0;
		while ((inLine = file.readLine())[0] != 0)
		{
			char* line;
			chompUTF8BOM((const char *)inLine, &line);
			char* div = dStrstr(line, separatorStr);
			if (div == NULL)
			{
				Con::errorf("Separator %s not found in line: %s", separatorStr, line);
				Con::errorf("Could not determine string name ID");
				continue;
			}
			*div = 0;
			char* text = div + dStrlen(separatorStr);

			langFile.addString((const UTF8*)text);

			if (mapStream)
			{
				String mapLine = String::ToString("$%s = %i;", line, stringId);
				mapStream->writeLine((const U8*)mapLine.c_str());
				String commentLine = String::ToString("// %s", text);
				mapStream->writeLine((const U8*)commentLine.c_str());
			}

			stringId++;
		}

		Torque::Path lsoPath = scriptFilenameBuffer;
		lsoPath.setExtension("lso");
		langFile.save(lsoPath.getFullPath());

		if (mapStream)
			delete mapStream;
	}
}
//end of the C++ changes for localization

Script-side Changes


Now the Torquescript changes. We're going to start by adding our language map files.

Create a "english.txt" in game/scripts and make its text:

txt_hello = Hello world
txt_language = This is the English language
txt_goodbye_going = Goodbye
txt_goodbye_staying = Goodbye
txt_last = This is the last string


You may want it to be UTF-8 without BOM


Add also a "korean.txt" with its contents except the "txt_..." translated like:

txt_hello = 안녕하새요
txt_language = 이곳은 한국어 이예요
txt_goodbye_going = 안녕히 계세요
txt_goodbye_staying = 안녕히 가세요
txt_last = 마지막이다


At the end of game/scripts/main.cs add:

new LangTable(mainLangTable); //lang_ localization
$I18N::default = mainLangTable.getId();

function updateLanguages()
{
    compileLanguage("./english.txt", 1); //that 1 exists to create languageMap.cs
    compileLanguage("./korean.txt", 0);

    mainLangTable.addLanguage("./english.lso", "English"); //language #0
    mainLangTable.addLanguage("./korean.lso", "Korean"); //language #1

    mainLangTable.setCurrentLanguage(0);
    exec("./languageMap.cs"); //this file is created by a previous compileLanguage
}
// This should only actually happen when the .txt is newer than the .lso
// but we can go ahead and call it ever time.
updateLanguages();

function selectLanguage(%language) //use here the #n as it's the order of inclusion
{
	mainLangTable.setCurrentLanguage(%language);
	Canvas.setContent(Canvas.getContent());
}


Finally, in game/main.cs inside function createCanvas(%windowTitle) before return true; add:

   Canvas.langTableMod = "default";

An Example of Use

The order of addition in game/scripts/main.cs sets the index of the language. In our case, English is 0 and Korean is 1.

Use switchLanguage(%languageIndex), where %languageIndex is the assigned language index, to select the current language.

In GUI controls, just use the text_id as assigned in the language map file (for example, txt_hello for the current language's "hello" text).

In script, call mainlangtable.getstring($txt_hello). It will return the localized text for the requested key.

In C++, get the LangTable and call getString() like the GUI does. It may be convoluted to program. Probably just using something similar to: UTF8 * GuiControl::getGUIString(S32 id).


To illustrate how to use this, we'll create a GUI to display our text, along with some buttons to make it happen.


First, the GUI file. Place it in scripts/gui/LangTest.gui:

//--- OBJECT WRITE BEGIN ---
%guiContent = new GuiWindowCtrl(LangTest) {
   text = "New Window";
   resizeWidth = "1";
   resizeHeight = "1";
   canMove = "1";
   canClose = "1";
   canMinimize = "1";
   canMaximize = "1";
   canCollapse = "0";
   edgeSnap = "1";
   margin = "0 0 0 0";
   padding = "0 0 0 0";
   anchorTop = "1";
   anchorBottom = "0";
   anchorLeft = "1";
   anchorRight = "0";
   position = "0 0";
   extent = "1024 768";
   minExtent = "8 2";
   horizSizing = "right";
   vertSizing = "bottom";
   profile = "GuiWindowProfile";
   visible = "1";
   active = "1";
   tooltipProfile = "GuiToolTipProfile";
   hovertime = "1000";
   isContainer = "1";
   canSave = "1";
   canSaveDynamicFields = "1";

   new GuiTextCtrl() {
      text = "Last";
      maxLength = "1024";
      margin = "0 0 0 0";
      padding = "0 0 0 0";
      anchorTop = "1";
      anchorBottom = "0";
      anchorLeft = "1";
      anchorRight = "0";
      position = "17 84";
      extent = "64 18";
      minExtent = "8 2";
      horizSizing = "right";
      vertSizing = "bottom";
      profile = "GuiTextProfile";
      visible = "1";
      active = "1";
      tooltipProfile = "GuiToolTipProfile";
      hovertime = "1000";
      isContainer = "1";
      canSave = "1";
      canSaveDynamicFields = "0";
   };
   new GuiTextCtrl(tbxLast) {
      text = "...";
      maxLength = "1024";
      margin = "0 0 0 0";
      padding = "0 0 0 0";
      anchorTop = "1";
      anchorBottom = "0";
      anchorLeft = "1";
      anchorRight = "0";
      position = "124 83";
      extent = "300 20";
      minExtent = "8 2";
      horizSizing = "right";
      vertSizing = "bottom";
      profile = "GuiTextProfile";
      visible = "1";
      active = "1";
      tooltipProfile = "GuiToolTipProfile";
      hovertime = "1000";
      isContainer = "1";
      canSave = "1";
      canSaveDynamicFields = "0";
   };
   new GuiTextCtrl() {
      text = "Hello World";
      maxLength = "1024";
      margin = "0 0 0 0";
      padding = "0 0 0 0";
      anchorTop = "1";
      anchorBottom = "0";
      anchorLeft = "1";
      anchorRight = "0";
      position = "17 40";
      extent = "64 18";
      minExtent = "8 2";
      horizSizing = "right";
      vertSizing = "bottom";
      profile = "GuiTextProfile";
      visible = "1";
      active = "1";
      tooltipProfile = "GuiToolTipProfile";
      hovertime = "1000";
      isContainer = "1";
      canSave = "1";
      canSaveDynamicFields = "0";
   };
   new GuiTextCtrl(tbxHelloWorld) {
      text = "...";
      maxLength = "1024";
      margin = "0 0 0 0";
      padding = "0 0 0 0";
      anchorTop = "1";
      anchorBottom = "0";
      anchorLeft = "1";
      anchorRight = "0";
      position = "124 39";
      extent = "300 20";
      minExtent = "8 2";
      horizSizing = "right";
      vertSizing = "bottom";
      profile = "GuiTextProfile";
      visible = "1";
      active = "1";
      tooltipProfile = "GuiToolTipProfile";
      hovertime = "1000";
      isContainer = "1";
      canSave = "1";
      canSaveDynamicFields = "0";
   };
   new GuiTextCtrl() {
      text = "Language";
      maxLength = "1024";
      margin = "0 0 0 0";
      padding = "0 0 0 0";
      anchorTop = "1";
      anchorBottom = "0";
      anchorLeft = "1";
      anchorRight = "0";
      position = "17 62";
      extent = "64 18";
      minExtent = "8 2";
      horizSizing = "right";
      vertSizing = "bottom";
      profile = "GuiTextProfile";
      visible = "1";
      active = "1";
      tooltipProfile = "GuiToolTipProfile";
      hovertime = "1000";
      isContainer = "1";
      canSave = "1";
      canSaveDynamicFields = "0";
   };
   new GuiTextCtrl(tbxLanguage) {
      text = "...";
      maxLength = "1024";
      margin = "0 0 0 0";
      padding = "0 0 0 0";
      anchorTop = "1";
      anchorBottom = "0";
      anchorLeft = "1";
      anchorRight = "0";
      position = "124 61";
      extent = "300 20";
      minExtent = "8 2";
      horizSizing = "right";
      vertSizing = "bottom";
      profile = "GuiTextProfile";
      visible = "1";
      active = "1";
      tooltipProfile = "GuiToolTipProfile";
      hovertime = "1000";
      isContainer = "1";
      canSave = "1";
      canSaveDynamicFields = "0";
   };
   new GuiTextCtrl(tbxGoodbye1) {
      text = "...";
      maxLength = "1024";
      margin = "0 0 0 0";
      padding = "0 0 0 0";
      anchorTop = "1";
      anchorBottom = "0";
      anchorLeft = "1";
      anchorRight = "0";
      position = "124 105";
      extent = "300 20";
      minExtent = "8 2";
      horizSizing = "right";
      vertSizing = "bottom";
      profile = "GuiTextProfile";
      visible = "1";
      active = "1";
      tooltipProfile = "GuiToolTipProfile";
      hovertime = "1000";
      isContainer = "1";
      canSave = "1";
      canSaveDynamicFields = "0";
   };
   new GuiTextCtrl() {
      text = "Goodbye 1";
      maxLength = "1024";
      margin = "0 0 0 0";
      padding = "0 0 0 0";
      anchorTop = "1";
      anchorBottom = "0";
      anchorLeft = "1";
      anchorRight = "0";
      position = "17 106";
      extent = "64 18";
      minExtent = "8 2";
      horizSizing = "right";
      vertSizing = "bottom";
      profile = "GuiTextProfile";
      visible = "1";
      active = "1";
      tooltipProfile = "GuiToolTipProfile";
      hovertime = "1000";
      isContainer = "1";
      canSave = "1";
      canSaveDynamicFields = "0";
   };
   new GuiTextCtrl(tbxGoodbye2) {
      text = "...";
      maxLength = "1024";
      margin = "0 0 0 0";
      padding = "0 0 0 0";
      anchorTop = "1";
      anchorBottom = "0";
      anchorLeft = "1";
      anchorRight = "0";
      position = "124 127";
      extent = "300 20";
      minExtent = "8 2";
      horizSizing = "right";
      vertSizing = "bottom";
      profile = "GuiTextProfile";
      visible = "1";
      active = "1";
      tooltipProfile = "GuiToolTipProfile";
      hovertime = "1000";
      isContainer = "1";
      canSave = "1";
      canSaveDynamicFields = "0";
   };
   new GuiTextCtrl() {
      text = "Goodbye 2";
      maxLength = "1024";
      margin = "0 0 0 0";
      padding = "0 0 0 0";
      anchorTop = "1";
      anchorBottom = "0";
      anchorLeft = "1";
      anchorRight = "0";
      position = "17 128";
      extent = "64 18";
      minExtent = "8 2";
      horizSizing = "right";
      vertSizing = "bottom";
      profile = "GuiTextProfile";
      visible = "1";
      active = "1";
      tooltipProfile = "GuiToolTipProfile";
      hovertime = "1000";
      isContainer = "1";
      canSave = "1";
      canSaveDynamicFields = "0";
   };
   new GuiButtonCtrl(btnEnglish) {
      text = "English";
      groupNum = "-1";
      buttonType = "PushButton";
      useMouseEvents = "0";
      position = "12 161";
      extent = "140 30";
      minExtent = "8 2";
      horizSizing = "right";
      vertSizing = "bottom";
      profile = "GuiButtonProfile";
      visible = "1";
      active = "1";
      tooltipProfile = "GuiToolTipProfile";
      hovertime = "1000";
      isContainer = "0";
      canSave = "1";
      canSaveDynamicFields = "0";
   };
   new GuiButtonCtrl(btnKorean) {
      text = "Korean";
      groupNum = "-1";
      buttonType = "PushButton";
      useMouseEvents = "0";
      position = "168 161";
      extent = "140 30";
      minExtent = "8 2";
      horizSizing = "right";
      vertSizing = "bottom";
      profile = "GuiButtonProfile";
      visible = "1";
      active = "1";
      tooltipProfile = "GuiToolTipProfile";
      hovertime = "1000";
      isContainer = "0";
      canSave = "1";
      canSaveDynamicFields = "0";
   };
   new GuiRadioCtrl(rbtnBatang) {
      text = "Batang";
      groupNum = "4";
      buttonType = "RadioButton";
      useMouseEvents = "0";
      position = "169 198";
      extent = "140 13";
      minExtent = "8 2";
      horizSizing = "right";
      vertSizing = "bottom";
      profile = "GuiRadioProfile";
      visible = "1";
      active = "1";
      tooltipProfile = "GuiToolTipProfile";
      hovertime = "1000";
      isContainer = "0";
      canSave = "1";
      canSaveDynamicFields = "0";
   };
   new GuiRadioCtrl(rbtnGungsuh) {
      text = "Gungsuh";
      groupNum = "4";
      buttonType = "RadioButton";
      useMouseEvents = "0";
      position = "169 217";
      extent = "140 13";
      minExtent = "8 2";
      horizSizing = "right";
      vertSizing = "bottom";
      profile = "GuiRadioProfile";
      visible = "1";
      active = "1";
      tooltipProfile = "GuiToolTipProfile";
      hovertime = "1000";
      isContainer = "0";
      canSave = "1";
      canSaveDynamicFields = "0";
   };
   new GuiRadioCtrl(rbtnDotum) {
      text = "Dotum";
      groupNum = "4";
      buttonType = "RadioButton";
      useMouseEvents = "0";
      position = "169 236";
      extent = "140 13";
      minExtent = "8 2";
      horizSizing = "right";
      vertSizing = "bottom";
      profile = "GuiRadioProfile";
      visible = "1";
      active = "1";
      tooltipProfile = "GuiToolTipProfile";
      hovertime = "1000";
      isContainer = "0";
      canSave = "1";
      canSaveDynamicFields = "0";
   };
};
//--- OBJECT WRITE END ---

Ensure that you have the Korean language pack installed so that the standard Korean fonts are available. Now, in art/gui/customProfiles.cs add our Korean font profiles:

if( !isObject( GuiKoreanTextProfile ) )
new GuiControlProfile( GuiKoreanTextProfile : GuiTextProfile )
{
    fontType = "BatangChe";
};

if( !isObject( GuiKoreanTextProfileGungsuh ) )
new GuiControlProfile( GuiKoreanTextProfileGungsuh : GuiTextProfile )
{
    fontType = "GungsuhChe";
};
if( !isObject( GuiKoreanTextProfileDotum ) )
new GuiControlProfile( GuiKoreanTextProfileDotum : GuiTextProfile )
{
    fontType = "DotumChe";
};

Next, add scripts/gui/LangTest.cs and copy in the following script:

exec("./LangTest.gui");

function btnEnglish::onClick(%this)
{
    selectLanguage(0);
    $SelectedLang = 0;
    tbxHelloWorld.profile = GuiTextProfile;
    tbxHelloWorld.setText(mainlangtable.getstring($txt_hello));
    tbxLanguage.profile = GuiTextProfile;
    tbxLanguage.setText(mainlangtable.getstring($txt_language));
    tbxGoodbye1.profile = GuiTextProfile;
    tbxGoodbye1.setText(mainlangtable.getstring($txt_goodbye_staying));
    tbxGoodbye2.profile = GuiTextProfile;
    tbxGoodbye2.setText(mainlangtable.getstring($txt_goodbye_going));
    tbxLast.profile = GuiTextProfile;
    tbxLast.setText(mainlangtable.getstring($txt_last));
}

function btnKorean::onClick(%this)
{
    selectLanguage(1);
    $SelectedLang = 1;
    tbxHelloWorld.profile = GuiKoreanTextProfile;
    tbxHelloWorld.setText(mainlangtable.getstring($txt_hello));
    tbxLanguage.profile = GuiKoreanTextProfile;
    tbxLanguage.setText(mainlangtable.getstring($txt_language));
    tbxGoodbye1.profile = GuiKoreanTextProfile;
    tbxGoodbye1.setText(mainlangtable.getstring($txt_goodbye_staying));
    tbxGoodbye2.profile = GuiKoreanTextProfile;
    tbxGoodbye2.setText(mainlangtable.getstring($txt_goodbye_going));
    tbxLast.profile = GuiKoreanTextProfile;
    tbxLast.setText(mainlangtable.getstring($txt_last));
}

function rbtnBatang::onClick(%this)
{
    if($SelectedLang == 1)
    {
        tbxHelloWorld.profile = GuiKoreanTextProfile;
        tbxLanguage.profile = GuiKoreanTextProfile;
        tbxGoodbye1.profile = GuiKoreanTextProfile;
        tbxGoodbye2.profile = GuiKoreanTextProfile;
        tbxLast.profile = GuiKoreanTextProfile;
    }
}

function rbtnGungsuh::onClick(%this)
{
    if($SelectedLang == 1)
    {
        tbxHelloWorld.profile = GuiKoreanTextProfileGungsuh;
        tbxLanguage.profile = GuiKoreanTextProfileGungsuh;
        tbxGoodbye1.profile = GuiKoreanTextProfileGungsuh;
        tbxGoodbye2.profile = GuiKoreanTextProfileGungsuh;
        tbxLast.profile = GuiKoreanTextProfileGungsuh;
    }
}

function rbtnDotum::onClick(%this)
{
    if($SelectedLang == 1)
    {
        tbxHelloWorld.profile = GuiKoreanTextProfileDotum;
        tbxLanguage.profile = GuiKoreanTextProfileDotum;
        tbxGoodbye1.profile = GuiKoreanTextProfileDotum;
        tbxGoodbye2.profile = GuiKoreanTextProfileDotum;
        tbxLast.profile = GuiKoreanTextProfileDotum;
    }
}

Finally, in scripts/client/init.cs after exec("scripts/gui/optionsDlg.cs"); (around line 98) add a line to load scripts/gui/LangTest.cs:

exec("scripts/gui/LangTest.cs");

Once that's in place, fire up T3D and give it a shot. Once the main menu is loaded up, open the console and type canvas.pushDialog(LangTest).


You should see the new GUI:


Click the English button to see the fields populated with English text:


Click the Korean button to see the fields populated with Korean text in the default BatangChe font:


Click the Gungsuh radio button to see the fields populated with Korean text in the GungsuhChe font:


Click the Dotum radio button to see the fields populated with Korean text in the DotumChe font:

Conclusion

And there you have it - probably not foolproof, but it gets the job done.


Special thanks to irei1as on Torque3D.org for finding the original resource and bringing it back to the light of day.