<Top of this site> <Top of programming pages in this site>

Functions to handle multiple octets character encoding scheme

libmoe-1.5.8.tar.gz (1523KB, 2004-11-21 18:34:48)
is gzipped tarball of a collection of functions to handle sequences of characters consisting of multiple octets. It includes a character encoding conversion tool which is initially written for debugging purpose of this library. In spite of my initial intention, I believe that it is very useful tool. You can view ChangeLog:
libmoe-1.5.8-ChangeLog.txt (31KB, 2004-11-21 18:34:48)
which is included in the above tarball.

The developement version:

libmoe-devel.tar.gz (1539KB, 2004-11-21 18:36:19)
and its ChangeLog
libmoe-devel-ChangeLog.txt (32KB, 2004-11-21 18:36:19)
are also available.

The main functionalities are to calculate from a character encoded in multiple octet, a non-negative integer, which is called Universal Code Point (UCP) for convinience of description in this document, including complete information about coded character set containing the character and codepoint of the character in the set, and to reproduce the orignal octet sequence from the integer.


To build and install this library, you need C compiler and libraries conforming to ANSI standard. Further

I strongly recommend to use GNU C compiler and GNU Make.

If you build with the included Makefile, you need to tell to your dynamic linker, the directory (/usr/local/lib) in which the shared library is installed.

If you are installing on a Linux box for example, add the line

to the file /etc/ld.so.conf unless it already contains such line, and then issue the command

Acceptable encodings

This library can handle the following subset of the ISO 2022 escape sequences:

Further it can handle the following non-ISO 2022 encodings: Characters with these encodings can be inserted into ISO 2022 encoded character sequences with leading escape sequence
1/11 2/5 2/1 2/X 3/Y
and trailing
1/11 2/5 4/0
where X * 0x10 + Y are integers assigned to encodings by the library.

Universal Code Point

The library classifies the coded character set (CCS) into 6 categories

The characters in Unicode are assigned the same codepoints as in Unicode.

For a character in an other CCS, it is somewhat difficult to describe how to determine UCP in natural language. Roughly speaking, we order all the codepoints into one sequence, in the order of above categorization and final octet of escape sequences designating the CCS. The UCP is logical or of the index (staring with 0) in the big sequence and of 1U << 21.

Internal multiple octet encoding

The library has support for a state-less encoding scheme which we call "x-moe-internal" to include all UCP in one document:

UCP less than 0x80:
UCP greater than or equal to 0x80:
11xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx.

Any questions or comments about this page are greatly appreciated.

Almost all contents in this site are written by Kiyokazu SUTO (i.e. me) unless especially noted. I want to put all of them into the PUBLIC DOMAIN, even though some lawyers mention that it is impossible in my country.