[Summary view] [Print] [Text view]

   1  =head1 NAME
   2  
   3  perlunitut - Perl Unicode Tutorial
   4  
   5  =head1 DESCRIPTION
   6  
   7  The days of just flinging strings around are over. It's well established that
   8  modern programs need to be capable of communicating funny accented letters, and
   9  things like euro symbols. This means that programmers need new habits. It's
  10  easy to program Unicode capable software, but it does require discipline to do
  11  it right.
  12  
  13  There's a lot to know about character sets, and text encodings. It's probably
  14  best to spend a full day learning all this, but the basics can be learned in
  15  minutes. 
  16  
  17  These are not the very basics, though. It is assumed that you already
  18  know the difference between bytes and characters, and realise (and accept!)
  19  that there are many different character sets and encodings, and that your
  20  program has to be explicit about them. Recommended reading is "The Absolute
  21  Minimum Every Software Developer Absolutely, Positively Must Know About Unicode
  22  and Character Sets (No Excuses!)" by Joel Spolsky, at
  23  L<http://joelonsoftware.com/articles/Unicode.html>.
  24  
  25  This tutorial speaks in rather absolute terms, and provides only a limited view
  26  of the wealth of character string related features that Perl has to offer. For
  27  most projects, this information will probably suffice.
  28  
  29  =head2 Definitions
  30  
  31  It's important to set a few things straight first. This is the most important
  32  part of this tutorial. This view may conflict with other information that you
  33  may have found on the web, but that's mostly because many sources are wrong.
  34  
  35  You may have to re-read this entire section a few times...
  36  
  37  =head3 Unicode
  38  
  39  B<Unicode> is a character set with room for lots of characters. The ordinal
  40  value of a character is called a B<code point>. 
  41  
  42  There are many, many code points, but computers work with bytes, and a byte can
  43  have only 256 values. Unicode has many more characters, so you need a method
  44  to make these accessible.
  45  
  46  Unicode is encoded using several competing encodings, of which UTF-8 is the
  47  most used. In a Unicode encoding, multiple subsequent bytes can be used to
  48  store a single code point, or simply: character.
  49  
  50  =head3 UTF-8
  51  
  52  B<UTF-8> is a Unicode encoding. Many people think that Unicode and UTF-8 are
  53  the same thing, but they're not. There are more Unicode encodings, but much of
  54  the world has standardized on UTF-8. 
  55  
  56  UTF-8 treats the first 128 codepoints, 0..127, the same as ASCII. They take
  57  only one byte per character. All other characters are encoded as two or more
  58  (up to six) bytes using a complex scheme. Fortunately, Perl handles this for
  59  us, so we don't have to worry about this.
  60  
  61  =head3 Text strings (character strings)
  62  
  63  B<Text strings>, or B<character strings> are made of characters. Bytes are
  64  irrelevant here, and so are encodings. Each character is just that: the
  65  character.
  66  
  67  Text strings are also called B<Unicode strings>, because in Perl, every text
  68  string is a Unicode string.
  69  
  70  On a text string, you would do things like:
  71  
  72      $text =~ s/foo/bar/;
  73      if ($string =~ /^\d+$/) { ... }
  74      $text = ucfirst $text;
  75      my $character_count = length $text;
  76  
  77  The value of a character (C<ord>, C<chr>) is the corresponding Unicode code
  78  point.
  79  
  80  =head3 Binary strings (byte strings)
  81  
  82  B<Binary strings>, or B<byte strings> are made of bytes. Here, you don't have
  83  characters, just bytes. All communication with the outside world (anything
  84  outside of your current Perl process) is done in binary.
  85  
  86  On a binary string, you would do things like:
  87  
  88      my (@length_content) = unpack "(V/a)*", $binary;
  89      $binary =~ s/\x00\x0F/\xFF\xF0/;  # for the brave :)
  90      print {$fh} $binary;
  91      my $byte_count = length $binary;
  92  
  93  =head3 Encoding
  94  
  95  B<Encoding> (as a verb) is the conversion from I<text> to I<binary>. To encode,
  96  you have to supply the target encoding, for example C<iso-8859-1> or C<UTF-8>.
  97  Some encodings, like the C<iso-8859> ("latin") range, do not support the full
  98  Unicode standard; characters that can't be represented are lost in the
  99  conversion.
 100  
 101  =head3 Decoding
 102  
 103  B<Decoding> is the conversion from I<binary> to I<text>. To decode, you have to
 104  know what encoding was used during the encoding phase. And most of all, it must
 105  be something decodable. It doesn't make much sense to decode a PNG image into a
 106  text string.
 107  
 108  =head3 Internal format
 109  
 110  Perl has an B<internal format>, an encoding that it uses to encode text strings
 111  so it can store them in memory. All text strings are in this internal format.
 112  In fact, text strings are never in any other format!
 113  
 114  You shouldn't worry about what this format is, because conversion is
 115  automatically done when you decode or encode.
 116  
 117  =head2 Your new toolkit
 118  
 119  Add to your standard heading the following line:
 120  
 121      use Encode qw(encode decode);
 122  
 123  Or, if you're lazy, just:
 124  
 125      use Encode;
 126  
 127  =head2 I/O flow (the actual 5 minute tutorial)
 128  
 129  The typical input/output flow of a program is:
 130  
 131      1. Receive and decode
 132      2. Process
 133      3. Encode and output
 134  
 135  If your input is binary, and is supposed to remain binary, you shouldn't decode
 136  it to a text string, of course. But in all other cases, you should decode it.
 137  
 138  Decoding can't happen reliably if you don't know how the data was encoded. If
 139  you get to choose, it's a good idea to standardize on UTF-8.
 140  
 141      my $foo   = decode('UTF-8', get 'http://example.com/');
 142      my $bar   = decode('ISO-8859-1', readline STDIN);
 143      my $xyzzy = decode('Windows-1251', $cgi->param('foo'));
 144  
 145  Processing happens as you knew before. The only difference is that you're now
 146  using characters instead of bytes. That's very useful if you use things like
 147  C<substr>, or C<length>.
 148  
 149  It's important to realize that there are no bytes in a text string. Of course,
 150  Perl has its internal encoding to store the string in memory, but ignore that.
 151  If you have to do anything with the number of bytes, it's probably best to move
 152  that part to step 3, just after you've encoded the string. Then you know
 153  exactly how many bytes it will be in the destination string.
 154  
 155  The syntax for encoding text strings to binary strings is as simple as decoding:
 156  
 157      $body = encode('UTF-8', $body);
 158  
 159  If you needed to know the length of the string in bytes, now's the perfect time
 160  for that. Because C<$body> is now a byte string, C<length> will report the
 161  number of bytes, instead of the number of characters. The number of
 162  characters is no longer known, because characters only exist in text strings.
 163  
 164      my $byte_count = length $body;
 165  
 166  And if the protocol you're using supports a way of letting the recipient know
 167  which character encoding you used, please help the receiving end by using that
 168  feature! For example, E-mail and HTTP support MIME headers, so you can use the
 169  C<Content-Type> header. They can also have C<Content-Length> to indicate the
 170  number of I<bytes>, which is always a good idea to supply if the number is
 171  known.
 172  
 173      "Content-Type: text/plain; charset=UTF-8",
 174      "Content-Length: $byte_count"
 175  
 176  =head1 SUMMARY
 177  
 178  Decode everything you receive, encode everything you send out. (If it's text
 179  data.)
 180  
 181  =head1 Q and A (or FAQ)
 182  
 183  After reading this document, you ought to read L<perlunifaq> too. 
 184  
 185  =head1 ACKNOWLEDGEMENTS
 186  
 187  Thanks to Johan Vromans from Squirrel Consultancy. His UTF-8 rants during the
 188  Amsterdam Perl Mongers meetings got me interested and determined to find out
 189  how to use character encodings in Perl in ways that don't break easily.
 190  
 191  Thanks to Gerard Goossen from TTY. His presentation "UTF-8 in the wild" (Dutch
 192  Perl Workshop 2006) inspired me to publish my thoughts and write this tutorial.
 193  
 194  Thanks to the people who asked about this kind of stuff in several Perl IRC
 195  channels, and have constantly reminded me that a simpler explanation was
 196  needed.
 197  
 198  Thanks to the people who reviewed this document for me, before it went public.
 199  They are: Benjamin Smith, Jan-Pieter Cornet, Johan Vromans, Lukas Mai, Nathan
 200  Gray.
 201  
 202  =head1 AUTHOR
 203  
 204  Juerd Waalboer <#####@juerd.nl>
 205  
 206  =head1 SEE ALSO
 207  
 208  L<perlunifaq>, L<perlunicode>, L<perluniintro>, L<Encode>
 209