Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements.  See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License.  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

=== Probe length: 20B ===
                            N  | --- ML ablation --------------------------------- | --- Baselines ----------------------- |
Charset                        | Stat R%   S%  | +ISO R%   S%  | +CJK R%   S%  | All  R%   S%  | ICU4J R%   S%  | juniv R%   S%  |
----------------------------------------------------------------------------------------------------------------------------------
Big5-HKSCS               5000  |  98.0  98.0  |  97.9  97.9  |  97.9  97.9  |  97.9  97.9  |   0.0   9.8  |   0.0  37.9  |
EUC-JP                   5000  |  54.5  54.5  |  54.6  54.6  |  54.7  54.7  |  54.7  54.7  |   0.0   0.0  |  69.7  69.7  |
EUC-KR                   5000  |  66.4  66.4  |  67.6  67.6  |  67.6  67.6  |  67.6  67.6  |   0.0   0.0  |  78.3  78.3  |
GB18030                  5000  |  29.8  29.8  |  30.7  30.7  |  30.8  30.8  |  30.8  30.8  |   0.1   0.1  |  40.6  40.6  |
IBM1047                 10000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0  86.8  |   0.0   0.0  |
IBM420-ltr              10000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0  80.9  |   0.0   0.0  |
IBM420-rtl              10000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0  84.2  |   0.0   0.0  |
IBM424-ltr               6195  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0  69.1  |   0.0   0.0  |
IBM424-rtl               4717  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0  69.6  |   0.0   0.0  |
IBM437                   7516  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |
IBM500                  10000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |  87.7  87.7  |   0.0   0.0  |
IBM850                  10000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |
IBM852                  10000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |
IBM855                  10000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |  90.7  90.7  |
IBM866                   8442  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |  43.0  43.0  |  93.2  93.2  |
ISO-2022-CN              5000  |   0.0   0.0  |  94.6  94.6  |  94.6  94.6  |  94.6  94.6  |  87.7  87.7  |   0.0   0.0  |
ISO-2022-JP              5000  |   0.0   0.0  |  94.0  94.0  |  94.0  94.0  |  94.0  94.0  |  77.9  77.9  |  93.9  93.9  |
ISO-2022-KR              5000  |   0.0   0.0  |  94.4  94.4  |  94.4  94.4  |  94.4  94.4  |  92.6  92.6  |  94.4  94.4  |
ISO-8859-16             10000  |  38.3  38.3  |  31.6  31.6  |  31.7  31.7  |  31.7  31.7  |   0.0   0.0  |   0.0   0.0  |
ISO-8859-3               5195  |  10.0  10.0  |  10.3  10.3  |  10.5  10.5  |  10.5  10.5  |   0.0   0.0  |   0.0   0.0  |
KOI8-R                   8411  |  57.0  66.6  |  57.0  66.6  |  57.1  66.7  |  57.1  66.7  |  59.7  59.7  |  94.6  94.6  |
KOI8-U                   5921  |  71.2  79.3  |  71.3  79.5  |  71.3  79.6  |  71.3  79.6  |   0.0  56.6  |   0.0  95.4  |
Shift_JIS                5000  |  84.3  84.3  |  86.6  86.6  |  86.6  86.6  |  86.6  86.6  |   0.0   0.0  |  71.6  71.6  |
US-ASCII                 5000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   2.3  |   0.0   0.0  |
UTF-16-BE                5000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |  68.7  68.7  |   0.0   0.0  |
UTF-16-LE                5000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |  69.4  69.4  |   0.0   0.0  |
UTF-32-BE                5000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  | 100.0 100.0  |   0.0   0.0  |
UTF-32-LE                5000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  | 100.0 100.0  |   0.0   0.0  |
UTF-8                    5000  |  61.4  61.4  |  61.1  61.1  |  61.5  61.5  |  61.5  61.5  |  77.6  77.6  |  78.2  78.2  |
windows-1250            10000  |   4.7   4.7  |   4.8   4.8  |   4.9   4.9  |   4.9   4.9  |   9.3  48.3  |   0.0   0.0  |
windows-1251            10000  |  82.4  82.4  |  82.5  82.5  |  82.6  82.6  |  82.6  82.6  |  56.1  56.4  |  71.4  71.5  |
windows-1252            10000  |   2.6   2.6  |  50.0  50.0  |  50.1  50.1  |  50.1  50.1  |   4.5  70.0  |   0.0  98.7  |
windows-1253            10000  |  55.2  55.2  |  55.4  55.4  |  55.5  55.5  |  55.5  55.5  |   1.9  63.3  |   0.1  80.8  |
windows-1254            10000  |  35.8  35.8  |  35.8  35.8  |  35.9  35.9  |  35.9  35.9  |   4.9  52.1  |   0.0   0.0  |
windows-1255            10000  |  86.7  86.7  |  86.7  86.7  |  86.7  86.7  |  86.7  86.7  |   3.5  27.2  |  93.9  96.2  |
windows-1256            10000  |  94.1  94.1  |  94.2  94.2  |  94.2  94.2  |  94.2  94.2  |  70.2  78.6  |   0.0   0.0  |
windows-1257            10000  |  18.8  18.8  |  17.0  17.0  |  17.1  17.1  |  17.1  17.1  |   0.0   0.0  |   0.0   0.0  |
windows-1258            10000  |  63.0  63.0  |  63.2  63.2  |  63.3  63.3  |  63.3  63.3  |   0.0   0.0  |   0.0   0.0  |
windows-874             10000  |  79.3  79.3  |  80.6  80.6  |  81.4  81.4  |  81.4  81.4  |   0.0   0.0  |   0.0   0.0  |
x-EUC-TW                 5000  |  99.4  99.4  |  98.3  98.3  |  98.3  98.3  |  98.3  98.3  |   0.0   0.0  |  38.7  38.7  |
x-ISO-2022-CN-CNS        5000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |
x-MacRoman              10000  |   6.1   6.1  |   6.2   6.2  |   6.3   6.3  |   6.3   6.3  |   0.0   0.0  |   0.0   0.0  |
x-mac-cyrillic          10000  |  68.8  68.8  |  69.2  69.2  |  69.3  69.3  |  69.3  69.3  |   0.0   0.0  |  49.3  49.3  |
------------------------------------------------------------------------------------------------------------------------
OVERALL                 326397  |  30.0  30.4  |  35.6  36.0  |  35.7  36.1  |  35.7  36.1  |  20.3  39.0  |  22.9  30.7  |
  Stat=model only | +ISO=+C1-correction | +CJK=+grammar | All=ML+rules | R%=strict | S%=soft
  µs/sample                    |        10.0  |         7.3  |         7.4  |         7.0  |        11.9  |         3.3  |

=== Probe length: 50B ===
                            N  | --- ML ablation --------------------------------- | --- Baselines ----------------------- |
Charset                        | Stat R%   S%  | +ISO R%   S%  | +CJK R%   S%  | All  R%   S%  | ICU4J R%   S%  | juniv R%   S%  |
----------------------------------------------------------------------------------------------------------------------------------
Big5-HKSCS               5000  |  99.4  99.4  |  99.3  99.3  |  99.3  99.3  |  99.3  99.3  |   0.0  95.9  |   0.0  69.1  |
EUC-JP                   5000  |  88.9  88.9  |  89.0  89.0  |  89.0  89.0  |  89.0  89.0  |  86.7  86.7  |  90.2  90.2  |
EUC-KR                   5000  |  92.7  92.7  |  93.3  93.3  |  93.3  93.3  |  93.3  93.3  |  89.4  89.4  |  96.0  96.0  |
GB18030                  5000  |  65.0  65.0  |  66.4  66.4  |  66.5  66.5  |  66.5  66.5  |  71.8  71.8  |  83.2  83.2  |
IBM1047                 10000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0  80.2  |   0.0   0.0  |
IBM420-ltr              10000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0  93.1  |   0.0   0.0  |
IBM420-rtl              10000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0  94.7  |   0.0   0.0  |
IBM424-ltr               6195  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0  86.1  |   0.0   0.0  |
IBM424-rtl               4717  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0  87.6  |   0.0   0.0  |
IBM437                   7516  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |
IBM500                  10000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |  80.2  80.2  |   0.0   0.0  |
IBM850                  10000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |
IBM852                  10000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |
IBM855                  10000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |  97.0  97.0  |
IBM866                   8442  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |  62.0  62.0  |  98.1  98.1  |
ISO-2022-CN              5000  |   0.0   0.0  |  98.6  98.6  |  98.6  98.6  |  98.6  98.6  |  98.5  98.5  |   0.0   0.0  |
ISO-2022-JP              5000  |   0.0   0.0  |  97.4  97.4  |  97.4  97.4  |  97.4  97.4  |  93.9  93.9  |  97.3  97.3  |
ISO-2022-KR              5000  |   0.0   0.0  |  98.9  98.9  |  98.9  98.9  |  98.9  98.9  |  98.8  98.8  |  98.9  98.9  |
ISO-8859-16             10000  |  89.3  89.3  |  80.5  80.5  |  80.7  80.7  |  80.7  80.7  |   0.0   0.0  |   0.0   0.0  |
ISO-8859-3               5195  |  50.8  50.8  |  49.8  49.8  |  49.9  49.9  |  49.9  49.9  |   0.0   0.0  |   0.0   0.0  |
KOI8-R                   8411  |  86.2  93.6  |  86.2  93.7  |  86.2  93.7  |  86.2  93.7  |  77.6  77.6  |  98.6  98.6  |
KOI8-U                   5921  |  92.7  97.4  |  92.7  97.4  |  92.7  97.5  |  92.7  97.5  |   0.0  74.8  |   0.0  98.2  |
Shift_JIS                5000  |  95.3  95.3  |  96.1  96.1  |  96.1  96.1  |  96.1  96.1  |  88.5  88.5  |  94.5  94.5  |
US-ASCII                 5000  |   0.0   0.5  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.3  |   0.0   0.0  |
UTF-16-BE                5000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |  85.7  85.7  |   0.0   0.0  |
UTF-16-LE                5000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |  87.6  87.6  |   0.0   0.0  |
UTF-32-BE                5000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  | 100.0 100.0  |   0.0   0.0  |
UTF-32-LE                5000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  | 100.0 100.0  |   0.0   0.0  |
UTF-8                    5000  |  84.4  84.4  |  88.3  88.3  |  88.5  88.5  |  88.5  88.5  |  92.4  92.4  |  93.3  93.3  |
windows-1250            10000  |  31.3  31.3  |  32.3  32.3  |  32.9  32.9  |  32.9  32.9  |  22.3  56.9  |   0.0   0.0  |
windows-1251            10000  |  95.8  95.8  |  95.8  95.8  |  95.9  95.9  |  95.9  95.9  |  76.0  76.1  |  80.9  80.9  |
windows-1252            10000  |  20.8  20.8  |  39.1  39.1  |  39.4  39.4  |  39.4  39.4  |  12.8  86.6  |   0.0  99.1  |
windows-1253            10000  |  90.2  90.2  |  90.3  90.3  |  90.3  90.3  |  90.3  90.3  |   8.2  86.8  |   0.1  91.5  |
windows-1254            10000  |  80.1  80.1  |  80.0  80.0  |  80.0  80.0  |  80.0  80.0  |  12.3  73.1  |   0.0   0.0  |
windows-1255            10000  |  97.8  97.8  |  97.8  97.8  |  97.8  97.8  |  97.8  97.8  |   9.4  40.2  |  98.1  99.4  |
windows-1256            10000  |  99.1  99.1  |  99.2  99.2  |  99.2  99.2  |  99.2  99.2  |  88.1  91.5  |   0.0   0.0  |
windows-1257            10000  |  72.0  72.0  |  67.2  67.2  |  67.3  67.3  |  67.3  67.3  |   0.0   0.0  |   0.0   0.0  |
windows-1258            10000  |  96.1  96.1  |  96.2  96.2  |  96.3  96.3  |  96.3  96.3  |   0.0   0.0  |   0.0   0.0  |
windows-874             10000  |  95.3  95.3  |  95.8  95.8  |  96.0  96.0  |  96.0  96.0  |   0.0   0.0  |   0.0   0.0  |
x-EUC-TW                 5000  |  99.5  99.5  |  99.3  99.3  |  99.3  99.3  |  99.3  99.3  |   0.0   0.0  |  67.1  67.1  |
x-ISO-2022-CN-CNS        5000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |
x-MacRoman              10000  |  58.5  58.5  |  56.4  56.4  |  56.6  56.6  |  56.6  56.6  |   0.0   0.0  |   0.0   0.0  |
x-mac-cyrillic          10000  |  81.0  81.0  |  81.2  81.2  |  81.2  81.2  |  81.2  81.2  |   0.0   0.0  |  62.3  62.3  |
------------------------------------------------------------------------------------------------------------------------
OVERALL                 326397  |  45.2  45.4  |  49.9  50.2  |  49.9  50.2  |  49.9  50.2  |  29.8  52.4  |  26.5  35.2  |
  Stat=model only | +ISO=+C1-correction | +CJK=+grammar | All=ML+rules | R%=strict | S%=soft
  µs/sample                    |        10.7  |         8.2  |         8.1  |         8.1  |        21.6  |         4.1  |

=== Probe length: 100B ===
                            N  | --- ML ablation --------------------------------- | --- Baselines ----------------------- |
Charset                        | Stat R%   S%  | +ISO R%   S%  | +CJK R%   S%  | All  R%   S%  | ICU4J R%   S%  | juniv R%   S%  |
----------------------------------------------------------------------------------------------------------------------------------
Big5-HKSCS               5000  |  99.7  99.7  |  99.8  99.8  |  99.8  99.8  |  99.8  99.8  |   0.0  97.6  |   0.0  69.7  |
EUC-JP                   5000  |  95.8  95.8  |  95.8  95.8  |  95.8  95.8  |  95.8  95.8  |  97.0  97.0  |  95.7  95.7  |
EUC-KR                   5000  |  95.3  95.3  |  95.4  95.4  |  95.4  95.4  |  95.4  95.4  |  98.4  98.4  |  99.1  99.1  |
GB18030                  5000  |  85.1  85.1  |  85.9  85.9  |  85.9  85.9  |  85.9  85.9  |  95.9  95.9  |  97.9  97.9  |
IBM1047                 10000  |  49.5  94.7  |  49.6  94.9  |  49.6  94.9  |  49.6  94.9  |   0.0  83.2  |   0.0   0.0  |
IBM420-ltr              10000  |  92.6  95.0  |  92.6  95.0  |  92.6  95.0  |  92.6  95.0  |   0.0  95.0  |   0.0   0.0  |
IBM420-rtl              10000  |  94.1  94.6  |  94.1  94.6  |  94.1  94.6  |  94.1  94.6  |   0.0  96.3  |   0.0   0.0  |
IBM424-ltr               6195  |  91.6  91.8  |  74.8  75.0  |  74.8  75.0  |  74.8  75.0  |   0.0  92.4  |   0.0   0.0  |
IBM424-rtl               4717  |  93.0  95.1  |  74.2  75.5  |  74.2  75.5  |  74.2  75.5  |   0.0  89.4  |   0.0   0.0  |
IBM437                   7516  |   0.0  79.0  |   0.0  77.8  |   0.0  77.8  |   0.0  77.8  |   0.0   0.0  |   0.0   0.0  |
IBM500                  10000  |  56.0  94.9  |  56.1  95.1  |  56.1  95.1  |  56.1  95.1  |  83.5  83.5  |   0.0   0.0  |
IBM850                  10000  |  79.8  79.8  |  79.5  79.5  |  79.5  79.5  |  79.5  79.5  |   0.0   0.0  |   0.0   0.0  |
IBM852                  10000  |  77.4  77.4  |  78.1  78.1  |  78.1  78.1  |  78.1  78.1  |   0.0   0.0  |   0.0   0.0  |
IBM855                  10000  |  94.6  94.6  |  94.7  94.7  |  94.7  94.7  |  94.7  94.7  |   0.0   0.0  |  99.0  99.0  |
IBM866                   8442  |  95.9  95.9  |  96.0  96.0  |  96.0  96.0  |  96.0  96.0  |  78.8  78.8  |  99.3  99.3  |
ISO-2022-CN              5000  |   0.0   0.0  |  99.1  99.1  |  99.1  99.1  |  99.1  99.1  |  99.0  99.0  |   0.0   0.0  |
ISO-2022-JP              5000  |   0.0   0.0  |  99.1  99.1  |  99.1  99.1  |  99.1  99.1  |  98.8  98.8  |  99.1  99.1  |
ISO-2022-KR              5000  |   0.0   0.0  |  99.6  99.6  |  99.6  99.6  |  99.6  99.6  |  99.6  99.6  |  99.6  99.6  |
ISO-8859-16             10000  |  92.0  92.0  |  89.1  89.1  |  89.1  89.1  |  89.1  89.1  |   0.0   0.0  |   0.0   0.0  |
ISO-8859-3               5195  |  78.4  78.4  |  78.5  78.5  |  78.5  78.5  |  78.5  78.5  |   0.0   0.0  |   0.0   0.0  |
KOI8-R                   8411  |  95.7  99.0  |  95.7  99.0  |  95.7  99.0  |  95.7  99.0  |  90.5  90.5  |  99.4  99.4  |
KOI8-U                   5921  |  96.9  99.5  |  96.9  99.5  |  96.9  99.5  |  96.9  99.5  |   0.0  85.4  |   0.0  98.9  |
Shift_JIS                5000  |  97.7  97.7  |  98.0  98.0  |  98.0  98.0  |  98.0  98.0  |  97.8  97.8  |  98.7  98.7  |
US-ASCII                 5000  |   0.0   0.8  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |
UTF-16-BE                5000  |  94.0  94.1  |  93.9  93.9  |  93.9  93.9  |  93.9  93.9  |  85.5  85.5  |   0.0   0.0  |
UTF-16-LE                5000  |  94.4  94.4  |  94.4  94.4  |  94.4  94.4  |  94.4  94.4  |  87.0  87.0  |   0.0   0.0  |
UTF-32-BE                5000  |  94.9  94.9  |  94.7  94.7  |  94.7  94.7  |  94.7  94.7  | 100.0 100.0  |   0.0   0.0  |
UTF-32-LE                5000  |  95.5  95.5  |  95.5  95.5  |  95.5  95.5  |  95.5  95.5  | 100.0 100.0  |   0.0   0.0  |
UTF-8                    5000  |  90.6  90.6  |  93.2  93.2  |  93.2  93.2  |  93.2  93.2  |  97.3  97.3  |  97.2  97.2  |
windows-1250            10000  |  55.3  55.3  |  56.4  56.4  |  56.5  56.5  |  56.5  56.5  |  34.4  62.5  |   0.0   0.0  |
windows-1251            10000  |  98.8  98.8  |  98.8  98.8  |  98.8  98.8  |  98.8  98.8  |  85.7  85.7  |  86.2  86.2  |
windows-1252            10000  |  34.4  34.4  |  40.0  40.0  |  40.4  40.4  |  40.4  40.4  |  21.3  91.4  |   0.0  99.4  |
windows-1253            10000  |  97.1  97.1  |  97.1  97.1  |  97.1  97.1  |  97.1  97.1  |  15.4  93.7  |   0.2  95.3  |
windows-1254            10000  |  93.1  93.1  |  93.0  93.0  |  93.0  93.0  |  93.0  93.0  |  21.2  83.8  |   0.0   0.0  |
windows-1255            10000  |  99.0  99.0  |  99.0  99.0  |  99.0  99.0  |  99.0  99.0  |  14.6  46.8  |  99.0  99.9  |
windows-1256            10000  |  99.6  99.6  |  99.6  99.6  |  99.6  99.6  |  99.6  99.6  |  94.6  96.1  |   0.0   0.0  |
windows-1257            10000  |  87.1  87.1  |  85.7  85.7  |  85.7  85.7  |  85.7  85.7  |   0.0   0.0  |   0.0   0.0  |
windows-1258            10000  |  97.0  97.0  |  97.0  97.0  |  97.0  97.0  |  97.0  97.0  |   0.0   0.0  |   0.0   0.0  |
windows-874             10000  |  97.7  97.7  |  97.7  97.7  |  97.7  97.7  |  97.7  97.7  |   0.0   0.0  |   0.0   0.0  |
x-EUC-TW                 5000  |  99.7  99.7  |  99.8  99.8  |  99.8  99.8  |  99.8  99.8  |   0.0   0.0  |  67.7  67.7  |
x-ISO-2022-CN-CNS        5000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |
x-MacRoman              10000  |  80.0  80.0  |  79.2  79.2  |  79.2  79.2  |  79.2  79.2  |   0.0   0.0  |   0.0   0.0  |
x-mac-cyrillic          10000  |  88.6  88.6  |  88.7  88.7  |  88.7  88.7  |  88.7  88.7  |   0.0   0.0  |  70.1  70.1  |
------------------------------------------------------------------------------------------------------------------------
OVERALL                 326397  |  78.0  82.6  |  82.1  86.7  |  82.1  86.7  |  82.1  86.7  |  33.4  56.3  |  27.6  36.4  |
  Stat=model only | +ISO=+C1-correction | +CJK=+grammar | All=ML+rules | R%=strict | S%=soft
  µs/sample                    |        10.8  |         8.2  |         8.0  |         7.9  |        36.3  |         5.7  |

=== Probe length: 200B ===
                            N  | --- ML ablation --------------------------------- | --- Baselines ----------------------- |
Charset                        | Stat R%   S%  | +ISO R%   S%  | +CJK R%   S%  | All  R%   S%  | ICU4J R%   S%  | juniv R%   S%  |
----------------------------------------------------------------------------------------------------------------------------------
Big5-HKSCS               5000  |  99.8  99.8  |  99.9  99.9  |  99.9  99.9  |  99.9  99.9  |   0.0  97.8  |   0.0  69.7  |
EUC-JP                   5000  |  98.4  98.4  |  98.4  98.4  |  98.4  98.4  |  98.4  98.4  |  99.2  99.2  |  97.9  97.9  |
EUC-KR                   5000  |  99.1  99.1  |  99.1  99.1  |  99.1  99.1  |  99.1  99.1  |  99.8  99.8  |  99.8  99.8  |
GB18030                  5000  |  94.0  94.0  |  94.5  94.5  |  94.5  94.5  |  94.5  94.5  |  98.4  98.4  |  99.4  99.4  |
IBM1047                 10000  |  52.7  95.7  |  52.8  95.8  |  52.8  95.8  |  52.8  95.8  |   0.0  86.8  |   0.0   0.0  |
IBM420-ltr              10000  |  95.3  96.1  |  95.3  96.1  |  95.3  96.1  |  95.3  96.1  |   0.0  96.8  |   0.0   0.0  |
IBM420-rtl              10000  |  95.3  95.4  |  95.3  95.4  |  95.3  95.4  |  95.3  95.4  |   0.0  97.3  |   0.0   0.0  |
IBM424-ltr               6195  |  94.9  94.9  |  87.8  87.8  |  87.8  87.8  |  87.8  87.8  |   0.0  93.4  |   0.0   0.0  |
IBM424-rtl               4717  |  95.6  96.0  |  86.7  87.2  |  86.7  87.2  |  86.7  87.2  |   0.0  87.3  |   0.0   0.0  |
IBM437                   7516  |   0.0  92.1  |   0.0  92.0  |   0.0  92.0  |   0.0  92.0  |   0.0   0.0  |   0.0   0.0  |
IBM500                  10000  |  56.0  96.0  |  56.0  96.0  |  56.0  96.0  |  56.0  96.0  |  86.8  86.8  |   0.0   0.0  |
IBM850                  10000  |  92.2  92.2  |  92.3  92.3  |  92.3  92.3  |  92.3  92.3  |   0.0   0.0  |   0.0   0.0  |
IBM852                  10000  |  90.0  90.0  |  90.7  90.7  |  90.7  90.7  |  90.7  90.7  |   0.0   0.0  |   0.0   0.0  |
IBM855                  10000  |  95.7  95.7  |  95.7  95.7  |  95.7  95.7  |  95.7  95.7  |   0.0   0.0  |  99.8  99.8  |
IBM866                   8442  |  96.2  96.2  |  96.2  96.2  |  96.2  96.2  |  96.2  96.2  |  89.5  89.5  |  99.7  99.7  |
ISO-2022-CN              5000  |   0.0   0.0  |  99.5  99.5  |  99.5  99.5  |  99.5  99.5  |  99.4  99.4  |   0.0   0.0  |
ISO-2022-JP              5000  |   0.0   0.0  |  99.6  99.6  |  99.6  99.6  |  99.6  99.6  |  99.4  99.4  |  99.6  99.6  |
ISO-2022-KR              5000  |   0.0   0.0  |  99.8  99.8  |  99.8  99.8  |  99.8  99.8  |  99.8  99.8  |  99.8  99.8  |
ISO-8859-16             10000  |  95.3  95.3  |  94.9  94.9  |  94.9  94.9  |  94.9  94.9  |   0.0   0.0  |   0.0   0.0  |
ISO-8859-3               5195  |  93.9  93.9  |  94.3  94.3  |  94.3  94.3  |  94.3  94.3  |   0.0   0.0  |   0.0   0.0  |
KOI8-R                   8411  |  98.6  99.8  |  98.6  99.8  |  98.6  99.8  |  98.6  99.8  |  95.8  95.8  |  99.7  99.7  |
KOI8-U                   5921  |  98.9  99.9  |  98.9  99.9  |  98.9  99.9  |  98.9  99.9  |   0.0  92.5  |   0.0  99.3  |
Shift_JIS                5000  |  99.3  99.3  |  99.4  99.4  |  99.4  99.4  |  99.4  99.4  |  99.7  99.7  |  99.7  99.7  |
US-ASCII                 5000  |   0.0   0.5  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |
UTF-16-BE                5000  |  94.5  94.5  |  94.4  94.4  |  94.4  94.4  |  94.4  94.4  |  84.9  84.9  |   0.0   0.0  |
UTF-16-LE                5000  |  94.4  94.4  |  94.4  94.4  |  94.4  94.4  |  94.4  94.4  |  86.8  86.8  |   0.0   0.0  |
UTF-32-BE                5000  |  95.2  95.2  |  95.0  95.0  |  95.0  95.0  |  95.0  95.0  | 100.0 100.0  |   0.0   0.0  |
UTF-32-LE                5000  |  95.5  95.5  |  95.5  95.5  |  95.5  95.5  |  95.5  95.5  | 100.0 100.0  |   0.0   0.0  |
UTF-8                    5000  |  96.1  96.1  |  96.7  96.7  |  96.7  96.7  |  96.7  96.7  |  99.5  99.5  |  98.1  98.1  |
windows-1250            10000  |  73.6  73.6  |  74.3  74.3  |  74.4  74.4  |  74.4  74.4  |  49.3  70.3  |   0.0   0.0  |
windows-1251            10000  |  99.8  99.8  |  99.8  99.8  |  99.8  99.8  |  99.8  99.8  |  92.6  92.6  |  89.1  89.1  |
windows-1252            10000  |  47.7  47.7  |  48.5  48.5  |  48.7  48.7  |  48.7  48.7  |  28.5  94.0  |   0.0  99.5  |
windows-1253            10000  |  99.4  99.4  |  99.4  99.4  |  99.4  99.4  |  99.4  99.4  |  22.0  97.2  |   0.3  97.6  |
windows-1254            10000  |  98.4  98.4  |  98.4  98.4  |  98.4  98.4  |  98.4  98.4  |  33.4  93.5  |   0.0   0.0  |
windows-1255            10000  |  99.8  99.8  |  99.8  99.8  |  99.8  99.8  |  99.8  99.8  |  20.2  57.9  |  99.7  99.9  |
windows-1256            10000  |  99.9  99.9  |  99.9  99.9  |  99.9  99.9  |  99.9  99.9  |  97.1  98.2  |   0.0   0.0  |
windows-1257            10000  |  95.1  95.1  |  95.1  95.1  |  95.1  95.1  |  95.1  95.1  |   0.0   0.0  |   0.0   0.0  |
windows-1258            10000  |  99.0  99.0  |  99.0  99.0  |  99.0  99.0  |  99.0  99.0  |   0.0   0.0  |   0.0   0.0  |
windows-874             10000  |  99.4  99.4  |  99.4  99.4  |  99.4  99.4  |  99.4  99.4  |   0.0   0.0  |   0.0   0.0  |
x-EUC-TW                 5000  |  99.9  99.9  |  99.9  99.9  |  99.9  99.9  |  99.9  99.9  |   0.0   0.0  |  67.7  67.7  |
x-ISO-2022-CN-CNS        5000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |
x-MacRoman              10000  |  93.5  93.5  |  93.5  93.5  |  93.5  93.5  |  93.5  93.5  |   0.0   0.0  |   0.0   0.0  |
x-mac-cyrillic          10000  |  93.5  93.5  |  93.5  93.5  |  93.5  93.5  |  93.5  93.5  |   0.0   0.0  |  74.9  74.9  |
------------------------------------------------------------------------------------------------------------------------
OVERALL                 326397  |  82.1  86.9  |  86.5  91.2  |  86.5  91.3  |  86.5  91.3  |  35.8  58.6  |  28.0  36.9  |
  Stat=model only | +ISO=+C1-correction | +CJK=+grammar | All=ML+rules | R%=strict | S%=soft
  µs/sample                    |        12.4  |         9.2  |         8.7  |         8.9  |        60.7  |         8.2  |

=== Probe length: full ===
                            N  | --- ML ablation --------------------------------- | --- Baselines ----------------------- |
Charset                        | Stat R%   S%  | +ISO R%   S%  | +CJK R%   S%  | All  R%   S%  | ICU4J R%   S%  | juniv R%   S%  |
----------------------------------------------------------------------------------------------------------------------------------
Big5-HKSCS               5000  |  99.9  99.9  |  99.9  99.9  |  99.9  99.9  |  99.9  99.9  |   0.0  97.8  |   0.0  69.7  |
EUC-JP                   5000  |  99.6  99.6  |  99.6  99.6  |  99.6  99.6  |  99.6  99.6  |  99.9  99.9  |  99.0  99.0  |
EUC-KR                   5000  |  99.7  99.7  |  99.7  99.7  |  99.7  99.7  |  99.7  99.7  | 100.0 100.0  |  99.9  99.9  |
GB18030                  5000  |  97.1  97.1  |  97.5  97.5  |  97.5  97.5  |  97.5  97.5  |  99.4  99.4  |  99.7  99.7  |
IBM1047                 10000  |  59.1  96.0  |  59.1  96.0  |  59.1  96.0  |  59.1  96.0  |   0.0  90.6  |   0.0   0.0  |
IBM420-ltr              10000  |  96.0  96.2  |  96.0  96.2  |  96.0  96.2  |  96.0  96.2  |   0.0  97.8  |   0.0   0.0  |
IBM420-rtl              10000  |  95.6  95.7  |  95.6  95.7  |  95.6  95.7  |  95.6  95.7  |   0.0  98.1  |   0.0   0.0  |
IBM424-ltr               6195  |  95.6  95.6  |  95.5  95.6  |  95.5  95.6  |  95.5  95.6  |   0.0  93.1  |   0.0   0.0  |
IBM424-rtl               4717  |  96.2  96.3  |  96.2  96.3  |  96.2  96.3  |  96.2  96.3  |   0.0  87.3  |   0.0   0.0  |
IBM437                   7516  |   0.0  95.9  |   0.0  96.0  |   0.0  96.0  |   0.0  96.0  |   0.0   0.0  |   0.0   0.0  |
IBM500                  10000  |  52.6  96.3  |  52.6  96.3  |  52.6  96.3  |  52.6  96.3  |  90.6  90.6  |   0.0   0.0  |
IBM850                  10000  |  95.8  95.8  |  95.9  95.9  |  95.9  95.9  |  95.9  95.9  |   0.0   0.0  |   0.0   0.0  |
IBM852                  10000  |  95.2  95.2  |  95.5  95.5  |  95.5  95.5  |  95.5  95.5  |   0.0   0.0  |   0.0   0.0  |
IBM855                  10000  |  95.9  95.9  |  95.9  95.9  |  95.9  95.9  |  95.9  95.9  |   0.0   0.0  |  99.9  99.9  |
IBM866                   8442  |  96.2  96.2  |  96.2  96.2  |  96.2  96.2  |  96.2  96.2  |  95.8  95.8  |  99.8  99.8  |
ISO-2022-CN              5000  |   0.0   0.0  |  99.8  99.8  |  99.8  99.8  |  99.8  99.8  |  99.7  99.7  |   0.0   0.0  |
ISO-2022-JP              5000  |   0.0   0.0  |  99.8  99.8  |  99.8  99.8  |  99.8  99.8  |  99.7  99.7  |  99.8  99.8  |
ISO-2022-KR              5000  |   0.0   0.0  |  99.9  99.9  |  99.9  99.9  |  99.9  99.9  |  99.9  99.9  |  99.9  99.9  |
ISO-8859-16             10000  |  98.5  98.5  |  98.5  98.5  |  98.5  98.5  |  98.5  98.5  |   0.0   0.0  |   0.0   0.0  |
ISO-8859-3               5195  |  98.4  98.4  |  98.7  98.7  |  98.7  98.7  |  98.7  98.7  |   0.0   0.0  |   0.0   0.0  |
KOI8-R                   8411  |  99.4  99.8  |  99.4  99.8  |  99.4  99.8  |  99.4  99.8  |  98.0  98.0  |  99.8  99.8  |
KOI8-U                   5921  |  99.5  99.9  |  99.5  99.9  |  99.5  99.9  |  99.5  99.9  |   0.0  96.5  |   0.0  99.8  |
Shift_JIS                5000  |  99.9  99.9  |  99.9  99.9  |  99.9  99.9  |  99.9  99.9  | 100.0 100.0  |  99.9  99.9  |
US-ASCII                 5000  |   0.0   0.2  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |
UTF-16-BE                5000  |  94.7  94.7  |  94.7  94.7  |  94.7  94.7  |  94.7  94.7  |  84.6  84.6  |   0.0   0.0  |
UTF-16-LE                5000  |  94.5  94.5  |  94.5  94.5  |  94.5  94.5  |  94.5  94.5  |  86.6  86.6  |   0.0   0.0  |
UTF-32-BE                5000  |  95.2  95.2  |  95.2  95.2  |  95.2  95.2  |  95.2  95.2  | 100.0 100.0  |   0.0   0.0  |
UTF-32-LE                5000  |  95.5  95.5  |  95.5  95.5  |  95.5  95.5  |  95.5  95.5  | 100.0 100.0  |   0.0   0.0  |
UTF-8                    5000  |  98.8  98.8  |  98.0  98.0  |  98.1  98.1  |  98.1  98.1  |  99.9  99.9  |  97.7  97.7  |
windows-1250            10000  |  83.5  83.5  |  83.7  83.7  |  83.8  83.8  |  83.8  83.8  |  65.4  78.8  |   0.0   0.0  |
windows-1251            10000  |  99.9  99.9  |  99.9  99.9  |  99.9  99.9  |  99.9  99.9  |  96.4  96.4  |  90.7  90.7  |
windows-1252            10000  |  63.8  63.8  |  64.0  64.0  |  64.0  64.0  |  64.0  64.0  |  41.4  96.2  |   0.0  99.6  |
windows-1253            10000  |  99.8  99.8  |  99.8  99.8  |  99.8  99.8  |  99.8  99.8  |  33.1  98.9  |   0.4  99.0  |
windows-1254            10000  |  99.3  99.3  |  99.3  99.3  |  99.3  99.3  |  99.3  99.3  |  48.7  97.6  |   0.0   0.0  |
windows-1255            10000  |  99.9  99.9  |  99.9  99.9  |  99.9  99.9  |  99.9  99.9  |  30.4  71.6  |  99.9 100.0  |
windows-1256            10000  |  99.9  99.9  |  99.9  99.9  |  99.9  99.9  |  99.9  99.9  |  98.1  99.0  |   0.0   0.0  |
windows-1257            10000  |  98.5  98.5  |  98.6  98.6  |  98.6  98.6  |  98.6  98.6  |   0.0   0.0  |   0.0   0.0  |
windows-1258            10000  |  99.6  99.6  |  99.6  99.6  |  99.6  99.6  |  99.6  99.6  |   0.0   0.0  |   0.0   0.0  |
windows-874             10000  |  99.7  99.7  |  99.8  99.8  |  99.8  99.8  |  99.8  99.8  |   0.0   0.0  |   0.0   0.0  |
x-EUC-TW                 5000  |  99.9  99.9  | 100.0 100.0  | 100.0 100.0  | 100.0 100.0  |   0.0   0.0  |  67.7  67.7  |
x-ISO-2022-CN-CNS        5000  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |   0.0   0.0  |
x-MacRoman              10000  |  98.5  98.5  |  98.6  98.6  |  98.7  98.7  |  98.7  98.7  |   0.0   0.0  |   0.0   0.0  |
x-mac-cyrillic          10000  |  96.9  96.9  |  96.9  96.9  |  96.9  96.9  |  96.9  96.9  |   0.0   0.0  |  75.0  75.0  |
------------------------------------------------------------------------------------------------------------------------
OVERALL                 326397  |  84.1  88.8  |  88.7  93.4  |  88.7  93.4  |  88.7  93.4  |  38.4  60.3  |  28.1  37.0  |
  Stat=model only | +ISO=+C1-correction | +CJK=+grammar | All=ML+rules | R%=strict | S%=soft
  µs/sample                    |        16.4  |        11.5  |        10.8  |        11.4  |       152.6  |        18.0  |
