| [ Index ] |
PHP Cross Reference of WordPress Trunk (Updated Daily) |
[Summary view] [Print] [Text view]
1 <?php 2 3 /** 4 * Class for efficiently looking up and mapping string keys to string values, with limits. 5 * 6 * @package WordPress 7 * @since 6.6.0 8 */ 9 10 /** 11 * WP_Token_Map class. 12 * 13 * Use this class in specific circumstances with a static set of lookup keys which map to 14 * a static set of transformed values. For example, this class is used to map HTML named 15 * character references to their equivalent UTF-8 values. 16 * 17 * This class works differently than code calling `in_array()` and other methods. It 18 * internalizes lookup logic and provides helper interfaces to optimize lookup and 19 * transformation. It provides a method for precomputing the lookup tables and storing 20 * them as PHP source code. 21 * 22 * All tokens and substitutions must be shorter than 256 bytes. 23 * 24 * Example: 25 * 26 * $smilies = WP_Token_Map::from_array( array( 27 * '8O' => '😯', 28 * ':(' => '🙁', 29 * ':)' => '🙂', 30 * ':?' => '😕', 31 * ) ); 32 * 33 * true === $smilies->contains( ':)' ); 34 * false === $smilies->contains( 'simile' ); 35 * 36 * '😕' === $smilies->read_token( 'Not sure :?.', 9, $length_of_smily_syntax ); 37 * 2 === $length_of_smily_syntax; 38 * 39 * ## Precomputing the Token Map. 40 * 41 * Creating the class involves some work sorting and organizing the tokens and their 42 * replacement values. In order to skip this, it's possible for the class to export 43 * its state and be used as actual PHP source code. 44 * 45 * Example: 46 * 47 * // Export with four spaces as the indent, only for the sake of this docblock. 48 * // The default indent is a tab character. 49 * $indent = ' '; 50 * echo $smilies->precomputed_php_source_table( $indent ); 51 * 52 * // Output, to be pasted into a PHP source file: 53 * WP_Token_Map::from_precomputed_table( 54 * array( 55 * "storage_version" => "6.6.0", 56 * "key_length" => 2, 57 * "groups" => "", 58 * "long_words" => array(), 59 * "small_words" => "8O\x00:)\x00:(\x00:?\x00", 60 * "small_mappings" => array( "😯", "🙂", "🙁", "😕" ) 61 * ) 62 * ); 63 * 64 * ## Large vs. small words. 65 * 66 * This class uses a short prefix called the "key" to optimize lookup of its tokens. 67 * This means that some tokens may be shorter than or equal in length to that key. 68 * Those words that are longer than the key are called "large" while those shorter 69 * than or equal to the key length are called "small." 70 * 71 * This separation of large and small words is incidental to the way this class 72 * optimizes lookup, and should be considered an internal implementation detail 73 * of the class. It may still be important to be aware of it, however. 74 * 75 * ## Determining Key Length. 76 * 77 * The choice of the size of the key length should be based on the data being stored in 78 * the token map. It should divide the data as evenly as possible, but should not create 79 * so many groups that a large fraction of the groups only contain a single token. 80 * 81 * For the HTML5 named character references, a key length of 2 was found to provide a 82 * sufficient spread and should be a good default for relatively large sets of tokens. 83 * 84 * However, for some data sets this might be too long. For example, a list of smilies 85 * may be too small for a key length of 2. Perhaps 1 would be more appropriate. It's 86 * best to experiment and determine empirically which values are appropriate. 87 * 88 * ## Generate Pre-Computed Source Code. 89 * 90 * Since the `WP_Token_Map` is designed for relatively static lookups, it can be 91 * advantageous to precompute the values and instantiate a table that has already 92 * sorted and grouped the tokens and built the lookup strings. 93 * 94 * This can be done with `WP_Token_Map::precomputed_php_source_table()`. 95 * 96 * Note that if there is a leading character that all tokens need, such as `&` for 97 * HTML named character references, it can be beneficial to exclude this from the 98 * token map. Instead, find occurrences of the leading character and then use the 99 * token map to see if the following characters complete the token. 100 * 101 * Example: 102 * 103 * $map = WP_Token_Map::from_array( array( 'simple_smile:' => '🙂', 'sob:' => '😭', 'soba:' => '🍜' ) ); 104 * echo $map->precomputed_php_source_table(); 105 * // Output 106 * WP_Token_Map::from_precomputed_table( 107 * array( 108 * "storage_version" => "6.6.0", 109 * "key_length" => 2, 110 * "groups" => "si\x00so\x00", 111 * "long_words" => array( 112 * // simple_smile:[🙂]. 113 * "\x0bmple_smile:\x04🙂", 114 * // soba:[🍜] sob:[😭]. 115 * "\x03ba:\x04🍜\x02b:\x04😭", 116 * ), 117 * "short_words" => "", 118 * "short_mappings" => array() 119 * } 120 * ); 121 * 122 * This precomputed value can be stored directly in source code and will skip the 123 * startup cost of generating the lookup strings. See `$html5_named_character_entities`. 124 * 125 * Note that any updates to the precomputed format should update the storage version 126 * constant. It would also be best to provide an update function to take older known 127 * versions and upgrade them in place when loading into `from_precomputed_table()`. 128 * 129 * ## Future Direction. 130 * 131 * It may be viable to dynamically increase the length limits such that there's no need to impose them. 132 * The limit appears because of the packing structure, which indicates how many bytes each segment of 133 * text in the lookup tables spans. If, however, care were taken to track the longest word length, then 134 * the packing structure could change its representation to allow for that. Each additional byte storing 135 * length, however, increases the memory overhead and lookup runtime. 136 * 137 * An alternative approach could be to borrow the UTF-8 variable-length encoding and store lengths of less 138 * than 127 as a single byte with the high bit unset, storing longer lengths as the combination of 139 * continuation bytes. 140 * 141 * Since it has not been shown during the development of this class that longer strings are required, this 142 * update is deferred until such a need is clear. 143 * 144 * @since 6.6.0 145 */ 146 class WP_Token_Map { 147 /** 148 * Denotes the version of the code which produces pre-computed source tables. 149 * 150 * This version will be used not only to verify pre-computed data, but also 151 * to upgrade pre-computed data from older versions. Choosing a name that 152 * corresponds to the WordPress release will help people identify where an 153 * old copy of data came from. 154 * 155 * @since 6.6.0 156 */ 157 const STORAGE_VERSION = '6.6.0-trunk'; 158 159 /** 160 * Maximum length for each key and each transformed value in the table (in bytes). 161 * 162 * @since 6.6.0 163 */ 164 const MAX_LENGTH = 256; 165 166 /** 167 * How many bytes of each key are used to form a group key for lookup. 168 * This also determines whether a word is considered short or long. 169 * 170 * @since 6.6.0 171 * 172 * @var int 173 */ 174 private $key_length = 2; 175 176 /** 177 * Stores an optimized form of the word set, where words are grouped 178 * by a prefix of the `$key_length` and then collapsed into a string. 179 * 180 * In each group, the keys and lookups form a packed data structure. 181 * The keys in the string are stripped of their "group key," which is 182 * the prefix of length `$this->key_length` shared by all of the items 183 * in the group. Each word in the string is prefixed by a single byte 184 * whose raw unsigned integer value represents how many bytes follow. 185 * 186 * ┌────────────────┬───────────────┬─────────────────┬────────┐ 187 * │ Length of rest │ Rest of key │ Length of value │ Value │ 188 * │ of key (bytes) │ │ (bytes) │ │ 189 * ├────────────────┼───────────────┼─────────────────┼────────┤ 190 * │ 0x08 │ nterDot; │ 0x02 │ · │ 191 * └────────────────┴───────────────┴─────────────────┴────────┘ 192 * 193 * In this example, the key `CenterDot;` has a group key `Ce`, leaving 194 * eight bytes for the rest of the key, `nterDot;`, and two bytes for 195 * the transformed value `·` (or U+B7 or "\xC2\xB7"). 196 * 197 * Example: 198 * 199 * // Stores array( 'CenterDot;' => '·', 'Cedilla;' => '¸' ). 200 * $groups = "Ce\x00"; 201 * $large_words = array( "\x08nterDot;\x02·\x06dilla;\x02¸" ) 202 * 203 * The prefixes appear in the `$groups` string, each followed by a null 204 * byte. This makes for quick lookup of where in the group string the key 205 * is found, and then a simple division converts that offset into the index 206 * in the `$large_words` array where the group string is to be found. 207 * 208 * This lookup data structure is designed to optimize cache locality and 209 * minimize indirect memory reads when matching strings in the set. 210 * 211 * @since 6.6.0 212 * 213 * @var array 214 */ 215 private $large_words = array(); 216 217 /** 218 * Stores the group keys for sequential string lookup. 219 * 220 * The offset into this string where the group key appears corresponds with the index 221 * into the group array where the rest of the group string appears. This is an optimization 222 * to improve cache locality while searching and minimize indirect memory accesses. 223 * 224 * @since 6.6.0 225 * 226 * @var string 227 */ 228 private $groups = ''; 229 230 /** 231 * Stores an optimized row of small words, where every entry is 232 * `$this->key_size + 1` bytes long and zero-extended. 233 * 234 * This packing allows for direct lookup of a short word followed 235 * by the null byte, if extended to `$this->key_size + 1`. 236 * 237 * Example: 238 * 239 * // Stores array( 'GT', 'LT', 'gt', 'lt' ). 240 * "GT\x00LT\x00gt\x00lt\x00" 241 * 242 * @since 6.6.0 243 * 244 * @var string 245 */ 246 private $small_words = ''; 247 248 /** 249 * Replacements for the small words, in the same order they appear. 250 * 251 * With the position of a small word it's possible to index the translation 252 * directly, as its position in the `$small_words` string corresponds to 253 * the index of the replacement in the `$small_mapping` array. 254 * 255 * Example: 256 * 257 * array( '>', '<', '>', '<' ) 258 * 259 * @since 6.6.0 260 * 261 * @var string[] 262 */ 263 private $small_mappings = array(); 264 265 /** 266 * Create a token map using an associative array of key/value pairs as the input. 267 * 268 * Example: 269 * 270 * $smilies = WP_Token_Map::from_array( array( 271 * '8O' => '😯', 272 * ':(' => '🙁', 273 * ':)' => '🙂', 274 * ':?' => '😕', 275 * ) ); 276 * 277 * @since 6.6.0 278 * 279 * @param array $mappings The keys transform into the values, both are strings. 280 * @param int $key_length Determines the group key length. Leave at the default value 281 * of 2 unless there's an empirical reason to change it. 282 * 283 * @return WP_Token_Map|null Token map, unless unable to create it. 284 */ 285 public static function from_array( array $mappings, int $key_length = 2 ): ?WP_Token_Map { 286 $map = new WP_Token_Map(); 287 $map->key_length = $key_length; 288 289 // Start by grouping words. 290 291 $groups = array(); 292 $shorts = array(); 293 foreach ( $mappings as $word => $mapping ) { 294 if ( 295 self::MAX_LENGTH <= strlen( $word ) || 296 self::MAX_LENGTH <= strlen( $mapping ) 297 ) { 298 _doing_it_wrong( 299 __METHOD__, 300 sprintf( 301 /* translators: 1: maximum byte length (a count) */ 302 __( 'Token Map tokens and substitutions must all be shorter than %1$d bytes.' ), 303 self::MAX_LENGTH 304 ), 305 '6.6.0' 306 ); 307 return null; 308 } 309 310 $length = strlen( $word ); 311 312 if ( $key_length >= $length ) { 313 $shorts[] = $word; 314 } else { 315 $group = substr( $word, 0, $key_length ); 316 317 if ( ! isset( $groups[ $group ] ) ) { 318 $groups[ $group ] = array(); 319 } 320 321 $groups[ $group ][] = array( substr( $word, $key_length ), $mapping ); 322 } 323 } 324 325 /* 326 * Sort the words to ensure that no smaller substring of a match masks the full match. 327 * For example, `Cap` should not match before `CapitalDifferentialD`. 328 */ 329 usort( $shorts, 'WP_Token_Map::longest_first_then_alphabetical' ); 330 foreach ( $groups as $group_key => $group ) { 331 usort( 332 $groups[ $group_key ], 333 static function ( array $a, array $b ): int { 334 return self::longest_first_then_alphabetical( $a[0], $b[0] ); 335 } 336 ); 337 } 338 339 // Finally construct the optimized lookups. 340 341 foreach ( $shorts as $word ) { 342 $map->small_words .= str_pad( $word, $key_length + 1, "\x00", STR_PAD_RIGHT ); 343 $map->small_mappings[] = $mappings[ $word ]; 344 } 345 346 $group_keys = array_keys( $groups ); 347 sort( $group_keys ); 348 349 foreach ( $group_keys as $group ) { 350 $map->groups .= "{$group}\x00"; 351 352 $group_string = ''; 353 354 foreach ( $groups[ $group ] as $group_word ) { 355 list( $word, $mapping ) = $group_word; 356 357 $word_length = pack( 'C', strlen( $word ) ); 358 $mapping_length = pack( 'C', strlen( $mapping ) ); 359 $group_string .= "{$word_length}{$word}{$mapping_length}{$mapping}"; 360 } 361 362 $map->large_words[] = $group_string; 363 } 364 365 return $map; 366 } 367 368 /** 369 * Creates a token map from a pre-computed table. 370 * This skips the initialization cost of generating the table. 371 * 372 * This function should only be used to load data created with 373 * WP_Token_Map::precomputed_php_source_tag(). 374 * 375 * @since 6.6.0 376 * 377 * @param array $state { 378 * Stores pre-computed state for directly loading into a Token Map. 379 * 380 * @type string $storage_version Which version of the code produced this state. 381 * @type int $key_length Group key length. 382 * @type string $groups Group lookup index. 383 * @type array $large_words Large word groups and packed strings. 384 * @type string $small_words Small words packed string. 385 * @type array $small_mappings Small word mappings. 386 * } 387 * 388 * @return WP_Token_Map Map with precomputed data loaded. 389 */ 390 public static function from_precomputed_table( $state ): ?WP_Token_Map { 391 $has_necessary_state = isset( 392 $state['storage_version'], 393 $state['key_length'], 394 $state['groups'], 395 $state['large_words'], 396 $state['small_words'], 397 $state['small_mappings'] 398 ); 399 400 if ( ! $has_necessary_state ) { 401 _doing_it_wrong( 402 __METHOD__, 403 __( 'Missing required inputs to pre-computed WP_Token_Map.' ), 404 '6.6.0' 405 ); 406 return null; 407 } 408 409 if ( self::STORAGE_VERSION !== $state['storage_version'] ) { 410 _doing_it_wrong( 411 __METHOD__, 412 /* translators: 1: version string, 2: version string. */ 413 sprintf( __( 'Loaded version \'%1$s\' incompatible with expected version \'%2$s\'.' ), $state['storage_version'], self::STORAGE_VERSION ), 414 '6.6.0' 415 ); 416 return null; 417 } 418 419 $map = new WP_Token_Map(); 420 421 $map->key_length = $state['key_length']; 422 $map->groups = $state['groups']; 423 $map->large_words = $state['large_words']; 424 $map->small_words = $state['small_words']; 425 $map->small_mappings = $state['small_mappings']; 426 427 return $map; 428 } 429 430 /** 431 * Indicates if a given word is a lookup key in the map. 432 * 433 * Example: 434 * 435 * true === $smilies->contains( ':)' ); 436 * false === $smilies->contains( 'simile' ); 437 * 438 * @since 6.6.0 439 * 440 * @param string $word Determine if this word is a lookup key in the map. 441 * @param string $case_sensitivity Optional. Pass 'ascii-case-insensitive' to ignore ASCII case when matching. Default 'case-sensitive'. 442 * @return bool Whether there's an entry for the given word in the map. 443 */ 444 public function contains( string $word, string $case_sensitivity = 'case-sensitive' ): bool { 445 if ( str_contains( $word, "\x00" ) ) { 446 return false; 447 } 448 449 $ignore_case = 'ascii-case-insensitive' === $case_sensitivity; 450 451 if ( $this->key_length >= strlen( $word ) ) { 452 if ( 0 === strlen( $this->small_words ) ) { 453 return false; 454 } 455 456 $term = str_pad( $word, $this->key_length + 1, "\x00", STR_PAD_RIGHT ); 457 $word_at = $ignore_case ? stripos( $this->small_words, $term ) : strpos( $this->small_words, $term ); 458 if ( false === $word_at ) { 459 return false; 460 } 461 462 return true; 463 } 464 465 $group_key = substr( $word, 0, $this->key_length ); 466 $group_at = $ignore_case ? stripos( $this->groups, $group_key ) : strpos( $this->groups, $group_key ); 467 if ( false === $group_at ) { 468 return false; 469 } 470 $group = $this->large_words[ $group_at / ( $this->key_length + 1 ) ]; 471 $group_length = strlen( $group ); 472 $slug = substr( $word, $this->key_length ); 473 $length = strlen( $slug ); 474 $at = 0; 475 476 while ( $at < $group_length ) { 477 $token_length = unpack( 'C', $group[ $at++ ] )[1]; 478 $token_at = $at; 479 $at += $token_length; 480 $mapping_length = unpack( 'C', $group[ $at++ ] )[1]; 481 $mapping_at = $at; 482 483 if ( $token_length === $length && 0 === substr_compare( $group, $slug, $token_at, $token_length, $ignore_case ) ) { 484 return true; 485 } 486 487 $at = $mapping_at + $mapping_length; 488 } 489 490 return false; 491 } 492 493 /** 494 * If the text starting at a given offset is a lookup key in the map, 495 * return the corresponding transformation from the map, else `false`. 496 * 497 * This function returns the translated string, but accepts an optional 498 * parameter `$matched_token_byte_length`, which communicates how many 499 * bytes long the lookup key was, if it found one. This can be used to 500 * advance a cursor in calling code if a lookup key was found. 501 * 502 * Example: 503 * 504 * false === $smilies->read_token( 'Not sure :?.', 0, $token_byte_length ); 505 * '😕' === $smilies->read_token( 'Not sure :?.', 9, $token_byte_length ); 506 * 2 === $token_byte_length; 507 * 508 * Example: 509 * 510 * while ( $at < strlen( $input ) ) { 511 * $next_at = strpos( $input, ':', $at ); 512 * if ( false === $next_at ) { 513 * break; 514 * } 515 * 516 * $smily = $smilies->read_token( $input, $next_at, $token_byte_length ); 517 * if ( false === $next_at ) { 518 * ++$at; 519 * continue; 520 * } 521 * 522 * $prefix = substr( $input, $at, $next_at - $at ); 523 * $at += $token_byte_length; 524 * $output .= "{$prefix}{$smily}"; 525 * } 526 * 527 * @since 6.6.0 528 * 529 * @param string $text String in which to search for a lookup key. 530 * @param int $offset Optional. How many bytes into the string where the lookup key ought to start. Default 0. 531 * @param int|null &$matched_token_byte_length Optional. Holds byte-length of found token matched, otherwise not set. Default null. 532 * @param string $case_sensitivity Optional. Pass 'ascii-case-insensitive' to ignore ASCII case when matching. Default 'case-sensitive'. 533 * @return string|null Mapped value of lookup key if found, otherwise `null`. 534 */ 535 public function read_token( string $text, int $offset = 0, &$matched_token_byte_length = null, $case_sensitivity = 'case-sensitive' ): ?string { 536 $ignore_case = 'ascii-case-insensitive' === $case_sensitivity; 537 $text_length = strlen( $text ); 538 539 // Search for a long word first, if the text is long enough, and if that fails, a short one. 540 if ( $text_length > $this->key_length ) { 541 /* 542 * Keys cannot contain null bytes, which is taken care of for the full words, 543 * but here it’s required to reject group keys with null bytes so that the 544 * lookup doesn’t get off track when scanning the group string. 545 */ 546 if ( strcspn( $text, "\x00", $offset, $this->key_length ) < $this->key_length ) { 547 return null; 548 } 549 550 $group_key = substr( $text, $offset, $this->key_length ); 551 $group_at = $ignore_case ? stripos( $this->groups, $group_key ) : strpos( $this->groups, $group_key ); 552 if ( false === $group_at ) { 553 // Perhaps a short word then. 554 return strlen( $this->small_words ) > 0 555 ? $this->read_small_token( $text, $offset, $matched_token_byte_length, $case_sensitivity ) 556 : null; 557 } 558 559 $group = $this->large_words[ $group_at / ( $this->key_length + 1 ) ]; 560 $group_length = strlen( $group ); 561 $at = 0; 562 while ( $at < $group_length ) { 563 $token_length = unpack( 'C', $group[ $at++ ] )[1]; 564 $token = substr( $group, $at, $token_length ); 565 $at += $token_length; 566 $mapping_length = unpack( 'C', $group[ $at++ ] )[1]; 567 $mapping_at = $at; 568 569 if ( 0 === substr_compare( $text, $token, $offset + $this->key_length, $token_length, $ignore_case ) ) { 570 $matched_token_byte_length = $this->key_length + $token_length; 571 return substr( $group, $mapping_at, $mapping_length ); 572 } 573 574 $at = $mapping_at + $mapping_length; 575 } 576 } 577 578 // Perhaps a short word then. 579 return strlen( $this->small_words ) > 0 580 ? $this->read_small_token( $text, $offset, $matched_token_byte_length, $case_sensitivity ) 581 : null; 582 } 583 584 /** 585 * Finds a match for a short word at the index. 586 * 587 * @since 6.6.0 588 * 589 * @param string $text String in which to search for a lookup key. 590 * @param int $offset Optional. How many bytes into the string where the lookup key ought to start. Default 0. 591 * @param int|null &$matched_token_byte_length Optional. Holds byte-length of found lookup key if matched, otherwise not set. Default null. 592 * @param string $case_sensitivity Optional. Pass 'ascii-case-insensitive' to ignore ASCII case when matching. Default 'case-sensitive'. 593 * @return string|null Mapped value of lookup key if found, otherwise `null`. 594 */ 595 private function read_small_token( string $text, int $offset = 0, &$matched_token_byte_length = null, $case_sensitivity = 'case-sensitive' ): ?string { 596 $ignore_case = 'ascii-case-insensitive' === $case_sensitivity; 597 $small_length = strlen( $this->small_words ); 598 $search_text = substr( $text, $offset, $this->key_length ); 599 if ( $ignore_case ) { 600 $search_text = strtoupper( $search_text ); 601 } 602 $starting_char = $search_text[0]; 603 604 $at = 0; 605 while ( $at < $small_length ) { 606 if ( 607 $starting_char !== $this->small_words[ $at ] && 608 ( ! $ignore_case || strtoupper( $this->small_words[ $at ] ) !== $starting_char ) 609 ) { 610 $at += $this->key_length + 1; 611 continue; 612 } 613 614 for ( $adjust = 1; $adjust < $this->key_length; $adjust++ ) { 615 if ( "\x00" === $this->small_words[ $at + $adjust ] ) { 616 $matched_token_byte_length = $adjust; 617 return $this->small_mappings[ $at / ( $this->key_length + 1 ) ]; 618 } 619 620 if ( 621 $search_text[ $adjust ] !== $this->small_words[ $at + $adjust ] && 622 ( ! $ignore_case || strtoupper( $this->small_words[ $at + $adjust ] !== $search_text[ $adjust ] ) ) 623 ) { 624 $at += $this->key_length + 1; 625 continue 2; 626 } 627 } 628 629 $matched_token_byte_length = $adjust; 630 return $this->small_mappings[ $at / ( $this->key_length + 1 ) ]; 631 } 632 633 return null; 634 } 635 636 /** 637 * Exports the token map into an associate array of key/value pairs. 638 * 639 * Example: 640 * 641 * $smilies->to_array() === array( 642 * '8O' => '😯', 643 * ':(' => '🙁', 644 * ':)' => '🙂', 645 * ':?' => '😕', 646 * ); 647 * 648 * @since 6.6.0 649 * 650 * @return array The lookup key/substitution values as an associate array. 651 */ 652 public function to_array(): array { 653 $tokens = array(); 654 655 $at = 0; 656 $small_mapping = 0; 657 $small_length = strlen( $this->small_words ); 658 while ( $at < $small_length ) { 659 $key = rtrim( substr( $this->small_words, $at, $this->key_length + 1 ), "\x00" ); 660 $value = $this->small_mappings[ $small_mapping++ ]; 661 $tokens[ $key ] = $value; 662 663 $at += $this->key_length + 1; 664 } 665 666 foreach ( $this->large_words as $index => $group ) { 667 $prefix = substr( $this->groups, $index * ( $this->key_length + 1 ), 2 ); 668 $group_length = strlen( $group ); 669 $at = 0; 670 while ( $at < $group_length ) { 671 $length = unpack( 'C', $group[ $at++ ] )[1]; 672 $key = $prefix . substr( $group, $at, $length ); 673 674 $at += $length; 675 $length = unpack( 'C', $group[ $at++ ] )[1]; 676 $value = substr( $group, $at, $length ); 677 678 $tokens[ $key ] = $value; 679 $at += $length; 680 } 681 } 682 683 return $tokens; 684 } 685 686 /** 687 * Export the token map for quick loading in PHP source code. 688 * 689 * This function has a specific purpose, to make loading of static token maps fast. 690 * It's used to ensure that the HTML character reference lookups add a minimal cost 691 * to initializing the PHP process. 692 * 693 * Example: 694 * 695 * echo $smilies->precomputed_php_source_table(); 696 * 697 * // Output. 698 * WP_Token_Map::from_precomputed_table( 699 * array( 700 * "storage_version" => "6.6.0", 701 * "key_length" => 2, 702 * "groups" => "", 703 * "long_words" => array(), 704 * "small_words" => "8O\x00:)\x00:(\x00:?\x00", 705 * "small_mappings" => array( "😯", "🙂", "🙁", "😕" ) 706 * ) 707 * ); 708 * 709 * @since 6.6.0 710 * 711 * @param string $indent Optional. Use this string for indentation, or rely on the default horizontal tab character. Default "\t". 712 * @return string Value which can be pasted into a PHP source file for quick loading of table. 713 */ 714 public function precomputed_php_source_table( string $indent = "\t" ): string { 715 $i1 = $indent; 716 $i2 = $i1 . $indent; 717 $i3 = $i2 . $indent; 718 719 $class_version = self::STORAGE_VERSION; 720 721 $output = self::class . "::from_precomputed_table(\n"; 722 $output .= "{$i1}array(\n"; 723 $output .= "{$i2}\"storage_version\" => \"{$class_version}\",\n"; 724 $output .= "{$i2}\"key_length\" => {$this->key_length},\n"; 725 726 $group_line = str_replace( "\x00", "\\x00", $this->groups ); 727 $output .= "{$i2}\"groups\" => \"{$group_line}\",\n"; 728 729 $output .= "{$i2}\"large_words\" => array(\n"; 730 731 $prefixes = explode( "\x00", $this->groups ); 732 foreach ( $prefixes as $index => $prefix ) { 733 if ( '' === $prefix ) { 734 break; 735 } 736 $group = $this->large_words[ $index ]; 737 $group_length = strlen( $group ); 738 $comment_line = "{$i3}//"; 739 $data_line = "{$i3}\""; 740 $at = 0; 741 while ( $at < $group_length ) { 742 $token_length = unpack( 'C', $group[ $at++ ] )[1]; 743 $token = substr( $group, $at, $token_length ); 744 $at += $token_length; 745 $mapping_length = unpack( 'C', $group[ $at++ ] )[1]; 746 $mapping = substr( $group, $at, $mapping_length ); 747 $at += $mapping_length; 748 749 $token_digits = str_pad( dechex( $token_length ), 2, '0', STR_PAD_LEFT ); 750 $mapping_digits = str_pad( dechex( $mapping_length ), 2, '0', STR_PAD_LEFT ); 751 752 $mapping = preg_replace_callback( 753 "~[\\x00-\\x1f\\x22\\x5c]~", 754 static function ( $match_result ) { 755 switch ( $match_result[0] ) { 756 case '"': 757 return '\\"'; 758 759 case '\\': 760 return '\\\\'; 761 762 default: 763 $hex = dechex( ord( $match_result[0] ) ); 764 return "\\x{$hex}"; 765 } 766 }, 767 $mapping 768 ); 769 770 $comment_line .= " {$prefix}{$token}[{$mapping}]"; 771 $data_line .= "\\x{$token_digits}{$token}\\x{$mapping_digits}{$mapping}"; 772 } 773 $comment_line .= ".\n"; 774 $data_line .= "\",\n"; 775 776 $output .= $comment_line; 777 $output .= $data_line; 778 } 779 780 $output .= "{$i2}),\n"; 781 782 $small_words = array(); 783 $small_length = strlen( $this->small_words ); 784 $at = 0; 785 while ( $at < $small_length ) { 786 $small_words[] = substr( $this->small_words, $at, $this->key_length + 1 ); 787 $at += $this->key_length + 1; 788 } 789 790 $small_text = str_replace( "\x00", '\x00', implode( '', $small_words ) ); 791 $output .= "{$i2}\"small_words\" => \"{$small_text}\",\n"; 792 793 $output .= "{$i2}\"small_mappings\" => array(\n"; 794 foreach ( $this->small_mappings as $mapping ) { 795 $output .= "{$i3}\"{$mapping}\",\n"; 796 } 797 $output .= "{$i2})\n"; 798 $output .= "{$i1})\n"; 799 $output .= ')'; 800 801 return $output; 802 } 803 804 /** 805 * Compares two strings, returning the longest, or whichever 806 * is first alphabetically if they are the same length. 807 * 808 * This is an important sort when building the token map because 809 * it should not form a match on a substring of a longer potential 810 * match. For example, it should not detect `Cap` when matching 811 * against the string `CapitalDifferentialD`. 812 * 813 * @since 6.6.0 814 * 815 * @param string $a First string to compare. 816 * @param string $b Second string to compare. 817 * @return int -1 or lower if `$a` is less than `$b`; 1 or greater if `$a` is greater than `$b`, and 0 if they are equal. 818 */ 819 private static function longest_first_then_alphabetical( string $a, string $b ): int { 820 if ( $a === $b ) { 821 return 0; 822 } 823 824 $length_a = strlen( $a ); 825 $length_b = strlen( $b ); 826 827 // Longer strings are less-than for comparison's sake. 828 if ( $length_a !== $length_b ) { 829 return $length_b - $length_a; 830 } 831 832 return strcmp( $a, $b ); 833 } 834 }
title
Description
Body
title
Description
Body
title
Description
Body
title
Body
| Generated : Tue Sep 22 08:20:31 2026 | Cross-referenced by PHPXref |