Starts by using token_get_all() but does a lot of extra processing
to insert information about the context of the token.
public static tokenizeString ( string $string, object $tokenizer, string $eolChar = '\n', integer $tabWidth = null, string $encoding = null ) : array | ||
$string | string | The string to tokenize. |
$tokenizer | object | A tokenizer class to use to tokenize the string. |
$eolChar | string | The EOL character to use for splitting strings. |
$tabWidth | integer | The number of spaces each tab respresents. |
$encoding | string | The charset of the sniffed file. |
return | array |