diff options
| author |   Albert-Jan Nijburg <albertjan@trinket.io> | 2017-05-31 15:00:21 +0100 |
|---|---|---|
| committer |   Victor Stinner <victor.stinner@gmail.com> | 2017-05-31 16:00:21 +0200 |
| commit | fc354f07855a9197e71f851ad930cbf5652f9160 (patch) | |
| tree | 5c7c6ad2a4de72993d916c321156572d4fa4635d /Parser | |
| parent | subprocess test_empty_env typo fix. (#1877) (diff) | |
| download | cpython-fc354f07855a9197e71f851ad930cbf5652f9160.tar.gz cpython-fc354f07855a9197e71f851ad930cbf5652f9160.tar.bz2 cpython-fc354f07855a9197e71f851ad930cbf5652f9160.zip | |
bpo-25324: copy tok_name before changing it (#1608)
* add test to check if were modifying token
* copy list so import tokenize doesnt have side effects on token
* shorten line
* add tokenize tokens to token.h to get them to show up in token
* move ERRORTOKEN back to its previous location, and fix nitpick
* copy comments from token.h automatically
* fix whitespace and make more pythonic
* change to fix comments from @haypo
* update token.rst and Misc/NEWS
* change wording
* some more wording changes
Diffstat (limited to 'Parser')
| -rw-r--r-- | Parser/tokenizer.c | 3 |
1 files changed, 3 insertions, 0 deletions
diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c index 5cc9533c3cd..7f2f3e60284 100644 --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -106,6 +106,9 @@ const char *_PyParser_TokenNames[] = { "AWAIT", "ASYNC", "<ERRORTOKEN>", + "COMMENT", + "NL", + "ENCODING" "<N_TOKENS>" }; |