PGNTRIM5.

DOC

18 APR 2006 - Documentation for freeware chess PGN normalization utility.

Author: Tom McCormick mccormit@sbcglobal.net Comments or questions are welcome.

BACKGROUND
-------------------

PGNTRIM5.EXE requires no installation routine, nor any .DLL file(s).

It runs from a Microsoft DOS prompt, or from a Microsoft Windows command prompt.
Under Windows NT/XP/Vista, just "Run" then enter "cmd" to start the command
processor.
The cmd command processor is an NT program provided by Microsoft as part of Windows
to emulate the old DOS (Disk Operating System) program executions. Multiple copies
of cmd may be run at the same time if you wish. "path" and other environment
variables work the same as when you are in the graphical interface.

User will be prompted for input and output filenames if none are in the command
tail. There are two additional (OPTIONAL) command tail parameters:
1. /MPL:1 to /MPL:7 determines the output moves per line
/MPL:1 is nice for students to add comments for review by an instructor.
2. /ALLTAGS will preserve all [Tag records: default is most important 12 tags,
only.

PGNTRIM5.EXE is a freeware Windows utility to correct most PGN syntax errors, and
to
direct games which need human review into a separate output file named
BADTRIM5.BAD. That
file usually furnishes enough information to the user so that a decision can be
made to
correct the input file and run again, or to accept the number of games rejected
from the
new output file. PGNTRIM5 never changes the original input file.

PGNTRIM5 does NOT detect illegal/impossible moves; but pgnscid or normal32 will
catch these.
I run pgntrim5 first against newly downloaded PGN files in order to clean up common
syntax
problems and ommisions, then I run the output file into pgnscid or normal32 to
catch any
illegal moves. This approach greatly reduces the amount of time needed to edit the
PGN
file for syntax errors before placing it into a database.

(P)ortable (G)ame (N)otation format is rather thoroughly defined and effective as a

means to
record and distribute recordings of chess game moves. This standard is available
over the
internet from several sources.

After 35 years in the computer systems business, I can say with certainty that I
have never
seen a standard that was followed by everyone. Or as the saying goes: "It is
wonderful
to have a standard, and there are so many nice ones to choose from!" In the case
of PGN,
there really are not many competing standards, but the problem of human errors in
remembering syntax (as when two knights can move to the same square...called
ambiguity),
entering an f instead of a g, or simply omitting items by oversight such as
omitting part of a move... especially omitting the "x" in captures such as entering
cd4 instead of cxd4 cause many difficulties. It is all too common for a viewing
program to find an "illegal move", a
self-capture, an impossible move (Nbg4 is my favorite), etc. Most pawn promotions
are
to a Queen, Knight, or Rook, but to a Bishop is legal, and can result in more than
one
"light-squared Bishop), and hence they would also need ambiguity coding.

People have been submitting "annofritzed" pgn games to internet websites. They
often like
to add their own comments and alternate moves. Such heavily commented games
frequently
reach more than 8,000 characters in length...and all too often contain unbalanced
alternate move tokens (..)..) for which nesting IS permitted, or they may contain
extra curly brace tokens {..} delimiting comments. Fritz will produce correct PGN
syntax when autofritzing, but humans seem driven to "improve or clarify" these and
they
frequently end up with these tokens unbalanced. These lengthy games are sent to the
BadTrim5.Bad so that they can receive human review.

Recently, PGN games have been appearing on the internet which are Fischer-Random
games.
This new form of chess is also called Chess960 because there are 960 possible ways
to
arrange pieces in the back rank as the game begins.

If there is no FEN statement, or other indication of this, then "illegal moves"

such as
1.Nb3 will pass through syntax checking, but will appear to be an illegal move!
Some
standard indication of Fisher-Random games needs to be added to the PGN standard.

Chess magazines and books are not immune from typograhical errors and omissions
such as leaving out moves entirely, leaving pieces off the diagrams, having two
black
Kings and no White King, displaying entirely the wrong diagram, etc.

Persons collecting PGN chess game records do not want to end up with such problems
that show up while a game is being studied! Normalization programs can detect most
PGN problems, fix some, and tell the user about the others so that they can be
manually edited, or the game discarded.

A PGN game recording example follows. Heading records are called "tags", and seven
of
them are required as a minimum....the first 7 shown below are required in any PGN
game.
Other tags are optional such as the "Opening" and "ECO" tags shown. All tags must
conform
to standard in order to be useful to a wide audience...Each tag must begin with
[ and end
with ], and the tag name must begin with one uppercase letter, the text must be
enclosed
within quotation marks, etc. It is somewhat surprising just how many PGN games
have
simple syntax errors in the tag records!

PGNTRIM5 will halt immediately if no carriage-return/linefeed character pairs are

found
as line delimiters. This is usually due to the PGN file originating on a Unix/Linux
system which only requires the linefeed, not both carriage-return and linefeed. The
user
is advised to run such PGN files through crlf.exe before attempting to use them on
a
Windows PC.

PGNTRIM5 will detect unbalanced (...) alternate move delimiters, and unbalanced
{...}
comment delimiters. Balanced alternate moves and comments (including nested
comments, which are legal) will be removed by PGNTRIM5. Unbalanced pairs of these
will
be rejected by PGNTRIM5, and the game sent to the BADTRIM5.BAD text file for
review.

Refer to the TEST.PGN file which accompanies distribution of PGNTRIM5. This may be
run through PGNTRIM5 to show the breadth of syntax errors it will correct.

Normalization programs detect deviations from standard, and either fix the problem,
notify
the user, or both. Missing tags, illegal moves or incomplete moves such as B7, a8,
or Rx
can not be fixed and are simply reorted to the user for editing or discarding the
game.

Other problems such as spacing errors can usually be fixed by a normalization

program;
so Nxg4Nbd7 (no space between White and Black halfmoves) can be fixed to
Nxg4 Nbd7, and O-O5. can be fixed to O-O 5. Castling must use alpha O, not
zeroes,
a normalization program can easily substitute to fix this. The correct symbol for
checkmate is #, not ++ or the word checkmate!

Avoid using a semicolon ANYWHERE in a PGN game file; the standard intends this to
signal
that the rest of the line is comments, and not essential. Semicolons within tags
cause
a tag syntax error. "Handling" semicolons within tags introduces other problems for
the
normalization programmer.

By normalizing PGN files before inserting them into you databases, you will improve
your
enjoyment considerably by avoiding syntax error interruptions.

[Event "Example PGN Chess Game Record"]

[Site "Moscow"]
[Date "2003.12.25"]
[Round "2"]
[White "Blaganov"]
[Black "Dufus"]
[Result "1-0"]
[Opening "Scandinavian"]

[ECO "B01"]

1.e4 d5 2.exd5 Qxd5 3.Nc3 Qd8 4.d4 Nf6

 {B01 Scandinavian}
5.Bc4 c6 6.Nf3 Bg4 7.Bxf7 Kxf7 8.Ne5 Kg8
9.Nxg4 Nbd7 10.Qe2 Nxg4 11.Qe6# 0-1

CARRIAGE-RETURNS AND LINE-FEEDS.

--------------------------------
PGNTRIM5 expects the input file to contain PGN data somewhat similar to the above.
Since it is a Microsoft Windows executable program, the incoming records should
be delimited by a pair of characters (carriage-return and linefeed) rather than
the Unix/Linux line delimiter of linefeed only. PGNTRIM5 will immediately notify
you if it can not read a PGN file from a Unix/Linux source. You can simply pass
the input file through CRLF.EXE or some such freeware utility to quickly make
the file accesible to a Windows PC. If you have crlf.exe, simply enter that
command using 'Run' from Windows or after execting command.exe (win9x) or
cmd.exe (Windows XP)....and then key in the input PGN filename which you want
to convert from having only linefeed delimiters to CR/LF delimiters for Windows.
NOTE: crlf.exe and possibly other such programs DO change the input file....
writing output to the SAME FILENAME...so make a copy first if you wish!

TOO MANY END-OF-FILE CHARACTERS IN ONE FILE.

--------------------------------------------

Windows will treat the first one as end-of-file regardless of the length specified
in the directory. XEOFF.EXE will safely remove the extra end-of-file characters,
and it will remove all form-feed characters in that file. There is no need for
form feeds in a PGN file.

PGNTRIM5.EXE
--------------------

Freeware stand-alone executable utility program to read a PGN-format input file,

identify as many errors as possible, and fix as many of those as possible. This
program has been run from the "DOS window" of many versions of Windows
including 95, 98, ME, and from the Windows XP "Command Prompt" window using the
DOS emulator cmd.exe which comes with Windows XP. If you do not understand the
previous sentence, then this program is probably not for you unless you have a
friend who can help you for a few minutes.

If you are collecting PGN files from internet sources, you will find PGNTRIM5
very useful in saving you much time and adding to your enjoyment when viewing
games with a viewer or database such as SCID, Bookup, or Chessbase. While it
does not catch impossible or illegal moves, it will catch a vast majority of
errors which frequently occur in PGN data, and it will correct most of those
syntax errors, and some ommissions such as [Round ""] into [Round "?"]. If
any of the seven required tag records are missing or illegible, then the game
is sent into the rejections file BADTRIM5.BAD.

Games which can not be fixed are placed into a separate output file for editing.
The original input file is NEVER changed by PGNTRIM5, a separate output file of
normalized PGN is always created (unless all the games go into the reject file)!
Game images in BADTRIM5.BAD contain an additional tag record titled [Warning
which provides the reason for placing the game image into the reject file. If
you edit the rejections file directly, you may delete these [Warning tags.
Processing continues as a rule, but if a game is unusually stinky, then the
game being processed when the fatal error occurred is written to fatalerr.pgn.
A game sent into fatalerr.pgn should be located within the original input file
and edited there, or removed, before reprocessing the input file.

PGNTRIM5 no longer uses a settings file (as did previous versions) since
permitting curly-brace comments, alternate move sequences, etc. greatly
compounded the programming difficulty of trying to fix syntax errors. The
desired number of moves per line in the output file, however, may still be
specified as a command line argument (or parameter) such as the example:
pgnTRIM5 infile1.pgn outfile.pgn /MPL:5 to obtain 5 moves per output line.

PGNTRIM5 will remove all {...} and (...) and ? and ! and $nnn commentary
data due to the high percentage of syntactical errors accompanying these.
If you want those comments, keep a copy of the input file as another file
name such as 04Linar.sav or 04Linar.cmt or some such. PGNTRIM5 never changes
the input file, but you might do so if you remove error games, or otherwise
edit the file yourself....so have a naming scheme to save files you want to
be able to return to. 04Linar.001 04Linar.002 etc. will accomplish this.
Suggestion, I use .TRM (for trim) as the output filetype.
For example, pgntrim5 Kasparov.pgn kasparov.trm
...then if the .BAD reject file contains little or nothing from this,
I rename the original file as with ren Kasparov.pgn Kasparov.sav
and then copy Kasparov.trm Kasparov.PGN /y
and then del Kasparov.trm

PGNTRIM5 will handle Danish characters "������" , and will remove

additional Fritz8 commentary in movestext such as [%eval, etc.

There is no limit as to the size of the input or output files provided you have
enough disk space for the output file. I have passed 43 million records
through PGNTRIM5 in one pass, however files that large are not convenient
to edit, sort, or otherwise work with. But if you have a huge PGN file, and you
wish to normalize it all at once, you may do so with PGNTRIM5.

TEST FILE to show what PGNTRIM5 will do.....

---------
A test data file is included in PGNTRIM5.ZIP; games numbered below 50 are
normalized by PGNTRIM5, and placed into whatever output filename you specify.
Games numbered 50 and above contain conditions which PGNTRIM5 cannot fix.
Such games are written into into the BADTRIM5.BAD rejections file, or added
to it if it already exists. To run the test data enter the following:
pgnTRIM5 test.pgn testout.pgn and some games will go into BADTRIM5.BAD.
Note that TEST.PGN contains several illegal moves simply to exercise PGNTRIM5
syntax corrections. NOTE: PGNTRIM5 does NOT catch illegal or impossible moves.

The PGN data in BADTRIM5.BAD is sometimes complete enough to edit, or will

normally contain enough information to point you to the specific game to edit
within the input file before reprocessing.

Here is an example "BEFORE" and "AFTER" using PGNTRIM5.

This very old game was annotated by the computer program
Fritz 6...a process called annofritzing. There are many
comments within curly braces {...}, NAG comments... $17,
move continations following an alternate move sequence,
and there are many nested alternate moves i.e., ( ( ( ) ) )

[Event "New Orleans"]

[Site "New Orleans"]
[Date "1849.??.??"]
[Round "?"]
[White "Morphy, Paul "]
[Black "J. MacConnell sr"]
[Result "1-0"]
[Annotator "Fritz 6 (6s)"]
[PlyCount "57"]
[EventDate "1849.??.??"]

1. e4 {C39: King's Gambit Accepted: 3 Nf3 g5 4 h4} e5 2. f4 exf4 3. Nf3 g5 4.

h4 g4 5. Ne5 h5 6. Bc4 Rh7 7. d4 d6 8. Nd3 f3 9. g3 (9. gxf3 Be7 10. Be3 Bxh4+
11. Kd2 Bg5 12. f4 Bf6 13. a3 c6 14. Nc3 Bh8 15. f5 Ne7 16. Qe2 Kf8 17. f6 Bxf6
18. Raf1 d5 19. Rxf6 dxc4 20. Ne5 Nd7 21. Nxd7+ Bxd7 22. Rh6 Rg7 23. R6xh5 Ng8
{Pektor,A-Zvara,P/Prague 1992/0-1 (48)}) 9... Nc6 10. Nf4 $146 (10. c3 Nge7 (
10... Nce7 11. Kf2 c6 12. Nf4 Qc7 13. Qb3 b5 14. Bd3 Rh8 15. Re1 Ng6 16. Nxg6
fxg6 17. e5 Ne7 18. Bxg6+ Kd8 19. Qf7 Nxg6 20. Qxg6 Qg7 21. Bg5+ Kc7 22. exd6+
Kb6 23. Bd8+ Ka6 24. Qxg7 Bxg7 25. Bc7 {
Abbe de Lionne & Morant-Maubisson & Auzout/Paris 1680/1-0 (40)}) 11. Nf4 a6 12.
a4 Bg7 13. Qb3 Bh8 14. Nxh5 Kf8 15. Nf4 Na5 16. Qa2 Nxc4 17. Qxc4 c6 18. Nd2 d5
19. exd5 cxd5 20. Qb4 Bf6 21. Nf1 Kg7 22. h5 Nc6 23. Qc5 Be6 24. Qa3 Qd7 {
Jannisson & Maubisson-Lionne & Morant/Paris 1680 (36)}) (10. Bb5 d5 11. Ne5
Bd7 12. Nxd7 Qxd7 $17 (12... Kxd7 $2 13. exd5 Bd6 14. Kf2 $18 (14. dxc6+ $6
bxc6 15. Ba4 Bxg3+ 16. Kf1 Rb8 $16 (16... Bxh4 $4 {
taking the pawn will bring Black grief} 17. Qd3 $18)))) 10... Bd7 (10... Nf6
11. Nc3 $17) 11. Nc3 Nf6 (11... Bg7 12. Be3 $17) 12. Be3 Ne7 (12... Bh6 13. Rf1
$17) 13. Kf2 c6 (13... Bh6 14. e5 dxe5 15. dxe5 $17) 14. Re1 Bg7 15. e5 dxe5
16. dxe5 Nfd5 (16... Nfg8 17. Ne4 Bxe5 18. Ng5 Bxf4 19. Nxh7 Bxe3+ 20. Rxe3 $11
) 17. Bxd5 (17. Nfxd5 Nxd5 18. Nxd5 cxd5 19. Qxd5 Bh8 $14) 17... cxd5 (17...
Nxd5 18. Ncxd5 cxd5 19. Nxd5 Be6 $14 (19... Bxe5 {
Black again will not be able to digest the pawn} 20. Bg5 f6 21. Nxf6+ Kf7 22.
Rxe5 (22. Qxd7+ $6 {is not possible} Qxd7 23. Nxd7 Bd4+ 24. Kf1 Kg6 $18) 22...
Qb6+ 23. Re3 $18)) 18. Bc5 (18. Ncxd5 Nxd5 (18... Bxe5 $2 {
is nothing because of} 19. Bb6 Qb8 20. Bd4 $18) 19. Nxd5 Be6 $14 (19... Bxe5 {
as before the pawn must remain untouched} 20. Bg5 f6 21. Nxf6+ Kf7 22. Rxe5
Qb6+ 23. Re3 $18)) 18... Bc6 (18... Rc8 19. Bxa7 Qa5 20. Bd4 $15) 19. b4 (19.
Qd3 Rh6 $11) 19... b6 (19... d4 $142 20. Qd3 Rh6 $15 (20... dxc3 21. Qxh7 Kf8
22. Rad1 $18 (22. Qxh5 $6 {is the less attractive alternative} Qd2+ 23. Kg1 Kg8
$18) (22. Nxh5 $4 {the pawn is indigestible} Qd2+ 23. Re2 Qxe2+ 24. Kg1 Qg2#)))
20. Bxe7 $14 Qxe7 {The isolani on e5 becomes a target} 21. Nfxd5 Qb7 $4 (21...
Bxd5 $142 {is just about the only chance} 22. Nxd5 Qd8 23. Nf6+ Bxf6 24. exf6+
Kf8 25. Qxd8+ Rxd8 $16) 22. Nf6+ $18 Bxf6 23. exf6+ Kf8 24. Qd6+ Kg8 25. Re7
Qc8 26. Rc7 Qf5 27. Qxc6 {Threatening mate: Qxa8} Qxc2+ (27... Rf8 {
does not save the day} 28. Nd5 Qe5 $18) 28. Ke3 Rd8 (28... Rf8 29. Rxa7 Qb2 30.
Ra8 Qxc3+ 31. Qxc3 Rxa8 32. Qc7 $18) 29. Rd1 $1 {
the end of the story. Threatening mate... how?} (29. Rd1 Rf8 30. Rxa7 $18) 1-0

[Event "New Orleans"]

[Site "New Orleans"]
[Date "1849.??.??"]
[Round "?"]
[White "Morphy, Paul "]
[Black "J. MacConnell sr"]
[Result "1-0"]
[Annotator "Fritz 6 6s "]
[PlyCount "57"]
[EventDate "1849.??.??"]

1.e4 e5 2.f4 exf4 3.Nf3 g5 4.h4 g4

5.Ne5 h5 6.Bc4 Rh7 7.d4 d6 8.Nd3 f3
9.g3 Nc6 10.Nf4 Bd7 11.Nc3 Nf6 12.Be3 Ne7
13.Kf2 c6 14.Re1 Bg7 15.e5 dxe5 16.dxe5 Nfd5
17.Bxd5 cxd5 18.Bc5 Bc6 19.b4 b6 20.Bxe7 Qxe7
21.Nfxd5 Qb7 22.Nf6+ Bxf6 23.exf6+ Kf8 24.Qd6+ Kg8
25.Re7 Qc8 26.Rc7 Qf5 27.Qxc6 Qxc2+ 28.Ke3 Rd8
29.Rd1 1-0

---------------------------------------------------------

Here is an example "BEFORE" and "AFTER" using PGNTRIM5.

This game was annotated by the computer program Fritz8.
There are many {[%emt 0:00:00]} elapsed-time remarks
which unfortunately use sqare braces within the movestext!!
Although these are also within curly brace pairs, using
[..] square braces within the moves text area is a violation
of common PGN good practice, if not the standard, itself.
PGNTRIM5 will remove these as shown in the example below.

[Event "Fritz8 commentary removal test file"]

[Site "Howie in the Hills, Florida"]
[Date "2004.05.28"]
[Round "?"]
[White "Fritz 8"]
[Black "McGillicuddy, Sean"]
[Result "1-0"]
[ECO "B06"]
[PlyCount "75"]
[Comment "Unfortunately, Fritz 8 also uses funky comment spacing"

{286MB, Fritz8.ctg, Intel 2.5 WinXP

} 1. Nf3 {[%emt 0:00:00]} g6 {
[%emt 0:00:00]} 2. e4 {[%emt 0:00:00]} Bg7 {[%emt 0:00:03]} 3. d4 {
[%emt 0:00:00]} d6 {[%emt 0:00:04]} 4. Nc3 {[%emt 0:00:00]} Nc6 {[%emt 0:00:12]
} 5. Bb5 {[%emt 0:00:01]} Bd7 {[%emt 0:00:02]} 6. O-O {[%emt 0:00:02]} a6 {
[%emt 0:00:05]} 7. Be2 {[%emt 0:00:01]} Bg4 {[%emt 0:00:17]} 8. Be3 {
[%emt 0:00:01]} Nf6 {[%emt 0:00:10]} 9. h3 {[%emt 0:00:02]} Bd7 {[%emt 0:00:04]
} 10. Qc1 {[%emt 0:00:01]} O-O {[%emt 0:00:25]} 11. Qb1 {[%emt 0:00:02]} e5 {
[%emt 0:00:23]} 12. dxe5 {[%emt 0:00:02]} dxe5 {[%emt 0:00:14]} 13. Kh1 {
[%emt 0:00:01]} Re8 {[%emt 0:00:14]} 14. a3 {[%emt 0:00:01]} b5 {[%emt 0:00:20]
} 15. Bc5 {[%emt 0:00:02]} Be6 {[%emt 0:00:12]} 16. Qc1 {[%emt 0:00:02]} Qc8 {
[%emt 0:00:10]} 17. Qd2 {[%emt 0:00:02]} Bxh3 {[%emt 0:00:19]} 18. gxh3 {
[%emt 0:00:05]} Qxh3+ {[%emt 0:00:02]} 19. Nh2 {[%emt 0:00:00]} Nd4 {
[%emt 0:00:14]} 20. Rfd1 {[%emt 0:00:04]} Rad8 {[%emt 0:00:09]} 21. Qd3 {
[%emt 0:00:03]} Qc8 {[%emt 0:00:37]} 22. b4 {[%emt 0:00:03]} h5 {[%emt 0:00:10]
} 23. Rac1 {[%emt 0:00:03]} Bh6 {[%emt 0:00:07]} 24. Rb1 {[%emt 0:00:04]} Bf4 {
[%emt 0:00:14]} 25. Bf1 {[%emt 0:00:02]} Kg7 {[%emt 0:00:10]} 26. a4 {
[%emt 0:00:05]} c6 {[%emt 0:00:06]} 27. Bg2 {[%emt 0:00:02]} Rh8 {
[%emt 0:00:31]} 28. Nf3 {[%emt 0:00:07]} h4 {[%emt 0:00:08]} 29. Ne2 {
[%emt 0:00:05]} h3 {[%emt 0:00:04]} 30. Nfxd4 {[%emt 0:00:02]} exd4 {
[%emt 0:00:09]} 31. Bf3 {[%emt 0:00:04]} Ng4 {[%emt 0:00:07]} 32. Bxg4 {
[%emt 0:00:02]} Qxg4 {[%emt 0:00:09]} 33. Rg1 {[%emt 0:00:04]} Qh4 {
[%emt 0:00:38]} 34. Bxd4+ {[%emt 0:00:09]} Kg8 {[%emt 0:00:19]} 35. Rbf1 {
[%emt 0:00:06]} Rh6 {[%emt 0:00:24]} 36. Ng3 {[%emt 0:00:03]} h2 {
[%emt 0:00:56]} 37. Rg2 {[%emt 0:00:02]} Be5 {[%emt 0:00:17]} 38. Nf5 {
[%emt 0:00:02]} 1-0

[Event "Fritz8 commentary removal test file"]

[Site "Howie in the Hills, Florida"]
[Date "2004.05.28"]
[Round "?"]
[White "Fritz 8"]
[Black "McGillicuddy, Sean"]
[Result "1-0"]
[ECO "B06"]
[PlyCount "75"]
[Comment "Unfortunately, Fritz 8 also uses funky comment spacing"

1.Nf3 g6 2.e4 Bg7 3.d4 d6 4.Nc3 Nc6

5.Bb5 Bd7 6.O-O a6 7.Be2 Bg4 8.Be3 Nf6
9.h3 Bd7 10.Qc1 O-O 11.Qb1 e5 12.dxe5 dxe5
13.Kh1 Re8 14.a3 b5 15.Bc5 Be6 16.Qc1 Qc8
17.Qd2 Bxh3 18.gxh3 Qxh3+ 19.Nh2 Nd4 20.Rfd1 Rad8
21.Qd3 Qc8 22.b4 h5 23.Rac1 Bh6 24.Rb1 Bf4
25.Bf1 Kg7 26.a4 c6 27.Bg2 Rh8 28.Nf3 h4
29.Ne2 h3 30.Nfxd4 exd4 31.Bf3 Ng4 32.Bxg4 Qxg4
33.Rg1 Qh4 34.Bxd4+ Kg8 35.Rbf1 Rh6 36.Ng3 h2
37.Rg2 Be5 38.Nf5 1-0