Detokeniser

//by Jon Ripley, January 2008, May 2011//

The //FNdetokenise// function listed below accepts a string containing either a tokenised program line or tokenised code and returns a string containing detokenised code: code format="bb4w" DEF FNdetokenise(S$) IF S$ = "" OR S$ = CHR$0+CHR$255+CHR$255 THEN ="" PRIVATE Keywd$ IF !^Keywd$=0 THEN DIM Keywd$(160) Keywd$ = "AND","DIV","EOR","MOD","OR","ERROR","LINE","OFF",\ \"STEP","SPC","TAB(","ELSE","THEN","","OPENIN","PTR",\       \"PAGE","TIME","LOMEM","HIMEM","ABS","ACS","ADVAL","ASC",\        \"ASN","ATN","BGET","COS","COUNT","DEG","ERL","ERR",\        \"EVAL","EXP","EXT","FALSE","FN","GET","INKEY","INSTR(",\ \"INT","LEN","LN","LOG","NOT","OPENUP","OPENOUT","PI",\ \"POINT(","POS","RAD","RND","SGN","SIN","SQR","TAN",\       \"TO","TRUE","USR","VAL","VPOS","CHR$","GET$","INKEY$",\        \"LEFT$(","MID$(","RIGHT$(","STR$","STRING$(","EOF","SUM","WHILE",\        \"CASE","WHEN","OF","ENDCASE","OTHERWISE","ENDIF","ENDWHILE","PTR",\        \"PAGE","TIME","LOMEM","HIMEM","SOUND","BPUT","CALL","CHAIN",\        \"CLEAR","CLOSE","CLG","CLS","DATA","DEF","DIM","DRAW",\        \"END","ENDPROC","ENVELOPE","FOR","GOSUB","GOTO","GCOL","IF",\        \"INPUT","LET","LOCAL","MODE","MOVE","NEXT","ON","VDU",\        \"PLOT","PRINT","PROC","READ","REM","REPEAT","REPORT","RESTORE",\        \"RETURN","RUN","STOP","COLOUR","TRACE","UNTIL","WIDTH","OSCLI",\        \"","CIRCLE","ELLIPSE","FILL","MOUSE","ORIGIN","QUIT","RECTANGLE",\        \"SWAP","SYS","TINT","WAIT","INSTALL","","PRIVATE","BY","EXIT"      ENDIF      LOCAL flag%, O$, i%, ch%      IF ASCS$ = LENS$ AND ASCRIGHT$(S$,1) = 13 THEN ch% = ASC MID$(S$, 2)+256*ASC MID$(S$, 3) IF ch% O$=STR$ch%+" " S$ = MID$(S$, 4, LENS$-4) IF S$="" THEN =LEFT$(O$, LENO$-1) ENDIF FOR i%=1 TO LENS$ IF ASCMID$(S$,i%)=34 flag%EOR=1 IF flag% AND 1 THEN O$ += MID$(S$, i%, 1) ELSE ch% = ASCMID$(S$, i%) CASE TRUE OF           WHEN ch% >= 17 AND ch% < 128:O$ += CHR$ch% WHEN ch% = &8D ch% = ASCMID$(S$,i%+1) lo% = ((ch% << 2) AND &C0) EOR ASC MID$(S$, i%+2) hi% = ((ch% << 4) AND &C0) EOR ASC MID$(S$, i%+3) O$+ = STR$(lo% + 256*hi%) i% += 3 OTHERWISE O$ += Keywd$(ch% EOR 128) ENDCASE ENDIF NEXT i%     =O$ code

Using FNdetokenise to list the error line
//FNdetokenise// is useful to list the source code of a line when one occurs.

Using FNdetokeniser to write a Quine
Using //FNdetokenise// it is possible to write a [|Quine], that is, a program which when run outputs its own listing: code format="bb4w" P% = PAGE WHILE (!P% AND &FFFFFF) <> &FFFF00 line$ = "" FOR i% = P% TO P%+?P%-1 line$ += CHR$?i% NEXT i%       PRINT FNdetokenise(line$) P% += ?P% ENDWHILE END code In the author's opinion using this method or ***LIST** to write a Quine is cheating.

Using the built-in detokeniser
The BBC BASIC for Windows interpreter has a built-in detokeniser accessed using the ***LIST** command that detokenises a specified file and outputs the listing on the current output device. Whilst it is possible to write a simpler looking detokeniser utilising ***LIST** the resulting routine is significantly slower due to the number of file operations required.

code format="bb4w" DEF FNdetokenise(S$) IF S$="" OR S$=CHR$0+CHR$255+CHR$255 THEN ="" IF LENS$>251 ERROR 100,"Line too long" LOCAL T%, fIn$, hFile%, hOut% DIM T% LOCAL 255 SYS "GetTempPath", 256, T%     SYS "GetTempFileName", $$T%, "tmp", 0, T%:fIn$=$$T% SYS "GetTempPath", 256, T%     SYS "GetTempFileName", $$T%, "tmp", 0, T%      hFile% = OPENOUT$$T%:IF hFile% = 0 ERROR 100,"Cannot create temp file" IF ASCS$=LENS$ AND ASC RIGHT$(S$,1)=13 THEN BPUT#hFile%, S$+CHR$0+CHR$255+CHR$255; ELSE BPUT#hFile%, CHR$(LEN(S$)+4)+CHR$0+CHR$0+S$+CHR$13+CHR$0+CHR$255+CHR$255; ENDIF CLOSE#hFile% hOut% = @vdu%?-130 hFile%=OPENUPfIn$:IF hFile%=0 ERROR 100,"Cannot create temp file" OSCLI "Output "+STR$hFile% OSCLI "List """+$$T%+"""" OSCLI "Output "+STR$hOut% PTR#hFile%=0:S$=GET$#hFile%:CLOSE#hFile% WHILE ASCS$=32:S$=MID$(S$,2):ENDWHILE SYS "DeleteFile", fIn$ SYS "DeleteFile", $$T% =S$ code


 * Do not use** this version in production code.