Flex Wiki Parser

Summary

Information about work in progress on FlexWiki parser

See_Also

Grammar meta language (there are a number of versions of the document marked Match expression syntax with various differences):

    // Match expression syntax
    Expr       ::= (ws* Choice)+ ws*                  
    Choice     ::= Bind (ws* '|' ws* Bind)*           // choice list
    Bind       ::= VarName '=' Factor                 // bind to factor semantic result
                 | Factor                             // no binding
    Factor     ::= Atom '??'                          // lazy zero or one
                 | Atom '*?'                          // lazy zero or more
                 | Atom '+?'                          // lazy one or more
                 | Atom '[' Int (',' ws* Int?)? ']?'  // lazy specified number
                 | Atom '?'                           // zero or one
                 | Atom '*'                           // zero or more
                 | Atom '+'                           // one or more
                 | Atom '[' Int (',' ws* Int?)? ']'   // specified number 
                 | Atom                               // one
    Atom       ::= Char '..' Char                     // char range
                 | String                             // two or more chars
                 | Char                               // single char
                 | SetName                            // named charset
                 | '_'                                // any char
                 | '^'                                // line start
                 | '$'                                // line end
                 | RuleName ('(' Params ')')?         // call another rule
                 | FuncCall 
                 | '(' Expr ')'                       // parentheses
   FuncCall    ::= '@' Name ('(' Params ')')?
   Params      ::= (Param (',' ws* Param)*)?
   Param       ::= VarName | Char | String
   Name        ::= ('A'..'Z' | 'a'..'z') {NC}* 
   VarName     ::= Name
   SetName     ::= '{' Name '}'
   Char        ::= '\'' ({SingleChar} | EscChar | HexChar | UnicodeChar) '\''
   String      ::= '\'' ({SingleChar} | EscChar | HexChar | UnicodeChar)[2,] '\''
   EscChar     ::= '\'' | '\"' | '\\' | '\0' | '\a' | '\b' | '\f' | '\n' | '\r' | '\t' | '\v'
   HexChar     ::= '\x' {HexDigit}[1,4]
   UnicodeChar ::= '\u' {HexDigit}[4]
                 | '\U' {HexDigit}[8]        
   Int         ::= {Digit}+
   ws          ::= ' ' | '\t' | '\r\n' | '\n' | '\r'

   // Character set definition set
   CharSet     ::= ws* Set ws* (('+' | '-') ws* Set ws*)* 
   Set         ::= SetName
                 | '[' ItemList ']'    
   SetItemList ::= SetItem (',' ws* SetItem)*
   SetItem     ::= Char '..' Char
                 | Char  
                 | '_'
 
   // Character sets 
   {NC}          = ['A'..'Z', 'a'..'z', '0'..'9', '_']
   {SingleChar}  = [_] - ['\'', '\\', '\n', '\r']
   {Digit}       = ['0'..'9']
   {HexDigit}    = ['0'..'9', 'A'..'F', 'a'..'f']

I (--JohnDavidson) believe the above is intended to be implemented by ParserExpressionReader, but that document has a number of inconsistencies (and sections that are commented out)

it is unclear if the comments in various DefineRule call are just a result of Test Driven Development or if they are considered redundant due to other rule definitions

I think they should become uncommented and result from Test Driven Development, but want confirmation

the ParserExpressionReaderTests::TestMatchSet returns CharSetName rather than the expected MatchSet

CharSetName is the correct value given the current logic, is this an error due to missing backtracking for a WomElement

Below is a sample of Flex Wiki grammar. It is far from completion and just shows some initial ideas:

<?xml version="1.0" encoding="utf-8" ?>
<wps:Grammar Name="FlexWiki" Version="1.0" xmlns:wps="urn:flexwiki-com:wom/2006/03">
  <wps:DefineSet Name="AZ" As="{Lu}" />
  <wps:DefineSet Name="az09" As="{Ll} + {Lt} + {Lo} + {Nd}" />
  <wps:DefineSet Name="Az09" As="{Lu} + {Ll} + {Lt} + {Lo} + {Nd}" />
  <wps:DefineSet Name="WordChar" As="['_', 'a'..'z', 'A'..'Z', '0'..'9']"/>
  <wps:DefineSet Name="NewLine" As="['\n', '\r']"/>
  <wps:DefineSet Name="LineChar" As="[_] - {NewLine}"/>

  <wps:StartTemplate>
    <wps:Choice>
      <wps:MatchTemplate Name="MultilineProperty"/>
      <wps:MatchTemplate Name="MultilineBehaviorProperty="/>
      <wps:MatchTemplate Name="Pre"/>
      <wps:MatchTemplate Name="HorizontalRule"/>
      <wps:MatchTemplate Name="PreLine"/>
      <wps:MatchTemplate Name="BulletList"/>
      <wps:MatchTemplate Name="Table"/>
      <wps:MatchTemplate Name="H7"/>
      <wps:MatchTemplate Name="H6"/>
      <wps:MatchTemplate Name="H5"/>
      <wps:MatchTemplate Name="H4"/>
      <wps:MatchTemplate Name="H3"/>
      <wps:MatchTemplate Name="H2"/>
      <wps:MatchTemplate Name="H1"/>
      <wps:MatchTemplate Name="Property"/>
      <wps:MatchTemplate Name="Paragraph"/>
    </wps:Choice>
  </wps:StartTemplate>

  <wps:Template Name="NamespaceName" Match="{AZ} {Az09}+" />
  <wps:Template Name="StartsWithMulticaps" Match="{AZ}[2,] {az09}+ {Az09}*" />
  <wps:Template Name="StartsWithOneCap" Match="{AZ} {az09}+ {Az09}*" />
  <wps:Template Name="UnbracketedWikiName" Match="'_'? StartsWithMulticaps | StartsWithOneCap" />
  <wps:Template Name="BracketedWikiName" Match="'[' ({Az09} | ' ')+ ']'" />
  <wps:Template Name="UnqualifiedWikiName" Match="UnbracketedWikiName | BracketedWikiName" />
  <wps:Template Name="QualifiedWikiName" Match="(NamespaceName '.' )* UnqualifiedWikiName" />
  <wps:Template Name="WikiName" Match="QualifiedWikiName | UnqualifiedWikiName | ForcedLocalWikiName" />
  <wps:Template Name="Identifier" Match="('A'..'Z'|'_') {WordChar}+"></wps:Template>

  <wps:Template Name="MultilineProperty" Match="^ leader=':'? Name=Identifier ':' '[' val=PropertyContent ']'"></wps:Template>
  <wps:Template Name="MultilineBehaviorProperty" Match="^ leader=':'? Name=Identifier ':' '{' val=Behavior '}'"></wps:Template>
  <wps:Template Name="Pre" Match="
    ^ '{@' preBlockKey={LineChar}* $ 
       content=_*? 
    ^ '}@' postBlockKey={LineChar}* $ 
    @Same(preBlockKey, LineChar)">
    <pre>
      <wps:ValueOf Select="content"/>
    </pre>
  </wps:Template>
  <wps:Template Name="HorizontalRule" Match="^ '----'">
    <hr/>
  </wps:Template>
  <wps:Template Name="IncludeTopic" Match="^ ['\t']* '{{' Name=WikiName '}}' {Space}* $"></wps:Template>
  <wps:Template Name="PreLine" Match="^ [' ','\t']+"></wps:Template>
  <wps:Template Name="BulletList" Match="^ indent=TAB+ '*' LineContent* $"></wps:Template>
  <wps:Template Name="BulletList" Match="^ indent=TAB+ '1.' LineContent* $"></wps:Template>
  <wps:Template Name="Table" Match="^ '||' (CellContent '||')+ $"></wps:Template>
  <wps:Template Name="H7" Match="^ '!!!!!!!' LineContent* $"></wps:Template>
  <wps:Template Name="H6" Match="^ '!!!!!!' LineContent* $"></wps:Template>
  <wps:Template Name="H5" Match="^ '!!!!!' LineContent* $"></wps:Template>
  <wps:Template Name="H4" Match="^ '!!!!' LineContent* $"></wps:Template>
  <wps:Template Name="H3" Match="^ '!!!' LineContent* $"></wps:Template>
  <wps:Template Name="H2" Match="^ '!!' LineContent* $"></wps:Template>
  <wps:Template Name="H1" Match="^ '!' LineContent* $"></wps:Template>
  <wps:Template Name="Property" Match="^ leader=':'? Name=Identifier ':' LineContent* $"></wps:Template>

  <wps:Template Name="Paragraph">
    <wps:LineStart/>
    <p>
      <wps:MatchTemplate Name="LineContent" Min="0" Max="unbounded"/>
    </p>
    <wps:LineEnd/>
  </wps:Template>

  <wps:Template Name="LineContent" Match="Behavior | Emoticon 
    | Deemphasis 
    | Textile('\'\'\'', 'strong') 
    | Textile('\'\'', 'em') 
    | Textile('_', 'em') 
    | Textile('*', 'strong') 
    | Textile('??', 'cite') 
    | Textile('-', 'del') 
    | Textile('+', 'ins')
    | Textile('^', 'sup') 
    | Textile('~', 'sub') 
    | Textile('@', 'code') 
    | Color 
    | HttpImage 
    | FileImage 
    | _"></wps:Template>

  <wps:Template Name="Behavior" Match="'@@' Behavior '@@'"></wps:Template>
  <wps:Template Name="Emoticon" Match="':)'">
    <img src="smile.gif"/>
  </wps:Template>
  <wps:Template Name="Deemphasis" Match="'`' c=LineContent+? '`'">
    <span class="Deemphasis">
      <wps:ValueOf Select="c"/>
    </span>
  </wps:Template>

  <wps:Template Name="Textile">
    <wps:Param Name="delimiter" />
    <wps:Param Name="tag" />
    <wps:Match Text="$delimiter"/>
    <wps:Element Name="$tag">
      <wps:MatchTemplate Name="LineContent" Min="0" Max="unbounded" Lazy="yes"/>
    </wps:Element>
    <wps:Match Text="$delimiter"/>
  </wps:Template>

  <wps:Template Name="Color" Match="'%' color or size '%' c=LineContent+? '%%'?">
    <wps:Call Name="Color">
      <wps:ValueOf Select="c"/>
    </wps:Call>
  </wps:Template>
  <wps:Template Name="HttpImage" Match="url=('http' 's'? '://' {NotSpace}* ('.jpg'|'.gif'|'.png'|'.jpeg'))">
    <img>
      <wps:Attribute Name="src">
        <wps:ValueOf Select="url"/>
      </wps:Attribute>
    </img>
  </wps:Template>
  <wps:Template Name="FileImage" Match="url=('file://' {NotSpace}* ('.jpg'|'.gif'|'.png'|'.jpeg'|'.doc'|'.xls'|'.ppt'|'.txt'))">
    <img>
      <wps:Attribute Name="src">
        <wps:ValueOf Select="url"/>
      </wps:Attribute>
    </img>
  </wps:Template>
</wps:Grammar>

Items for clarification in FlexWiki grammar xml for reasons of consistency (--JohnDavidson):

'As=' in DefineSet is never explained
'{Lu}' while understood to mean uppercase letter set is not related to any defined "CharSetName"

or is '{Lu}' the char set name

see above for '{Ll}', '{Lt}', '{Lo}' and '{Nd}'
would the code fragment below be a valid test (if not what should the assert be for the given Grammar input)

        [Test]
        public void TestRuleMatch()
        {
            StringReader textReader = new StringReader(
                @"<?xml version=""1.0"" encoding=""utf-8"" ?>
				  <wps:Grammar version=""1.0"" xmlns:wps=""urn:flexwiki-com:wom/2006/04"">
				    <wps:DefineSet Name="AZ" As="{Lu}" />
				  </wps:Grammar>");
            XmlTextReader xmlReader = new XmlTextReader(textReader);
            ParserXmlGrammarReader grammarReader = new ParserXmlGrammarReader(xmlReader);
            WomDocument grammar = grammarReader.Read();
            Assert.AreEqual(@"<Grammar><DefineSet Name=""AZ""><As Text=""{Lu}"" /></DefineSet></Grammar>",
                grammar.Xml);
        }

would the code fragment below be a valid test (if not what should the assert be for the given Grammar input)

        [Test]
        public void TestRuleMatch()
        {
            StringReader textReader = new StringReader(
                @"<?xml version=""1.0"" encoding=""utf-8"" ?>
				    <wps:Grammar version=""1.0"" xmlns:wps=""urn:flexwiki-com:wom/2006/04"">
				      <wps:StartTemplate>
    						<wps:Choice>
      						  <wps:MatchTemplate Name="MultilineProperty"/>
      						  <wps:MatchTemplate Name="MultilineBehaviorProperty="/>
      						  <wps:MatchTemplate Name="Pre"/>
      						  <wps:MatchTemplate Name="HorizontalRule"/>
    						</wps:Choice>
  				      </wps:StartTemplate>
				    </wps:Grammar>");
            XmlTextReader xmlReader = new XmlTextReader(textReader);
            ParserXmlGrammarReader grammarReader = new ParserXmlGrammarReader(xmlReader);
            WomDocument grammar = grammarReader.Read();
            Assert.AreEqual(@"<Grammar>
								<StartTemplate>
								  <Choice>
								    <Match Name=""MultilineProperty"">
								    <Match Name=""MultilineBehaviorProperty="">
								    <Match Name=""Pre"">
								    <Match Name=""HorizontalRule"">
								  </Choice>
								</StartTemplate>
						      </Grammar>",
                grammar.Xml);
            Assert.AreEqual("<Context><Choice><Match Text=\"a\" /><Match Text=\"b\" /></Choice></Context>",
                grammar.Xml);
        }

in the above why is there an equals sign in MultilineBehaviorProperty=
in the above should there not be a 'Name=' as an attribute to StartTemplate or is one not required because it is the only instance of StartTemplate
LineContent and CellContent are referenced in a Match constraint but not defined prior to use, are they inherent properties or is the Template Name="LineContent" below the first reference out of order (with no definition for CellContent)
In the Template Name="Paragraph" there are references to 'LineStart' and 'LineEnd' without prior definition - are these using the definitions from ParserExpressionReader

	Show Changes
	Edit
	Print
	Recent Changes
	Subscriptions
	Lost and Found
	Find References
	Rename
	Administration Page