<html>
<head>
<style><!--
body {background-color:#ffffff;}
.file {border:1px solid #eeeeee;margin-top:1em;margin-bottom:1em;}
.pathname {font-family:monospace; float:right;}
.fileheader {margin-bottom:.5em;}
.diff {margin:0;}
.tasklist {padding:4px;border:1px dashed #000000;margin-top:1em;}
.tasklist ul {margin-top:0;margin-bottom:0;}
tr.alt {background-color:#eeeeee}
#added {background-color:#ddffdd;}
#addedchars {background-color:#99ff99;font-weight:bolder;}
tr.alt #added {background-color:#ccf7cc;}
#removed {background-color:#ffdddd;}
#removedchars {background-color:#ff9999;font-weight:bolder;}
tr.alt #removed {background-color:#f7cccc;}
#copied {background-color:#ccccff;}
tr.alt #copied {background-color:#bbbbf7;}
#info {color:#888888;}
#context {background-color:#eeeeee;}
td {padding-left:.3em;padding-right:.3em;}
tr.head {border-bottom-width:1px;border-bottom-style:solid;}
tr.head td {padding:0;padding-top:.2em;}
.task {background-color:#ffff00;}
.comment {white-space:-moz-pre-wrap;white-space:-pre-wrap;white-space:-o-pre-wrap;white-space:pre-wrap;word-wrap:break-word;padding:4px;border:1px dashed #000000;background-color:#ffffdd}
.error {color:red;}
hr {border-width:0px;height:2px;background:black;}
--></style>
</head>
<body>
<table cellspacing="0" cellpadding="0" border="0" rules="cols">
<tr class="head"><td colspan="5">Commit in <b><tt>as2api/trunk/as2api</tt></b></td></tr>
<tr><td><tt>lib/<a href="#file1">api_loader.rb</a></tt> </td><td></td><td align="right" id="added">+2</td><td align="right" id="removed">-3</td><td nowrap="nowrap" align="center">376 -> 377</td></tr>
<tr class="alt"><td><tt>lib/output/html/<a href="#file2">sources.rb</a></tt> </td><td></td><td align="right" id="added">+1</td><td align="right" id="removed">-2</td><td nowrap="nowrap" align="center">376 -> 377</td></tr>
<tr><td><tt>lib/parse/<a href="#file3"><span id="removed">as_io.rb</span></a></tt> </td><td></td><td></td><td align="right" id="removed">-51</td><td nowrap="nowrap">376 removed</td></tr>
<tr class="alt"><td><tt> /<a href="#file4">aslexer.rb</a></tt> </td><td></td><td align="right" id="added">+28</td><td align="right" id="removed">-45</td><td nowrap="nowrap" align="center">376 -> 377</td></tr>
<tr><td><tt> /<a href="#file5">doccomment_lexer.rb</a></tt> </td><td></td><td align="right" id="added">+6</td><td align="right" id="removed">-6</td><td nowrap="nowrap" align="center">376 -> 377</td></tr>
<tr class="alt"><td><tt> /<a href="#file6">lexer.rb</a></tt> </td><td></td><td align="right" id="added">+9</td><td align="right" id="removed">-9</td><td nowrap="nowrap" align="center">376 -> 377</td></tr>
<tr><td><tt>test/<a href="#file7">tc_doc_comment.rb</a></tt> </td><td></td><td></td><td align="right" id="removed">-1</td><td nowrap="nowrap" align="center">376 -> 377</td></tr>
<tr class="alt"><td><tt> /<a href="#file8">tc_doccomment_lexer.rb</a></tt> </td><td></td><td align="right" id="added">+1</td><td></td><td nowrap="nowrap" align="center">376 -> 377</td></tr>
<tr><td></td><td></td><td align="right" id="added">+47</td><td align="right" id="removed">-117</td><td></td></tr>
</table>
<small id="info">1 removed + 7 modified, total 8 files</small><br />
<pre class="comment">
Slurp entire .as file into a StringScanner at once, rather than streaming it line-by-line.
This simplifies the code a bit, and speeds up lexing. Handling of #include
lines has been moved into the lexer (rather than beeing a preprocessing step)
to make this possible.
This change has the potential to degrade performance if the files to be loaded
are very 'large' (for some unspecified definition), but that seems unlikely to
be the case most of the time. I can't spot a 'streaming' alternative to
StringScanner.
If StringScanner is compatable with the Ruby mmap wrapper, making use of mmap,
when available, might well provide a further boost, and would make the lack of
streaming a non-issue.
</pre>
<hr /><a name="file1" /><div class="file">
<span class="pathname">as2api/trunk/as2api/lib</span><br />
<div class="fileheader"><big><b>api_loader.rb</b></big> <small id="info">376 -> 377</small></div>
<pre class="diff"><small id="info">--- trunk/as2api/lib/api_loader.rb 2006-04-07 18:13:25 UTC (rev 376)
+++ trunk/as2api/lib/api_loader.rb 2006-04-09 12:33:54 UTC (rev 377)
@@ -30,8 +30,7 @@
</small></pre><pre class="diff" id="context">
def simple_parse(input, source)
</pre><pre class="diff" id="removed">- as_io = ASIO.new(input)
- lex = ActionScript::Parse::ASLexer.new(as_io)
</pre><pre class="diff" id="added">+ lex = ActionScript::Parse::ASLexer.new(input)
</pre><pre class="diff" id="context"> lex.source = source
skip = DocASLexer.new(lex)
parse = DocASParser.new(skip)
</pre><pre class="diff"><small id="info">@@ -286,7 +285,7 @@
</small></pre><pre class="diff" id="context"> comment_data = CommentData.new
input = StringIO.new(comment_token.body)
</pre><pre class="diff" id="removed">- input.lineno = comment_token.lineno
</pre><pre class="diff" id="added">+ input.lineno = comment_token.lineno<span id="addedchars"> - 1</span>
</pre><pre class="diff" id="context"> lexer = ActionScript::ParseDoc::DocCommentLexer.new(input)
lexer.source = @source
parser = ActionScript::ParseDoc::DocCommentParser.new(lexer)
</pre></div>
<hr /><a name="file2" /><div class="file">
<span class="pathname">as2api/trunk/as2api/lib/output/html</span><br />
<div class="fileheader"><big><b>sources.rb</b></big> <small id="info">376 -> 377</small></div>
<pre class="diff"><small id="info">--- trunk/as2api/lib/output/html/sources.rb 2006-04-07 18:13:25 UTC (rev 376)
+++ trunk/as2api/lib/output/html/sources.rb 2006-04-09 12:33:54 UTC (rev 377)
@@ -197,9 +197,8 @@
</small></pre><pre class="diff" id="context"> File.open(File.join(file)) do |io|
begin
is_utf8 = detect_bom?(io)
</pre><pre class="diff" id="removed">- as_io = ASIO.new(io)
</pre><pre class="diff" id="context"> highlight = CodeHighlighter.new
</pre><pre class="diff" id="removed">- highlight.highlight(<span id="removedchars">as_</span>io, self)
</pre><pre class="diff" id="added">+ highlight.highlight(io, self)
</pre><pre class="diff" id="context"> rescue =>e
$stderr.puts "#{file}: #{e.message}\n#{e.backtrace.join("\n")}"
end
</pre></div>
<hr /><a name="file3" /><div class="file">
<span class="pathname" id="removed">as2api/trunk/as2api/lib/parse</span><br />
<div class="fileheader" id="removed"><big><b>as_io.rb</b></big> <small id="info">removed after 376</small></div>
<pre class="diff"><small id="info">--- trunk/as2api/lib/parse/as_io.rb 2006-04-07 18:13:25 UTC (rev 376)
+++ trunk/as2api/lib/parse/as_io.rb 2006-04-09 12:33:54 UTC (rev 377)
@@ -1,51 +0,0 @@
</small></pre><pre class="diff" id="removed">-#
-# Part of as2api - http://www.badgers-in-foil.co.uk/projects/as2api/
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-#
-
-
-# An IO-like object for reading from ActionScrpt source code.
-# It's job is to filter out #include directives, so that these don't need to
-# be handled at lexer or parser levels.
-class ASIO
- def initialize(io)
- @io = io
- end
-
- def eof?
- @io.eof?
- end
-
- def readline
- @io.each_line do |line|
- return line unless handle_directives(line)
- end
- end
-
- def lineno
- @io.lineno
- end
-
- private
-
- def handle_directives(line)
- if line =~ /\s*#include/
- # TODO: Implement #include. We just ignore, at the moment
- return true
- end
- return false
- end
-end
</pre></div>
<hr /><a name="file4" /><div class="file">
<span class="pathname">as2api/trunk/as2api/lib/parse</span><br />
<div class="fileheader"><big><b>aslexer.rb</b></big> <small id="info">376 -> 377</small></div>
<pre class="diff"><small id="info">--- trunk/as2api/lib/parse/aslexer.rb 2006-04-07 18:13:25 UTC (rev 376)
+++ trunk/as2api/lib/parse/aslexer.rb 2006-04-09 12:33:54 UTC (rev 377)
@@ -43,6 +43,9 @@
</small></pre><pre class="diff" id="context"> end
end
</pre><pre class="diff" id="added">+class IncludeToken < ASToken
+end
+
</pre><pre class="diff" id="context"> class WhitespaceToken < ASToken
end
</pre><pre class="diff"><small id="info">@@ -241,17 +244,17 @@
</small></pre><pre class="diff" id="context">
class ASLexer < AbstractLexer
</pre><pre class="diff" id="removed">- def lex_simple_token(class_sym, match, io)
- ActionScript::Parse.const_get(class_sym).new(io.lineno-1)
</pre><pre class="diff" id="added">+ def lex_simple_token(class_sym, match)
+ ActionScript::Parse.const_get(class_sym).new(@lineno)
</pre><pre class="diff" id="context"> end
</pre><pre class="diff" id="removed">- def lex_key_or_ident_token(match<span id="removedchars">, io</span>)
</pre><pre class="diff" id="added">+ def lex_key_or_ident_token(match)
</pre><pre class="diff" id="context"> body = match[0]
class_sym = @@keyword_tokens[body]
if class_sym
</pre><pre class="diff" id="removed">- lex_simple_token(class_sym, match<span id="removedchars">, io</span>)
</pre><pre class="diff" id="added">+ lex_simple_token(class_sym, match)
</pre><pre class="diff" id="context"> else
</pre><pre class="diff" id="removed">- lex_simplebody_token(:IdentifierToken, match<span id="removedchars">, io</span>)
</pre><pre class="diff" id="added">+ lex_simplebody_token(:IdentifierToken, match)
</pre><pre class="diff" id="context"> end
end
</pre><pre class="diff"><small id="info">@@ -259,53 +262,31 @@
</small></pre><pre class="diff" id="context"> @@keyword_tokens = toks
end
</pre><pre class="diff" id="removed">- def lex_simplebody_token(class_sym, match, io)
- ActionScript::Parse.const_get(class_sym).new(match[0], io.lineno-1)
</pre><pre class="diff" id="added">+ def lex_simplebody_token(class_sym, match)
+ ActionScript::Parse.const_get(class_sym).new(match[0], @lineno)
</pre><pre class="diff" id="context"> end
</pre><pre class="diff" id="removed">- def lex_singlelinecoomment_token(class_sym, match, io)
- SingleLineCommentToken.new(match[1], io.lineno-1)
</pre><pre class="diff" id="added">+ def lex_singlelinecoomment_token(class_sym, match)
+ SingleLineCommentToken.new(match[1], @lineno)
</pre><pre class="diff" id="context"> end
</pre><pre class="diff" id="removed">- def lex_multilinecomment_token(class_sym, match, io)
- lineno = io.lineno-1
- line = match.post_match
- comment = ''
- until line =~ /\*\//o
- comment << line
- line = io.readline;
- end
- comment << $`
- match.string = $'
- MultiLineCommentToken.new(comment, lineno)
</pre><pre class="diff" id="added">+ def lex_multilinecomment_token(class_sym, match)
+ lineno = @lineno
+ comment = match.scan_until(/\*\//o)
+ raise "#{@lineno}:unexpected EOF in comment" if comment.nil?
+ MultiLineCommentToken.new(comment[0, comment.length-2], lineno)
</pre><pre class="diff" id="context"> end
</pre><pre class="diff" id="removed">- def lex_string1_token(class_sym, match, io)
- lineno = io.lineno-1
- line = match.post_match
- str = ''
- until line =~ /#{STRING_END1}/o
- str << line
- line = io.readline;
- raise "#{lineno}:unexpected EOF in string" if line.nil?
- end
- str << $1
- match.string = $'
- StringToken.new(str, lineno)
</pre><pre class="diff" id="added">+ def lex_string1_token(class_sym, match)
+ str = match.scan_until(/#{STRING_END1}/o)
+ raise "#{@lineno}:unexpected EOF in string" if str.nil?
+ StringToken.new(str[0, str.length-1], @lineno)
</pre><pre class="diff" id="context"> end
</pre><pre class="diff" id="removed">- def lex_string2_token(class_sym, match, io)
- lineno = io.lineno-1
- line = match.post_match
- str = ''
- until line =~ /#{STRING_END2}/o
- str << line
- line = io.readline;
- raise "#{lineno}:unexpected EOF in string" if line.nil?
- end
- str << $1
- match.string = $'
- StringToken.new(str, lineno)
</pre><pre class="diff" id="added">+ def lex_string2_token(class_sym, match)
+ str = match.scan_until(/#{STRING_END2}/o)
+ raise "#{@lineno}:unexpected EOF in string" if str.nil?
+ StringToken.new(str[0, str.length-1], @lineno)
</pre><pre class="diff" id="context"> end
</pre><pre class="diff"><small id="info">@@ -318,6 +299,8 @@
</small></pre><pre class="diff" id="context"> # behaviour
builder.add_match(WHITESPACE, :lex_simplebody_token, :WhitespaceToken)
</pre><pre class="diff" id="added">+ builder.add_match("^#include [^\r\n]*", :lex_simplebody_token, :IncludeToken)
+
</pre><pre class="diff" id="context"> builder.add_match(SINGLE_LINE_COMMENT, :lex_singlelinecoomment_token, :SingleLineCommentToken)
builder.add_match(OMULTI_LINE_COMMENT, :lex_multilinecomment_token, :MultiLineCommentToken)
</pre><pre class="diff"><small id="info">@@ -376,7 +359,7 @@
</small></pre><pre class="diff" id="context"> protected
def skip?(tok)
</pre><pre class="diff" id="removed">- tok.is_a?(CommentToken) || tok.is_a?(WhitespaceToken)
</pre><pre class="diff" id="added">+ tok.is_a?(CommentToken) || tok.is_a?(WhitespaceToken)<span id="addedchars"> || tok.is_a?(IncludeToken)</span>
</pre><pre class="diff" id="context"> end
def notify(tok)
</pre></div>
<hr /><a name="file5" /><div class="file">
<span class="pathname">as2api/trunk/as2api/lib/parse</span><br />
<div class="fileheader"><big><b>doccomment_lexer.rb</b></big> <small id="info">376 -> 377</small></div>
<pre class="diff"><small id="info">--- trunk/as2api/lib/parse/doccomment_lexer.rb 2006-04-07 18:13:25 UTC (rev 376)
+++ trunk/as2api/lib/parse/doccomment_lexer.rb 2006-04-09 12:33:54 UTC (rev 377)
@@ -47,16 +47,16 @@
</small></pre><pre class="diff" id="context"> end
class DocCommentLexer < ActionScript::Parse::AbstractLexer
</pre><pre class="diff" id="removed">- def lex_simple_token(class_sym, match, io)
- ActionScript::ParseDoc.const_get(class_sym).new(io.lineno-1)
</pre><pre class="diff" id="added">+ def lex_simple_token(class_sym, match)
+ ActionScript::ParseDoc.const_get(class_sym).new(@lineno)
</pre><pre class="diff" id="context"> end
</pre><pre class="diff" id="removed">- def lex_simplebody_token(class_sym, match, io)
- ActionScript::ParseDoc.const_get(class_sym).new(match[0], io.lineno-1)
</pre><pre class="diff" id="added">+ def lex_simplebody_token(class_sym, match)
+ ActionScript::ParseDoc.const_get(class_sym).new(match[0], @lineno)
</pre><pre class="diff" id="context"> end
</pre><pre class="diff" id="removed">- def lex_simplecapture_token(class_sym, match, io)
- ActionScript::ParseDoc.const_get(class_sym).new(match[1], io.lineno-1)
</pre><pre class="diff" id="added">+ def lex_simplecapture_token(class_sym, match)
+ ActionScript::ParseDoc.const_get(class_sym).new(match[1], @lineno)
</pre><pre class="diff" id="context"> end
end
</pre></div>
<hr /><a name="file6" /><div class="file">
<span class="pathname">as2api/trunk/as2api/lib/parse</span><br />
<div class="fileheader"><big><b>lexer.rb</b></big> <small id="info">376 -> 377</small></div>
<pre class="diff"><small id="info">--- trunk/as2api/lib/parse/lexer.rb 2006-04-07 18:13:25 UTC (rev 376)
+++ trunk/as2api/lib/parse/lexer.rb 2006-04-09 12:33:54 UTC (rev 377)
@@ -33,9 +33,7 @@
</small></pre><pre class="diff" id="context"> def body
@body
end
</pre><pre class="diff" id="removed">- def lineno
- @lineno
- end
</pre><pre class="diff" id="added">+ attr_accessor :lineno
</pre><pre class="diff" id="context"> def to_s
@body
end
</pre><pre class="diff"><small id="info">@@ -50,6 +48,7 @@
</small></pre><pre class="diff" id="context"> @tokens = Array.new
@eof = false
@source = nil
</pre><pre class="diff" id="added">+ @lineno = io.lineno + 1
</pre><pre class="diff" id="context"> end
attr_accessor :source
</pre><pre class="diff"><small id="info">@@ -77,6 +76,7 @@
</small></pre><pre class="diff" id="context"> end
def emit(token)
</pre><pre class="diff" id="added">+ @lineno += token.body.scan(/\r\n|\r|\n/).length
</pre><pre class="diff" id="context"> token.source = @source
@tokens << token
end
</pre><pre class="diff"><small id="info">@@ -145,23 +145,23 @@
</small></pre><pre class="diff" id="context"> def build_lexer(target_class)
text = <<-EOS
def fill
</pre><pre class="diff" id="removed">- line = StringScanner.new(@io.readline)
- until line.eos?
</pre><pre class="diff" id="added">+ input = StringScanner.new(@io.read)
+ until input.eos?
</pre><pre class="diff" id="context"> EOS
@matches.each_with_index do |token_match, index|
re, lex_method, tok_class = token_match
</pre><pre class="diff" id="removed">- text << "if <span id="removedchars">line</span>.scan(/#{re}/)\n"
</pre><pre class="diff" id="added">+ text << "if <span id="addedchars">input</span>.scan(/#{re}/)\n"
</pre><pre class="diff" id="context"> if tok_class
</pre><pre class="diff" id="removed">- text << " emit(#{lex_method.to_s}(:#{tok_class.to_s}, <span id="removedchars">line, @io</span>))\n"
</pre><pre class="diff" id="added">+ text << " emit(#{lex_method.to_s}(:#{tok_class.to_s}, <span id="addedchars">input</span>))\n"
</pre><pre class="diff" id="context"> else
</pre><pre class="diff" id="removed">- text << " emit(#{lex_method.to_s}(<span id="removedchars">line, @io</span>))\n"
</pre><pre class="diff" id="added">+ text << " emit(#{lex_method.to_s}(<span id="addedchars">input</span>))\n"
</pre><pre class="diff" id="context"> end
text << " next\n"
text << "end\n"
end
text << <<-EOS
# no previous regexp matched,
</pre><pre class="diff" id="removed">- parse_error(<span id="removedchars">line</span>.rest)
</pre><pre class="diff" id="added">+ parse_error(<span id="addedchars">input</span>.rest)
</pre><pre class="diff" id="context"> end
end
EOS
</pre></div>
<hr /><a name="file7" /><div class="file">
<span class="pathname">as2api/trunk/as2api/test</span><br />
<div class="fileheader"><big><b>tc_doc_comment.rb</b></big> <small id="info">376 -> 377</small></div>
<pre class="diff"><small id="info">--- trunk/as2api/test/tc_doc_comment.rb 2006-04-07 18:13:25 UTC (rev 376)
+++ trunk/as2api/test/tc_doc_comment.rb 2006-04-09 12:33:54 UTC (rev 377)
@@ -104,7 +104,6 @@
</small></pre><pre class="diff" id="context"> comment_data = CommentData.new
input = StringIO.new(text)
</pre><pre class="diff" id="removed">- input.lineno = 1
</pre><pre class="diff" id="context"> lexer = ActionScript::ParseDoc::DocCommentLexer.new(input)
lexer.source = caller.last
parser = ActionScript::ParseDoc::DocCommentParser.new(lexer)
</pre></div>
<hr /><a name="file8" /><div class="file">
<span class="pathname">as2api/trunk/as2api/test</span><br />
<div class="fileheader"><big><b>tc_doccomment_lexer.rb</b></big> <small id="info">376 -> 377</small></div>
<pre class="diff"><small id="info">--- trunk/as2api/test/tc_doccomment_lexer.rb 2006-04-07 18:13:25 UTC (rev 376)
+++ trunk/as2api/test/tc_doccomment_lexer.rb 2006-04-09 12:33:54 UTC (rev 377)
@@ -65,6 +65,7 @@
</small></pre><pre class="diff" id="context"> tok = lex.get_next
assert_equal(expected.class, tok.class)
assert_equal(expected.body, tok.body)
</pre><pre class="diff" id="added">+ assert_equal(expected.lineno, tok.lineno)
</pre><pre class="diff" id="context"> end
assert_nil(lex.get_next)
end
</pre></div>
<center><small><a href="http://www.badgers-in-foil.co.uk/projects/cvsspam/" title="commit -> email">CVSspam</a> 0.2.12</small></center>
</body></html>