On Wed, Nov 5, 2008 at 8:45 AM, David Holroyd <span dir="ltr"><<a href="mailto:dave@badgers-in-foil.co.uk">dave@badgers-in-foil.co.uk</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I don't remember ever having seen something like this.<br>
<br>
Could it somehow be related to end-of-line-sequence handling (i.e. are<br>
extra windows-style carriage-return characters at the end of the line in<br>
question)?<br>
</blockquote><div><br>It doesn't seem to be the case.<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><br>
I seem to recall that the Ruby 'puts' method doesn't append a newline to<br>
the string if there's one there already -- maybe windows line endings<br>
cause this logic to foul up in one of the code paths?</blockquote><div><br>I'm not saying it's impossible, but it doesn't seem to me to be related: basically a line from one file gets mingled in the diff of another file.<br>
I'm more and more inclined to think that this is because of concurrent execution in background, but it's not clear to me how.<br> <br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Would be good to see the actual data, if possible? (If you want to send<br>
a private mail, I can treat it in confidence).</blockquote><div><br>I'll send you some data privately.<br><br>Here I can show the changes that I've made to collect_diffs.rb and cvsspam.rb, together with the shell script that is invoked when a commit is made (triggered by a configuration line in loginfo).<br>
<br><br><span style="font-family: courier new,monospace;">--- cvsspam-0.2.12/collect_diffs.rb.original 2008-10-17 12:07:37.000000000 -0400</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">+++ collect_diffs.rb 2008-10-31 12:29:07.000000000 -0400</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">@@ -147,7 +147,7 @@</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> raise "missing data dir (#{$tmpdir}/#{$dirtemplate}-XXXXXX)" if $datadir==nil</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">- line = $stdin.gets</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">+ line = $inputfile.gets</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> unless line =~ /^Update of (.+)/</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> fail "Log preamble looks suspect (doesn't start 'Update of ...')"</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> end</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">@@ -167,7 +167,7 @@</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> end</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> # look for the start of the user's comment</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">- $stdin.each do |line|</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">+ $inputfile.each do |line|</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> break if line =~ /^Log Message/</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> end</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">@@ -176,14 +176,14 @@</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> end</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> File.open("#{$datadir}/logfile", File::WRONLY|File::CREAT|File::APPEND) do |file|</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">- $stdin.each do |line|</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">+ $inputfile.each do |line|</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> # remove any trailing whitespace; we don't want the split() below to</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> # produce empty trailing items due to '\r' at the end of the line</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> # (if input is 'DOS' style),</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> line.sub!(/\s*$/, "")</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> # 'Mac' clients sending logs to a unix server may denote end-of-line with</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">- # a carriage-return, defeating the $stdin.each above (i.e. the whole log</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">+ # a carriage-return, defeating the $inputfile.each above (i.e. the whole log</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> # message will appear as a single line containing '\r's). We handle this</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> # case explicitly here, so that cvsspam.rb's Subject header generation</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> # doesn't break,</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">@@ -264,8 +264,19 @@</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> end</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> end</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">+ remove_inputfile()</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> end</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">+def remove_inputfile()</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">+ if $inputfile != $stdin</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">+ blah "Closing inputfile"</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">+ $inputfile.close</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">+ end</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">+ unless $filename.nil?</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">+ blah "If I wasn't debugging I'd have removed #{$filename}"</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">+ File.delete($filename) unless $debug</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">+ end</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">+end</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> # sometimes, CVS would exit with an error like,</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> #</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">@@ -334,7 +345,9 @@</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> $config = nil</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> $cvs_prog = "cvs"</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">-$debug = false</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">+$debug = true</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">+$inputfile = $stdin</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">+$filename = nil</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> $diff_ignore_keywords = false</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> $task_keywords = []</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">@@ -342,7 +355,6 @@</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> fail "$CVSROOT not defined. It should be when I am invoked from CVSROOT/loginfo"</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> end</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">-</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> def handle_operation?(args)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> # The CVS 1.12.x series pass an argument with the value "- New directory"</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> # whereas previous versions passed "some/path - New directory". The newer</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">@@ -360,12 +372,6 @@</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> end</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">-unless handle_operation?(ARGV)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">- consume_stdin()</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">- exit</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">-end</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">-</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">-</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> require 'getoptlong'</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> opts = GetoptLong.new(</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">@@ -373,7 +379,8 @@</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> [ "--config", "-c", GetoptLong::REQUIRED_ARGUMENT ],</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> [ "--debug", "-d", GetoptLong::NO_ARGUMENT ],</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> [ "--from", "-u", GetoptLong::REQUIRED_ARGUMENT ],</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">- [ "--charset", GetoptLong::REQUIRED_ARGUMENT ]</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">+ [ "--charset", GetoptLong::REQUIRED_ARGUMENT ],</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">+ [ "--file", "-f", GetoptLong::OPTIONAL_ARGUMENT ]</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> )</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> # arguments to pass though to 'cvsspam.rb'</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">@@ -387,6 +394,17 @@</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> end</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> $config = arg if opt=="--config"</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> $debug = true if opt == "--debug"</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">+ if opt == "--file"</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">+ $filename = arg</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">+ $inputfile = File.new(arg)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">+ blah "inputfile is #{$inputfile.path}"</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">+ end</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">+end</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">+</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">+unless handle_operation?(ARGV)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">+ consume_stdin()</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">+ remove_inputfile()</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">+ exit</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> end</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> blah("CVSROOT is #{ENV['CVSROOT']}")</span><br><br><br><br><br><br><br><br><br><br><span style="font-family: courier new,monospace;">--- cvsspam-0.2.12/cvsspam.rb.original 2008-10-17 12:40:22.000000000 -0400</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">+++ cvsspam.rb 2008-10-31 12:36:36.000000000 -0400</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">@@ -226,7 +226,9 @@</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> @line = @io.gets</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> return false if @line == nil</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> unless @line[0,1] == "#"</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">- raise "#{$logfile}:#{@io.lineno} line did not begin with '#': #{@line}"</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">+ raise "#{$logfile}:#{@io.lineno} line did not begin with '#': #{@line}" unless $debug</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">+ blah "#{$logfile}:#{@io.lineno} line did not begin with '#' => SKIPPING: #{@line}"</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">+ return advance</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> end</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> return true</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> end</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">@@ -1281,7 +1283,6 @@</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> $arg_charset = arg if opt == "--charset"</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> end</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">-</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> if ARGV.length != 1</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> if ARGV.length > 1</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> $stderr.puts "extra arguments not needed: #{ARGV[1, ARGV.length-1].join(', ')}"</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">@@ -1761,10 +1762,11 @@</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> unless from.address =~ /@/</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> from.address = "#{from.address}@#{$hostname}"</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> end</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">- smtp = Net::SMTP.new(@smtp_host)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">+ smtp = Net::SMTP.new(@smtp_host, 25)</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> blah("connecting to '#{@smtp_host}'")</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">- smtp.start()</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">- smtp.ready(from.address, recipients.map{|addr| addr.address}) do |mail|</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">+ #smtp.set_debug_output $stderr</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">+ smtp.start('__omissis__', '__omissis__', '__omissis__', :login)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">+ smtp.ready('__omissis__', recipients.map{|addr| addr.address}) do |mail|</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> ctx = MailContext.new(IOAdapter.new(mail))</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> ctx.header("To", recipients.map{|addr| addr.encoded}.join(','))</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> blah("Mail From: <#{from}>")</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">@@ -1772,6 +1774,7 @@</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> ctx.header("Date", Time.now.utc.strftime(DATE_HEADER_FORMAT))</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> yield ctx</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> end</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">+ smtp.finish</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> end</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> end</span><br style="font-family: courier new,monospace;"><br><br><br><br><br><br><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">[cvsspam]# cat collect_diffs_in_background.sh</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">#!/bin/bash</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">date >> $CVSROOT/collect_diffs.log</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">echo $USER >> $CVSROOT/collect_diffs.log</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">echo $1 >> $CVSROOT/collect_diffs.log</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">tmpfile="/tmp/tempfile.$$.${RANDOM%1000}"</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">cat > "$tmpfile"</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">echo "$tmpfile" >> $CVSROOT/collect_diffs.log</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">nohup /usr/local/lib/cvsspam/collect_diffs.rb --debug --file "$tmpfile" --from $USER "$@" >> $CVSROOT/collect_diffs.log 2>&1 &</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">[cvsspam]#</span><br style="font-family: courier new,monospace;"><br><br><br></div></div>So as you can see the differences are:<br>*) Upon commit the script collect_diffs_in_background.sh is called (the relevant line in loginfo looks like the following)<br>
ALL /usr/local/lib/cvsspam/collect_diffs_in_background.sh %{sVv}<br>*) I use SMTP with recipients configured in the conf file (I have omitted the credentials in the diff above, of course)<br>*) In order to collect input from cvs and pass it to collect_diffs.rb in background I use a temporary file with a random name ==> collect_diffs.rb then reads from the file rather than from stdin<br>
*) In debugging mode, if a line doesn't begin with '#' I simply skip to the next one<br><br>After avoiding to raise an exception and turning debugging on the issue happened again once. I'll send you the relevant files privately.<br>
<br>Thanks<br>Giuliano<br>