Bug 4119 - Malformed UTF-8 character problem in globus-ftp-client tests
: Malformed UTF-8 character problem in globus-ftp-client tests
Status: NEW
: GridFTP
GridFTP
: 4.0.1
: All All
: P3 normal
: ---
Assigned To:
:
:
:
:
  Show dependency treegraph
 
Reported: 2005-12-20 15:16 by
Modified: 2005-12-20 17:17 (History)


Attachments


Note

You need to log in before you can comment on or make changes to this bug.


Description From 2005-12-20 15:16:36
Some of the globus-ftp-client tests get "malformed UTF-8 character" errors when
they are run on certain systems with UTF-8 locales.  


I tried the patch that Charles suggested, and my results were similar to JJK's.
(i.e. extended-put reports "Wide character in print, <> line 1").


Then I changed compare_local_files() to use File::compare.  It seems to
eliminate the "malformed UTF-8 character" and "Wide character in print" errors.
I'm not sure why there are still failures, probably unrelated.

sub compare_local_files($$)
{
    my($a,$b) = @_;
    my $diffs;

    if (compare($a, $b) ne 0) {
                return "\n# Differences via File::compare between $a and $b.\n";
        } else {
                $diffs = "";
        }
    return $diffs;
}

This is the results:

Failed 7/18 test scripts, 61.11% okay. 879/2567 subtests failed, 65.76% okay.
Failed Test                       Stat Wstat Total Fail  Failed  List of Failed
-------------------------------------------------------------------------------
globus-ftp-client-caching-get-tes               89    1   1.12%  1
globus-ftp-client-caching-transfe               90    1   1.11%  1
globus-ftp-client-extended-get-te    1   256   873  873 100.00%  1-873
globus-ftp-client-get-test.pl                  124    1   0.81%  1
globus-ftp-client-multiple-block-               89    1   1.12%  1
globus-ftp-client-put-test.pl                  124    1   0.81%  1
globus-ftp-client-transfer-test.p               98    1   1.02%  1
 (2 subtests UNEXPECTEDLY SUCCEEDED).

My platform:
 cat /etc/redhat-release 
Red Hat Enterprise Linux WS release 3 (Taroon Update 5)

My locale:
 locale
LANG=en_US.UTF-8
...

Regards,
Tom Seelbach - Grid Development - seelbach@us.ibm.com

owner-discuss@globus.org wrote on 12/07/2005 06:24:43 PM:

> On Dec 7, 2005, at 5:12 PM, Charles Bacon wrote:
> 
> > --- FtpTestLib.pm.orig  2005-12-07 17:11:37.000000000 -0600
> > +++ FtpTestLib.pm       2005-12-07 17:11:51.000000000 -0600
> > @@ -142,7 +142,7 @@
> >      if(-B $a or -B $b)
> >      {
> > -       $diffs = `perl -pe 's/\\[restart plugin\\].*\\n//' < $b |  
> > (cmp '$a' - 2>&1 ) | sed -e 's/^/# /'`;
> > +       $diffs = `binmode(STDIN); perl -pe 's/\\[restart plugin\\].* 
> > \\n//' < $b | (cmp '$a' - 2>&1 ) | sed -e 's/^/# /'`;
> >      }
> >      else
> >      {
> 
> Ahem.  That's obviously a bad patch.
> +       $diffs = `perl -pe 'binmode(STDIN); s/\\[restart plugin\\].*\ 
> \n//' < $b | (cmp '$a' - 2>&1 ) | sed -e 's/^/# /'`;
> 
> I'm also not sure if you might also have to do that to the STDOUT for  
> the pipe.
> 
> 
> Charles
> 
> -
> To Unsubscribe: send mail to majordomo@globus.org
> with "unsubscribe discuss" in the body of the message
> 


More Details from the discuss@globus.org:


Jan Just Keijser wrote:

> Tried it, did not work. If I add only
>  binmode(STDIN);
> I see
>
> globus-ftp-client-extended-get-test..........dubious
>        Test returned status 1 (wstat 256, 0x100)
> DIED. FAILED tests 1-873
>        Failed 873/873 tests, 0.00% okay
> globus-ftp-client-extended-put-test..........# Test 1 got: '
> # Test failed
> # ./globus-ftp-client-extended-put-test -P 0 -d 
> gsiftp://localhost:45174/tmp/fileZYuX1J < /bin/sh >/dev/null 2>&1
> #
> # Differences between /bin/sh and /tmp/fileWlnxN2.
> # /bin/sh - differ: byte 609, line 2
>
> i.e. extended-get dies completely and the extended-put gives 
> differences; if I add
>  binmode(STDIN); binmode(STDOUT);
> then
> - extended-get still dies completely
> - extended-put now reports:
>
> Wide character in print, <> line 1.
>
> A quick and dirty hack
>  $diffs = `LC_ALL=C perl -pe ....
> seems to work for extended-put but the 'extended-get' test STILL dies, 
> although beats me why....
>
ah well that last part was easy: the script
  ./globus-ftp-client-extended-get-test.pl
was printing out
  wrong data
all the time because it uses its own compare_data function... Adding
  use open IN => ':raw';
to the script fixes it. Another way to get the extended-put-test to work 
is to simply redirect stderr to /dev/null; perl seems to be barfing about
  Malformed UTF-8 characters
but it still *does* write the (correct) output file. Adding
  binmode(STDIN);
and the likes seems to mess up the output file: it grows in size, for 
starters.

cheers,

JJK
------- Comment #1 From 2005-12-20 17:17:35 -------
That compare func won't work since any header data needs to be stripped out 
before comparing.

FYI, this only affects perl 5.8.0.  5.8.1 reverted to the pre-5.8.0 behavior of 
opening file handles.  See http://search.cpan.org/dist/perl/pod/perl581delta.pod

I've read over the thread on discuss and I have a machine I can reproduce this 
on, so I will look into it further.