From davide.cittaro at ifom-ieo-campus.it Thu Oct 4 03:24:18 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Thu, 4 Oct 2007 12:24:18 +0200 Subject: [Genome-mirror] track errors and deleted tracks Message-ID: Dear all, some users here expected strange errors while uploading a custom track. What happens, as described, is that if they try to upload a 10- column GFF file the track is in unknown format (and this is ok). They then delete a column (the 10th, after the group column) and re-upload it. It happens that they have this error: Internal error (Expecting at least 12 words line 388 of ../trash/ct/ ct_genome_616a_4b2820.bed got 11 ): removing custom tracks It happens that the previous upload has been somehow cached and the new upload makes something going wrong: all the tracks are deleted! This means that if one uploads 100 tracks and makes something wrong with the 101st, he has to do everything from the beginning... Some tests have been made: After this error: Error File 'perc_picchi_orc3_s95_10.txt' - Unrecognized format line 1 of custom track: chr19 NimbleScan 1881502_ratio.gff._0.95 40698 41022 1.55333333333333 + . 100 1.5533333333333 (note: chrom names are case sensitive) they have this error: Internal error (Expecting at least 12 words line 388 of ../trash/ct/ ct_genome_616a_4b2820.bed got 11 ): removing custom tracks once removed the 10th column. The same fatal error happens if one changes the file name or adds a description header. If one removes an additional column (resulting in a non recognized file): Internal error (Unrecognized format type=bed 12 . line 2 of ../trash/ ct/ct_genome_647c_4b4f20.bed ): removing custom tracks I personally believe it would be great if future releases will prevent deletion of all tracks in case an unattended error happens... Also, is there some workaround? Meaning some way to 'empty the cache'... d /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro at ifom-ieo-campus.it */ From kayla at soe.ucsc.edu Thu Oct 4 17:03:10 2007 From: kayla at soe.ucsc.edu (Kayla Smith) Date: Thu, 04 Oct 2007 17:03:10 -0700 Subject: [Genome-mirror] track errors and deleted tracks In-Reply-To: References: Message-ID: <47057F3E.3030303@cse.ucsc.edu> Hello Davide, Thank you for pointing out this error. I've passed this on to our developers and we will address this failure situation in the future. Kayla Smith UCSC Genome Bioinformatics Group Davide Cittaro wrote: > Dear all, > some users here expected strange errors while uploading a custom > track. What happens, as described, is that if they try to upload a 10- > column GFF file the track is in unknown format (and this is ok). They > then delete a column (the 10th, after the group column) and re-upload > it. It happens that they have this error: > > Internal error (Expecting at least 12 words line 388 of ../trash/ct/ > ct_genome_616a_4b2820.bed got 11 ): removing custom tracks > > It happens that the previous upload has been somehow cached and the > new upload makes something going wrong: all the tracks are deleted! > This means that if one uploads 100 tracks and makes something wrong > with the 101st, he has to do everything from the beginning... > Some tests have been made: > After this error: > Error File 'perc_picchi_orc3_s95_10.txt' - Unrecognized format line 1 > of custom track: chr19 NimbleScan 1881502_ratio.gff._0.95 40698 41022 > 1.55333333333333 + . 100 1.5533333333333 (note: chrom names are case > sensitive) > > they have this error: > Internal error (Expecting at least 12 words line 388 of ../trash/ct/ > ct_genome_616a_4b2820.bed got 11 ): removing custom tracks > once removed the 10th column. > The same fatal error happens if one changes the file name or adds a > description header. > If one removes an additional column (resulting in a non recognized > file): > Internal error (Unrecognized format type=bed 12 . line 2 of ../trash/ > ct/ct_genome_647c_4b4f20.bed ): removing custom tracks > > I personally believe it would be great if future releases will > prevent deletion of all tracks in case an unattended error happens... > Also, is there some workaround? Meaning some way to 'empty the cache'... > > d > > /* > Davide Cittaro > HPC and Bioinformatics Systems @ Informatics Core > > IFOM - Istituto FIRC di Oncologia Molecolare > via adamello, 16 > 20139 Milano > Italy > > tel.: +39(02)574303007 > e-mail: davide.cittaro at ifom-ieo-campus.it > */ > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror From kayla at soe.ucsc.edu Thu Oct 4 17:16:21 2007 From: kayla at soe.ucsc.edu (Kayla Smith) Date: Thu, 04 Oct 2007 17:16:21 -0700 Subject: [Genome-mirror] Notice of some mm9 chain/net data Message-ID: <47058255.3080501@cse.ucsc.edu> Hello Mirror Sites: There will be quite a bit of net and chain data involving the mm9 database coming out shortly, and some which has been recently released. In total there will be 30 new pairs of nets and chains between mouse (mm9) and other organisms, some of which are replacing mouse (mm8) data. Net/Chain data recently released: database: mm9 size: 8.2G (tables) 1G (files) database: canFam2 size: 2.5G (tables) 64M (files) database: oryLat1 size: 230M (tables) 3.6M (files) database: gasAcu1 size: 183M (tables) 3.4M (files) database: fr2 size: 167M (tables) 3.3M (files) database: monDom4 size: 5.2G (tables) 20M (files) total: ~17G Net/Chain data to be released soon: database: mm9 size: 13G (tables) 226M (files) database: bosTau3 size: 3.3G (tables) 55M (files) database: equCab1 size: 3.7G (tables) 66M (files) database: ornAna1 size: 1.1G (tables) 12M (files) database: panTro2 size: 3.1G (tables) 74M (files) database: galGal3 size: 576M (tables) 6.4M (files) database: anoCar1 size: 1.0G (tables) 6.7M (files) database: xenTro2 size: 1.9G (tables) 7.2M (files) Total: ~28G Please be prepared to receive this data if you choose. If you have any questions or concerns please don't hesitate to contact us. Thank You, Kayla Smith UCSC Genome Bioinformatics Group From kayla at soe.ucsc.edu Wed Oct 10 12:46:47 2007 From: kayla at soe.ucsc.edu (Kayla Smith) Date: Wed, 10 Oct 2007 12:46:47 -0700 Subject: [Genome-mirror] notice of release of new sea urchin assembly Message-ID: <470D2C27.7070203@cse.ucsc.edu> Mirror email for strPur2: Hello mirror sites, We are intending to release the new sea urchin assembly, strPur2, soon. Mirror sites should be prepared to host the following data: ~ 1.5G strPur2 tables ~ 2.1G /gbdb/strPur2/* files Total: ~ 3.6G of new data. Please let us know if you have any questions or concerns. Kayla Smith UCSC Genome Bioinformatics Group From kuhn at soe.ucsc.edu Wed Oct 10 18:03:45 2007 From: kuhn at soe.ucsc.edu (Robert Kuhn) Date: Wed, 10 Oct 2007 18:03:45 -0700 Subject: [Genome-mirror] mm9 UCSC Genes release upcoming Message-ID: <200710110103.SAA19604@moondance.cse.ucsc.edu> To the mirrors: We are about to release the UCSC Genes track, including the Gene Sorter, Proteome Browser, VisiGene and eight tracks of Expression data. The release includes some 80 Mb of files in /gbdb and 3.1 Gb of tables. thanks, --b0b kuhn ucsc genome bioinformatics group From bl246 at hermes.cam.ac.uk Fri Oct 12 06:21:41 2007 From: bl246 at hermes.cam.ac.uk (B. Liu) Date: Fri, 12 Oct 2007 14:21:41 +0100 (BST) Subject: [Genome-mirror] Funny Help Html Page In-Reply-To: <200710110103.SAA19604@moondance.cse.ucsc.edu> References: <200710110103.SAA19604@moondance.cse.ucsc.edu> Message-ID: Hi guys, Attention to http://genome.ucsc.edu/goldenPath/help/hgWiggleTrackHelp.html please. May be more, but I didn't check others. Thanks Bin From ann at soe.ucsc.edu Fri Oct 12 08:39:26 2007 From: ann at soe.ucsc.edu (Ann Zweig) Date: Fri, 12 Oct 2007 08:39:26 -0700 Subject: [Genome-mirror] Funny Help Html Page In-Reply-To: References: <200710110103.SAA19604@moondance.cse.ucsc.edu> Message-ID: <470F952E.9030502@soe.ucsc.edu> Hello Bin, I'm not exactly sure what you're referring to here. This page is linked to from many others -- it offers an explanation to users about how to configure wiggle graphs. Regards, ---------- Ann Zweig UCSC Genome Bioinformatics Group http://genome.ucsc.edu B. Liu wrote: > Hi guys, > > Attention to http://genome.ucsc.edu/goldenPath/help/hgWiggleTrackHelp.html please. May be more, but I didn't check others. > > Thanks > > Bin > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror From ann at soe.ucsc.edu Fri Oct 12 15:21:33 2007 From: ann at soe.ucsc.edu (Ann Zweig) Date: Fri, 12 Oct 2007 15:21:33 -0700 Subject: [Genome-mirror] upcoming release of danRer5 assembly Message-ID: <470FF36D.2060308@soe.ucsc.edu> Hello Mirror Sites, Next week, we will be releasing the latest zebrafish assembly (danRer5) to the public Genome Browser. Please be prepared to host approximately this much data: danRer5 database: 731MB files (/gbdb/danRer5/*): 682MB At the same time, we will be releasing nets and chains from four other databases to the danRer5 assembly. This will consist of the following data: fr2 database: 62MB tetNig1 database: 47MB mm9 database: 47MB oryLat1 database: 191MB Regards, Ann Zweig. From ann at soe.ucsc.edu Fri Oct 12 15:31:49 2007 From: ann at soe.ucsc.edu (Ann Zweig) Date: Fri, 12 Oct 2007 15:31:49 -0700 Subject: [Genome-mirror] upcoming release of danRer5 assembly In-Reply-To: <470FF36D.2060308@soe.ucsc.edu> References: <470FF36D.2060308@soe.ucsc.edu> Message-ID: <470FF5D5.4080104@soe.ucsc.edu> Hello again, Mirror Sites, Sorry, I used the wrong conversion factor. Here's an accounting of the real amount of upcoming data: danRer5 database: 5.7GB files (/gbdb/danRer5/*): 682MB fr2 database: 0.5GB tetNig1 database: 0.4GB mm9 database: 0.4GB oryLat1 database: 1.5GB Regards, Ann Zweig. Ann Zweig wrote: > Hello Mirror Sites, > > Next week, we will be releasing the latest zebrafish assembly (danRer5) to the > public Genome Browser. Please be prepared to host approximately this much data: > > danRer5 database: 731MB > files (/gbdb/danRer5/*): 682MB > > At the same time, we will be releasing nets and chains from four other > databases to the danRer5 assembly. This will consist of the following data: > > fr2 database: 62MB > tetNig1 database: 47MB > mm9 database: 47MB > oryLat1 database: 191MB > > > Regards, > Ann Zweig. > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror From bl246 at hermes.cam.ac.uk Mon Oct 15 01:56:22 2007 From: bl246 at hermes.cam.ac.uk (B. Liu) Date: Mon, 15 Oct 2007 09:56:22 +0100 (BST) Subject: [Genome-mirror] Funny Help Html Page In-Reply-To: <470F952E.9030502@soe.ucsc.edu> References: <200710110103.SAA19604@moondance.cse.ucsc.edu> <470F952E.9030502@soe.ucsc.edu> Message-ID: Hi Ann, Nothing serious to be honest, only because the topbar goes to the bottom of the page. Regards Bin On Fri, 12 Oct 2007, Ann Zweig wrote: > Hello Bin, > > I'm not exactly sure what you're referring to here. This page is > linked to from many others -- it offers an explanation to users about how to > configure wiggle graphs. > > > Regards, > > ---------- > Ann Zweig > UCSC Genome Bioinformatics Group > http://genome.ucsc.edu > > > B. Liu wrote: >> Hi guys, >> >> Attention to http://genome.ucsc.edu/goldenPath/help/hgWiggleTrackHelp.html >> please. May be more, but I didn't check others. >> >> Thanks >> >> Bin >> _______________________________________________ >> Genome-mirror mailing list >> Genome-mirror at soe.ucsc.edu >> http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From ann at soe.ucsc.edu Mon Oct 15 14:21:45 2007 From: ann at soe.ucsc.edu (Ann Zweig) Date: Mon, 15 Oct 2007 14:21:45 -0700 Subject: [Genome-mirror] Funny Help Html Page In-Reply-To: References: <200710110103.SAA19604@moondance.cse.ucsc.edu> <470F952E.9030502@soe.ucsc.edu> Message-ID: <4713D9E9.7070201@cse.ucsc.edu> Hello again, Bin, It sounds like you may be seeing something unusual -- my page looks just fine with the topbar at the top. Can you please send me a screen shot of the page as you see it, along with information about your o/s and Internet browser. Thanks, Ann Zweig. B. Liu wrote: > Hi Ann, > > Nothing serious to be honest, only because the topbar goes to the bottom > of the page. > > Regards > > Bin > > On Fri, 12 Oct 2007, Ann Zweig wrote: > >> Hello Bin, >> >> I'm not exactly sure what you're referring to here. This page is >> linked to from many others -- it offers an explanation to users about >> how to configure wiggle graphs. >> >> >> Regards, >> >> ---------- >> Ann Zweig >> UCSC Genome Bioinformatics Group >> http://genome.ucsc.edu >> >> >> B. Liu wrote: >>> Hi guys, >>> >>> Attention to >>> http://genome.ucsc.edu/goldenPath/help/hgWiggleTrackHelp.html please. >>> May be more, but I didn't check others. >>> >>> Thanks >>> >>> Bin >>> _______________________________________________ >>> Genome-mirror mailing list >>> Genome-mirror at soe.ucsc.edu >>> http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror >> From ann at soe.ucsc.edu Mon Oct 15 14:23:19 2007 From: ann at soe.ucsc.edu (Ann Zweig) Date: Mon, 15 Oct 2007 14:23:19 -0700 Subject: [Genome-mirror] you can drop some old tables pointing towards danRer4 Message-ID: <4713DA47.7060605@cse.ucsc.edu> Hello mirror sites, Once you have updated your assembly databases, you will be able to safely drop the following tables from the following databases: from fr2 database: *DanRer4* (5 tables) from tetNig1 database: *DanRer4* (55 tables) from oryLat1 database: *DanRer4* (53 tables) Regards, Ann Zweig. From bthenry at u.washington.edu Mon Oct 15 14:17:48 2007 From: bthenry at u.washington.edu (Brendan Henry) Date: Mon, 15 Oct 2007 14:17:48 -0700 Subject: [Genome-mirror] cgi compiling errors Message-ID: <4713D8FC.5070205@u.washington.edu> Hello, I am trying to compile cgi-bin executables for a mirror site of UCSC's browser. I notice that when running 'make compile' in kent/src/hg, there are errors which prevent successful building of important binaries such as 'hgTracks' while still exiting with a return code of 0. I am trying to compile CGI_VERSION "167" on GNU/Linux kernel 2.6.9-42.ELsmp on x86_64 architecture with gcc version 4.1.1. Here is sample output: gcc -O -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DMACHTYPE_x86_64 -DJK_WARN -Wall -Werror -I../inc -I../../inc -I../../../inc -I../../../../inc -I../../../../../inc -o correlate.o -c correlate.c cc1: warnings being treated as errors correlate.c: In function ?doCorrelateMore?: correlate.c:528: warning: ?groupList? may be used uninitialized in this function make[1]: *** [correlate.o] Error 1 make[1]: Leaving directory `/usr/local/jksrc/kent/src/hg/hgTables' Another example: gcc -O -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DMACHTYPE_x86_64 -DJK_WARN -Wall -Werror -I../inc -I../../inc -I../../../inc -I../../../../inc -I../../../../../inc -o expRatioTracks.o -c expRatioTracks.c cc1: warnings being treated as errors expRatioTracks.c: In function ?expRatioDrawItems?: expRatioTracks.c:1524: warning: ?pixCountArray? may be used uninitialized in this function expRatioTracks.c:1523: warning: ?pixScoreArray? may be used uninitialized in this function make: *** [expRatioTracks.o] Error 1 Any insight would be helpful! Thanks, Brendan Henry -- Brendan Henry Senior Computer Specialist Department of Genome Sciences University of Washington bthenry at u.washington.edu From m.pheasant at imb.uq.edu.au Tue Oct 16 03:54:18 2007 From: m.pheasant at imb.uq.edu.au (Michael Pheasant) Date: Tue, 16 Oct 2007 12:54:18 +0200 Subject: [Genome-mirror] Problems with slow RSYNC transfers from UCSC Message-ID: <44bb49500710160354sc3631dape9c8fa08763a6ed2@mail.gmail.com> Hi all We are trying to maintain a full (public) mirror in Australia including all 'download' data (rsync targets 'gbdb', 'mysql', and 'genome'), hosted at a university. The reason for this that Australian researchers get slow speeds of around 170KB/sec maximum downloading from UCSC, whereas they will get very fast download speeds from our mirror. It takes days or weeks to complete the 'mysql' and 'gbdb' targets and is a big problem taking even longer for the 'genome' target. We can get faster speeds from different US sites; for example, we can get around double that rate downloading data from a Stanford rsync server (similar TCP/IP route from AU), so it is not simply a problem with rsync or all traffic on our international link. I am now at a university in France, and initial testing shows that we get the same slow speeds to UCSC here, and again almost double the speed to Stanford, and as well we can get 1MB/s and more from an rsync server at SDSC (also on CENIC network) (I havent been able to test this from AU), so the slow rsync speed doesnt seem to be a problem limited to Australia. However, it seems that US-based servers get much greater speeds when doing rsync from UCSC - until a few years ago we rented a US-hosted server and were getting speeds of at least 500 KB/s between this server and UCSC and I suspect US academic peer networks will see even faster speeds. Furthermore, the overall throughput is not limited, just individual connections. For example, whether I have 1 or 5 rsync connections open, I get a maximum of 170KB/s on each rsync. This is also true of FTP. The slow download speeds are particularly a problem with the large files that are updated on a daily basis (such as daily EST genbank data and all its derived data) since we need to get everything synchronised between daily updates. (Files such as goldenPath/bigZips/*.fa.gz, goldenPath/database/{est,xenoMrna,gbStatus,...}.txt.gz, mysql/*/{est,xenoMrna,gbStatus,...}.MYD). The problem seems worse when rsyncing large compressed files, particularly rsync target 'genome' stuff (/bigZips/ & /database/) which is updated on a daily basis since we seem to end up downloading entire files for each update. When we download mysql/*MYD files which are updated daily (eg. tables with EST data) we generally get very fast results since updates appear to be appended to the file and rsync only needs to transfer the trailing changed portions. I note that with gzip there is a flag '--rsyncable Make rsync-friendly archive' which may help with this. Traceroutes I have done (see below) indicate the slow Australian and French routes to UCSC go through a common point (hpr-ucsc--svl-egm.cenic.net, the first host with UCSC in the name). However, the US server (faster) does not appears to use this route (the first route with UCSC in the name is ucsc-ucsc1--dc-oak-dc1-egm.cenic.net) (note that this is an old traceroute). Also, the French route to UCSC, Stanford, and SDSC all enter CENIC the same way; the difference in the routes is all between CENIC and the rsync hosts. This is also observed for the Australian routes (not shown). This seems to indicate that international TCPIP routes into UCSC follow different and slower paths than US based routes, and that the problem for international mirrors (at minimum, France and Australia) may be somewhere between the UCSC-CENIC interface for international (university) traffic ( i.e., aarnet.net.au/pacificwave.net for Australian universities and Geant2.net/Internet2.edu for French universities). I realise that any traceroutes I can do are not necessarily accurate representations of the route, but I think they are instructive when taken together with the rsync download speeds to different hosts. I would like to know if we are the only ones experiencing this problem: - Do any mirrors outside the US get faster rsync downloads than we do ? (more than say 170KB/s?) - Do any mirrors inside the US get slow downloads? (less than say 500KB/s?) Also; - would UCSC increase the max rsync clients (again!) say from 15 to 20 or more (or perhaps allow 5 or 6 or even more from the same IP address) so that international mirrors can run more rsyncs in parallel to overcome this slow per-connection speed? (in particular we want to do mysql/, gbdb/, genome/, genome/*/bigZips/ and genome/*/database/ in parallel) I believe this would in fact reduce the load on your servers and network since we would be able to complete an update in a day or two, rather than having just a few connections open for weeks on end continually refreshing old data waiting for everything to be in sync before the next daily update. - can we test the '--rsyncable' gzip flag when you create compressed files for rsync to see if it helps the mirrors (assuming you are not already using it)? Cheers Mike Pheasant ==== Very fast rsync: Traceroute from France to SDSC (wwwPDB)========= traceroute to rsync.wwpdb.org (198.202.122.181), 30 hops max, 40 byte packets 1 crc-rc1-ge-1-2-0-65.u-strasbg.fr (130.79.47.253) 0.915 ms 1.087 ms 1.276 ms 2 strasbourg-g3-3.cssi.renater.fr (193.51.184.42) 0.510 ms 0.537 ms 0.497 ms 3 nancy-pos2-0.cssi.renater.fr (193.51.180.41) 38.250 ms 38.344 ms 38.463 ms 4 reims-pos1-0.cssi.renater.fr (193.51.179.137) 8.412 ms 8.600 ms 8.883ms 5 * nri-b-pos6-0.cssi.renater.fr (193.51.179.149) 8.012 ms 7.967 ms 6 renater.rt1.par.fr.geant2.net (62.40.124.69) 8.181 ms 8.340 ms 8.301ms 7 so-7-3-0.rt1.gen.ch.geant2.net (62.40.112.29) 16.874 ms 16.862 ms 16.864 ms 8 so-7-2-0.rt1.fra.de.geant2.net (62.40.112.22) 24.941 ms 24.990 ms 24.942 ms 9 abilene-wash-gw.rt1.fra.de.geant2.net (62.40.125.18) 117.936 ms 117.883 ms 118.437 ms 10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 137.316 ms 137.302ms 137.220 ms 11 so-0-2-0.0.rtr.hous.net.internet2.edu (64.57.28.43) 154.646 ms 154.577ms 154.667 ms 12 so-3-0-0.0.rtr.losa.net.internet2.edu (64.57.28.44) 186.533 ms 187.578ms 187.560 ms 13 hpr-lax-hpr--i2-newnet.cenic.net (137.164.26.132) 186.753 ms 189.032ms * 14 riv-hpr--lax-hpr-10ge.cenic.net (137.164.25.5) 193.032 ms * * 15 hpr-sdsc-sdsc1--riv-hpr-ge.cenic.net (137.164.27.50) 206.951 ms 206.854 ms 206.840 ms 16 lightning.sdsc.edu (132.249.30.6) 192.749 ms 192.779 ms 192.774 ms ==== Fast rsync: Traceroute from France to Stanford========= traceroute to genome-ftp.stanford.edu (171.65.76.47), 30 hops max, 40 byte packets 1 crc-rc1-ge-1-2-0-65.u-strasbg.fr ( 130.79.47.253) 0.881 ms 0.980 ms 1.000 ms 2 strasbourg-g3-3.cssi.renater.fr ( 193.51.184.42) 0.500 ms 0.497 ms 0.472 ms 3 nancy-pos2-0.cssi.renater.fr ( 193.51.180.41) 8.336 ms 8.423 ms 8.569ms 4 reims-pos1-0.cssi.renater.fr (193.51.179.137) 8.600 ms 8.783 ms 8.896ms 5 nri-b-pos6-0.cssi.renater.fr ( 193.51.179.149) 7.972 ms 7.935 ms 7.904 ms 6 renater.rt1.par.fr.geant2.net ( 62.40.124.69) 8.213 ms 8.183 ms 8.266ms 7 so-7-3-0.rt1.gen.ch.geant2.net ( 62.40.112.29) 16.765 ms 16.866 ms 16.779 ms 8 so-7-2-0.rt1.fra.de.geant2.net (62.40.112.22) 25.118 ms 24.979 ms 24.922 ms 9 abilene-wash-gw.rt1.fra.de.geant2.net ( 62.40.125.18) 139.126 ms 139.163 ms 135.276 ms 10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 131.089 ms 131.084ms 131.095 ms 11 so-0-2-0.0.rtr.hous.net.internet2.edu ( 64.57.28.43) 154.667 ms 154.695 ms 154.517 ms 12 so-3-0-0.0.rtr.losa.net.internet2.edu (64.57.28.44) 186.611 ms 186.570ms 186.422 ms 13 hpr-lax-hpr--i2-newnet.cenic.net ( 137.164.26.132) 278.881 ms 269.707ms 266.963 ms 14 svl-hpr--lax-hpr-10ge.cenic.net (137.164.25.13 ) 191.474 ms 191.432 ms * 15 hpr-stan-ge--svl-hpr.cenic.net ( 137.164.27.162) 191.573 ms 191.851ms 191.622 ms 16 bbrb-i2.Stanford.EDU (171.64.1.136) 192.174 ms 191.888 ms 191.804 ms ==== Slow rsync: Traceroute from France to UCSC========= traceroute to hgdownload.cse.ucsc.edu ( 128.114.119.140), 30 hops max, 40 byte packets 1 crc-rc1-ge-1-2-0-65.u-strasbg.fr ( 130.79.47.253) 2.106 ms 2.075 ms 2.051 ms 2 strasbourg-g3-3.cssi.renater.fr ( 193.51.184.42) 1.466 ms 1.441 ms 1.400 ms 3 nancy-pos2-0.cssi.renater.fr ( 193.51.180.41) 15.768 ms 15.921 ms 16.097 ms 4 reims-pos1-0.cssi.renater.fr ( 193.51.179.137) 12.477 ms 12.603 ms 12.744 ms 5 nri-b-pos6-0.cssi.renater.fr ( 193.51.179.149) 8.546 ms 8.532 ms 8.491 ms 6 renater.rt1.par.fr.geant2.net ( 62.40.124.69) 8.819 ms 8.232 ms 8.194ms 7 so-7-3-0.rt1.gen.ch.geant2.net ( 62.40.112.29) 16.913 ms 16.815 ms 16.795 ms 8 so-7-2-0.rt1.fra.de.geant2.net ( 62.40.112.22) 24.893 ms 24.925 ms 25.057 ms 9 abilene-wash-gw.rt1.fra.de.geant2.net ( 62.40.125.18) 131.087 ms 130.878 ms 130.469 ms 10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 136.166 ms 133.322ms 133.242 ms 11 so-0-2-0.0.rtr.hous.net.internet2.edu (64.57.28.43 ) 154.750 ms 154.452 ms 154.438 ms 12 so-3-0-0.0.rtr.losa.net.internet2.edu ( 64.57.28.44) 197.920 ms 197.889 ms 197.847 ms 13 hpr-lax-hpr--i2-newnet.cenic.net ( 137.164.26.132) 183.765 ms * * 14 svl-hpr--lax-hpr-10ge.cenic.net ( 137.164.25.13) 191.428 ms 191.484 ms * 15 hpr-ucsc--svl-egm.cenic.net ( 137.164.27.86) 193.071 ms 193.075 ms 193.034 ms 16 comm-g-GE3-4.ucsc.edu (128.114.0.65) 193.016 ms 192.962 ms 192.941 ms 17 comm-d3-g-GE1-0-12.ucsc.edu ( 128.114.110.1) 193.267 ms 193.398 ms 193.267 ms 18 hgdownload.cse.ucsc.edu ( 128.114.119.140) 193.088 ms 192.974 ms 192.998 ms ==== Slow rsync: Traceroute from Australia to UCSC========= Tracing route to 128.114.119.140 1 ge-1-0-9.bb1.a.adl.aarnet.net.au ( 203.21.37.17) 0.627ms 0.497 ms 0.481 ms 2 so-0-1-0.bb1.a.mel.aarnet.net.au (202.158.194.18) 9.490ms 9.549 ms 9.488 ms 3 so-0-1-0.bb1.b.syd.aarnet.net.au (202.158.194.34) 21.494ms 21.808 ms 21.493 ms 4 pos1-0.bb1.b.sea.aarnet.net.au ( 202.158.194.94) 186.473 ms 186.177 ms 270.973 ms 5 cenichpr-1-is-std-779.snvaca.pacificwave.net (207.231.248.129 ) 204.455 ms 204.472 ms 204.459 ms 6 hpr-ucsc--svl-egm.cenic.net ( 137.164.27.86) 205.963 ms 205.785 ms 205.732 ms 7 isb-g-GE2-2.ucsc.edu ( 128.114.0.45) 206.094 ms 205.613 ms 205.963 ms 8 comm-d3-g-GE1-0-11.ucsc.edu (128.114.110.5) 206.450 ms 206.154 ms 205.959 ms 9 hgdownload.cse.ucsc.edu ( 128.114.119.140) 206.466 ms 205.583 ms 205.959 ms ==== Fast rsync: Traceroute from rented US-based server to UCSC=========== traceroute to hgdownload.cse.ucsc.edu ( 128.114.119.140), 30 hops max, 38 byte packets 1 1.87.1243.static.theplanet.com ( 67.18.135.1) 0.489 ms 0.465ms 0.508 ms 2 gi3-6.dsr02.dllstx4.theplanet.com ( 67.19.255.133) 0.375 ms 0.239ms 0.251 ms 3 vl42.dsr01.dllstx3.theplanet.com (70.85.127.89) 0.508 ms 0.765ms 0.633 ms 4 25.7f.5546.static.theplanet.com ( 70.85.127.37) 0.492 ms 0.361ms 0.382 ms 5 dal-ix.he.net (206.223.118.37) 0.507 ms 0.475ms 0.502 ms 6 pos5-0.gsr12012.lax.he.net ( 66.160.184.5) 35.030 ms 35.257ms 35.129 ms 7 lax-px1--hurricane-ge.cenic.net (198.32.251.85) 35.544 ms 35.388ms 35.420 ms 8 dc-sac-dc1--lax-dc1-pos.cenic.net ( 137.164.22.127) 45.014 ms 45.126ms 45.020 ms 9 dc-oak-dc1--csac-dc1-ge.cenic.net (137.164.22.110 ) 47.092 ms 46.942ms 46.840 ms 10 dc-oak-dc2--oak-dc1-p2p-2.cenic.net ( 137.164.22.195) 46.953 ms 46.813ms 46.969 ms 11 ucsc-ucsc1--dc-oak-dc1-egm.cenic.net ( 137.164.23.13) 47.595 ms 47.458ms 47.488 ms 12 comm-g-GE3-5.ucsc.edu (128.114.0.214) 47.479 ms 47.463ms 47.351 ms 13 comm-d3-g-GE1-0-12.ucsc.edu ( 128.114.110.1) 47.742 ms 47.586ms 47.616 ms 14 hgdownload.cse.ucsc.edu ( 128.114.119.140) 47.478 ms 47.465ms 47.475 ms From kuhn at soe.ucsc.edu Tue Oct 16 09:18:44 2007 From: kuhn at soe.ucsc.edu (Robert Kuhn) Date: Tue, 16 Oct 2007 09:18:44 -0700 Subject: [Genome-mirror] Problems with slow RSYNC transfers from UCSC Message-ID: <200710161618.JAA17815@moondance.cse.ucsc.edu> Hi, Mike, I am forwarding your message to our system admins. thanks a lot for your careful analysis. --b0b kuhn > From genome-mirror-bounces at soe.ucsc.edu Tue Oct 16 09:09:06 2007 > To: "genome mirror" > Subject: [Genome-mirror] Problems with slow RSYNC transfers from UCSC > > Hi all > > We are trying to maintain a full (public) mirror in Australia including all > 'download' data (rsync targets 'gbdb', 'mysql', and 'genome'), hosted at a > university. The reason for this that Australian researchers get slow speeds > of around 170KB/sec maximum downloading from UCSC, whereas they will get > very fast download speeds from our mirror. It takes days or weeks to > complete the 'mysql' and 'gbdb' targets and is a big problem taking even > longer for the 'genome' target. We can get faster speeds from different US > sites; for example, we can get around double that rate downloading data from > a Stanford rsync server (similar TCP/IP route from AU), so it is not simply > a problem with rsync or all traffic on our international link. > > I am now at a university in France, and initial testing shows that we get > the same slow speeds to UCSC here, and again almost double the speed to > Stanford, and as well we can get 1MB/s and more from an rsync server at SDSC > (also on CENIC network) (I havent been able to test this from AU), so the > slow rsync speed doesnt seem to be a problem limited to Australia. However, > it seems that US-based servers get much greater speeds when doing rsync from > UCSC - until a few years ago we rented a US-hosted server and were getting > speeds of at least 500 KB/s between this server and UCSC and I suspect US > academic peer networks will see even faster speeds. > > Furthermore, the overall throughput is not limited, just individual > connections. For example, whether I have 1 or 5 rsync connections open, I > get a maximum of 170KB/s on each rsync. This is also true of FTP. The slow > download speeds are particularly a problem with the large files that are > updated on a daily basis (such as daily EST genbank data and all its derived > data) since we need to get everything synchronised between daily updates. > (Files such as goldenPath/bigZips/*.fa.gz, > goldenPath/database/{est,xenoMrna,gbStatus,...}.txt.gz, > mysql/*/{est,xenoMrna,gbStatus,...}.MYD). > > The problem seems worse when rsyncing large compressed files, particularly > rsync target 'genome' stuff (/bigZips/ & /database/) which is updated on a > daily basis since we seem to end up downloading entire files for each > update. When we download mysql/*MYD files which are updated daily (eg. > tables with EST data) we generally get very fast results since updates > appear to be appended to the file and rsync only needs to transfer the > trailing changed portions. I note that with gzip there is a flag > '--rsyncable Make rsync-friendly archive' which may help with this. > > Traceroutes I have done (see below) indicate the slow Australian and French > routes to UCSC go through a common point (hpr-ucsc--svl-egm.cenic.net, the > first host with UCSC in the name). However, the US server (faster) does not > appears to use this route (the first route with UCSC in the name is > ucsc-ucsc1--dc-oak-dc1-egm.cenic.net) (note that this is an old traceroute). > Also, the French route to UCSC, Stanford, and SDSC all enter CENIC the same > way; the difference in the routes is all between CENIC and the rsync hosts. > This is also observed for the Australian routes (not shown). This seems to > indicate that international TCPIP routes into UCSC follow different and > slower paths than US based routes, and that the problem for international > mirrors (at minimum, France and Australia) may be somewhere between the > UCSC-CENIC interface for international (university) traffic ( i.e., > aarnet.net.au/pacificwave.net for Australian universities and > Geant2.net/Internet2.edu for French universities). I realise that any > traceroutes I can do are not necessarily accurate representations of the > route, but I think they are instructive when taken together with the rsync > download speeds to different hosts. > > > I would like to know if we are the only ones experiencing this problem: > > - Do any mirrors outside the US get faster rsync downloads than we do ? > (more than say 170KB/s?) > > - Do any mirrors inside the US get slow downloads? (less than say 500KB/s?) > > > Also; > > - would UCSC increase the max rsync clients (again!) say from 15 to 20 or > more (or perhaps allow 5 or 6 or even more from the same IP address) so that > international mirrors can run more rsyncs in parallel to overcome this slow > per-connection speed? (in particular we want to do mysql/, gbdb/, genome/, > genome/*/bigZips/ and genome/*/database/ in parallel) > I believe this would in fact reduce the load on your servers and network > since we would be able to complete an update in a day or two, rather than > having just a few connections open for weeks on end continually refreshing > old data waiting for everything to be in sync before the next daily update. > > - can we test the '--rsyncable' gzip flag when you create compressed files > for rsync to see if it helps the mirrors (assuming you are not already using > it)? > > > Cheers > > Mike Pheasant > > > ==== Very fast rsync: Traceroute from France to SDSC (wwwPDB)========= > > traceroute to rsync.wwpdb.org (198.202.122.181), 30 hops max, 40 byte > packets > 1 crc-rc1-ge-1-2-0-65.u-strasbg.fr (130.79.47.253) 0.915 ms 1.087 ms > 1.276 ms > 2 strasbourg-g3-3.cssi.renater.fr (193.51.184.42) 0.510 ms 0.537 ms > 0.497 ms > 3 nancy-pos2-0.cssi.renater.fr (193.51.180.41) 38.250 ms 38.344 ms > 38.463 ms > 4 reims-pos1-0.cssi.renater.fr (193.51.179.137) 8.412 ms 8.600 ms 8.883ms > 5 * nri-b-pos6-0.cssi.renater.fr (193.51.179.149) 8.012 ms 7.967 ms > 6 renater.rt1.par.fr.geant2.net (62.40.124.69) 8.181 ms 8.340 ms 8.301ms > 7 so-7-3-0.rt1.gen.ch.geant2.net (62.40.112.29) 16.874 ms 16.862 ms > 16.864 ms > 8 so-7-2-0.rt1.fra.de.geant2.net (62.40.112.22) 24.941 ms 24.990 ms > 24.942 ms > 9 abilene-wash-gw.rt1.fra.de.geant2.net (62.40.125.18) 117.936 ms > 117.883 ms 118.437 ms > 10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 137.316 ms 137.302ms > 137.220 ms > 11 so-0-2-0.0.rtr.hous.net.internet2.edu (64.57.28.43) 154.646 ms 154.577ms > 154.667 ms > 12 so-3-0-0.0.rtr.losa.net.internet2.edu (64.57.28.44) 186.533 ms 187.578ms > 187.560 ms > 13 hpr-lax-hpr--i2-newnet.cenic.net (137.164.26.132) 186.753 ms 189.032ms * > 14 riv-hpr--lax-hpr-10ge.cenic.net (137.164.25.5) 193.032 ms * * > 15 hpr-sdsc-sdsc1--riv-hpr-ge.cenic.net (137.164.27.50) 206.951 ms > 206.854 ms 206.840 ms > 16 lightning.sdsc.edu (132.249.30.6) 192.749 ms 192.779 ms 192.774 ms > > > ==== Fast rsync: Traceroute from France to Stanford========= > > traceroute to genome-ftp.stanford.edu (171.65.76.47), 30 hops max, 40 byte > packets > 1 crc-rc1-ge-1-2-0-65.u-strasbg.fr ( 130.79.47.253) 0.881 ms 0.980 ms > 1.000 ms > 2 strasbourg-g3-3.cssi.renater.fr ( 193.51.184.42) 0.500 ms 0.497 ms > 0.472 ms > 3 nancy-pos2-0.cssi.renater.fr ( 193.51.180.41) 8.336 ms 8.423 ms 8.569ms > 4 reims-pos1-0.cssi.renater.fr (193.51.179.137) 8.600 ms 8.783 ms 8.896ms > 5 nri-b-pos6-0.cssi.renater.fr ( 193.51.179.149) 7.972 ms 7.935 ms > 7.904 ms > 6 renater.rt1.par.fr.geant2.net ( 62.40.124.69) 8.213 ms 8.183 ms 8.266ms > 7 so-7-3-0.rt1.gen.ch.geant2.net ( 62.40.112.29) 16.765 ms 16.866 ms > 16.779 ms > 8 so-7-2-0.rt1.fra.de.geant2.net (62.40.112.22) 25.118 ms 24.979 ms > 24.922 ms > 9 abilene-wash-gw.rt1.fra.de.geant2.net ( 62.40.125.18) 139.126 ms > 139.163 ms 135.276 ms > 10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 131.089 ms 131.084ms > 131.095 ms > 11 so-0-2-0.0.rtr.hous.net.internet2.edu ( 64.57.28.43) 154.667 ms > 154.695 ms 154.517 ms > 12 so-3-0-0.0.rtr.losa.net.internet2.edu (64.57.28.44) 186.611 ms 186.570ms > 186.422 ms > 13 hpr-lax-hpr--i2-newnet.cenic.net ( 137.164.26.132) 278.881 ms 269.707ms > 266.963 ms > 14 svl-hpr--lax-hpr-10ge.cenic.net (137.164.25.13 ) 191.474 ms 191.432 ms > * > 15 hpr-stan-ge--svl-hpr.cenic.net ( 137.164.27.162) 191.573 ms 191.851ms > 191.622 ms > 16 bbrb-i2.Stanford.EDU (171.64.1.136) 192.174 ms 191.888 ms 191.804 ms > > > ==== Slow rsync: Traceroute from France to UCSC========= > > traceroute to hgdownload.cse.ucsc.edu ( 128.114.119.140), 30 hops max, 40 > byte packets > 1 crc-rc1-ge-1-2-0-65.u-strasbg.fr ( 130.79.47.253) 2.106 ms 2.075 ms > 2.051 ms > 2 strasbourg-g3-3.cssi.renater.fr ( 193.51.184.42) 1.466 ms 1.441 ms > 1.400 ms > 3 nancy-pos2-0.cssi.renater.fr ( 193.51.180.41) 15.768 ms 15.921 ms > 16.097 ms > 4 reims-pos1-0.cssi.renater.fr ( 193.51.179.137) 12.477 ms 12.603 ms > 12.744 ms > 5 nri-b-pos6-0.cssi.renater.fr ( 193.51.179.149) 8.546 ms 8.532 ms > 8.491 ms > 6 renater.rt1.par.fr.geant2.net ( 62.40.124.69) 8.819 ms 8.232 ms 8.194ms > 7 so-7-3-0.rt1.gen.ch.geant2.net ( 62.40.112.29) 16.913 ms 16.815 ms > 16.795 ms > 8 so-7-2-0.rt1.fra.de.geant2.net ( 62.40.112.22) 24.893 ms 24.925 ms > 25.057 ms > 9 abilene-wash-gw.rt1.fra.de.geant2.net ( 62.40.125.18) 131.087 ms > 130.878 ms 130.469 ms > 10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 136.166 ms 133.322ms > 133.242 ms > 11 so-0-2-0.0.rtr.hous.net.internet2.edu (64.57.28.43 ) 154.750 ms > 154.452 ms 154.438 ms > 12 so-3-0-0.0.rtr.losa.net.internet2.edu ( 64.57.28.44) 197.920 ms > 197.889 ms 197.847 ms > 13 hpr-lax-hpr--i2-newnet.cenic.net ( 137.164.26.132) 183.765 ms * * > 14 svl-hpr--lax-hpr-10ge.cenic.net ( 137.164.25.13) 191.428 ms 191.484 ms > * > 15 hpr-ucsc--svl-egm.cenic.net ( 137.164.27.86) 193.071 ms 193.075 ms > 193.034 ms > 16 comm-g-GE3-4.ucsc.edu (128.114.0.65) 193.016 ms 192.962 ms 192.941 ms > 17 comm-d3-g-GE1-0-12.ucsc.edu ( 128.114.110.1) 193.267 ms 193.398 ms > 193.267 ms > 18 hgdownload.cse.ucsc.edu ( 128.114.119.140) 193.088 ms 192.974 ms > 192.998 ms > > > ==== Slow rsync: Traceroute from Australia to UCSC========= > > Tracing route to 128.114.119.140 > 1 ge-1-0-9.bb1.a.adl.aarnet.net.au ( 203.21.37.17) 0.627ms > 0.497 ms 0.481 ms > 2 so-0-1-0.bb1.a.mel.aarnet.net.au (202.158.194.18) 9.490ms > 9.549 ms 9.488 ms > 3 so-0-1-0.bb1.b.syd.aarnet.net.au (202.158.194.34) 21.494ms > 21.808 ms 21.493 ms > 4 pos1-0.bb1.b.sea.aarnet.net.au ( 202.158.194.94) 186.473 ms > 186.177 ms 270.973 ms > 5 cenichpr-1-is-std-779.snvaca.pacificwave.net (207.231.248.129 ) 204.455 ms > 204.472 ms 204.459 ms > 6 hpr-ucsc--svl-egm.cenic.net ( 137.164.27.86) 205.963 ms > 205.785 ms 205.732 ms > 7 isb-g-GE2-2.ucsc.edu ( 128.114.0.45) 206.094 ms > 205.613 ms 205.963 ms > 8 comm-d3-g-GE1-0-11.ucsc.edu (128.114.110.5) 206.450 ms > 206.154 ms 205.959 ms > 9 hgdownload.cse.ucsc.edu ( 128.114.119.140) 206.466 ms > 205.583 ms 205.959 ms > > > ==== Fast rsync: Traceroute from rented US-based server to UCSC=========== > > traceroute to hgdownload.cse.ucsc.edu ( 128.114.119.140), 30 hops max, 38 > byte packets > 1 1.87.1243.static.theplanet.com ( 67.18.135.1) 0.489 ms > 0.465ms > 0.508 ms > 2 gi3-6.dsr02.dllstx4.theplanet.com ( 67.19.255.133) 0.375 ms > 0.239ms > 0.251 ms > 3 vl42.dsr01.dllstx3.theplanet.com (70.85.127.89) 0.508 ms 0.765ms > 0.633 ms > 4 25.7f.5546.static.theplanet.com ( 70.85.127.37) 0.492 ms > 0.361ms > 0.382 ms > 5 dal-ix.he.net (206.223.118.37) 0.507 ms 0.475ms > 0.502 ms > 6 pos5-0.gsr12012.lax.he.net ( 66.160.184.5) 35.030 ms 35.257ms > 35.129 ms > 7 lax-px1--hurricane-ge.cenic.net (198.32.251.85) 35.544 ms 35.388ms > 35.420 ms > 8 dc-sac-dc1--lax-dc1-pos.cenic.net ( 137.164.22.127) 45.014 ms 45.126ms > 45.020 ms > 9 dc-oak-dc1--csac-dc1-ge.cenic.net (137.164.22.110 ) 47.092 ms 46.942ms > 46.840 ms > 10 dc-oak-dc2--oak-dc1-p2p-2.cenic.net ( 137.164.22.195) 46.953 ms 46.813ms > 46.969 ms > 11 ucsc-ucsc1--dc-oak-dc1-egm.cenic.net ( 137.164.23.13) 47.595 ms 47.458ms > 47.488 ms > 12 comm-g-GE3-5.ucsc.edu (128.114.0.214) 47.479 ms 47.463ms > 47.351 ms > 13 comm-d3-g-GE1-0-12.ucsc.edu ( 128.114.110.1) 47.742 ms 47.586ms > 47.616 ms > 14 hgdownload.cse.ucsc.edu ( 128.114.119.140) 47.478 ms 47.465ms > 47.475 ms > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From hiram at soe.ucsc.edu Tue Oct 16 10:15:48 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Tue, 16 Oct 2007 10:15:48 -0700 Subject: [Genome-mirror] cgi compiling errors In-Reply-To: <4713D8FC.5070205@u.washington.edu> References: <4713D8FC.5070205@u.washington.edu> Message-ID: <4714F1C4.4030406@soe.ucsc.edu> Good Morning Brendan: Can you forward me the output of your 'gcc --version' command. I want to get these errors fixed in the source and would like to use the gcc version that exhibits this behavior. --Hiram Brendan Henry wrote: > Hello, > > I am trying to compile cgi-bin executables for a mirror site of UCSC's > browser. I notice that when running 'make compile' in kent/src/hg, there > are errors which prevent successful building of important binaries such > as 'hgTracks' while still exiting with a return code of 0. I am trying > to compile CGI_VERSION "167" on GNU/Linux kernel 2.6.9-42.ELsmp on > x86_64 architecture with gcc version 4.1.1. > > Here is sample output: > > gcc -O -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE > -DMACHTYPE_x86_64 -DJK_WARN -Wall -Werror -I../inc -I../../inc > -I../../../inc -I../../../../inc -I../../../../../inc -o correlate.o -c > correlate.c > cc1: warnings being treated as errors > correlate.c: In function ?doCorrelateMore?: > correlate.c:528: warning: ?groupList? may be used uninitialized in this > function > make[1]: *** [correlate.o] Error 1 > make[1]: Leaving directory `/usr/local/jksrc/kent/src/hg/hgTables' > > Another example: > > gcc -O -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE > -DMACHTYPE_x86_64 -DJK_WARN -Wall -Werror -I../inc -I../../inc > -I../../../inc -I../../../../inc -I../../../../../inc -o > expRatioTracks.o -c expRatioTracks.c > cc1: warnings being treated as errors > expRatioTracks.c: In function ?expRatioDrawItems?: > expRatioTracks.c:1524: warning: ?pixCountArray? may be used > uninitialized in this function > expRatioTracks.c:1523: warning: ?pixScoreArray? may be used > uninitialized in this function > make: *** [expRatioTracks.o] Error 1 > > Any insight would be helpful! > > Thanks, > Brendan Henry > From hiram at soe.ucsc.edu Tue Oct 16 11:34:03 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Tue, 16 Oct 2007 11:34:03 -0700 Subject: [Genome-mirror] cgi compiling errors In-Reply-To: <4713D8FC.5070205@u.washington.edu> References: <4713D8FC.5070205@u.washington.edu> Message-ID: <4715041B.7030007@soe.ucsc.edu> Thanks for the error report Brendan: I have fixed up the errors like this when using gcc 4.1.1 The public CVS server will have these changes in an hour or two. Also thanks to Michael Pheasant who pointed out these same errors earlier. If I had fixed them then, we would have been done with this sooner. --Hiram Brendan Henry wrote: > Hello, > > I am trying to compile cgi-bin executables for a mirror site of UCSC's > browser. I notice that when running 'make compile' in kent/src/hg, there > are errors which prevent successful building of important binaries such > as 'hgTracks' while still exiting with a return code of 0. I am trying > to compile CGI_VERSION "167" on GNU/Linux kernel 2.6.9-42.ELsmp on > x86_64 architecture with gcc version 4.1.1. From reiner.schulz at kcl.ac.uk Wed Oct 17 05:12:52 2007 From: reiner.schulz at kcl.ac.uk (Reiner Schulz) Date: Wed, 17 Oct 2007 13:12:52 +0100 Subject: [Genome-mirror] Problems with slow RSYNC transfers from UCSC In-Reply-To: <200710161618.JAA17815@moondance.cse.ucsc.edu> References: <200710161618.JAA17815@moondance.cse.ucsc.edu> Message-ID: <4715FC44.2030409@kcl.ac.uk> hi Mike, Robert, i am experiencing similarly slow rsync's of our partial mirror here at KCL, UK. but that's mostly due to KCL's limited bandwidth to the outside world in general (155Mb/s ATM). however, a while back i suggested half-jokingly that UCSC could perhaps take up google's offer to help w/ academic computing/storage: >>>>>>>>>> is the UCSC browser team planning to cooperate w/ google like the Archimedes Palimpsest people did so that perhaps one day one can have the entire UCSC browser drop-shipped instead of it trickling down the net? :) relevant refs: http://news.bbc.co.uk/2/hi/technology/6425975.stm http://searchstorage.techtarget.com/originalContent/0,289142,sid5_gci1246719,00.html <<<<<<<<<< i'd think that running an rsync service for UCSC's genome browser on google hardware falls within the same academic remit. has that possibility been explored? cheers, Reiner Robert Kuhn wrote: > Hi, Mike, > > I am forwarding your message to our system admins. thanks a lot > for your careful analysis. > > --b0b kuhn > >> From genome-mirror-bounces at soe.ucsc.edu Tue Oct 16 09:09:06 2007 >> To: "genome mirror" >> Subject: [Genome-mirror] Problems with slow RSYNC transfers from UCSC >> >> Hi all >> >> We are trying to maintain a full (public) mirror in Australia including all >> 'download' data (rsync targets 'gbdb', 'mysql', and 'genome'), hosted at a >> university. The reason for this that Australian researchers get slow speeds >> of around 170KB/sec maximum downloading from UCSC, whereas they will get >> very fast download speeds from our mirror. It takes days or weeks to >> complete the 'mysql' and 'gbdb' targets and is a big problem taking even >> longer for the 'genome' target. We can get faster speeds from different US >> sites; for example, we can get around double that rate downloading data from >> a Stanford rsync server (similar TCP/IP route from AU), so it is not simply >> a problem with rsync or all traffic on our international link. >> >> I am now at a university in France, and initial testing shows that we get >> the same slow speeds to UCSC here, and again almost double the speed to >> Stanford, and as well we can get 1MB/s and more from an rsync server at SDSC >> (also on CENIC network) (I havent been able to test this from AU), so the >> slow rsync speed doesnt seem to be a problem limited to Australia. However, >> it seems that US-based servers get much greater speeds when doing rsync from >> UCSC - until a few years ago we rented a US-hosted server and were getting >> speeds of at least 500 KB/s between this server and UCSC and I suspect US >> academic peer networks will see even faster speeds. >> >> Furthermore, the overall throughput is not limited, just individual >> connections. For example, whether I have 1 or 5 rsync connections open, I >> get a maximum of 170KB/s on each rsync. This is also true of FTP. The slow >> download speeds are particularly a problem with the large files that are >> updated on a daily basis (such as daily EST genbank data and all its derived >> data) since we need to get everything synchronised between daily updates. >> (Files such as goldenPath/bigZips/*.fa.gz, >> goldenPath/database/{est,xenoMrna,gbStatus,...}.txt.gz, >> mysql/*/{est,xenoMrna,gbStatus,...}.MYD). >> >> The problem seems worse when rsyncing large compressed files, particularly >> rsync target 'genome' stuff (/bigZips/ & /database/) which is updated on a >> daily basis since we seem to end up downloading entire files for each >> update. When we download mysql/*MYD files which are updated daily (eg. >> tables with EST data) we generally get very fast results since updates >> appear to be appended to the file and rsync only needs to transfer the >> trailing changed portions. I note that with gzip there is a flag >> '--rsyncable Make rsync-friendly archive' which may help with this. >> >> Traceroutes I have done (see below) indicate the slow Australian and French >> routes to UCSC go through a common point (hpr-ucsc--svl-egm.cenic.net, the >> first host with UCSC in the name). However, the US server (faster) does not >> appears to use this route (the first route with UCSC in the name is >> ucsc-ucsc1--dc-oak-dc1-egm.cenic.net) (note that this is an old traceroute). >> Also, the French route to UCSC, Stanford, and SDSC all enter CENIC the same >> way; the difference in the routes is all between CENIC and the rsync hosts. >> This is also observed for the Australian routes (not shown). This seems to >> indicate that international TCPIP routes into UCSC follow different and >> slower paths than US based routes, and that the problem for international >> mirrors (at minimum, France and Australia) may be somewhere between the >> UCSC-CENIC interface for international (university) traffic ( i.e., >> aarnet.net.au/pacificwave.net for Australian universities and >> Geant2.net/Internet2.edu for French universities). I realise that any >> traceroutes I can do are not necessarily accurate representations of the >> route, but I think they are instructive when taken together with the rsync >> download speeds to different hosts. >> >> >> I would like to know if we are the only ones experiencing this problem: >> >> - Do any mirrors outside the US get faster rsync downloads than we do ? >> (more than say 170KB/s?) >> >> - Do any mirrors inside the US get slow downloads? (less than say 500KB/s?) >> >> >> Also; >> >> - would UCSC increase the max rsync clients (again!) say from 15 to 20 or >> more (or perhaps allow 5 or 6 or even more from the same IP address) so that >> international mirrors can run more rsyncs in parallel to overcome this slow >> per-connection speed? (in particular we want to do mysql/, gbdb/, genome/, >> genome/*/bigZips/ and genome/*/database/ in parallel) >> I believe this would in fact reduce the load on your servers and network >> since we would be able to complete an update in a day or two, rather than >> having just a few connections open for weeks on end continually refreshing >> old data waiting for everything to be in sync before the next daily update. >> >> - can we test the '--rsyncable' gzip flag when you create compressed files >> for rsync to see if it helps the mirrors (assuming you are not already using >> it)? >> >> >> Cheers >> >> Mike Pheasant >> >> >> ==== Very fast rsync: Traceroute from France to SDSC (wwwPDB)========= >> >> traceroute to rsync.wwpdb.org (198.202.122.181), 30 hops max, 40 byte >> packets >> 1 crc-rc1-ge-1-2-0-65.u-strasbg.fr (130.79.47.253) 0.915 ms 1.087 ms >> 1.276 ms >> 2 strasbourg-g3-3.cssi.renater.fr (193.51.184.42) 0.510 ms 0.537 ms >> 0.497 ms >> 3 nancy-pos2-0.cssi.renater.fr (193.51.180.41) 38.250 ms 38.344 ms >> 38.463 ms >> 4 reims-pos1-0.cssi.renater.fr (193.51.179.137) 8.412 ms 8.600 ms 8.883ms >> 5 * nri-b-pos6-0.cssi.renater.fr (193.51.179.149) 8.012 ms 7.967 ms >> 6 renater.rt1.par.fr.geant2.net (62.40.124.69) 8.181 ms 8.340 ms 8.301ms >> 7 so-7-3-0.rt1.gen.ch.geant2.net (62.40.112.29) 16.874 ms 16.862 ms >> 16.864 ms >> 8 so-7-2-0.rt1.fra.de.geant2.net (62.40.112.22) 24.941 ms 24.990 ms >> 24.942 ms >> 9 abilene-wash-gw.rt1.fra.de.geant2.net (62.40.125.18) 117.936 ms >> 117.883 ms 118.437 ms >> 10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 137.316 ms 137.302ms >> 137.220 ms >> 11 so-0-2-0.0.rtr.hous.net.internet2.edu (64.57.28.43) 154.646 ms 154.577ms >> 154.667 ms >> 12 so-3-0-0.0.rtr.losa.net.internet2.edu (64.57.28.44) 186.533 ms 187.578ms >> 187.560 ms >> 13 hpr-lax-hpr--i2-newnet.cenic.net (137.164.26.132) 186.753 ms 189.032ms * >> 14 riv-hpr--lax-hpr-10ge.cenic.net (137.164.25.5) 193.032 ms * * >> 15 hpr-sdsc-sdsc1--riv-hpr-ge.cenic.net (137.164.27.50) 206.951 ms >> 206.854 ms 206.840 ms >> 16 lightning.sdsc.edu (132.249.30.6) 192.749 ms 192.779 ms 192.774 ms >> >> >> ==== Fast rsync: Traceroute from France to Stanford========= >> >> traceroute to genome-ftp.stanford.edu (171.65.76.47), 30 hops max, 40 byte >> packets >> 1 crc-rc1-ge-1-2-0-65.u-strasbg.fr ( 130.79.47.253) 0.881 ms 0.980 ms >> 1.000 ms >> 2 strasbourg-g3-3.cssi.renater.fr ( 193.51.184.42) 0.500 ms 0.497 ms >> 0.472 ms >> 3 nancy-pos2-0.cssi.renater.fr ( 193.51.180.41) 8.336 ms 8.423 ms 8.569ms >> 4 reims-pos1-0.cssi.renater.fr (193.51.179.137) 8.600 ms 8.783 ms 8.896ms >> 5 nri-b-pos6-0.cssi.renater.fr ( 193.51.179.149) 7.972 ms 7.935 ms >> 7.904 ms >> 6 renater.rt1.par.fr.geant2.net ( 62.40.124.69) 8.213 ms 8.183 ms 8.266ms >> 7 so-7-3-0.rt1.gen.ch.geant2.net ( 62.40.112.29) 16.765 ms 16.866 ms >> 16.779 ms >> 8 so-7-2-0.rt1.fra.de.geant2.net (62.40.112.22) 25.118 ms 24.979 ms >> 24.922 ms >> 9 abilene-wash-gw.rt1.fra.de.geant2.net ( 62.40.125.18) 139.126 ms >> 139.163 ms 135.276 ms >> 10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 131.089 ms 131.084ms >> 131.095 ms >> 11 so-0-2-0.0.rtr.hous.net.internet2.edu ( 64.57.28.43) 154.667 ms >> 154.695 ms 154.517 ms >> 12 so-3-0-0.0.rtr.losa.net.internet2.edu (64.57.28.44) 186.611 ms 186.570ms >> 186.422 ms >> 13 hpr-lax-hpr--i2-newnet.cenic.net ( 137.164.26.132) 278.881 ms 269.707ms >> 266.963 ms >> 14 svl-hpr--lax-hpr-10ge.cenic.net (137.164.25.13 ) 191.474 ms 191.432 ms >> * >> 15 hpr-stan-ge--svl-hpr.cenic.net ( 137.164.27.162) 191.573 ms 191.851ms >> 191.622 ms >> 16 bbrb-i2.Stanford.EDU (171.64.1.136) 192.174 ms 191.888 ms 191.804 ms >> >> >> ==== Slow rsync: Traceroute from France to UCSC========= >> >> traceroute to hgdownload.cse.ucsc.edu ( 128.114.119.140), 30 hops max, 40 >> byte packets >> 1 crc-rc1-ge-1-2-0-65.u-strasbg.fr ( 130.79.47.253) 2.106 ms 2.075 ms >> 2.051 ms >> 2 strasbourg-g3-3.cssi.renater.fr ( 193.51.184.42) 1.466 ms 1.441 ms >> 1.400 ms >> 3 nancy-pos2-0.cssi.renater.fr ( 193.51.180.41) 15.768 ms 15.921 ms >> 16.097 ms >> 4 reims-pos1-0.cssi.renater.fr ( 193.51.179.137) 12.477 ms 12.603 ms >> 12.744 ms >> 5 nri-b-pos6-0.cssi.renater.fr ( 193.51.179.149) 8.546 ms 8.532 ms >> 8.491 ms >> 6 renater.rt1.par.fr.geant2.net ( 62.40.124.69) 8.819 ms 8.232 ms 8.194ms >> 7 so-7-3-0.rt1.gen.ch.geant2.net ( 62.40.112.29) 16.913 ms 16.815 ms >> 16.795 ms >> 8 so-7-2-0.rt1.fra.de.geant2.net ( 62.40.112.22) 24.893 ms 24.925 ms >> 25.057 ms >> 9 abilene-wash-gw.rt1.fra.de.geant2.net ( 62.40.125.18) 131.087 ms >> 130.878 ms 130.469 ms >> 10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 136.166 ms 133.322ms >> 133.242 ms >> 11 so-0-2-0.0.rtr.hous.net.internet2.edu (64.57.28.43 ) 154.750 ms >> 154.452 ms 154.438 ms >> 12 so-3-0-0.0.rtr.losa.net.internet2.edu ( 64.57.28.44) 197.920 ms >> 197.889 ms 197.847 ms >> 13 hpr-lax-hpr--i2-newnet.cenic.net ( 137.164.26.132) 183.765 ms * * >> 14 svl-hpr--lax-hpr-10ge.cenic.net ( 137.164.25.13) 191.428 ms 191.484 ms >> * >> 15 hpr-ucsc--svl-egm.cenic.net ( 137.164.27.86) 193.071 ms 193.075 ms >> 193.034 ms >> 16 comm-g-GE3-4.ucsc.edu (128.114.0.65) 193.016 ms 192.962 ms 192.941 ms >> 17 comm-d3-g-GE1-0-12.ucsc.edu ( 128.114.110.1) 193.267 ms 193.398 ms >> 193.267 ms >> 18 hgdownload.cse.ucsc.edu ( 128.114.119.140) 193.088 ms 192.974 ms >> 192.998 ms >> >> >> ==== Slow rsync: Traceroute from Australia to UCSC========= >> >> Tracing route to 128.114.119.140 >> 1 ge-1-0-9.bb1.a.adl.aarnet.net.au ( 203.21.37.17) 0.627ms >> 0.497 ms 0.481 ms >> 2 so-0-1-0.bb1.a.mel.aarnet.net.au (202.158.194.18) 9.490ms >> 9.549 ms 9.488 ms >> 3 so-0-1-0.bb1.b.syd.aarnet.net.au (202.158.194.34) 21.494ms >> 21.808 ms 21.493 ms >> 4 pos1-0.bb1.b.sea.aarnet.net.au ( 202.158.194.94) 186.473 ms >> 186.177 ms 270.973 ms >> 5 cenichpr-1-is-std-779.snvaca.pacificwave.net (207.231.248.129 ) 204.455 ms >> 204.472 ms 204.459 ms >> 6 hpr-ucsc--svl-egm.cenic.net ( 137.164.27.86) 205.963 ms >> 205.785 ms 205.732 ms >> 7 isb-g-GE2-2.ucsc.edu ( 128.114.0.45) 206.094 ms >> 205.613 ms 205.963 ms >> 8 comm-d3-g-GE1-0-11.ucsc.edu (128.114.110.5) 206.450 ms >> 206.154 ms 205.959 ms >> 9 hgdownload.cse.ucsc.edu ( 128.114.119.140) 206.466 ms >> 205.583 ms 205.959 ms >> >> >> ==== Fast rsync: Traceroute from rented US-based server to UCSC=========== >> >> traceroute to hgdownload.cse.ucsc.edu ( 128.114.119.140), 30 hops max, 38 >> byte packets >> 1 1.87.1243.static.theplanet.com ( 67.18.135.1) 0.489 ms >> 0.465ms >> 0.508 ms >> 2 gi3-6.dsr02.dllstx4.theplanet.com ( 67.19.255.133) 0.375 ms >> 0.239ms >> 0.251 ms >> 3 vl42.dsr01.dllstx3.theplanet.com (70.85.127.89) 0.508 ms 0.765ms >> 0.633 ms >> 4 25.7f.5546.static.theplanet.com ( 70.85.127.37) 0.492 ms >> 0.361ms >> 0.382 ms >> 5 dal-ix.he.net (206.223.118.37) 0.507 ms 0.475ms >> 0.502 ms >> 6 pos5-0.gsr12012.lax.he.net ( 66.160.184.5) 35.030 ms 35.257ms >> 35.129 ms >> 7 lax-px1--hurricane-ge.cenic.net (198.32.251.85) 35.544 ms 35.388ms >> 35.420 ms >> 8 dc-sac-dc1--lax-dc1-pos.cenic.net ( 137.164.22.127) 45.014 ms 45.126ms >> 45.020 ms >> 9 dc-oak-dc1--csac-dc1-ge.cenic.net (137.164.22.110 ) 47.092 ms 46.942ms >> 46.840 ms >> 10 dc-oak-dc2--oak-dc1-p2p-2.cenic.net ( 137.164.22.195) 46.953 ms 46.813ms >> 46.969 ms >> 11 ucsc-ucsc1--dc-oak-dc1-egm.cenic.net ( 137.164.23.13) 47.595 ms 47.458ms >> 47.488 ms >> 12 comm-g-GE3-5.ucsc.edu (128.114.0.214) 47.479 ms 47.463ms >> 47.351 ms >> 13 comm-d3-g-GE1-0-12.ucsc.edu ( 128.114.110.1) 47.742 ms 47.586ms >> 47.616 ms >> 14 hgdownload.cse.ucsc.edu ( 128.114.119.140) 47.478 ms 47.465ms >> 47.475 ms >> _______________________________________________ >> Genome-mirror mailing list >> Genome-mirror at soe.ucsc.edu >> http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror >> > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror -- (*)->[]->()->[]->(**)->[]->()->[]->(*)->[]->()->[]->()->[]->()->[]->()->[] (Humboldt University Berlin, Germany)->[]-> ... (University of Maryland, USA)->[]-> ... (King's College London, UK) https://josh.umds.ac.uk/~rschulz From m.pheasant at imb.uq.edu.au Wed Oct 17 06:17:56 2007 From: m.pheasant at imb.uq.edu.au (Michael Pheasant) Date: Wed, 17 Oct 2007 15:17:56 +0200 Subject: [Genome-mirror] Problems with slow RSYNC transfers from UCSC In-Reply-To: <4715FC44.2030409@kcl.ac.uk> References: <200710161618.JAA17815@moondance.cse.ucsc.edu> <4715FC44.2030409@kcl.ac.uk> Message-ID: <44bb49500710170617p3d4be635u58a93a73e058b94@mail.gmail.com> Hi Reiner, You can do a simple test to see if the problem is just KCL bandwidth. Compare the speeds of downloading these two files: rsync -P --port=33444 rsync.wwpdb.org::ftp/ls-lR . rsync -P rsync://hgdownload.cse.ucsc.edu/genome/goldenPath/hg18/bigZips/upstream1000.zip . It would be intersting to see what max speeds you get from both. I can get over 1MB/s from rsync.wwpdb.org (SDSC, for me it seems to be on a similar route to UCSC) and 170Kb/s to UCSC. An italian mirror confirmed to me that they also get 170KB/s to UCSC. Cheers Mike On 10/17/07, Reiner Schulz wrote: > > hi Mike, Robert, > > i am experiencing similarly slow rsync's of our partial mirror here at > KCL, UK. but that's mostly due to KCL's limited bandwidth to the outside > world in general (155Mb/s ATM). > > however, a while back i suggested half-jokingly that UCSC could perhaps > take up google's offer to help w/ academic computing/storage: > > >>>>>>>>>> > is the UCSC browser team planning to cooperate w/ google like the > Archimedes Palimpsest people did so that perhaps one day one can have > the entire UCSC browser drop-shipped instead of it trickling down the > net? :) > relevant refs: > http://news.bbc.co.uk/2/hi/technology/6425975.stm > > http://searchstorage.techtarget.com/originalContent/0,289142,sid5_gci1246719,00.html > <<<<<<<<<< > > i'd think that running an rsync service for UCSC's genome browser on > google hardware falls within the same academic remit. has that > possibility been explored? > > cheers, > > Reiner > > Robert Kuhn wrote: > > Hi, Mike, > > > > I am forwarding your message to our system admins. thanks a lot > > for your careful analysis. > > > > --b0b kuhn > > > >> From genome-mirror-bounces at soe.ucsc.edu Tue Oct 16 09:09:06 2007 > >> To: "genome mirror" > >> Subject: [Genome-mirror] Problems with slow RSYNC transfers from UCSC > >> > >> Hi all > >> > >> We are trying to maintain a full (public) mirror in Australia including > all > >> 'download' data (rsync targets 'gbdb', 'mysql', and 'genome'), hosted > at a > >> university. The reason for this that Australian researchers get slow > speeds > >> of around 170KB/sec maximum downloading from UCSC, whereas they will > get > >> very fast download speeds from our mirror. It takes days or weeks to > >> complete the 'mysql' and 'gbdb' targets and is a big problem taking > even > >> longer for the 'genome' target. We can get faster speeds from different > US > >> sites; for example, we can get around double that rate downloading data > from > >> a Stanford rsync server (similar TCP/IP route from AU), so it is not > simply > >> a problem with rsync or all traffic on our international link. > >> > >> I am now at a university in France, and initial testing shows that we > get > >> the same slow speeds to UCSC here, and again almost double the speed to > >> Stanford, and as well we can get 1MB/s and more from an rsync server at > SDSC > >> (also on CENIC network) (I havent been able to test this from AU), so > the > >> slow rsync speed doesnt seem to be a problem limited to Australia. > However, > >> it seems that US-based servers get much greater speeds when doing rsync > from > >> UCSC - until a few years ago we rented a US-hosted server and were > getting > >> speeds of at least 500 KB/s between this server and UCSC and I suspect > US > >> academic peer networks will see even faster speeds. > >> > >> Furthermore, the overall throughput is not limited, just individual > >> connections. For example, whether I have 1 or 5 rsync connections open, > I > >> get a maximum of 170KB/s on each rsync. This is also true of FTP. The > slow > >> download speeds are particularly a problem with the large files that > are > >> updated on a daily basis (such as daily EST genbank data and all its > derived > >> data) since we need to get everything synchronised between daily > updates. > >> (Files such as goldenPath/bigZips/*.fa.gz, > >> goldenPath/database/{est,xenoMrna,gbStatus,...}.txt.gz, > >> mysql/*/{est,xenoMrna,gbStatus,...}.MYD). > >> > >> The problem seems worse when rsyncing large compressed files, > particularly > >> rsync target 'genome' stuff (/bigZips/ & /database/) which is updated > on a > >> daily basis since we seem to end up downloading entire files for each > >> update. When we download mysql/*MYD files which are updated daily (eg. > >> tables with EST data) we generally get very fast results since updates > >> appear to be appended to the file and rsync only needs to transfer the > >> trailing changed portions. I note that with gzip there is a flag > >> '--rsyncable Make rsync-friendly archive' which may help with this. > >> > >> Traceroutes I have done (see below) indicate the slow Australian and > French > >> routes to UCSC go through a common point (hpr-ucsc--svl-egm.cenic.net, > the > >> first host with UCSC in the name). However, the US server (faster) does > not > >> appears to use this route (the first route with UCSC in the name is > >> ucsc-ucsc1--dc-oak-dc1-egm.cenic.net) (note that this is an old > traceroute). > >> Also, the French route to UCSC, Stanford, and SDSC all enter CENIC the > same > >> way; the difference in the routes is all between CENIC and the rsync > hosts. > >> This is also observed for the Australian routes (not shown). This seems > to > >> indicate that international TCPIP routes into UCSC follow different and > >> slower paths than US based routes, and that the problem for > international > >> mirrors (at minimum, France and Australia) may be somewhere between the > >> UCSC-CENIC interface for international (university) traffic ( i.e., > >> aarnet.net.au/pacificwave.net for Australian universities and > >> Geant2.net/Internet2.edu for French universities). I realise that any > >> traceroutes I can do are not necessarily accurate representations of > the > >> route, but I think they are instructive when taken together with the > rsync > >> download speeds to different hosts. > >> > >> > >> I would like to know if we are the only ones experiencing this problem: > >> > >> - Do any mirrors outside the US get faster rsync downloads than we do ? > >> (more than say 170KB/s?) > >> > >> - Do any mirrors inside the US get slow downloads? (less than say > 500KB/s?) > >> > >> > >> Also; > >> > >> - would UCSC increase the max rsync clients (again!) say from 15 to 20 > or > >> more (or perhaps allow 5 or 6 or even more from the same IP address) so > that > >> international mirrors can run more rsyncs in parallel to overcome this > slow > >> per-connection speed? (in particular we want to do mysql/, gbdb/, > genome/, > >> genome/*/bigZips/ and genome/*/database/ in parallel) > >> I believe this would in fact reduce the load on your servers and > network > >> since we would be able to complete an update in a day or two, rather > than > >> having just a few connections open for weeks on end continually > refreshing > >> old data waiting for everything to be in sync before the next daily > update. > >> > >> - can we test the '--rsyncable' gzip flag when you create compressed > files > >> for rsync to see if it helps the mirrors (assuming you are not already > using > >> it)? > >> > >> > >> Cheers > >> > >> Mike Pheasant > >> > >> > >> ==== Very fast rsync: Traceroute from France to SDSC (wwwPDB)========= > >> > >> traceroute to rsync.wwpdb.org (198.202.122.181), 30 hops max, 40 byte > >> packets > >> 1 crc-rc1-ge-1-2-0-65.u-strasbg.fr (130.79.47.253) 0.915 ms 1.087ms > >> 1.276 ms > >> 2 strasbourg-g3-3.cssi.renater.fr (193.51.184.42) 0.510 ms 0.537 ms > >> 0.497 ms > >> 3 nancy-pos2-0.cssi.renater.fr (193.51.180.41) 38.250 ms 38.344 ms > >> 38.463 ms > >> 4 reims-pos1-0.cssi.renater.fr (193.51.179.137) 8.412 ms 8.600 ms > 8.883ms > >> 5 * nri-b-pos6-0.cssi.renater.fr (193.51.179.149) 8.012 ms 7.967 ms > >> 6 renater.rt1.par.fr.geant2.net (62.40.124.69) 8.181 ms 8.340 ms > 8.301ms > >> 7 so-7-3-0.rt1.gen.ch.geant2.net (62.40.112.29) 16.874 ms 16.862 ms > >> 16.864 ms > >> 8 so-7-2-0.rt1.fra.de.geant2.net (62.40.112.22) 24.941 ms 24.990 ms > >> 24.942 ms > >> 9 abilene-wash-gw.rt1.fra.de.geant2.net (62.40.125.18) 117.936 ms > >> 117.883 ms 118.437 ms > >> 10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 137.316 ms > 137.302ms > >> 137.220 ms > >> 11 so-0-2-0.0.rtr.hous.net.internet2.edu (64.57.28.43) 154.646 ms > 154.577ms > >> 154.667 ms > >> 12 so-3-0-0.0.rtr.losa.net.internet2.edu (64.57.28.44) 186.533 ms > 187.578ms > >> 187.560 ms > >> 13 hpr-lax-hpr--i2-newnet.cenic.net (137.164.26.132) 186.753 ms > 189.032ms * > >> 14 riv-hpr--lax-hpr-10ge.cenic.net (137.164.25.5) 193.032 ms * * > >> 15 hpr-sdsc-sdsc1--riv-hpr-ge.cenic.net (137.164.27.50) 206.951 ms > >> 206.854 ms 206.840 ms > >> 16 lightning.sdsc.edu (132.249.30.6) 192.749 ms 192.779 ms 192.774ms > >> > >> > >> ==== Fast rsync: Traceroute from France to Stanford========= > >> > >> traceroute to genome-ftp.stanford.edu (171.65.76.47), 30 hops max, 40 > byte > >> packets > >> 1 crc-rc1-ge-1-2-0-65.u-strasbg.fr ( 130.79.47.253) 0.881 ms 0.980ms > >> 1.000 ms > >> 2 strasbourg-g3-3.cssi.renater.fr ( 193.51.184.42) 0.500 ms 0.497ms > >> 0.472 ms > >> 3 nancy-pos2-0.cssi.renater.fr ( 193.51.180.41) 8.336 ms 8.423 ms > 8.569ms > >> 4 reims-pos1-0.cssi.renater.fr (193.51.179.137) 8.600 ms 8.783 ms > 8.896ms > >> 5 nri-b-pos6-0.cssi.renater.fr ( 193.51.179.149) 7.972 ms 7.935 ms > >> 7.904 ms > >> 6 renater.rt1.par.fr.geant2.net ( 62.40.124.69) 8.213 ms 8.183 ms > 8.266ms > >> 7 so-7-3-0.rt1.gen.ch.geant2.net ( 62.40.112.29) 16.765 ms 16.866ms > >> 16.779 ms > >> 8 so-7-2-0.rt1.fra.de.geant2.net (62.40.112.22) 25.118 ms 24.979 ms > >> 24.922 ms > >> 9 abilene-wash-gw.rt1.fra.de.geant2.net ( 62.40.125.18) 139.126 ms > >> 139.163 ms 135.276 ms > >> 10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 131.089 ms > 131.084ms > >> 131.095 ms > >> 11 so-0-2-0.0.rtr.hous.net.internet2.edu ( 64.57.28.43) 154.667 ms > >> 154.695 ms 154.517 ms > >> 12 so-3-0-0.0.rtr.losa.net.internet2.edu (64.57.28.44) 186.611 ms > 186.570ms > >> 186.422 ms > >> 13 hpr-lax-hpr--i2-newnet.cenic.net ( 137.164.26.132) 278.881 ms > 269.707ms > >> 266.963 ms > >> 14 svl-hpr--lax-hpr-10ge.cenic.net (137.164.25.13 ) 191.474 ms > 191.432 ms > >> * > >> 15 hpr-stan-ge--svl-hpr.cenic.net ( 137.164.27.162) 191.573 ms > 191.851ms > >> 191.622 ms > >> 16 bbrb-i2.Stanford.EDU (171.64.1.136) 192.174 ms 191.888 ms > 191.804 ms > >> > >> > >> ==== Slow rsync: Traceroute from France to UCSC========= > >> > >> traceroute to hgdownload.cse.ucsc.edu ( 128.114.119.140), 30 hops max, > 40 > >> byte packets > >> 1 crc-rc1-ge-1-2-0-65.u-strasbg.fr ( 130.79.47.253) 2.106 ms 2.075ms > >> 2.051 ms > >> 2 strasbourg-g3-3.cssi.renater.fr ( 193.51.184.42) 1.466 ms 1.441ms > >> 1.400 ms > >> 3 nancy-pos2-0.cssi.renater.fr ( 193.51.180.41) 15.768 ms 15.921 ms > >> 16.097 ms > >> 4 reims-pos1-0.cssi.renater.fr ( 193.51.179.137) 12.477 ms 12.603ms > >> 12.744 ms > >> 5 nri-b-pos6-0.cssi.renater.fr ( 193.51.179.149) 8.546 ms 8.532 ms > >> 8.491 ms > >> 6 renater.rt1.par.fr.geant2.net ( 62.40.124.69) 8.819 ms 8.232 ms > 8.194ms > >> 7 so-7-3-0.rt1.gen.ch.geant2.net ( 62.40.112.29) 16.913 ms 16.815ms > >> 16.795 ms > >> 8 so-7-2-0.rt1.fra.de.geant2.net ( 62.40.112.22) 24.893 ms 24.925ms > >> 25.057 ms > >> 9 abilene-wash-gw.rt1.fra.de.geant2.net ( 62.40.125.18) 131.087 ms > >> 130.878 ms 130.469 ms > >> 10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 136.166 ms > 133.322ms > >> 133.242 ms > >> 11 so-0-2-0.0.rtr.hous.net.internet2.edu (64.57.28.43 ) 154.750 ms > >> 154.452 ms 154.438 ms > >> 12 so-3-0-0.0.rtr.losa.net.internet2.edu ( 64.57.28.44) 197.920 ms > >> 197.889 ms 197.847 ms > >> 13 hpr-lax-hpr--i2-newnet.cenic.net ( 137.164.26.132) 183.765 ms * * > >> 14 svl-hpr--lax-hpr-10ge.cenic.net ( 137.164.25.13) 191.428 ms > 191.484 ms > >> * > >> 15 hpr-ucsc--svl-egm.cenic.net ( 137.164.27.86) 193.071 ms 193.075ms > >> 193.034 ms > >> 16 comm-g-GE3-4.ucsc.edu (128.114.0.65) 193.016 ms 192.962 ms > 192.941 ms > >> 17 comm-d3-g-GE1-0-12.ucsc.edu ( 128.114.110.1) 193.267 ms 193.398ms > >> 193.267 ms > >> 18 hgdownload.cse.ucsc.edu ( 128.114.119.140) 193.088 ms 192.974 ms > >> 192.998 ms > >> > >> > >> ==== Slow rsync: Traceroute from Australia to UCSC========= > >> > >> Tracing route to 128.114.119.140 > >> 1 ge-1-0-9.bb1.a.adl.aarnet.net.au ( 203.21.37.17) > 0.627ms > >> 0.497 ms 0.481 ms > >> 2 so-0-1-0.bb1.a.mel.aarnet.net.au (202.158.194.18) > 9.490ms > >> 9.549 ms 9.488 ms > >> 3 so-0-1-0.bb1.b.syd.aarnet.net.au (202.158.194.34) > 21.494ms > >> 21.808 ms 21.493 ms > >> 4 pos1-0.bb1.b.sea.aarnet.net.au ( 202.158.194.94) > 186.473 ms > >> 186.177 ms 270.973 ms > >> 5 cenichpr-1-is-std-779.snvaca.pacificwave.net (207.231.248.129 ) > 204.455 ms > >> 204.472 ms 204.459 ms > >> 6 hpr-ucsc--svl-egm.cenic.net ( 137.164.27.86) > 205.963 ms > >> 205.785 ms 205.732 ms > >> 7 isb-g-GE2-2.ucsc.edu ( 128.114.0.45) > 206.094 ms > >> 205.613 ms 205.963 ms > >> 8 comm-d3-g-GE1-0-11.ucsc.edu (128.114.110.5) > 206.450 ms > >> 206.154 ms 205.959 ms > >> 9 hgdownload.cse.ucsc.edu ( 128.114.119.140) > 206.466 ms > >> 205.583 ms 205.959 ms > >> > >> > >> ==== Fast rsync: Traceroute from rented US-based server to > UCSC=========== > >> > >> traceroute to hgdownload.cse.ucsc.edu ( 128.114.119.140), 30 hops max, > 38 > >> byte packets > >> 1 1.87.1243.static.theplanet.com ( 67.18.135.1) 0.489 ms > >> 0.465ms > >> 0.508 ms > >> 2 gi3-6.dsr02.dllstx4.theplanet.com ( 67.19.255.133) 0.375 ms > >> 0.239ms > >> 0.251 ms > >> 3 vl42.dsr01.dllstx3.theplanet.com (70.85.127.89) 0.508 ms > 0.765ms > >> 0.633 ms > >> 4 25.7f.5546.static.theplanet.com ( 70.85.127.37) 0.492 ms > >> 0.361ms > >> 0.382 ms > >> 5 dal-ix.he.net (206.223.118.37) 0.507 ms > 0.475ms > >> 0.502 ms > >> 6 pos5-0.gsr12012.lax.he.net ( 66.160.184.5) 35.030 ms > 35.257ms > >> 35.129 ms > >> 7 lax-px1--hurricane-ge.cenic.net (198.32.251.85) 35.544 ms > 35.388ms > >> 35.420 ms > >> 8 dc-sac-dc1--lax-dc1-pos.cenic.net ( 137.164.22.127) 45.014 ms > 45.126ms > >> 45.020 ms > >> 9 dc-oak-dc1--csac-dc1-ge.cenic.net (137.164.22.110 ) 47.092 ms > 46.942ms > >> 46.840 ms > >> 10 dc-oak-dc2--oak-dc1-p2p-2.cenic.net ( 137.164.22.195) 46.953 ms > 46.813ms > >> 46.969 ms > >> 11 ucsc-ucsc1--dc-oak-dc1-egm.cenic.net ( 137.164.23.13) 47.595 ms > 47.458ms > >> 47.488 ms > >> 12 comm-g-GE3-5.ucsc.edu (128.114.0.214) 47.479 ms > 47.463ms > >> 47.351 ms > >> 13 comm-d3-g-GE1-0-12.ucsc.edu ( 128.114.110.1) 47.742 ms > 47.586ms > >> 47.616 ms > >> 14 hgdownload.cse.ucsc.edu ( 128.114.119.140) 47.478 ms > 47.465ms > >> 47.475 ms > >> _______________________________________________ > >> Genome-mirror mailing list > >> Genome-mirror at soe.ucsc.edu > >> http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > >> > > _______________________________________________ > > Genome-mirror mailing list > > Genome-mirror at soe.ucsc.edu > > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > > -- > (*)->[]->()->[]->(**)->[]->()->[]->(*)->[]->()->[]->()->[]->()->[]->()->[] > > (Humboldt University Berlin, Germany)->[]-> ... > (University of Maryland, USA)->[]-> ... > (King's College London, UK) > > https://josh.umds.ac.uk/~rschulz > From galt at soe.ucsc.edu Wed Oct 17 10:03:47 2007 From: galt at soe.ucsc.edu (Galt Barber) Date: Wed, 17 Oct 2007 10:03:47 -0700 (PDT) Subject: [Genome-mirror] Problems with slow RSYNC transfers from UCSC In-Reply-To: <4715FC44.2030409@kcl.ac.uk> References: <200710161618.JAA17815@moondance.cse.ucsc.edu> <4715FC44.2030409@kcl.ac.uk> Message-ID: If there were enough mirror sites, perhaps a torrent approach would work? -Galt On Wed, 17 Oct 2007, Reiner Schulz wrote: > hi Mike, Robert, > > i am experiencing similarly slow rsync's of our partial mirror here at > KCL, UK. but that's mostly due to KCL's limited bandwidth to the outside > world in general (155Mb/s ATM). > > however, a while back i suggested half-jokingly that UCSC could perhaps > take up google's offer to help w/ academic computing/storage: > > >>>>>>>>>> > is the UCSC browser team planning to cooperate w/ google like the > Archimedes Palimpsest people did so that perhaps one day one can have > the entire UCSC browser drop-shipped instead of it trickling down the > net? :) > relevant refs: > http://news.bbc.co.uk/2/hi/technology/6425975.stm > http://searchstorage.techtarget.com/originalContent/0,289142,sid5_gci1246719,00.html > <<<<<<<<<< > > i'd think that running an rsync service for UCSC's genome browser on > google hardware falls within the same academic remit. has that > possibility been explored? > > cheers, > > Reiner > > Robert Kuhn wrote: > > Hi, Mike, > > > > I am forwarding your message to our system admins. thanks a lot > > for your careful analysis. > > > > --b0b kuhn > > > >> From genome-mirror-bounces at soe.ucsc.edu Tue Oct 16 09:09:06 2007 > >> To: "genome mirror" > >> Subject: [Genome-mirror] Problems with slow RSYNC transfers from UCSC > >> > >> Hi all > >> > >> We are trying to maintain a full (public) mirror in Australia including all > >> 'download' data (rsync targets 'gbdb', 'mysql', and 'genome'), hosted at a > >> university. The reason for this that Australian researchers get slow speeds > >> of around 170KB/sec maximum downloading from UCSC, whereas they will get > >> very fast download speeds from our mirror. It takes days or weeks to > >> complete the 'mysql' and 'gbdb' targets and is a big problem taking even > >> longer for the 'genome' target. We can get faster speeds from different US > >> sites; for example, we can get around double that rate downloading data from > >> a Stanford rsync server (similar TCP/IP route from AU), so it is not simply > >> a problem with rsync or all traffic on our international link. > >> > >> I am now at a university in France, and initial testing shows that we get > >> the same slow speeds to UCSC here, and again almost double the speed to > >> Stanford, and as well we can get 1MB/s and more from an rsync server at SDSC > >> (also on CENIC network) (I havent been able to test this from AU), so the > >> slow rsync speed doesnt seem to be a problem limited to Australia. However, > >> it seems that US-based servers get much greater speeds when doing rsync from > >> UCSC - until a few years ago we rented a US-hosted server and were getting > >> speeds of at least 500 KB/s between this server and UCSC and I suspect US > >> academic peer networks will see even faster speeds. > >> > >> Furthermore, the overall throughput is not limited, just individual > >> connections. For example, whether I have 1 or 5 rsync connections open, I > >> get a maximum of 170KB/s on each rsync. This is also true of FTP. The slow > >> download speeds are particularly a problem with the large files that are > >> updated on a daily basis (such as daily EST genbank data and all its derived > >> data) since we need to get everything synchronised between daily updates. > >> (Files such as goldenPath/bigZips/*.fa.gz, > >> goldenPath/database/{est,xenoMrna,gbStatus,...}.txt.gz, > >> mysql/*/{est,xenoMrna,gbStatus,...}.MYD). > >> > >> The problem seems worse when rsyncing large compressed files, particularly > >> rsync target 'genome' stuff (/bigZips/ & /database/) which is updated on a > >> daily basis since we seem to end up downloading entire files for each > >> update. When we download mysql/*MYD files which are updated daily (eg. > >> tables with EST data) we generally get very fast results since updates > >> appear to be appended to the file and rsync only needs to transfer the > >> trailing changed portions. I note that with gzip there is a flag > >> '--rsyncable Make rsync-friendly archive' which may help with this. > >> > >> Traceroutes I have done (see below) indicate the slow Australian and French > >> routes to UCSC go through a common point (hpr-ucsc--svl-egm.cenic.net, the > >> first host with UCSC in the name). However, the US server (faster) does not > >> appears to use this route (the first route with UCSC in the name is > >> ucsc-ucsc1--dc-oak-dc1-egm.cenic.net) (note that this is an old traceroute). > >> Also, the French route to UCSC, Stanford, and SDSC all enter CENIC the same > >> way; the difference in the routes is all between CENIC and the rsync hosts. > >> This is also observed for the Australian routes (not shown). This seems to > >> indicate that international TCPIP routes into UCSC follow different and > >> slower paths than US based routes, and that the problem for international > >> mirrors (at minimum, France and Australia) may be somewhere between the > >> UCSC-CENIC interface for international (university) traffic ( i.e., > >> aarnet.net.au/pacificwave.net for Australian universities and > >> Geant2.net/Internet2.edu for French universities). I realise that any > >> traceroutes I can do are not necessarily accurate representations of the > >> route, but I think they are instructive when taken together with the rsync > >> download speeds to different hosts. > >> > >> > >> I would like to know if we are the only ones experiencing this problem: > >> > >> - Do any mirrors outside the US get faster rsync downloads than we do ? > >> (more than say 170KB/s?) > >> > >> - Do any mirrors inside the US get slow downloads? (less than say 500KB/s?) > >> > >> > >> Also; > >> > >> - would UCSC increase the max rsync clients (again!) say from 15 to 20 or > >> more (or perhaps allow 5 or 6 or even more from the same IP address) so that > >> international mirrors can run more rsyncs in parallel to overcome this slow > >> per-connection speed? (in particular we want to do mysql/, gbdb/, genome/, > >> genome/*/bigZips/ and genome/*/database/ in parallel) > >> I believe this would in fact reduce the load on your servers and network > >> since we would be able to complete an update in a day or two, rather than > >> having just a few connections open for weeks on end continually refreshing > >> old data waiting for everything to be in sync before the next daily update. > >> > >> - can we test the '--rsyncable' gzip flag when you create compressed files > >> for rsync to see if it helps the mirrors (assuming you are not already using > >> it)? > >> > >> > >> Cheers > >> > >> Mike Pheasant > >> > >> > >> ==== Very fast rsync: Traceroute from France to SDSC (wwwPDB)========= > >> > >> traceroute to rsync.wwpdb.org (198.202.122.181), 30 hops max, 40 byte > >> packets > >> 1 crc-rc1-ge-1-2-0-65.u-strasbg.fr (130.79.47.253) 0.915 ms 1.087 ms > >> 1.276 ms > >> 2 strasbourg-g3-3.cssi.renater.fr (193.51.184.42) 0.510 ms 0.537 ms > >> 0.497 ms > >> 3 nancy-pos2-0.cssi.renater.fr (193.51.180.41) 38.250 ms 38.344 ms > >> 38.463 ms > >> 4 reims-pos1-0.cssi.renater.fr (193.51.179.137) 8.412 ms 8.600 ms 8.883ms > >> 5 * nri-b-pos6-0.cssi.renater.fr (193.51.179.149) 8.012 ms 7.967 ms > >> 6 renater.rt1.par.fr.geant2.net (62.40.124.69) 8.181 ms 8.340 ms 8.301ms > >> 7 so-7-3-0.rt1.gen.ch.geant2.net (62.40.112.29) 16.874 ms 16.862 ms > >> 16.864 ms > >> 8 so-7-2-0.rt1.fra.de.geant2.net (62.40.112.22) 24.941 ms 24.990 ms > >> 24.942 ms > >> 9 abilene-wash-gw.rt1.fra.de.geant2.net (62.40.125.18) 117.936 ms > >> 117.883 ms 118.437 ms > >> 10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 137.316 ms 137.302ms > >> 137.220 ms > >> 11 so-0-2-0.0.rtr.hous.net.internet2.edu (64.57.28.43) 154.646 ms 154.577ms > >> 154.667 ms > >> 12 so-3-0-0.0.rtr.losa.net.internet2.edu (64.57.28.44) 186.533 ms 187.578ms > >> 187.560 ms > >> 13 hpr-lax-hpr--i2-newnet.cenic.net (137.164.26.132) 186.753 ms 189.032ms * > >> 14 riv-hpr--lax-hpr-10ge.cenic.net (137.164.25.5) 193.032 ms * * > >> 15 hpr-sdsc-sdsc1--riv-hpr-ge.cenic.net (137.164.27.50) 206.951 ms > >> 206.854 ms 206.840 ms > >> 16 lightning.sdsc.edu (132.249.30.6) 192.749 ms 192.779 ms 192.774 ms > >> > >> > >> ==== Fast rsync: Traceroute from France to Stanford========= > >> > >> traceroute to genome-ftp.stanford.edu (171.65.76.47), 30 hops max, 40 byte > >> packets > >> 1 crc-rc1-ge-1-2-0-65.u-strasbg.fr ( 130.79.47.253) 0.881 ms 0.980 ms > >> 1.000 ms > >> 2 strasbourg-g3-3.cssi.renater.fr ( 193.51.184.42) 0.500 ms 0.497 ms > >> 0.472 ms > >> 3 nancy-pos2-0.cssi.renater.fr ( 193.51.180.41) 8.336 ms 8.423 ms 8.569ms > >> 4 reims-pos1-0.cssi.renater.fr (193.51.179.137) 8.600 ms 8.783 ms 8.896ms > >> 5 nri-b-pos6-0.cssi.renater.fr ( 193.51.179.149) 7.972 ms 7.935 ms > >> 7.904 ms > >> 6 renater.rt1.par.fr.geant2.net ( 62.40.124.69) 8.213 ms 8.183 ms 8.266ms > >> 7 so-7-3-0.rt1.gen.ch.geant2.net ( 62.40.112.29) 16.765 ms 16.866 ms > >> 16.779 ms > >> 8 so-7-2-0.rt1.fra.de.geant2.net (62.40.112.22) 25.118 ms 24.979 ms > >> 24.922 ms > >> 9 abilene-wash-gw.rt1.fra.de.geant2.net ( 62.40.125.18) 139.126 ms > >> 139.163 ms 135.276 ms > >> 10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 131.089 ms 131.084ms > >> 131.095 ms > >> 11 so-0-2-0.0.rtr.hous.net.internet2.edu ( 64.57.28.43) 154.667 ms > >> 154.695 ms 154.517 ms > >> 12 so-3-0-0.0.rtr.losa.net.internet2.edu (64.57.28.44) 186.611 ms 186.570ms > >> 186.422 ms > >> 13 hpr-lax-hpr--i2-newnet.cenic.net ( 137.164.26.132) 278.881 ms 269.707ms > >> 266.963 ms > >> 14 svl-hpr--lax-hpr-10ge.cenic.net (137.164.25.13 ) 191.474 ms 191.432 ms > >> * > >> 15 hpr-stan-ge--svl-hpr.cenic.net ( 137.164.27.162) 191.573 ms 191.851ms > >> 191.622 ms > >> 16 bbrb-i2.Stanford.EDU (171.64.1.136) 192.174 ms 191.888 ms 191.804 ms > >> > >> > >> ==== Slow rsync: Traceroute from France to UCSC========= > >> > >> traceroute to hgdownload.cse.ucsc.edu ( 128.114.119.140), 30 hops max, 40 > >> byte packets > >> 1 crc-rc1-ge-1-2-0-65.u-strasbg.fr ( 130.79.47.253) 2.106 ms 2.075 ms > >> 2.051 ms > >> 2 strasbourg-g3-3.cssi.renater.fr ( 193.51.184.42) 1.466 ms 1.441 ms > >> 1.400 ms > >> 3 nancy-pos2-0.cssi.renater.fr ( 193.51.180.41) 15.768 ms 15.921 ms > >> 16.097 ms > >> 4 reims-pos1-0.cssi.renater.fr ( 193.51.179.137) 12.477 ms 12.603 ms > >> 12.744 ms > >> 5 nri-b-pos6-0.cssi.renater.fr ( 193.51.179.149) 8.546 ms 8.532 ms > >> 8.491 ms > >> 6 renater.rt1.par.fr.geant2.net ( 62.40.124.69) 8.819 ms 8.232 ms 8.194ms > >> 7 so-7-3-0.rt1.gen.ch.geant2.net ( 62.40.112.29) 16.913 ms 16.815 ms > >> 16.795 ms > >> 8 so-7-2-0.rt1.fra.de.geant2.net ( 62.40.112.22) 24.893 ms 24.925 ms > >> 25.057 ms > >> 9 abilene-wash-gw.rt1.fra.de.geant2.net ( 62.40.125.18) 131.087 ms > >> 130.878 ms 130.469 ms > >> 10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 136.166 ms 133.322ms > >> 133.242 ms > >> 11 so-0-2-0.0.rtr.hous.net.internet2.edu (64.57.28.43 ) 154.750 ms > >> 154.452 ms 154.438 ms > >> 12 so-3-0-0.0.rtr.losa.net.internet2.edu ( 64.57.28.44) 197.920 ms > >> 197.889 ms 197.847 ms > >> 13 hpr-lax-hpr--i2-newnet.cenic.net ( 137.164.26.132) 183.765 ms * * > >> 14 svl-hpr--lax-hpr-10ge.cenic.net ( 137.164.25.13) 191.428 ms 191.484 ms > >> * > >> 15 hpr-ucsc--svl-egm.cenic.net ( 137.164.27.86) 193.071 ms 193.075 ms > >> 193.034 ms > >> 16 comm-g-GE3-4.ucsc.edu (128.114.0.65) 193.016 ms 192.962 ms 192.941 ms > >> 17 comm-d3-g-GE1-0-12.ucsc.edu ( 128.114.110.1) 193.267 ms 193.398 ms > >> 193.267 ms > >> 18 hgdownload.cse.ucsc.edu ( 128.114.119.140) 193.088 ms 192.974 ms > >> 192.998 ms > >> > >> > >> ==== Slow rsync: Traceroute from Australia to UCSC========= > >> > >> Tracing route to 128.114.119.140 > >> 1 ge-1-0-9.bb1.a.adl.aarnet.net.au ( 203.21.37.17) 0.627ms > >> 0.497 ms 0.481 ms > >> 2 so-0-1-0.bb1.a.mel.aarnet.net.au (202.158.194.18) 9.490ms > >> 9.549 ms 9.488 ms > >> 3 so-0-1-0.bb1.b.syd.aarnet.net.au (202.158.194.34) 21.494ms > >> 21.808 ms 21.493 ms > >> 4 pos1-0.bb1.b.sea.aarnet.net.au ( 202.158.194.94) 186.473 ms > >> 186.177 ms 270.973 ms > >> 5 cenichpr-1-is-std-779.snvaca.pacificwave.net (207.231.248.129 ) 204.455 ms > >> 204.472 ms 204.459 ms > >> 6 hpr-ucsc--svl-egm.cenic.net ( 137.164.27.86) 205.963 ms > >> 205.785 ms 205.732 ms > >> 7 isb-g-GE2-2.ucsc.edu ( 128.114.0.45) 206.094 ms > >> 205.613 ms 205.963 ms > >> 8 comm-d3-g-GE1-0-11.ucsc.edu (128.114.110.5) 206.450 ms > >> 206.154 ms 205.959 ms > >> 9 hgdownload.cse.ucsc.edu ( 128.114.119.140) 206.466 ms > >> 205.583 ms 205.959 ms > >> > >> > >> ==== Fast rsync: Traceroute from rented US-based server to UCSC=========== > >> > >> traceroute to hgdownload.cse.ucsc.edu ( 128.114.119.140), 30 hops max, 38 > >> byte packets > >> 1 1.87.1243.static.theplanet.com ( 67.18.135.1) 0.489 ms > >> 0.465ms > >> 0.508 ms > >> 2 gi3-6.dsr02.dllstx4.theplanet.com ( 67.19.255.133) 0.375 ms > >> 0.239ms > >> 0.251 ms > >> 3 vl42.dsr01.dllstx3.theplanet.com (70.85.127.89) 0.508 ms 0.765ms > >> 0.633 ms > >> 4 25.7f.5546.static.theplanet.com ( 70.85.127.37) 0.492 ms > >> 0.361ms > >> 0.382 ms > >> 5 dal-ix.he.net (206.223.118.37) 0.507 ms 0.475ms > >> 0.502 ms > >> 6 pos5-0.gsr12012.lax.he.net ( 66.160.184.5) 35.030 ms 35.257ms > >> 35.129 ms > >> 7 lax-px1--hurricane-ge.cenic.net (198.32.251.85) 35.544 ms 35.388ms > >> 35.420 ms > >> 8 dc-sac-dc1--lax-dc1-pos.cenic.net ( 137.164.22.127) 45.014 ms 45.126ms > >> 45.020 ms > >> 9 dc-oak-dc1--csac-dc1-ge.cenic.net (137.164.22.110 ) 47.092 ms 46.942ms > >> 46.840 ms > >> 10 dc-oak-dc2--oak-dc1-p2p-2.cenic.net ( 137.164.22.195) 46.953 ms 46.813ms > >> 46.969 ms > >> 11 ucsc-ucsc1--dc-oak-dc1-egm.cenic.net ( 137.164.23.13) 47.595 ms 47.458ms > >> 47.488 ms > >> 12 comm-g-GE3-5.ucsc.edu (128.114.0.214) 47.479 ms 47.463ms > >> 47.351 ms > >> 13 comm-d3-g-GE1-0-12.ucsc.edu ( 128.114.110.1) 47.742 ms 47.586ms > >> 47.616 ms > >> 14 hgdownload.cse.ucsc.edu ( 128.114.119.140) 47.478 ms 47.465ms > >> 47.475 ms > >> _______________________________________________ > >> Genome-mirror mailing list > >> Genome-mirror at soe.ucsc.edu > >> http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > >> > > _______________________________________________ > > Genome-mirror mailing list > > Genome-mirror at soe.ucsc.edu > > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > > -- > (*)->[]->()->[]->(**)->[]->()->[]->(*)->[]->()->[]->()->[]->()->[]->()->[] > > (Humboldt University Berlin, Germany)->[]-> ... > (University of Maryland, USA)->[]-> ... > (King's College London, UK) > > https://josh.umds.ac.uk/~rschulz > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From kuhn at soe.ucsc.edu Thu Oct 18 09:57:28 2007 From: kuhn at soe.ucsc.edu (Robert Kuhn) Date: Thu, 18 Oct 2007 09:57:28 -0700 Subject: [Genome-mirror] Problems with slow RSYNC transfers from UCSC Message-ID: <200710181657.JAA20364@moondance.cse.ucsc.edu> Reiner, These are interesting ideas. We will consider them. Thanks for your input! --b0b kuhn ucsc genome bioinformatics group > From reiner.schulz at kcl.ac.uk Wed Oct 17 05:13:06 2007 > To: Robert Kuhn > Cc: genome-mirror at soe.ucsc.edu, m.pheasant at imb.uq.edu.au > Subject: Re: [Genome-mirror] Problems with slow RSYNC transfers from UCSC > > hi Mike, Robert, > > i am experiencing similarly slow rsync's of our partial mirror here at > KCL, UK. but that's mostly due to KCL's limited bandwidth to the outside > world in general (155Mb/s ATM). > > however, a while back i suggested half-jokingly that UCSC could perhaps > take up google's offer to help w/ academic computing/storage: > > >>>>>>>>>> > is the UCSC browser team planning to cooperate w/ google like the > Archimedes Palimpsest people did so that perhaps one day one can have > the entire UCSC browser drop-shipped instead of it trickling down the > net? :) > relevant refs: > http://news.bbc.co.uk/2/hi/technology/6425975.stm > http://searchstorage.techtarget.com/originalContent/0,289142,sid5_gci1246719,00.html > <<<<<<<<<< > > i'd think that running an rsync service for UCSC's genome browser on > google hardware falls within the same academic remit. has that > possibility been explored? > > cheers, > > Reiner > > Robert Kuhn wrote: > > Hi, Mike, > > > > I am forwarding your message to our system admins. thanks a lot > > for your careful analysis. > > > > --b0b kuhn > > > >> From genome-mirror-bounces at soe.ucsc.edu Tue Oct 16 09:09:06 2007 > >> To: "genome mirror" > >> Subject: [Genome-mirror] Problems with slow RSYNC transfers from UCSC > >> > >> Hi all > >> > >> We are trying to maintain a full (public) mirror in Australia including all > >> 'download' data (rsync targets 'gbdb', 'mysql', and 'genome'), hosted at a > >> university. The reason for this that Australian researchers get slow speeds > >> of around 170KB/sec maximum downloading from UCSC, whereas they will get > >> very fast download speeds from our mirror. It takes days or weeks to > >> complete the 'mysql' and 'gbdb' targets and is a big problem taking even > >> longer for the 'genome' target. We can get faster speeds from different US > >> sites; for example, we can get around double that rate downloading data from > >> a Stanford rsync server (similar TCP/IP route from AU), so it is not simply > >> a problem with rsync or all traffic on our international link. > >> > >> I am now at a university in France, and initial testing shows that we get > >> the same slow speeds to UCSC here, and again almost double the speed to > >> Stanford, and as well we can get 1MB/s and more from an rsync server at SDSC > >> (also on CENIC network) (I havent been able to test this from AU), so the > >> slow rsync speed doesnt seem to be a problem limited to Australia. However, > >> it seems that US-based servers get much greater speeds when doing rsync from > >> UCSC - until a few years ago we rented a US-hosted server and were getting > >> speeds of at least 500 KB/s between this server and UCSC and I suspect US > >> academic peer networks will see even faster speeds. > >> > >> Furthermore, the overall throughput is not limited, just individual > >> connections. For example, whether I have 1 or 5 rsync connections open, I > >> get a maximum of 170KB/s on each rsync. This is also true of FTP. The slow > >> download speeds are particularly a problem with the large files that are > >> updated on a daily basis (such as daily EST genbank data and all its derived > >> data) since we need to get everything synchronised between daily updates. > >> (Files such as goldenPath/bigZips/*.fa.gz, > >> goldenPath/database/{est,xenoMrna,gbStatus,...}.txt.gz, > >> mysql/*/{est,xenoMrna,gbStatus,...}.MYD). > >> > >> The problem seems worse when rsyncing large compressed files, particularly > >> rsync target 'genome' stuff (/bigZips/ & /database/) which is updated on a > >> daily basis since we seem to end up downloading entire files for each > >> update. When we download mysql/*MYD files which are updated daily (eg. > >> tables with EST data) we generally get very fast results since updates > >> appear to be appended to the file and rsync only needs to transfer the > >> trailing changed portions. I note that with gzip there is a flag > >> '--rsyncable Make rsync-friendly archive' which may help with this. > >> > >> Traceroutes I have done (see below) indicate the slow Australian and French > >> routes to UCSC go through a common point (hpr-ucsc--svl-egm.cenic.net, the > >> first host with UCSC in the name). However, the US server (faster) does not > >> appears to use this route (the first route with UCSC in the name is > >> ucsc-ucsc1--dc-oak-dc1-egm.cenic.net) (note that this is an old traceroute). > >> Also, the French route to UCSC, Stanford, and SDSC all enter CENIC the same > >> way; the difference in the routes is all between CENIC and the rsync hosts. > >> This is also observed for the Australian routes (not shown). This seems to > >> indicate that international TCPIP routes into UCSC follow different and > >> slower paths than US based routes, and that the problem for international > >> mirrors (at minimum, France and Australia) may be somewhere between the > >> UCSC-CENIC interface for international (university) traffic ( i.e., > >> aarnet.net.au/pacificwave.net for Australian universities and > >> Geant2.net/Internet2.edu for French universities). I realise that any > >> traceroutes I can do are not necessarily accurate representations of the > >> route, but I think they are instructive when taken together with the rsync > >> download speeds to different hosts. > >> > >> > >> I would like to know if we are the only ones experiencing this problem: > >> > >> - Do any mirrors outside the US get faster rsync downloads than we do ? > >> (more than say 170KB/s?) > >> > >> - Do any mirrors inside the US get slow downloads? (less than say 500KB/s?) > >> > >> > >> Also; > >> > >> - would UCSC increase the max rsync clients (again!) say from 15 to 20 or > >> more (or perhaps allow 5 or 6 or even more from the same IP address) so that > >> international mirrors can run more rsyncs in parallel to overcome this slow > >> per-connection speed? (in particular we want to do mysql/, gbdb/, genome/, > >> genome/*/bigZips/ and genome/*/database/ in parallel) > >> I believe this would in fact reduce the load on your servers and network > >> since we would be able to complete an update in a day or two, rather than > >> having just a few connections open for weeks on end continually refreshing > >> old data waiting for everything to be in sync before the next daily update. > >> > >> - can we test the '--rsyncable' gzip flag when you create compressed files > >> for rsync to see if it helps the mirrors (assuming you are not already using > >> it)? > >> > >> > >> Cheers > >> > >> Mike Pheasant > >> > >> > >> ==== Very fast rsync: Traceroute from France to SDSC (wwwPDB)========= > >> > >> traceroute to rsync.wwpdb.org (198.202.122.181), 30 hops max, 40 byte > >> packets > >> 1 crc-rc1-ge-1-2-0-65.u-strasbg.fr (130.79.47.253) 0.915 ms 1.087 ms > >> 1.276 ms > >> 2 strasbourg-g3-3.cssi.renater.fr (193.51.184.42) 0.510 ms 0.537 ms > >> 0.497 ms > >> 3 nancy-pos2-0.cssi.renater.fr (193.51.180.41) 38.250 ms 38.344 ms > >> 38.463 ms > >> 4 reims-pos1-0.cssi.renater.fr (193.51.179.137) 8.412 ms 8.600 ms 8.883ms > >> 5 * nri-b-pos6-0.cssi.renater.fr (193.51.179.149) 8.012 ms 7.967 ms > >> 6 renater.rt1.par.fr.geant2.net (62.40.124.69) 8.181 ms 8.340 ms 8.301ms > >> 7 so-7-3-0.rt1.gen.ch.geant2.net (62.40.112.29) 16.874 ms 16.862 ms > >> 16.864 ms > >> 8 so-7-2-0.rt1.fra.de.geant2.net (62.40.112.22) 24.941 ms 24.990 ms > >> 24.942 ms > >> 9 abilene-wash-gw.rt1.fra.de.geant2.net (62.40.125.18) 117.936 ms > >> 117.883 ms 118.437 ms > >> 10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 137.316 ms 137.302ms > >> 137.220 ms > >> 11 so-0-2-0.0.rtr.hous.net.internet2.edu (64.57.28.43) 154.646 ms 154.577ms > >> 154.667 ms > >> 12 so-3-0-0.0.rtr.losa.net.internet2.edu (64.57.28.44) 186.533 ms 187.578ms > >> 187.560 ms > >> 13 hpr-lax-hpr--i2-newnet.cenic.net (137.164.26.132) 186.753 ms 189.032ms * > >> 14 riv-hpr--lax-hpr-10ge.cenic.net (137.164.25.5) 193.032 ms * * > >> 15 hpr-sdsc-sdsc1--riv-hpr-ge.cenic.net (137.164.27.50) 206.951 ms > >> 206.854 ms 206.840 ms > >> 16 lightning.sdsc.edu (132.249.30.6) 192.749 ms 192.779 ms 192.774 ms > >> > >> > >> ==== Fast rsync: Traceroute from France to Stanford========= > >> > >> traceroute to genome-ftp.stanford.edu (171.65.76.47), 30 hops max, 40 byte > >> packets > >> 1 crc-rc1-ge-1-2-0-65.u-strasbg.fr ( 130.79.47.253) 0.881 ms 0.980 ms > >> 1.000 ms > >> 2 strasbourg-g3-3.cssi.renater.fr ( 193.51.184.42) 0.500 ms 0.497 ms > >> 0.472 ms > >> 3 nancy-pos2-0.cssi.renater.fr ( 193.51.180.41) 8.336 ms 8.423 ms 8.569ms > >> 4 reims-pos1-0.cssi.renater.fr (193.51.179.137) 8.600 ms 8.783 ms 8.896ms > >> 5 nri-b-pos6-0.cssi.renater.fr ( 193.51.179.149) 7.972 ms 7.935 ms > >> 7.904 ms > >> 6 renater.rt1.par.fr.geant2.net ( 62.40.124.69) 8.213 ms 8.183 ms 8.266ms > >> 7 so-7-3-0.rt1.gen.ch.geant2.net ( 62.40.112.29) 16.765 ms 16.866 ms > >> 16.779 ms > >> 8 so-7-2-0.rt1.fra.de.geant2.net (62.40.112.22) 25.118 ms 24.979 ms > >> 24.922 ms > >> 9 abilene-wash-gw.rt1.fra.de.geant2.net ( 62.40.125.18) 139.126 ms > >> 139.163 ms 135.276 ms > >> 10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 131.089 ms 131.084ms > >> 131.095 ms > >> 11 so-0-2-0.0.rtr.hous.net.internet2.edu ( 64.57.28.43) 154.667 ms > >> 154.695 ms 154.517 ms > >> 12 so-3-0-0.0.rtr.losa.net.internet2.edu (64.57.28.44) 186.611 ms 186.570ms > >> 186.422 ms > >> 13 hpr-lax-hpr--i2-newnet.cenic.net ( 137.164.26.132) 278.881 ms 269.707ms > >> 266.963 ms > >> 14 svl-hpr--lax-hpr-10ge.cenic.net (137.164.25.13 ) 191.474 ms 191.432 ms > >> * > >> 15 hpr-stan-ge--svl-hpr.cenic.net ( 137.164.27.162) 191.573 ms 191.851ms > >> 191.622 ms > >> 16 bbrb-i2.Stanford.EDU (171.64.1.136) 192.174 ms 191.888 ms 191.804 ms > >> > >> > >> ==== Slow rsync: Traceroute from France to UCSC========= > >> > >> traceroute to hgdownload.cse.ucsc.edu ( 128.114.119.140), 30 hops max, 40 > >> byte packets > >> 1 crc-rc1-ge-1-2-0-65.u-strasbg.fr ( 130.79.47.253) 2.106 ms 2.075 ms > >> 2.051 ms > >> 2 strasbourg-g3-3.cssi.renater.fr ( 193.51.184.42) 1.466 ms 1.441 ms > >> 1.400 ms > >> 3 nancy-pos2-0.cssi.renater.fr ( 193.51.180.41) 15.768 ms 15.921 ms > >> 16.097 ms > >> 4 reims-pos1-0.cssi.renater.fr ( 193.51.179.137) 12.477 ms 12.603 ms > >> 12.744 ms > >> 5 nri-b-pos6-0.cssi.renater.fr ( 193.51.179.149) 8.546 ms 8.532 ms > >> 8.491 ms > >> 6 renater.rt1.par.fr.geant2.net ( 62.40.124.69) 8.819 ms 8.232 ms 8.194ms > >> 7 so-7-3-0.rt1.gen.ch.geant2.net ( 62.40.112.29) 16.913 ms 16.815 ms > >> 16.795 ms > >> 8 so-7-2-0.rt1.fra.de.geant2.net ( 62.40.112.22) 24.893 ms 24.925 ms > >> 25.057 ms > >> 9 abilene-wash-gw.rt1.fra.de.geant2.net ( 62.40.125.18) 131.087 ms > >> 130.878 ms 130.469 ms > >> 10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 136.166 ms 133.322ms > >> 133.242 ms > >> 11 so-0-2-0.0.rtr.hous.net.internet2.edu (64.57.28.43 ) 154.750 ms > >> 154.452 ms 154.438 ms > >> 12 so-3-0-0.0.rtr.losa.net.internet2.edu ( 64.57.28.44) 197.920 ms > >> 197.889 ms 197.847 ms > >> 13 hpr-lax-hpr--i2-newnet.cenic.net ( 137.164.26.132) 183.765 ms * * > >> 14 svl-hpr--lax-hpr-10ge.cenic.net ( 137.164.25.13) 191.428 ms 191.484 ms > >> * > >> 15 hpr-ucsc--svl-egm.cenic.net ( 137.164.27.86) 193.071 ms 193.075 ms > >> 193.034 ms > >> 16 comm-g-GE3-4.ucsc.edu (128.114.0.65) 193.016 ms 192.962 ms 192.941 ms > >> 17 comm-d3-g-GE1-0-12.ucsc.edu ( 128.114.110.1) 193.267 ms 193.398 ms > >> 193.267 ms > >> 18 hgdownload.cse.ucsc.edu ( 128.114.119.140) 193.088 ms 192.974 ms > >> 192.998 ms > >> > >> > >> ==== Slow rsync: Traceroute from Australia to UCSC========= > >> > >> Tracing route to 128.114.119.140 > >> 1 ge-1-0-9.bb1.a.adl.aarnet.net.au ( 203.21.37.17) 0.627ms > >> 0.497 ms 0.481 ms > >> 2 so-0-1-0.bb1.a.mel.aarnet.net.au (202.158.194.18) 9.490ms > >> 9.549 ms 9.488 ms > >> 3 so-0-1-0.bb1.b.syd.aarnet.net.au (202.158.194.34) 21.494ms > >> 21.808 ms 21.493 ms > >> 4 pos1-0.bb1.b.sea.aarnet.net.au ( 202.158.194.94) 186.473 ms > >> 186.177 ms 270.973 ms > >> 5 cenichpr-1-is-std-779.snvaca.pacificwave.net (207.231.248.129 ) 204.455 ms > >> 204.472 ms 204.459 ms > >> 6 hpr-ucsc--svl-egm.cenic.net ( 137.164.27.86) 205.963 ms > >> 205.785 ms 205.732 ms > >> 7 isb-g-GE2-2.ucsc.edu ( 128.114.0.45) 206.094 ms > >> 205.613 ms 205.963 ms > >> 8 comm-d3-g-GE1-0-11.ucsc.edu (128.114.110.5) 206.450 ms > >> 206.154 ms 205.959 ms > >> 9 hgdownload.cse.ucsc.edu ( 128.114.119.140) 206.466 ms > >> 205.583 ms 205.959 ms > >> > >> > >> ==== Fast rsync: Traceroute from rented US-based server to UCSC=========== > >> > >> traceroute to hgdownload.cse.ucsc.edu ( 128.114.119.140), 30 hops max, 38 > >> byte packets > >> 1 1.87.1243.static.theplanet.com ( 67.18.135.1) 0.489 ms > >> 0.465ms > >> 0.508 ms > >> 2 gi3-6.dsr02.dllstx4.theplanet.com ( 67.19.255.133) 0.375 ms > >> 0.239ms > >> 0.251 ms > >> 3 vl42.dsr01.dllstx3.theplanet.com (70.85.127.89) 0.508 ms 0.765ms > >> 0.633 ms > >> 4 25.7f.5546.static.theplanet.com ( 70.85.127.37) 0.492 ms > >> 0.361ms > >> 0.382 ms > >> 5 dal-ix.he.net (206.223.118.37) 0.507 ms 0.475ms > >> 0.502 ms > >> 6 pos5-0.gsr12012.lax.he.net ( 66.160.184.5) 35.030 ms 35.257ms > >> 35.129 ms > >> 7 lax-px1--hurricane-ge.cenic.net (198.32.251.85) 35.544 ms 35.388ms > >> 35.420 ms > >> 8 dc-sac-dc1--lax-dc1-pos.cenic.net ( 137.164.22.127) 45.014 ms 45.126ms > >> 45.020 ms > >> 9 dc-oak-dc1--csac-dc1-ge.cenic.net (137.164.22.110 ) 47.092 ms 46.942ms > >> 46.840 ms > >> 10 dc-oak-dc2--oak-dc1-p2p-2.cenic.net ( 137.164.22.195) 46.953 ms 46.813ms > >> 46.969 ms > >> 11 ucsc-ucsc1--dc-oak-dc1-egm.cenic.net ( 137.164.23.13) 47.595 ms 47.458ms > >> 47.488 ms > >> 12 comm-g-GE3-5.ucsc.edu (128.114.0.214) 47.479 ms 47.463ms > >> 47.351 ms > >> 13 comm-d3-g-GE1-0-12.ucsc.edu ( 128.114.110.1) 47.742 ms 47.586ms > >> 47.616 ms > >> 14 hgdownload.cse.ucsc.edu ( 128.114.119.140) 47.478 ms 47.465ms > >> 47.475 ms > >> _______________________________________________ > >> Genome-mirror mailing list > >> Genome-mirror at soe.ucsc.edu > >> http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > >> > > _______________________________________________ > > Genome-mirror mailing list > > Genome-mirror at soe.ucsc.edu > > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > > -- > (*)->[]->()->[]->(**)->[]->()->[]->(*)->[]->()->[]->()->[]->()->[]->()->[] > > (Humboldt University Berlin, Germany)->[]-> ... > (University of Maryland, USA)->[]-> ... > (King's College London, UK) > > https://josh.umds.ac.uk/~rschulz > From mike at bioinformatics.com.au Thu Oct 18 04:33:00 2007 From: mike at bioinformatics.com.au (mike pheasant) Date: Thu, 18 Oct 2007 13:33:00 +0200 Subject: [Genome-mirror] Problems with slow RSYNC transfers from UCSC In-Reply-To: <47173452.1070402@kcl.ac.uk> References: <200710161618.JAA17815@moondance.cse.ucsc.edu> <4715FC44.2030409@kcl.ac.uk> <44bb49500710170617p3d4be635u58a93a73e058b94@mail.gmail.com> <47173452.1070402@kcl.ac.uk> Message-ID: <44bb49500710180433w76c87c45n61ae41f628d2218a@mail.gmail.com> Thanks Reiner Interesting you get double the speed from PDB than UCSC. Even though your UCSC is a bit slower than mine (120 vs 170), your speed to PDB is faster than both of our UCSC connections. Although we both use different routes to the US, and into cenic.net, the first traceroute hostname we have with UCSC in it is the same for both of us: hpr-ucsc--svl-egm.cenic.net (137.164.27.86) Also, both your traceroutes are bascially the same up to cenic.net ( hpr-lax-hpr--nlr-packenet.cenic.net (137.164.26.130) So I think there is something slowing our UCSC rsync traffic down, between hpr-ucsc--svl-egm.cenic.net and the UCSC rsync server. Cheers Mike On 10/18/07, Reiner Schulz wrote: > > hi Mike, > > PDB: 240.60kB/s > UCSC: 121.74kB/s > > and here are the traceroute's: > > traceroute to ftp-remediated-v3.rcsb.org (198.202.122.182), 64 hops max, > 40 byte packets > 1 159.92.26.3 (159.92.26.3) 0 ms 0 ms 0 ms > 2 192.168.150.13 (192.168.150.13) 0 ms 0 ms 0 ms > 3 192.168.10.10 (192.168.10.10) 1 ms 0 ms 0 ms > 4 192.168.10.14 (192.168.10.14) 0 ms 0 ms 0 ms > 5 137.73.1.1 (137.73.1.1) 1 ms 2 ms 5 ms > 6 137.73.0.14 (137.73.0.14) 78 ms 91 ms 117 ms > 7 137.73.1.105 (137.73.1.105) 85 ms 54 ms 81 ms > 8 kcl-gsr.lmn.net.uk (194.83.101.9) 81 ms 87 ms 78 ms > 9 po1-0.ulcc-gsr.lmn.net.uk (194.83.100.14) 65 ms 41 ms 41 ms > 10 so-1-0-0.lond-sbr1.ja.net (146.97.42.61) 47 ms * 19 ms > 11 so-0-0-0.lond-sbr3.ja.net (146.97.33.134) 22 ms 32 ms 27 ms > 12 lond-sbr5.ja.net (146.97.33.6) 20 ms 14 ms 10 ms > 13 po1-0.gn2-gw1.ja.net (146.97.35.98) 12 ms 14 ms 22 ms > 14 janet.rt1.lon.uk.geant2.net (62.40.124.197) 21 ms 17 ms 28 ms > 15 so-2-0-0.rt1.ams.nl.geant2.net (62.40.112.137) 40 ms 39 ms 55 ms > 16 so-7-0-0.rt1.nyc.us.geant2.net (62.40.112.134) 116 ms 116 ms 138 ms > 17 216.24.184.85 (216.24.184.85) 148 ms 132 ms 128 ms > 18 wash-newy-98.layer3.nlr.net (216.24.186.23) 142 ms 117 ms 120 ms > 19 atla-wash-64.layer3.nlr.net (216.24.186.20) 120 ms 133 ms 146 ms > 20 hous-atla-70.layer3.nlr.net (216.24.186.8) 160 ms 204 ms 173 ms > 21 losa-hous-87.layer3.nlr.net (216.24.186.30) 254 ms 265 ms 235 ms > 22 hpr-lax-hpr--nlr-packenet.cenic.net (137.164.26.130) 284 ms 277 ms > 274 ms23 riv-hpr--lax-hpr-10ge.cenic.net (137.164.25.5) 263 ms 262 > ms 241 ms > 24 hpr-sdsc-sdsc1--riv-hpr-ge.cenic.net (137.164.27.50) 250 ms 269 ms > 235 ms25 lightning.sdsc.edu (132.249.30.6) 238 ms 267 ms 265 ms > 26 * * * > 27 198.202.122.182 (198.202.122.182) 191 ms 205 ms 217 ms > > traceroute to hgdownload.cse.ucsc.edu (128.114.119.140), 64 hops max, 40 > byte packets > 1 159.92.26.3 (159.92.26.3) 0 ms 0 ms 0 ms > 2 192.168.150.13 (192.168.150.13) 0 ms 0 ms 0 ms > 3 192.168.10.10 (192.168.10.10) 0 ms 1 ms 0 ms > 4 137.73.1.13 (137.73.1.13) 1 ms 1 ms 1 ms > 5 137.73.0.14 (137.73.0.14) 31 ms 23 ms 42 ms > 6 137.73.1.105 (137.73.1.105) 32 ms 31 ms 63 ms > 7 * kcl-gsr.lmn.net.uk (194.83.101.9) 52 ms 56 ms > 8 po1-0.ulcc-gsr.lmn.net.uk (194.83.100.14) 49 ms 50 ms 55 ms > 9 so-1-0-0.lond-sbr1.ja.net (146.97.42.61) 59 ms 74 ms 91 ms > 10 so-0-0-0.lond-sbr3.ja.net (146.97.33.134) 72 ms 41 ms 28 ms > 11 lond-sbr5.ja.net (146.97.33.6) 34 ms 52 ms 46 ms > 12 po1-0.gn2-gw1.ja.net (146.97.35.98) 27 ms 20 ms 11 ms > 13 janet.rt1.lon.uk.geant2.net (62.40.124.197) 6 ms 8 ms 7 ms > 14 so-2-0-0.rt1.ams.nl.geant2.net (62.40.112.137) 15 ms 29 ms 38 ms > 15 so-7-0-0.rt1.nyc.us.geant2.net (62.40.112.134) 158 ms 148 ms 175 ms > 16 216.24.184.85 (216.24.184.85) 119 ms 109 ms 120 ms > 17 wash-newy-98.layer3.nlr.net (216.24.186.23) 169 ms 159 ms 152 ms > 18 atla-wash-64.layer3.nlr.net (216.24.186.20) 160 ms 174 ms 202 ms > 19 hous-atla-70.layer3.nlr.net (216.24.186.8) 203 ms 165 ms 151 ms > 20 losa-hous-87.layer3.nlr.net (216.24.186.30) 179 ms 194 ms 177 ms > 21 hpr-lax-hpr--nlr-packenet.cenic.net (137.164.26.130) 245 ms * 222 ms > 22 svl-hpr--lax-hpr-10ge.cenic.net (137.164.25.13) 184 ms 189 ms 186 > ms > 23 hpr-ucsc--svl-egm.cenic.net (137.164.27.86) 192 ms 206 ms 184 ms > 24 comm-g-GE3-4.ucsc.edu (128.114.0.65) 188 ms 220 ms 199 ms > 25 comm-d3-g-GE1-0-12.ucsc.edu (128.114.110.1) 220 ms 185 ms 188 ms > 26 hgdownload.cse.ucsc.edu (128.114.119.140) 198 ms 187 ms 189 ms > > Michael Pheasant wrote: > > Hi Reiner, > > > > You can do a simple test to see if the problem is just KCL bandwidth. > > Compare the speeds of downloading these two files: > > > > rsync -P --port=33444 rsync.wwpdb.org::ftp/ls-lR . > > rsync -P > > > rsync://hgdownload.cse.ucsc.edu/genome/goldenPath/hg18/bigZips/upstream1000.zip > > . > > > > It would be intersting to see what max speeds you get from both. I can > get > > over 1MB/s from rsync.wwpdb.org (SDSC, for me it seems to be on a > similar > > route to UCSC) and 170Kb/s to UCSC. An italian mirror confirmed to me > that > > they also get 170KB/s to UCSC. > > > > Cheers > > > > Mike > > > > On 10/17/07, Reiner Schulz wrote: > >> hi Mike, Robert, > >> > >> i am experiencing similarly slow rsync's of our partial mirror here at > >> KCL, UK. but that's mostly due to KCL's limited bandwidth to the > outside > >> world in general (155Mb/s ATM). > >> > >> however, a while back i suggested half-jokingly that UCSC could perhaps > >> take up google's offer to help w/ academic computing/storage: > >> > >> is the UCSC browser team planning to cooperate w/ google like the > >> Archimedes Palimpsest people did so that perhaps one day one can have > >> the entire UCSC browser drop-shipped instead of it trickling down the > >> net? :) > >> relevant refs: > >> http://news.bbc.co.uk/2/hi/technology/6425975.stm > >> > >> > http://searchstorage.techtarget.com/originalContent/0,289142,sid5_gci1246719,00.html > >> <<<<<<<<<< > >> > >> i'd think that running an rsync service for UCSC's genome browser on > >> google hardware falls within the same academic remit. has that > >> possibility been explored? > >> > >> cheers, > >> > >> Reiner > >> > >> Robert Kuhn wrote: > >>> Hi, Mike, > >>> > >>> I am forwarding your message to our system admins. thanks a lot > >>> for your careful analysis. > >>> > >>> --b0b kuhn > >>> > >>>> From genome-mirror-bounces at soe.ucsc.edu Tue Oct 16 09:09:06 2007 > >>>> To: "genome mirror" > >>>> Subject: [Genome-mirror] Problems with slow RSYNC transfers from UCSC > >>>> > >>>> Hi all > >>>> > >>>> We are trying to maintain a full (public) mirror in Australia > including > >> all > >>>> 'download' data (rsync targets 'gbdb', 'mysql', and 'genome'), hosted > >> at a > >>>> university. The reason for this that Australian researchers get slow > >> speeds > >>>> of around 170KB/sec maximum downloading from UCSC, whereas they will > >> get > >>>> very fast download speeds from our mirror. It takes days or weeks to > >>>> complete the 'mysql' and 'gbdb' targets and is a big problem taking > >> even > >>>> longer for the 'genome' target. We can get faster speeds from > different > >> US > >>>> sites; for example, we can get around double that rate downloading > data > >> from > >>>> a Stanford rsync server (similar TCP/IP route from AU), so it is not > >> simply > >>>> a problem with rsync or all traffic on our international link. > >>>> > >>>> I am now at a university in France, and initial testing shows that we > >> get > >>>> the same slow speeds to UCSC here, and again almost double the speed > to > >>>> Stanford, and as well we can get 1MB/s and more from an rsync server > at > >> SDSC > >>>> (also on CENIC network) (I havent been able to test this from AU), so > >> the > >>>> slow rsync speed doesnt seem to be a problem limited to Australia. > >> However, > >>>> it seems that US-based servers get much greater speeds when doing > rsync > >> from > >>>> UCSC - until a few years ago we rented a US-hosted server and were > >> getting > >>>> speeds of at least 500 KB/s between this server and UCSC and I > suspect > >> US > >>>> academic peer networks will see even faster speeds. > >>>> > >>>> Furthermore, the overall throughput is not limited, just individual > >>>> connections. For example, whether I have 1 or 5 rsync connections > open, > >> I > >>>> get a maximum of 170KB/s on each rsync. This is also true of FTP. The > >> slow > >>>> download speeds are particularly a problem with the large files that > >> are > >>>> updated on a daily basis (such as daily EST genbank data and all its > >> derived > >>>> data) since we need to get everything synchronised between daily > >> updates. > >>>> (Files such as goldenPath/bigZips/*.fa.gz, > >>>> goldenPath/database/{est,xenoMrna,gbStatus,...}.txt.gz, > >>>> mysql/*/{est,xenoMrna,gbStatus,...}.MYD). > >>>> > >>>> The problem seems worse when rsyncing large compressed files, > >> particularly > >>>> rsync target 'genome' stuff (/bigZips/ & /database/) which is updated > >> on a > >>>> daily basis since we seem to end up downloading entire files for each > >>>> update. When we download mysql/*MYD files which are updated daily > (eg. > >>>> tables with EST data) we generally get very fast results since > updates > >>>> appear to be appended to the file and rsync only needs to transfer > the > >>>> trailing changed portions. I note that with gzip there is a flag > >>>> '--rsyncable Make rsync-friendly archive' which may help with this. > >>>> > >>>> Traceroutes I have done (see below) indicate the slow Australian and > >> French > >>>> routes to UCSC go through a common point (hpr-ucsc--svl-egm.cenic.net > , > >> the > >>>> first host with UCSC in the name). However, the US server (faster) > does > >> not > >>>> appears to use this route (the first route with UCSC in the name is > >>>> ucsc-ucsc1--dc-oak-dc1-egm.cenic.net) (note that this is an old > >> traceroute). > >>>> Also, the French route to UCSC, Stanford, and SDSC all enter CENIC > the > >> same > >>>> way; the difference in the routes is all between CENIC and the rsync > >> hosts. > >>>> This is also observed for the Australian routes (not shown). This > seems > >> to > >>>> indicate that international TCPIP routes into UCSC follow different > and > >>>> slower paths than US based routes, and that the problem for > >> international > >>>> mirrors (at minimum, France and Australia) may be somewhere between > the > >>>> UCSC-CENIC interface for international (university) traffic ( i.e., > >>>> aarnet.net.au/pacificwave.net for Australian universities and > >>>> Geant2.net/Internet2.edu for French universities). I realise that any > >>>> traceroutes I can do are not necessarily accurate representations of > >> the > >>>> route, but I think they are instructive when taken together with the > >> rsync > >>>> download speeds to different hosts. > >>>> > >>>> > >>>> I would like to know if we are the only ones experiencing this > problem: > >>>> > >>>> - Do any mirrors outside the US get faster rsync downloads than we do > ? > >>>> (more than say 170KB/s?) > >>>> > >>>> - Do any mirrors inside the US get slow downloads? (less than say > >> 500KB/s?) > >>>> > >>>> Also; > >>>> > >>>> - would UCSC increase the max rsync clients (again!) say from 15 to > 20 > >> or > >>>> more (or perhaps allow 5 or 6 or even more from the same IP address) > so > >> that > >>>> international mirrors can run more rsyncs in parallel to overcome > this > >> slow > >>>> per-connection speed? (in particular we want to do mysql/, gbdb/, > >> genome/, > >>>> genome/*/bigZips/ and genome/*/database/ in parallel) > >>>> I believe this would in fact reduce the load on your servers and > >> network > >>>> since we would be able to complete an update in a day or two, rather > >> than > >>>> having just a few connections open for weeks on end continually > >> refreshing > >>>> old data waiting for everything to be in sync before the next daily > >> update. > >>>> - can we test the '--rsyncable' gzip flag when you create compressed > >> files > >>>> for rsync to see if it helps the mirrors (assuming you are not > already > >> using > >>>> it)? > >>>> > >>>> > >>>> Cheers > >>>> > >>>> Mike Pheasant > >>>> > >>>> > >>>> ==== Very fast rsync: Traceroute from France to SDSC > (wwwPDB)========= > >>>> > >>>> traceroute to rsync.wwpdb.org (198.202.122.181), 30 hops max, 40 byte > >>>> packets > >>>> 1 crc-rc1-ge-1-2-0-65.u-strasbg.fr (130.79.47.253) 0.915 ms > 1.087ms > >>>> 1.276 ms > >>>> 2 strasbourg-g3-3.cssi.renater.fr (193.51.184.42) 0.510 ms 0.537ms > >>>> 0.497 ms > >>>> 3 nancy-pos2-0.cssi.renater.fr (193.51.180.41) 38.250 ms 38.344ms > >>>> 38.463 ms > >>>> 4 reims-pos1-0.cssi.renater.fr (193.51.179.137) 8.412 ms 8.600 ms > >> 8.883ms > >>>> 5 * nri-b-pos6-0.cssi.renater.fr (193.51.179.149) 8.012 ms 7.967ms > >>>> 6 renater.rt1.par.fr.geant2.net (62.40.124.69) 8.181 ms 8.340 ms > >> 8.301ms > >>>> 7 so-7-3-0.rt1.gen.ch.geant2.net (62.40.112.29) 16.874 ms 16.862ms > >>>> 16.864 ms > >>>> 8 so-7-2-0.rt1.fra.de.geant2.net (62.40.112.22) 24.941 ms 24.990ms > >>>> 24.942 ms > >>>> 9 abilene-wash-gw.rt1.fra.de.geant2.net (62.40.125.18) 117.936 ms > >>>> 117.883 ms 118.437 ms > >>>> 10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 137.316 ms > >> 137.302ms > >>>> 137.220 ms > >>>> 11 so-0-2-0.0.rtr.hous.net.internet2.edu (64.57.28.43) 154.646 ms > >> 154.577ms > >>>> 154.667 ms > >>>> 12 so-3-0-0.0.rtr.losa.net.internet2.edu (64.57.28.44) 186.533 ms > >> 187.578ms > >>>> 187.560 ms > >>>> 13 hpr-lax-hpr--i2-newnet.cenic.net (137.164.26.132) 186.753 ms > >> 189.032ms * > >>>> 14 riv-hpr--lax-hpr-10ge.cenic.net (137.164.25.5) 193.032 ms * * > >>>> 15 hpr-sdsc-sdsc1--riv-hpr-ge.cenic.net (137.164.27.50) 206.951 ms > >>>> 206.854 ms 206.840 ms > >>>> 16 lightning.sdsc.edu (132.249.30.6) 192.749 ms 192.779 ms > 192.774ms > >>>> > >>>> > >>>> ==== Fast rsync: Traceroute from France to Stanford========= > >>>> > >>>> traceroute to genome-ftp.stanford.edu (171.65.76.47), 30 hops max, 40 > >> byte > >>>> packets > >>>> 1 crc-rc1-ge-1-2-0-65.u-strasbg.fr ( 130.79.47.253) 0.881 ms > 0.980ms > >>>> 1.000 ms > >>>> 2 strasbourg-g3-3.cssi.renater.fr ( 193.51.184.42) 0.500 ms > 0.497ms > >>>> 0.472 ms > >>>> 3 nancy-pos2-0.cssi.renater.fr ( 193.51.180.41) 8.336 ms 8.423 ms > >> 8.569ms > >>>> 4 reims-pos1-0.cssi.renater.fr (193.51.179.137) 8.600 ms 8.783 ms > >> 8.896ms > >>>> 5 nri-b-pos6-0.cssi.renater.fr ( 193.51.179.149) 7.972 ms 7.935ms > >>>> 7.904 ms > >>>> 6 renater.rt1.par.fr.geant2.net ( 62.40.124.69) 8.213 ms 8.183 ms > >> 8.266ms > >>>> 7 so-7-3-0.rt1.gen.ch.geant2.net ( 62.40.112.29) 16.765 ms > 16.866ms > >>>> 16.779 ms > >>>> 8 so-7-2-0.rt1.fra.de.geant2.net (62.40.112.22) 25.118 ms 24.979ms > >>>> 24.922 ms > >>>> 9 abilene-wash-gw.rt1.fra.de.geant2.net ( 62.40.125.18) 139.126 ms > >>>> 139.163 ms 135.276 ms > >>>> 10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 131.089 ms > >> 131.084ms > >>>> 131.095 ms > >>>> 11 so-0-2-0.0.rtr.hous.net.internet2.edu ( 64.57.28.43) 154.667 ms > >>>> 154.695 ms 154.517 ms > >>>> 12 so-3-0-0.0.rtr.losa.net.internet2.edu (64.57.28.44) 186.611 ms > >> 186.570ms > >>>> 186.422 ms > >>>> 13 hpr-lax-hpr--i2-newnet.cenic.net ( 137.164.26.132) 278.881 ms > >> 269.707ms > >>>> 266.963 ms > >>>> 14 svl-hpr--lax-hpr-10ge.cenic.net (137.164.25.13 ) 191.474 ms > >> 191.432 ms > >>>> * > >>>> 15 hpr-stan-ge--svl-hpr.cenic.net ( 137.164.27.162) 191.573 ms > >> 191.851ms > >>>> 191.622 ms > >>>> 16 bbrb-i2.Stanford.EDU (171.64.1.136) 192.174 ms 191.888 ms > >> 191.804 ms > >>>> > >>>> ==== Slow rsync: Traceroute from France to UCSC========= > >>>> > >>>> traceroute to hgdownload.cse.ucsc.edu ( 128.114.119.140), 30 hops > max, > >> 40 > >>>> byte packets > >>>> 1 crc-rc1-ge-1-2-0-65.u-strasbg.fr ( 130.79.47.253) 2.106 ms > 2.075ms > >>>> 2.051 ms > >>>> 2 strasbourg-g3-3.cssi.renater.fr ( 193.51.184.42) 1.466 ms > 1.441ms > >>>> 1.400 ms > >>>> 3 nancy-pos2-0.cssi.renater.fr ( 193.51.180.41) 15.768 ms 15.921ms > >>>> 16.097 ms > >>>> 4 reims-pos1-0.cssi.renater.fr ( 193.51.179.137) 12.477 ms > 12.603ms > >>>> 12.744 ms > >>>> 5 nri-b-pos6-0.cssi.renater.fr ( 193.51.179.149) 8.546 ms 8.532ms > >>>> 8.491 ms > >>>> 6 renater.rt1.par.fr.geant2.net ( 62.40.124.69) 8.819 ms 8.232 ms > >> 8.194ms > >>>> 7 so-7-3-0.rt1.gen.ch.geant2.net ( 62.40.112.29) 16.913 ms > 16.815ms > >>>> 16.795 ms > >>>> 8 so-7-2-0.rt1.fra.de.geant2.net ( 62.40.112.22) 24.893 ms > 24.925ms > >>>> 25.057 ms > >>>> 9 abilene-wash-gw.rt1.fra.de.geant2.net ( 62.40.125.18) 131.087 ms > >>>> 130.878 ms 130.469 ms > >>>> 10 so-0-0-0.0.rtr.atla.net.internet2.edu (64.57.28.6) 136.166 ms > >> 133.322ms > >>>> 133.242 ms > >>>> 11 so-0-2-0.0.rtr.hous.net.internet2.edu (64.57.28.43 ) 154.750 ms > >>>> 154.452 ms 154.438 ms > >>>> 12 so-3-0-0.0.rtr.losa.net.internet2.edu ( 64.57.28.44) 197.920 ms > >>>> 197.889 ms 197.847 ms > >>>> 13 hpr-lax-hpr--i2-newnet.cenic.net ( 137.164.26.132) 183.765 ms * > * > >>>> 14 svl-hpr--lax-hpr-10ge.cenic.net ( 137.164.25.13) 191.428 ms > >> 191.484 ms > >>>> * > >>>> 15 hpr-ucsc--svl-egm.cenic.net ( 137.164.27.86) 193.071 ms > 193.075ms > >>>> 193.034 ms > >>>> 16 comm-g-GE3-4.ucsc.edu (128.114.0.65) 193.016 ms 192.962 ms > >> 192.941 ms > >>>> 17 comm-d3-g-GE1-0-12.ucsc.edu ( 128.114.110.1) 193.267 ms > 193.398ms > >>>> 193.267 ms > >>>> 18 hgdownload.cse.ucsc.edu ( 128.114.119.140) 193.088 ms 192.974ms > >>>> 192.998 ms > >>>> > >>>> > >>>> ==== Slow rsync: Traceroute from Australia to UCSC========= > >>>> > >>>> Tracing route to 128.114.119.140 > >>>> 1 ge-1-0-9.bb1.a.adl.aarnet.net.au ( 203.21.37.17) > >> 0.627ms > >>>> 0.497 ms 0.481 ms > >>>> 2 so-0-1-0.bb1.a.mel.aarnet.net.au (202.158.194.18) > >> 9.490ms > >>>> 9.549 ms 9.488 ms > >>>> 3 so-0-1-0.bb1.b.syd.aarnet.net.au (202.158.194.34) > >> 21.494ms > >>>> 21.808 ms 21.493 ms > >>>> 4 pos1-0.bb1.b.sea.aarnet.net.au ( 202.158.194.94) > >> 186.473 ms > >>>> 186.177 ms 270.973 ms > >>>> 5 cenichpr-1-is-std-779.snvaca.pacificwave.net (207.231.248.129 ) > >> 204.455 ms > >>>> 204.472 ms 204.459 ms > >>>> 6 hpr-ucsc--svl-egm.cenic.net ( 137.164.27.86) > >> 205.963 ms > >>>> 205.785 ms 205.732 ms > >>>> 7 isb-g-GE2-2.ucsc.edu ( 128.114.0.45) > >> 206.094 ms > >>>> 205.613 ms 205.963 ms > >>>> 8 comm-d3-g-GE1-0-11.ucsc.edu (128.114.110.5) > >> 206.450 ms > >>>> 206.154 ms 205.959 ms > >>>> 9 hgdownload.cse.ucsc.edu ( 128.114.119.140) > >> 206.466 ms > >>>> 205.583 ms 205.959 ms > >>>> > >>>> > >>>> ==== Fast rsync: Traceroute from rented US-based server to > >> UCSC=========== > >>>> traceroute to hgdownload.cse.ucsc.edu ( 128.114.119.140), 30 hops > max, > >> 38 > >>>> byte packets > >>>> 1 1.87.1243.static.theplanet.com ( 67.18.135.1) 0.489 ms > >>>> 0.465ms > >>>> 0.508 ms > >>>> 2 gi3-6.dsr02.dllstx4.theplanet.com ( 67.19.255.133) 0.375 ms > >>>> 0.239ms > >>>> 0.251 ms > >>>> 3 vl42.dsr01.dllstx3.theplanet.com (70.85.127.89) 0.508 ms > >> 0.765ms > >>>> 0.633 ms > >>>> 4 25.7f.5546.static.theplanet.com ( 70.85.127.37) 0.492 ms > >>>> 0.361ms > >>>> 0.382 ms > >>>> 5 dal-ix.he.net (206.223.118.37) 0.507 ms > >> 0.475ms > >>>> 0.502 ms > >>>> 6 pos5-0.gsr12012.lax.he.net ( 66.160.184.5) 35.030 ms > >> 35.257ms > >>>> 35.129 ms > >>>> 7 lax-px1--hurricane-ge.cenic.net (198.32.251.85) 35.544 ms > >> 35.388ms > >>>> 35.420 ms > >>>> 8 dc-sac-dc1--lax-dc1-pos.cenic.net ( 137.164.22.127) 45.014 ms > >> 45.126ms > >>>> 45.020 ms > >>>> 9 dc-oak-dc1--csac-dc1-ge.cenic.net (137.164.22.110 ) 47.092 ms > >> 46.942ms > >>>> 46.840 ms > >>>> 10 dc-oak-dc2--oak-dc1-p2p-2.cenic.net ( 137.164.22.195) 46.953 ms > >> 46.813ms > >>>> 46.969 ms > >>>> 11 ucsc-ucsc1--dc-oak-dc1-egm.cenic.net ( 137.164.23.13) 47.595 ms > >> 47.458ms > >>>> 47.488 ms > >>>> 12 comm-g-GE3-5.ucsc.edu (128.114.0.214) 47.479 ms > >> 47.463ms > >>>> 47.351 ms > >>>> 13 comm-d3-g-GE1-0-12.ucsc.edu ( 128.114.110.1) 47.742 ms > >> 47.586ms > >>>> 47.616 ms > >>>> 14 hgdownload.cse.ucsc.edu ( 128.114.119.140) 47.478 ms > >> 47.465ms > >>>> 47.475 ms > >>>> _______________________________________________ > >>>> Genome-mirror mailing list > >>>> Genome-mirror at soe.ucsc.edu > >>>> http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > >>>> > >>> _______________________________________________ > >>> Genome-mirror mailing list > >>> Genome-mirror at soe.ucsc.edu > >>> http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > >> -- > >> > (*)->[]->()->[]->(**)->[]->()->[]->(*)->[]->()->[]->()->[]->()->[]->()->[] > >> > >> (Humboldt University Berlin, Germany)->[]-> ... > >> (University of Maryland, USA)->[]-> ... > >> (King's College London, UK) > >> > >> https://josh.umds.ac.uk/~rschulz > >> > > _______________________________________________ > > Genome-mirror mailing list > > Genome-mirror at soe.ucsc.edu > > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > > -- > (*)->[]->()->[]->(**)->[]->()->[]->(*)->[]->()->[]->()->[]->()->[]->()->[] > > (Humboldt University Berlin, Germany)->[]-> ... > (University of Maryland, USA)->[]-> ... > (King's College London, UK) > > https://josh.umds.ac.uk/~rschulz > From reiner.schulz at kcl.ac.uk Thu Oct 18 03:24:18 2007 From: reiner.schulz at kcl.ac.uk (Reiner Schulz) Date: Thu, 18 Oct 2007 11:24:18 +0100 Subject: [Genome-mirror] Problems with slow RSYNC transfers from UCSC In-Reply-To: <44bb49500710170617p3d4be635u58a93a73e058b94@mail.gmail.com> References: <200710161618.JAA17815@moondance.cse.ucsc.edu> <4715FC44.2030409@kcl.ac.uk> <44bb49500710170617p3d4be635u58a93a73e058b94@mail.gmail.com> Message-ID: <47173452.1070402@kcl.ac.uk> hi Mike, PDB: 240.60kB/s UCSC: 121.74kB/s and here are the traceroute's: traceroute to ftp-remediated-v3.rcsb.org (198.202.122.182), 64 hops max, 40 byte packets 1 159.92.26.3 (159.92.26.3) 0 ms 0 ms 0 ms 2 192.168.150.13 (192.168.150.13) 0 ms 0 ms 0 ms 3 192.168.10.10 (192.168.10.10) 1 ms 0 ms 0 ms 4 192.168.10.14 (192.168.10.14) 0 ms 0 ms 0 ms 5 137.73.1.1 (137.73.1.1) 1 ms 2 ms 5 ms 6 137.73.0.14 (137.73.0.14) 78 ms 91 ms 117 ms 7 137.73.1.105 (137.73.1.105) 85 ms 54 ms 81 ms 8 kcl-gsr.lmn.net.uk (194.83.101.9) 81 ms 87 ms 78 ms 9 po1-0.ulcc-gsr.lmn.net.uk (194.83.100.14) 65 ms 41 ms 41 ms 10 so-1-0-0.lond-sbr1.ja.net (146.97.42.61) 47 ms * 19 ms 11 so-0-0-0.lond-sbr3.ja.net (146.97.33