From m.pheasant at imb.uq.edu.au Tue Apr 3 05:46:29 2007 From: m.pheasant at imb.uq.edu.au (Michael Pheasant) Date: Tue, 03 Apr 2007 22:46:29 +1000 Subject: [Genome-mirror] Rsync error in panTro1 Message-ID: Hi, My rsyncs of mysql keep failing on /mysql/panTro1/ Other databases including /mysql/panTro2/ work fine. Is there a problem with one of the files on your end? Or is it my end? (cant see any thing odd here). Thanks, Mike. The command is: rsync -P --delete -Lrtvz --exclude-from=/home/ucsc/new-scripts/rsync-mysql-exclude rsync://hgdownload.cse.ucsc.edu/mysql/panTro1/ /data/ucsc_mirror_mysql/panTro1/ The output is: receiving file list ... 1759 files to consider 1700 files... xenoEst.MYD 6532398432 84% 46.13MB/s 0:00:25 rsync: read error: Connection reset by peer (104) rsync error: error in rsync protocol data stream (code 12) at io.c(515) rsync: connection unexpectedly closed (31984 bytes received so far) [generator] rsync error: error in rsync protocol data stream (code 12) at io.c(359) __________________________________ Michael Pheasant, PhD Bioinformatics Research Institute for Molecular Bioscience The University of Queensland Brisbane QLD 4067 Australia P: +61-7-3346-2080 M: +61-405-679-541 http://www.imb.uq.edu.au From jlchang at broad.mit.edu Tue Apr 3 12:33:36 2007 From: jlchang at broad.mit.edu (Jean Chang) Date: Tue, 3 Apr 2007 15:33:36 -0400 Subject: [Genome-mirror] Genome-mirror Digest, Vol 14, Issue 1 In-Reply-To: References: Message-ID: <80163A29-09C8-4D3F-8D7C-9A7F61311F8A@broad.mit.edu> Michael, I had the same issue. I turned off compression ( the -z flag) and let the rsync get past the xenoEst.MYD file and that seemed to work. I'm guessing the compression generated a funny character string that was killing the rsync... Once it get's past the file, I kill the rsync and start up again with compression enabled. > xenoEst.MYD > 6532398432 84% 46.13MB/s 0:00:25 > rsync: read error: Connection reset by peer (104) > rsync error: error in rsync protocol data stream (code 12) at io.c > (515) > rsync: connection unexpectedly closed (31984 bytes received so far) > [generator] > rsync error: error in rsync protocol data stream (code 12) at io.c > (359) Hope this helps, Jean PS. I've been getting a permissions error for some Xenopus files. I'm re-doing the rsync to replicated the error and then I'll send it in if it's reproducible - just an FYI. From jlchang at broad.mit.edu Wed Apr 4 12:15:57 2007 From: jlchang at broad.mit.edu (Jean Chang) Date: Wed, 4 Apr 2007 15:15:57 -0400 Subject: [Genome-mirror] Genome-mirror Digest, Vol 14, Issue 2 In-Reply-To: References: Message-ID: <784CB701-00F0-406A-9C5F-72BDC7BAFE2D@broad.mit.edu> Hi all, As I mentioned yesterday, > PS. I've been getting a permissions error for some Xenopus files. I'm > re-doing the rsync to replicated the error and then I'll send it in > if it's reproducible - just an FYI. I'm running: nohup rsync -avP --delete --max-delete=20 rsync:// hgdownload.cse.ucsc.edu/mysql/ /sysman/scratch/jlchang/UCSCmirror/ mysql/ & and I get a series of: rsync: send_files failed to open "/xenTro2/trackDb.MYI" (in mysql): Permission denied (13) rsync: send_files failed to open "/xenTro2/trackDb.frm" (in mysql): Permission denied (13) sent 67945392 bytes received 17405701478 bytes 218859.67 bytes/sec total size is 982653977749 speedup is 56.24 rsync error: some files could not be transferred (code 23) at main.c (1138) Any suggestions? Thanks in advance, Jean From hiram at soe.ucsc.edu Wed Apr 4 13:39:31 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Wed, 04 Apr 2007 13:39:31 -0700 Subject: [Genome-mirror] Genome-mirror Digest, Vol 14, Issue 2 In-Reply-To: <784CB701-00F0-406A-9C5F-72BDC7BAFE2D@broad.mit.edu> References: <784CB701-00F0-406A-9C5F-72BDC7BAFE2D@broad.mit.edu> Message-ID: <46140D03.9020605@soe.ucsc.edu> Good Afternoon Jean: I can't find any error here that would cause this. Can you check your destination directory UCSCmirror/xenTro2/ to see if it has files in it that you can not write to ? And what does your rsync say with the --version argument ? $ rsync --version rsync version 2.6.8 protocol version 29 --Hiram Jean Chang wrote: > Hi all, > > As I mentioned yesterday, > >> PS. I've been getting a permissions error for some Xenopus files. I'm >> re-doing the rsync to replicated the error and then I'll send it in >> if it's reproducible - just an FYI. > > > I'm running: > nohup rsync -avP --delete --max-delete=20 rsync:// > hgdownload.cse.ucsc.edu/mysql/ /sysman/scratch/jlchang/UCSCmirror/ > mysql/ & > > and I get a series of: > rsync: send_files failed to open "/xenTro2/trackDb.MYI" (in mysql): > Permission denied (13) > rsync: send_files failed to open "/xenTro2/trackDb.frm" (in mysql): > Permission denied (13) > > sent 67945392 bytes received 17405701478 bytes 218859.67 bytes/sec > total size is 982653977749 speedup is 56.24 > rsync error: some files could not be transferred (code 23) at main.c > (1138) > > Any suggestions? > > Thanks in advance, > > Jean From m.pheasant at imb.uq.edu.au Wed Apr 4 14:16:11 2007 From: m.pheasant at imb.uq.edu.au (Michael Pheasant) Date: Thu, 05 Apr 2007 07:16:11 +1000 Subject: [Genome-mirror] Genome-mirror Digest, Vol 14, Issue 1 In-Reply-To: <80163A29-09C8-4D3F-8D7C-9A7F61311F8A@broad.mit.edu> References: <80163A29-09C8-4D3F-8D7C-9A7F61311F8A@broad.mit.edu> Message-ID: Thanks Jean, Turning of 'z' flag gets me past xenoEst file in panTro1 as well. Very strange. Mike > Michael, > > I had the same issue. I turned off compression ( the -z flag) and let > the rsync get past the xenoEst.MYD file and that seemed to work. I'm > guessing the compression generated a funny character string that was > killing the rsync... Once it get's past the file, I kill the rsync > and start up again with compression enabled. > >> xenoEst.MYD >> 6532398432 84% 46.13MB/s 0:00:25 >> rsync: read error: Connection reset by peer (104) >> rsync error: error in rsync protocol data stream (code 12) at io.c >> (515) >> rsync: connection unexpectedly closed (31984 bytes received so far) >> [generator] >> rsync error: error in rsync protocol data stream (code 12) at io.c >> (359) > > Hope this helps, > > Jean > > PS. I've been getting a permissions error for some Xenopus files. I'm > re-doing the rsync to replicated the error and then I'll send it in > if it's reproducible - just an FYI. > > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror __________________________________ Michael Pheasant, PhD Bioinformatics Research Institute for Molecular Bioscience The University of Queensland Brisbane QLD 4067 Australia P: +61-7-3346-2080 M: +61-405-679-541 http://www.imb.uq.edu.au From ebr26 at student.canterbury.ac.nz Wed Apr 4 22:01:48 2007 From: ebr26 at student.canterbury.ac.nz (Emmanuel Buschiazzo) Date: Thu, 05 Apr 2007 17:01:48 +1200 Subject: [Genome-mirror] cvs installation kent source Message-ID: <1175749308.11293.1.camel@localhost.localdomain> Hello, Has the CVS password changed from "genome"? It would not accept this one when I try it... Thanks in advance for your help! With best regards, Emmanuel. From ann at soe.ucsc.edu Thu Apr 5 15:17:10 2007 From: ann at soe.ucsc.edu (Ann Zweig) Date: Thu, 05 Apr 2007 15:17:10 -0700 Subject: [Genome-mirror] cvs installation kent source In-Reply-To: <1175749308.11293.1.camel@localhost.localdomain> References: <1175749308.11293.1.camel@localhost.localdomain> Message-ID: <46157566.70704@soe.ucsc.edu> Hello Emmanuel, No, the password has not changed. Perhaps it was a temporary glitch. We are able to log in fine from outside our environment. Please give it another try. Here's a link to the detailed instructions: http://genome.ucsc.edu/admin/cvs.html Regards, ---------- Ann Zweig UCSC Genome Bioinformatics Group http://genome.ucsc.edu Emmanuel Buschiazzo wrote: > Hello, > > Has the CVS password changed from "genome"? > It would not accept this one when I try it... > > Thanks in advance for your help! > > With best regards, > > Emmanuel. > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror From donnak at soe.ucsc.edu Fri Apr 6 12:15:05 2007 From: donnak at soe.ucsc.edu (Donna Karolchik) Date: Fri, 6 Apr 2007 12:15:05 -0700 Subject: [Genome-mirror] Possible inconsistencies with hg18 Known Genes today Message-ID: <014001c7787f$e530d170$0ba8a8c0@donnakLT> We are in the process of releasing a new UCSC-generated gene prediction annotation (UCSC Genes) on the public hg18 Genome Browser today. While this is being staged (Friday 6 April, 12-5pm PDT), you may experience problems with the hg18 Known Genes and UCSC Genes tracks, including broken links, error messages, and problems with gene name searches. I will post an update message later today when the new annotation has been fully released to the public site. Apologies for the last-minute warning on this, and thanks for your patience during this transition. -Donna ----------------------------------- Donna Karolchik UCSC Genome Bioinformatics Group http://genome.ucsc.edu From kuhn at soe.ucsc.edu Fri Apr 6 12:30:32 2007 From: kuhn at soe.ucsc.edu (Robert Kuhn) Date: Fri, 6 Apr 2007 12:30:32 -0700 Subject: [Genome-mirror] new proteins db Message-ID: <200704061930.MAA19851@moondance.cse.ucsc.edu> Dear mirrors: We are releasing a new database, which is an update of the proteins data supporting the new hg18 Known Genes track (and KG on other assemblies to follow). The new database is: proteins070202 [7.1 Gb] It should be available on the downloads server on Monday. thanks, --b0b kuhn ucsc genome bioinformatics group From donnak at soe.ucsc.edu Fri Apr 6 17:14:43 2007 From: donnak at soe.ucsc.edu (Donna Karolchik) Date: Fri, 6 Apr 2007 17:14:43 -0700 Subject: [Genome-mirror] New UCSC gene prediction set released Message-ID: <02ca01c778a9$c04c9900$0ba8a8c0@donnakLT> We are pleased to announce the release of a new gene prediction set, UCSC Genes, on the latest human Genome Browser (hg18, NCBI Build 36). This annotation, which includes putative non-coding genes as well as protein-coding genes and 99.9% of RefSeq genes, is the next generation of the Known Genes set that UCSC has been providing for several years and supersedes the existing Known Genes annotation on the hg18 assembly. The UCSC Genes is a moderately conservative prediction set based on data from RefSeq, GenBank, and UniProt. Each entry requires the support of one GenBank RNA sequence plus at least one additional line of evidence, with the exception of RefSeq RNAs, which require no additional evidence. Some of the non-coding transcripts in the set may actually code for protein, but the evidence for the associated protein is weak at best. Compared to RefSeq, this gene set generally has about 10% more protein-coding genes, approximately five times as many putative non-coding genes, and about twice as many splice variants. The UCSC Genes set is produced using a computational pipeline developed at UCSC by Jim Kent, Chuck Sugnet and Mark Diekhans. For detailed information about the process used to construct the genes set, see the track description page. In upcoming months, we plan to release UCSC Genes sets on other organisms in addition to human. As part of this change, we are now using our own UCSC Genes accession numbers as the primary key into the underlying knownGene table, rather than the GenBank mRNA accessions we used in the previous Known Genes prediction set. Note that this may affect external sites with URLs that link into our genes track using the older-style accessions. We will continue to provide the older Known Genes track on hg18 under the name "Old Known Genes". You may find the following tables useful in referencing the older gene set and converting between the two sets: - knownGeneOld2: new name for table underlying the old Known Genes (previously called knownGene) - kgXrefOld2: new name for table that contains data for converting old Known Genes IDs to other IDs (previously called kgXref) - kg2ToKg3: data for converting old Known Genes IDs to the newer UCSC Genes IDs We'd like to acknowledge the many people affiliated with the UCSC Genome Bioinformatics group who worked hard to release this new annotation: developers Jim Kent, Mark Diekhans, and Fan Hsu (with technical support from several other engineers in the group); David Haussler; our splendid QA team -- Archana Thakkapallayil, Ann Zweig, Robert Kuhn, Kayla Smith, and Brooke Rhead; our build engineer -- Andy Pohl; and our sysadmin group. We'd also like to thank Chuck Sugnet for his input, the people and organizations maintaining the RefSeq, UniProt, and GenBank databases, and the scientists worldwide who have contributed to them. If you have any questions about this new release, feel free to contact us at genome at soe.ucsc.edu (general questions) or genome-mirror at soe.ucsc.edu (mirror-specific questions). -Donna ----------------------------------- Donna Karolchik UCSC Genome Bioinformatics Group http://genome.ucsc.edu From jlchang at broad.mit.edu Sun Apr 8 17:49:26 2007 From: jlchang at broad.mit.edu (Jean Chang) Date: Sun, 8 Apr 2007 20:49:26 -0400 Subject: [Genome-mirror] rsync issues - empty or too-small directories Message-ID: <7D1158C4-9CA5-4894-A91E-BA3DFF2BE123@broad.mit.edu> Hiram, I am mirroring this data as a role user, "apsg". I don't think there are any permissions problems here since apsg is running the rsync. drwxr-sr-x 2 apsg apsg 8192 Apr 2 21:30 xenTro2/ and the problem is more than xenTro2 If I check the amount of data (using du -k) for the various mysql directories, empty or suspiciously small directories include: 28 danRer1 0 go 140 hgcentarchive 184 hgcentral 0 hgw1.pid 0 hgw2.pid 0 hgw7.pid 28 ib_arch_log_0000000000 4 innodb.status.15447 4 innodb.status.2809 4 innodb.status.2831 4 innodb.status.2837 4 innodb.status.2898 4 innodb.status.2937 4 innodb.status.5435 8 lost+found 8 proteins040515 8 proteins050415 8 proteins051015 8 proteins060115 0 proteome 8 rheMac1 8 rheMac2 8 rn3 8 sep30 8 sp040315 8 sp040515 8 sp050415 8 sp051015 8 sp060115 8 strPur1 8 test 0 uniProt 8 xenTro1 Note that xenTro2 isn't even on the above list: 186324 xenTro2 because it has one file in it, -rw-rw-r-- 1 apsg apsg 190591076 Mar 16 22:48 all_est.MYD To answer your other question: rsync --version rsync version 2.6.3pre1 protocol version 28 Copyright (C) 1996-2004 by Andrew Tridgell and others Capabilities: 64-bit files, socketpairs, hard links, acls, symlinks, batchfiles, inplace, IPv6, 64-bit system inums, 64-bit internal inums, SLP Any ideas? Help! Thanks, Jean From hiram at soe.ucsc.edu Mon Apr 9 09:26:00 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Mon, 09 Apr 2007 09:26:00 -0700 Subject: [Genome-mirror] rsync issues - empty or too-small directories In-Reply-To: <7D1158C4-9CA5-4894-A91E-BA3DFF2BE123@broad.mit.edu> References: <7D1158C4-9CA5-4894-A91E-BA3DFF2BE123@broad.mit.edu> Message-ID: <461A6918.8000001@soe.ucsc.edu> Good Morning Jean: This is a good mystery you have here. At this point all I can do is speculate. rsync is a reliable tool and we depend upon it here completely to push our data around. The best thing about it, if it ever encounters any kind of transmission difficulty and exits with a failure, when it is simply rerun in the same way, it will recover from where it was interrupted and complete the transaction. Potential problems: 1. firewall interference or timeouts causing the connection to be dropped 2. version incompatibilities between rsync client and server our server rsync is version 2.6.9 protocol version 29 on a FreeBSD system we have used clients as old as 2.6.3 protocol version 28 with no difficulty 3. receiving filesystems are not enabled for large files over 2 Gb You can always try a single directory rsync with -v verbose on to see what it says it is doing, for example: rsync -v -a --progress --stats rsync://hgdownload.cse.ucsc.edu/mysql/xenTro2/ . Or just the files it complains about: rsync -v -a --progress --stats rsync://hgdownload.cse.ucsc.edu/mysql/xenTro2/track* . --Hiram Jean Chang wrote: > Hiram, > > I am mirroring this data as a role user, "apsg". I don't think there > are any permissions problems here since apsg is running the rsync. > > drwxr-sr-x 2 apsg apsg 8192 Apr 2 21:30 xenTro2/ > > and the problem is more than xenTro2 > > If I check the amount of data (using du -k) for the various mysql > directories, empty or suspiciously small directories include: > 28 danRer1 > 0 go > 140 hgcentarchive > 184 hgcentral > 0 hgw1.pid > 0 hgw2.pid > 0 hgw7.pid > 28 ib_arch_log_0000000000 > 4 innodb.status.15447 > 4 innodb.status.2809 > 4 innodb.status.2831 > 4 innodb.status.2837 > 4 innodb.status.2898 > 4 innodb.status.2937 > 4 innodb.status.5435 > 8 lost+found > 8 proteins040515 > 8 proteins050415 > 8 proteins051015 > 8 proteins060115 > 0 proteome > 8 rheMac1 > 8 rheMac2 > 8 rn3 > 8 sep30 > 8 sp040315 > 8 sp040515 > 8 sp050415 > 8 sp051015 > 8 sp060115 > 8 strPur1 > 8 test > 0 uniProt > 8 xenTro1 > > Note that xenTro2 isn't even on the above list: > > 186324 xenTro2 > > because it has one file in it, > > -rw-rw-r-- 1 apsg apsg 190591076 Mar 16 22:48 all_est.MYD > > To answer your other question: > rsync --version > rsync version 2.6.3pre1 protocol version 28 > Copyright (C) 1996-2004 by Andrew Tridgell and others > > Capabilities: 64-bit files, socketpairs, hard links, acls, symlinks, > batchfiles, > inplace, IPv6, 64-bit system inums, 64-bit internal > inums, SLP > > Any ideas? Help! > > Thanks, > > Jean From aamp at ucsc.edu Tue Apr 10 08:49:50 2007 From: aamp at ucsc.edu (Andy Pohl) Date: Tue, 10 Apr 2007 08:49:50 -0700 Subject: [Genome-mirror] v156 browser available Message-ID: <9fa943760704100849y1355e5edp1fd00f87a7f32fa9@mail.gmail.com> Hello, We skipped releasing v155 because there was too much to patch up for that one. So we instead we just worked another week to make v156. v156 source is at: http://hgdownload.cse.ucsc.edu/admin/jksrc.zip or labeled with source number: http://hgdownload.cse.ucsc.edu/admin/jksrc.v156.zip cartDump cartReset das hgc hgBlat hgConvert hgCustom hgEncodeDataVersions hgGateway hgGene hgGenome hgLiftOver hgNear hgPcr hgSession hgTables hgText hgTracks hgTrackUi hgVisiGene mkEncodeFrameset pbGateway pbGlobal pbTracks phyloGif /usr/local/apache/cgi-bin/all.joiner /usr/local/apache/cgi-bin/hgNearData/* /usr/local/apache/cgi-bin/hgGeneData/* /usr/local/apache/cgi-bin/hgcData/* /usr/local/apache/cgi-bin/hgCgiData/* /usr/local/apache/cgi-bin/loader/* Contents of v156 may be seen at: http://genome-test.cse.ucsc.edu/builds/versions.html Please check both v156-preview and v156-final to see all the code and data changes in this release. Andy Pohl UCSC CBSE From davide.cittaro at ifom-ieo-campus.it Tue Apr 10 13:32:16 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Tue, 10 Apr 2007 22:32:16 +0200 Subject: [Genome-mirror] a couple of questions on mirroring GB Message-ID: <44057B97-264F-466D-8F42-8DBF6D538404@ifom-ieo-campus.it> Hi all, I have some doubts on mirror procedures. In your announce page you usually list the bug fixes and the binaries affected. Since the entire suite is made of hundreds of binaries, I wonder if there are updates for binaries that are not "cgi" stuff. As a consequence I wonder if I have to restart the gfServer I normally keep alive. The last question is about the "trash" directory. Can I safely remove files from there (in other words: can I empty the trash)? Thanks Davide /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro at ifom-ieo-campus.it */ From hiram at soe.ucsc.edu Tue Apr 10 14:41:59 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Tue, 10 Apr 2007 14:41:59 -0700 Subject: [Genome-mirror] a couple of questions on mirroring GB In-Reply-To: <44057B97-264F-466D-8F42-8DBF6D538404@ifom-ieo-campus.it> References: <44057B97-264F-466D-8F42-8DBF6D538404@ifom-ieo-campus.it> Message-ID: <461C04A7.5030804@soe.ucsc.edu> Good Afternoon Davide: If you are curious about any potential changes to critical binaries you use other than the CGI binaries, you can always take a look in their CVS history to see when they last changed. For example, in kent/src/gfServer running a 'cvs log gfServer.c' I see the most recent three changes indicate that the most significant thing that happened to gfServer.c was most likely in December of 2005: ---------------------------- revision 1.54 date: 2007/03/31 19:38:13; author: markd; state: Exp; lines: +3 -2 fixed various issues detected by GCC 4.3 ---------------------------- revision 1.53 date: 2006/11/16 16:19:00; author: angie; state: Exp; lines: +2 -13 Resolved -Wall warning: unused variable. ---------------------------- revision 1.52 date: 2005/12/15 16:38:23; author: kent; state: Exp; lines: +13 -12 Adding maxGap to options table, and alphabetizing that table. ---------------------------- Of course there could also be fixes in the two libraries that link with this binary. But actual fixes to library functions are rare. They may happen once in a while, but usually the only changes to library functions are additional functions added. As for cleaning your trash files, yes please do. Operate a simple cron job that has a find and remove based on age. We have two different age times here for trash files, 8 hours for most files, but 48 hours for custom track trash files. And the age time is since last access, thus if it is being used over time it will not disappear. Our find and remove is actually two commands to implement the two age times: find /trash \! \( -regex "/trash/ct/.*" -or -regex "/trash/hgSs/.*" \) -type f -amin +480 -exec rm -f {} \; find /trash \( -regex "/trash/ct/.*" -or -regex "/trash/hgSs/.*" \) -type f -amin +2880 -exec rm -f {} \; --Hiram Davide Cittaro wrote: > Hi all, I have some doubts on mirror procedures. > In your announce page you usually list the bug fixes and the binaries > affected. Since the entire suite is made of hundreds of binaries, I > wonder if there are updates for binaries that are not "cgi" stuff. > As a consequence I wonder if I have to restart the gfServer I > normally keep alive. > The last question is about the "trash" directory. Can I safely remove > files from there (in other words: can I empty the trash)? > > Thanks > > Davide > /* > Davide Cittaro > HPC and Bioinformatics Systems @ Informatics Core From ann at soe.ucsc.edu Tue Apr 10 14:52:29 2007 From: ann at soe.ucsc.edu (Ann Zweig) Date: Tue, 10 Apr 2007 14:52:29 -0700 Subject: [Genome-mirror] releasing fr2 assembly Message-ID: <461C071D.7030401@soe.ucsc.edu> Hello mirror sites, Later this week, we are intending to release the newest fugu assembly: fr2. Please be prepared to host the following new data: fr2 MySQL database 5.0 G files in /gbdb/fr2/* 2.1 G Additionally there are net and chain tables from other organisms that will be released to the respective annotation databases. These tables are named netFr2 and *chainFr2*. The size of these tables is as follows: hg18 207 MB tetNig1 669 MB gasAcu1 732 MB danRer4 1232 MB galGal3 87 MB =============== total: 2.8 GB Regards, ---------- Ann Zweig UCSC Genome Bioinformatics Group http://genome.ucsc.edu From galt at soe.ucsc.edu Tue Apr 10 15:09:58 2007 From: galt at soe.ucsc.edu (Galt Barber) Date: Tue, 10 Apr 2007 15:09:58 -0700 (PDT) Subject: [Genome-mirror] a couple of questions on mirroring GB In-Reply-To: <461C04A7.5030804@soe.ucsc.edu> References: <44057B97-264F-466D-8F42-8DBF6D538404@ifom-ieo-campus.it> <461C04A7.5030804@soe.ucsc.edu> Message-ID: BLAT belongs to Jim Kent. He releases new versions from time to time. Our gfServers are not updated weekly or at any regular interval. -Galt On Tue, 10 Apr 2007, Hiram Clawson wrote: > Good Afternoon Davide: > > If you are curious about any potential changes to critical binaries > you use other than the CGI binaries, you can always take a look > in their CVS history to see when they last changed. For example, > in kent/src/gfServer running a 'cvs log gfServer.c' I see the most > recent three changes indicate that the most significant thing that > happened to gfServer.c was most likely in December of 2005: > ---------------------------- > revision 1.54 > date: 2007/03/31 19:38:13; author: markd; state: Exp; lines: +3 -2 > fixed various issues detected by GCC 4.3 > ---------------------------- > revision 1.53 > date: 2006/11/16 16:19:00; author: angie; state: Exp; lines: +2 -13 > Resolved -Wall warning: unused variable. > ---------------------------- > revision 1.52 > date: 2005/12/15 16:38:23; author: kent; state: Exp; lines: +13 -12 > Adding maxGap to options table, and alphabetizing that table. > ---------------------------- > > Of course there could also be fixes in the two libraries that link with > this binary. But actual fixes to library functions are rare. They may > happen once in a while, but usually the only changes to library functions > are additional functions added. > > As for cleaning your trash files, yes please do. Operate a simple cron > job that has a find and remove based on age. We have two different > age times here for trash files, 8 hours for most files, but 48 hours for > custom track trash files. And the age time is since last access, thus > if it is being used over time it will not disappear. Our find and remove > is actually two commands to implement the two age times: > > find /trash \! \( -regex "/trash/ct/.*" -or -regex "/trash/hgSs/.*" \) -type f -amin +480 -exec rm -f {} \; > find /trash \( -regex "/trash/ct/.*" -or -regex "/trash/hgSs/.*" \) -type f -amin +2880 -exec rm -f {} \; > > --Hiram > > Davide Cittaro wrote: > > Hi all, I have some doubts on mirror procedures. > > In your announce page you usually list the bug fixes and the binaries > > affected. Since the entire suite is made of hundreds of binaries, I > > wonder if there are updates for binaries that are not "cgi" stuff. > > As a consequence I wonder if I have to restart the gfServer I > > normally keep alive. > > The last question is about the "trash" directory. Can I safely remove > > files from there (in other words: can I empty the trash)? > > > > Thanks > > > > Davide > > /* > > Davide Cittaro > > HPC and Bioinformatics Systems @ Informatics Core > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From hiram at soe.ucsc.edu Tue Apr 10 17:04:12 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Tue, 10 Apr 2007 17:04:12 -0700 Subject: [Genome-mirror] custom track database feature Message-ID: <461C25FC.7090201@soe.ucsc.edu> Good Afternoon Mirror Sites: With the release of v156 browser code, we now have a new method of keeping custom tracks in a combination of trash file and database format. A small /trash/ct/ file is maintained to provide pointers to database tables where the data is stored. This feature can be turned on by appropriate configuration items in the cgi-bin/hg.conf file. Please note the discussion of this function and how to get it properly turned on in either of these two formats: 1. In the source tree file src/product/README.customTracks.dataBase 2. In the Wiki page: http://genomewiki.ucsc.edu/index.php/Using_custom_track_database Currently the source code supports both the existing file-based format and the database format. At some point in the near future, the support of the file-based format will most likely become obsolete. Once you make this switch, you don't want to go back to the older format, otherwise existing custom tracks at that time would be lost. --Hiram From davide.cittaro at ifom-ieo-campus.it Wed Apr 11 01:19:38 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Wed, 11 Apr 2007 10:19:38 +0200 Subject: [Genome-mirror] new proteins db In-Reply-To: <200704061930.MAA19851@moondance.cse.ucsc.edu> References: <200704061930.MAA19851@moondance.cse.ucsc.edu> Message-ID: On Apr 6, 2007, at 9:30 PM, Robert Kuhn wrote: > > Dear mirrors: > > proteins070202 > [7.1 Gb] is the database "proteome" the same of proteins070202? Better, is proteome a symlink to the latest proteins* database? d /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro at ifom-ieo-campus.it */ From reiner.schulz at kcl.ac.uk Wed Apr 11 01:13:55 2007 From: reiner.schulz at kcl.ac.uk (Reiner Schulz) Date: Wed, 11 Apr 2007 09:13:55 +0100 Subject: [Genome-mirror] rsync issues - empty or too-small directories In-Reply-To: <461A6918.8000001@soe.ucsc.edu> References: <7D1158C4-9CA5-4894-A91E-BA3DFF2BE123@broad.mit.edu> <461A6918.8000001@soe.ucsc.edu> Message-ID: <461C98C3.7030807@kcl.ac.uk> Jean, i have been experiencing the same rsync problems (protocol errors and denied permission) on my end: rsync version 2.6.6 protocol version 29 x86_64 GNU/Linux (Ubuntu server 6.06 LTS, 2.6.15-28-amd64-server #1 SMP kernel) Reiner Hiram Clawson wrote: > Good Morning Jean: > > This is a good mystery you have here. At this point all I can do is speculate. > rsync is a reliable tool and we depend upon it here completely to push our > data around. The best thing about it, if it ever encounters any kind of > transmission difficulty and exits with a failure, when it is simply rerun > in the same way, it will recover from where it was interrupted and complete > the transaction. > > Potential problems: > 1. firewall interference or timeouts causing the connection to be dropped > 2. version incompatibilities between rsync client and server > our server rsync is version 2.6.9 protocol version 29 on a FreeBSD system > we have used clients as old as 2.6.3 protocol version 28 with no difficulty > 3. receiving filesystems are not enabled for large files over 2 Gb > > You can always try a single directory rsync with -v verbose on to see what > it says it is doing, for example: > rsync -v -a --progress --stats rsync://hgdownload.cse.ucsc.edu/mysql/xenTro2/ . > > Or just the files it complains about: > rsync -v -a --progress --stats rsync://hgdownload.cse.ucsc.edu/mysql/xenTro2/track* . > > --Hiram > > Jean Chang wrote: >> Hiram, >> >> I am mirroring this data as a role user, "apsg". I don't think there >> are any permissions problems here since apsg is running the rsync. >> >> drwxr-sr-x 2 apsg apsg 8192 Apr 2 21:30 xenTro2/ >> >> and the problem is more than xenTro2 >> >> If I check the amount of data (using du -k) for the various mysql >> directories, empty or suspiciously small directories include: >> 28 danRer1 >> 0 go >> 140 hgcentarchive >> 184 hgcentral >> 0 hgw1.pid >> 0 hgw2.pid >> 0 hgw7.pid >> 28 ib_arch_log_0000000000 >> 4 innodb.status.15447 >> 4 innodb.status.2809 >> 4 innodb.status.2831 >> 4 innodb.status.2837 >> 4 innodb.status.2898 >> 4 innodb.status.2937 >> 4 innodb.status.5435 >> 8 lost+found >> 8 proteins040515 >> 8 proteins050415 >> 8 proteins051015 >> 8 proteins060115 >> 0 proteome >> 8 rheMac1 >> 8 rheMac2 >> 8 rn3 >> 8 sep30 >> 8 sp040315 >> 8 sp040515 >> 8 sp050415 >> 8 sp051015 >> 8 sp060115 >> 8 strPur1 >> 8 test >> 0 uniProt >> 8 xenTro1 >> >> Note that xenTro2 isn't even on the above list: >> >> 186324 xenTro2 >> >> because it has one file in it, >> >> -rw-rw-r-- 1 apsg apsg 190591076 Mar 16 22:48 all_est.MYD >> >> To answer your other question: >> rsync --version >> rsync version 2.6.3pre1 protocol version 28 >> Copyright (C) 1996-2004 by Andrew Tridgell and others >> >> Capabilities: 64-bit files, socketpairs, hard links, acls, symlinks, >> batchfiles, >> inplace, IPv6, 64-bit system inums, 64-bit internal >> inums, SLP >> >> Any ideas? Help! >> >> Thanks, >> >> Jean > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror -- (*)->[]->()->[]->(**)->[]->()->[]->(*)->[]->()->[]->()->[]->()->[]->()->[] (Humboldt University Berlin, Germany)->[]-> ... (University of Maryland, USA)->[]-> ... (King's College London, UK) https://josh.umds.ac.uk/~rschulz From kuhn at soe.ucsc.edu Wed Apr 11 08:45:29 2007 From: kuhn at soe.ucsc.edu (Robert Kuhn) Date: Wed, 11 Apr 2007 08:45:29 -0700 Subject: [Genome-mirror] new proteins db Message-ID: <200704111545.IAA24621@moondance.cse.ucsc.edu> Davide, yes, "proteome" is a symlink to the latest proteinsYYMMDD database. The earlier databases are used by the browser per the table, gdbPdb, in hgcentral. Similarly, "uniProt" is a symlink to the corresponding spYYMMDD databse. --b0b > From davide.cittaro at ifom-ieo-campus.it Wed Apr 11 01:19:49 2007 > Cc: genome-mirror at soe.ucsc.edu > Subject: Re: [Genome-mirror] new proteins db > To: Robert Kuhn > > > On Apr 6, 2007, at 9:30 PM, Robert Kuhn wrote: > > > > > Dear mirrors: > > > > proteins070202 > > [7.1 Gb] > > is the database "proteome" the same of proteins070202? Better, is > proteome a symlink to the latest proteins* database? > > d > > /* > Davide Cittaro > HPC and Bioinformatics Systems @ Informatics Core > > IFOM - Istituto FIRC di Oncologia Molecolare > via adamello, 16 > 20139 Milano > Italy > > tel.: +39(02)574303007 > e-mail: davide.cittaro at ifom-ieo-campus.it > */ > > > > --Apple-Mail-3-403883467 > Content-Transfer-Encoding: quoted-printable > Content-Type: text/html; > charset=ISO-8859-1 > > -khtml-line-break: after-white-space; ">
On Apr 6, 2007, at = > 9:30 PM, Robert Kuhn wrote:

class=3D"Apple-interchange-newline">
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = > margin-left: 0px; min-height: 14px; ">
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Dear = > mirrors:
margin-bottom: 0px; margin-left: 0px; min-height: 14px; ">
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = > margin-left: 0px; "> style=3D"white-space:pre"> proteins070202
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = > margin-left: 0px; "> style=3D"white-space:pre"> [7.1 = > Gb]

is the database "proteome" the same = > of proteins070202? Better, is proteome a symlink to the latest proteins* = > database?

class=3D"khtml-block-placeholder">
d

class=3D"Apple-style-span" style=3D"border-collapse: separate; = > border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; = > font-size: 12px; font-style: normal; font-variant: normal; font-weight: = > normal; letter-spacing: normal; line-height: normal; text-align: auto; = > -khtml-text-decorations-in-effect: none; text-indent: 0px; = > -apple-text-size-adjust: auto; text-transform: none; orphans: 2; = > white-space: normal; widows: 2; word-spacing: 0px; "> class=3D"Apple-style-span" style=3D"border-collapse: separate; = > border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; = > font-size: 12px; font-style: normal; font-variant: normal; font-weight: = > normal; letter-spacing: normal; line-height: normal; text-align: auto; = > -khtml-text-decorations-in-effect: none; text-indent: 0px; = > -apple-text-size-adjust: auto; text-transform: none; orphans: 2; = > white-space: normal; widows: 2; word-spacing: 0px; ">
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = > margin-left: 0px; ">/*
0px; margin-bottom: 0px; margin-left: 0px; ">Davide Cittaro
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = > margin-left: 0px; ">HPC and Bioinformatics Systems @ Informatics = > Core
margin-bottom: 0px; margin-left: 0px; font: normal normal normal = > 12px/normal Helvetica; min-height: 14px; ">
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = > margin-left: 0px; ">IFOM - Istituto FIRC di Oncologia = > Molecolare
margin-bottom: 0px; margin-left: 0px; ">via adamello, 16
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = > margin-left: 0px; ">20139 Milano
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; = > ">Italy
margin-bottom: 0px; margin-left: 0px; font: normal normal normal = > 12px/normal Helvetica; min-height: 14px; ">
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = > margin-left: 0px; ">tel.: +39(02)574303007
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; = > margin-left: 0px; ">*/

class=3D"Apple-interchange-newline"> = >

= > > --Apple-Mail-3-403883467-- > From m.pheasant at imb.uq.edu.au Wed Apr 11 02:59:23 2007 From: m.pheasant at imb.uq.edu.au (Michael Pheasant) Date: Wed, 11 Apr 2007 19:59:23 +1000 Subject: [Genome-mirror] rsync issues - empty or too-small directories In-Reply-To: <461C98C3.7030807@kcl.ac.uk> References: <7D1158C4-9CA5-4894-A91E-BA3DFF2BE123@broad.mit.edu> <461A6918.8000001@soe.ucsc.edu> <461C98C3.7030807@kcl.ac.uk> Message-ID: Hi, I also have been experienceing rsync errors. Previously it happened with the same file on panTro1. Jean contacted me, experiencing the same thing and said the workaround is to remove the 'z' compress flag. This works. To try to fix the problem, I upgraded my rsync from 2.6.3 to the latest stable version 2.6.9, but the problem remains, the latest error was on mm5/xenoExt.MYD. I remove the 'z' flag to get past this file, then I get the problem with mm6/xenoEst.MYD. You almost get through the file, then it just times out with an error (last reported progress: 6413126720 99% 33.82MB/s) Conceivably its actually the next file xenoExt.MYI but the error means we never see the output. So, it seems to be something to do with the 'z' flag, and maybe at the UCSC end since more than one site is experienceing it with mutliple versions of rsync client. *) Could maybe the xenoExt.MYD files be so large (6GB+) that the compression is taking too long and the client socket is timing out? Cheers, Mike The last two errors: rsync: connection unexpectedly closed (1366455 bytes received so far) [generator] rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(453) [generator=2.6.9] rsync: writefd_unbuffered failed to write 4092 bytes [generator]: Broken pipe (32)rsync: read error: Connection reset by peer (104) rsync error: error in rsync protocol data stream (code 12) at io.c(1122) [generator=2.6.9] rsync error: error in rsync protocol data stream (code 12) at io.c(604) [receiver=2.6.9] rsync: connection unexpectedly closed (337552087 bytes received so far) [receiver] rsync: writefd_unbuffered failed to write 4092 bytes [generator]: Connection reset by peer (104) rsync error: error in rsync protocol data stream (code 12) at io.c(1122) [generator=2.6.9] rsync error: error in rsync protocol data stream (code 12) at io.c(453) [receiver=2.6.9] > Jean, i have been experiencing the same rsync problems (protocol errors > and denied permission) on my end: > rsync version 2.6.6 protocol version 29 > x86_64 GNU/Linux (Ubuntu server 6.06 LTS, 2.6.15-28-amd64-server #1 SMP > kernel) > > Reiner > > Hiram Clawson wrote: >> Good Morning Jean: >> >> This is a good mystery you have here. At this point all I can do is >>speculate. >> rsync is a reliable tool and we depend upon it here completely to push our >> data around. The best thing about it, if it ever encounters any kind of >> transmission difficulty and exits with a failure, when it is simply rerun >> in the same way, it will recover from where it was interrupted and complete >> the transaction. >> >> Potential problems: >> 1. firewall interference or timeouts causing the connection to be dropped >> 2. version incompatibilities between rsync client and server >> our server rsync is version 2.6.9 protocol version 29 on a FreeBSD system >> we have used clients as old as 2.6.3 protocol version 28 with no difficulty >> 3. receiving filesystems are not enabled for large files over 2 Gb >> >> You can always try a single directory rsync with -v verbose on to see what >> it says it is doing, for example: >> rsync -v -a --progress --stats >>rsync://hgdownload.cse.ucsc.edu/mysql/xenTro2/ . >> >> Or just the files it complains about: >> rsync -v -a --progress --stats >>rsync://hgdownload.cse.ucsc.edu/mysql/xenTro2/track* . >> >> --Hiram >> >> Jean Chang wrote: >>> Hiram, >>> >>> I am mirroring this data as a role user, "apsg". I don't think there >>> are any permissions problems here since apsg is running the rsync. >>> >>> drwxr-sr-x 2 apsg apsg 8192 Apr 2 21:30 xenTro2/ >>> >>> and the problem is more than xenTro2 >>> >>> If I check the amount of data (using du -k) for the various mysql >>> directories, empty or suspiciously small directories include: >>> 28 danRer1 >>> 0 go >>> 140 hgcentarchive >>> 184 hgcentral >>> 0 hgw1.pid >>> 0 hgw2.pid >>> 0 hgw7.pid >>> 28 ib_arch_log_0000000000 >>> 4 innodb.status.15447 >>> 4 innodb.status.2809 >>> 4 innodb.status.2831 >>> 4 innodb.status.2837 >>> 4 innodb.status.2898 >>> 4 innodb.status.2937 >>> 4 innodb.status.5435 >>> 8 lost+found >>> 8 proteins040515 >>> 8 proteins050415 >>> 8 proteins051015 >>> 8 proteins060115 >>> 0 proteome >>> 8 rheMac1 >>> 8 rheMac2 >>> 8 rn3 >>> 8 sep30 >>> 8 sp040315 >>> 8 sp040515 >>> 8 sp050415 >>> 8 sp051015 >>> 8 sp060115 >>> 8 strPur1 >>> 8 test >>> 0 uniProt >>> 8 xenTro1 >>> >>> Note that xenTro2 isn't even on the above list: >>> >>> 186324 xenTro2 >>> >>> because it has one file in it, >>> >>> -rw-rw-r-- 1 apsg apsg 190591076 Mar 16 22:48 all_est.MYD >>> >>> To answer your other question: >>> rsync --version >>> rsync version 2.6.3pre1 protocol version 28 >>> Copyright (C) 1996-2004 by Andrew Tridgell and others >>> >>> Capabilities: 64-bit files, socketpairs, hard links, acls, symlinks, >>> batchfiles, >>> inplace, IPv6, 64-bit system inums, 64-bit internal >>> inums, SLP >>> >>> Any ideas? Help! >>> >>> Thanks, >>> >>> Jean >> _______________________________________________ >> Genome-mirror mailing list >> Genome-mirror at soe.ucsc.edu >> http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > > -- > (*)->[]->()->[]->(**)->[]->()->[]->(*)->[]->()->[]->()->[]->()->[]->()->[] > > (Humboldt University Berlin, Germany)->[]-> ... > (University of Maryland, USA)->[]-> ... > (King's College London, UK) > > https://josh.umds.ac.uk/~rschulz > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror __________________________________ Michael Pheasant, PhD Bioinformatics Research Institute for Molecular Bioscience The University of Queensland Brisbane QLD 4067 Australia P: +61-7-3346-2080 M: +61-405-679-541 http://www.imb.uq.edu.au From hiram at soe.ucsc.edu Wed Apr 11 13:25:10 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Wed, 11 Apr 2007 13:25:10 -0700 Subject: [Genome-mirror] rsync issues - empty or too-small directories In-Reply-To: References: <7D1158C4-9CA5-4894-A91E-BA3DFF2BE123@broad.mit.edu> <461A6918.8000001@soe.ucsc.edu> <461C98C3.7030807@kcl.ac.uk> Message-ID: <461D4426.70201@soe.ucsc.edu> We're checking into the situation at this end to see if anything might be haywire. One possibility that has come up is that your rsync might be happening at the time these files are being updated here. That would cause an outage at your end. We may find something here. --Hiram Michael Pheasant wrote: > Hi, > > I also have been experienceing rsync errors. > > Previously it happened with the same file on panTro1. > Jean contacted me, experiencing the same thing and said the workaround is to > remove the 'z' compress flag. > This works. > From ann at soe.ucsc.edu Wed Apr 11 15:14:13 2007 From: ann at soe.ucsc.edu (Ann Zweig) Date: Wed, 11 Apr 2007 15:14:13 -0700 Subject: [Genome-mirror] new fugu assembly released Message-ID: <461D5DB5.9010003@cse.ucsc.edu> Hi Mirror folks, As promised earlier this week (see email entitled "releasing fr2 assembly"), we have released fr2 to the public website. The data (both mysql and gbdb) should be available to you later today. Also note that *after* you have downloaded all of the fr2 data, you should feel free to drop the chain and net tables from other organisms to fr1 (with this release they have been replaced by chain and net tables to fr2). You can safely drop the following tables: chr%chainFr1% netFr1 from these databases: hg18 99 tables galGal3 115 tables danRer4 57 tables gasAcu1 47 tables tetNig1 55 tables Thanks, Ann Zweig. From ebr26 at student.canterbury.ac.nz Wed Apr 11 18:43:31 2007 From: ebr26 at student.canterbury.ac.nz (Emmanuel Buschiazzo) Date: Thu, 12 Apr 2007 13:43:31 +1200 Subject: [Genome-mirror] kent source installation Message-ID: <461D8EC3.2050007@student.canterbury.ac.nz> Hello, and thank you for your previous help! I am still in the process of installing the kent source on my computer. So far, I have been successful up to the 'make libs' command, and the 4 libraries are indeed placed into ~/src/lib/ $MACHTYPE (i386). Then, going to src/hg and typing 'make compile' bring troubles. Here's what I get on Terminal: [ebr26 at localhost hg]$ make compile cd lib && make make[1]: Entering directory `/home/ebr26/kent/src/hg/lib' ar rcus ../../lib//jkhgap.a acemblyClass.o affyAllExonProbe.o affyAtlas.o affy10KDetails.o affy120KDetails.o affyOffset.o affyPairs.o agp.o agpFrag.o agpGap.o alignSeqSizes.o altGraph.o altGraphX.o ancientRref.o axtInfo.o axtLib.o bactigPos.o bdgpExprLink.o bdgpGeneInfo.o bed.o bed5FloatScore.o bed5Pval.o bed6FloatScore.o bed12Source.o bedCart.o bgiGeneInfo.o [.................] wiggleUtils.o wikiLink.o zdobnovSynt.o oreganno.o oregannoUi.o gvUi.o gv.o make[1]: Leaving directory `/home/ebr26/kent/src/hg/lib' make[1]: Entering directory `/home/ebr26/kent/src/hg/cartReset' gcc cartReset.o ../../lib//jkhgap.a ../../lib//jkweb.a -lm /usr/lib/mysql/libmysqlclient.a -lz gcc: ../../lib//jkweb.a: No such file or directory make[1]: *** [compile] Error 1 make[1]: Leaving directory `/home/ebr26/kent/src/hg/cartReset' make[1]: Entering directory `/home/ebr26/kent/src/hg/das' gcc das.o ../../lib//jkhgap.a ../../lib//jkweb.a -lm /usr/lib/mysql/libmysqlclient.a -lz gcc: ../../lib//jkweb.a: No such file or directory make[1]: *** [compile] Error 1 etc......................................................................... or when I try program by program, e.g.: [ebr26 at localhost liftOver]$ make make: *** No rule to make target `../../lib//jkweb.a', needed by `liftOver'. Stop. [ebr26 at localhost liftOver]$ The jkweb.a library is definitely present where it should be, so I don't know what to do. Thank you again in advance for any help! Emmanuel. From hiram at soe.ucsc.edu Thu Apr 12 09:41:08 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Thu, 12 Apr 2007 09:41:08 -0700 Subject: [Genome-mirror] kent source installation In-Reply-To: <461D8EC3.2050007@student.canterbury.ac.nz> References: <461D8EC3.2050007@student.canterbury.ac.nz> Message-ID: <461E6124.9030905@soe.ucsc.edu> Good Morning Emmanuel: It appears you do not have a MACHTYPE defined properly in your shell environment. Remember to 'export' MACHTYPE if you are using a bash shell. Note the reference to your libraries is constructed: ../../lib//jkhgap.a This should instead read: ../../lib/i386/jkhgap.a Please note the build instructions in the source tree file: src/product/README.building.source --Hiram Emmanuel Buschiazzo wrote: > Hello, and thank you for your previous help! > > I am still in the process of installing the kent source on my computer. > So far, I have been successful up to the 'make libs' command, and the 4 > libraries are indeed placed into ~/src/lib/ > $MACHTYPE (i386). > > Then, going to src/hg and typing 'make compile' bring troubles. > > Here's what I get on Terminal: > > [ebr26 at localhost hg]$ make compile > cd lib && make > make[1]: Entering directory `/home/ebr26/kent/src/hg/lib' > ar rcus ../../lib//jkhgap.a acemblyClass.o affyAllExonProbe.o > affyAtlas.o affy10KDetails.o affy120KDetails.o affyOffset.o affyPairs.o > agp.o agpFrag.o agpGap.o alignSeqSizes.o altGraph.o altGraphX.o > ancientRref.o axtInfo.o axtLib.o bactigPos.o bdgpExprLink.o > bdgpGeneInfo.o bed.o bed5FloatScore.o bed5Pval.o bed6FloatScore.o > bed12Source.o bedCart.o bgiGeneInfo.o > [.................] > wiggleUtils.o wikiLink.o zdobnovSynt.o oreganno.o oregannoUi.o gvUi.o gv.o > make[1]: Leaving directory `/home/ebr26/kent/src/hg/lib' > make[1]: Entering directory `/home/ebr26/kent/src/hg/cartReset' > gcc cartReset.o ../../lib//jkhgap.a ../../lib//jkweb.a -lm > /usr/lib/mysql/libmysqlclient.a -lz > gcc: ../../lib//jkweb.a: No such file or directory > make[1]: *** [compile] Error 1 > make[1]: Leaving directory `/home/ebr26/kent/src/hg/cartReset' > make[1]: Entering directory `/home/ebr26/kent/src/hg/das' > gcc das.o ../../lib//jkhgap.a ../../lib//jkweb.a -lm > /usr/lib/mysql/libmysqlclient.a -lz > gcc: ../../lib//jkweb.a: No such file or directory > make[1]: *** [compile] Error 1 > etc......................................................................... > > or when I try program by program, e.g.: > [ebr26 at localhost liftOver]$ make > make: *** No rule to make target `../../lib//jkweb.a', needed by > `liftOver'. Stop. > [ebr26 at localhost liftOver]$ > > The jkweb.a library is definitely present where it should be, so I don't > know what to do. > > Thank you again in advance for any help! > > Emmanuel. From davide.cittaro at ifom-ieo-campus.it Wed Apr 18 06:21:23 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Wed, 18 Apr 2007 15:21:23 +0200 Subject: [Genome-mirror] configure output error Message-ID: <2A8DFA0A-B58B-4BDC-9B24-A4C66B110887@ifom-ieo-campus.it> Dear UCSC people, we had a Internal server error as we try to configure the output of genome browser. It is hard to check which parameter raises the error but, if we check all the boxes Display chromosome ideogram above main graphic Show light blue vertical guidelines Display labels to the left of items in tracks Display description above each track Show track controls under main graphic Next/previous item navigation Next/previous exon navigation Enable track re-ordering and click on "show all" in Configure Tracks, we go in Internal Server Error. Apache log says Premature end of script headers: hgTracks, referer: http:// genome.ifom-ieo-campus.it/cgi-bin/hgTracks? hgsid=7465&hgTracksConfigPage=configure+tracks+and+display and nothing more... any hint? d /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro at ifom-ieo-campus.it */ From hiram at soe.ucsc.edu Wed Apr 18 10:08:56 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Wed, 18 Apr 2007 10:08:56 -0700 Subject: [Genome-mirror] configure output error In-Reply-To: <2A8DFA0A-B58B-4BDC-9B24-A4C66B110887@ifom-ieo-campus.it> References: <2A8DFA0A-B58B-4BDC-9B24-A4C66B110887@ifom-ieo-campus.it> Message-ID: <462650A8.20407@soe.ucsc.edu> Good Morning Davide: Let's see if we can narrow this down a bit better. Which version of the browser does it say on the title line of your WEB browser when you are on the genome browser page that works ? Do you get into this situation with a particular combination of tracks, or merely turning on these options while in the default location on hg18 ? Can you fetch your session id cart contents from your hgcentral db: $ hgsql -e "select * from sessionDb where id=7465;" hgcentral I'll check and see if we've made any hgcentral structure changes recently. --Hiram Davide Cittaro wrote: > Dear UCSC people, we had a Internal server error as we try to > configure the output of genome browser. > It is hard to check which parameter raises the error but, if we check > all the boxes > > > Display chromosome ideogram above main graphic > Show light blue vertical guidelines > Display labels to the left of items in tracks > Display description above each track > Show track controls under main graphic > Next/previous item navigation > Next/previous exon navigation > Enable track re-ordering > > and click on "show all" in Configure Tracks, we go in Internal Server > Error. Apache log says > > Premature end of script headers: hgTracks, referer: http:// > genome.ifom-ieo-campus.it/cgi-bin/hgTracks? > hgsid=7465&hgTracksConfigPage=configure+tracks+and+display > > and nothing more... any hint? > > d > /* > Davide Cittaro > HPC and Bioinformatics Systems @ Informatics Core > > IFOM - Istituto FIRC di Oncologia Molecolare > via adamello, 16 > 20139 Milano > Italy > > tel.: +39(02)574303007 > e-mail: davide.cittaro at ifom-ieo-campus.it > */ From hiram at soe.ucsc.edu Thu Apr 19 12:45:38 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Thu, 19 Apr 2007 12:45:38 -0700 Subject: [Genome-mirror] Mirror setup question In-Reply-To: References: Message-ID: <4627C6E2.9020507@soe.ucsc.edu> Good Morning Jean: All of these tables you mention should be in hgcentral. Even the userDb table should be in hgcentral: Tables_in_hgcentral blatServers clade dbDb defaultDb gdbPdb genomeClade liftOverChain namedSessionDb sessionDb userDb The hgcentarchive is for archived databases that are no longer active. I'd have to check how the browser software accesses the archive assemblies, but in your case you should have no access to those. Your hg.conf file specifies these databases and the hosts on which to find them. --Hiram >> Subject: >> mirror setup >> From: >> Jean Chang >> Date: >> Thu, 19 Apr 2007 10:25:38 -0400 >> To: >> genome-mirror at soe.ucsc.edu >> >> Hi all, >> >> I'm confused. I've put hgcentral in place and I'm getting an error that >> myhguser can't insert into userDb BUT userDb isn't in hgcentral, it's in >> hgcentarchive - what am I missing? >> >> Thanks, >> >> Jean From maximilianh at gmail.com Fri Apr 20 03:42:42 2007 From: maximilianh at gmail.com (Maximilian Haeussler) Date: Fri, 20 Apr 2007 11:42:42 +0100 Subject: [Genome-mirror] chained tblastn vs web-blat (hyperlinks) vs local blat vs web-blat (psl) Message-ID: <76f031ae0704200342x2f0ec16dmfd3a0ec98a41ce0b@mail.gmail.com> Hi gurus, I've received a set of protein sequences from a publication and I'm trying to put them on the genome (same species, supposed to match 100%) on my mirror site (http://genome.ciona.cnrs-gif.fr). Using the web-based BLAT it would work, but the file is too big. Local blat, as usual, gave me different results: shorter exons. Yes, I know your documentation on http://genome.ucsc.edu/FAQ/FAQblat.html#blat5 about this topic and I've tried the exact recommended command line, but it didn't change the result). The commands that I've tried were local blat and also translated gfServer/gfClient with -minScore=0 -minIdentity=0. With both, the results are always different from the standard blat-webserver with output set to "hyperlinks" (shorter exons) Strange enough, even web-blat comes in different tastes: output as psl / pasting as a new track versus simply clicking "submit" on the blat-webform gives two different results. Confused, I've looked into makeHg17.doc and tried to repeat what you're doing when you're aligning fly proteins to vertebrates. I thought that the following might do what I want: ## blastall -p tblastn -i julie.fa -d ../genome/blast/ci.fa -e 1 -o julie.blast -F F -m 0 -a 2 -M BLOSUM80 -W 2 blastToPsl julie.blast julie.psl simpleChain -prot -outPsl -maxGap=40000 julie.psl julie.chain.psl sort -rn julie.chain.psl | pslUniq stdin julie.chain.best.psl ## But this gives me a third result now, again with smaller exons and some exons completely missing. a) Blast seems to be less sensitive than blat for sequences that are supposed to be 100% matches (??) b) the web-based BLAT is doing something different when choosing "hyperlink" versus "psl output" (??) That doesn't make much sense, so I must have made a mistake somewhere... can you point me into the right direction? Here is an example with all the results on various tracks: http://genome.ucsc.edu/cgi-bin/hgTracks?db=ci1&position=Scaffold_71:17154-75591&hgt.customText=http://max.butterbrot.org/biofiles/temp.bed Thanks a lot in advance! cheers, Max here is a sequence to paste into web-blat to compare with the psl-tracks: >ci0100149827_ci149827_Hemicentin MNTVIAALLAVACLAASCIGQSRRREPLDIRGKIRGNITGDTYTPDLTGITPGSSSLAFVFDVTGSMWDDLKQV RGGAAQILETTRNRQEKPLFNYVLIPFADPVVGPISVTTDPATFITALKGLYVEGGGDCPEMTISAIREALVIS LPGSFIYVFTDARSKDYLQVPEILQIIQRKQSQVVFVLTGDCNERDHDGYRAYERIAATSSGQVFQISREQVSE VVEFIRVSVQAHKVMLLSTDHDTPSPSQHEWVLPLDPNLEELTLSISGPDPLLSIFDPDGIEITEQTGLIPQLD LEQVFIGTITEPKPGLWRIVAGSSGEHTLRVTGLSTLDFQARFSRKPTLNWTDTEYQPVANIPSYILLNATGLN QPGRFKRMKLLDIAGRPLRTLRLGTPPDPNQPYLYTVEPFMTPDVPFYIAVEGITDDGYNFIRTTKNAILNIIP TAPVVQVKANQPGYFDYPVEIPCNITSLLPYHVQWSREVNGIYVNLHSPRLDSATDYYVIDEVSEASEGMYRCY ANNSEGNDYGLTFLDVEERPPVVTTDGNVTVTPGNSVVLTCHIQSNVQYSLRWSRVDGRNLDRRRIRKLRNGSL LIQRAMPGDEAGYQCTAVNIGGESNDVIGLIVQRPPTVIIRPSQKSTFSEGGRIRIRCKANGLPRPKLKWLKGS NYLFDIGKIRISRSGNLLEIINANQEDGGLYTCQATNRAGTDQASVEWVFTEHPVVQIPEPEMLVLEGTTATLR CVTSGVPPPVVLWSKDGNNIVAGRSFEINVKGSLVILQVARSDAGVFVCTARNPGGSDHGNITLTVGSLPLFVE PLPHDIGVDFGKLLNIHIPCNAIGNPEPTVTWHKEDPDSDEYNVMGEADLLLSNILSISSVELDDGATYICTAT NSFGRSTTTAHVTITGTAHPEVDVGDLRPHVIVGETIIIPCRIVAGNPRPIQRWKRRRVAFNPTGRVYINEASS VVITQAVTSDAGAYFCHATNVIGHEQGRVDLDVWVPPTITSTDTLFTTIEHIPVSMQCVATGNPEPVVVWRKDG DPRPLNKLSGYEVSGDGTLTILNPDHENEGSYSCQSTNAAGVDSFDVHLEIYLRPQVDSSVDDYIVTIGHSVTL QCDVSDSDPPPTFTWRKGEESITGDEDGISITADGNLTLSSATLDDAGFYTCVATNIAGSAEKNVRLTVQVPPS IRRGPTLVKVLKSETASLQCQATGLPTPTITWSRGNVLIDENFNEPGTSELRIRNTQLSDDDLYVCNAVNPAGR DSSSVTLDVQSRPSLDLLDICVPSVTNNNCTLTPVELGRLTLPCNASGDPRPTIRWYHDGEELNGRDPNVDIDA EGTLTLYVVTADHNGQYVCEAENERGIIRKVTDVLVLEHPDIKGGGNVERITVLEGEDVDLTCEADAVPPPTIT WYGGEDQTQTVLEEREHIEFIENGQTFRITSARVSDTGLYKCTAVNKVGETHKFIRLDVYVPPTIRDNEVVSVQ TVVVDESIDLHCYVDGIPFPAIRWYTGVNEVIPDDERVQFIDRDQTLRINSALVSDTGSYKCIATNVAGESSKD FDLNVHVPPFIKGNDENHVVTLDTTLVLRCLTNGIPKPAIVWTLDDDPITSRAGMRIVQDNQVIEFSNIQIDDV From reiner.schulz at kcl.ac.uk Fri Apr 20 04:05:57 2007 From: reiner.schulz at kcl.ac.uk (Reiner Schulz) Date: Fri, 20 Apr 2007 12:05:57 +0100 Subject: [Genome-mirror] /gbdb file size error Message-ID: <46289E95.1000900@kcl.ac.uk> when accessing the track display for the mm8 genome at the default position chr12:57,612,909-57,630,019 on our local mirror, i receive the following error message: 'External file /gbdb/mm8/multiz17way/anno/maf/chr12.maf cannot be opened or has wrong size. Old size 1144349117, new size 1144458830, error Success' both /gbdb/mm8/ and /var/lib/mysql/mm8/ are up-to-date (have just been rsync'ed successfully). the error does NOT occur with different coordinates like, for example, chr12:56,612,909-56,630,019, or different genomes like rn4 or hg18. however, mm8 is the only genome for which we have additional custom tables in the database, but how could that cause the above problem? cheers, Reiner -- (*)->[]->()->[]->(**)->[]->()->[]->(*)->[]->()->[]->()->[]->()->[]->()->[] (Humboldt University Berlin, Germany)->[]-> ... (University of Maryland, USA)->[]-> ... (King's College London, UK) https://josh.umds.ac.uk/~rschulz From hiram at soe.ucsc.edu Fri Apr 20 08:48:41 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Fri, 20 Apr 2007 08:48:41 -0700 Subject: [Genome-mirror] /gbdb file size error In-Reply-To: <46289E95.1000900@kcl.ac.uk> References: <46289E95.1000900@kcl.ac.uk> Message-ID: <4628E0D9.3090507@soe.ucsc.edu> Good Morning Reiner: Please verify the table extFile in your mm8 database is up to date. These sizes are checked between the entries in extFile and the actual file. For the example you mention, the file is: -rw-rw-r-- 1 1144458830 Mar 27 15:30 /gbdb/mm8/multiz17way/anno/maf/chr12.maf And the extFile table entry is: id name path size 17888510 chr12.maf /gbdb/mm8/multiz17way/anno/maf/chr12.maf 1144458830 --Hiram Reiner Schulz wrote: > when accessing the track display for the mm8 genome at the default > position chr12:57,612,909-57,630,019 on our local mirror, i receive the > following error message: > 'External file /gbdb/mm8/multiz17way/anno/maf/chr12.maf cannot be opened > or has wrong size. Old size 1144349117, new size 1144458830, error Success' > both /gbdb/mm8/ and /var/lib/mysql/mm8/ are up-to-date (have just been > rsync'ed successfully). the error does NOT occur with different > coordinates like, for example, chr12:56,612,909-56,630,019, or different > genomes like rn4 or hg18. however, mm8 is the only genome for which we > have additional custom tables in the database, but how could that cause > the above problem? > > cheers, > > Reiner From reiner.schulz at kcl.ac.uk Sat Apr 21 05:29:23 2007 From: reiner.schulz at kcl.ac.uk (Reiner Schulz) Date: Sat, 21 Apr 2007 13:29:23 +0100 Subject: [Genome-mirror] /gbdb file size error In-Reply-To: <4628E0D9.3090507@soe.ucsc.edu> References: <46289E95.1000900@kcl.ac.uk> <4628E0D9.3090507@soe.ucsc.edu> Message-ID: <462A03A3.8080005@kcl.ac.uk> hi Hiram, extFile in mm8 checks out: mysql> select * from extFile where name like '%chr12%'; +----------+-----------+------------------------------------------+------------+| id | name | path | size |+----------+-----------+------------------------------------------+------------+| 17888510 | chr12.maf | /gbdb/mm8/multiz17way/anno/maf/chr12.maf | 1144458830 | +----------+-----------+------------------------------------------+------------+1 row in set (0.00 sec) rschulz at atlas:~/tmp$ ls -l /gbdb/mm8/multiz17way/anno/maf/chr12.maf -rw-rw-r-- 1 rschulz oakey 1144458830 2007-03-27 23:30 /gbdb/mm8/multiz17way/anno/maf/chr12.maf i'll do a little poking in the source to see where this ominous 'Old size 1144349117' comes from. if in the meantime, i should check anything else, please let me know. much appreciated, Reiner Hiram Clawson wrote: > Good Morning Reiner: > > Please verify the table extFile in your mm8 database is up to date. > These sizes are checked between the entries in extFile and the > actual file. For the example you mention, the file is: > -rw-rw-r-- 1 1144458830 Mar 27 15:30 > /gbdb/mm8/multiz17way/anno/maf/chr12.maf > And the extFile table entry is: > id name path size > 17888510 chr12.maf /gbdb/mm8/multiz17way/anno/maf/chr12.maf 1144458830 > > --Hiram > > Reiner Schulz wrote: >> when accessing the track display for the mm8 genome at the default >> position chr12:57,612,909-57,630,019 on our local mirror, i receive the >> following error message: >> 'External file /gbdb/mm8/multiz17way/anno/maf/chr12.maf cannot be opened >> or has wrong size. Old size 1144349117, new size 1144458830, error >> Success' >> both /gbdb/mm8/ and /var/lib/mysql/mm8/ are up-to-date (have just been >> rsync'ed successfully). the error does NOT occur with different >> coordinates like, for example, chr12:56,612,909-56,630,019, or different >> genomes like rn4 or hg18. however, mm8 is the only genome for which we >> have additional custom tables in the database, but how could that cause >> the above problem? >> >> cheers, >> >> Reiner -- (*)->[]->()->[]->(**)->[]->()->[]->(*)->[]->()->[]->()->[]->()->[]->()->[] (Humboldt University Berlin, Germany)->[]-> ... (University of Maryland, USA)->[]-> ... (King's College London, UK) https://josh.umds.ac.uk/~rschulz From reiner.schulz at kcl.ac.uk Sat Apr 21 05:55:11 2007 From: reiner.schulz at kcl.ac.uk (Reiner Schulz) Date: Sat, 21 Apr 2007 13:55:11 +0100 Subject: [Genome-mirror] /gbdb file size error In-Reply-To: <462A03A3.8080005@kcl.ac.uk> References: <46289E95.1000900@kcl.ac.uk> <4628E0D9.3090507@soe.ucsc.edu> <462A03A3.8080005@kcl.ac.uk> Message-ID: <462A09AF.5040200@kcl.ac.uk> no need to look any further: a restart of mysql got rid of the problem. i suspect it must have had to do w/ connection and query caching since the size value returned by the query in hExtFileNameC (hdb.c) was indeed different (and wrong) from the value returned when issued the query in the mysql command line client. r Reiner Schulz wrote: > hi Hiram, > > extFile in mm8 checks out: > > mysql> select * from extFile where name like '%chr12%'; > +----------+-----------+------------------------------------------+------------+| > id | name | path | size > > |+----------+-----------+------------------------------------------+------------+| > 17888510 | chr12.maf | /gbdb/mm8/multiz17way/anno/maf/chr12.maf | > 1144458830 | > +----------+-----------+------------------------------------------+------------+1 > row in set (0.00 sec) > > rschulz at atlas:~/tmp$ ls -l /gbdb/mm8/multiz17way/anno/maf/chr12.maf > -rw-rw-r-- 1 rschulz oakey 1144458830 2007-03-27 23:30 > /gbdb/mm8/multiz17way/anno/maf/chr12.maf > > i'll do a little poking in the source to see where this ominous 'Old > size 1144349117' comes from. if in the meantime, i should check anything > else, please let me know. > > much appreciated, > > Reiner > > Hiram Clawson wrote: >> Good Morning Reiner: >> >> Please verify the table extFile in your mm8 database is up to date. >> These sizes are checked between the entries in extFile and the >> actual file. For the example you mention, the file is: >> -rw-rw-r-- 1 1144458830 Mar 27 15:30 >> /gbdb/mm8/multiz17way/anno/maf/chr12.maf >> And the extFile table entry is: >> id name path size >> 17888510 chr12.maf /gbdb/mm8/multiz17way/anno/maf/chr12.maf 1144458830 >> >> --Hiram >> >> Reiner Schulz wrote: >>> when accessing the track display for the mm8 genome at the default >>> position chr12:57,612,909-57,630,019 on our local mirror, i receive the >>> following error message: >>> 'External file /gbdb/mm8/multiz17way/anno/maf/chr12.maf cannot be opened >>> or has wrong size. Old size 1144349117, new size 1144458830, error >>> Success' >>> both /gbdb/mm8/ and /var/lib/mysql/mm8/ are up-to-date (have just been >>> rsync'ed successfully). the error does NOT occur with different >>> coordinates like, for example, chr12:56,612,909-56,630,019, or different >>> genomes like rn4 or hg18. however, mm8 is the only genome for which we >>> have additional custom tables in the database, but how could that cause >>> the above problem? >>> >>> cheers, >>> >>> Reiner > -- (*)->[]->()->[]->(**)->[]->()->[]->(*)->[]->()->[]->()->[]->()->[]->()->[] (Humboldt University Berlin, Germany)->[]-> ... (University of Maryland, USA)->[]-> ... (King's College London, UK) https://josh.umds.ac.uk/~rschulz From aamp at ucsc.edu Sat Apr 21 08:01:49 2007 From: aamp at ucsc.edu (Andy Pohl) Date: Sat, 21 Apr 2007 08:01:49 -0700 Subject: [Genome-mirror] v157 Genome Browser available Message-ID: <9fa943760704210801k658f508cv3e9705c6bf651435@mail.gmail.com> Hello, v156 source is at: http://hgdownload.cse.ucsc.edu/admin/jksrc.zip or labeled with source number: http://hgdownload.cse.ucsc.edu/admin/jksrc.v157.zip cartDump cartReset das hgc hgBlat hgConvert hgCustom hgEncodeDataVersions hgGateway hgGene hgGenome hgLiftOver hgNear hgPcr hgSession hgTables hgText hgTracks hgTrackUi hgVisiGene mkEncodeFrameset pbGateway pbGlobal pbTracks phyloGif /usr/local/apache/cgi-bin/all.joiner /usr/local/apache/cgi-bin/hgNearData/* /usr/local/apache/cgi-bin/hgGeneData/* /usr/local/apache/cgi-bin/hgcData/* /usr/local/apache/cgi-bin/hgCgiData/* /usr/local/apache/cgi-bin/loader/* Contents of v157 may be seen at: http://genome-test.cse.ucsc.edu/builds/versions.html Please check both v157-preview and v157-final to see all the code and data changes in this release. Have a nice weekend, Andy Pohl UCSC CBSE From m.oshea at prion.ucl.ac.uk Sun Apr 22 13:16:56 2007 From: m.oshea at prion.ucl.ac.uk (Marie OShea (PRION)) Date: Sun, 22 Apr 2007 21:16:56 +0100 Subject: [Genome-mirror] Build release dates Message-ID: <43D60A06C3FC3744BCA0D8D06551D321903AF8@prion-owa1.prion.ucl.ac.uk> Dear Sir/Madam, Would it be possible to send me a list of UCSC Genome Browser Build release dates please. I need to find out release numbers for the mouse genome from the start of 2004 please. I see that there is a release log available on the website but it doesn't go as far as the start of 2004. I'd really appreciate if you could make the information available to me. Kind regards, Marie O'Shea From archanat at soe.ucsc.edu Mon Apr 23 08:48:46 2007 From: archanat at soe.ucsc.edu (Archana Thakkapallayil) Date: Mon, 23 Apr 2007 08:48:46 -0700 Subject: [Genome-mirror] Build release dates In-Reply-To: <43D60A06C3FC3744BCA0D8D06551D321903AF8@prion-owa1.prion.ucl.ac.uk> References: <43D60A06C3FC3744BCA0D8D06551D321903AF8@prion-owa1.prion.ucl.ac.uk> Message-ID: <462CD55E.6010509@soe.ucsc.edu> Hello Marie, We have a list in the FAQ (Assembly Releases and Versions section) of each organism, assembly, date, and which build it refers to. This is located at: http://genome.ucsc.edu/FAQ/FAQreleases#release1 I hope this helps you find what you are looking for. Let us know if you have further questions. Regards, Archana UCSC Genome Bioinformatics Group Marie OShea (PRION) wrote: > Dear Sir/Madam, > > Would it be possible to send me a list of UCSC Genome Browser Build release dates please. I need to find out release numbers for the mouse genome from the start of 2004 please. I see that there is a release log available on the website but it doesn't go as far as the start of 2004. > > I'd really appreciate if you could make the information available to me. > > Kind regards, > > Marie O'Shea > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From reiner.schulz at kcl.ac.uk Mon Apr 23 06:05:56 2007 From: reiner.schulz at kcl.ac.uk (Reiner Schulz) Date: Mon, 23 Apr 2007 14:05:56 +0100 Subject: [Genome-mirror] Comparative Genomics: Chain tracks Message-ID: <462CAF34.3080403@kcl.ac.uk> clicking on an entry in one of the chain tracks results in a server error on my mirror: 'Premature end of script headers: hgc' the mirror contains mm8, hg18 and rn4, and no other genomes. the complete /gbdb and /var/lib/mysql/[mm8|hg18|rn4] are regularly rsync'ed. just wondering if this is a common problem w/ an already known solution. i suspect that i'm missing some data that are necessary and not contained in /gbdb or /var/lib/mysql/[mm8|hg18|rn4]. much appreciated, Reiner -- (*)->[]->()->[]->(**)->[]->()->[]->(*)->[]->()->[]->()->[]->()->[]->()->[] (Humboldt University Berlin, Germany)->[]-> ... (University of Maryland, USA)->[]-> ... (King's College London, UK) https://josh.umds.ac.uk/~rschulz From jlchang at broad.mit.edu Mon Apr 23 12:25:06 2007 From: jlchang at broad.mit.edu (Jean Chang) Date: Mon, 23 Apr 2007 15:25:06 -0400 Subject: [Genome-mirror] rsync issues - empty or too-small directories In-Reply-To: References: Message-ID: <7E207FD6-71C6-4B0E-8540-7C33BC53485B@broad.mit.edu> Hi all, Just a quick update: upgrading to rsync 2.6.9 and turning off the -z option seemed to resolve my rsync issues. Regards, Jean From galt at soe.ucsc.edu Mon Apr 23 14:14:56 2007 From: galt at soe.ucsc.edu (Galt Barber) Date: Mon, 23 Apr 2007 14:14:56 -0700 (PDT) Subject: [Genome-mirror] chained tblastn vs web-blat (hyperlinks) vs local blat vs web-blat (psl) In-Reply-To: <76f031ae0704200342x2f0ec16dmfd3a0ec98a41ce0b@mail.gmail.com> References: <76f031ae0704200342x2f0ec16dmfd3a0ec98a41ce0b@mail.gmail.com> Message-ID: Hi, Maximillian! What you have discovered is a minor problem in the custom-track loader. If one strips off the psl header which would otherwise carry the word "protein", then the code doesn't know it is protein. Since the block sizes are expressed relative to your original protein query, the blocksizes are all off by 3x which produces the "shortened" exons you saw. (Internally, most custom tracks types are converted to a bed file). I have added a detection method based on a simple pattern that hgBlat uses for protein psls, and preliminary tests look good. Assuming everything goes well with testing the patch, you should see new patched cgis for hgTracks and hgCustom and maybe hgTables soonish which should fix it. Thanks for reporting this, and sorry for any inconvenience this problem caused. Meanwhile, you may be able to work-around it by re-generating your psls with the psl-header intact. You could probably just make a fake header with this line: psLayout version 4 protein DNAX Take care! -Galt On Fri, 20 Apr 2007, Maximilian Haeussler wrote: > Hi gurus, > > I've received a set of protein sequences from a publication and I'm > trying to put them on the genome (same species, supposed to match > 100%) on my mirror site (http://genome.ciona.cnrs-gif.fr). > > Using the web-based BLAT it would work, but the file is too big. > > Local blat, as usual, gave me different results: shorter exons. Yes, > I know your documentation on > http://genome.ucsc.edu/FAQ/FAQblat.html#blat5 about this topic and > I've tried the exact recommended command line, but it didn't change > the result). The commands that I've tried were local blat and also > translated gfServer/gfClient with -minScore=0 -minIdentity=0. With > both, the results are always different from the standard > blat-webserver with output set to "hyperlinks" (shorter exons) > > Strange enough, even web-blat comes in different tastes: output as psl > / pasting as a new track versus simply clicking "submit" on the > blat-webform gives two different results. > > Confused, I've looked into makeHg17.doc and tried to repeat what > you're doing when you're aligning fly proteins to vertebrates. I > thought that the following might do what I want: > > ## > blastall -p tblastn -i julie.fa -d ../genome/blast/ci.fa -e 1 -o > julie.blast -F F -m 0 -a 2 -M BLOSUM80 -W 2 > blastToPsl julie.blast julie.psl > simpleChain -prot -outPsl -maxGap=40000 julie.psl julie.chain.psl > sort -rn julie.chain.psl | pslUniq stdin julie.chain.best.psl > ## > > But this gives me a third result now, again with smaller exons and > some exons completely missing. > > a) Blast seems to be less sensitive than blat for sequences that are > supposed to be 100% matches (??) > b) the web-based BLAT is doing something different when choosing > "hyperlink" versus "psl output" (??) > > That doesn't make much sense, so I must have made a mistake > somewhere... can you point me into the right direction? > > Here is an example with all the results on various tracks: > http://genome.ucsc.edu/cgi-bin/hgTracks?db=ci1&position=Scaffold_71:17154-75591&hgt.customText=http://max.butterbrot.org/biofiles/temp.bed > > Thanks a lot in advance! > cheers, > Max > > here is a sequence to paste into web-blat to compare with the psl-tracks: > >ci0100149827_ci149827_Hemicentin > MNTVIAALLAVACLAASCIGQSRRREPLDIRGKIRGNITGDTYTPDLTGITPGSSSLAFVFDVTGSMWDDLKQV > RGGAAQILETTRNRQEKPLFNYVLIPFADPVVGPISVTTDPATFITALKGLYVEGGGDCPEMTISAIREALVIS > LPGSFIYVFTDARSKDYLQVPEILQIIQRKQSQVVFVLTGDCNERDHDGYRAYERIAATSSGQVFQISREQVSE > VVEFIRVSVQAHKVMLLSTDHDTPSPSQHEWVLPLDPNLEELTLSISGPDPLLSIFDPDGIEITEQTGLIPQLD > LEQVFIGTITEPKPGLWRIVAGSSGEHTLRVTGLSTLDFQARFSRKPTLNWTDTEYQPVANIPSYILLNATGLN > QPGRFKRMKLLDIAGRPLRTLRLGTPPDPNQPYLYTVEPFMTPDVPFYIAVEGITDDGYNFIRTTKNAILNIIP > TAPVVQVKANQPGYFDYPVEIPCNITSLLPYHVQWSREVNGIYVNLHSPRLDSATDYYVIDEVSEASEGMYRCY > ANNSEGNDYGLTFLDVEERPPVVTTDGNVTVTPGNSVVLTCHIQSNVQYSLRWSRVDGRNLDRRRIRKLRNGSL > LIQRAMPGDEAGYQCTAVNIGGESNDVIGLIVQRPPTVIIRPSQKSTFSEGGRIRIRCKANGLPRPKLKWLKGS > NYLFDIGKIRISRSGNLLEIINANQEDGGLYTCQATNRAGTDQASVEWVFTEHPVVQIPEPEMLVLEGTTATLR > CVTSGVPPPVVLWSKDGNNIVAGRSFEINVKGSLVILQVARSDAGVFVCTARNPGGSDHGNITLTVGSLPLFVE > PLPHDIGVDFGKLLNIHIPCNAIGNPEPTVTWHKEDPDSDEYNVMGEADLLLSNILSISSVELDDGATYICTAT > NSFGRSTTTAHVTITGTAHPEVDVGDLRPHVIVGETIIIPCRIVAGNPRPIQRWKRRRVAFNPTGRVYINEASS > VVITQAVTSDAGAYFCHATNVIGHEQGRVDLDVWVPPTITSTDTLFTTIEHIPVSMQCVATGNPEPVVVWRKDG > DPRPLNKLSGYEVSGDGTLTILNPDHENEGSYSCQSTNAAGVDSFDVHLEIYLRPQVDSSVDDYIVTIGHSVTL > QCDVSDSDPPPTFTWRKGEESITGDEDGISITADGNLTLSSATLDDAGFYTCVATNIAGSAEKNVRLTVQVPPS > IRRGPTLVKVLKSETASLQCQATGLPTPTITWSRGNVLIDENFNEPGTSELRIRNTQLSDDDLYVCNAVNPAGR > DSSSVTLDVQSRPSLDLLDICVPSVTNNNCTLTPVELGRLTLPCNASGDPRPTIRWYHDGEELNGRDPNVDIDA > EGTLTLYVVTADHNGQYVCEAENERGIIRKVTDVLVLEHPDIKGGGNVERITVLEGEDVDLTCEADAVPPPTIT > WYGGEDQTQTVLEEREHIEFIENGQTFRITSARVSDTGLYKCTAVNKVGETHKFIRLDVYVPPTIRDNEVVSVQ > TVVVDESIDLHCYVDGIPFPAIRWYTGVNEVIPDDERVQFIDRDQTLRINSALVSDTGSYKCIATNVAGESSKD > FDLNVHVPPFIKGNDENHVVTLDTTLVLRCLTNGIPKPAIVWTLDDDPITSRAGMRIVQDNQVIEFSNIQIDDV > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From avpoliakov at lbl.gov Mon Apr 23 14:51:43 2007 From: avpoliakov at lbl.gov (Alexander Poliakov) Date: Mon, 23 Apr 2007 14:51:43 -0700 Subject: [Genome-mirror] faToNib Message-ID: <462D2A6F.5080307@lbl.gov> Hello, I tried to convert a sequence from FASTA to NIB format using "faToNib", but all I got from the program was the following error. needHugeMen: Out of huge memory - request size 1073741824 bytes The sequence is ~740MB long and the machine has 4GB RAM. Could you please help me make it work? Thanks, Alex From hiram at soe.ucsc.edu Mon Apr 23 15:18:17 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Mon, 23 Apr 2007 15:18:17 -0700 Subject: [Genome-mirror] faToNib In-Reply-To: <462D2A6F.5080307@lbl.gov> References: <462D2A6F.5080307@lbl.gov> Message-ID: <462D30A9.9070005@soe.ucsc.edu> Good Afternoon Alex: What type of CPU and operating system are you working on ? Does your login shell have process size limits ? See what the command: ulimit can tell you about your process limits. I just tried this on chr1 of Opossum, 729 Mb and it converts with a peak memory usage of just over 1 Gb. Have you considered using the 2bit file format ? All the kent utilities can also work with the 2bit file format and it is a bit more efficient that the nib format. --Hiram Alexander Poliakov wrote: > Hello, > > I tried to convert a sequence from FASTA to NIB format using "faToNib", > but all I got from the program was the following error. > > needHugeMen: Out of huge memory - request size 1073741824 bytes > > The sequence is ~740MB long and the machine has 4GB RAM. Could you > please help me make it work? > > Thanks, > Alex From avpoliakov at lbl.gov Mon Apr 23 15:37:12 2007 From: avpoliakov at lbl.gov (Alexander Poliakov) Date: Mon, 23 Apr 2007 15:37:12 -0700 Subject: [Genome-mirror] faToNib In-Reply-To: <462D30A9.9070005@soe.ucsc.edu> References: <462D2A6F.5080307@lbl.gov> <462D30A9.9070005@soe.ucsc.edu> Message-ID: <462D3518.2050804@lbl.gov> Dear Hiram, > I just tried this on chr1 of Opossum, 729 Mb and it converts > with a peak memory usage of just over 1 Gb. That is exactly what I was trying to do - chr1 of opossum. I am using Fedora Core 6 on a machine with an Intel Xeon CPU. Here is what I get from tcsh: cputime unlimited filesize unlimited datasize unlimited stacksize unlimited coredumpsize 0 kbytes memoryuse unlimited vmemoryuse unlimited descriptors 1024 memorylocked unlimited maxproc 65536 And here is what I get from "free": total used free shared buffers cached Mem: 3631028 2214164 1416864 0 165472 1440940 -/+ buffers/cache: 607752 3023276 Swap: 4192956 0 4192956 > Have you considered using the 2bit file format ? There are a few programs developed in our lab that work with NIB files, and I do not want to rewrite them at the moment. Thanks, Alex Hiram Clawson wrote: > Good Afternoon Alex: > > What type of CPU and operating system are you working on ? > Does your login shell have process size limits ? See what > the command: ulimit > can tell you about your process limits. > > I just tried this on chr1 of Opossum, 729 Mb and it converts > with a peak memory usage of just over 1 Gb. > > Have you considered using the 2bit file format ? All the kent > utilities can also work with the 2bit file format and it is a bit > more efficient that the nib format. > > --Hiram > > Alexander Poliakov wrote: > >> Hello, >> >> I tried to convert a sequence from FASTA to NIB format using >> "faToNib", but all I got from the program was the following error. >> >> needHugeMen: Out of huge memory - request size 1073741824 bytes >> >> The sequence is ~740MB long and the machine has 4GB RAM. Could you >> please help me make it work? >> >> Thanks, >> Alex From hiram at soe.ucsc.edu Mon Apr 23 16:04:50 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Mon, 23 Apr 2007 16:04:50 -0700 Subject: [Genome-mirror] faToNib In-Reply-To: <462D3518.2050804@lbl.gov> References: <462D2A6F.5080307@lbl.gov> <462D30A9.9070005@soe.ucsc.edu> <462D3518.2050804@lbl.gov> Message-ID: <462D3B92.6040603@soe.ucsc.edu> Alex: Check your kent/src/lib/memalloc.c and see if its definition of maxAlloc matches this: /* 128*8*1024*1024 == 1073741824 == 2^30 on 32 bit machines,size_t == 4 bytes*/ /* on 64 bit machines, size_t = 8 bytes, 2^30 * 2 * 2 = 2^32 == 4 Gb */ static size_t maxAlloc = 128*8*1024*1024*(sizeof(size_t)/4)*(sizeof(size_t)/4); Although the HugeMem allocator doesn't even check for a limit, it just goes ahead and asks for the memory: if ((pt = mhStack->alloc(size)) == NULL) errAbort("needHugeMen: Out of huge memory - request size %llu bytes\n", (unsigned long long)size); where mhStack->alloc() would be the defaultAlloc unless you have carefulMemHandler in place, which translates to a simple malloc() --Hiram Alexander Poliakov wrote: > Dear Hiram, > > > I just tried this on chr1 of Opossum, 729 Mb and it converts > > with a peak memory usage of just over 1 Gb. > > That is exactly what I was trying to do - chr1 of opossum. I am using > Fedora Core 6 on a machine with an Intel Xeon CPU. Here is what I get > from tcsh: > > cputime unlimited > filesize unlimited > datasize unlimited > stacksize unlimited > coredumpsize 0 kbytes > memoryuse unlimited > vmemoryuse unlimited > descriptors 1024 > memorylocked unlimited > maxproc 65536 > > And here is what I get from "free": > > total used free shared buffers cached > Mem: 3631028 2214164 1416864 0 165472 1440940 > -/+ buffers/cache: 607752 3023276 > Swap: 4192956 0 4192956 > > > Have you considered using the 2bit file format ? > > There are a few programs developed in our lab that work with NIB files, > and I do not want to rewrite them at the moment. > > Thanks, > Alex > > Hiram Clawson wrote: >> Good Afternoon Alex: >> >> What type of CPU and operating system are you working on ? >> Does your login shell have process size limits ? See what >> the command: ulimit >> can tell you about your process limits. >> >> I just tried this on chr1 of Opossum, 729 Mb and it converts >> with a peak memory usage of just over 1 Gb. >> >> Have you considered using the 2bit file format ? All the kent >> utilities can also work with the 2bit file format and it is a bit >> more efficient that the nib format. >> >> --Hiram >> >> Alexander Poliakov wrote: >> >>> Hello, >>> >>> I tried to convert a sequence from FASTA to NIB format using >>> "faToNib", but all I got from the program was the following error. >>> >>> needHugeMen: Out of huge memory - request size 1073741824 bytes >>> >>> The sequence is ~740MB long and the machine has 4GB RAM. Could you >>> please help me make it work? >>> >>> Thanks, >>> Alex > From avpoliakov at lbl.gov Tue Apr 24 08:52:39 2007 From: avpoliakov at lbl.gov (Alexander Poliakov) Date: Tue, 24 Apr 2007 08:52:39 -0700 Subject: [Genome-mirror] faToNib In-Reply-To: <462D3B92.6040603@soe.ucsc.edu> References: <462D2A6F.5080307@lbl.gov> <462D30A9.9070005@soe.ucsc.edu> <462D3518.2050804@lbl.gov> <462D3B92.6040603@soe.ucsc.edu> Message-ID: <462E27C7.2050101@lbl.gov> > Check your kent/src/lib/memalloc.c and see if its definition of maxAlloc > matches this: > /* 128*8*1024*1024 == 1073741824 == 2^30 on 32 bit machines,size_t == 4 > bytes*/ > /* on 64 bit machines, size_t = 8 bytes, 2^30 * 2 * 2 = 2^32 == 4 Gb */ > static size_t maxAlloc = > 128*8*1024*1024*(sizeof(size_t)/4)*(sizeof(size_t)/4); Yes, it matches. So, is there any way to convert a ~700MB fasta file to nib besides writing my own program for that? Thanks, Alex Hiram Clawson wrote: > Alex: > > Check your kent/src/lib/memalloc.c and see if its definition of maxAlloc > matches this: > /* 128*8*1024*1024 == 1073741824 == 2^30 on 32 bit machines,size_t == 4 > bytes*/ > /* on 64 bit machines, size_t = 8 bytes, 2^30 * 2 * 2 = 2^32 == 4 Gb */ > static size_t maxAlloc = > 128*8*1024*1024*(sizeof(size_t)/4)*(sizeof(size_t)/4); > > Although the HugeMem allocator doesn't even check for a limit, it just > goes ahead and asks for the memory: > if ((pt = mhStack->alloc(size)) == NULL) > errAbort("needHugeMen: Out of huge memory - request size %llu bytes\n", > (unsigned long long)size); > where mhStack->alloc() would be the defaultAlloc unless you have > carefulMemHandler in place, which translates to a simple malloc() > > --Hiram > > Alexander Poliakov wrote: > >> Dear Hiram, >> >> > I just tried this on chr1 of Opossum, 729 Mb and it converts >> > with a peak memory usage of just over 1 Gb. >> >> That is exactly what I was trying to do - chr1 of opossum. I am using >> Fedora Core 6 on a machine with an Intel Xeon CPU. Here is what I get >> from tcsh: >> >> cputime unlimited >> filesize unlimited >> datasize unlimited >> stacksize unlimited >> coredumpsize 0 kbytes >> memoryuse unlimited >> vmemoryuse unlimited >> descriptors 1024 >> memorylocked unlimited >> maxproc 65536 >> >> And here is what I get from "free": >> >> total used free shared buffers cached >> Mem: 3631028 2214164 1416864 0 165472 1440940 >> -/+ buffers/cache: 607752 3023276 >> Swap: 4192956 0 4192956 >> >> > Have you considered using the 2bit file format ? >> >> There are a few programs developed in our lab that work with NIB >> files, and I do not want to rewrite them at the moment. >> >> Thanks, >> Alex >> >> Hiram Clawson wrote: >> >>> Good Afternoon Alex: >>> >>> What type of CPU and operating system are you working on ? >>> Does your login shell have process size limits ? See what >>> the command: ulimit >>> can tell you about your process limits. >>> >>> I just tried this on chr1 of Opossum, 729 Mb and it converts >>> with a peak memory usage of just over 1 Gb. >>> >>> Have you considered using the 2bit file format ? All the kent >>> utilities can also work with the 2bit file format and it is a bit >>> more efficient that the nib format. >>> >>> --Hiram >>> >>> Alexander Poliakov wrote: >>> >>>> Hello, >>>> >>>> I tried to convert a sequence from FASTA to NIB format using >>>> "faToNib", but all I got from the program was the following error. >>>> >>>> needHugeMen: Out of huge memory - request size 1073741824 bytes >>>> >>>> The sequence is ~740MB long and the machine has 4GB RAM. Could you >>>> please help me make it work? >>>> >>>> Thanks, >>>> Alex From hiram at soe.ucsc.edu Tue Apr 24 10:48:07 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Tue, 24 Apr 2007 10:48:07 -0700 Subject: [Genome-mirror] faToNib In-Reply-To: <462E27C7.2050101@lbl.gov> References: <462D2A6F.5080307@lbl.gov> <462D30A9.9070005@soe.ucsc.edu> <462D3518.2050804@lbl.gov> <462D3B92.6040603@soe.ucsc.edu> <462E27C7.2050101@lbl.gov> Message-ID: <462E42D7.4090200@soe.ucsc.edu> Good Morning Alex: I don't know what is going on here. I ran faToNib on a 32 bit machine here and it has no difficulty. You could try changing the maxAlloc equation to include another factor of 2 to get up to 2^31 allowed in your environment and see if that gets it to go, although that really shouldn't be an issue. Something in your system is preventing a large program from running. In your source tree, what does it say in the file src/hg/inc/versionInfo.h ? We are currently on version 157, is yours much older ? Attached is a simple test program I use to verify memory limits in running programs. You compile: $ gcc -o memhogtest memhogtest.c And run it, to test 1 Gb, for example: $ ./memhogtest 1024 --Hiram Alexander Poliakov wrote: > > Check your kent/src/lib/memalloc.c and see if its definition of maxAlloc > > matches this: > > /* 128*8*1024*1024 == 1073741824 == 2^30 on 32 bit machines,size_t == 4 > > bytes*/ > > /* on 64 bit machines, size_t = 8 bytes, 2^30 * 2 * 2 = 2^32 == 4 Gb */ > > static size_t maxAlloc = > > 128*8*1024*1024*(sizeof(size_t)/4)*(sizeof(size_t)/4); > > Yes, it matches. So, is there any way to convert a ~700MB fasta file to > nib besides writing my own program for that? > > Thanks, > Alex > > Hiram Clawson wrote: >> Alex: >> >> Check your kent/src/lib/memalloc.c and see if its definition of >> maxAlloc matches this: >> /* 128*8*1024*1024 == 1073741824 == 2^30 on 32 bit machines,size_t == >> 4 bytes*/ >> /* on 64 bit machines, size_t = 8 bytes, 2^30 * 2 * 2 = 2^32 == 4 Gb */ >> static size_t maxAlloc = >> 128*8*1024*1024*(sizeof(size_t)/4)*(sizeof(size_t)/4); >> >> Although the HugeMem allocator doesn't even check for a limit, it just >> goes ahead and asks for the memory: >> if ((pt = mhStack->alloc(size)) == NULL) >> errAbort("needHugeMen: Out of huge memory - request size %llu >> bytes\n", >> (unsigned long long)size); >> where mhStack->alloc() would be the defaultAlloc unless you have >> carefulMemHandler in place, which translates to a simple malloc() >> >> --Hiram >> >> Alexander Poliakov wrote: >> >>> Dear Hiram, >>> >>> > I just tried this on chr1 of Opossum, 729 Mb and it converts >>> > with a peak memory usage of just over 1 Gb. >>> >>> That is exactly what I was trying to do - chr1 of opossum. I am using >>> Fedora Core 6 on a machine with an Intel Xeon CPU. Here is what I get >>> from tcsh: >>> >>> cputime unlimited >>> filesize unlimited >>> datasize unlimited >>> stacksize unlimited >>> coredumpsize 0 kbytes >>> memoryuse unlimited >>> vmemoryuse unlimited >>> descriptors 1024 >>> memorylocked unlimited >>> maxproc 65536 >>> >>> And here is what I get from "free": >>> >>> total used free shared buffers >>> cached >>> Mem: 3631028 2214164 1416864 0 165472 >>> 1440940 >>> -/+ buffers/cache: 607752 3023276 >>> Swap: 4192956 0 4192956 >>> >>> > Have you considered using the 2bit file format ? >>> >>> There are a few programs developed in our lab that work with NIB >>> files, and I do not want to rewrite them at the moment. >>> >>> Thanks, >>> Alex >>> >>> Hiram Clawson wrote: >>> >>>> Good Afternoon Alex: >>>> >>>> What type of CPU and operating system are you working on ? >>>> Does your login shell have process size limits ? See what >>>> the command: ulimit >>>> can tell you about your process limits. >>>> >>>> I just tried this on chr1 of Opossum, 729 Mb and it converts >>>> with a peak memory usage of just over 1 Gb. >>>> >>>> Have you considered using the 2bit file format ? All the kent >>>> utilities can also work with the 2bit file format and it is a bit >>>> more efficient that the nib format. >>>> >>>> --Hiram >>>> >>>> Alexander Poliakov wrote: >>>> >>>>> Hello, >>>>> >>>>> I tried to convert a sequence from FASTA to NIB format using >>>>> "faToNib", but all I got from the program was the following error. >>>>> >>>>> needHugeMen: Out of huge memory - request size 1073741824 bytes >>>>> >>>>> The sequence is ~740MB long and the machine has 4GB RAM. Could you >>>>> please help me make it work? >>>>> >>>>> Thanks, >>>>> Alex > From avpoliakov at lbl.gov Tue Apr 24 11:02:44 2007 From: avpoliakov at lbl.gov (Alexander Poliakov) Date: Tue, 24 Apr 2007 11:02:44 -0700 Subject: [Genome-mirror] faToNib In-Reply-To: <462E42D7.4090200@soe.ucsc.edu> References: <462D2A6F.5080307@lbl.gov> <462D30A9.9070005@soe.ucsc.edu> <462D3518.2050804@lbl.gov> <462D3B92.6040603@soe.ucsc.edu> <462E27C7.2050101@lbl.gov> <462E42D7.4090200@soe.ucsc.edu> Message-ID: <462E4644.9080602@lbl.gov> Hi Hiram, > In your source tree, what does it say in the > file src/hg/inc/versionInfo.h ? 157 > Attached is a simple test program I use to verify memory limits in > running programs. I tried it with 1024 and 2048, and it ran OK. So, what is so different about faToNib? Any ideas? Thanks, Alex Hiram Clawson wrote: > Good Morning Alex: > > I don't know what is going on here. I ran faToNib on a 32 bit machine > here and it has no difficulty. You could try changing the maxAlloc > equation to include another factor of 2 to get up to 2^31 allowed > in your environment and see if that gets it to go, although that really > shouldn't be an issue. Something in your system is preventing a large > program from running. > > In your source tree, what does it say in the > file src/hg/inc/versionInfo.h ? > We are currently on version 157, is yours much older ? > > Attached is a simple test program I use to verify memory limits in > running programs. You compile: > $ gcc -o memhogtest memhogtest.c > And run it, to test 1 Gb, for example: > $ ./memhogtest 1024 > > --Hiram > > Alexander Poliakov wrote: > >> > Check your kent/src/lib/memalloc.c and see if its definition of >> maxAlloc >> > matches this: >> > /* 128*8*1024*1024 == 1073741824 == 2^30 on 32 bit machines,size_t >> == 4 >> > bytes*/ >> > /* on 64 bit machines, size_t = 8 bytes, 2^30 * 2 * 2 = 2^32 == 4 >> Gb */ >> > static size_t maxAlloc = >> > 128*8*1024*1024*(sizeof(size_t)/4)*(sizeof(size_t)/4); >> >> Yes, it matches. So, is there any way to convert a ~700MB fasta file >> to nib besides writing my own program for that? >> >> Thanks, >> Alex >> >> Hiram Clawson wrote: >> >>> Alex: >>> >>> Check your kent/src/lib/memalloc.c and see if its definition of >>> maxAlloc matches this: >>> /* 128*8*1024*1024 == 1073741824 == 2^30 on 32 bit machines,size_t == >>> 4 bytes*/ >>> /* on 64 bit machines, size_t = 8 bytes, 2^30 * 2 * 2 = 2^32 == 4 Gb */ >>> static size_t maxAlloc = >>> 128*8*1024*1024*(sizeof(size_t)/4)*(sizeof(size_t)/4); >>> >>> Although the HugeMem allocator doesn't even check for a limit, it just >>> goes ahead and asks for the memory: >>> if ((pt = mhStack->alloc(size)) == NULL) >>> errAbort("needHugeMen: Out of huge memory - request size %llu >>> bytes\n", >>> (unsigned long long)size); >>> where mhStack->alloc() would be the defaultAlloc unless you have >>> carefulMemHandler in place, which translates to a simple malloc() >>> >>> --Hiram >>> >>> Alexander Poliakov wrote: >>> >>>> Dear Hiram, >>>> >>>> > I just tried this on chr1 of Opossum, 729 Mb and it converts >>>> > with a peak memory usage of just over 1 Gb. >>>> >>>> That is exactly what I was trying to do - chr1 of opossum. I am >>>> using Fedora Core 6 on a machine with an Intel Xeon CPU. Here is >>>> what I get from tcsh: >>>> >>>> cputime unlimited >>>> filesize unlimited >>>> datasize unlimited >>>> stacksize unlimited >>>> coredumpsize 0 kbytes >>>> memoryuse unlimited >>>> vmemoryuse unlimited >>>> descriptors 1024 >>>> memorylocked unlimited >>>> maxproc 65536 >>>> >>>> And here is what I get from "free": >>>> >>>> total used free shared buffers >>>> cached >>>> Mem: 3631028 2214164 1416864 0 165472 >>>> 1440940 >>>> -/+ buffers/cache: 607752 3023276 >>>> Swap: 4192956 0 4192956 >>>> >>>> > Have you considered using the 2bit file format ? >>>> >>>> There are a few programs developed in our lab that work with NIB >>>> files, and I do not want to rewrite them at the moment. >>>> >>>> Thanks, >>>> Alex >>>> >>>> Hiram Clawson wrote: >>>> >>>>> Good Afternoon Alex: >>>>> >>>>> What type of CPU and operating system are you working on ? >>>>> Does your login shell have process size limits ? See what >>>>> the command: ulimit >>>>> can tell you about your process limits. >>>>> >>>>> I just tried this on chr1 of Opossum, 729 Mb and it converts >>>>> with a peak memory usage of just over 1 Gb. >>>>> >>>>> Have you considered using the 2bit file format ? All the kent >>>>> utilities can also work with the 2bit file format and it is a bit >>>>> more efficient that the nib format. >>>>> >>>>> --Hiram >>>>> >>>>> Alexander Poliakov wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> I tried to convert a sequence from FASTA to NIB format using >>>>>> "faToNib", but all I got from the program was the following error. >>>>>> >>>>>> needHugeMen: Out of huge memory - request size 1073741824 bytes >>>>>> >>>>>> The sequence is ~740MB long and the machine has 4GB RAM. Could you >>>>>> please help me make it work? >>>>>> >>>>>> Thanks, >>>>>> Alex From archanat at soe.ucsc.edu Wed Apr 25 13:22:31 2007 From: archanat at soe.ucsc.edu (Archana Thakkapallayil) Date: Wed, 25 Apr 2007 13:22:31 -0700 Subject: [Genome-mirror] Release of new assembly: oryLat1 (Medaka) Message-ID: <462FB887.2030300@soe.ucsc.edu> Hello mirror sites, This message is to announce that we are preparing to release the new Medaka assembly, oryLat1, soon. Mirror sites should be prepared to host: ~11 Gb ( oryLat1 tables ) ~3.4 Gb ( files in /gbdb/oryLat1/* ) Additionally there are net and chain tables from other organisms that will be released to the respective annotation databases. These tables are named netOryLat1 and *chainOryLat1*. The size of these tables is as follows: hg18 281 MB tetNig1 911 MB gasAcu1 1500 MB danRer4 5216 MB fr2 1194 MB total: ~9 GB Thanks, Archana UCSC Genome Bioinformatics Group From reiner.schulz at kcl.ac.uk Fri Apr 27 05:28:01 2007 From: reiner.schulz at kcl.ac.uk (Reiner Schulz) Date: Fri, 27 Apr 2007 13:28:01 +0100 Subject: [Genome-mirror] Comparative Genomics: Chain tracks In-Reply-To: <462CAF34.3080403@kcl.ac.uk> References: <462CAF34.3080403@kcl.ac.uk> Message-ID: <4631EC51.30605@kcl.ac.uk> just FYI, that was entirely my fault (broke the code w/ a patch). r Reiner Schulz wrote: > clicking on an entry in one of the chain tracks results in a server > error on my mirror: 'Premature end of script headers: hgc' > the mirror contains mm8, hg18 and rn4, and no other genomes. the > complete /gbdb and /var/lib/mysql/[mm8|hg18|rn4] are regularly rsync'ed. > just wondering if this is a common problem w/ an already known solution. > i suspect that i'm missing some data that are necessary and not > contained in /gbdb or /var/lib/mysql/[mm8|hg18|rn4]. > > much appreciated, > > Reiner -- (*)->[]->()->[]->(**)->[]->()->[]->(*)->[]->()->[]->()->[]->()->[]->()->[] (Humboldt University Berlin, Germany)->[]-> ... (University of Maryland, USA)->[]-> ... (King's College London, UK) https://josh.umds.ac.uk/~rschulz From davide.cittaro at ifom-ieo-campus.it Fri Apr 27 02:44:42 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Fri, 27 Apr 2007 11:44:42 +0200 Subject: [Genome-mirror] genome browser table browser errors Message-ID: Dear UCSC people, I think I have some issues with mysql permissions and/or hgTables binary. As I click on "paste list" or "upload list" button in main "Tables" page I get this error: Can't start query: create temporary table tmpknownGene select name from knownGene limit 100000 mySQL error 1044: Access denied for user 'genome'@'%' to database 'hg18' Apparently this happens with every table: Can't start query: create temporary table tmprefGene select name from refGene limit 100000 Can't start query: create temporary table tmpchr1_chainMonDom4 select qName from chr1_chainMonDom4 limit 100000 And so on... I've checked the ''genome'@'%' permissions, that are: GRANT SELECT, EXECUTE ON `hg%`.* TO 'genome'@'%' for hg% databases, as I found in your mysql server. The gbrowser user (the one that acts behind the software) is GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, ALTER ON `hg%`.* TO 'gbrowser'@'localhost' Apparently I cannot use paste/upload features (but I can do it on your main server). Can anybody help me on this? Thanks d /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro at ifom-ieo-campus.it */ From davide.cittaro at ifom-ieo-campus.it Fri Apr 27 02:44:42 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Fri, 27 Apr 2007 11:44:42 +0200 Subject: [Genome-mirror] genome browser table browser errors #2 Message-ID: <15D12566-5A73-41E3-A024-C5EAB964FA2B@ifom-ieo-campus.it> I forgot to say that I'm running v157 of genome browser d > Dear UCSC people, I think I have some issues with mysql permissions > and/or hgTables binary. > As I click on "paste list" or "upload list" button in main "Tables" > page I get this error: > Can't start query: create temporary table tmpknownGene select name > from knownGene limit 100000 > > mySQL error 1044: Access denied for user 'genome'@'%' to database > 'hg18' > > Apparently this happens with every table: > Can't start query: create temporary table tmprefGene select name > from refGene limit 100000 > > Can't start query: create temporary table tmpchr1_chainMonDom4 > select qName from chr1_chainMonDom4 limit 100000 > > And so on... I've checked the ''genome'@'%' permissions, that are: > > GRANT SELECT, EXECUTE ON `hg%`.* TO 'genome'@'%' > > for hg% databases, as I found in your mysql server. The gbrowser > user (the one that acts behind the software) is > > GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, ALTER ON `hg% > `.* TO 'gbrowser'@'localhost' > > Apparently I cannot use paste/upload features (but I can do it on > your main server). Can anybody help me on this? > > Thanks > > d /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro at ifom-ieo-campus.it */ From davide.cittaro at ifom-ieo-campus.it Fri Apr 27 02:56:28 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Fri, 27 Apr 2007 11:56:28 +0200 Subject: [Genome-mirror] genome browser table browser errors #2 In-Reply-To: <15D12566-5A73-41E3-A024-C5EAB964FA2B@ifom-ieo-campus.it> References: <15D12566-5A73-41E3-A024-C5EAB964FA2B@ifom-ieo-campus.it> Message-ID: Apparently v156 of genome browser doesn't have this problem d On Apr 27, 2007, at 11:53 AM, Davide Cittaro wrote: > I forgot to say that I'm running v157 of genome browser > > d > >> Dear UCSC people, I think I have some issues with mysql >> permissions and/or hgTables binary. >> As I click on "paste list" or "upload list" button in main >> "Tables" page I get this error: >> Can't start query: create temporary table tmpknownGene select name >> from knownGene limit 100000 >> >> mySQL error 1044: Access denied for user 'genome'@'%' to database >> 'hg18' >> >> Apparently this happens with every table: >> Can't start query: create temporary table tmprefGene select name >> from refGene limit 100000 >> >> Can't start query: create temporary table tmpchr1_chainMonDom4 >> select qName from chr1_chainMonDom4 limit 100000 >> >> And so on... I've checked the ''genome'@'%' permissions, that are: >> >> GRANT SELECT, EXECUTE ON `hg%`.* TO 'genome'@'%' >> >> for hg% databases, as I found in your mysql server. The gbrowser >> user (the one that acts behind the software) is >> >> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, ALTER ON `hg% >> `.* TO 'gbrowser'@'localhost' >> >> Apparently I cannot use paste/upload features (but I can do it on >> your main server). Can anybody help me on this? >> >> Thanks >> >> d > /* > > Davide Cittaro > > HPC and Bioinformatics Systems @ Informatics Core > > IFOM - Istituto FIRC di Oncologia Molecolare > via adamello, 16 > 20139 Milano > Italy > > tel.: +39(02)574303007 > e-mail: davide.cittaro at ifom-ieo-campus.it > */ > > /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro at ifom-ieo-campus.it */ From angie at soe.ucsc.edu Fri Apr 27 10:44:12 2007 From: angie at soe.ucsc.edu (Angie Hinrichs) Date: Fri, 27 Apr 2007 10:44:12 -0700 (PDT) Subject: [Genome-mirror] genome browser table browser errors #2 In-Reply-To: References: <15D12566-5A73-41E3-A024-C5EAB964FA2B@ifom-ieo-campus.it> Message-ID: Hi Davide, Sorry about the inconvenience! As you have guessed, new code has been added in v157 that requires the CGI mysql user to create a temporary table, and that code is reached by the paste/upload page (that is the only place in the CGIs, I believe). It is a performance tweak for the function of selecting random rows from a table. Some of our tables have several million rows, so a plain old sort-by-rand() on the table takes a very long time (like 15-20s). To prevent that, we create a temporary table of limited size and randomize that (max 1-1.5s). I played around a bit with other approaches e.g. computing random numbers less than the row count and then 'limit offset,1' but that also was quite slow. It so happened that our local mysql permissions already included Create_tmp_table_priv so this didn't require a change here. Our public mysql server is a little more restrictive. We will have to document this new requirement. Angie On Fri, 27 Apr 2007, Davide Cittaro wrote: > Apparently v156 of genome browser doesn't have this problem > > > d > On Apr 27, 2007, at 11:53 AM, Davide Cittaro wrote: > > > I forgot to say that I'm running v157 of genome browser > > > > d > > > >> Dear UCSC people, I think I have some issues with mysql > >> permissions and/or hgTables binary. > >> As I click on "paste list" or "upload list" button in main > >> "Tables" page I get this error: > >> Can't start query: create temporary table tmpknownGene select name > >> from knownGene limit 100000 > >> > >> mySQL error 1044: Access denied for user 'genome'@'%' to database > >> 'hg18' > >> > >> Apparently this happens with every table: > >> Can't start query: create temporary table tmprefGene select name > >> from refGene limit 100000 > >> > >> Can't start query: create temporary table tmpchr1_chainMonDom4 > >> select qName from chr1_chainMonDom4 limit 100000 > >> > >> And so on... I've checked the ''genome'@'%' permissions, that are: > >> > >> GRANT SELECT, EXECUTE ON `hg%`.* TO 'genome'@'%' > >> > >> for hg% databases, as I found in your mysql server. The gbrowser > >> user (the one that acts behind the software) is > >> > >> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, ALTER ON `hg% > >> `.* TO 'gbrowser'@'localhost' > >> > >> Apparently I cannot use paste/upload features (but I can do it on > >> your main server). Can anybody help me on this? > >> > >> Thanks > >> > >> d > > /* > > > > Davide Cittaro > > > > HPC and Bioinformatics Systems @ Informatics Core > > > > IFOM - Istituto FIRC di Oncologia Molecolare > > via adamello, 16 > > 20139 Milano > > Italy > > > > tel.: +39(02)574303007 > > e-mail: davide.cittaro at ifom-ieo-campus.it > > */ > > > > > > /* > Davide Cittaro > HPC and Bioinformatics Systems @ Informatics Core > > IFOM - Istituto FIRC di Oncologia Molecolare > via adamello, 16 > 20139 Milano > Italy > > tel.: +39(02)574303007 > e-mail: davide.cittaro at ifom-ieo-campus.it > */ > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From davide.cittaro at ifom-ieo-campus.it Fri Apr 27 11:46:07 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Fri, 27 Apr 2007 20:46:07 +0200 Subject: [Genome-mirror] genome browser table browser errors #2 In-Reply-To: References: <15D12566-5A73-41E3-A024-C5EAB964FA2B@ifom-ieo-campus.it> Message-ID: Hi Angie, thanks for the answer On Apr 27, 2007, at 7:44 PM, Angie Hinrichs wrote: > Hi Davide, > > Sorry about the inconvenience! I see that the duke university mirror has the same problem :-( > As you have guessed, new code has been > added in v157 that requires the CGI mysql user to create a temporary > table, and that code is reached by the paste/upload page (that is the > only place in the CGIs, I believe). It is a performance tweak for the > function of selecting random rows from a table. Some of our tables > have several million rows, so a plain old sort-by-rand() on the table > takes a very long time (like 15-20s). To prevent that, we create a > temporary table of limited size and randomize that (max 1-1.5s). I > played around a bit with other approaches e.g. computing random > numbers less than the row count and then 'limit offset,1' but > that also was quite slow. > I understand that and I'm happy to see that your project is growing and shaping better and better. > It so happened that our local mysql permissions already included > Create_tmp_table_priv so this didn't require a change here. Our > public mysql server is a little more restrictive. We will have to > document this new requirement. :-) I think so! I understand that your public mysql server is not the one your genome browser relies on... Meanwhile I'll take a look to mysql documents to understand how to give that grant to 'genome' user, hoping that is the only one who needs fixes. d /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro at ifom-ieo-campus.it */ From hiram at soe.ucsc.edu Fri Apr 27 11:53:44 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Fri, 27 Apr 2007 11:53:44 -0700 Subject: [Genome-mirror] speaking of browser mirror configurations In-Reply-To: References: <15D12566-5A73-41E3-A024-C5EAB964FA2B@ifom-ieo-campus.it> Message-ID: <463246B8.1010507@soe.ucsc.edu> I'm curious. Did anyone try to turn on the custom tracks database business I mentioned here in email a couple weeks back ? --Hiram From donnak at soe.ucsc.edu Fri Apr 27 13:07:34 2007 From: donnak at soe.ucsc.edu (Donna Karolchik) Date: Fri, 27 Apr 2007 13:07:34 -0700 Subject: [Genome-mirror] we want your feedback! Message-ID: <00f701c78907$b6231f80$0ba8a8c0@donnakLT> hi all, We've put together a survey to collect feedback from the Genome Browser community. This is your chance to let us know what changes you'd like to see to the Genome Browser tools, data, and web site, and what you like about the current implementation. Please take a few minutes to participate in our survey: http://www.surveymonkey.com/s.asp?u=881163743177. You will also find a survey link on the Genome Browser home page at http://genome.ucsc.edu. The survey is open through the end of May. Thanks for your participation! -Donna ----------------------------------- Donna Karolchik UCSC Genome Bioinformatics Group http://genome.ucsc.edu From tsfurey at duke.edu Fri Apr 27 13:20:09 2007 From: tsfurey at duke.edu (Terry Furey) Date: Fri, 27 Apr 2007 16:20:09 -0400 (EDT) Subject: [Genome-mirror] genome browser table browser errors #2 In-Reply-To: References: <15D12566-5A73-41E3-A024-C5EAB964FA2B@ifom-ieo-campus.it> Message-ID: David and all - Thanks for bringing this to my attention. I think that we have made the necessary priveledge changes and that all is working again. David - I'd appreciate it if you could do a quick test like before and see if things work, now. In an unrelated problem, I have noticed that the proteins database has been updated to proteins070202, but I am not getting this database currently in my rsyncs. Is this database on the rsync server? It may just be a problem on my side. Thanks, Terry On Fri, 27 Apr 2007, Davide Cittaro wrote: > Hi Angie, thanks for the answer > On Apr 27, 2007, at 7:44 PM, Angie Hinrichs wrote: > >> Hi Davide, >> >> Sorry about the inconvenience! > > I see that the duke university mirror has the same problem :-( > >> As you have guessed, new code has been >> added in v157 that requires the CGI mysql user to create a temporary >> table, and that code is reached by the paste/upload page (that is the >> only place in the CGIs, I believe). It is a performance tweak for the >> function of selecting random rows from a table. Some of our tables >> have several million rows, so a plain old sort-by-rand() on the table >> takes a very long time (like 15-20s). To prevent that, we create a >> temporary table of limited size and randomize that (max 1-1.5s). I >> played around a bit with other approaches e.g. computing random >> numbers less than the row count and then 'limit offset,1' but >> that also was quite slow. >> > > I understand that and I'm happy to see that your project is growing > and shaping better and better. > >> It so happened that our local mysql permissions already included >> Create_tmp_table_priv so this didn't require a change here. Our >> public mysql server is a little more restrictive. We will have to >> document this new requirement. > > :-) I think so! I understand that your public mysql server is not the > one your genome browser relies on... > Meanwhile I'll take a look to mysql documents to understand how to > give that grant to 'genome' user, hoping that is the only one who > needs fixes. > > d > > /* > Davide Cittaro > HPC and Bioinformatics Systems @ Informatics Core > > IFOM - Istituto FIRC di Oncologia Molecolare > via adamello, 16 > 20139 Milano > Italy > > tel.: +39(02)574303007 > e-mail: davide.cittaro at ifom-ieo-campus.it > */ > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From ann at soe.ucsc.edu Fri Apr 27 13:29:40 2007 From: ann at soe.ucsc.edu (Ann Zweig) Date: Fri, 27 Apr 2007 13:29:40 -0700 Subject: [Genome-mirror] genome browser table browser errors #2 In-Reply-To: References: <15D12566-5A73-41E3-A024-C5EAB964FA2B@ifom-ieo-campus.it> Message-ID: <46325D34.8040801@cse.ucsc.edu> Hi Terry, Yes, the proteins070202 database is available at: hgdownload.cse.ucsc.edu/goldenPath/proteinDB/proteins070202/database/ Regards, ---------- Ann Zweig UCSC Genome Bioinformatics Group http://genome.ucsc.edu Terry Furey wrote: > David and all - > > Thanks for bringing this to my attention. I think that we have made the > necessary priveledge changes and that all is working again. David - I'd > appreciate it if you could do a quick test like before and see if things > work, now. > > In an unrelated problem, I have noticed that the proteins database has > been updated to proteins070202, but I am not getting this database > currently in my rsyncs. Is this database on the rsync server? It may > just be a problem on my side. > > Thanks, > > Terry > > > On Fri, 27 Apr 2007, Davide Cittaro wrote: > >> Hi Angie, thanks for the answer >> On Apr 27, 2007, at 7:44 PM, Angie Hinrichs wrote: >> >>> Hi Davide, >>> >>> Sorry about the inconvenience! >> I see that the duke university mirror has the same problem :-( >> >>> As you have guessed, new code has been >>> added in v157 that requires the CGI mysql user to create a temporary >>> table, and that code is reached by the paste/upload page (that is the >>> only place in the CGIs, I believe). It is a performance tweak for the >>> function of selecting random rows from a table. Some of our tables >>> have several million rows, so a plain old sort-by-rand() on the table >>> takes a very long time (like 15-20s). To prevent that, we create a >>> temporary table of limited size and randomize that (max 1-1.5s). I >>> played around a bit with other approaches e.g. computing random >>> numbers less than the row count and then 'limit offset,1' but >>> that also was quite slow. >>> >> I understand that and I'm happy to see that your project is growing >> and shaping better and better. >> >>> It so happened that our local mysql permissions already included >>> Create_tmp_table_priv so this didn't require a change here. Our >>> public mysql server is a little more restrictive. We will have to >>> document this new requirement. >> :-) I think so! I understand that your public mysql server is not the >> one your genome browser relies on... >> Meanwhile I'll take a look to mysql documents to understand how to >> give that grant to 'genome' user, hoping that is the only one who >> needs fixes. >> >> d >> >> /* >> Davide Cittaro >> HPC and Bioinformatics Systems @ Informatics Core >> >> IFOM - Istituto FIRC di Oncologia Molecolare >> via adamello, 16 >> 20139 Milano >> Italy >> >> tel.: +39(02)574303007 >> e-mail: davide.cittaro at ifom-ieo-campus.it >> */ >> >> >> _______________________________________________ >> Genome-mirror mailing list >> Genome-mirror at soe.ucsc.edu >> http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror >> > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror From davide.cittaro at ifom-ieo-campus.it Fri Apr 27 14:09:18 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Fri, 27 Apr 2007 23:09:18 +0200 Subject: [Genome-mirror] speaking of browser mirror configurations In-Reply-To: <463246B8.1010507@soe.ucsc.edu> References: <15D12566-5A73-41E3-A024-C5EAB964FA2B@ifom-ieo-campus.it> <463246B8.1010507@soe.ucsc.edu> Message-ID: Hi Hiram On Apr 27, 2007, at 8:53 PM, Hiram Clawson wrote: > I'm curious. Did anyone try to turn on the custom tracks database > business I mentioned here in email a couple weeks back ? > I had a look at the wiki page and I've added the track database in my to-do list. If you care I can write to genome-mirror once everything will be up and running. d > --Hiram > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro at ifom-ieo-campus.it */ From davide.cittaro at ifom-ieo-campus.it Fri Apr 27 14:27:02 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Fri, 27 Apr 2007 23:27:02 +0200 Subject: [Genome-mirror] genome browser table browser errors #2 In-Reply-To: References: <15D12566-5A73-41E3-A024-C5EAB964FA2B@ifom-ieo-campus.it> Message-ID: On Apr 27, 2007, at 10:20 PM, Terry Furey wrote: > David and all - > > Thanks for bringing this to my attention. I think that we have > made the necessary priveledge changes and that all is working > again. David - I'd appreciate it if you could do a quick test like > before and see if things work, now. > Yep! It works. Which changes have you applied? d /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro at ifom-ieo-campus.it */ From davide.cittaro at ifom-ieo-campus.it Fri Apr 27 14:55:44 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Fri, 27 Apr 2007 23:55:44 +0200 Subject: [Genome-mirror] genome browser table browser errors #2 In-Reply-To: References: <15D12566-5A73-41E3-A024-C5EAB964FA2B@ifom-ieo-campus.it> Message-ID: Hi all, even here it has been fixed with the "CREATE TEMPORARY TABLES" privilege... it's time to sleep now :-) d On Apr 27, 2007, at 11:27 PM, Davide Cittaro wrote: > > On Apr 27, 2007, at 10:20 PM, Terry Furey wrote: > >> David and all - >> >> Thanks for bringing this to my attention. I think that we have >> made the necessary priveledge changes and that all is working >> again. David - I'd appreciate it if you could do a quick test like >> before and see if things work, now. >> > > Yep! It works. Which changes have you applied? > > d > > /* > Davide Cittaro > HPC and Bioinformatics Systems @ Informatics Core > > IFOM - Istituto FIRC di Oncologia Molecolare > via adamello, 16 > 20139 Milano > Italy > > tel.: +39(02)574303007 > e-mail: davide.cittaro at ifom-ieo-campus.it > */ > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro at ifom-ieo-campus.it */