From jizu at osa.sunysb.edu Mon Nov 5 10:08:09 2007 From: jizu at osa.sunysb.edu (jizu) Date: Mon, 05 Nov 2007 13:08:09 -0500 Subject: [Genome-mirror] different BLAT result using local mirror Message-ID: <472F5C09.3070806@osa.sunysb.edu> Hi gurus, We are having a strange problem with our local BLAT mirror. For example, if submitting the following sequence to search against hg18 GGCCAGGCGCAGTCCCTTACACCTGTAATCTCAGCACTTTGGGAGGCTGA GGCAGGAGGATCGCCTGAGCCTGGGAGATGAGGCTGCAGTGAGCTATGAC AGCACCACTGCACTCCAGCCTGGGCAACAGAGAGAGACGCTGTCTCAAA Top Hits on UCSC site ACTIONS QUERY SCORE START END QSIZE IDENTITY CHRO STRAND START END SPAN --------------------------------------------------------------------------------------------------- browser details YourSeq 149 1 149 149 100.0% 3 + 38840 38988 149 browser details YourSeq 131 1 149 149 94.0% 3 + 75370223 75370371 149 browser details YourSeq 106 2 149 149 86.4% 3 - 186543951 186544231 281 browser details YourSeq 106 1 149 149 88.0% 3 + 150710511 150762774 52264 browser details YourSeq 106 1 146 149 87.0% 19 + 6067438 6067724 287 browser details YourSeq 106 1 149 149 86.4% 11 + 38670539 38670824 286 browser details YourSeq 104 6 149 149 86.8% 8 - 119094470 119094788 319 browser details YourSeq 103 1 143 149 86.8% 17 + 36242009 36242332 324 browser details YourSeq 102 1 149 149 87.7% 7 - 68792288 68792914 627 browser details YourSeq 101 22 149 149 90.5% 6 - 144777560 144777821 262 browser details YourSeq 99 1 149 149 89.7% 12 - 121959207 121959606 400 browser details YourSeq 99 40 149 149 95.5% 1 + 81524722 81524832 111 browser details YourSeq 98 1 149 149 90.3% 2 - 50150508 50150791 284 browser details YourSeq 98 1 149 149 90.3% 18 - 12613168 12613451 284 browser details YourSeq 97 22 135 149 94.6% 10 - 98515786 98516036 251 Top Hits on mirror (http://genomics01.arcan.stonybrook.edu/) ACTIONS QUERY SCORE START END QSIZE IDENTITY CHRO STRAND START END SPAN --------------------------------------------------------------------------------------------------- browser details YourSeq 149 1 149 149 100.0% 3 + 38840 38988 149 browser details YourSeq 131 1 149 149 94.0% 3 + 75370223 75370371 149 browser details YourSeq 96 21 145 149 89.6% 3 - 180780284 180780534 251 browser details YourSeq 94 1 149 149 85.7% 7 + 44009162 44009311 150 browser details YourSeq 90 19 127 149 92.7% 6_cox_hap1 + 2471579 2471826 248 browser details YourSeq 90 19 149 149 89.5% 16 + 87341011 87341427 417 browser details YourSeq 89 2 132 149 86.4% 16 + 56723646 56724089 444 browser details YourSeq 86 19 149 149 91.3% 1 - 21165569 21165835 267 browser details YourSeq 86 19 135 149 90.0% 7 + 67680926 67681175 250 browser details YourSeq 85 22 149 149 87.0% 9 - 104811323 104811585 263 browser details YourSeq 85 44 149 149 93.9% 19 - 2743238 2743773 536 browser details YourSeq 85 22 149 149 88.3% 1 - 70562010 70562275 266 browser details YourSeq 84 19 124 149 90.6% 7 - 26890375 26890616 242 browser details YourSeq 83 38 149 149 92.0% 17 + 40814748 40815021 274 The files under cgi-bin, gbdb, mysql are rsync-ed daily. One possible difference between UCSC and our mirror is the gfServer command. We start gfServer using default parameters: cd /gbdb/hg18/nib gfServer start genomics01 17781 *.nib & gfServer start genomics01 17780 -trans -mask *.nib & Do you have any suggestions regarding the different result? Regagrds, Jizu Zhi Stony Brook University From galt at soe.ucsc.edu Mon Nov 5 10:43:08 2007 From: galt at soe.ucsc.edu (Galt Barber) Date: Mon, 5 Nov 2007 10:43:08 -0800 (PST) Subject: [Genome-mirror] different BLAT result using local mirror In-Reply-To: <472F5C09.3070806@osa.sunysb.edu> References: <472F5C09.3070806@osa.sunysb.edu> Message-ID: We use -stepsize 5 with our untranslated BLAT server. http://hgwdev.cse.ucsc.edu/FAQ/FAQblat#blat5 gfServer (this is how the UCSC web-based blat servers are configured): * blat server (capable of PCR): gfServer start blatMachine portX -stepSize=5 -log=untrans.log database.2bit * translated blat server: gfServer start blatMachine portY -trans -mask -log=trans.log database.2bit -Galt On Mon, 5 Nov 2007, jizu wrote: > Hi gurus, > > We are having a strange problem with our local BLAT mirror. For example, > if submitting the following sequence to search against hg18 > > GGCCAGGCGCAGTCCCTTACACCTGTAATCTCAGCACTTTGGGAGGCTGA > GGCAGGAGGATCGCCTGAGCCTGGGAGATGAGGCTGCAGTGAGCTATGAC > AGCACCACTGCACTCCAGCCTGGGCAACAGAGAGAGACGCTGTCTCAAA > > Top Hits on UCSC site > > ACTIONS QUERY SCORE START END QSIZE IDENTITY CHRO > STRAND START END SPAN > --------------------------------------------------------------------------------------------------- > browser details YourSeq 149 1 149 149 100.0% 3 > + 38840 38988 149 > browser details YourSeq 131 1 149 149 94.0% 3 > + 75370223 75370371 149 > browser details YourSeq 106 2 149 149 86.4% 3 - > 186543951 186544231 281 > browser details YourSeq 106 1 149 149 88.0% 3 + > 150710511 150762774 52264 > browser details YourSeq 106 1 146 149 87.0% 19 > + 6067438 6067724 287 > browser details YourSeq 106 1 149 149 86.4% 11 > + 38670539 38670824 286 > browser details YourSeq 104 6 149 149 86.8% 8 - > 119094470 119094788 319 > browser details YourSeq 103 1 143 149 86.8% 17 > + 36242009 36242332 324 > browser details YourSeq 102 1 149 149 87.7% 7 > - 68792288 68792914 627 > browser details YourSeq 101 22 149 149 90.5% 6 - > 144777560 144777821 262 > browser details YourSeq 99 1 149 149 89.7% 12 - > 121959207 121959606 400 > browser details YourSeq 99 40 149 149 95.5% 1 > + 81524722 81524832 111 > browser details YourSeq 98 1 149 149 90.3% 2 > - 50150508 50150791 284 > browser details YourSeq 98 1 149 149 90.3% 18 > - 12613168 12613451 284 > browser details YourSeq 97 22 135 149 94.6% 10 > - 98515786 98516036 251 > > Top Hits on mirror (http://genomics01.arcan.stonybrook.edu/) > > ACTIONS QUERY SCORE START END QSIZE IDENTITY CHRO > STRAND START END SPAN > --------------------------------------------------------------------------------------------------- > browser details YourSeq 149 1 149 149 100.0% 3 > + 38840 38988 149 > browser details YourSeq 131 1 149 149 94.0% 3 > + 75370223 75370371 149 > browser details YourSeq 96 21 145 149 89.6% 3 - > 180780284 180780534 251 > browser details YourSeq 94 1 149 149 85.7% 7 > + 44009162 44009311 150 > browser details YourSeq 90 19 127 149 92.7% > 6_cox_hap1 + 2471579 2471826 248 > browser details YourSeq 90 19 149 149 89.5% 16 > + 87341011 87341427 417 > browser details YourSeq 89 2 132 149 86.4% 16 > + 56723646 56724089 444 > browser details YourSeq 86 19 149 149 91.3% 1 > - 21165569 21165835 267 > browser details YourSeq 86 19 135 149 90.0% 7 > + 67680926 67681175 250 > browser details YourSeq 85 22 149 149 87.0% 9 - > 104811323 104811585 263 > browser details YourSeq 85 44 149 149 93.9% 19 > - 2743238 2743773 536 > browser details YourSeq 85 22 149 149 88.3% 1 > - 70562010 70562275 266 > browser details YourSeq 84 19 124 149 90.6% 7 > - 26890375 26890616 242 > browser details YourSeq 83 38 149 149 92.0% 17 > + 40814748 40815021 274 > > The files under cgi-bin, gbdb, mysql are rsync-ed daily. One possible > difference between UCSC and our mirror is the gfServer command. We start > gfServer using default parameters: > cd /gbdb/hg18/nib > gfServer start genomics01 17781 *.nib & > gfServer start genomics01 17780 -trans -mask *.nib & > > Do you have any suggestions regarding the different result? > > Regagrds, > Jizu Zhi > Stony Brook University > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From xguo2 at bsd.uchicago.edu Mon Nov 5 12:07:55 2007 From: xguo2 at bsd.uchicago.edu (Xinghua Guo) Date: Mon, 5 Nov 2007 14:07:55 -0600 Subject: [Genome-mirror] My mirroring error: @ERROR: max connections (15) reached -- try again later Message-ID: Dear Sir, I couldn't do any rsync from Saturday. Would you please help me on this? Thanks a lot! Jerry Updating ce1 @ERROR: max connections (15) reached -- try again later rsync: connection unexpectedly closed (0 bytes received so far) [receiver] rsync error: error in rsync protocol data stream (code 12) at io.c(359) Updating ce2 @ERROR: max connections (15) reached -- try again later rsync: connection unexpectedly closed (0 bytes received so far) [receiver] rsync error: error in rsync protocol data stream (code 12) at io.c(359) This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal. Thank you. From rhead at soe.ucsc.edu Mon Nov 5 14:07:26 2007 From: rhead at soe.ucsc.edu (Brooke Rhead) Date: Mon, 05 Nov 2007 14:07:26 -0800 Subject: [Genome-mirror] My mirroring error: @ERROR: max connections (15) reached -- try again later In-Reply-To: References: Message-ID: <472F941E.30602@soe.ucsc.edu> Hello Jerry, This error is due to a large number of rsync connections to our hgdownload server. We limit the number of connections that can be made at any one time. If you keep trying, you should eventually get a connection. Here is a previous explanation of the same phenomenon: https://www.soe.ucsc.edu/pipermail/genome-mirror/2007-September/000591.html I hope this helps. -- Brooke Rhead UCSC Genome Bioinformatics Group Xinghua Guo wrote: > > > Dear Sir, > > > > I couldn't do any rsync from Saturday. > > Would you please help me on this? > > > > Thanks a lot! > > > > Jerry > > > > Updating ce1 > > > > @ERROR: max connections (15) reached -- try again later > > rsync: connection unexpectedly closed (0 bytes received so far) > [receiver] > > rsync error: error in rsync protocol data stream (code 12) at io.c(359) > > > > Updating ce2 > > > > @ERROR: max connections (15) reached -- try again later > > rsync: connection unexpectedly closed (0 bytes received so far) > [receiver] > > rsync error: error in rsync protocol data stream (code 12) at io.c(359) > > > This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal. Thank you. > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror From davide.cittaro at ifom-ieo-campus.it Thu Nov 8 01:31:18 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Thu, 8 Nov 2007 10:31:18 +0100 Subject: [Genome-mirror] genbank rsync? Message-ID: <045EF95A-5369-4409-8E31-ABEEDB0953D2@ifom-ieo-campus.it> Dear mirror people, I've noticed that our local mirror is downloading /gbdb/genbank since approx one week and it seems to be a ~70 Gb download :-( possibly replacing the old /gbdb/genbank. Is this normal? d /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro at ifom-ieo-campus.it */ From kayla at soe.ucsc.edu Thu Nov 8 15:14:18 2007 From: kayla at soe.ucsc.edu (Kayla Smith) Date: Thu, 08 Nov 2007 15:14:18 -0800 Subject: [Genome-mirror] genbank rsync? In-Reply-To: <045EF95A-5369-4409-8E31-ABEEDB0953D2@ifom-ieo-campus.it> References: <045EF95A-5369-4409-8E31-ABEEDB0953D2@ifom-ieo-campus.it> Message-ID: <4733984A.7050901@cse.ucsc.edu> Hello Davide, This is due to a recent GenBank update. We are updating the appropriate tables over the weekend. Sorry for the inconvenience. Please don't hesitate to contact us again if you require further assistance. Kayla Smith UCSC Genome Bioinformatics Group Davide Cittaro wrote: > Dear mirror people, > I've noticed that our local mirror is downloading /gbdb/genbank since > approx one week and it seems to be a ~70 Gb download :-( possibly > replacing the old /gbdb/genbank. > Is this normal? > > d > /* > Davide Cittaro > HPC and Bioinformatics Systems @ Informatics Core > > IFOM - Istituto FIRC di Oncologia Molecolare > via adamello, 16 > 20139 Milano > Italy > > tel.: +39(02)574303007 > e-mail: davide.cittaro at ifom-ieo-campus.it > */ > > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror From xguo2 at bsd.uchicago.edu Fri Nov 9 08:08:04 2007 From: xguo2 at bsd.uchicago.edu (Xinghua Guo) Date: Fri, 9 Nov 2007 10:08:04 -0600 Subject: [Genome-mirror] How to set archivecentral options in the hg.conf file In-Reply-To: <472FA5F4.8030002@soe.ucsc.edu> References: <472F941E.30602@soe.ucsc.edu> <472FA5F4.8030002@soe.ucsc.edu> Message-ID: Dear Brooke, I got this error message when I click "Batch Coordinate Conversion (liftOver)" In http://genomebrowser.uchicago.edu/util.html I just cannot imagine what's the format to set the option. I copied the hg.conf from mirror instruction, and made some change. What else configuration should I do? I've downloaded the databases, and the site works well. But every morning when I check the rsync log(from cron job), I am disappointed to find some errors, for one database or another, like: Updating monDom4 rsync: send_files failed to open "/monDom4/author.MYD" (in mysql): Permission denied (13) rsync: send_files failed to open "/monDom4/author.MYI" (in mysql): Permission denied (13) Updating panTro1 rsync: link_stat "/panTro1/." (in mysql) failed: Permission denied (13) rsync error: some files could not be transferred (code 23) at main.c(723) receiving file list ... 0 files to consider client: nothing to do: perhaps you need to specify some filenames or the --recursive option? Thanks for help! Jerry This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal. Thank you. From ann at soe.ucsc.edu Fri Nov 9 10:57:13 2007 From: ann at soe.ucsc.edu (Ann Zweig) Date: Fri, 09 Nov 2007 10:57:13 -0800 Subject: [Genome-mirror] How to set archivecentral options in the hg.conf file In-Reply-To: References: <472F941E.30602@soe.ucsc.edu> <472FA5F4.8030002@soe.ucsc.edu> Message-ID: <4734AD89.4000807@soe.ucsc.edu> Hello Jerry, The error that you are seeing when you try to use liftOver on your mirror site is a known issue. We are working on a code fix so that this issue is resolved for mirrors. There is something you can try (sorry, no guarantee that it will work) in your hg.conf file. Edit your hg.conf file so that the archivecentral and archivebackup settings point to the same place as your central settings. If you copied your hg.conf directly from the mirror instructions, you will have a block like this: central.db=hgcentral central.host=localhost central.user=username central.password=password central.domain=.mydomain.mytoplevel Add two blocks like this: archivecentral.db= archivecentral.host= archivecentral.user= archivecentral.password= archivebackup.db= archivebackup.host= archivebackup.user= archivebackup.password= As for your rsync problems, those are likely temporary glitches. Do they always fail on the same databases? How long have you been seeing these error messages? Regards, ---------- Ann Zweig UCSC Genome Bioinformatics Group http://genome.ucsc.edu Xinghua Guo wrote: > Dear Brooke, > > I got this error message when I click "Batch Coordinate Conversion > (liftOver)" > In http://genomebrowser.uchicago.edu/util.html > > I just cannot imagine what's the format to set the option. > > I copied the hg.conf from mirror instruction, and made some change. What > else configuration should I do? > > I've downloaded the databases, and the site works well. But every > morning when I check the rsync log(from cron job), I am disappointed to > find some errors, for one database or another, like: > > Updating monDom4 > > rsync: send_files failed to open "/monDom4/author.MYD" (in mysql): > Permission denied (13) > rsync: send_files failed to open "/monDom4/author.MYI" (in mysql): > Permission denied (13) > > Updating panTro1 > > rsync: link_stat "/panTro1/." (in mysql) failed: Permission denied (13) > rsync error: some files could not be transferred (code 23) at > main.c(723) > receiving file list ... > 0 files to consider > client: nothing to do: perhaps you need to specify some filenames or the > --recursive option? > > > Thanks for help! > > Jerry > > This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal. Thank you. > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror From ann at soe.ucsc.edu Fri Nov 9 11:39:33 2007 From: ann at soe.ucsc.edu (Ann Zweig) Date: Fri, 09 Nov 2007 11:39:33 -0800 Subject: [Genome-mirror] How to set archivecentral options in the hg.conf file In-Reply-To: <4734AD89.4000807@soe.ucsc.edu> References: <472F941E.30602@soe.ucsc.edu> <472FA5F4.8030002@soe.ucsc.edu> <4734AD89.4000807@soe.ucsc.edu> Message-ID: <4734B775.5050508@soe.ucsc.edu> Hello again, Jerry, One of our system admins has a suggestion as to why you might be seeing these rsync errors. "As far as I understand, this problem seems to happen when people are downloading files while we're updating the files themselves. I think there's a brief period of time that the updated files are not owned by mysql, and therefore you get a "Permission denied" when you try to access them. If you try again (even a few minutes later), it works." Regards, ---------- Ann Zweig UCSC Genome Bioinformatics Group http://genome.ucsc.edu Ann Zweig wrote: > Hello Jerry, > > The error that you are seeing when you try to use liftOver on your > mirror site is a known issue. We are working on a code fix so that this > issue is resolved for mirrors. > > There is something you can try (sorry, no guarantee that it will > work) in your hg.conf file. Edit your hg.conf file so that the > archivecentral and archivebackup settings point to the same place as > your central settings. > > If you copied your hg.conf directly from the mirror instructions, > you will have a block like this: > > central.db=hgcentral > central.host=localhost > central.user=username > central.password=password > central.domain=.mydomain.mytoplevel > > Add two blocks like this: > > archivecentral.db= > archivecentral.host= > archivecentral.user= > archivecentral.password= > > archivebackup.db= > archivebackup.host= > archivebackup.user= > archivebackup.password= > > As for your rsync problems, those are likely temporary glitches. Do > they always fail on the same databases? How long have you been seeing > these error messages? > > > Regards, > > ---------- > Ann Zweig > UCSC Genome Bioinformatics Group > http://genome.ucsc.edu > > > > > Xinghua Guo wrote: >> Dear Brooke, >> >> I got this error message when I click "Batch Coordinate Conversion >> (liftOver)" In http://genomebrowser.uchicago.edu/util.html >> >> I just cannot imagine what's the format to set the option. >> >> I copied the hg.conf from mirror instruction, and made some change. What >> else configuration should I do? >> >> I've downloaded the databases, and the site works well. But every >> morning when I check the rsync log(from cron job), I am disappointed to >> find some errors, for one database or another, like: >> >> Updating monDom4 >> >> rsync: send_files failed to open "/monDom4/author.MYD" (in mysql): >> Permission denied (13) >> rsync: send_files failed to open "/monDom4/author.MYI" (in mysql): >> Permission denied (13) >> >> Updating panTro1 >> >> rsync: link_stat "/panTro1/." (in mysql) failed: Permission denied (13) >> rsync error: some files could not be transferred (code 23) at >> main.c(723) >> receiving file list ... 0 files to consider >> client: nothing to do: perhaps you need to specify some filenames or the >> --recursive option? >> >> >> Thanks for help! >> >> Jerry >> >> This email is intended only for the use of the individual or entity to >> which it is addressed and may contain information that is privileged >> and confidential. If the reader of this email message is not the >> intended recipient, you are hereby notified that any dissemination, >> distribution, or copying of this communication is prohibited. If you >> have received this email in error, please notify the sender and >> destroy/delete all copies of the transmittal. Thank you. >> >> _______________________________________________ >> Genome-mirror mailing list >> Genome-mirror at soe.ucsc.edu >> http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From drs20 at cam.ac.uk Tue Nov 13 02:36:28 2007 From: drs20 at cam.ac.uk (david sargan) Date: Tue, 13 Nov 2007 10:36:28 +0000 Subject: [Genome-mirror] error messages on links and new requests Message-ID: When I attempt to use links from many of your browser pages to additional information, or to make a new request for display into the position/ search box, I am getting the following message most of the time "External file /gbdb/genbank/./data/processed/genbank.160.0/full/est.dn.fa cannot be opened or has wrong size. Old size 689586873, new size -1, error No such file or directory". This occurs with either Firefox or Safari as browser. Is there anything I can do? -- David R Sargan MA PhD Senior Lecturer in Molecular Pathology Director of Graduate Studies Department of Veterinary Medicine University of Cambridge Madingley Road Cambridge CB3 0ES UK. Tel: +44-1223-337686 Fax: +44-1223-337610 From ann at soe.ucsc.edu Tue Nov 13 09:39:02 2007 From: ann at soe.ucsc.edu (Ann Zweig) Date: Tue, 13 Nov 2007 09:39:02 -0800 Subject: [Genome-mirror] error messages on links and new requests In-Reply-To: References: Message-ID: <4739E136.9080506@soe.ucsc.edu> Hello David, Is this happening on your own mirror site, or on our public site (genome.ucsc.edu)? If it is happening on your mirror site, then I have a likely explanation. We recently did an update to the GenBank files and tracks. If you haven't downloaded the latest assembly databases (especially the GenBank-related tables like gbExtFile), you need to do that. That should clear up the problem. Regards, ---------- Ann Zweig UCSC Genome Bioinformatics Group http://genome.ucsc.edu david sargan wrote: > When I attempt to use links from many of your browser pages to > additional information, or to make a new request for display into the > position/ search box, I am getting the following message most of the > time > "External file > /gbdb/genbank/./data/processed/genbank.160.0/full/est.dn.fa cannot be > opened or has wrong size. Old size 689586873, new size -1, error No > such file or directory". > This occurs with either Firefox or Safari as browser. Is there > anything I can do? > From ann at soe.ucsc.edu Tue Nov 13 17:22:39 2007 From: ann at soe.ucsc.edu (Ann Zweig) Date: Tue, 13 Nov 2007 17:22:39 -0800 Subject: [Genome-mirror] GenBank update notice ... a little late Message-ID: <473A4DDF.7010906@soe.ucsc.edu> Hello Mirror Sites, In case you hadn't already noticed, last week we made a major update to the GenBank tables in most assembly databases. GenBank released a new update (version 1.160) a few weeks ago. We responded by updating the GenBank tables in all assembly databases. This update was extensive -- it took us nearly a week to download and update all tables! I can imagine that it had a similar impact on your rsync process. In the future, we will send you an email warning you that a deluge of new tables are on their way. Sorry for any inconvenience this may have caused to you. Regards, ---------- Ann Zweig UCSC Genome Bioinformatics Group http://genome.ucsc.edu From davide.cittaro at ifom-ieo-campus.it Fri Nov 16 02:48:09 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Fri, 16 Nov 2007 11:48:09 +0100 Subject: [Genome-mirror] new genome release Message-ID: <3C6A5CAB-451E-496D-9980-A5DD591FBBCA@ifom-ieo-campus.it> According to http://genome-test.cse.ucsc.edu/builds/versions.html we are now at v171, but it seems I can't download it from http://hgdownload.cse.ucsc.edu/admin/jksrc.v171.zip I can only get v170, and it seems to me that even this one has not been announced (but I may be wrong...) d /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro at ifom-ieo-campus.it */ From bl246 at hermes.cam.ac.uk Fri Nov 16 05:21:21 2007 From: bl246 at hermes.cam.ac.uk (B. Liu) Date: Fri, 16 Nov 2007 13:21:21 +0000 (GMT) Subject: [Genome-mirror] Zoom Function Problem Message-ID: Dear All, We found a problem related with zoom function on our mirror site. When I choose a position, say chr8:12000000-123000000, hide some tracks for example SNPs, and enable my local tracks, then I click zoom1.5x, then the browser goes back to the default tracks, loses the tracks I set few minutes ago, i.e. the hidden SNPs track comes back, and enabled local tracks are all disappeared. It is fine on the UCSC site, I could figure out where goes wrong, help please. Thanks Bin From ann at soe.ucsc.edu Fri Nov 16 09:12:53 2007 From: ann at soe.ucsc.edu (Ann Zweig) Date: Fri, 16 Nov 2007 09:12:53 -0800 Subject: [Genome-mirror] new genome release In-Reply-To: <3C6A5CAB-451E-496D-9980-A5DD591FBBCA@ifom-ieo-campus.it> References: <3C6A5CAB-451E-496D-9980-A5DD591FBBCA@ifom-ieo-campus.it> Message-ID: <473DCF95.7080309@cse.ucsc.edu> Hello Davide, We are scheduled to release v171 later today. Regards, ---------- Ann Zweig UCSC Genome Bioinformatics Group http://genome.ucsc.edu Davide Cittaro wrote: > According to http://genome-test.cse.ucsc.edu/builds/versions.html we > are now at v171, but it seems I can't download it from > http://hgdownload.cse.ucsc.edu/admin/jksrc.v171.zip > I can only get v170, and it seems to me that even this one has not > been announced (but I may be wrong...) > > d > > /* > Davide Cittaro > HPC and Bioinformatics Systems @ Informatics Core > > IFOM - Istituto FIRC di Oncologia Molecolare > via adamello, 16 > 20139 Milano > Italy > > tel.: +39(02)574303007 > e-mail: davide.cittaro at ifom-ieo-campus.it > */ > > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror From hiram at soe.ucsc.edu Fri Nov 16 09:59:53 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Fri, 16 Nov 2007 09:59:53 -0800 Subject: [Genome-mirror] Zoom Function Problem In-Reply-To: References: Message-ID: <473DDA99.30802@soe.ucsc.edu> Good Morning Bin: Your browser hg.conf is not configured properly to allow the browser to construct correct cookie names. Please note the discussion of the setting central.domain in your hg.conf file, and the required write permissions to your MySQL central.db database as specified in your hg.conf file. From the source code example file in src/product/ex.hg.conf: > # Be sure this user has UPDATE AND INSERT privs for hgcentral > # The central.domain will allow the browser cookie-cart > # function to work. Set it to your domain. It can be > # central.domain=local > # when the same machine is used for MySQL and the Apache > # WEB server. Depends upon how your DNS is configured. > # > central.user=readwrite > central.password=update > central.domain=.cam.ac.uk The domain setting depends upon how your machine is set up and what it finds when it asks its network what its domain name is and how specific you want your cookies to be identified. This domain name is used as the cookie name back to a user's WEB browser. --Hiram B. Liu wrote: > Dear All, > > We found a problem related with zoom function on our mirror site. When I choose a position, say chr8:12000000-123000000, hide some tracks for example SNPs, and enable my local tracks, then I click zoom1.5x, then the browser goes back to the default tracks, loses the tracks I set few minutes ago, i.e. the hidden SNPs track comes back, and enabled local tracks are all disappeared. It is fine on the UCSC site, I could figure out where goes wrong, help please. > > Thanks > > Bin From kayla at soe.ucsc.edu Fri Nov 16 15:09:01 2007 From: kayla at soe.ucsc.edu (Kayla Smith) Date: Fri, 16 Nov 2007 15:09:01 -0800 Subject: [Genome-mirror] new Conservation track for mm9 Message-ID: <473E230D.2020700@cse.ucsc.edu> Hello Mirror Sites, Please be warned that we will soon be placing a large amount of data (about 55 GB) on the hgnfs1 (/gbdb) server. This is to support the new upcoming 30-way Conservation track on mm9. Here is a list of the new files: /gbdb/mm9/multiz30way/anno/maf/*.maf (49G) /gbdb/mm9/multiz30way/*.wib (5.4G) Please let us know if you have any questions or concerns. Thanks, Kayla Smith UCSC Genome Bioinformatics Group From aamp at ucsc.edu Fri Nov 16 15:49:56 2007 From: aamp at ucsc.edu (Andy Pohl) Date: Fri, 16 Nov 2007 15:49:56 -0800 Subject: [Genome-mirror] v171 Genome Browser Available Message-ID: <9fa943760711161549k5cb8e520te391786111b0f32f@mail.gmail.com> Hello, The v171 source is at: http://hgdownload.cse.ucsc.edu/admin/jksrc.zip or labeled with source number: http://hgdownload.cse.ucsc.edu/admin/jksrc.v171.zip cartDump cartReset das hgc hgBlat hgConvert hgCustom hgEncodeDataVersions hgGateway hgGene hgGenome hgLiftOver hgNear hgPcr hgSession hgTables hgText hgTracks hgTrackUi hgVisiGene mkEncodeFrameset pbGateway pbGlobal pbTracks phyloGif /usr/local/apache/cgi-bin/all.joiner /usr/local/apache/cgi-bin/hgNearData/* /usr/local/apache/cgi-bin/hgGeneData/* /usr/local/apache/cgi-bin/hgcData/* /usr/local/apache/cgi-bin/hgCgiData/* /usr/local/apache/cgi-bin/loader/* /usr/local/apache/cgi-bin/visiGeneData/* Contents of v171 may be seen at: http://genome-test.cse.ucsc.edu/builds/versions.html Please check both v171-preview and v171-final to see all the code and data changes in this release. Have a nice weekend! Andy Pohl UCSC CBSE From bl246 at hermes.cam.ac.uk Mon Nov 19 03:16:00 2007 From: bl246 at hermes.cam.ac.uk (B. Liu) Date: Mon, 19 Nov 2007 11:16:00 +0000 (GMT) Subject: [Genome-mirror] Zoom Function Problem In-Reply-To: <473DDA99.30802@soe.ucsc.edu> References: <473DDA99.30802@soe.ucsc.edu> Message-ID: Dear Hiram, Thanks for the direction, it turns out that I got a https address lately, and I didn't change the https address in the httpd.conf file and the central.domain in the hg.conf file, the whole web site still functions apart from the cookies. Cheers Bin On Fri, 16 Nov 2007, Hiram Clawson wrote: > Good Morning Bin: > > Your browser hg.conf is not configured properly to allow the browser to > construct correct cookie names. > > Please note the discussion of the setting central.domain in your hg.conf > file, > and the required write permissions to your MySQL central.db database as > specified in your hg.conf file. From the source code example file in > src/product/ex.hg.conf: > >> # Be sure this user has UPDATE AND INSERT privs for hgcentral >> # The central.domain will allow the browser cookie-cart >> # function to work. Set it to your domain. It can be >> # central.domain=local >> # when the same machine is used for MySQL and the Apache >> # WEB server. Depends upon how your DNS is configured. >> # >> central.user=readwrite >> central.password=update >> central.domain=.cam.ac.uk > > The domain setting depends upon how your machine is set up and > what it finds when it asks its network what its domain name is and > how specific you want your cookies to be identified. This domain name > is used as the cookie name back to a user's WEB browser. > > --Hiram From eschen at bcm.tmc.edu Mon Nov 19 09:05:58 2007 From: eschen at bcm.tmc.edu (Chen, Edward S.) Date: Mon, 19 Nov 2007 11:05:58 -0600 Subject: [Genome-mirror] Realistic sizes for MySQL tables? Message-ID: I am in the process of setting up a UCSC mirror, and was looking for a realistic estimate of the size of all the MySQL tables (ie: how much disk will I need /var/lib/mysql on my MySQL machine) Following the documentation, I was expecting to need 400-500GB, but after rsyncing hg18 alone, 99GB of the allocated space is already gone, and I know hg16 and the mm* are each close to that in size. Thank you very much, Ed Chen, Ph.D. Baylor College of Medicine Bioinformatics Research Center From Janet_Yang at dfci.harvard.edu Mon Nov 19 08:53:19 2007 From: Janet_Yang at dfci.harvard.edu (Yang, Janet Q.) Date: Mon, 19 Nov 2007 11:53:19 -0500 Subject: [Genome-mirror] About procedure for creating a mirror site for UCSC genome Browser Message-ID: Dear Administrator, I am janet yang, form Dana Farber Cancer Institute, and now I am following the UCSC procedure for creating a mirror site for UCSC genome Browser in our Lab website. I am following the webpage: http://genome.ucsc.edu/admin/mirror.html In Step-by-Step Details: I am fine with steps 1 to 4. Step 5: Set up database (400 Gb MySQL data), On your sheet: you state: "Use the following table to identify the freeze date ($FREEZEDATE) and the database version ($DBVERSION) for each genome assembly " My problems: Do I need to make $FREEZEDATE?, $DBVERSION here? How do I do it? Can you provide a intact step for create $FREEZEDATE? $DBVERSION, or I do not need to do any thing here until 5a, and 5b steps? a. You state: After connecting to the MySQL server, create a database called "$DBVERSION" corresponding to $FREEZEDATE in the table above. I did this step used your command: mysql> create database $DBVERSION; My problems: How do I correspond to $FREEZEDATE table? b. You state: Create tables for the "$DBVERSION" database: i. You state: "Move to $WEBROOT/goldenPath/$FREEZEDATE/database/ directory. All the files for creating the "$DBVERSION" database are in this directory" My problem is: I only have $WEBROOT/goldenPath, how can I create $FREEZEDATE/database directory? just manually do mkdir $FREEZEDATE under $WEBROOT/goldenPath? Can you provide intact command for how to create /$FREEZEDATE/database/ directory? ii.You state: Create and load data into the tables: I selected the way "Optionally, you can download the MySQL tables via rsync directly into your /var/lib/mysql/ database hierarchy" I type this command (is it correct ?): rsync -avzP rsync://hgdownload.cse.ucsc.edu/mysql/ /var/lib/mysql/ Overall I do have problems to understand this whole step5-"Set up database". Can you provide detail steps to help me down load your database? My mysql data in /var/lib/mysql, my mysql database username=root, password=jayang, and hostname=miron.dfci.harvard.edu. Thank you very much! Janet Yang Bioinformatic Analyst Dana Farber Cancer Institute Boston, Mass 617- 632 - 4283 (office) 978 - 886 - 2638 (cell) The information transmitted in this electronic communication is intended only for the person or entity to whom it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this information in error, please contact the Compliance HelpLine at 800-856-1983 and properly dispose of this information. From zhangc at cshl.edu Sat Nov 17 08:19:39 2007 From: zhangc at cshl.edu (Zhang, Chaolin) Date: Sat, 17 Nov 2007 11:19:39 -0500 Subject: [Genome-mirror] multiz28way Message-ID: Hi, I am updating my local UCSC genome browser mirror to include multiz28way conservation. Before that I have already the data of multiz17way conservation. Now, I have downloaded the following files: /gbdb/hg18/multiz28way multiz28wayFrames.frm multiz28way.frm multiz28waySummary.frm phastCons28way.frm phastCons28wayPlacMammal.frm multiz28wayFrames.MYD multiz28way.MYD multiz28waySummary.MYD phastCons28way.MYD phastCons28wayPlacMammal.MYD multiz28wayFrames.MYI multiz28way.MYI multiz28waySummary.MYI phastCons28way.MYI phastCons28wayPlacMammal.MYI The trackDb.ra for hg18 is also fine. But still, there are some problems. When I set the display mode of the 28way conservation to dense or squish, the browser works fine, but when I set the mode to pack or full, it report an error as Database inconsistency - no external file with id 27023304 It seems that some database files are still missing. Could you have a look? Thanks! Chaolin From galt at soe.ucsc.edu Mon Nov 19 10:00:00 2007 From: galt at soe.ucsc.edu (Galt Barber) Date: Mon, 19 Nov 2007 10:00:00 -0800 (PST) Subject: [Genome-mirror] Realistic sizes for MySQL tables? In-Reply-To: References: Message-ID: I think we are up to about 750GB by now. -Galt On Mon, 19 Nov 2007, Chen, Edward S. wrote: > I am in the process of setting up a UCSC mirror, and was looking for a realistic estimate of the size of all the MySQL tables (ie: how much disk will I need /var/lib/mysql on my MySQL machine) Following the documentation, I was expecting to need 400-500GB, but after rsyncing hg18 alone, 99GB of > the allocated space is already gone, and I know hg16 and the mm* are each close to that in size. > > > > Thank you very much, > > Ed Chen, Ph.D. > > Baylor College of Medicine > > Bioinformatics Research Center > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From hiram at soe.ucsc.edu Mon Nov 19 10:16:15 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Mon, 19 Nov 2007 10:16:15 -0800 Subject: [Genome-mirror] Realistic sizes for MySQL tables? In-Reply-To: References: Message-ID: <4741D2EF.1050201@soe.ucsc.edu> Good Morning Dr. Chen: You can test the size of an rsync download with the -n (== -dry-run) option and --stats: $ rsync -na --stats rsync://hgdownload.cse.ucsc.edu/mysql/ # Number of files: 101720 # total size is 1276098293742 That is 1.2 Tb for the mysql tables. For the /gbdb/ directory of files: $ rsync -na --stats rsync://hgdownload.cse.ucsc.edu/gbdb/ # Number of files: 123794 # total size is 684190240905 which is 637 Gb. To verify a single database size, for example hg18: rsync -na --progress --stats rsync://hgdownload.cse.ucsc.edu/mysql/hg18/ rsync -na --stats rsync://hgdownload.cse.ucsc.edu/gbdb/hg18/ Shows 97 Gb for MySQL tables, and 140 Gb for /gbdb/hg18/ directories. Although there are other files in /gbdb/ that are required for all assemblies in the browser to function. For example, /gbdb/genbank/ at 28 Gb. --Hiram Chen, Edward S. wrote: > I am in the process of setting up a UCSC mirror, and was looking for a realistic estimate of the size of all the MySQL tables (ie: how much disk will I need /var/lib/mysql on my MySQL machine) Following the documentation, I was expecting to need 400-500GB, but after rsyncing hg18 alone, 99GB of > the allocated space is already gone, and I know hg16 and the mm* are each close to that in size. > > > > Thank you very much, > > Ed Chen, Ph.D. > > Baylor College of Medicine > > Bioinformatics Research Center > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From hiram at soe.ucsc.edu Mon Nov 19 10:56:25 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Mon, 19 Nov 2007 10:56:25 -0800 Subject: [Genome-mirror] About procedure for creating a mirror site for UCSC genome Browser In-Reply-To: References: Message-ID: <4741DC59.40107@soe.ucsc.edu> Good Morning Janet: There are two methods possible to load a database in the genome browser. You can either download the text sql files as mentioned in the section three of the mirror document, or the actual database binary files from the 'mysql' directory of the rsync. rsync -na --stats rsync://hgdownload.cse.ucsc.edu/mysql/ The FREEZEDATE and DBVERSION table discussion merely relates a directory name in the goldenPath/ directory with the actual database name. All new genome assemblies have the same directory name in goldenPath as is their database names. The directories in goldenPath are populated via the rsync of the text files as mentioned in section 3 of the document. The databases are created from those text files as mentioned in section 5. There are some additional notes in the source tree to assist with the setup. Download the source tree: http://genome.ucsc.edu/admin/cvs.html and review the README text files in kent/src/product/README* It may be helpful to review notes from other users who have set up mirror sites, for example: http://genomewiki.ucsc.edu/index.php/Minimal_Browser_Installation http://genomewiki.ucsc.edu/index.php/Browser_Installation Please respond to the genome-mirror email list with any additional queries. --Hiram Yang, Janet Q. wrote: > Dear Administrator, > > I am janet yang, form Dana Farber Cancer Institute, and now I am following the > UCSC procedure for creating a mirror site for UCSC genome Browser in our Lab > website. > > I am following the webpage: http://genome.ucsc.edu/admin/mirror.html > > In Step-by-Step Details: > > I am fine with steps 1 to 4. > > Step 5: Set up database > (400 Gb MySQL data), > > On your sheet: you state: "Use the following table to identify the freeze date > ($FREEZEDATE) and the database version ($DBVERSION) for each genome assembly > " > > My problems: Do I need to make $FREEZEDATE?, $DBVERSION here? How do I do > it? Can you provide a intact step for create $FREEZEDATE? $DBVERSION, or I do > not need to do any thing here until 5a, and 5b steps? > > a. You state: After connecting to the MySQL server, create a database > called "$DBVERSION" corresponding to $FREEZEDATE in the table > above. > > I did this step used your command: mysql> create database $DBVERSION; > My problems: How do I correspond to $FREEZEDATE table? > > b. You state: Create tables for the "$DBVERSION" database: > > i. You state: "Move to $WEBROOT/goldenPath/$FREEZEDATE/database/ directory. > > All the files for creating the "$DBVERSION" database are in > this directory" > > My problem is: I only have $WEBROOT/goldenPath, how can I create > $FREEZEDATE/database directory? just manually do mkdir $FREEZEDATE under > $WEBROOT/goldenPath? > > Can you provide intact command for how to create > /$FREEZEDATE/database/ directory? > > ii.You state: Create and load data into the tables: > > I selected the way "Optionally, you can download the MySQL tables via rsync > directly into your /var/lib/mysql/ database hierarchy" > > I type this command (is it correct ?): rsync -avzP > rsync://hgdownload.cse.ucsc.edu/mysql/ /var/lib/mysql/ > > Overall I do have problems to understand this whole step5-"Set up database". > Can you provide detail steps to help me down load your database? > > My mysql data in /var/lib/mysql, my mysql database username=root, > password=jayang, and hostname=miron.dfci.harvard.edu. > > Thank you very much! > > Janet Yang > > Bioinformatic Analyst > > Dana Farber Cancer Institute > > Boston, Mass > > 617- 632 - 4283 (office) > > 978 - 886 - 2638 (cell) From kuhn at soe.ucsc.edu Mon Nov 19 14:03:21 2007 From: kuhn at soe.ucsc.edu (Robert Kuhn) Date: Mon, 19 Nov 2007 14:03:21 -0800 Subject: [Genome-mirror] Realistic sizes for MySQL tables? Message-ID: <200711192203.OAA15423@moondance.cse.ucsc.edu> we are currently at 832 Gb in MySQL for all genome assemblies, not including proteins and uniProt. 1216 Gb if you include the indices (which you should). > From genome-mirror-bounces at soe.ucsc.edu Mon Nov 19 10:03:50 2007 > To: "Chen, Edward S." > Cc: genome-mirror at soe.ucsc.edu > Subject: Re: [Genome-mirror] Realistic sizes for MySQL tables? > > > I think we are up to about 750GB by now. > > -Galt > > > On Mon, 19 Nov 2007, Chen, Edward S. wrote: > > > I am in the process of setting up a UCSC mirror, and was looking for a realistic estimate of the size of all the MySQL tables (ie: how much disk will I need /var/lib/mysql on my MySQL machine) Following the documentation, I was expecting to need 400-500GB, but after rsyncing hg18 alone, 99GB of > > the allocated space is already gone, and I know hg16 and the mm* are each close to that in size. > > > > > > > > Thank you very much, > > > > Ed Chen, Ph.D. > > > > Baylor College of Medicine > > > > Bioinformatics Research Center > > > > _______________________________________________ > > Genome-mirror mailing list > > Genome-mirror at soe.ucsc.edu > > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From kayla at soe.ucsc.edu Mon Nov 19 17:31:35 2007 From: kayla at soe.ucsc.edu (Kayla Smith) Date: Mon, 19 Nov 2007 17:31:35 -0800 (PST) Subject: [Genome-mirror] multiz28way In-Reply-To: References: Message-ID: Chaolin, One piece of the puzzle for getting this working is to update your hg18.extFile table. Check that it has 110 rows in it, some of the rows in this table mention the 28 way conservation files. The error you are seeing "no external file with id 27023304" corresponds to the file /gbdb/hg18/multiz28way/anno/maf/chr17.maf Try checking to make sure that file exists. I hope this is helpful to you. Please don't hesitate to write back if we can be of further assistance or if this don't solve the problem. Kayla Smith UCSC Genome Bioinformatics Group On Sat, 17 Nov 2007, Zhang, Chaolin wrote: > Hi, > > I am updating my local UCSC genome browser mirror to include multiz28way conservation. Before that I have already the data of multiz17way conservation. Now, I have downloaded the following files: > > /gbdb/hg18/multiz28way > multiz28wayFrames.frm multiz28way.frm multiz28waySummary.frm phastCons28way.frm phastCons28wayPlacMammal.frm > multiz28wayFrames.MYD multiz28way.MYD multiz28waySummary.MYD phastCons28way.MYD phastCons28wayPlacMammal.MYD > multiz28wayFrames.MYI multiz28way.MYI multiz28waySummary.MYI phastCons28way.MYI phastCons28wayPlacMammal.MYI > > The trackDb.ra for hg18 is also fine. But still, there are some problems. When I set the display mode of the 28way conservation to dense or squish, the browser works fine, but when I set the mode to pack or full, it report an error as > > Database inconsistency - no external file with id 27023304 > > It seems that some database files are still missing. Could you have a look? Thanks! > > Chaolin > > > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From donnak at soe.ucsc.edu Mon Nov 19 18:46:19 2007 From: donnak at soe.ucsc.edu (Donna Karolchik) Date: Mon, 19 Nov 2007 18:46:19 -0800 Subject: [Genome-mirror] downloads server system maintenance on 24 Nov. Message-ID: <01fa01c82b20$31403b20$13a8a8c0@donnakLT> The Genome Browser downloads server -- hgdownload.cse.ucsc.edu -- will be down for system maintenance and upgrades on Saturday 24 Nov. from approximately 10:00 am to 1:00 pm PST. Please plan your access accordingly. Our apologies for any inconvenience this may cause. -Donna ----------------------------------- Donna Karolchik Genome Browser Project Manager UCSC Genome Bioinformatics Group http://genome.ucsc.edu From zhangc at cshl.edu Tue Nov 20 06:11:39 2007 From: zhangc at cshl.edu (Zhang, Chaolin) Date: Tue, 20 Nov 2007 09:11:39 -0500 Subject: [Genome-mirror] multiz28way In-Reply-To: References: Message-ID: Hi Kayla, Thanks for the help. I have downloaded the hg18.extFile and it works now. Chaolin -----Original Message----- From: Kayla Smith [mailto:kayla at soe.ucsc.edu] Sent: Monday, November 19, 2007 8:32 PM To: Zhang, Chaolin Cc: genome-mirror at soe.ucsc.edu Subject: Re: [Genome-mirror] multiz28way Chaolin, One piece of the puzzle for getting this working is to update your hg18.extFile table. Check that it has 110 rows in it, some of the rows in this table mention the 28 way conservation files. The error you are seeing "no external file with id 27023304" corresponds to the file /gbdb/hg18/multiz28way/anno/maf/chr17.maf Try checking to make sure that file exists. I hope this is helpful to you. Please don't hesitate to write back if we can be of further assistance or if this don't solve the problem. Kayla Smith UCSC Genome Bioinformatics Group On Sat, 17 Nov 2007, Zhang, Chaolin wrote: > Hi, > > I am updating my local UCSC genome browser mirror to include multiz28way conservation. Before that I have already the data of multiz17way conservation. Now, I have downloaded the following files: > > /gbdb/hg18/multiz28way > multiz28wayFrames.frm multiz28way.frm multiz28waySummary.frm phastCons28way.frm phastCons28wayPlacMammal.frm > multiz28wayFrames.MYD multiz28way.MYD multiz28waySummary.MYD phastCons28way.MYD phastCons28wayPlacMammal.MYD > multiz28wayFrames.MYI multiz28way.MYI multiz28waySummary.MYI phastCons28way.MYI phastCons28wayPlacMammal.MYI > > The trackDb.ra for hg18 is also fine. But still, there are some problems. When I set the display mode of the 28way conservation to dense or squish, the browser works fine, but when I set the mode to pack or full, it report an error as > > Database inconsistency - no external file with id 27023304 > > It seems that some database files are still missing. Could you have a look? Thanks! > > Chaolin > > > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From davide.cittaro at ifom-ieo-campus.it Mon Nov 26 01:11:25 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Mon, 26 Nov 2007 10:11:25 +0100 Subject: [Genome-mirror] one more question about genbank and refseq Message-ID: <193EC060-46D7-494B-8079-8138EA5E80E8@ifom-ieo-campus.it> Dear team, How can I know which genbank/refseq version I'm using in each database I have? Is this info somewhere in mysql tables so that I can have it with a query? d /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro at ifom-ieo-campus.it */ From hiram at soe.ucsc.edu Mon Nov 26 09:11:44 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Mon, 26 Nov 2007 09:11:44 -0800 Subject: [Genome-mirror] one more question about genbank and refseq In-Reply-To: <193EC060-46D7-494B-8079-8138EA5E80E8@ifom-ieo-campus.it> References: <193EC060-46D7-494B-8079-8138EA5E80E8@ifom-ieo-campus.it> Message-ID: <474AFE50.9070605@soe.ucsc.edu> Good Morning Davide: I don't know if this information is in a MySQL table, but you can see what data is in use by taking a look at the directory: /gbdb/genbank/data/processed/ drwxr-xr-x 38 genbank genbank 1024 Nov 26 04:20 genbank.162.0 drwxr-xr-x 51 genbank genbank 1536 Nov 6 07:20 refseq.25 drwxr-xr-x 15 genbank genbank 512 Nov 26 04:20 refseq.26 --Hiram Davide Cittaro wrote: > Dear team, > How can I know which genbank/refseq version I'm using in each database > I have? Is this info somewhere in mysql tables so that I can have it > with a query? > > d > /* > Davide Cittaro > HPC and Bioinformatics Systems @ Informatics Core > > IFOM - Istituto FIRC di Oncologia Molecolare > via adamello, 16 > 20139 Milano > Italy > > tel.: +39(02)574303007 > e-mail: davide.cittaro at ifom-ieo-campus.it > */ > > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From davide.cittaro at ifom-ieo-campus.it Wed Nov 28 00:40:35 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Wed, 28 Nov 2007 09:40:35 +0100 Subject: [Genome-mirror] one more question about genbank and refseq In-Reply-To: <474AFE50.9070605@soe.ucsc.edu> References: <193EC060-46D7-494B-8079-8138EA5E80E8@ifom-ieo-campus.it> <474AFE50.9070605@soe.ucsc.edu> Message-ID: <71381681-69C1-4869-976B-A0F5A4E80627@ifom-ieo-campus.it> On Nov 26, 2007, at 6:11 PM, Hiram Clawson wrote: > Good Morning Davide: > > I don't know if this information is in a MySQL table, but you > can see what data is in use by taking a look at the directory: > /gbdb/genbank/data/processed/ > > drwxr-xr-x 38 genbank genbank 1024 Nov 26 04:20 genbank.162.0 > drwxr-xr-x 51 genbank genbank 1536 Nov 6 07:20 refseq.25 > drwxr-xr-x 15 genbank genbank 512 Nov 26 04:20 refseq.26 My directory list gives this: drwxr-xr-x 65 root root 1536 Nov 16 12:54 genbank.158.0 drwxr-xr-x 62 root root 1536 Nov 15 12:53 genbank.159.0 drwxr-xr-x 121 root root 2560 Nov 14 12:53 genbank.160.0 drwxr-xr-x 73 root root 1536 Nov 13 12:49 genbank.161.0 drwxr-xr-x 39 root root 1024 Nov 27 11:20 genbank.162.0 drwxr-xr-x 56 root root 1536 Nov 12 12:47 refseq.21 drwxr-xr-x 57 root root 1536 Nov 2 18:10 refseq.22 drwxr-xr-x 7 root root 1536 Nov 27 21:30 refseq.23 drwxr-xr-x 51 root root 1536 Nov 6 16:20 refseq.25 drwxr-xr-x 15 root root 512 Nov 26 13:20 refseq.26 I need to eliminate some of them to free some space, so I would like to know how can I decide if a version is useful or not for my mirror subset. Is the list you posted the one for the online UCSC genome browser? d /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro at ifom-ieo-campus.it */ From davide.cittaro at ifom-ieo-campus.it Wed Nov 28 01:43:21 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Wed, 28 Nov 2007 10:43:21 +0100 Subject: [Genome-mirror] hgCustom buffer overflow Message-ID: Dear GB people, we have found that hgCustom cgi executable goes into a buffer overflow if one tries to upload a track with a name longer than 52 characters. We have tried this with bed tracks. You can take a look to your apache error log, since we have tested also on your public web site, there should be some lines like this: [/cgi-bin/hgCustom] buffer overflow, size 64, format: %s_%s, referer: http://genome.ifom-ieo-campus.it/cgi-bin/hgCustom?hgsid=11331 with a different hostname/hgsid, of course. We haven't tried to exploit this (yet ^__^). Cheers d /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro at ifom-ieo-campus.it */ From rhead at soe.ucsc.edu Wed Nov 28 12:30:52 2007 From: rhead at soe.ucsc.edu (Brooke Rhead) Date: Wed, 28 Nov 2007 12:30:52 -0800 Subject: [Genome-mirror] one more question about genbank and refseq In-Reply-To: <71381681-69C1-4869-976B-A0F5A4E80627@ifom-ieo-campus.it> References: <193EC060-46D7-494B-8079-8138EA5E80E8@ifom-ieo-campus.it> <474AFE50.9070605@soe.ucsc.edu> <71381681-69C1-4869-976B-A0F5A4E80627@ifom-ieo-campus.it> Message-ID: <474DCFFC.7020104@soe.ucsc.edu> Hello Davide, Yes, the list Hiram posted: genbank.162.0 refseq.25 refseq.26 is the current set of directories on the public UCSC Genome Browser site. -- Brooke Rhead UCSC Genome Bioinformatics Group Davide Cittaro wrote: > On Nov 26, 2007, at 6:11 PM, Hiram Clawson wrote: > >> Good Morning Davide: >> >> I don't know if this information is in a MySQL table, but you >> can see what data is in use by taking a look at the directory: >> /gbdb/genbank/data/processed/ >> >> drwxr-xr-x 38 genbank genbank 1024 Nov 26 04:20 genbank.162.0 >> drwxr-xr-x 51 genbank genbank 1536 Nov 6 07:20 refseq.25 >> drwxr-xr-x 15 genbank genbank 512 Nov 26 04:20 refseq.26 > > > My directory list gives this: > > drwxr-xr-x 65 root root 1536 Nov 16 12:54 genbank.158.0 > drwxr-xr-x 62 root root 1536 Nov 15 12:53 genbank.159.0 > drwxr-xr-x 121 root root 2560 Nov 14 12:53 genbank.160.0 > drwxr-xr-x 73 root root 1536 Nov 13 12:49 genbank.161.0 > drwxr-xr-x 39 root root 1024 Nov 27 11:20 genbank.162.0 > drwxr-xr-x 56 root root 1536 Nov 12 12:47 refseq.21 > drwxr-xr-x 57 root root 1536 Nov 2 18:10 refseq.22 > drwxr-xr-x 7 root root 1536 Nov 27 21:30 refseq.23 > drwxr-xr-x 51 root root 1536 Nov 6 16:20 refseq.25 > drwxr-xr-x 15 root root 512 Nov 26 13:20 refseq.26 > > I need to eliminate some of them to free some space, so I would like > to know how can I decide if a version is useful or not for my mirror > subset. Is the list you posted the one for the online UCSC genome > browser? > > d > > > /* > Davide Cittaro > HPC and Bioinformatics Systems @ Informatics Core > > IFOM - Istituto FIRC di Oncologia Molecolare > via adamello, 16 > 20139 Milano > Italy > > tel.: +39(02)574303007 > e-mail: davide.cittaro at ifom-ieo-campus.it > */ > > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror From rhead at soe.ucsc.edu Wed Nov 28 12:34:12 2007 From: rhead at soe.ucsc.edu (Brooke Rhead) Date: Wed, 28 Nov 2007 12:34:12 -0800 Subject: [Genome-mirror] hgCustom buffer overflow In-Reply-To: References: Message-ID: <474DD0C4.2050001@soe.ucsc.edu> Hello Davide, Thank you for bringing this to our attention. We did find your buffer overflow error in our error logs. We are working on a fix. -- Brooke Rhead UCSC Genome Bioinformatics Group Davide Cittaro wrote: > Dear GB people, > we have found that hgCustom cgi executable goes into a buffer overflow > if one tries to upload a track with a name longer than 52 characters. > We have tried this with bed tracks. > You can take a look to your apache error log, since we have tested > also on your public web site, there should be some lines like this: > > [/cgi-bin/hgCustom] buffer overflow, size 64, format: %s_%s, > referer: http://genome.ifom-ieo-campus.it/cgi-bin/hgCustom?hgsid=11331 > > with a different hostname/hgsid, of course. > We haven't tried to exploit this (yet ^__^). > > Cheers > > d > /* > Davide Cittaro > HPC and Bioinformatics Systems @ Informatics Core > > IFOM - Istituto FIRC di Oncologia Molecolare > via adamello, 16 > 20139 Milano > Italy > > tel.: +39(02)574303007 > e-mail: davide.cittaro at ifom-ieo-campus.it > */ > > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror From hiram at soe.ucsc.edu Wed Nov 28 13:18:10 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Wed, 28 Nov 2007 13:18:10 -0800 Subject: [Genome-mirror] one more question about genbank and refseq In-Reply-To: <71381681-69C1-4869-976B-A0F5A4E80627@ifom-ieo-campus.it> References: <193EC060-46D7-494B-8079-8138EA5E80E8@ifom-ieo-campus.it> <474AFE50.9070605@soe.ucsc.edu> <71381681-69C1-4869-976B-A0F5A4E80627@ifom-ieo-campus.it> Message-ID: <474DDB12.7010908@soe.ucsc.edu> Good Afternoon Davide: To verify exactly which /gbdb/genbank/ files are in use by your browsers, take a look at the gbExtFile tables in your assemblies. This table lists the actual files used by that browser. --Hiram Davide Cittaro wrote: > > On Nov 26, 2007, at 6:11 PM, Hiram Clawson wrote: > >> Good Morning Davide: >> >> I don't know if this information is in a MySQL table, but you >> can see what data is in use by taking a look at the directory: >> /gbdb/genbank/data/processed/ >> >> drwxr-xr-x 38 genbank genbank 1024 Nov 26 04:20 genbank.162.0 >> drwxr-xr-x 51 genbank genbank 1536 Nov 6 07:20 refseq.25 >> drwxr-xr-x 15 genbank genbank 512 Nov 26 04:20 refseq.26 > > My directory list gives this: > > drwxr-xr-x 65 root root 1536 Nov 16 12:54 genbank.158.0 > drwxr-xr-x 62 root root 1536 Nov 15 12:53 genbank.159.0 > drwxr-xr-x 121 root root 2560 Nov 14 12:53 genbank.160.0 > drwxr-xr-x 73 root root 1536 Nov 13 12:49 genbank.161.0 > drwxr-xr-x 39 root root 1024 Nov 27 11:20 genbank.162.0 > drwxr-xr-x 56 root root 1536 Nov 12 12:47 refseq.21 > drwxr-xr-x 57 root root 1536 Nov 2 18:10 refseq.22 > drwxr-xr-x 7 root root 1536 Nov 27 21:30 refseq.23 > drwxr-xr-x 51 root root 1536 Nov 6 16:20 refseq.25 > drwxr-xr-x 15 root root 512 Nov 26 13:20 refseq.26 > > I need to eliminate some of them to free some space, so I would like to > know how can I decide if a version is useful or not for my mirror > subset. Is the list you posted the one for the online UCSC genome browser? > > d