From rhead at soe.ucsc.edu Mon Jul 2 15:06:43 2007 From: rhead at soe.ucsc.edu (Brooke Rhead) Date: Mon, 02 Jul 2007 15:06:43 -0700 Subject: [Genome-mirror] custom track questions In-Reply-To: <2D19BDC6-8EE8-43CB-8691-864D219A65BB@broad.mit.edu> References: <52E54868-6875-40B9-B268-8934CF5E4DA6@broad.mit.edu> <4684039F.8070701@soe.ucsc.edu> <2D19BDC6-8EE8-43CB-8691-864D219A65BB@broad.mit.edu> Message-ID: <468976F3.6060408@soe.ucsc.edu> Hi Jean, I spoke to one of our developers, and he thinks that perhaps the two missing files (customTrackLoad.html and customTrackManage.html) could be keeping the hgCustom cgi from working properly. Our system administrators just raised the limit on rsync connections to 15. Please let us know if you are still not able to rsync. Also, in case it isn't totally clear from the documentation, some .bed and .wib files are expected to appear in the trash/ct directory after the custom track database is implemented. The .wib files will contain binary data, and the .bed files will contain pointers to their corresponding database tables. Please let us know if you still have problems after getting the two files. -- Brooke Rhead UCSC Genome Bioinformatics Group Jean Chang wrote: > Brooke, > > I tried to follow the directions but I'm confounded. > > I have done the following: > create a customTrash database > added the customTracks lines to hg.conf > created ctdbuser and granted priveleges > created a tmpdir > > When I try to upload custom tracks, I'm still just getting .bed, .wig > and .wib files created in my trash/ct directory. Can you tell me what > additional information you'd need to point me to where I should be > looking to figure out what I'm missing in trying to get custom tracks > working? > > The only errors I'm seeing in my apache error_log is a lack of > customTrackLoad.html and customTrackManage.html but when I try an rsync > to get these files, I get: > > @ERROR: max connections (10) reached -- try again later > rsync: connection unexpectedly closed (0 bytes received so far) [receiver] > rsync error: error in rsync protocol data stream (code 12) at io.c(359) > > I just wanted to check if your rsync server maxxed out or if I have > another problem on my end. > > Thanks, > > Jean > > On Jun 28, 2007, at 2:53 PM, Brooke Rhead wrote: > >> Hello Jean, >> >> There is a good page about the custom track database on our wiki, here: >> >> http://genomewiki.cse.ucsc.edu/index.php/Using_custom_track_database >> >> The same information is also available in the kent source tree at >> /src/product/README.customTracks.dataBase. >> >> I hope this information helps. >> >> -- >> Brooke Rhead >> UCSC Genome Bioinformatics Group >> >> >> Jean Chang wrote: >>> Hi, >>> Where would I find information about how uploaded custom tracks are >>> handled on the server side? and ways to have custom tracks persist >>> at our internal mirror? >>> Also, what are the best sources of information about implementing >>> the custom track database? >>> Thanks, >>> Jean >>> _______________________________________________ >>> Genome-mirror mailing list >>> Genome-mirror at soe.ucsc.edu >>> http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror From rhead at soe.ucsc.edu Mon Jul 2 16:58:42 2007 From: rhead at soe.ucsc.edu (Brooke Rhead) Date: Mon, 02 Jul 2007 16:58:42 -0700 Subject: [Genome-mirror] custom track questions In-Reply-To: <468976F3.6060408@soe.ucsc.edu> References: <52E54868-6875-40B9-B268-8934CF5E4DA6@broad.mit.edu> <4684039F.8070701@soe.ucsc.edu> <2D19BDC6-8EE8-43CB-8691-864D219A65BB@broad.mit.edu> <468976F3.6060408@soe.ucsc.edu> Message-ID: <46899132.4010009@soe.ucsc.edu> Hi again Jean, Another developer has pointed out that you can verify that the custom track database is functioning from the command line, using a special hg.conf file. Instructions for doing so are here: http://genomewiki.cse.ucsc.edu/index.php/Using_custom_track_database#Command_line_access Another suggestion is to double-check that the custom track lines in your hg.conf file make sense and have been entered correctly. Incorrect hg.conf lines could cause the problem you describe. -- Brooke Rhead UCSC Genome Bioinformatics Group Brooke Rhead wrote: > Hi Jean, > > I spoke to one of our developers, and he thinks that perhaps the two > missing files (customTrackLoad.html and customTrackManage.html) could be > keeping the hgCustom cgi from working properly. > > Our system administrators just raised the limit on rsync connections to > 15. Please let us know if you are still not able to rsync. > > Also, in case it isn't totally clear from the documentation, some .bed > and .wib files are expected to appear in the trash/ct directory after > the custom track database is implemented. The .wib files will contain > binary data, and the .bed files will contain pointers to their > corresponding database tables. > > Please let us know if you still have problems after getting the two files. > > -- > Brooke Rhead > UCSC Genome Bioinformatics Group > > > Jean Chang wrote: >> Brooke, >> >> I tried to follow the directions but I'm confounded. >> >> I have done the following: >> create a customTrash database >> added the customTracks lines to hg.conf >> created ctdbuser and granted priveleges >> created a tmpdir >> >> When I try to upload custom tracks, I'm still just getting .bed, .wig >> and .wib files created in my trash/ct directory. Can you tell me what >> additional information you'd need to point me to where I should be >> looking to figure out what I'm missing in trying to get custom tracks >> working? >> >> The only errors I'm seeing in my apache error_log is a lack of >> customTrackLoad.html and customTrackManage.html but when I try an >> rsync to get these files, I get: >> >> @ERROR: max connections (10) reached -- try again later >> rsync: connection unexpectedly closed (0 bytes received so far) >> [receiver] >> rsync error: error in rsync protocol data stream (code 12) at io.c(359) >> >> I just wanted to check if your rsync server maxxed out or if I have >> another problem on my end. >> >> Thanks, >> >> Jean >> >> On Jun 28, 2007, at 2:53 PM, Brooke Rhead wrote: >> >>> Hello Jean, >>> >>> There is a good page about the custom track database on our wiki, here: >>> >>> http://genomewiki.cse.ucsc.edu/index.php/Using_custom_track_database >>> >>> The same information is also available in the kent source tree at >>> /src/product/README.customTracks.dataBase. >>> >>> I hope this information helps. >>> >>> -- >>> Brooke Rhead >>> UCSC Genome Bioinformatics Group >>> >>> >>> Jean Chang wrote: >>>> Hi, >>>> Where would I find information about how uploaded custom tracks are >>>> handled on the server side? and ways to have custom tracks persist >>>> at our internal mirror? >>>> Also, what are the best sources of information about implementing >>>> the custom track database? >>>> Thanks, >>>> Jean >>>> _______________________________________________ >>>> Genome-mirror mailing list >>>> Genome-mirror at soe.ucsc.edu >>>> http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From rhead at soe.ucsc.edu Tue Jul 3 16:26:47 2007 From: rhead at soe.ucsc.edu (Brooke Rhead) Date: Tue, 03 Jul 2007 16:26:47 -0700 Subject: [Genome-mirror] custom track questions In-Reply-To: References: <52E54868-6875-40B9-B268-8934CF5E4DA6@broad.mit.edu> <4684039F.8070701@soe.ucsc.edu> <2D19BDC6-8EE8-43CB-8691-864D219A65BB@broad.mit.edu> <468976F3.6060408@soe.ucsc.edu> <46899132.4010009@soe.ucsc.edu> Message-ID: <468ADB37.40800@soe.ucsc.edu> Hi Jean, I have some answers from our engineers, as well as some suggestions of things you could check to troubleshoot this problem. > (I didn't run the create statement with hgsql b/c I can't > find it - I just used mysql to create a database... perhaps that's the > problem?) hgsql is just a wrapper program around mysql that gets your username, password, etc. from ~/.hg.conf. Using mysql rather than hgsql is fine. (If you are interested, the program source resides in the src/hg/hgsql/ directory. When you type 'make' in that directory (assuming you have already built the libraries) the executable file will be placed in $HOME/bin/$MACHTYPE.) > I didn't realize the .wib files would persist. If the standard trash > cleaner script deletes them, they will get regenerated? No, old files cleaned out from trash are NOT regenerated automatically. If they are deleted, then the original custom track file will have to be resubmitted in hgCustom in order to view the custom track. > I rsynced the files, and they discovered that I do actually have them. > hgCustom is looking in /usr/local/apache/htdocs/goldenPath/help for them > rather than in $HOME/goldenPath/help - not sure why. I made a symlink > and the html files are loading but no dice on the custom track database. This definitely would be a problem (we confirmed yesterday that the cgi aborts with an error when these files are missing), but since you created symlinks and the html files are loading now, this was not the only problem. I also have several trouble-shooting questions for you: 1. When you load a custom-track, do you see any messages in your apache logs? e.g., in /usr/local/apache/logs/error_log or wherever they are on your apache server. Running tail -f on that file while loading a custom track may help you find errors generated by hgCustom. 2. In mysql (or hgsql), what is the result from these commands?: use mysql show grants for ctdbuser; select * from db where Db = "customTrash"; 3. Do you see any error messages anywhere else? 4. Do the custom tracks work (even if they are not yet going into customTrash db)? 5. Do you have recent cgis (not older than a couple of months)? 6. Did the mysql permissions work for the customTrash db and user? 7. Can you log into the customTrash db with mysql? e.g. mysql -u ctdbuser -p customTrash Enter the password ctdbpasswd when prompted. 8. Can you create and delete a test table in customTrash db? -- Brooke Rhead UCSC Genome Bioinformatics Group Jean Chang wrote: > Brooke, > > I'll describe what I've tried, in-line below: > >> Another developer has pointed out that you can verify that the custom >> track database is functioning from the command line, using a special >> hg.conf file. Instructions for doing so are here: >> >> http://genomewiki.cse.ucsc.edu/index.php/Using_custom_track_database#Command_line_access >> >> >> Another suggestion is to double-check that the custom track lines in >> your hg.conf file make sense and have been entered correctly. >> Incorrect hg.conf lines could cause the problem you describe. > > I tried following the directions but I cannot find hgsql anywhere... > From the genomewiki, I can see it should be a Kent source utility but I > cannot find it in my copy of /kent/src > > FYI, the lines I added to hg.conf are: > customTracks.host=localhost > customTracks.user=ctdbuser > customTracks.pasword=ctdbpasswd > customTracks.useAll=yes > customTracks.tmpdir=/srv/www/tmp > > After i upload data, I see .bed, .wig and .wib files in $HOME/trash/ct > and the customTrash database (in /mysql/data/customTrash) is > unpopulated. (I didn't run the create statement with hgsql b/c I can't > find it - I just used mysql to create a database... perhaps that's the > problem?) > >>> I spoke to one of our developers, and he thinks that perhaps the two >>> missing files (customTrackLoad.html and customTrackManage.html) could >>> be keeping the hgCustom cgi from working properly. > > I rsynced the files, and they discovered that I do actually have them. > hgCustom is looking in /usr/local/apache/htdocs/goldenPath/help for them > rather than in $HOME/goldenPath/help - not sure why. I made a symlink > and the html files are loading but no dice on the custom track database. > >>> Our system administrators just raised the limit on rsync connections >>> to 15. Please let us know if you are still not able to rsync. > > I am able to rsync now - thanks! > >>> Also, in case it isn't totally clear from the documentation, some >>> .bed and .wib files are expected to appear in the trash/ct directory >>> after the custom track database is implemented. The .wib files will >>> contain binary data, and the .bed files will contain pointers to >>> their corresponding database tables. > > I didn't realize the .wib files would persist. If the standard trash > cleaner script deletes them, they will get regenerated? > >>> Please let us know if you still have problems after getting the two >>> files. > > Sorry to be so dense about this. Thanks in advance for your help! > > Regards, > > Jean From forgetta at gmail.com Wed Jul 4 08:00:48 2007 From: forgetta at gmail.com (Vincenzo Forgetta) Date: Wed, 04 Jul 2007 11:00:48 -0400 Subject: [Genome-mirror] hgBlat size limit Message-ID: <468BB620.1060805@gmail.com> Hi all, Not sure if I posted this in the right forum ... I'm trying to increase the query size limit of hgBlat. I modified the maxSingleSize, maxTotalSize, and maxSeqCount to number that will allow me to web-blat sequences in excess of 500Kb (ran make, etc to install the binary in cgu-bin). However, when I try to webblat either by pasting a sequence or submitting a file, it only aligns ~40Kb of the query. Did I miss a variable's value that needed to be changed? Thanks. Vince From jlchang at broad.mit.edu Tue Jul 3 09:46:31 2007 From: jlchang at broad.mit.edu (Jean Chang) Date: Tue, 3 Jul 2007 12:46:31 -0400 Subject: [Genome-mirror] custom track questions In-Reply-To: <46899132.4010009@soe.ucsc.edu> References: <52E54868-6875-40B9-B268-8934CF5E4DA6@broad.mit.edu> <4684039F.8070701@soe.ucsc.edu> <2D19BDC6-8EE8-43CB-8691-864D219A65BB@broad.mit.edu> <468976F3.6060408@soe.ucsc.edu> <46899132.4010009@soe.ucsc.edu> Message-ID: Brooke, I'll describe what I've tried, in-line below: > Another developer has pointed out that you can verify that the > custom track database is functioning from the command line, using a > special hg.conf file. Instructions for doing so are here: > > http://genomewiki.cse.ucsc.edu/index.php/ > Using_custom_track_database#Command_line_access > > Another suggestion is to double-check that the custom track lines > in your hg.conf file make sense and have been entered correctly. > Incorrect hg.conf lines could cause the problem you describe. I tried following the directions but I cannot find hgsql anywhere... From the genomewiki, I can see it should be a Kent source utility but I cannot find it in my copy of /kent/src FYI, the lines I added to hg.conf are: customTracks.host=localhost customTracks.user=ctdbuser customTracks.pasword=ctdbpasswd customTracks.useAll=yes customTracks.tmpdir=/srv/www/tmp After i upload data, I see .bed, .wig and .wib files in $HOME/trash/ ct and the customTrash database (in /mysql/data/customTrash) is unpopulated. (I didn't run the create statement with hgsql b/c I can't find it - I just used mysql to create a database... perhaps that's the problem?) >> I spoke to one of our developers, and he thinks that perhaps the >> two missing files (customTrackLoad.html and >> customTrackManage.html) could be keeping the hgCustom cgi from >> working properly. I rsynced the files, and they discovered that I do actually have them. hgCustom is looking in /usr/local/apache/htdocs/goldenPath/help for them rather than in $HOME/goldenPath/help - not sure why. I made a symlink and the html files are loading but no dice on the custom track database. >> Our system administrators just raised the limit on rsync >> connections to 15. Please let us know if you are still not able >> to rsync. I am able to rsync now - thanks! >> Also, in case it isn't totally clear from the documentation, >> some .bed and .wib files are expected to appear in the trash/ct >> directory after the custom track database is implemented. >> The .wib files will contain binary data, and the .bed files will >> contain pointers to their corresponding database tables. I didn't realize the .wib files would persist. If the standard trash cleaner script deletes them, they will get regenerated? >> Please let us know if you still have problems after getting the >> two files. Sorry to be so dense about this. Thanks in advance for your help! Regards, Jean From galt at soe.ucsc.edu Wed Jul 4 22:20:53 2007 From: galt at soe.ucsc.edu (Galt Barber) Date: Wed, 4 Jul 2007 22:20:53 -0700 (PDT) Subject: [Genome-mirror] hgBlat size limit In-Reply-To: <468BB620.1060805@gmail.com> References: <468BB620.1060805@gmail.com> Message-ID: This is the limit of gfServer. -maxNtSize=N Maximum size of untranslated DNA query sequence Default is 40000 You really can't expand it much for practical purposes. The algorithm has a step that is O(n*n) in size, so really long alignments won't work. Trying to align enormous chunks of near-perfect matches does not work well. You could align just the ends of, for example, a contig with blat and infer the middle. Big genes work because blat is only aligning the exons individually and then a finishing step chains the exons together. -Galt On Wed, 4 Jul 2007, Vincenzo Forgetta wrote: > Hi all, > > Not sure if I posted this in the right forum ... > > I'm trying to increase the query size limit of hgBlat. I modified the > maxSingleSize, maxTotalSize, and maxSeqCount to number that will allow > me to web-blat sequences in excess of 500Kb (ran make, etc to install > the binary in cgu-bin). However, when I try to webblat either by pasting > a sequence or submitting a file, it only aligns ~40Kb of the query. > Did I miss a variable's value that needed to be changed? > > Thanks. > > Vince > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From davide.cittaro at ifom-ieo-campus.it Fri Jul 6 00:42:26 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Fri, 6 Jul 2007 09:42:26 +0200 Subject: [Genome-mirror] gfServer max client Message-ID: Hi Genome people, I would like to know how gfServer handles simultaneous access to a port. How many clients can be connected? How can I configure it? What happens to connection over the (supposed) limit? Thanks d /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro at ifom-ieo-campus.it */ From galt at soe.ucsc.edu Fri Jul 6 12:02:54 2007 From: galt at soe.ucsc.edu (Galt Barber) Date: Fri, 6 Jul 2007 12:02:54 -0700 (PDT) Subject: [Genome-mirror] gfServer max client In-Reply-To: References: Message-ID: from gfServer.c: (line 522) /* Set up socket. Get ready to listen to it. */ socketHandle = netAcceptingSocket(port, 100); 100 is the queue size, so if the load backs up, it can't queue more than 100 waiting requests, but that seems to work fine normally. It does not seem to use fork() like some unix daemons do. I know that Jim's BLATZ, an extension of BLAT that will have higher sensitivity but higher RAM requirements does use fork(). I don't know how well gfServer currently scales over new multi-cpu systems, but Jim understands all the issues involved very well. If you are hitting a problem from this, is it because you are creating long-running queries? Typically, gfServer responds to multiple queries per second. -Galt On Fri, 6 Jul 2007, Davide Cittaro wrote: > Hi Genome people, I would like to know how gfServer handles > simultaneous access to a port. How many clients can be connected? How > can I configure it? What happens to connection over the (supposed) > limit? > > Thanks > > d > > /* > Davide Cittaro > HPC and Bioinformatics Systems @ Informatics Core > > IFOM - Istituto FIRC di Oncologia Molecolare > via adamello, 16 > 20139 Milano > Italy > > tel.: +39(02)574303007 > e-mail: davide.cittaro at ifom-ieo-campus.it > */ > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From galt at soe.ucsc.edu Sat Jul 7 13:18:42 2007 From: galt at soe.ucsc.edu (Galt Barber) Date: Sat, 7 Jul 2007 13:18:42 -0700 (PDT) Subject: [Genome-mirror] gfServer max client In-Reply-To: <4CD25A84-D143-4506-8962-4381D9BFD5C8@ifom-ieo-campus.it> References: <4CD25A84-D143-4506-8962-4381D9BFD5C8@ifom-ieo-campus.it> Message-ID: It should be ok to increase the size of the queue, however, we have multiple people using gfServer more or less at the same time all the time with hgBlat/gfServer. Maybe your user should submit his requests one at a time. Usually gfServer can satisfy 10 requests per second (sequentially). -Galt On Sat, 7 Jul 2007, Davide Cittaro wrote: > Hi Galt, and thanks for the answer > > On Jul 6, 2007, at 9:02 PM, Galt Barber wrote: > > > I don't know how well gfServer currently scales over > > new multi-cpu systems, but Jim understands > > all the issues involved very well. > > > > I thought it is not threaded and, since you told me it doesn't use > forks, I believe that it doesn't scales at all on multi-core or multi- > CPU machines. > > > If you are hitting a problem from this, > > is it because you are creating long-running queries? > > Typically, gfServer responds to multiple queries > > per second. > > Don't worry, I have an user here that has submitted tons of gfClient > instances, and the gfServer takes some time to give me access to its > statistics. Now I understand why. > Do you think that increasing the queue size would harm the gfServer > functionality? > > Thanks > > d > > /* > Davide Cittaro > HPC and Bioinformatics Systems @ Informatics Core > > IFOM - Istituto FIRC di Oncologia Molecolare > via adamello, 16 > 20139 Milano > Italy > > tel.: +39(02)574303007 > e-mail: davide.cittaro at ifom-ieo-campus.it > */ > > > From galt at soe.ucsc.edu Sat Jul 7 13:30:10 2007 From: galt at soe.ucsc.edu (Galt Barber) Date: Sat, 7 Jul 2007 13:30:10 -0700 (PDT) Subject: [Genome-mirror] gfServer max client In-Reply-To: References: <4CD25A84-D143-4506-8962-4381D9BFD5C8@ifom-ieo-campus.it> Message-ID: Also, if your user is doing many queries, stand-alone commandline-line blat is good for batch jobs, as I'm sure you're aware. -Galt On Sat, 7 Jul 2007, Galt Barber wrote: > > It should be ok to increase the size of the queue, > however, we have multiple people using gfServer > more or less at the same time all the time > with hgBlat/gfServer. Maybe your user should > submit his requests one at a time. Usually > gfServer can satisfy 10 requests per second > (sequentially). > > -Galt > > > On Sat, 7 Jul 2007, Davide Cittaro wrote: > > > Hi Galt, and thanks for the answer > > > > On Jul 6, 2007, at 9:02 PM, Galt Barber wrote: > > > > > I don't know how well gfServer currently scales over > > > new multi-cpu systems, but Jim understands > > > all the issues involved very well. > > > > > > > I thought it is not threaded and, since you told me it doesn't use > > forks, I believe that it doesn't scales at all on multi-core or multi- > > CPU machines. > > > > > If you are hitting a problem from this, > > > is it because you are creating long-running queries? > > > Typically, gfServer responds to multiple queries > > > per second. > > > > Don't worry, I have an user here that has submitted tons of gfClient > > instances, and the gfServer takes some time to give me access to its > > statistics. Now I understand why. > > Do you think that increasing the queue size would harm the gfServer > > functionality? > > > > Thanks > > > > d > > > > /* > > Davide Cittaro > > HPC and Bioinformatics Systems @ Informatics Core > > > > IFOM - Istituto FIRC di Oncologia Molecolare > > via adamello, 16 > > 20139 Milano > > Italy > > > > tel.: +39(02)574303007 > > e-mail: davide.cittaro at ifom-ieo-campus.it > > */ > > > > > > > From davide.cittaro at ifom-ieo-campus.it Sat Jul 7 02:33:13 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Sat, 7 Jul 2007 11:33:13 +0200 Subject: [Genome-mirror] gfServer max client In-Reply-To: References: Message-ID: <4CD25A84-D143-4506-8962-4381D9BFD5C8@ifom-ieo-campus.it> Hi Galt, and thanks for the answer On Jul 6, 2007, at 9:02 PM, Galt Barber wrote: > I don't know how well gfServer currently scales over > new multi-cpu systems, but Jim understands > all the issues involved very well. > I thought it is not threaded and, since you told me it doesn't use forks, I believe that it doesn't scales at all on multi-core or multi- CPU machines. > If you are hitting a problem from this, > is it because you are creating long-running queries? > Typically, gfServer responds to multiple queries > per second. Don't worry, I have an user here that has submitted tons of gfClient instances, and the gfServer takes some time to give me access to its statistics. Now I understand why. Do you think that increasing the queue size would harm the gfServer functionality? Thanks d /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro at ifom-ieo-campus.it */ From archanat at soe.ucsc.edu Mon Jul 9 12:58:42 2007 From: archanat at soe.ucsc.edu (Archana Thakkapallayil) Date: Mon, 09 Jul 2007 12:58:42 -0700 Subject: [Genome-mirror] Release of new assembly: Drosophila melanogaster (dm3) Message-ID: <46929372.1050906@soe.ucsc.edu> Hello mirror sites, This message is to announce that we are preparing to release the new Drosophila assembly, dm3, soon. Mirror sites should be prepared to host: ~15 GB ( dm3 tables ) ~4.1 GB ( files in /gbdb/dm3/* ) Additionally there are net and chain tables from other organisms that will be released to the respective annotation databases. These tables are named netDm3 and *chainDm3*. The size of these tables is as follows: anoGam1 : 205 MB dp3 : 367 MB droPer1 : 1024 MB droAna2 : 3995 MB droEre1 : 5744 MB droYak2 : 1110 MB droSec1 : 2404 MB droSim1 : 732 MB total : ~15 GB Thanks, Archana UCSC Genome Bioinformatics Group From aamp at ucsc.edu Fri Jul 13 16:20:45 2007 From: aamp at ucsc.edu (Andy Pohl) Date: Fri, 13 Jul 2007 16:20:45 -0700 Subject: [Genome-mirror] v163 Genome Browser available Message-ID: <9fa943760707131620n1d8aacfdq9661ca1b353c4e00@mail.gmail.com> Hello, The v163 source is at: http://hgdownload.cse.ucsc.edu/admin/jksrc.zip or labeled with source number: http://hgdownload.cse.ucsc.edu/admin/jksrc.v163.zip cartDump cartReset das hgc hgBlat hgConvert hgCustom hgEncodeDataVersions hgGateway hgGene hgGenome hgLiftOver hgNear hgPcr hgSession hgTables hgText hgTracks hgTrackUi hgVisiGene mkEncodeFrameset pbGateway pbGlobal pbTracks phyloGif /usr/local/apache/cgi-bin/all.joiner /usr/local/apache/cgi-bin/hgNearData/* /usr/local/apache/cgi-bin/hgGeneData/* /usr/local/apache/cgi-bin/hgcData/* /usr/local/apache/cgi-bin/hgCgiData/* /usr/local/apache/cgi-bin/loader/* /usr/local/apache/cgi-bin/visiGeneData/* Contents of v163 may be seen at: http://genome-test.cse.ucsc.edu/builds/versions.html Please check both v163-preview and v163-final to see all the code and data changes in this release. Have a nice weekend! Andy Pohl UCSC CBSE From Lucie.Friedlander at sanofi-aventis.com Mon Jul 16 06:03:12 2007 From: Lucie.Friedlander at sanofi-aventis.com (Lucie.Friedlander at sanofi-aventis.com) Date: Mon, 16 Jul 2007 15:03:12 +0200 Subject: [Genome-mirror] Mirroring the UCSC genom browser Message-ID: <56D2D492E52B534EA14D1380D8EF0E6E3DF4BC@CLPW44.f2.enterprise> Dear all, I am working in a pharmaceutical industry Sanotif-aventis and I will be interesting in having a mirror of the UCSC Genom browser. Do we need specific aggreement and commercial payement ? Is it technically possible to add your own set of genes or annotations into the system ? thank you for your answer. Regards. Lucie Friedlander SCDM Paris - Scientific Computing & Data Management SANOFI-AVENTIS 13, quai Jules Guesde F-94400 VITRY-SUR-SEINE Tel: +33 (0)1.58.93.88.39 Fax: +33 (0)&. 58.93.34.78 From donnak at soe.ucsc.edu Mon Jul 16 16:48:06 2007 From: donnak at soe.ucsc.edu (Donna Karolchik) Date: Mon, 16 Jul 2007 16:48:06 -0700 Subject: [Genome-mirror] Mirroring the UCSC genom browser References: <56D2D492E52B534EA14D1380D8EF0E6E3DF4BC@CLPW44.f2.enterprise> Message-ID: <02fc01c7c803$d7304370$0ba8a8c0@donnakLT> Dear Lucie, Thanks for your inquiry. Yes, it is possible to add your own genes and other annotations to the Genome Browser, either by loading them as custom tracks (which does not require access to the code) or by building them into the Genome Browser track set (which requires modifications to the software). Commerical mirroring of our software does require a license. You will find licensing information at http://genome.ucsc.edu/license/, which includes a link to our software licensing agreement. Please feel free to contact me if you have any questions after reviewing this information. The licensing includes documentation that will assist you in adding annotations to the browser. Regards, -Donna ----------------------------------- Donna Karolchik UCSC Genome Bioinformatics Group http://genome.ucsc.edu ----- Original Message ----- From: To: Sent: Monday, July 16, 2007 6:03 AM Subject: [Genome-mirror] Mirroring the UCSC genom browser > Dear all, > I am working in a pharmaceutical industry Sanotif-aventis and I > will be > interesting in having a mirror of the UCSC Genom browser. > Do we need specific aggreement and commercial payement ? > Is it technically possible to add your own set of genes or > annotations > into the system ? > thank you for your answer. > Regards. > > Lucie Friedlander > SCDM Paris - Scientific Computing & Data Management > SANOFI-AVENTIS > 13, quai Jules Guesde > F-94400 VITRY-SUR-SEINE > Tel: +33 (0)1.58.93.88.39 > Fax: +33 (0)&. 58.93.34.78 > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From thiago.venancio at gmail.com Thu Jul 19 09:28:12 2007 From: thiago.venancio at gmail.com (Thiago Venancio) Date: Thu, 19 Jul 2007 13:28:12 -0300 Subject: [Genome-mirror] aedes genome Message-ID: <44255ea80707190928v460a7b0ex7d3aa7bf1b5c63de@mail.gmail.com> Hi all. Are you intending to put the recently published Aedes genome online in a near future ? Thanks in advance. Thiago -- "Innovation distinguishes between a leader and a follower." Steve Jobs ======================== Thiago Motta Venancio, MSc PhD student in Bioinformatics University of Sao Paulo ======================== From archanat at soe.ucsc.edu Fri Jul 20 11:29:29 2007 From: archanat at soe.ucsc.edu (Archana Thakkapallayil) Date: Fri, 20 Jul 2007 11:29:29 -0700 Subject: [Genome-mirror] aedes genome In-Reply-To: <44255ea80707190928v460a7b0ex7d3aa7bf1b5c63de@mail.gmail.com> References: <44255ea80707190928v460a7b0ex7d3aa7bf1b5c63de@mail.gmail.com> Message-ID: <46A0FF09.5070702@soe.ucsc.edu> Hello Thiago, Currently, we don't have any plans to add this genome to our Browser. We are sorry about that. Regards, Archana, UCSC Genome Bioinformatics Group Thiago Venancio wrote: > Hi all. > > Are you intending to put the recently published Aedes genome online in a > near future ? > > Thanks in advance. > > Thiago > > From hiram at soe.ucsc.edu Fri Jul 20 11:38:37 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Fri, 20 Jul 2007 11:38:37 -0700 Subject: [Genome-mirror] aedes genome In-Reply-To: <46A0FF09.5070702@soe.ucsc.edu> References: <44255ea80707190928v460a7b0ex7d3aa7bf1b5c63de@mail.gmail.com> <46A0FF09.5070702@soe.ucsc.edu> Message-ID: <46A1012D.8020203@soe.ucsc.edu> Good Morning Thiago: You can view the Aedes genome at VectorBase: http://aaegypti.vectorbase.org/Genome/Home/ --Hiram Archana Thakkapallayil wrote: > Hello Thiago, > > Currently, we don't have any plans to add this genome to our Browser. We > are sorry about that. > > Regards, > > Archana, > UCSC Genome Bioinformatics Group > > Thiago Venancio wrote: >> Hi all. >> >> Are you intending to put the recently published Aedes genome online in a >> near future ? >> >> Thanks in advance. >> >> Thiago >> >> > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From kuhn at soe.ucsc.edu Mon Jul 23 10:56:39 2007 From: kuhn at soe.ucsc.edu (Robert Kuhn) Date: Mon, 23 Jul 2007 10:56:39 -0700 Subject: [Genome-mirror] dropped some tables Message-ID: <200707231756.KAA07888@moondance.cse.ucsc.edu> Dear mirrors: For those of you who may be dropping tables manually, be advised that we dropped some tables (1.8 Gb) from the old hg15 database. They fit the description, *hg15Mm3L. thanks, --b0b kuhn ucsc genome bioinformatics group From yoyoq at yahoo.com Wed Jul 25 14:30:14 2007 From: yoyoq at yahoo.com (jp d) Date: Wed, 25 Jul 2007 14:30:14 -0700 (PDT) Subject: [Genome-mirror] slash at hgGateway Message-ID: <981414.82441.qm@web50410.mail.re2.yahoo.com> hi, sorry to bother, but setting up our mirror of the browser we noticed something odd about hgGateway. It appears to happen on the main site as well. http://genome.ucsc.edu/cgi-bin/hgGateway works fine, but with a trailing slash http://genome.ucsc.edu/cgi-bin/hgGateway/ further submissions don't work pressing submit from that point adds an extra cgi-bin to the path. i know we should just not use the last slash. but i though i would let you know its like when you tell your doctor, "it hurts when i do this" and he says "well, don't do that" thanks jpd From ann at soe.ucsc.edu Wed Jul 25 14:54:32 2007 From: ann at soe.ucsc.edu (Ann Zweig) Date: Wed, 25 Jul 2007 14:54:32 -0700 Subject: [Genome-mirror] slash at hgGateway In-Reply-To: <981414.82441.qm@web50410.mail.re2.yahoo.com> References: <981414.82441.qm@web50410.mail.re2.yahoo.com> Message-ID: <46A7C698.7080607@cse.ucsc.edu> Hello jpd, My advice to you is: "well, don't do that"! That said, this should probably be documented in our mirror site instructions. Thanks for pointing it out. Regards, ---------- Ann Zweig UCSC Genome Bioinformatics Group http://genome.ucsc.edu jp d wrote: > hi, > sorry to bother, but setting up our mirror > of the browser we noticed something odd > about hgGateway. > It appears to happen on the main site as well. > http://genome.ucsc.edu/cgi-bin/hgGateway > works fine, but with a trailing slash > http://genome.ucsc.edu/cgi-bin/hgGateway/ > further submissions don't work > pressing submit from that point adds an extra > cgi-bin to the path. > > i know we should just not use the last slash. > but i though i would let you know > > its like when you tell your doctor, > "it hurts when i do this" > and he says > "well, don't do that" > > thanks > jpd > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror From itamarsi at ekmd.huji.ac.il Thu Jul 26 06:42:47 2007 From: itamarsi at ekmd.huji.ac.il (itamar simon) Date: Thu, 26 Jul 2007 16:42:47 +0300 Subject: [Genome-mirror] mapping u133plus to genes Message-ID: I am trying to get the genomic coordinates of all the genes mapped to the Affymetrix U133 Plus array (using the hg17 release). I did it by getting the coordinates from the known genes table and the mapping from the KnownToU133Plus2 table. However I found that not all Affymetrix U133Plus 2 probes exists in this table even for probes that are correctly mapped in your browser. For example probes 232361_s_at and 225645_at are both mapped to the EHF gene on chromosome 11 but are absent from the KnownToU133Plus2 table. Please let me know how I can get the coordinates for all Affy probes. Thanks a lot Itamar _______________________ Itamar Simon Department of Molecular Biology Hebrew University Medical School Hadassah Ein Kerem Jerusalem Israel 91120 Tel 972-2-6758544 Fax 972-2-6758992 email itamarsi at ekmd.huji.ac.il From hiram at soe.ucsc.edu Thu Jul 26 09:15:58 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Thu, 26 Jul 2007 09:15:58 -0700 Subject: [Genome-mirror] slash at hgGateway In-Reply-To: <981414.82441.qm@web50410.mail.re2.yahoo.com> References: <981414.82441.qm@web50410.mail.re2.yahoo.com> Message-ID: <46A8C8BE.1020802@soe.ucsc.edu> Good Morning jpd: As you noted in other email, please leave the slash off the cgi binary reference. I'm more curious though what prompted you to think of putting a slash at the end of a cgi binary reference. If you found an example of this somewhere in our WEB pages, we should know about that so we can eliminate it. I do recognize the a trailing slash is a good idea in a URL that is referencing a directory, but it doesn't have the same meaning when applied to a cgi binary reference. --Hiram jp d wrote: > hi, > sorry to bother, but setting up our mirror > of the browser we noticed something odd > about hgGateway. > It appears to happen on the main site as well. > http://genome.ucsc.edu/cgi-bin/hgGateway > works fine, but with a trailing slash > http://genome.ucsc.edu/cgi-bin/hgGateway/ > further submissions don't work > pressing submit from that point adds an extra > cgi-bin to the path. > > i know we should just not use the last slash. > but i though i would let you know > > its like when you tell your doctor, > "it hurts when i do this" > and he says > "well, don't do that" > > thanks > jpd From yoyoq at yahoo.com Thu Jul 26 09:23:55 2007 From: yoyoq at yahoo.com (jp d) Date: Thu, 26 Jul 2007 09:23:55 -0700 (PDT) Subject: [Genome-mirror] slash at hgGateway In-Reply-To: <46A8C8BE.1020802@soe.ucsc.edu> Message-ID: <109680.92954.qm@web50407.mail.re2.yahoo.com> hi, no, no link for it. a user just typed it in out of habit. --- Hiram Clawson wrote: > Good Morning jpd: > > As you noted in other email, please leave the slash > off the cgi > binary reference. > > I'm more curious though what prompted you to think > of putting > a slash at the end of a cgi binary reference. If > you found an example > of this somewhere in our WEB pages, we should know > about that > so we can eliminate it. I do recognize the a > trailing slash is a good > idea in a URL that is referencing a directory, but > it doesn't have > the same meaning when applied to a cgi binary > reference. > > --Hiram > > jp d wrote: > > hi, > > sorry to bother, but setting up our mirror > > of the browser we noticed something odd > > about hgGateway. > > It appears to happen on the main site as well. > > http://genome.ucsc.edu/cgi-bin/hgGateway > > works fine, but with a trailing slash > > http://genome.ucsc.edu/cgi-bin/hgGateway/ > > further submissions don't work > > pressing submit from that point adds an extra > > cgi-bin to the path. > > > > i know we should just not use the last slash. > > but i though i would let you know > > > > its like when you tell your doctor, > > "it hurts when i do this" > > and he says > > "well, don't do that" > > > > thanks > > jpd > From kayla at soe.ucsc.edu Thu Jul 26 13:47:59 2007 From: kayla at soe.ucsc.edu (Kayla Smith) Date: Thu, 26 Jul 2007 13:47:59 -0700 Subject: [Genome-mirror] mapping u133plus to genes In-Reply-To: References: Message-ID: <46A9087F.9040107@cse.ucsc.edu> Itamar, The reason you are not seeing the probe 232361_s_at in the knownToU133Plus2 table is because there is a "better" probe that aligns to the EHF gene (NM_012153), namely, the probe "222932_at". The coordinates for all of the U133Plus2 affymetrix probes can be found in the affyU133Plus2 table. The relevant fields are tName, tStart, tEnd. qName is the name of the probes. I hope this information is helpful to you. Please don't hesitate to contact us again if you require further assistance. Kayla Smith UCSC Bioinformatics Group itamar simon wrote: > I am trying to get the genomic coordinates of all the genes mapped to > the Affymetrix U133 Plus array (using the hg17 release). I did it by > getting the coordinates from the known genes table and the mapping from > the KnownToU133Plus2 table. However I found that not all Affymetrix > U133Plus 2 probes exists in this table even for probes that are > correctly mapped in your browser. For example probes 232361_s_at and > 225645_at are both mapped to the EHF gene on chromosome 11 but are > absent from the KnownToU133Plus2 table. > > Please let me know how I can get the coordinates for all Affy probes. > > Thanks a lot > > Itamar > > > > _______________________ > > Itamar Simon > > Department of Molecular Biology > > Hebrew University Medical School > > Hadassah Ein Kerem > > Jerusalem > > Israel 91120 > > Tel 972-2-6758544 > > Fax 972-2-6758992 > > email itamarsi at ekmd.huji.ac.il > > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror From kayla at soe.ucsc.edu Thu Jul 26 13:50:08 2007 From: kayla at soe.ucsc.edu (Kayla Smith) Date: Thu, 26 Jul 2007 13:50:08 -0700 Subject: [Genome-mirror] 5 new worms coming soon Message-ID: <46A90900.3020707@cse.ucsc.edu> Hello mirror sites, This message is to announce that we are preparing to release five new worm assemblies, ce4, cb3, caeRem2, caePb1, and priPac1 next week. Mirror sites should be prepared to host: ~19 GB ( worm tables ) ~1.2 GB ( files in /gbdb/{worms}/* ) ce4: tables: 4706M files (/gbdb/ce4/*): 493M cb3: tables: 2004M files (/gbdb/cb3/*): 59M caeRem2: tables: 4603M files (/gbdb/caeRem2/*): 86M caePb1: tables: 6156M files (/gbdb/caePb1/*): 314M priPac1: tables: 1534M files (/gbdb/priPac1/*): 243M Thanks, Kayla Smith UCSC Genome Bioinformatics Group From aamp at ucsc.edu Mon Jul 30 13:46:41 2007 From: aamp at ucsc.edu (Andy Pohl) Date: Mon, 30 Jul 2007 13:46:41 -0700 Subject: [Genome-mirror] v164 Genome Browser Available Message-ID: <9fa943760707301346o6e00f8e2tf9cea1aa06431ed3@mail.gmail.com> Hello, The v164 source is at: http://hgdownload.cse.ucsc.edu/admin/jksrc.zip or labeled with source number: http://hgdownload.cse.ucsc.edu/admin/jksrc.v164.zip cartDump cartReset das hgc hgBlat hgConvert hgCustom hgEncodeDataVersions hgGateway hgGene hgGenome hgLiftOver hgNear hgPcr hgSession hgTables hgText hgTracks hgTrackUi hgVisiGene mkEncodeFrameset pbGateway pbGlobal pbTracks phyloGif /usr/local/apache/cgi-bin/all.joiner /usr/local/apache/cgi-bin/hgNearData/* /usr/local/apache/cgi-bin/hgGeneData/* /usr/local/apache/cgi-bin/hgcData/* /usr/local/apache/cgi-bin/hgCgiData/* /usr/local/apache/cgi-bin/loader/* /usr/local/apache/cgi-bin/visiGeneData/* Contents of v164 may be seen at: http://genome-test.cse.ucsc.edu/builds/versions.html Please check both v164-preview and v164-final to see all the code and data changes in this release. Have a nice week! Andy Pohl UCSC CBSE