From davide.cittaro at ifom-ieo-campus.it Tue Jan 2 00:44:30 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Tue, 2 Jan 2007 09:44:30 +0100 Subject: [Genome-mirror] performance tuning and other questions In-Reply-To: <6009f9d65b4e2ea86d717abb2adcc342@soe.ucsc.edu> References: <7224D442-755C-45A5-930D-E936C5163C35@ifom-ieo-campus.it> <6009f9d65b4e2ea86d717abb2adcc342@soe.ucsc.edu> Message-ID: <3809ED02-9EBD-49A0-947A-13C730BE9194@ifom-ieo-campus.it> Hi again, I'm back to work after NYD parties! On Dec 29, 2006, at 7:24 PM, Hiram Clawson wrote: > I suspect the performance situation you are observing may > be due to memory size. Run a 'top' display and watch your > memory usage during one of these requests that takes a lot > of time. As example I tried to visualize the entire chromosome 22 on hg18. On linux top says that hgTracks asks for 227Mb RAM, on FreeBSD 150Mb. Consider that the top command has a different output on these two systems. On both machines I have enough memory to satisfy this RAM allocation and some free memory remains anyway. On linux I see that hgTracks uses really few CPU, 1% and the machine load doesn't increase much. On FreeBSD I see a almost doubled load average and hgTracks using ~70% CPU. > > Our public WEB site machines are working with 8 Gb of memory. > AMD Opteron processor 850, 2.4Ghz > That is a "normal" machine (I mean, nothing supernatural!) > Or could there perhaps be a missing index on one of your SQL > tables ? I don't think so, as they query the same MySQL server. d > > --Hiram > > On 2006 Dec 29, , at 8:40 AM, Davide Cittaro wrote: > >> Hi again, I've tried to migrate the cgi stuff on a linux machine >> running on itanium2 processors. It takes much less time to visualize >> an entire chromosome (~3 min). Since a single ia64 CPU is not faster >> than a single Xeon (and hgTracks runs on a CPU) I don't think that >> this is a hardware issue. I suspect that the entire cgi binaries (at >> least hgTracks) simply run better on linux than freebsd. Can anybody >> confirm this? Is hgTracks somehow optimized for linux? >> >> d >> On Dec 28, 2006, at 3:18 PM, Davide Cittaro wrote: >> >>> Hi all, I've noticed that hgTracks process takes (here) some time to >>> process a very large data request whereas on your public servers it >>> is ~5 times faster, the corresponding mysql query is this: >>> >>> select * from snp126 where chrom='chr19' and (bin>=585 and bin<=1071 >>> or bin>=73 and bin<=133 or bin>=9 and bin<=16 or bin=1 or bin=0 or >>> bin=4681 ) and chromStart<63811651 and chromEnd>0 >>> >>> a rather big slice of chromosome 19 on hg18. >>> Actually mysql takes ~14s to process the query and most of the time >>> is taken by hgTracks to process it. >>> Therefore I have a couple of questions about overall performance: >>> - what kind of machine are you using to serve the public genome >>> browser? At least CPU/RAM (I suppose you are running linux). Here is >>> FreeBSD on a single Xeon and 4Gb RAM >>> - Are there some parameters (even at build time) I can tune to >>> increase performance? >>> - Is there a threaded hgTracks version? >>> >>> Thanks >>> >>> d >>> >>> /* >>> Davide Cittaro >>> HPC and Bioinformatics Systems @ Informatics Core >>> >>> IFOM - Istituto FIRC di Oncologia Molecolare >>> via adamello, 16 >>> 20139 Milano >>> Italy >>> >>> tel.: +39(02)574303355 >>> e-mail: davide.cittaro at ifom-ieo-campus.it >>> */ >>> >>> >>> _______________________________________________ >>> Genome-mirror mailing list >>> Genome-mirror at soe.ucsc.edu >>> http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror >> >> /* >> Davide Cittaro >> HPC and Bioinformatics Systems @ Informatics Core >> >> IFOM - Istituto FIRC di Oncologia Molecolare >> via adamello, 16 >> 20139 Milano >> Italy >> >> tel.: +39(02)574303355 >> e-mail: davide.cittaro at ifom-ieo-campus.it >> */ >> >> >> _______________________________________________ >> Genome-mirror mailing list >> Genome-mirror at soe.ucsc.edu >> http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror >> > /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303355 e-mail: davide.cittaro at ifom-ieo-campus.it */ From davide.cittaro at ifom-ieo-campus.it Tue Jan 2 08:14:54 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Tue, 2 Jan 2007 17:14:54 +0100 Subject: [Genome-mirror] performance tuning and other questions In-Reply-To: <3809ED02-9EBD-49A0-947A-13C730BE9194@ifom-ieo-campus.it> References: <7224D442-755C-45A5-930D-E936C5163C35@ifom-ieo-campus.it> <6009f9d65b4e2ea86d717abb2adcc342@soe.ucsc.edu> <3809ED02-9EBD-49A0-947A-13C730BE9194@ifom-ieo-campus.it> Message-ID: I performed the "worst case ever" test. I've tried to visualize chr22 in a freebsd 6.1 virtual machine on my mac, running with 768 Mb RAM and using the same mysql server of the other tests... It takes 41 minutes to have results, almost the same time I had on FreeBSD on a "real" machine. I suspect FreeBSD is not the best place where mirror Genome Browser :-) d On Jan 2, 2007, at 9:44 AM, Davide Cittaro wrote: > Hi again, I'm back to work after NYD parties! > > On Dec 29, 2006, at 7:24 PM, Hiram Clawson wrote: >> I suspect the performance situation you are observing may >> be due to memory size. Run a 'top' display and watch your >> memory usage during one of these requests that takes a lot >> of time. > > As example I tried to visualize the entire chromosome 22 on hg18. > On linux top says that hgTracks asks for 227Mb RAM, on FreeBSD 150Mb. > Consider that the top command has a different output on these two > systems. On both machines I have enough memory to satisfy this RAM > allocation and some free memory remains anyway. On linux I see that > hgTracks uses really few CPU, 1% and the machine load doesn't > increase much. On FreeBSD I see a almost doubled load average and > hgTracks using ~70% CPU. > >> >> Our public WEB site machines are working with 8 Gb of memory. >> AMD Opteron processor 850, 2.4Ghz >> > > That is a "normal" machine (I mean, nothing supernatural!) > >> Or could there perhaps be a missing index on one of your SQL >> tables ? > > I don't think so, as they query the same MySQL server. > > d > >> >> --Hiram >> >> On 2006 Dec 29, , at 8:40 AM, Davide Cittaro wrote: >> >>> Hi again, I've tried to migrate the cgi stuff on a linux machine >>> running on itanium2 processors. It takes much less time to visualize >>> an entire chromosome (~3 min). Since a single ia64 CPU is not faster >>> than a single Xeon (and hgTracks runs on a CPU) I don't think that >>> this is a hardware issue. I suspect that the entire cgi binaries (at >>> least hgTracks) simply run better on linux than freebsd. Can anybody >>> confirm this? Is hgTracks somehow optimized for linux? >>> >>> d >>> On Dec 28, 2006, at 3:18 PM, Davide Cittaro wrote: >>> >>>> Hi all, I've noticed that hgTracks process takes (here) some >>>> time to >>>> process a very large data request whereas on your public servers it >>>> is ~5 times faster, the corresponding mysql query is this: >>>> >>>> select * from snp126 where chrom='chr19' and (bin>=585 and >>>> bin<=1071 >>>> or bin>=73 and bin<=133 or bin>=9 and bin<=16 or bin=1 or bin=0 or >>>> bin=4681 ) and chromStart<63811651 and chromEnd>0 >>>> >>>> a rather big slice of chromosome 19 on hg18. >>>> Actually mysql takes ~14s to process the query and most of the time >>>> is taken by hgTracks to process it. >>>> Therefore I have a couple of questions about overall performance: >>>> - what kind of machine are you using to serve the public genome >>>> browser? At least CPU/RAM (I suppose you are running linux). >>>> Here is >>>> FreeBSD on a single Xeon and 4Gb RAM >>>> - Are there some parameters (even at build time) I can tune to >>>> increase performance? >>>> - Is there a threaded hgTracks version? >>>> >>>> Thanks >>>> >>>> d >>>> >>>> /* >>>> Davide Cittaro >>>> HPC and Bioinformatics Systems @ Informatics Core >>>> >>>> IFOM - Istituto FIRC di Oncologia Molecolare >>>> via adamello, 16 >>>> 20139 Milano >>>> Italy >>>> >>>> tel.: +39(02)574303355 >>>> e-mail: davide.cittaro at ifom-ieo-campus.it >>>> */ >>>> >>>> >>>> _______________________________________________ >>>> Genome-mirror mailing list >>>> Genome-mirror at soe.ucsc.edu >>>> http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror >>> >>> /* >>> Davide Cittaro >>> HPC and Bioinformatics Systems @ Informatics Core >>> >>> IFOM - Istituto FIRC di Oncologia Molecolare >>> via adamello, 16 >>> 20139 Milano >>> Italy >>> >>> tel.: +39(02)574303355 >>> e-mail: davide.cittaro at ifom-ieo-campus.it >>> */ >>> >>> >>> _______________________________________________ >>> Genome-mirror mailing list >>> Genome-mirror at soe.ucsc.edu >>> http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror >>> >> > > /* > Davide Cittaro > HPC and Bioinformatics Systems @ Informatics Core > > IFOM - Istituto FIRC di Oncologia Molecolare > via adamello, 16 > 20139 Milano > Italy > > tel.: +39(02)574303355 > e-mail: davide.cittaro at ifom-ieo-campus.it > */ > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303355 e-mail: davide.cittaro at ifom-ieo-campus.it */ From hiram at soe.ucsc.edu Tue Jan 2 09:41:32 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Tue, 02 Jan 2007 09:41:32 -0800 Subject: [Genome-mirror] performance tuning and other questions In-Reply-To: References: <7224D442-755C-45A5-930D-E936C5163C35@ifom-ieo-campus.it> <6009f9d65b4e2ea86d717abb2adcc342@soe.ucsc.edu> <3809ED02-9EBD-49A0-947A-13C730BE9194@ifom-ieo-campus.it> Message-ID: <459A994C.9060705@soe.ucsc.edu> Good Morning Davide: There is something highly suspicious about this situation. The FreeBSD should be able to run this business the same or even better than Linux. You have some kind of unusual bottleneck in the business there. Your Mac is already running BSD. That is what the Mac OS is. There is no need to run a virtual machine for BSD on a Mac. (unless you want some particular version of BSD) You can also use the /bin/time/ command to run the hgTracks: /usr/bin/time --verbose ./hgTracks db=hg18 position=chr22:1-49,691,432 (assuming this is the GNU time command) The --verbose flag will display lots of statistics about the command: Command being timed: "./hgTracks db=hg18 position=chr22:1-49,691,432" User time (seconds): 4.71 System time (seconds): 2.22 Percent of CPU this job got: 72% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:09.51 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 0 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 58556 Voluntary context switches: 82036 Involuntary context switches: 46 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 A note in the man page explains all the zeros you see in this Linux version output: > BUGS > Not all resources are measured by all versions of Unix, so some of the > values might be reported as zero. The present selection was mostly > inspired by the data provided by 4.2 or 4.3BSD. Evidently you should get a lot more numbers out of running it on BSD. --Hiram Davide Cittaro wrote: > I performed the "worst case ever" test. I've tried to visualize chr22 in > a freebsd 6.1 virtual machine on my mac, running with 768 Mb RAM and > using the same mysql server of the other tests... It takes 41 minutes to > have results, almost the same time I had on FreeBSD on a "real" machine. > I suspect FreeBSD is not the best place where mirror Genome Browser :-) > > d > On Jan 2, 2007, at 9:44 AM, Davide Cittaro wrote: > >> Hi again, I'm back to work after NYD parties! >> >> On Dec 29, 2006, at 7:24 PM, Hiram Clawson wrote: >>> I suspect the performance situation you are observing may >>> be due to memory size. Run a 'top' display and watch your >>> memory usage during one of these requests that takes a lot >>> of time. >> >> As example I tried to visualize the entire chromosome 22 on hg18. >> On linux top says that hgTracks asks for 227Mb RAM, on FreeBSD 150Mb. >> Consider that the top command has a different output on these two >> systems. On both machines I have enough memory to satisfy this RAM >> allocation and some free memory remains anyway. On linux I see that >> hgTracks uses really few CPU, 1% and the machine load doesn't >> increase much. On FreeBSD I see a almost doubled load average and >> hgTracks using ~70% CPU. >> >>> >>> Our public WEB site machines are working with 8 Gb of memory. >>> AMD Opteron processor 850, 2.4Ghz >>> >> >> That is a "normal" machine (I mean, nothing supernatural!) >> >>> Or could there perhaps be a missing index on one of your SQL >>> tables ? >> >> I don't think so, as they query the same MySQL server. >> >> d >> >>> >>> --Hiram >>> >>> On 2006 Dec 29, , at 8:40 AM, Davide Cittaro wrote: >>> >>>> Hi again, I've tried to migrate the cgi stuff on a linux machine >>>> running on itanium2 processors. It takes much less time to visualize >>>> an entire chromosome (~3 min). Since a single ia64 CPU is not faster >>>> than a single Xeon (and hgTracks runs on a CPU) I don't think that >>>> this is a hardware issue. I suspect that the entire cgi binaries (at >>>> least hgTracks) simply run better on linux than freebsd. Can anybody >>>> confirm this? Is hgTracks somehow optimized for linux? >>>> >>>> d >>>> On Dec 28, 2006, at 3:18 PM, Davide Cittaro wrote: >>>> >>>>> Hi all, I've noticed that hgTracks process takes (here) some time to >>>>> process a very large data request whereas on your public servers it >>>>> is ~5 times faster, the corresponding mysql query is this: >>>>> >>>>> select * from snp126 where chrom='chr19' and (bin>=585 and bin<=1071 >>>>> or bin>=73 and bin<=133 or bin>=9 and bin<=16 or bin=1 or bin=0 or >>>>> bin=4681 ) and chromStart<63811651 and chromEnd>0 >>>>> >>>>> a rather big slice of chromosome 19 on hg18. >>>>> Actually mysql takes ~14s to process the query and most of the time >>>>> is taken by hgTracks to process it. >>>>> Therefore I have a couple of questions about overall performance: >>>>> - what kind of machine are you using to serve the public genome >>>>> browser? At least CPU/RAM (I suppose you are running linux). Here is >>>>> FreeBSD on a single Xeon and 4Gb RAM >>>>> - Are there some parameters (even at build time) I can tune to >>>>> increase performance? >>>>> - Is there a threaded hgTracks version? >>>>> >>>>> Thanks >>>>> >>>>> d From davide.cittaro at ifom-ieo-campus.it Tue Jan 2 15:18:48 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Wed, 3 Jan 2007 00:18:48 +0100 Subject: [Genome-mirror] performance tuning and other questions In-Reply-To: <459A994C.9060705@soe.ucsc.edu> References: <7224D442-755C-45A5-930D-E936C5163C35@ifom-ieo-campus.it> <6009f9d65b4e2ea86d717abb2adcc342@soe.ucsc.edu> <3809ED02-9EBD-49A0-947A-13C730BE9194@ifom-ieo-campus.it> <459A994C.9060705@soe.ucsc.edu> Message-ID: <5EDB648D-87AD-4F3E-84CC-772010CD71EF@ifom-ieo-campus.it> Good Morning to you, Hiram On Jan 2, 2007, at 6:41 PM, Hiram Clawson wrote: > > Your Mac is already running BSD. That is what the Mac OS is. > There is no need to run a virtual machine for BSD on a Mac. > (unless you want some particular version of BSD) > I know it, I just wanted to see if newer version of FreeBSD has some improvements (6.x is pretty different from 5.x). I'll test GB also on Mac OS X to see how it behaves. I talked with some colleague who told me that it could be due to getc and similar calls that are implemented in different ways on linux and other unix. Since hgTracks is single threaded I've substituted every getc and fgetc with getc_unlocked hoping in a speed bump (I also modified jk*libs), without any improvement. The same colleague told me about strict POSIX compliance of *BSD, OS X and Solaris in contrast with coarser compliance of Linux (i.e. in getc calls). I would like to know if anybody in this mailing list hosts a local mirror on OS X or Solaris. > You can also use the /bin/time/ command to run the hgTracks: > /usr/bin/time --verbose ./hgTracks db=hg18 position=chr22:1-49,691,432 > (assuming this is the GNU time command) > /usr/bin/time -l -o ~/time ./hgTracks db=hg18 position=chr22:1-49,691,432 9.56 real 1.52 user 0.32 sys 78396 maximum resident set size 1147 average shared memory size 55266 average unshared data size 125 average unshared stack size 19299 page reclaims 0 page faults 0 swaps 0 block input operations 0 block output operations 1106 messages sent 5000 messages received 0 signals received 4961 voluntary context switches 260 involuntary context switches Mmm, it says 9.56 real, but I can assure it took much more (~45 min) d From c.simons at imb.uq.edu.au Tue Jan 2 20:54:24 2007 From: c.simons at imb.uq.edu.au (Cas) Date: Wed, 3 Jan 2007 14:54:24 +1000 Subject: [Genome-mirror] (no subject) Message-ID: Hi, I just wanted to ask if it was possible for you to put the danRer4 -> xenTro2 net file (danRer4.xenTro2.net.gz) up on the downloads server please? Thank you for your help, Cas. From ann at soe.ucsc.edu Wed Jan 3 16:17:06 2007 From: ann at soe.ucsc.edu (Ann Zweig) Date: Wed, 03 Jan 2007 16:17:06 -0800 Subject: [Genome-mirror] (no subject) In-Reply-To: References: Message-ID: <459C4782.3080103@cse.ucsc.edu> Hello Cas, Yes, I will make sure the file gets out there in the next few days (and will send you an email when it's there). This list is used more for sites who are mirroring the UCSC Genome Browser. A more appropriate list for questions such as yours is our genome mail list: genome at soe.ucsc.edu. Regards, ---------- Ann Zweig UCSC Genome Bioinformatics Group http://genome.ucsc.edu Cas wrote: > Hi, > > I just wanted to ask if it was possible for you to put the danRer4 -> > xenTro2 net file (danRer4.xenTro2.net.gz) up on the downloads server > please? > > Thank you for your help, > > Cas. > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror From davide.cittaro at ifom-ieo-campus.it Thu Jan 4 06:11:05 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Thu, 4 Jan 2007 15:11:05 +0100 Subject: [Genome-mirror] performance tuning and other questions In-Reply-To: <5EDB648D-87AD-4F3E-84CC-772010CD71EF@ifom-ieo-campus.it> References: <7224D442-755C-45A5-930D-E936C5163C35@ifom-ieo-campus.it> <6009f9d65b4e2ea86d717abb2adcc342@soe.ucsc.edu> <3809ED02-9EBD-49A0-947A-13C730BE9194@ifom-ieo-campus.it> <459A994C.9060705@soe.ucsc.edu> <5EDB648D-87AD-4F3E-84CC-772010CD71EF@ifom-ieo-campus.it> Message-ID: <261D0D06-02E5-4F9B-8FB8-8CB23DC026D8@ifom-ieo-campus.it> A real case test 2923.84 real 1988.24 user 8.18 sys 157364 maximum resident set size 1279 average shared memory size 16163 average unshared data size 127 average unshared stack size 39606 page reclaims 0 page faults 0 swaps 362 block input operations 1 block output operations 40625 messages sent 47244 messages received 0 signals received 46067 voluntary context switches 265372 involuntary context switches On Jan 3, 2007, at 12:18 AM, Davide Cittaro wrote: > /usr/bin/time -l -o ~/time ./hgTracks db=hg18 > position=chr22:1-49,691,432 > 9.56 real 1.52 user 0.32 sys > 78396 maximum resident set size > 1147 average shared memory size > 55266 average unshared data size > 125 average unshared stack size > 19299 page reclaims > 0 page faults > 0 swaps > 0 block input operations > 0 block output operations > 1106 messages sent > 5000 messages received > 0 signals received > 4961 voluntary context switches > 260 involuntary context switches /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303355 e-mail: davide.cittaro at ifom-ieo-campus.it */ From hiram at soe.ucsc.edu Thu Jan 4 10:12:56 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Thu, 4 Jan 2007 10:12:56 -0800 Subject: [Genome-mirror] performance tuning and other questions In-Reply-To: <261D0D06-02E5-4F9B-8FB8-8CB23DC026D8@ifom-ieo-campus.it> References: <7224D442-755C-45A5-930D-E936C5163C35@ifom-ieo-campus.it> <6009f9d65b4e2ea86d717abb2adcc342@soe.ucsc.edu> <3809ED02-9EBD-49A0-947A-13C730BE9194@ifom-ieo-campus.it> <459A994C.9060705@soe.ucsc.edu> <5EDB648D-87AD-4F3E-84CC-772010CD71EF@ifom-ieo-campus.it> <261D0D06-02E5-4F9B-8FB8-8CB23DC026D8@ifom-ieo-campus.it> Message-ID: <284216a6046d0c75903ee06c6eeb9368@soe.ucsc.edu> Look at the huge number of involuntary context switches. Your system is extremely busy with some other task. On 2007 Jan 04, , at 6:11 AM, Davide Cittaro wrote: > A real case test > > 2923.84 real? ? ? 1988.24 user? ? ? ?? 8.18 sys > ? ? 157364? maximum resident set size > ? ? ? 1279? average shared memory size > ? ?? 16163? average unshared data size > ? ? ?? 127? average unshared stack size > ? ?? 39606? page reclaims > ? ? ? ?? 0? page faults > ? ? ? ?? 0? swaps > ? ? ?? 362? block input operations > ? ? ? ?? 1? block output operations > ? ?? 40625? messages sent > ? ?? 47244? messages received > ? ? ? ?? 0? signals received > ? ?? 46067? voluntary context switches > ? ? 265372? involuntary context switches > > On Jan 3, 2007, at 12:18 AM, Davide Cittaro wrote: > >> /usr/bin/time -l -o ~/time ./hgTracks db=hg18 >> position=chr22:1-49,691,432 >> ? 9.56 real 1.52 user 0.32 sys >> ? 78396 maximum resident set size >> ? 1147 average shared memory size >> ? 55266 average unshared data size >> ? 125 average unshared stack size >> ? 19299 page reclaims >> ? 0 page faults >> ? 0 swaps >> ? 0 block input operations >> ? 0 block output operations >> ? 1106 messages sent >> ? 5000 messages received >> ? 0 signals received >> ? 4961 voluntary context switches >> ? 260 involuntary context switches > > /* > Davide Cittaro > HPC and Bioinformatics Systems @ Informatics Core > > IFOM - Istituto FIRC di Oncologia Molecolare > via adamello, 16 > 20139 Milano > Italy > > tel.: +39(02)574303355 > e-mail:?davide.cittaro at ifom-ieo-campus.it > */ > > > From davide.cittaro at ifom-ieo-campus.it Thu Jan 4 11:49:43 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Thu, 4 Jan 2007 20:49:43 +0100 Subject: [Genome-mirror] performance tuning and other questions In-Reply-To: <284216a6046d0c75903ee06c6eeb9368@soe.ucsc.edu> References: <7224D442-755C-45A5-930D-E936C5163C35@ifom-ieo-campus.it> <6009f9d65b4e2ea86d717abb2adcc342@soe.ucsc.edu> <3809ED02-9EBD-49A0-947A-13C730BE9194@ifom-ieo-campus.it> <459A994C.9060705@soe.ucsc.edu> <5EDB648D-87AD-4F3E-84CC-772010CD71EF@ifom-ieo-campus.it> <261D0D06-02E5-4F9B-8FB8-8CB23DC026D8@ifom-ieo-campus.it> <284216a6046d0c75903ee06c6eeb9368@soe.ucsc.edu> Message-ID: <66064D37-E4C1-4CDD-9CD3-B1A9CF5541D9@ifom-ieo-campus.it> On Jan 4, 2007, at 7:12 PM, Hiram Clawson wrote: > Look at the huge number of involuntary context switches. > Your system is extremely busy with some other task. > I've seen that, but also I installed everything on OS X and it takes the same time even without that number of icsw. BTW, I'm in contact with some FreeBSD developers who, hopefully, will help me to find out what's going on d > > On 2007 Jan 04, , at 6:11 AM, Davide Cittaro wrote: > >> A real case test >> >> 2923.84 real 1988.24 user 8.18 sys >> 157364 maximum resident set size >> 1279 average shared memory size >> 16163 average unshared data size >> 127 average unshared stack size >> 39606 page reclaims >> 0 page faults >> 0 swaps >> 362 block input operations >> 1 block output operations >> 40625 messages sent >> 47244 messages received >> 0 signals received >> 46067 voluntary context switches >> 265372 involuntary context switches >> >> On Jan 3, 2007, at 12:18 AM, Davide Cittaro wrote: >> >>> /usr/bin/time -l -o ~/time ./hgTracks db=hg18 >>> position=chr22:1-49,691,432 >>> 9.56 real 1.52 user 0.32 sys >>> 78396 maximum resident set size >>> 1147 average shared memory size >>> 55266 average unshared data size >>> 125 average unshared stack size >>> 19299 page reclaims >>> 0 page faults >>> 0 swaps >>> 0 block input operations >>> 0 block output operations >>> 1106 messages sent >>> 5000 messages received >>> 0 signals received >>> 4961 voluntary context switches >>> 260 involuntary context switches >> >> /* >> Davide Cittaro >> HPC and Bioinformatics Systems @ Informatics Core >> >> IFOM - Istituto FIRC di Oncologia Molecolare >> via adamello, 16 >> 20139 Milano >> Italy >> >> tel.: +39(02)574303355 >> e-mail: davide.cittaro at ifom-ieo-campus.it >> */ >> >> >> > From hartera at soe.ucsc.edu Thu Jan 4 16:03:30 2007 From: hartera at soe.ucsc.edu (Rachel Harte) Date: Thu, 4 Jan 2007 16:03:30 -0800 (PST) Subject: [Genome-mirror] Is there a quick way to edit cutter.c file so that it knows the number of times each restriction enzyme site appear in the genome. Message-ID: Dear Weerayuth, Firstly, sorry for the delay in replying to your question - we have been catching up with the holiday backlog of questions. The Restriction Enzymes track is created on the fly so when you select a position on the Browser, the Restriction Enzymes found in that region are computed and displayed at that moment. I am not sure then that it is so easy to edit the code to calculate the number of times each restriction enzyme appears in the whole genome. However you could download the genome sequence and find the positions of all of the restriction enzymes and then calculate this number from the results. Here is an answer to a previous question that will help you to do this: https://www.soe.ucsc.edu/pipermail/genome/2006-May/010707.html The restriction enzyme sequences may be found in a database called hgFixed and they can be downloaded using the Table Browser. To reach this, click on the Tables link on the top blue bar of the Genome Browser pages. >From the "group" pulldown menu, select "All Tables" and then select hgFixed as the database. Then select hgFixed.cutters as the table and you can download the name and seq fields of the table to use in creating a FASTA file of restriction enzyme sequences. I am also checking with the developer of this track to see if he has any suggestions of how you could edit the Browser code to get a calculation of the number of times each restrictin enzyme cuts in a genome. I hope that this helps you. Please let us know if you have further questions. Rachel Weerayuth Kittichotirat weerayuth at gmail.com Wed Dec 27 10:25:47 PST 2006 Good morning, Specifically, I am working with an oral bacteria genome which is about 2.1megabase. The UCSC Genome Browser has been very useful to me as I have been using it to visualize this small genome, its annotation, etc. However, now that I would like to be able to specifically visualize restriction enzyme sites according to number of times they appear in the whole genome, eg. only show restriction tracks if they appear less than 10 times in the whole genome. I have spent the past few days looking at the source code and believe that the functionality of restriction enzyme sites track is written in the " cutter.c" file located in the /src/hg/lib folder. However, with my limited c programming skill, so far it was a miseral experience trying to edit that file by myself. Therefore, I would like to ask the developer team and everyone whether is there a quick way to edit cutter.c file (and/or other source files) so that it knows number of times each restriction enzyme site appear in the whole genome and have the functionality of being able to display only sites that appear less than certain number of times. All suggestions are, as always, greatly appreciated. Have a happy holidays everyone. Regards, WK Rachel Harte UCSC Genome Bioinformatics Group http://genome.ucsc.edu From davide.cittaro at ifom-ieo-campus.it Fri Jan 5 06:15:46 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Fri, 5 Jan 2007 15:15:46 +0100 Subject: [Genome-mirror] hgTracks bug (was: performance tuning and other questions) Message-ID: <78252D54-DC3D-499C-B989-F6FC4753B679@ifom-ieo-campus.it> Hi all, we (me and my chief Andrea Cocito) have found a bug in hgTracks that slows its execution on *NIX systems other than Linux. We have found that compare function snp125ExtendedColorCmp (variation.c:350) for qsort (common.c:336) never handle the case two elements are equal. I don't know how qsort is implemented on linux, but on other *NIX it has some optimizations that make qsort "going crazy" if compare function pass equal elements as reversed or ordered: int snp125ExtendedColorCmp(const void *va, const void *vb) /* Compare to sort based on color -- black first, red last */ { [...] if (ca==MG_RED) return 1; if (cb==MG_RED) return -1; if (ca==MG_GREEN) return 1; if (cb==MG_GREEN) return -1; [...] } if ca and cb are MG_RED, this function says once that ca is greater than cb and that cb is greater than ca when called in "reversed" way. We have modified it adding the "equal case" handling and compiled everything. Now it perfectly runs on our system. Since comparing are a tricotomy, we think it would be necessary to apply this patch even on linux systems, it shouldn't be harmful (although we haven't tested it on linux). BTW, here is the patch: [patch start] ~/kent/src/hg/hgTracks diff -Naur variation.c variation.c.old --- variation.c Fri Jan 5 15:14:20 2007 +++ variation.c.old Thu Dec 7 23:19:07 2006 @@ -356,13 +356,6 @@ const Color cb = b->color; /* order is important here */ -/* Found missing handler of equal elements - added here at the beginning - by Andrea Cocito and Davide Cittaro - IFOM, Milan, Italy - Jan 5th, 2006 */ -if (ca==cb) - return 0; if (ca==MG_RED) return 1; if (cb==MG_RED) [patch end] I suggest to everybody who is hosting an UCSC mirror on FreeBSD, OS X or possibly Solaris to apply this patch HTH Davide Cittaro /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303355 e-mail: davide.cittaro at ifom-ieo-campus.it */ From davide.cittaro at ifom-ieo-campus.it Fri Jan 5 07:20:23 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Fri, 5 Jan 2007 16:20:23 +0100 Subject: [Genome-mirror] follow up on hgTracks bug Message-ID: <1613FAE0-FF41-43B2-97C9-F87BC761E0B1@ifom-ieo-campus.it> The function slSort uses array which relies on the library function qsort() and passes to it snp125ExtendedColorCmp as compare function. snp125ExtendedColorCmp failed in correctly returning 0 for identical colors, thus violating the POSIX specified semantics for a valid comparison function passed to qsort, in example a compare(a, b) in which both a and b are "red" returned 1 (a > b) while obviously compare(b, a) returned again 1 (b > a) and this is against posix specification IEEE Std 1003.1 that implies trichotomic ordering: "When the same objects (consisting of width bytes, irrespective of their current positions in the array) are passed more than once to the comparison function, the results shall be consistent with one another. That is, they shall define a total ordering on the array.". A function stating that a>b and b>a does not define a total ordering of the array. The linux libc uses a vanilla implementation of the qsort algorithm (which in some cases can degenerate to an O(n**2) time complexity), the *BSD libc implements qsort with a median selection twak to avoid O (N**2) degeneration but this exposes it to a severe performance degradation with a compare function that does not respect the trichotomic principle. The solution is either to use a different ordering algorithm (in example mergesort) or to fix the comparison function (such that it correctly returns zero when two objects in the list have the same color) or, at best, redefining the process to split the list in an array of sublists (one per color) and then concatenate them; the latter would be the faster way since it could be done in place without copying the list into an array etc. /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303355 e-mail: davide.cittaro at ifom-ieo-campus.it */ From davide.cittaro at ifom-ieo-campus.it Fri Jan 5 07:51:14 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Fri, 5 Jan 2007 16:51:14 +0100 Subject: [Genome-mirror] hgTables error Message-ID: <6CF413FD-4058-4AA0-B929-A02BDDFFCD2F@ifom-ieo-campus.it> Hi there, suddenly I have this error while trying to open hgTables: Need at least one database before . in table spec line 2048 of all.joiner I haven't modified all.joiner file and it is the same you host on your main site... What does this mean? And, more important, any hint to fix it? d /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303355 e-mail: davide.cittaro at ifom-ieo-campus.it */ From heather at soe.ucsc.edu Fri Jan 5 08:04:46 2007 From: heather at soe.ucsc.edu (Heather Trumbower) Date: Fri, 5 Jan 2007 08:04:46 -0800 (PST) Subject: [Genome-mirror] hgTables error In-Reply-To: <6CF413FD-4058-4AA0-B929-A02BDDFFCD2F@ifom-ieo-campus.it> References: <6CF413FD-4058-4AA0-B929-A02BDDFFCD2F@ifom-ieo-campus.it> Message-ID: David: Have you checked whether this is coming from kent/src/hg/lib/joiner.c parseTableSpec() table->dbList = parseDatabaseList(lf, spec); if (table->dbList == NULL) errAbort("Need at least one database before . in table spec line %d of %s", lf->lineIx, lf->fileName); Heather Trumbower UCSC Genome Bioinformatics Group On Fri, 5 Jan 2007, Davide Cittaro wrote: > Hi there, suddenly I have this error while trying to open hgTables: > > Need at least one database before . in table spec line 2048 of > all.joiner > > I haven't modified all.joiner file and it is the same you host on > your main site... > What does this mean? And, more important, any hint to fix it? > > d > /* > Davide Cittaro > HPC and Bioinformatics Systems @ Informatics Core > > IFOM - Istituto FIRC di Oncologia Molecolare > via adamello, 16 > 20139 Milano > Italy > > tel.: +39(02)574303355 > e-mail: davide.cittaro at ifom-ieo-campus.it > */ > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From heather at soe.ucsc.edu Fri Jan 5 08:11:21 2007 From: heather at soe.ucsc.edu (Heather Trumbower) Date: Fri, 5 Jan 2007 08:11:21 -0800 (PST) Subject: [Genome-mirror] hgTracks bug (was: performance tuning and other questions) In-Reply-To: <78252D54-DC3D-499C-B989-F6FC4753B679@ifom-ieo-campus.it> References: <78252D54-DC3D-499C-B989-F6FC4753B679@ifom-ieo-campus.it> Message-ID: Davide: Thanks very much. Our apologies for the error. I have patched if (ca==cb) return 0; into sortSnp125ExtendedColorCmp() in hgTracks/variation.c, it is revision 1.118. We are doing release v149 today, it will be included in that, and therefore available very quickly to all mirror sites. Thanks again. Heather Trumbower UCSC Genome Bioinformatics Group On Fri, 5 Jan 2007, Davide Cittaro wrote: > Hi all, we (me and my chief Andrea Cocito) have found a bug in > hgTracks that slows its execution on *NIX systems other than Linux. > We have found that compare function snp125ExtendedColorCmp > (variation.c:350) for qsort (common.c:336) never handle the case two > elements are equal. I don't know how qsort is implemented on linux, > but on other *NIX it has some optimizations that make qsort "going > crazy" if compare function pass equal elements as reversed or ordered: > > int snp125ExtendedColorCmp(const void *va, const void *vb) > /* Compare to sort based on color -- black first, red last */ > { > [...] > if (ca==MG_RED) > return 1; > if (cb==MG_RED) > return -1; > if (ca==MG_GREEN) > return 1; > if (cb==MG_GREEN) > return -1; > [...] > } > > if ca and cb are MG_RED, this function says once that ca is greater > than cb and that cb is greater than ca when called in "reversed" way. > We have modified it adding the "equal case" handling and compiled > everything. Now it perfectly runs on our system. > Since comparing are a tricotomy, we think it would be necessary to > apply this patch even on linux systems, it shouldn't be harmful > (although we haven't tested it on linux). > BTW, here is the patch: > > [patch start] > > ~/kent/src/hg/hgTracks diff -Naur variation.c variation.c.old > --- variation.c Fri Jan 5 15:14:20 2007 > +++ variation.c.old Thu Dec 7 23:19:07 2006 > @@ -356,13 +356,6 @@ > const Color cb = b->color; > /* order is important here */ > -/* Found missing handler of equal elements > - added here at the beginning > - by Andrea Cocito and Davide Cittaro > - IFOM, Milan, Italy > - Jan 5th, 2006 */ > -if (ca==cb) > - return 0; > if (ca==MG_RED) > return 1; > if (cb==MG_RED) > > [patch end] > > I suggest to everybody who is hosting an UCSC mirror on FreeBSD, OS X > or possibly Solaris to apply this patch > HTH > > Davide Cittaro > > /* > Davide Cittaro > HPC and Bioinformatics Systems @ Informatics Core > > IFOM - Istituto FIRC di Oncologia Molecolare > via adamello, 16 > 20139 Milano > Italy > > tel.: +39(02)574303355 > e-mail: davide.cittaro at ifom-ieo-campus.it > */ > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From heather at soe.ucsc.edu Fri Jan 5 08:18:04 2007 From: heather at soe.ucsc.edu (Heather Trumbower) Date: Fri, 5 Jan 2007 08:18:04 -0800 (PST) Subject: [Genome-mirror] follow up on hgTracks bug In-Reply-To: <1613FAE0-FF41-43B2-97C9-F87BC761E0B1@ifom-ieo-campus.it> References: <1613FAE0-FF41-43B2-97C9-F87BC761E0B1@ifom-ieo-campus.it> Message-ID: Davide: Thanks for the additional analysis. As I wrote earlier, our immediate response is that we have patched the comparison function to include the equivalence case. I will consult with my colleagues regarding whether we might consider using different ordering algorithms, for example mergesort as you suggest. Heather Trumbower UCSC Genome Bioinformatics Group On Fri, 5 Jan 2007, Davide Cittaro wrote: > > The function slSort uses array which relies on the library function > qsort() and passes to it snp125ExtendedColorCmp as compare function. > > snp125ExtendedColorCmp failed in correctly returning 0 for identical > colors, thus violating the POSIX specified semantics for a valid > comparison function passed to qsort, in example a compare(a, b) in > which both a and b are "red" returned 1 (a > b) while obviously > compare(b, a) returned again 1 (b > a) and this is against posix > specification IEEE Std 1003.1 that implies trichotomic ordering: > "When the same objects (consisting of width bytes, irrespective of > their current positions in the array) are passed more than once to > the comparison function, the results shall be consistent with one > another. That is, they shall define a total ordering on the array.". > A function stating that a>b and b>a does not define a total ordering > of the array. > > The linux libc uses a vanilla implementation of the qsort algorithm > (which in some cases can degenerate to an O(n**2) time complexity), > the *BSD libc implements qsort with a median selection twak to avoid O > (N**2) degeneration but this exposes it to a severe performance > degradation with a compare function that does not respect the > trichotomic principle. > > The solution is either to use a different ordering algorithm (in > example mergesort) or to fix the comparison function (such that it > correctly returns zero when two objects in the list have the same > color) or, at best, redefining the process to split the list in an > array of sublists (one per color) and then concatenate them; the > latter would be the faster way since it could be done in place > without copying the list into an array etc. > > /* > Davide Cittaro > HPC and Bioinformatics Systems @ Informatics Core > > IFOM - Istituto FIRC di Oncologia Molecolare > via adamello, 16 > 20139 Milano > Italy > > tel.: +39(02)574303355 > e-mail: davide.cittaro at ifom-ieo-campus.it > */ > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From davide.cittaro at ifom-ieo-campus.it Fri Jan 5 08:21:22 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Fri, 5 Jan 2007 17:21:22 +0100 Subject: [Genome-mirror] hgTables error In-Reply-To: References: <6CF413FD-4058-4AA0-B929-A02BDDFFCD2F@ifom-ieo-campus.it> Message-ID: I was trying to debug it... I recompiled with gdb symbols and it worked... I've recompiled without and it works... Pretty strange, huh? Ok, I think we can forget about this... d On Jan 5, 2007, at 5:04 PM, Heather Trumbower wrote: > David: > > Have you checked whether this is coming from kent/src/hg/lib/ > joiner.c parseTableSpec() > > table->dbList = parseDatabaseList(lf, spec); > if (table->dbList == NULL) > errAbort("Need at least one database before . in table spec > line %d of %s", > lf->lineIx, lf->fileName); > > Heather Trumbower > UCSC Genome Bioinformatics Group > > > On Fri, 5 Jan 2007, Davide Cittaro wrote: > >> Hi there, suddenly I have this error while trying to open hgTables: >> >> Need at least one database before . in table spec line 2048 of >> all.joiner >> >> I haven't modified all.joiner file and it is the same you host on >> your main site... >> What does this mean? And, more important, any hint to fix it? >> >> d >> /* >> Davide Cittaro >> HPC and Bioinformatics Systems @ Informatics Core >> >> IFOM - Istituto FIRC di Oncologia Molecolare >> via adamello, 16 >> 20139 Milano >> Italy >> >> tel.: +39(02)574303355 >> e-mail: davide.cittaro at ifom-ieo-campus.it >> */ >> >> >> _______________________________________________ >> Genome-mirror mailing list >> Genome-mirror at soe.ucsc.edu >> http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror >> /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303355 e-mail: davide.cittaro at ifom-ieo-campus.it */ From davide.cittaro at ifom-ieo-campus.it Fri Jan 5 08:27:45 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Fri, 5 Jan 2007 17:27:45 +0100 Subject: [Genome-mirror] follow up on hgTracks bug In-Reply-To: References: <1613FAE0-FF41-43B2-97C9-F87BC761E0B1@ifom-ieo-campus.it> Message-ID: Hi Heather, On Jan 5, 2007, at 5:18 PM, Heather Trumbower wrote: > > I will consult with my colleagues regarding whether we might > consider using different ordering algorithms, for example mergesort > as you suggest. > qsort is great, and with an appropriate compare function works perfectly. What we have thought is rather different. You have a qsort call to sort many kind of data. In this particular case, you have only 5 classes and therefore it may be faster to split your list by color into 5 different lists and then concatenate them. In this way you don't have to sort anything. cheers d > Heather Trumbower > UCSC Genome Bioinformatics Group > > > On Fri, 5 Jan 2007, Davide Cittaro wrote: > >> >> The function slSort uses array which relies on the library function >> qsort() and passes to it snp125ExtendedColorCmp as compare function. >> >> snp125ExtendedColorCmp failed in correctly returning 0 for identical >> colors, thus violating the POSIX specified semantics for a valid >> comparison function passed to qsort, in example a compare(a, b) in >> which both a and b are "red" returned 1 (a > b) while obviously >> compare(b, a) returned again 1 (b > a) and this is against posix >> specification IEEE Std 1003.1 that implies trichotomic ordering: >> "When the same objects (consisting of width bytes, irrespective of >> their current positions in the array) are passed more than once to >> the comparison function, the results shall be consistent with one >> another. That is, they shall define a total ordering on the array.". >> A function stating that a>b and b>a does not define a total ordering >> of the array. >> >> The linux libc uses a vanilla implementation of the qsort algorithm >> (which in some cases can degenerate to an O(n**2) time complexity), >> the *BSD libc implements qsort with a median selection twak to >> avoid O >> (N**2) degeneration but this exposes it to a severe performance >> degradation with a compare function that does not respect the >> trichotomic principle. >> >> The solution is either to use a different ordering algorithm (in >> example mergesort) or to fix the comparison function (such that it >> correctly returns zero when two objects in the list have the same >> color) or, at best, redefining the process to split the list in an >> array of sublists (one per color) and then concatenate them; the >> latter would be the faster way since it could be done in place >> without copying the list into an array etc. >> >> /* >> Davide Cittaro >> HPC and Bioinformatics Systems @ Informatics Core >> >> IFOM - Istituto FIRC di Oncologia Molecolare >> via adamello, 16 >> 20139 Milano >> Italy >> >> tel.: +39(02)574303355 >> e-mail: davide.cittaro at ifom-ieo-campus.it >> */ >> >> >> _______________________________________________ >> Genome-mirror mailing list >> Genome-mirror at soe.ucsc.edu >> http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror >> /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303355 e-mail: davide.cittaro at ifom-ieo-campus.it */ From heather at soe.ucsc.edu Fri Jan 5 08:33:45 2007 From: heather at soe.ucsc.edu (Heather Trumbower) Date: Fri, 5 Jan 2007 08:33:45 -0800 (PST) Subject: [Genome-mirror] follow up on hgTracks bug In-Reply-To: References: <1613FAE0-FF41-43B2-97C9-F87BC761E0B1@ifom-ieo-campus.it> Message-ID: Davide: Got it, thanks. I'm hoping to refactor this code in the next few weeks, I appreciate your input. Heather On Fri, 5 Jan 2007, Davide Cittaro wrote: > What we have though is rather different. You have a qsort call to sort many > kinds of data. In this particular case, you have only 5 classes and therefore > it may be faster to split your list by color into 5 different lists and then > concatenate them. In this way you don't have to sort anything. From kuhn at soe.ucsc.edu Fri Jan 5 11:21:11 2007 From: kuhn at soe.ucsc.edu (Robert Kuhn) Date: Fri, 5 Jan 2007 11:21:11 -0800 Subject: [Genome-mirror] drop some files Message-ID: <200701051921.LAA29368@moondance.cse.ucsc.edu> mirror sites, we have dropped a number of directories from /gbdb directory because we no longer support the functionality for which they were used. most are from archived assemblies anyway, but you may wish to drop any you find on your system. here is the list. thanks, --b0b kuhn ucsc genome bioinformatics group > ./hg10/axtBestMm2 > ./hg12/axtBestHg12 > ./hg12/axtBestMm2 > ./hg13/axtBestHg13 > ./hg13/axtBestMm2 > ./hg13/axtBestMm3 > ./hg13/axtBestRn1 > ./hg13/axtBestRn2 > ./hg15/axtBestMm3 > ./hg16/axtBestMm3 > ./hg16/axtBestMm4 > ./hg16/axtBestRn3 > ./mm2/axtBestHg12 > ./mm2/axtBestHg13 > ./mm2/axtBestMm2 > ./mm2/axtBestRn1 > ./mm3/axtBestHg13 > ./mm3/axtBestMm3 > ./mm3/axtBestRn2 > ./mm4/axtBestHg16 > ./mm4/axtBestRn3 > ./rn2/axtBestHg13 > [48 Gb] From maronsza at fhcrc.org Fri Jan 5 12:40:45 2007 From: maronsza at fhcrc.org (Aronszajn, Mark) Date: Fri, 5 Jan 2007 12:40:45 -0800 Subject: [Genome-mirror] minimum requirements for a mirror Message-ID: Hi, We want to test a particular installation set-up of the GenomeBrowser here at Fred Hutchinson. It would be as follows: the application, web server and MySQL server would all be running on a linux machine, but a large share of the storage would be on NFS-mounted drives on a different machine (a Sun). Briefly, one reason we want to test this is that we already have tried running the app & servers on the same machine (the Sun) and found that the resulting GeneBrowser performance was somewhat slower than the UCSC instance itself, despite being local, and we thought that the app & servers might run faster on the linux machine. I have a few questions about this idea, and it would be great if someone could help us: 1) Any opinion on the idea that the app & servers might indeed run faster on Linux than on Sun (Solaris)? 2) The Linux box currently has 4 GBs RAM, 2 X 1.26 GHz processors. What are reasonable minimum requirements for RAM and processor speed, and number of processors on the machine that would be running the app & servers? 3) Has anyone heard if there are any serious drawbacks to having much of the storage (MySQL Table spaces, /gbdb and location for Genome db text dumps) on a mounted drive? 4) We have a version of RedHat running on the linux machine right now, 2.6.10-1.9_FC2smp #1, that is earlier than the version you all recommend for installing the GeneBrowser binary. Would it be feasible to build & install from source and stick with the RedHat version, or should we bite the bullet and upgrade the version? 5) Is there a mailing list now for people (multitudes surely) who are struggling with GenomeBrowser mirror installs? Thanks in advance for any feedback or answers you can give us to these questions, Mark Aronszajn Biocomputing Fred Hutchinson Cancer Research Center From hartera at soe.ucsc.edu Mon Jan 8 10:22:15 2007 From: hartera at soe.ucsc.edu (Rachel Harte) Date: Mon, 8 Jan 2007 10:22:15 -0800 (PST) Subject: [Genome-mirror] v149 Genome Browser available Message-ID: v149 source is at: http://hgdownload.cse.ucsc.edu/admin/jksrc.zip or labeled with source number: http://hgdownload.cse.ucsc.edu/admin/jksrc.v149.zip cartDump cartReset das hgc hgBlat hgConvert hgCustom hgEncodeDataVersions hgGateway hgGene hgLiftOver hgNear hgPcr hgTables hgText hgTracks hgTrackUi hgVisiGene mkEncodeFrameset pbGateway pbGlobal pbTracks phyloGif /usr/local/apache/cgi-bin/all.joiner /usr/local/apache/cgi-bin/hgNearData/* /usr/local/apache/cgi-bin/hgGeneData/* /usr/local/apache/cgi-bin/hgcData/* /usr/local/apache/cgi-bin/hgCgiData/* Contents of v149 may be seen at: http://genome-test.cse.ucsc.edu/builds/versions.html Please check both v149-preview and v149-final to see all the code and data changes in this release. Rachel Harte UCSC Genome Bioinformatics Group http://genome.ucsc.edu From maronsza at fhcrc.org Mon Jan 8 11:51:54 2007 From: maronsza at fhcrc.org (Aronszajn, Mark) Date: Mon, 8 Jan 2007 11:51:54 -0800 Subject: [Genome-mirror] hardware requirements apart from storage Message-ID: Hi Patrick, I don't know if you all got my latest inquiry... I'm embarrassed to admit that I had forgotten about your reply to a similar inquiry I sent last May. At any rate, there are a few new questions about getting a local mirror set up here at Fred Hutchinson. If you're still working with Genome Browser mirroring questions & issues, could you give a hand? We want to test a particular installation set-up of the Genome Browser here at Fred Hutchinson. It would be as follows: the application, web server and MySQL server would all be running on a linux machine, but a large share of the storage would be on NFS-mounted drives on a different machine (a Sun). Briefly, one reason we want to test this is that we already have tried running the app & servers on the same machine (the Sun) and found that the resulting GeneBrowser performance was somewhat slower than the UCSC instance itself, despite being local, and we thought that the app & servers might run faster on the linux machine. I have a few questions about this idea, and it would be great if someone could help us: 1) Any opinion on the idea that the app & servers might indeed run faster on Linux than on Sun (Solaris)? 2) Has anyone heard if there are any serious drawbacks to having much of the storage (MySQL Table spaces, /gbdb and location for Genome db text dumps) on a mounted drive? 3) We have a version of RedHat running on the linux machine right now, 2.6.10-1.9_FC2smp #1, that is earlier than the version you all recommend for installing the GeneBrowser binary. Would it be feasible to build & install from source and stick with the RedHat version, or should we bite the bullet and upgrade the version? Thanks in advance for any feedback or answers you can give us to these questions, Mark Aronszajn Biocomputing Fred Hutchinson Cancer Research Center From davide.cittaro at ifom-ieo-campus.it Mon Jan 8 15:24:50 2007 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Tue, 9 Jan 2007 00:24:50 +0100 Subject: [Genome-mirror] hardware requirements apart from storage In-Reply-To: References: Message-ID: Hi Mark, I can tell you our experience On Jan 8, 2007, at 8:51 PM, Aronszajn, Mark wrote: > > 1) Any opinion on the idea that the app & servers might indeed run > faster on Linux than on Sun (Solaris)? > I've never tried on Solaris but I suspect it has the same problems that other *NIX have (*BSD, OS X, not linux), a recent bugfix solved this. Also, I can tell now (after bugfixes) that FreeBSD slightly outperforms linux (Single Xeon 4 Gb RAM versus quad itanium2 8 Gb RAM respectively). > 2) Has anyone heard if there are any serious drawbacks to having > much of > the storage (MySQL Table spaces, /gbdb and location for Genome db text > dumps) on a mounted drive? It's not a good idea to mount mysql tables on NFS as you have to handle several locks on NFS, it may be better to run MySQL on your Solaris machine and then connect to it at yoursever:3306 I have MySQL tables and /gbdb on a fiber channel little disk and export /gbdb area with NFS to all blat servers and other machines on which users have an account (read only) /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core IFOM - Istituto FIRC di Oncologia Molecolare via adamello, 16 20139 Milano Italy tel.: +39(02)574303355 e-mail: davide.cittaro at ifom-ieo-campus.it */ From hiram at soe.ucsc.edu Mon Jan 8 16:21:28 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Mon, 08 Jan 2007 16:21:28 -0800 Subject: [Genome-mirror] [ hardware requirements apart from storage In-Reply-To: <45A2CE7F.4020607@cse.ucsc.edu> References: <45A2CE7F.4020607@cse.ucsc.edu> Message-ID: <45A2E008.1060102@soe.ucsc.edu> Good Afternoon Mark: It is not a big deal to compile the code on any version of Linux and use it there. If that combination works better than your Sun system, go with it. Your system performance may depend upon some tuning parameters of the MySQL system itself. Its usage of caching and so forth can have an effect on its performance. Attached below is a dump of our usual MySQL settings. You might compare this list with yours to see if there is any significant difference in anything. We have found that MySQL versions past 4.0 are somewhat slower for some functions. It could also be a matter of how much memory is available. Some of the genome browser operations can take up to a couple of Gb of memory to comfortably complete. Our CPUs here are AMD Opteron's: cpu family : 15 model : 5 model name : AMD Opteron(tm) Processor 246 HE stepping : 10 cpu MHz : 1992.692 cache size : 1024 KB fpu : yes fpu_exception : yes cpuid level : 1 wp : yes bogomips : 3990.71 TLB size : 1088 4K pages clflush size : 64 cache_alignment : 64 I don't know how those CPUs would compare to your Sun system. We operate our system with the /gbdb/ filesystem being served up from a freeBSD NFS fileserver on a Sun SUNV20Z amd64 system. Our MySQL databases are on local disks to the WEB browser, separate filesystem from everything else to avoid disk I/O contention. It has been pointed out here by others, that it isn't a good idea to operate MySQL on mounted NFS partitions since there are several NFS locks in operation. You may get acceptable performance using the network connection to the remote MySQL server and connecting via yourserver:3306 (I see Davide Cittaro has mentioned this in another answer to your query) --Hiram > -------- Original Message -------- > Subject: [Genome-mirror] hardware requirements apart from storage > Date: Mon, 8 Jan 2007 11:51:54 -0800 > From: Aronszajn, Mark > To: > > Hi Patrick, > > I don't know if you all got my latest inquiry... I'm embarrassed to > admit that I had forgotten about your reply to a similar inquiry I sent > last May. At any rate, there are a few new questions about getting a > local mirror set up here at Fred Hutchinson. > > If you're still working with Genome Browser mirroring questions & > issues, could you give a hand? > > We want to test a particular installation set-up of the Genome Browser > here at Fred Hutchinson. It would be as follows: the application, web > server and MySQL server would all be running on a linux machine, but a > large share of the storage would be on NFS-mounted drives on a different > machine (a Sun). Briefly, one reason we want to test this is that we > already have tried running the app & servers on the same machine (the > Sun) and found that the resulting GeneBrowser performance was somewhat > slower than the UCSC instance itself, despite being local, and we > thought that the app & servers might run faster on the linux machine. I > have a few questions about this idea, and it would be great if someone > could help us: > > 1) Any opinion on the idea that the app & servers might indeed run > faster on Linux than on Sun (Solaris)? > > 2) Has anyone heard if there are any serious drawbacks to having much of > the storage (MySQL Table spaces, /gbdb and location for Genome db text > dumps) on a mounted drive? > > 3) We have a version of RedHat running on the linux machine right now, > 2.6.10-1.9_FC2smp #1, that is earlier than the version you all recommend > for installing the GeneBrowser binary. Would it be feasible to build & > install from source and stick with the RedHat version, or should we bite > the bullet and upgrade the version? > > Thanks in advance for any feedback or answers you can give us to these > questions, > > Mark Aronszajn > > Biocomputing > Fred Hutchinson Cancer Research Center $ hgsql -e "show variables;" mysql Variable_name Value _____________ _________ back_log 50 basedir / binlog_cache_size 32768 bulk_insert_buffer_size 8388608 character_set latin1 character_sets latin1 big5 czech euc_kr gb2312 gbk latin1_de sjis tis620 ujis dec8 dos german1 hp8 koi8_ru latin2 swe7 usa7 cp1251 danish hebrew win1251 estonia hungarian koi8_ukr win1251ukr greek win1250 croat cp1257 latin5 concurrent_insert ON connect_timeout 5 convert_character_set datadir /var/lib/mysql/ default_week_format 0 delay_key_write ON delayed_insert_limit 100 delayed_insert_timeout 300 delayed_queue_size 1000 flush OFF flush_time 0 ft_boolean_syntax + -><()~*:""&| ft_max_word_len 254 ft_max_word_len_for_sort 20 ft_min_word_len 4 ft_stopword_file (built-in) have_bdb NO have_crypt YES have_innodb YES have_isam YES have_openssl NO have_query_cache YES have_raid NO have_symlink YES init_file innodb_additional_mem_pool_size 1048576 innodb_autoextend_increment 8 innodb_buffer_pool_size 8388608 innodb_data_file_path ibdata1:10M:autoextend innodb_data_home_dir innodb_fast_shutdown ON innodb_file_io_threads 4 innodb_flush_log_at_trx_commit 1 innodb_flush_method innodb_force_recovery 0 innodb_force_recovery 0 innodb_lock_wait_timeout 50 innodb_log_arch_dir ./ innodb_log_archive OFF innodb_log_buffer_size 1048576 innodb_log_file_size 5242880 innodb_log_files_in_group 2 innodb_log_group_home_dir ./ innodb_max_dirty_pages_pct 90 innodb_max_purge_lag 0 innodb_mirrored_log_groups 1 innodb_table_locks ON innodb_thread_concurrency 8 interactive_timeout 28800 join_buffer_size 131072 key_buffer_size 268435456 language /usr/share/mysql/english/ large_files_support ON license GPL local_infile ON locked_in_memory OFF log OFF log_bin OFF log_error log_slave_updates OFF log_slow_queries ON log_update OFF log_warnings 1 long_query_time 10 low_priority_updates OFF lower_case_file_system OFF lower_case_table_names 0 max_allowed_packet 1048576 max_binlog_cache_size 4294967295 max_binlog_size 1073741824 max_connect_errors 10 max_connections 300 max_delayed_threads 20 max_heap_table_size 16777216 max_insert_delayed_threads 20 max_join_size 18446744073709551615 max_relay_log_size 0 max_seeks_for_key 4294967295 max_sort_length 1024 max_tmp_tables 32 max_user_connections 0 max_write_lock_count 4294967295 myisam_max_extra_sort_file_size 268435456 myisam_max_sort_file_size 9223372036854775807 myisam_recover_options OFF myisam_repair_threads 1 myisam_sort_buffer_size 8388608 net_buffer_length 16384 net_read_timeout 30 net_retry_count 10 net_write_timeout 60 new OFF open_files_limit 1510 port 3306 protocol_version 10 query_alloc_block_size 8192 query_cache_limit 1048576 query_cache_size 67108864 query_cache_type ON query_cache_wlock_invalidate OFF query_prealloc_size 8192 range_alloc_block_size 2048 read_buffer_size 1044480 read_only OFF read_rnd_buffer_size 262144 rpl_recovery_rank 0 server_id 0 skip_external_locking ON skip_networking OFF skip_show_database OFF slave_net_timeout 3600 slow_launch_time 2 sort_buffer_size 4194296 sql_mode 0 table_cache 512 table_type MYISAM thread_cache_size 0 thread_stack 196608 timezone PST tmp_table_size 67108864 tmpdir /tmp/ transaction_alloc_block_size 8192 transaction_prealloc_size 4096 tx_isolation REPEATABLE-READ version 4.0.27-standard-log version_comment Official MySQL RPM version_compile_os unknown-linux-gnu wait_timeout 28800 From hartera at soe.ucsc.edu Tue Jan 9 10:51:26 2007 From: hartera at soe.ucsc.edu (Rachel Harte) Date: Tue, 9 Jan 2007 10:51:26 -0800 (PST) Subject: [Genome-mirror] altSplice error In-Reply-To: <5c2c43620612201116k3683f111lc4e0b306177debc6@mail.gmail.com> References: <5c2c43620612201116k3683f111lc4e0b306177debc6@mail.gmail.com> Message-ID: Dear Pete, Sorry for the delay in getting back to you. The person who developed the altSplice code has left UCSC. We contacted him and now he has Found and checked in the files required. Also he modified the source code to comply with the -Werror compiler flags. If you download a new source tree using the public CVS then you should get the latest source code: http://genome.ucsc.edu/admin/cvs.html The download site when we post the jksrc.zip source code is only updated every two weeks so this will not contain these new changes for another 2 weeks. That is why I am directing you to the CVS to get new source code. Please let us know if you find any problems. Rachel Rachel Harte UCSC Genome Bioinformatics Group http://genome.ucsc.edu On Wed, 20 Dec 2006, Pete Shepard wrote: > Dear UCSC folks, > > I am attemptng to install the /jksrc//kent/src tree > > > I can run a make clean and make cgi in /jksrc/kent/src as well as a make > > Everything seems to work fine. > > However, when I do a make n /kent/src/hg/altSplice/altSplice > > I get the following error: > > gcc -O -I../inc -I../../inc -I../../../inc -o sequenceForBed > sequenceForBed.c ../../../lib/i686/jkhgap.a ../../../lib/i686/jkweb.a > ../lib/i686/libSpliceGraph.a -L/usr/include/mysql -lmysqlclient -lm -lz > gcc: ../lib/i686/libSpliceGraph.a: No such file or directory > make: *** [sequenceForBed] Error 1 > > Any ideas? > > Thanks again > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From m.pheasant at imb.uq.edu.au Tue Jan 9 17:30:06 2007 From: m.pheasant at imb.uq.edu.au (Michael Pheasant) Date: Wed, 10 Jan 2007 11:30:06 +1000 Subject: [Genome-mirror] rsync /genome permissions errors Message-ID: Hi, I am getting some rsync permissions errors: rsync --progress --partial -rltvz rsync://hgdownload.cse.ucsc.edu/genome/goldenPath/ /..mypath.../goldenPath/ rsync: readlink "/goldenPath/droVir3/bigZips" (in genome) failed: Permission denied (13) rsync: readlink "/goldenPath/droMoj3/bigZips" (in genome) failed: Permission denied (13) rsync: readlink "/goldenPath/droWil1/bigZips" (in genome) failed: Permission denied (13) rsync: readlink "/goldenPath/droGri2/bigZips" (in genome) failed: Permission denied (13) Seems to be on your side? Regards, Mike. __________________________________ Michael Pheasant, PhD Bioinformatics Research Officer Institute for Molecular Bioscience The University of Queensland Brisbane QLD 4067 Australia P: +61-7-3346-2080 M: +61-405-679-541 http://www.imb.uq.edu.au From chesman at soe.ucsc.edu Wed Jan 10 09:07:03 2007 From: chesman at soe.ucsc.edu (Chester Manuel) Date: Wed, 10 Jan 2007 09:07:03 -0800 Subject: [Genome-mirror] rsync /genome permissions errors In-Reply-To: References: Message-ID: <45A51D37.4080102@soe.ucsc.edu> Michael Pheasant wrote: > Hi, > > I am getting some rsync permissions errors: > > rsync --progress --partial -rltvz > rsync://hgdownload.cse.ucsc.edu/genome/goldenPath/ > /..mypath.../goldenPath/ > > rsync: readlink "/goldenPath/droVir3/bigZips" (in > genome) failed: Permission denied (13) > rsync: readlink "/goldenPath/droMoj3/bigZips" (in > genome) failed: Permission denied (13) > rsync: readlink "/goldenPath/droWil1/bigZips" (in > genome) failed: Permission denied (13) > rsync: readlink "/goldenPath/droGri2/bigZips" (in > genome) failed: Permission denied (13) > > Seems to be on your side? > > Regards, > > Mike. > > __________________________________ > Michael Pheasant, PhD > Bioinformatics Research Officer > Institute for Molecular Bioscience > The University of Queensland > Brisbane QLD 4067 Australia > P: +61-7-3346-2080 > M: +61-405-679-541 > http://www.imb.uq.edu.au > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > Hi Michael, Please try now since permissions have been updated. -- Chester Manuel Systems Administrator Center for Biomolecular Science and Engineering Engineering 2 Building, Suite 501 Mail Stop CBSE/ITI University of California Santa Cruz 1156 High St. Santa Cruz, CA 95064 From peter.shepard at gmail.com Thu Jan 11 15:20:58 2007 From: peter.shepard at gmail.com (Pete Shepard) Date: Thu, 11 Jan 2007 15:20:58 -0800 Subject: [Genome-mirror] altSplice error In-Reply-To: References: <5c2c43620612201116k3683f111lc4e0b306177debc6@mail.gmail.com> Message-ID: <5c2c43620701111520g42b3ec4ex5c6ec2e79dd0de32@mail.gmail.com> Hello, I download the new CVS and run a make in /home/pedro/kent/src/hg/altSplice/altSplice/tests and get the same error as before, blat input/chromTest.fa input/alt3Prime5PrimeCassAgx.fa \ -fine -tileSize=6 input/alt3Prime5PrimeCassAgx.psl Loaded 10640 letters in 1 sequences Searched 3994 bases in 5 sequences blat input/chromTest.fa input/simpleCass.fa \ -fine -tileSize=6 input/simpleCass.psl Loaded 10640 letters in 1 sequences Searched 781 bases in 2 sequences blat input/chromTest.fa input/altEndsStarts.fa \ -fine -tileSize=6 input/altEndsStarts.psl Loaded 10640 letters in 1 sequences Searched 2096 bases in 3 sequences blat input/chromTest.fa input/mutExclusive.fa \ -fine -tileSize=6 input/mutExclusive.psl Loaded 10640 letters in 1 sequences Searched 1475 bases in 2 sequences blat input/chromTest.fa input/altRetInt.fa \ -fine -tileSize=6 input/altRetInt.psl Can't open input/altRetInt.fa to read: No such file or directory make: *** [in] Error 255 I also still need to comment out the -Werror in /kent/src/inc/common.mk in order to get other programs to compile. Thanks, Peter On 1/9/07, Rachel Harte wrote: > > Dear Pete, > > Sorry for the delay in getting back to you. The person who developed the > altSplice code has left UCSC. We contacted him and now he has Found and > checked in the files required. Also he modified the source code to > comply with the -Werror compiler flags. > > If you download a new source tree using the public CVS then you should > get the latest source code: > http://genome.ucsc.edu/admin/cvs.html > > The download site when we post the jksrc.zip source code is only updated > every two weeks so this will not contain these new changes for another 2 > weeks. That is why I am directing you to the CVS to get new source code. > > Please let us know if you find any problems. > > Rachel > > Rachel Harte > UCSC Genome Bioinformatics Group > http://genome.ucsc.edu > > > On Wed, 20 Dec 2006, Pete Shepard wrote: > > > Dear UCSC folks, > > > > I am attemptng to install the /jksrc//kent/src tree > > > > > > I can run a make clean and make cgi in /jksrc/kent/src as well as a make > > > > Everything seems to work fine. > > > > However, when I do a make n /kent/src/hg/altSplice/altSplice > > > > I get the following error: > > > > gcc -O -I../inc -I../../inc -I../../../inc -o sequenceForBed > > sequenceForBed.c ../../../lib/i686/jkhgap.a ../../../lib/i686/jkweb.a > > ../lib/i686/libSpliceGraph.a -L/usr/include/mysql -lmysqlclient -lm -lz > > gcc: ../lib/i686/libSpliceGraph.a: No such file or directory > > make: *** [sequenceForBed] Error 1 > > > > Any ideas? > > > > Thanks again > > _______________________________________________ > > Genome-mirror mailing list > > Genome-mirror at soe.ucsc.edu > > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > > > From galt at soe.ucsc.edu Thu Jan 11 15:47:10 2007 From: galt at soe.ucsc.edu (Galt Barber) Date: Thu, 11 Jan 2007 15:47:10 -0800 (PST) Subject: [Genome-mirror] altSplice error In-Reply-To: <5c2c43620701111520g42b3ec4ex5c6ec2e79dd0de32@mail.gmail.com> References: <5c2c43620612201116k3683f111lc4e0b306177debc6@mail.gmail.com> <5c2c43620701111520g42b3ec4ex5c6ec2e79dd0de32@mail.gmail.com> Message-ID: Sorry about the lack of clarity. You need the very latest source to get the change now before the next official release. Therefore do NOT use the -rbeta which picks up the latest released version. referring to http://genome.ucsc.edu/admin/cvs.html, don't do this: cvs co -rbeta kent do this cvs co kent I just tried it in my source sandbox, and the altSplice compiles without errors. -Galt On Thu, 11 Jan 2007, Pete Shepard wrote: > Hello, > > I download the new CVS and run a make in > /home/pedro/kent/src/hg/altSplice/altSplice/tests > > and get the same error as before, > > blat input/chromTest.fa input/alt3Prime5PrimeCassAgx.fa \ > -fine -tileSize=6 input/alt3Prime5PrimeCassAgx.psl > Loaded 10640 letters in 1 sequences > Searched 3994 bases in 5 sequences > blat input/chromTest.fa input/simpleCass.fa \ > -fine -tileSize=6 input/simpleCass.psl > Loaded 10640 letters in 1 sequences > Searched 781 bases in 2 sequences > blat input/chromTest.fa input/altEndsStarts.fa \ > -fine -tileSize=6 input/altEndsStarts.psl > Loaded 10640 letters in 1 sequences > Searched 2096 bases in 3 sequences > blat input/chromTest.fa input/mutExclusive.fa \ > -fine -tileSize=6 input/mutExclusive.psl > Loaded 10640 letters in 1 sequences > Searched 1475 bases in 2 sequences > blat input/chromTest.fa input/altRetInt.fa \ > -fine -tileSize=6 input/altRetInt.psl > Can't open input/altRetInt.fa to read: No such file or directory > make: *** [in] Error 255 > > I also still need to comment out the -Werror in /kent/src/inc/common.mk in > order to get other programs to compile. > > Thanks, > > Peter > > On 1/9/07, Rachel Harte wrote: > > > > Dear Pete, > > > > Sorry for the delay in getting back to you. The person who developed the > > altSplice code has left UCSC. We contacted him and now he has Found and > > checked in the files required. Also he modified the source code to > > comply with the -Werror compiler flags. > > > > If you download a new source tree using the public CVS then you should > > get the latest source code: > > http://genome.ucsc.edu/admin/cvs.html > > > > The download site when we post the jksrc.zip source code is only updated > > every two weeks so this will not contain these new changes for another 2 > > weeks. That is why I am directing you to the CVS to get new source code. > > > > Please let us know if you find any problems. > > > > Rachel > > > > Rachel Harte > > UCSC Genome Bioinformatics Group > > http://genome.ucsc.edu > > > > > > On Wed, 20 Dec 2006, Pete Shepard wrote: > > > > > Dear UCSC folks, > > > > > > I am attemptng to install the /jksrc//kent/src tree > > > > > > > > > I can run a make clean and make cgi in /jksrc/kent/src as well as a make > > > > > > Everything seems to work fine. > > > > > > However, when I do a make n /kent/src/hg/altSplice/altSplice > > > > > > I get the following error: > > > > > > gcc -O -I../inc -I../../inc -I../../../inc -o sequenceForBed > > > sequenceForBed.c ../../../lib/i686/jkhgap.a ../../../lib/i686/jkweb.a > > > ../lib/i686/libSpliceGraph.a -L/usr/include/mysql -lmysqlclient -lm -lz > > > gcc: ../lib/i686/libSpliceGraph.a: No such file or directory > > > make: *** [sequenceForBed] Error 1 > > > > > > Any ideas? > > > > > > Thanks again > > > _______________________________________________ > > > Genome-mirror mailing list > > > Genome-mirror at soe.ucsc.edu > > > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > > > > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From peter.shepard at gmail.com Thu Jan 11 17:12:49 2007 From: peter.shepard at gmail.com (Pete Shepard) Date: Thu, 11 Jan 2007 17:12:49 -0800 Subject: [Genome-mirror] altSplice error In-Reply-To: References: <5c2c43620612201116k3683f111lc4e0b306177debc6@mail.gmail.com> <5c2c43620701111520g42b3ec4ex5c6ec2e79dd0de32@mail.gmail.com> Message-ID: <5c2c43620701111712l4e811049td51b98f2f76645e6@mail.gmail.com> Thanks Galt, That got me past that step now I get the following error even if I run this as su? blat input/chromTest.fa input/altRetInt.fa \ -fine -tileSize=6 input/altRetInt.psl Loaded 10640 letters in 1 sequences Searched 1616 bases in 2 sequences faToNib input/chromTest.fa input/chromTest.nib Writing 10640 bases in 5328 bytes ../altSplice -consensus -db=hg16 -beds=input/testLoci.bed \ -agxOut=tmp.agx -chromNib=input/chromTest.nib \ -skipTissues -localMem -pslFile=input/simpleCass.psl \>& /dev/null cannot read in connection setting from configuration file. /bin/sh: /dev/null: Permission denied make: *** [simpleCassAgx] Error 126 Thanks again On 1/11/07, Galt Barber wrote: > > > Sorry about the lack of clarity. > You need the very latest source to get the change now > before the next official release. Therefore do NOT > use the -rbeta which picks up the latest released version. > > referring to http://genome.ucsc.edu/admin/cvs.html, > don't do this: > cvs co -rbeta kent > > do this > cvs co kent > > I just tried it in my source sandbox, and the altSplice > compiles without errors. > > -Galt > > > On Thu, 11 Jan 2007, Pete Shepard wrote: > > > Hello, > > > > I download the new CVS and run a make in > > /home/pedro/kent/src/hg/altSplice/altSplice/tests > > > > and get the same error as before, > > > > blat input/chromTest.fa input/alt3Prime5PrimeCassAgx.fa \ > > -fine -tileSize=6 input/alt3Prime5PrimeCassAgx.psl > > Loaded 10640 letters in 1 sequences > > Searched 3994 bases in 5 sequences > > blat input/chromTest.fa input/simpleCass.fa \ > > -fine -tileSize=6 input/simpleCass.psl > > Loaded 10640 letters in 1 sequences > > Searched 781 bases in 2 sequences > > blat input/chromTest.fa input/altEndsStarts.fa \ > > -fine -tileSize=6 input/altEndsStarts.psl > > Loaded 10640 letters in 1 sequences > > Searched 2096 bases in 3 sequences > > blat input/chromTest.fa input/mutExclusive.fa \ > > -fine -tileSize=6 input/mutExclusive.psl > > Loaded 10640 letters in 1 sequences > > Searched 1475 bases in 2 sequences > > blat input/chromTest.fa input/altRetInt.fa \ > > -fine -tileSize=6 input/altRetInt.psl > > Can't open input/altRetInt.fa to read: No such file or directory > > make: *** [in] Error 255 > > > > I also still need to comment out the -Werror in /kent/src/inc/common.mk > in > > order to get other programs to compile. > > > > Thanks, > > > > Peter > > > > On 1/9/07, Rachel Harte wrote: > > > > > > Dear Pete, > > > > > > Sorry for the delay in getting back to you. The person who developed > the > > > altSplice code has left UCSC. We contacted him and now he has Found > and > > > checked in the files required. Also he modified the source code to > > > comply with the -Werror compiler flags. > > > > > > If you download a new source tree using the public CVS then you should > > > get the latest source code: > > > http://genome.ucsc.edu/admin/cvs.html > > > > > > The download site when we post the jksrc.zip source code is only > updated > > > every two weeks so this will not contain these new changes for another > 2 > > > weeks. That is why I am directing you to the CVS to get new source > code. > > > > > > Please let us know if you find any problems. > > > > > > Rachel > > > > > > Rachel Harte > > > UCSC Genome Bioinformatics Group > > > http://genome.ucsc.edu > > > > > > > > > On Wed, 20 Dec 2006, Pete Shepard wrote: > > > > > > > Dear UCSC folks, > > > > > > > > I am attemptng to install the /jksrc//kent/src tree > > > > > > > > > > > > I can run a make clean and make cgi in /jksrc/kent/src as well as a > make > > > > > > > > Everything seems to work fine. > > > > > > > > However, when I do a make n /kent/src/hg/altSplice/altSplice > > > > > > > > I get the following error: > > > > > > > > gcc -O -I../inc -I../../inc -I../../../inc -o sequenceForBed > > > > sequenceForBed.c ../../../lib/i686/jkhgap.a > ../../../lib/i686/jkweb.a > > > > ../lib/i686/libSpliceGraph.a -L/usr/include/mysql -lmysqlclient -lm > -lz > > > > gcc: ../lib/i686/libSpliceGraph.a: No such file or directory > > > > make: *** [sequenceForBed] Error 1 > > > > > > > > Any ideas? > > > > > > > > Thanks again > > > > _______________________________________________ > > > > Genome-mirror mailing list > > > > Genome-mirror at soe.ucsc.edu > > > > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > > > > > > > > > _______________________________________________ > > Genome-mirror mailing list > > Genome-mirror at soe.ucsc.edu > > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > > > From galt at soe.ucsc.edu Thu Jan 11 17:30:55 2007 From: galt at soe.ucsc.edu (Galt Barber) Date: Thu, 11 Jan 2007 17:30:55 -0800 (PST) Subject: [Genome-mirror] altSplice error In-Reply-To: <5c2c43620701111712l4e811049td51b98f2f76645e6@mail.gmail.com> References: <5c2c43620612201116k3683f111lc4e0b306177debc6@mail.gmail.com> <5c2c43620701111520g42b3ec4ex5c6ec2e79dd0de32@mail.gmail.com> <5c2c43620701111712l4e811049td51b98f2f76645e6@mail.gmail.com> Message-ID: > -skipTissues -localMem -pslFile=input/simpleCass.psl \>& /dev/null You can't have the \ followed by more stuff after. Either remove the \ or put everything after it on the next line. When I look in the simpleCass test target in the makefile, there is no backslash there. -Galt On Thu, 11 Jan 2007, Pete Shepard wrote: > Thanks Galt, > > That got me past that step now I get the following error even if I run this > as su? > > blat input/chromTest.fa input/altRetInt.fa \ > -fine -tileSize=6 input/altRetInt.psl > Loaded 10640 letters in 1 sequences > Searched 1616 bases in 2 sequences > faToNib input/chromTest.fa input/chromTest.nib > Writing 10640 bases in 5328 bytes > ../altSplice -consensus -db=hg16 -beds=input/testLoci.bed \ > -agxOut=tmp.agx -chromNib=input/chromTest.nib \ > -skipTissues -localMem -pslFile=input/simpleCass.psl \>& /dev/null > cannot read in connection setting from configuration file. > /bin/sh: /dev/null: Permission denied > make: *** [simpleCassAgx] Error 126 > > > Thanks again > > On 1/11/07, Galt Barber wrote: > > > > > > Sorry about the lack of clarity. > > You need the very latest source to get the change now > > before the next official release. Therefore do NOT > > use the -rbeta which picks up the latest released version. > > > > referring to http://genome.ucsc.edu/admin/cvs.html, > > don't do this: > > cvs co -rbeta kent > > > > do this > > cvs co kent > > > > I just tried it in my source sandbox, and the altSplice > > compiles without errors. > > > > -Galt > > > > > > On Thu, 11 Jan 2007, Pete Shepard wrote: > > > > > Hello, > > > > > > I download the new CVS and run a make in > > > /home/pedro/kent/src/hg/altSplice/altSplice/tests > > > > > > and get the same error as before, > > > > > > blat input/chromTest.fa input/alt3Prime5PrimeCassAgx.fa \ > > > -fine -tileSize=6 input/alt3Prime5PrimeCassAgx.psl > > > Loaded 10640 letters in 1 sequences > > > Searched 3994 bases in 5 sequences > > > blat input/chromTest.fa input/simpleCass.fa \ > > > -fine -tileSize=6 input/simpleCass.psl > > > Loaded 10640 letters in 1 sequences > > > Searched 781 bases in 2 sequences > > > blat input/chromTest.fa input/altEndsStarts.fa \ > > > -fine -tileSize=6 input/altEndsStarts.psl > > > Loaded 10640 letters in 1 sequences > > > Searched 2096 bases in 3 sequences > > > blat input/chromTest.fa input/mutExclusive.fa \ > > > -fine -tileSize=6 input/mutExclusive.psl > > > Loaded 10640 letters in 1 sequences > > > Searched 1475 bases in 2 sequences > > > blat input/chromTest.fa input/altRetInt.fa \ > > > -fine -tileSize=6 input/altRetInt.psl > > > Can't open input/altRetInt.fa to read: No such file or directory > > > make: *** [in] Error 255 > > > > > > I also still need to comment out the -Werror in /kent/src/inc/common.mk > > in > > > order to get other programs to compile. > > > > > > Thanks, > > > > > > Peter > > > > > > On 1/9/07, Rachel Harte wrote: > > > > > > > > Dear Pete, > > > > > > > > Sorry for the delay in getting back to you. The person who developed > > the > > > > altSplice code has left UCSC. We contacted him and now he has Found > > and > > > > checked in the files required. Also he modified the source code to > > > > comply with the -Werror compiler flags. > > > > > > > > If you download a new source tree using the public CVS then you should > > > > get the latest source code: > > > > http://genome.ucsc.edu/admin/cvs.html > > > > > > > > The download site when we post the jksrc.zip source code is only > > updated > > > > every two weeks so this will not contain these new changes for another > > 2 > > > > weeks. That is why I am directing you to the CVS to get new source > > code. > > > > > > > > Please let us know if you find any problems. > > > > > > > > Rachel > > > > > > > > Rachel Harte > > > > UCSC Genome Bioinformatics Group > > > > http://genome.ucsc.edu > > > > > > > > > > > > On Wed, 20 Dec 2006, Pete Shepard wrote: > > > > > > > > > Dear UCSC folks, > > > > > > > > > > I am attemptng to install the /jksrc//kent/src tree > > > > > > > > > > > > > > > I can run a make clean and make cgi in /jksrc/kent/src as well as a > > make > > > > > > > > > > Everything seems to work fine. > > > > > > > > > > However, when I do a make n /kent/src/hg/altSplice/altSplice > > > > > > > > > > I get the following error: > > > > > > > > > > gcc -O -I../inc -I../../inc -I../../../inc -o sequenceForBed > > > > > sequenceForBed.c ../../../lib/i686/jkhgap.a > > ../../../lib/i686/jkweb.a > > > > > ../lib/i686/libSpliceGraph.a -L/usr/include/mysql -lmysqlclient -lm > > -lz > > > > > gcc: ../lib/i686/libSpliceGraph.a: No such file or directory > > > > > make: *** [sequenceForBed] Error 1 > > > > > > > > > > Any ideas? > > > > > > > > > > Thanks again > > > > > _______________________________________________ > > > > > Genome-mirror mailing list > > > > > Genome-mirror at soe.ucsc.edu > > > > > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > > > > > > > > > > > > _______________________________________________ > > > Genome-mirror mailing list > > > Genome-mirror at soe.ucsc.edu > > > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > > > > > > From peter.shepard at gmail.com Fri Jan 12 08:18:10 2007 From: peter.shepard at gmail.com (Pete Shepard) Date: Fri, 12 Jan 2007 08:18:10 -0800 Subject: [Genome-mirror] altSplice error In-Reply-To: References: <5c2c43620612201116k3683f111lc4e0b306177debc6@mail.gmail.com> <5c2c43620701111520g42b3ec4ex5c6ec2e79dd0de32@mail.gmail.com> <5c2c43620701111712l4e811049td51b98f2f76645e6@mail.gmail.com> Message-ID: <5c2c43620701120818g3343e4d5oc2df1845d4cfdda4@mail.gmail.com> Ok, I now get the following error: ../altSplice -consensus -db=hg16 -beds=input/testLoci.bed \ -agxOut=tmp.agx -chromNib=input/chromTest.nib \ -skipTissues -localMem -pslFile=input/simpleCass.psl >& /dev/null make: *** [simpleCassAgx] Error 255 Thanks On 1/11/07, Galt Barber wrote: > > > > -skipTissues -localMem -pslFile=input/simpleCass.psl \>& > /dev/null > > You can't have the \ followed by more stuff after. > Either remove the \ or put everything after it on the next line. > When I look in the simpleCass test target in the makefile, > there is no backslash there. > > -Galt > > > On Thu, 11 Jan 2007, Pete Shepard wrote: > > > Thanks Galt, > > > > That got me past that step now I get the following error even if I run > this > > as su? > > > > blat input/chromTest.fa input/altRetInt.fa \ > > -fine -tileSize=6 input/altRetInt.psl > > Loaded 10640 letters in 1 sequences > > Searched 1616 bases in 2 sequences > > faToNib input/chromTest.fa input/chromTest.nib > > Writing 10640 bases in 5328 bytes > > ../altSplice -consensus -db=hg16 -beds=input/testLoci.bed \ > > -agxOut=tmp.agx -chromNib=input/chromTest.nib \ > > -skipTissues -localMem -pslFile=input/simpleCass.psl \>& > /dev/null > > cannot read in connection setting from configuration file. > > /bin/sh: /dev/null: Permission denied > > make: *** [simpleCassAgx] Error 126 > > > > > > Thanks again > > > > On 1/11/07, Galt Barber wrote: > > > > > > > > > Sorry about the lack of clarity. > > > You need the very latest source to get the change now > > > before the next official release. Therefore do NOT > > > use the -rbeta which picks up the latest released version. > > > > > > referring to http://genome.ucsc.edu/admin/cvs.html, > > > don't do this: > > > cvs co -rbeta kent > > > > > > do this > > > cvs co kent > > > > > > I just tried it in my source sandbox, and the altSplice > > > compiles without errors. > > > > > > -Galt > > > > > > > > > On Thu, 11 Jan 2007, Pete Shepard wrote: > > > > > > > Hello, > > > > > > > > I download the new CVS and run a make in > > > > /home/pedro/kent/src/hg/altSplice/altSplice/tests > > > > > > > > and get the same error as before, > > > > > > > > blat input/chromTest.fa input/alt3Prime5PrimeCassAgx.fa \ > > > > -fine -tileSize=6 input/alt3Prime5PrimeCassAgx.psl > > > > Loaded 10640 letters in 1 sequences > > > > Searched 3994 bases in 5 sequences > > > > blat input/chromTest.fa input/simpleCass.fa \ > > > > -fine -tileSize=6 input/simpleCass.psl > > > > Loaded 10640 letters in 1 sequences > > > > Searched 781 bases in 2 sequences > > > > blat input/chromTest.fa input/altEndsStarts.fa \ > > > > -fine -tileSize=6 input/altEndsStarts.psl > > > > Loaded 10640 letters in 1 sequences > > > > Searched 2096 bases in 3 sequences > > > > blat input/chromTest.fa input/mutExclusive.fa \ > > > > -fine -tileSize=6 input/mutExclusive.psl > > > > Loaded 10640 letters in 1 sequences > > > > Searched 1475 bases in 2 sequences > > > > blat input/chromTest.fa input/altRetInt.fa \ > > > > -fine -tileSize=6 input/altRetInt.psl > > > > Can't open input/altRetInt.fa to read: No such file or directory > > > > make: *** [in] Error 255 > > > > > > > > I also still need to comment out the -Werror in > /kent/src/inc/common.mk > > > in > > > > order to get other programs to compile. > > > > > > > > Thanks, > > > > > > > > Peter > > > > > > > > On 1/9/07, Rachel Harte wrote: > > > > > > > > > > Dear Pete, > > > > > > > > > > Sorry for the delay in getting back to you. The person who > developed > > > the > > > > > altSplice code has left UCSC. We contacted him and now he has > Found > > > and > > > > > checked in the files required. Also he modified the source code to > > > > > comply with the -Werror compiler flags. > > > > > > > > > > If you download a new source tree using the public CVS then you > should > > > > > get the latest source code: > > > > > http://genome.ucsc.edu/admin/cvs.html > > > > > > > > > > The download site when we post the jksrc.zip source code is only > > > updated > > > > > every two weeks so this will not contain these new changes for > another > > > 2 > > > > > weeks. That is why I am directing you to the CVS to get new source > > > code. > > > > > > > > > > Please let us know if you find any problems. > > > > > > > > > > Rachel > > > > > > > > > > Rachel Harte > > > > > UCSC Genome Bioinformatics Group > > > > > http://genome.ucsc.edu > > > > > > > > > > > > > > > On Wed, 20 Dec 2006, Pete Shepard wrote: > > > > > > > > > > > Dear UCSC folks, > > > > > > > > > > > > I am attemptng to install the /jksrc//kent/src tree > > > > > > > > > > > > > > > > > > I can run a make clean and make cgi in /jksrc/kent/src as well > as a > > > make > > > > > > > > > > > > Everything seems to work fine. > > > > > > > > > > > > However, when I do a make n /kent/src/hg/altSplice/altSplice > > > > > > > > > > > > I get the following error: > > > > > > > > > > > > gcc -O -I../inc -I../../inc -I../../../inc -o sequenceForBed > > > > > > sequenceForBed.c ../../../lib/i686/jkhgap.a > > > ../../../lib/i686/jkweb.a > > > > > > ../lib/i686/libSpliceGraph.a -L/usr/include/mysql -lmysqlclient > -lm > > > -lz > > > > > > gcc: ../lib/i686/libSpliceGraph.a: No such file or directory > > > > > > make: *** [sequenceForBed] Error 1 > > > > > > > > > > > > Any ideas? > > > > > > > > > > > > Thanks again > > > > > > _______________________________________________ > > > > > > Genome-mirror mailing list > > > > > > Genome-mirror at soe.ucsc.edu > > > > > > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > > > > > > > > > > > > > > > _______________________________________________ > > > > Genome-mirror mailing list > > > > Genome-mirror at soe.ucsc.edu > > > > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > > > > > > > > > > From galt at soe.ucsc.edu Fri Jan 12 11:53:31 2007 From: galt at soe.ucsc.edu (Galt Barber) Date: Fri, 12 Jan 2007 11:53:31 -0800 (PST) Subject: [Genome-mirror] altSplice error In-Reply-To: <5c2c43620701120818g3343e4d5oc2df1845d4cfdda4@mail.gmail.com> References: <5c2c43620612201116k3683f111lc4e0b306177debc6@mail.gmail.com> <5c2c43620701111520g42b3ec4ex5c6ec2e79dd0de32@mail.gmail.com> <5c2c43620701111712l4e811049td51b98f2f76645e6@mail.gmail.com> <5c2c43620701120818g3343e4d5oc2df1845d4cfdda4@mail.gmail.com> Message-ID: Remove the >& /dev/null so you can see any messages being produced and then run it. -db=hg16 means that it will be looking for a mysql database called "hg16", and it will need various tables. If you run ../altSplice with no parameters, it will show you the help for the program: -db -- Database (i.e. hg15) to load psl records from. When I tried to run without -db, it says: Must set -db flag. Try -help for usage. I think it may be using the database to get all the mRNA and EST alignments. I see references to chrN_mrna, chrN_intronEst. Under some conditions, refSeqAli table may also be used. "... cluster mrna's and ests onto genomic sequence and try to determine various types of alternative splicing. Currently cassette exons are particularly highlighted. Only spliced ests are used, as they tend to be less noisy and are also easier to orient correctly using splice sites from genomic sequence." -Galt On Fri, 12 Jan 2007, Pete Shepard wrote: > Ok, > > I now get the following error: > > ../altSplice -consensus -db=hg16 -beds=input/testLoci.bed \ > -agxOut=tmp.agx -chromNib=input/chromTest.nib \ > -skipTissues -localMem -pslFile=input/simpleCass.psl >& /dev/null > make: *** [simpleCassAgx] Error 255 > > > Thanks > > On 1/11/07, Galt Barber wrote: > > > > > > > -skipTissues -localMem -pslFile=input/simpleCass.psl \>& > > /dev/null > > > > You can't have the \ followed by more stuff after. > > Either remove the \ or put everything after it on the next line. > > When I look in the simpleCass test target in the makefile, > > there is no backslash there. > > > > -Galt > > > > > > On Thu, 11 Jan 2007, Pete Shepard wrote: > > > > > Thanks Galt, > > > > > > That got me past that step now I get the following error even if I run > > this > > > as su? > > > > > > blat input/chromTest.fa input/altRetInt.fa \ > > > -fine -tileSize=6 input/altRetInt.psl > > > Loaded 10640 letters in 1 sequences > > > Searched 1616 bases in 2 sequences > > > faToNib input/chromTest.fa input/chromTest.nib > > > Writing 10640 bases in 5328 bytes > > > ../altSplice -consensus -db=hg16 -beds=input/testLoci.bed \ > > > -agxOut=tmp.agx -chromNib=input/chromTest.nib \ > > > -skipTissues -localMem -pslFile=input/simpleCass.psl \>& > > /dev/null > > > cannot read in connection setting from configuration file. > > > /bin/sh: /dev/null: Permission denied > > > make: *** [simpleCassAgx] Error 126 > > > > > > > > > Thanks again > > > > > > On 1/11/07, Galt Barber wrote: > > > > > > > > > > > > Sorry about the lack of clarity. > > > > You need the very latest source to get the change now > > > > before the next official release. Therefore do NOT > > > > use the -rbeta which picks up the latest released version. > > > > > > > > referring to http://genome.ucsc.edu/admin/cvs.html, > > > > don't do this: > > > > cvs co -rbeta kent > > > > > > > > do this > > > > cvs co kent > > > > > > > > I just tried it in my source sandbox, and the altSplice > > > > compiles without errors. > > > > > > > > -Galt > > > > > > > > > > > > On Thu, 11 Jan 2007, Pete Shepard wrote: > > > > > > > > > Hello, > > > > > > > > > > I download the new CVS and run a make in > > > > > /home/pedro/kent/src/hg/altSplice/altSplice/tests > > > > > > > > > > and get the same error as before, > > > > > > > > > > blat input/chromTest.fa input/alt3Prime5PrimeCassAgx.fa \ > > > > > -fine -tileSize=6 input/alt3Prime5PrimeCassAgx.psl > > > > > Loaded 10640 letters in 1 sequences > > > > > Searched 3994 bases in 5 sequences > > > > > blat input/chromTest.fa input/simpleCass.fa \ > > > > > -fine -tileSize=6 input/simpleCass.psl > > > > > Loaded 10640 letters in 1 sequences > > > > > Searched 781 bases in 2 sequences > > > > > blat input/chromTest.fa input/altEndsStarts.fa \ > > > > > -fine -tileSize=6 input/altEndsStarts.psl > > > > > Loaded 10640 letters in 1 sequences > > > > > Searched 2096 bases in 3 sequences > > > > > blat input/chromTest.fa input/mutExclusive.fa \ > > > > > -fine -tileSize=6 input/mutExclusive.psl > > > > > Loaded 10640 letters in 1 sequences > > > > > Searched 1475 bases in 2 sequences > > > > > blat input/chromTest.fa input/altRetInt.fa \ > > > > > -fine -tileSize=6 input/altRetInt.psl > > > > > Can't open input/altRetInt.fa to read: No such file or directory > > > > > make: *** [in] Error 255 > > > > > > > > > > I also still need to comment out the -Werror in > > /kent/src/inc/common.mk > > > > in > > > > > order to get other programs to compile. > > > > > > > > > > Thanks, > > > > > > > > > > Peter > > > > > > > > > > On 1/9/07, Rachel Harte wrote: > > > > > > > > > > > > Dear Pete, > > > > > > > > > > > > Sorry for the delay in getting back to you. The person who > > developed > > > > the > > > > > > altSplice code has left UCSC. We contacted him and now he has > > Found > > > > and > > > > > > checked in the files required. Also he modified the source code to > > > > > > comply with the -Werror compiler flags. > > > > > > > > > > > > If you download a new source tree using the public CVS then you > > should > > > > > > get the latest source code: > > > > > > http://genome.ucsc.edu/admin/cvs.html > > > > > > > > > > > > The download site when we post the jksrc.zip source code is only > > > > updated > > > > > > every two weeks so this will not contain these new changes for > > another > > > > 2 > > > > > > weeks. That is why I am directing you to the CVS to get new source > > > > code. > > > > > > > > > > > > Please let us know if you find any problems. > > > > > > > > > > > > Rachel > > > > > > > > > > > > Rachel Harte > > > > > > UCSC Genome Bioinformatics Group > > > > > > http://genome.ucsc.edu > > > > > > > > > > > > > > > > > > On Wed, 20 Dec 2006, Pete Shepard wrote: > > > > > > > > > > > > > Dear UCSC folks, > > > > > > > > > > > > > > I am attemptng to install the /jksrc//kent/src tree > > > > > > > > > > > > > > > > > > > > > I can run a make clean and make cgi in /jksrc/kent/src as well > > as a > > > > make > > > > > > > > > > > > > > Everything seems to work fine. > > > > > > > > > > > > > > However, when I do a make n /kent/src/hg/altSplice/altSplice > > > > > > > > > > > > > > I get the following error: > > > > > > > > > > > > > > gcc -O -I../inc -I../../inc -I../../../inc -o sequenceForBed > > > > > > > sequenceForBed.c ../../../lib/i686/jkhgap.a > > > > ../../../lib/i686/jkweb.a > > > > > > > ../lib/i686/libSpliceGraph.a -L/usr/include/mysql -lmysqlclient > > -lm > > > > -lz > > > > > > > gcc: ../lib/i686/libSpliceGraph.a: No such file or directory > > > > > > > make: *** [sequenceForBed] Error 1 > > > > > > > > > > > > > > Any ideas? > > > > > > > > > > > > > > Thanks again > > > > > > > _______________________________________________ > > > > > > > Genome-mirror mailing list > > > > > > > Genome-mirror at soe.ucsc.edu > > > > > > > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > Genome-mirror mailing list > > > > > Genome-mirror at soe.ucsc.edu > > > > > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > > > > > > > > > > > > > > > From rhead at soe.ucsc.edu Tue Jan 16 16:48:19 2007 From: rhead at soe.ucsc.edu (Brooke Rhead) Date: Tue, 16 Jan 2007 16:48:19 -0800 Subject: [Genome-mirror] felCat3 to be released soon Message-ID: <45AD7253.5070905@soe.ucsc.edu> Hello mirrors, This is to let you know that we will soon (likely within the week) be releasing a new cat database, felCat3. Be prepared to host ~23G of data: ~5G data in /gbdb/felCat3 directory ~11G data in felCat3 tables ~6.5G data in hg18, mm8 and canFam2 tables The tables in other databases are called chr*_chainFelCat3Link, chr*_chainFelCat3, and netFelCat3. The total size of these tables in each database is as follows: hg18 2,043 MB mm8 1,392 MB canFam2 3,051 MB total table size in other databases: ~6.5G. Please let us know if you have any questions or concerns. -- Brooke Rhead UCSC Genome Bioinformatics Group From hartera at soe.ucsc.edu Mon Jan 22 19:38:02 2007 From: hartera at soe.ucsc.edu (Rachel Harte) Date: Mon, 22 Jan 2007 19:38:02 -0800 (PST) Subject: [Genome-mirror] v150 Genome Browser available Message-ID: v150 source is at: http://hgdownload.cse.ucsc.edu/admin/jksrc.zip or labeled with source number: http://hgdownload.cse.ucsc.edu/admin/jksrc.v150.zip cartDump cartReset das hgc hgBlat hgConvert hgCustom hgEncodeDataVersions hgGateway hgGene hgLiftOver hgNear hgPcr hgTables hgText hgTracks hgTrackUi hgVisiGene mkEncodeFrameset pbGateway pbGlobal pbTracks phyloGif /usr/local/apache/cgi-bin/all.joiner /usr/local/apache/cgi-bin/hgNearData/* /usr/local/apache/cgi-bin/hgGeneData/* /usr/local/apache/cgi-bin/hgcData/* /usr/local/apache/cgi-bin/hgCgiData/* Contents of v150 may be seen at: http://genome-test.cse.ucsc.edu/builds/versions.html Please check both v150-preview and v150-final to see all the code and data changes in this release. Rachel Harte UCSC Genome Bioinformatics Group http://genome.ucsc.edu From liaojy at mail2.sysu.edu.cn Mon Jan 29 20:36:27 2007 From: liaojy at mail2.sysu.edu.cn (liaojy) Date: Tue, 30 Jan 2007 12:36:27 +0800 Subject: [Genome-mirror] A question about mirror site Message-ID: <000001c74428$36758300$880ca8c0@leocomputer> Hi, I have a question about creating UCSC mirror site. In "Procedure for Creating a Mirror Site for the UCSC Genome Browser" page, step-by-Step Details, step 4-"Obtain the /gbdb data file area(400Gb)", I want to ask is this 400Gb file contain all species' data? Could I only download a specie's data of this 400Gb? Sincerely, LiaoJianyou From kayla at soe.ucsc.edu Tue Jan 30 16:50:17 2007 From: kayla at soe.ucsc.edu (Kayla Smith) Date: Tue, 30 Jan 2007 16:50:17 -0800 Subject: [Genome-mirror] A question about mirror site In-Reply-To: <000001c74428$36758300$880ca8c0@leocomputer> References: <000001c74428$36758300$880ca8c0@leocomputer> Message-ID: <45BFE7C9.9050107@cse.ucsc.edu> Liao, You don't have to download the whole /gbdb directory. There are some useful instructions on our wiki which describe Partial mirrors here: http://genomewiki.ucsc.edu/index.php/Mirror I hope this helps to answer your question. Please don't hesitate to contact us again if you require further assistance. Kayla Smith UCSC Genome Bioinformatics Group liaojy wrote: > Hi, > > I have a question about creating UCSC mirror site. In "Procedure for > Creating a Mirror Site for the UCSC Genome Browser" page, step-by-Step > Details, step 4-"Obtain the /gbdb data file area(400Gb)", I want to ask > is this 400Gb file contain all species' data? Could I only download a > specie's data of this 400Gb? > > Sincerely, > LiaoJianyou > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From - Tue From hiram at soe.ucsc.edu Tue Jan 30 17:03:46 2007 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Tue, 30 Jan 2007 17:03:46 -0800 Subject: [Genome-mirror] A question about mirror site In-Reply-To: <45BFE7C9.9050107@cse.ucsc.edu> References: <000001c74428$36758300$880ca8c0@leocomputer> <45BFE7C9.9050107@cse.ucsc.edu> Message-ID: <45BFEAF2.4050702@soe.ucsc.edu> Good Afternoon Liao: There are a variety of situations for Mirror sites. See also these instructions for a minimal mirror site setup: http://genomewiki.ucsc.edu/index.php/Browser_installation --Hiram