From J.Hendricks at med.umcg.nl Fri Feb 1 02:44:05 2008 From: J.Hendricks at med.umcg.nl (J.Hendricks at med.umcg.nl) Date: Fri, 1 Feb 2008 11:44:05 +0100 Subject: [Genome-mirror] Re Message-ID: <20080201114405046.00000001780@129-125-166-182> I an encounter this error mention when trying to uploud my file: Error File 'IgHV locus.txt' - line 3 of var chr6 137962840 138007867 Brf1* 1000 - 137962840 138007867 0 2 45027, 0, in custom input: name="Immunoglobulin heavy chain V region" description="IgH V region" visibility=2 itemRGB="On" chr6 137962840 138007867 Brf1* 1000 - 137962840 138007867 0 2 45027, 0, missing = in var/val pair Please advice From hiram at soe.ucsc.edu Fri Feb 1 09:41:33 2008 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Fri, 01 Feb 2008 09:41:33 -0800 Subject: [Genome-mirror] Re In-Reply-To: <20080201114405046.00000001780@129-125-166-182> References: <20080201114405046.00000001780@129-125-166-182> Message-ID: <47A359CD.1080208@soe.ucsc.edu> Good Morning: Can you please copy us on the first 4 or 5 lines of this file that has the problem. I suspect your "track" line may be broken up into more than a single line. The "track" line needs to be a single line. Or, it actually looks like your line that should begin: chr6 137962840 138007867 Brf1* ... has been joined to the track line above it. How do you produce this text file ? What type of editor on what type of operating system ? --Hiram J.Hendricks at med.umcg.nl wrote: > I an encounter this error mention when trying to uploud my file: > Error File 'IgHV locus.txt' - > line 3 of var chr6 137962840 138007867 Brf1* 1000 - 137962840 138007867 0 2 45027, 0, > in custom input: name="Immunoglobulin heavy chain V region" > description="IgH V region" visibility=2 itemRGB="On" > chr6 137962840 138007867 Brf1* 1000 - 137962840 138007867 0 2 45027, 0, missing = in var/val pair > > Please advice > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From zhangc at cshl.edu Mon Feb 4 13:30:35 2008 From: zhangc at cshl.edu (Zhang, Chaolin) Date: Mon, 4 Feb 2008 16:30:35 -0500 Subject: [Genome-mirror] wiggle files Message-ID: Hi, I have a local mirror of the genome browser. I have a question about loading wiggle files into the database. I used to be able to load those files with wigEncode and hgLoadWiggle. However, it stops working now when I run hgLoadWiggle. The program can write something into the database, but then stuck, and the server is dead. I remembered the last time I updated the kent source tree was nov 2007. The last time I could load the wiggle file is dec 2007, which is weird. Is there any explanation for this? Thanks very much for your time! Chaolin From hiram at soe.ucsc.edu Mon Feb 4 14:16:51 2008 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Mon, 04 Feb 2008 14:16:51 -0800 Subject: [Genome-mirror] wiggle files In-Reply-To: References: Message-ID: <47A78ED3.5030906@soe.ucsc.edu> Good Afternoon Chaolin: There is nothing new in the code for a long time regarding these types of files. You can run wigEncode and get the result from that in a file. (the .wig and .wib files) The hgLoadWiggle of that result is a simple database operation that creates the table and loads the .wig data file from wigEncode into that table. It isn't a special operation. You should see a temporary .tab file being created that is nothing but the .wig data file with an extra bin column added in front of everything else. The load is then a simple "load data local infile" of that temporary .tab file. You can verify your MySQL system while this is happening with a show processlist command: hgsql -e "show processlist;" mysql --Hiram Zhang, Chaolin wrote: > Hi, > > I have a local mirror of the genome browser. I have a question about loading wiggle files into the database. I used to be able to load those files with wigEncode and hgLoadWiggle. However, it stops working now when I run hgLoadWiggle. The program can write something into the database, but then stuck, and the server is dead. I remembered the last time I updated the kent source tree was nov 2007. The last time I could load the wiggle file is dec 2007, which is weird. Is there any explanation for this? > > Thanks very much for your time! > > Chaolin > > > > > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From zhangc at cshl.edu Mon Feb 4 14:18:43 2008 From: zhangc at cshl.edu (Zhang, Chaolin) Date: Mon, 4 Feb 2008 17:18:43 -0500 Subject: [Genome-mirror] wiggle files In-Reply-To: <47A78ED3.5030906@soe.ucsc.edu> References: <47A78ED3.5030906@soe.ucsc.edu> Message-ID: Thanks very much Hiram! Chaolin -----Original Message----- From: Hiram Clawson [mailto:hiram at soe.ucsc.edu] Sent: Monday, February 04, 2008 5:17 PM To: Zhang, Chaolin Cc: genome-mirror at soe.ucsc.edu Subject: Re: [Genome-mirror] wiggle files Good Afternoon Chaolin: There is nothing new in the code for a long time regarding these types of files. You can run wigEncode and get the result from that in a file. (the .wig and .wib files) The hgLoadWiggle of that result is a simple database operation that creates the table and loads the .wig data file from wigEncode into that table. It isn't a special operation. You should see a temporary .tab file being created that is nothing but the .wig data file with an extra bin column added in front of everything else. The load is then a simple "load data local infile" of that temporary .tab file. You can verify your MySQL system while this is happening with a show processlist command: hgsql -e "show processlist;" mysql --Hiram Zhang, Chaolin wrote: > Hi, > > I have a local mirror of the genome browser. I have a question about loading wiggle files into the database. I used to be able to load those files with wigEncode and hgLoadWiggle. However, it stops working now when I run hgLoadWiggle. The program can write something into the database, but then stuck, and the server is dead. I remembered the last time I updated the kent source tree was nov 2007. The last time I could load the wiggle file is dec 2007, which is weird. Is there any explanation for this? > > Thanks very much for your time! > > Chaolin > > > > > > > _______________________________________________ > Genome-mirror mailing list > Genome-mirror at soe.ucsc.edu > http://www.soe.ucsc.edu/mailman/listinfo/genome-mirror > From vcpatel at genetics.emory.edu Tue Feb 5 07:35:31 2008 From: vcpatel at genetics.emory.edu (Viren Patel) Date: Tue, 05 Feb 2008 10:35:31 -0500 Subject: [Genome-mirror] Trouble updating cgi-bin Message-ID: <47A88243.2080406@genetics.emory.edu> After moving from Apple OS X 10.4.11 (Tiger) on PPC platform to Apple OS X 10.5.1 (Leopard) on Intel platform, updating cgi-bin gives the following error: gcc -O -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DMACHTYPE_i386 -DJK_WARN -Wall -Werror -Wno-unused-variable -Wno-long-double -I../inc -I../../inc -I../../../inc -I../../../../inc -I../../../../../inc -I/usr/local/include/mysql -o hgFind.o -c hgFind.c cc1: warnings being treated as errors hgFind.c: In function 'searchSpecial': hgFind.c:1204: warning: 'type' may be used uninitialized in this function make[1]: *** [hgFind.o] Error 1 make: *** [hgLib] Error 2 and the build fails. The suggested solution from the mailing lists is to remove -Werror from kent/src/inc/common.mk. However isn't the point of this option to flag such errors? Also, automated update scripts (such as what I use to do the update) break :-( Would it be possible to patch the code so that line 1024 in hgFind.c reads: char *type = NULL; This seems to be simple enough but I don't presume to know why this is the way it is. Maybe there is a reason to the madness... :-), in which case might I suggest to remove -Werror from common.mk (at least for OS X). Any other suggestions are most welcome. Viren -- Viren Patel, Application Developer/Analyst Sr. Department of Human Genetics Emory University School of Medicine From hiram at soe.ucsc.edu Tue Feb 5 10:26:34 2008 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Tue, 05 Feb 2008 10:26:34 -0800 Subject: [Genome-mirror] Trouble updating cgi-bin In-Reply-To: <47A88243.2080406@genetics.emory.edu> References: <47A88243.2080406@genetics.emory.edu> Message-ID: <47A8AA5A.6050309@soe.ucsc.edu> Good Morning Viren: You are correct, the -Werror is a *good* thing. I need to figure out how to reproduce this here. Can you forward to me the output of your gcc --version printout. I have a gcc 3.4.6 and 4.1.1 here and neither one of them indicate this error on this line. --Hiram Viren Patel wrote: > After moving from Apple OS X 10.4.11 (Tiger) on PPC platform to Apple OS > X 10.5.1 (Leopard) on Intel platform, updating cgi-bin gives the > following error: > > gcc -O -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE > -DMACHTYPE_i386 -DJK_WARN -Wall -Werror -Wno-unused-variable > -Wno-long-double -I../inc -I../../inc -I../../../inc -I../../../../inc > -I../../../../../inc -I/usr/local/include/mysql -o hgFind.o -c hgFind.c > cc1: warnings being treated as errors > hgFind.c: In function 'searchSpecial': > hgFind.c:1204: warning: 'type' may be used uninitialized in this function > make[1]: *** [hgFind.o] Error 1 > make: *** [hgLib] Error 2 > > and the build fails. The suggested solution from the mailing lists is to > remove -Werror from kent/src/inc/common.mk. However isn't the point of > this option to flag such errors? Also, automated update scripts (such as > what I use to do the update) break :-( > > > Would it be possible to patch the code so that line 1024 in hgFind.c reads: > > char *type = NULL; > > This seems to be simple enough but I don't presume to know why this is > the way it is. Maybe there is a reason to the madness... :-), in which > case might I suggest to remove -Werror from common.mk (at least for OS > X). Any other suggestions are most welcome. > > > Viren > > -- > Viren Patel, Application Developer/Analyst Sr. > Department of Human Genetics > Emory University School of Medicine From hiram at soe.ucsc.edu Wed Feb 6 10:24:40 2008 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Wed, 06 Feb 2008 10:24:40 -0800 Subject: [Genome-mirror] Trouble updating cgi-bin In-Reply-To: <47A9F9DD.8090400@genetics.emory.edu> References: <47A88243.2080406@genetics.emory.edu> <47A8AA5A.6050309@soe.ucsc.edu> <47A9F9DD.8090400@genetics.emory.edu> Message-ID: <47A9FB68.9060606@soe.ucsc.edu> Good Morning Viren: This wouldn't be the first time that gcc on the Mac does something different than gcc elsewhere. We have always had unusual differences with the Mac gcc versions. I don't know why that is. We can fix the particular error you have found, but since we can't reproduce this here we can't find perhaps others, nor keep this from happening again. I'm sorry, but you will have to let us know if this happens elsewhere. --Hiram Viren Patel wrote: > Hiram, > > Here is the gcc --version output. I believe gcc is part of XCode 3.0 > which is the version for Leopard. I am not sure why they are using 4.0.1 > if there is a newer version available. > > > gcc --version > i686-apple-darwin9-gcc-4.0.1 (GCC) 4.0.1 (Apple Inc. build 5465) > Copyright (C) 2005 Free Software Foundation, Inc. > This is free software; see the source for copying conditions. There is NO > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > Viren > > Hiram Clawson wrote: >> Good Morning Viren: You are correct, the -Werror is a *good* thing. >> >> I need to figure out how to reproduce this here. Can you forward >> to me the output of your gcc --version printout. >> I have a gcc 3.4.6 and 4.1.1 here and neither one of them indicate >> this error on this line. >> >> --Hiram >> >> Viren Patel wrote: >>> After moving from Apple OS X 10.4.11 (Tiger) on PPC platform to Apple >>> OS X 10.5.1 (Leopard) on Intel platform, updating cgi-bin gives the >>> following error: >>> >>> gcc -O -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE >>> -DMACHTYPE_i386 -DJK_WARN -Wall -Werror -Wno-unused-variable >>> -Wno-long-double -I../inc -I../../inc -I../../../inc >>> -I../../../../inc -I../../../../../inc -I/usr/local/include/mysql -o >>> hgFind.o -c hgFind.c >>> cc1: warnings being treated as errors >>> hgFind.c: In function 'searchSpecial': >>> hgFind.c:1204: warning: 'type' may be used uninitialized in this >>> function >>> make[1]: *** [hgFind.o] Error 1 >>> make: *** [hgLib] Error 2 >>> >>> and the build fails. The suggested solution from the mailing lists is >>> to remove -Werror from kent/src/inc/common.mk. However isn't the >>> point of this option to flag such errors? Also, automated update >>> scripts (such as what I use to do the update) break :-( >>> >>> >>> Would it be possible to patch the code so that line 1024 in hgFind.c >>> reads: >>> >>> char *type = NULL; >>> >>> This seems to be simple enough but I don't presume to know why this >>> is the way it is. Maybe there is a reason to the madness... :-), in >>> which case might I suggest to remove -Werror from common.mk (at least >>> for OS X). Any other suggestions are most welcome. >>> >>> >>> Viren >>> >>> -- >>> Viren Patel, Application Developer/Analyst Sr. >>> Department of Human Genetics >>> Emory University School of Medicine From vcpatel at genetics.emory.edu Wed Feb 6 10:18:05 2008 From: vcpatel at genetics.emory.edu (Viren Patel) Date: Wed, 06 Feb 2008 13:18:05 -0500 Subject: [Genome-mirror] Trouble updating cgi-bin In-Reply-To: <47A8AA5A.6050309@soe.ucsc.edu> References: <47A88243.2080406@genetics.emory.edu> <47A8AA5A.6050309@soe.ucsc.edu> Message-ID: <47A9F9DD.8090400@genetics.emory.edu> Hiram, Here is the gcc --version output. I believe gcc is part of XCode 3.0 which is the version for Leopard. I am not sure why they are using 4.0.1 if there is a newer version available. gcc --version i686-apple-darwin9-gcc-4.0.1 (GCC) 4.0.1 (Apple Inc. build 5465) Copyright (C) 2005 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Viren Hiram Clawson wrote: > Good Morning Viren: You are correct, the -Werror is a *good* thing. > > I need to figure out how to reproduce this here. Can you forward > to me the output of your gcc --version printout. > I have a gcc 3.4.6 and 4.1.1 here and neither one of them indicate > this error on this line. > > --Hiram > > Viren Patel wrote: >> After moving from Apple OS X 10.4.11 (Tiger) on PPC platform to Apple >> OS X 10.5.1 (Leopard) on Intel platform, updating cgi-bin gives the >> following error: >> >> gcc -O -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE >> -DMACHTYPE_i386 -DJK_WARN -Wall -Werror -Wno-unused-variable >> -Wno-long-double -I../inc -I../../inc -I../../../inc -I../../../../inc >> -I../../../../../inc -I/usr/local/include/mysql -o hgFind.o -c hgFind.c >> cc1: warnings being treated as errors >> hgFind.c: In function 'searchSpecial': >> hgFind.c:1204: warning: 'type' may be used uninitialized in this function >> make[1]: *** [hgFind.o] Error 1 >> make: *** [hgLib] Error 2 >> >> and the build fails. The suggested solution from the mailing lists is >> to remove -Werror from kent/src/inc/common.mk. However isn't the point >> of this option to flag such errors? Also, automated update scripts >> (such as what I use to do the update) break :-( >> >> >> Would it be possible to patch the code so that line 1024 in hgFind.c >> reads: >> >> char *type = NULL; >> >> This seems to be simple enough but I don't presume to know why this is >> the way it is. Maybe there is a reason to the madness... :-), in which >> case might I suggest to remove -Werror from common.mk (at least for OS >> X). Any other suggestions are most welcome. >> >> >> Viren >> >> -- >> Viren Patel, Application Developer/Analyst Sr. >> Department of Human Genetics >> Emory University School of Medicine > From aamp at ucsc.edu Mon Feb 11 15:55:42 2008 From: aamp at ucsc.edu (Andy Pohl) Date: Mon, 11 Feb 2008 15:55:42 -0800 Subject: [Genome-mirror] v175 Genome Browser Available Message-ID: <9fa943760802111555r58db1156h95a93159ed0d527@mail.gmail.com> Hello, The v175 source is at: http://hgdownload.cse.ucsc.edu/admin/jksrc.zip or labeled with source number: http://hgdownload.cse.ucsc.edu/admin/jksrc.v175.zip cartDump cartReset das hgc hgBlat hgConvert hgCustom hgEncodeDataVersions hgGateway hgGene hgGenome hgLiftOver hgNear hgPcr hgSession hgTables hgText hgTracks hgTrackUi hgVisiGene mkEncodeFrameset pbGateway pbGlobal pbTracks phyloGif /usr/local/apache/cgi-bin/all.joiner /usr/local/apache/cgi-bin/hgNearData/* /usr/local/apache/cgi-bin/hgGeneData/* /usr/local/apache/cgi-bin/hgcData/* /usr/local/apache/cgi-bin/hgCgiData/* /usr/local/apache/cgi-bin/loader/* /usr/local/apache/cgi-bin/visiGeneData/* Contents of v175 may be seen at: http://genome-test.cse.ucsc.edu/builds/versions.html Please check both v175-preview and v175-final to see all the code and data changes in this release. Have a nice week! Andy Pohl UCSC CBSE From rhead at soe.ucsc.edu Fri Feb 15 11:19:09 2008 From: rhead at soe.ucsc.edu (Brooke Rhead) Date: Fri, 15 Feb 2008 11:19:09 -0800 Subject: [Genome-mirror] 14G more data coming soon to /gbdb/hg18 Message-ID: <47B5E5AD.4060300@soe.ucsc.edu> Hello mirror sites, Early next week, we will be adding this 14 gigabyte file to the hg18 /gbdb area: /gbdb/hg18/snp/snp128.fa This is to support the upcoming human "SNPs (128)" track. Please let us know if you have any questions or concerns. -- Brooke Rhead UCSC Genome Bioinformatics Group From davide.cittaro at ifom-ieo-campus.it Tue Feb 19 02:56:56 2008 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Tue, 19 Feb 2008 11:56:56 +0100 Subject: [Genome-mirror] handling custom tracks Message-ID: Dear mirror people, I've noticed that uploading some "wrong" gff files as ct (with wrong number of columns) raises an error on our mirror and not on your public server. It also happened that the user lost all custom tracks... :-( I'm trying to track the issue, but meanwhile I would like to ask a couple of questions: 1- is it possibile to recover an user session with all its custom tracks? How? I've seen that hguid and hgsid can be helpful but I can't manage them in a useful way 2- some time ago I've read a mail where it was discussed a method to have custom tracks in mysql tables rather than ct_* files on the filesystem. I would like to set this up, I miss instructions and also I would like to know if (and how) I can import custom tracks I already have on the filesystem. Thanks in advance d /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core Cogentech - Consortium for Genomic Technologies via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro at ifom-ieo-campus.it */ From hiram at soe.ucsc.edu Tue Feb 19 10:09:29 2008 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Tue, 19 Feb 2008 10:09:29 -0800 Subject: [Genome-mirror] handling custom tracks In-Reply-To: References: Message-ID: <47BB1B59.3080101@soe.ucsc.edu> Good Morning Davide: There is no actual method to recover custom tracks once they are lost. You might be able to hack an entry into a user's sessionDb or userDb to point to a custom track file in the trash filesystem. That may work, but it would be tricky. The database setup doesn't help in this regard. It has the same problem. An entry in the user's session or user Db has a pointer to the trash file, which has pointers to the database table names. Once that relationship is broken, it would be difficult to reestablish it. You are correct in your observation that when loading a track that produces errors also causes the loss of existing custom tracks. It would be nice to fix this situation. Again, the database setup doesn't help, it has the same difficulty. For the instructions on how to turn on the database implementation, refer to: http://genomewiki.ucsc.edu/index.php/Using_custom_track_database And in the source tree, the file: README.customTracks.dataBase which should be a copy of the Wiki page. It would be good for mirror sites to move on to the database implementation. Since we have turned that on here in April of 2007, we no longer operate or test the file-based system. The file-based system could be broken at any time and we would be unaware of that situation. --Hiram Davide Cittaro wrote: > Dear mirror people, > I've noticed that uploading some "wrong" gff files as ct (with wrong > number of columns) raises an error on our mirror and not on your > public server. It also happened that the user lost all custom > tracks... :-( > I'm trying to track the issue, but meanwhile I would like to ask a > couple of questions: > > 1- is it possibile to recover an user session with all its custom > tracks? How? I've seen that hguid and hgsid can be helpful but I can't > manage them in a useful way > 2- some time ago I've read a mail where it was discussed a method to > have custom tracks in mysql tables rather than ct_* files on the > filesystem. I would like to set this up, I miss instructions and also > I would like to know if (and how) I can import custom tracks I already > have on the filesystem. > > Thanks in advance > > d From hiram at soe.ucsc.edu Tue Feb 19 10:43:10 2008 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Tue, 19 Feb 2008 10:43:10 -0800 Subject: [Genome-mirror] custom track database In-Reply-To: <47BB1B59.3080101@soe.ucsc.edu> References: <47BB1B59.3080101@soe.ucsc.edu> Message-ID: <47BB233E.7040100@soe.ucsc.edu> Good Morning Mirror Sites: I wanted to highlight and emphasize this note in my previous message to Davide: > It would be good for mirror sites to move on to the database > implementation. Since we have turned that on here in April > of 2007, we no longer operate or test the file-based system. > The file-based system could be broken at any time and we > would be unaware of that situation. See also: http://genomewiki.ucsc.edu/index.php/Using_custom_track_database And in the source tree, the file: README.customTracks.dataBase which should be a copy of the Wiki page. Consider yourself, hereby, forewarned. --Hiram From davide.cittaro at ifom-ieo-campus.it Wed Feb 20 01:35:58 2008 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Wed, 20 Feb 2008 10:35:58 +0100 Subject: [Genome-mirror] handling custom tracks In-Reply-To: <47BB1B59.3080101@soe.ucsc.edu> References: <47BB1B59.3080101@soe.ucsc.edu> Message-ID: On Feb 19, 2008, at 7:09 PM, Hiram Clawson wrote: > > For the instructions on how to turn on the database implementation, > refer to: > http://genomewiki.ucsc.edu/index.php/Using_custom_track_database > And in the source tree, the file: README.customTracks.dataBase > which should be a copy of the Wiki page. I've done with this, I've got only an error: LOAD DATA LOCAL INFILE '/data/httpd/genome.ifom-ieo-campus.it/tmp/ loadBed_genome_40ec_bf3ab0.tab' INTO TABLE t1_genome_40e5_bf3ab0 I've enabled --local-infile either in mysql-server and mysql-client in my.cnf (you should add this to your wiki!). It happens that hgsql -e "LOAD DATA LOCAL INFILE '/data/httpd/genome.ifom-ieo- campus.it/tmp/loadBed_genome_40ec_bf3ab0.tab' INTO TABLE t1_genome_40e5_bf3ab0;" customTrash works. Furthermore cli user and web user are the same to the database... any hint? d /* Davide Cittaro HPC and Bioinformatics Systems @ Informatics Core Cogentech - Consortium for Genomic Technologies via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro at ifom-ieo-campus.it */ From hiram at soe.ucsc.edu Wed Feb 20 12:05:08 2008 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Wed, 20 Feb 2008 12:05:08 -0800 Subject: [Genome-mirror] handling custom tracks In-Reply-To: References: <47BB1B59.3080101@soe.ucsc.edu> Message-ID: <47BC87F4.2040206@soe.ucsc.edu> Good Afternoon Davide: That "LOCAL" thing always causes trouble. In theory it should always work whether the client is on a different machine than the server or on the same machine. It got turned off by default in MySQL versions later than the one we use here for security concerns: http://dev.mysql.com/doc/refman/5.1/en/load-data-local.html So you are correct, one needs to enable that specifically when starting the MySQL system. I'll place a note in the Wiki. Also, please feel free to place your own notes and observations in the Wiki. That's what I'm hoping it would be used for. --Hiram Davide Cittaro wrote: > > On Feb 19, 2008, at 7:09 PM, Hiram Clawson wrote: >> >> For the instructions on how to turn on the database implementation, >> refer to: >> http://genomewiki.ucsc.edu/index.php/Using_custom_track_database >> And in the source tree, the file: README.customTracks.dataBase >> which should be a copy of the Wiki page. > > > I've done with this, I've got only an error: > > LOAD DATA LOCAL INFILE > '/data/httpd/genome.ifom-ieo-campus.it/tmp/loadBed_genome_40ec_bf3ab0.tab' > INTO TABLE t1_genome_40e5_bf3ab0 > > I've enabled --local-infile either in mysql-server and mysql-client in > my.cnf (you should add this to your wiki!). It happens that > > hgsql -e "LOAD DATA LOCAL INFILE > '/data/httpd/genome.ifom-ieo-campus.it/tmp/loadBed_genome_40ec_bf3ab0.tab' > INTO TABLE t1_genome_40e5_bf3ab0;" customTrash > > works. Furthermore cli user and web user are the same to the database... > any hint? > > d From kuhn at soe.ucsc.edu Fri Feb 22 14:33:20 2008 From: kuhn at soe.ucsc.edu (Robert Kuhn) Date: Fri, 22 Feb 2008 14:33:20 -0800 Subject: [Genome-mirror] rsync of UCSC data Message-ID: <200802222233.OAA05868@sundance.cse.ucsc.edu> Dear mirrors, To make our resources as efficient as possible, we are interested in learning how our mirror users access our data. In particular, we would like to know which mirrors download data via: rsync -a rsync://hgdownload.cse.ucsc.edu/genome/ or via: rsync -a rsync://hgdownload.cse.ucsc.edu/mysql/ (or both). Could you please respond directly to me (kuhn at soe.ucsc.edu) and let me know how you access our data? thanks, --b0b kuhn ucsc genome bioinformatics group From davide.cittaro at ifom-ieo-campus.it Mon Feb 25 02:48:11 2008 From: davide.cittaro at ifom-ieo-campus.it (Davide Cittaro) Date: Mon, 25 Feb 2008 11:48:11 +0100 Subject: [Genome-mirror] again on custom tracks, local infile and trash Message-ID: <9708F230-E1BE-448B-8C58-78106C63EB43@ifom-ieo-campus.it> Hi mirror people, I've done with custom tracks in mysql. I would like to warn that mysql (either the server and the client) has to be built with --enable-local- infile option in configure script. Adding --local-infile=1 wouldn't work if there's no support at compile time. Ideally this should be written in your public wiki. On the other side I have some questions concerning maintenance. I have users working on their CT for months, so I can't just delete. dbTrash tells me which tracks have been used in the past N hours, and that's ok, I see I can drop tables from customTrash selectively. I see that files in tmp directory (defined in customTracks.tmpdir) can be deleted safely, they are just used to load mysql... is this right? Ok, and what about trash directory on the filesystem? Once every user has their CT in mysql, can I simply free this directory? I see that CT positions are within the corresponding table in customTrash... I'm asking this because in your public wiki still there are instruction to keep trash directory clean... Thanks in advance d /* Davide Cittaro Cogentech - Consortium for Genomic Technologies via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-mail: davide.cittaro at ifom-ieo-campus.it */ From hiram at soe.ucsc.edu Mon Feb 25 09:59:10 2008 From: hiram at soe.ucsc.edu (Hiram Clawson) Date: Mon, 25 Feb 2008 09:59:10 -0800 Subject: [Genome-mirror] again on custom tracks, local infile and trash In-Reply-To: <9708F230-E1BE-448B-8C58-78106C63EB43@ifom-ieo-campus.it> References: <9708F230-E1BE-448B-8C58-78106C63EB43@ifom-ieo-campus.it> Message-ID: <47C301EE.9030804@soe.ucsc.edu> Good Morning Davide: I have added your recommended note about MySQL building on the Wiki page instructions. Please feel free to add any notes you believe would make things clearer in the Wiki pages. Cleaning trash is your own business. You can set whatever policy you believe is useful to your site. The Wiki page mentions 48 hours, that is the policy we have here. If you start out the trash filesystem with a couple of Tb of disk space, you probably wouldn't have to clean it for a year or more. The tmp files in customTracks.tmpdir should be disappearing after their use when a track is successfully loaded. If the track fails to load, there will be files left in this directory. The cleaning policy for this tmpdir can be pretty much immediate, there is no use for those files beyond their immediate creation and loading into the databse. This is as mentioned in the Wiki page. However, the small files in the UCSC browser directory hierarchy under the /trash/ directory should not be deleted unless you want to clean the tracks from there and from the database. These small files are the pointers to the database table names. If either the database table or the file in trash/ is removed, the custom track disappears from use. This is as mentioned in the Wiki page. --Hiram Davide Cittaro wrote: > Hi mirror people, > I've done with custom tracks in mysql. I would like to warn that mysql > (either the server and the client) has to be built with --enable-local- > infile option in configure script. Adding --local-infile=1 wouldn't > work if there's no support at compile time. Ideally this should be > written in your public wiki. > > On the other side I have some questions concerning maintenance. > I have users working on their CT for months, so I can't just delete. > dbTrash tells me which tracks have been used in the past N hours, and > that's ok, I see I can drop tables from customTrash selectively. I see > that files in tmp directory (defined in customTracks.tmpdir) can be > deleted safely, they are just used to load mysql... is this right? > Ok, and what about trash directory on the filesystem? Once every user > has their CT in mysql, can I simply free this directory? I see that CT > positions are within the corresponding table in customTrash... I'm > asking this because in your public wiki still there are instruction to > keep trash directory clean... > > Thanks in advance > > d From aamp at ucsc.edu Tue Feb 26 11:45:23 2008 From: aamp at ucsc.edu (Andy Pohl) Date: Tue, 26 Feb 2008 11:45:23 -0800 Subject: [Genome-mirror] v176 Genome Browser available Message-ID: <9fa943760802261145p127e1bd5se436b30d9d1b94ef@mail.gmail.com> Hello, Sorry, I forgot to e-mail this the other day. The v176 source is at: http://hgdownload.cse.ucsc.edu/admin/jksrc.zip or labeled with source number: http://hgdownload.cse.ucsc.edu/admin/jksrc.v176.zip cartDump cartReset das hgc hgBlat hgConvert hgCustom hgEncodeDataVersions hgGateway hgGene hgGenome hgLiftOver hgNear hgPcr hgSession hgTables hgTracks hgTrackUi hgVisiGene mkEncodeFrameset pbGateway pbGlobal pbTracks phyloGif /usr/local/apache/cgi-bin/all.joiner /usr/local/apache/cgi-bin/hgNearData/* /usr/local/apache/cgi-bin/hgGeneData/* /usr/local/apache/cgi-bin/hgcData/* /usr/local/apache/cgi-bin/hgCgiData/* /usr/local/apache/cgi-bin/loader/* /usr/local/apache/cgi-bin/visiGeneData/* Contents of v176 may be seen at: http://genome-test.cse.ucsc.edu/builds/versions.html Please check both v176-preview and v176-final to see all the code and data changes in this release. Have a nice week! Andy Pohl UCSC CBSE