Install a local mirror

From wubrowse wiki
Jump to: navigation, search

Preparing your server

sudo apt-get install make
sudo apt-get install gcc

sudo apt-get install apache2
sudo apt-get install mysql-server
sudo apt-get install libmysqlclient-dev libmysqld-dev
sudo apt-get install unzip
sudo apt-get install r-base

# this is for 32-bit systems
# sudo apt-get install lib32z1-dev lib32ncurses5-dev

# for 64-bit systems
sudo apt-get install lib64z1-dev lib64ncurses5-dev

To enable HTTPS support, do following: 

sudo apt-get install libssl-dev
sudo apt-get install libghc-curl-dev
sudo apt-get install libssh2-1-dev libssh2-1

To enable pdf output: 

sudo apt-get install librsvg2-bin

May need to enable Apache CGI 

sudo a2enmod cgi
sudo service apache2 restart

Get the newest code

git clone https://github.com/epgg/eg.git
  • the code source could be put anywhere on your server

Compile the cgi-bin program

  • after getting the newest code, go to the cig-bin folder, then go to the jsmn-example and jsmn folder, type make 
cd cgi-bin/jsmn-example
make
cd ../jsmn
make
  • the go to parent folder, type make again 
cd ..
make
    • please note you need edit subtleKnife.c and postdeposit.c first
    • edit these 2 .c files with correct database host, trash directory
    • in subtleKnife.c, change the following: 
#define WWWT "/var/www/browser/t"
#define BINdir "."
#define Mysqlserver "localhost"
#define Mysqluser "hguser"
#define Mysqlpswd "hguser"
    • in postdeposit.c, chane the following: 
#define WWWT "/var/www/browser/t"
  • you could recompile the files after making these changes as well
  • you will get the subtleKnife and postdeposit file which are executable, these 2 files need be put the Apache cgi-bin folder
  • for Ubuntu system, it would be /usr/lib/cgi-bin by default 
cp subtleKnife postdeposit /usr/lib/cgi-bin/
  • optional, this python script is used to convert UCSC trackDb format to our datahub json format: 
 cp script/ucsc/ucsc2jsonhub.py /usr/lib/cgi-bin/

Directories

sudo mkdir /srv/epgg
chown MYNAME /srv/epgg
mkdir /srv/epgg/data

cd /srv/epgg/data
mkdir data
cd data/
mkdir subtleKnife
cd subtleKnife
mkdir seq

"/srv/epgg/data/data/subtleKnife/seq" is the directory hosting genome sequence files.

Say if you want to create genome assembly for hg19: 

mkdir /srv/epgg/data/data/subtleKnife/hg19
mkdir /srv/epgg/data/data/subtleKnife/hg19/config
mkdir /srv/epgg/data/data/subtleKnife/hg19/session
chown www-data.www-data /srv/epgg/data/data/subtleKnife/hg19/session

You need put gene related track files for erg under "/srv/epgg/data/data/subtleKnife/hg19".

Note: since ubuntu 14.04, the default Apache path is "/var/www/html" instead of "/var/www/". This is hard-coded in CGI code files subtleKnife.c and postdeposit.c.

If you have different paths, update these two files (just search for "www" in the source code).

Two trash directories, both are apache-writable: 

mkdir /var/www/html/browser/t
mkdir /srv/epgg/data/trash
sudo chown www-data.www-data /var/www/html/browser/t /srv/epgg/data/trash
sudo ln -s /srv/epgg/data/trash /usr/lib/

Install browser

Put html, javascript files to web folder

  • after git clone 
mv css/ index.html js/ images/ /var/www/html/browser/

Edit file "treeoflife" to add/remove genomes 

mv config/treeoflife /srv/epgg/data/data/subtleKnife/

hub and tracks config file

  • put tracks.json, publichub.json file to the config folder for each genome assembly, for example, for human hg19 
cp config/hg19/tracks.json /srv/epgg/data/data/subtleKnife/hg19/config
cp config/hg19/publichub.json /srv/epgg/data/data/subtleKnife/hg19/config
  • you need download the gene related tracks file to the folder /srv/epgg/data/data/subtleKnife/hg19/, for human hg19
    • open tracks.json file, download following file to /srv/epgg/data/data/subtleKnife/hg19 
http://egg.wustl.edu/d/hg19/refGene.gz
http://egg.wustl.edu/d/hg19/refGene.gz.tbi
http://egg.wustl.edu/d/hg19/gencodeV17.gz
http://egg.wustl.edu/d/hg19/gencodeV17.gz.tbi
http://egg.wustl.edu/d/hg19/xenoRefGene.gz
http://egg.wustl.edu/d/hg19/xenoRefGene.gz.tbi

load database

  • create a username called hguser with password hguser (login to mysql console with root first): 
mysql –u root -p
> GRANT ALL ON *.* TO 'hguser'@'localhost' IDENTIFIED BY 'hguser' WITH GRANT OPTION;
> FLUSH PRIVILEGES;
  • use human hg19 for example
  • go to your mysql console, create a database called hg19: 
> create database hg19;
  • load sessionUtils.sql from the config folder 
cd config
cat sessionUtils.sql | mysql -u hguser -p hg19
  • go the hg19 folder, load the makeDb.sql to hg19 database 
cd hg19
cat makeDb.sql | mysql -u hguser -p hg19
    • if you see this error: 
ERROR 1148 (42000) at line 51: The used command is not allowed with this MySQL version
    • add the following option: 
cat makeDb.sql | mysql -u hguser -p hg19 --local-infile=1
  • fetch the newest database load file from github for gene information: 
git clone https://github.com/epgg/load.git
  • go to hg19 folder, load the load.sql file to hg19 database 
cat load.sql | mysql -u hguser -p hg19
    • if you see same error, adding --local-infile=1

Supporting bigWig and bigBed

Enable the embedding feature

    • Edit /etc/apache2/apache2.conf, add last three lines to enable CORS 
<Location /cgi-bin>
Header set Access-Control-Allow-Origin "*"
</Location>
  • Run these shell commands to take effect 
a2enmod headers
service apache2 restart

Add a private genome

Archive

  • the following are instructions for old releases, no need to follow

Supporting bigWig

Skip this part to disable bigWig support. You can always use bedGraph tracks instead.

Check if you comply with their licensing requirement before you download the UCSC browser source code. Certain amount of labor is required to compile it (libpng is only needed to get it compiled without complaints). 

sudo apt-get install libpng12-dev
wget http://hgdownload.soe.ucsc.edu/admin/jksrc.zip
unzip jksrc.zip 
cd kent/src/
export MACHTYPE="x86_64"
export USE_SSL=1
make libs
mkdir utils/bwquery
cd utils/bwquery
mv /srv/epgg/data/subtleKnife/bwquery/* .
make
mv bwquery /usr/lib/cgi-bin/

Done compiling bwquery. The bwquery binary file will be called by subtleKnife for bigWig access. To test the bwquery binary, run this command: 

/usr/lib/cgi-bin/bwquery http://vizhub.wustl.edu/hubSample/hg19/sample.bigWig chr7 27053199 27373567 20 bwOutput.txt 1
more bwOutput.txt

(You could download the kent source tree to anywhere. the steps above assume you compile bwquery under kent source tree, If you want to put bwquery out of kent source tee, after compile kent source, make soft link of inc and lib folder to the 2nd parent folder of bwquery) 

ln -s kent_folder/src/lib 2nd_parent_folder_of_bwquery
ln -s kent_folder/src/inc 2nd_parent_folder_of_bwquery

The browser is now enabled to access bigWig tracks.

Pull data

  • Update: you only need pull gene related track file
  • So don't need to follow
  • Pull data from our server to rebuild the genome of your choice
  • See instructions in Pull annotation data

Get the "call super enhancer" app

  • no need to follow

This app is disabled on the public browser site at the moment. You can get it enabled in your locally installed browser.

Make sure "samtools" can be executed at command line without specifying path. 

cd /srv/epgg/data
wget http://cgs.wustl.edu/~xzhou/file/samtools-0.1.19.tar.bz2
bunzip2 samtools-0.1.19.tar.bz2
tar xvf samtools-0.1.19.tar
cd samtools-0.1.19
make
sudo ln -s /srv/epgg/data/samtools-0.1.19/samtools /usr/local/bin/

Compile "bamliquidator": 

cd ~/subtleKnife/script/bamliquidator
./make.sh
  • Put the binary into command search PATH 
sudo mv bamliquidator /usr/local/bin/
  • Edit "/etc/at.deny", delete "www-data" so that Apache process can submit batch jobs 
sudo apt-get install sendmail-bin
  • Make sure the www-data user can execute these commands 
sudo su
su www-data
samtools
bamliquidator
/usr/sbin/sendmail
batch

Download a version of "rose" modified by Xin Zhou 

mkdir /srv/epgg/data/rose
cd /srv/epgg/data/rose
wget http://cgs.wustl.edu/~xzhou/rose.tgz
tar zxvf rose.tgz

Tabix and Samtools

  • Updated, no need to follow

Tabix and Samtools are core utility libraries used by the browser. You must compile the two programs from source code but not installing them from a software repository (as via apt-get on Ubuntu). Tabix and Samtools are used as libraries for compiling browser's CGI program called "subtleKnife", and their binaries (tabix and samtools) are not used by the browser.


For the moment, we need non-standard versions of Samtools and Tabix to enable HTTPS support. 

cd /srv/epgg/data/
wget http://cgs.wustl.edu/~xzhou/samtools-legacy.tgz
tar zxvf samtools-legacy.tgz
cd samtools-legacy
make

Make sure samtools works properly 

./samtools view http://vizhub.wustl.edu/hubSample/hg19/bam1.bam

Download a version of tabix modified by Daofeng Li and compile. However the Tabix we're using here does not work well on 32-bit computers It has trouble accessing files with size 4GB or larger. It works all right on 64-bit systems. 

cd /srv/epgg/data
wget http://cgs.wustl.edu/~xzhou/tabix-lidaof.tgz
tar zxvf tabix-lidaof.tgz
cd tabix-lidaof/
make

Make sure tabix works properly. 

./tabix -f http://vizhub.wustl.edu/hubSample/hg19/refGene.gz chr7:27053199-27373567
Retrieved from "https://wiki.wubrowse.org/index.php?title=Install_a_local_mirror&oldid=1510"