Running STRUCTURE on Katak

By Jean-Sebastien Moore
2014-04-08

Install STRUCTURE in your personal Katak folder

Open a terminal window and connect to the server, who will prompt for a password:

    ssh yourfolder@katak.ibis.ulaval.ca

Create folder (make directory; mkdir) for STRUCTURE:

    mkdir structure

Change directory (cd) to the STRUCTURE folder:

    cd ~/structure/

Download the UNIX version of STRUCTURE without GUI from the web using wget (webget):

    wget http://pritchardlab.stanford.edu/structure_software/release_versions/v2.3.4/release/structure_linux_console.tar.gz

The file structure_linux_console.tar.gz should appear in the folder.

Decompress the file:

    tar xvfz structure_linux_console.tar.gz

A new folder named console will appear. Go look into it with the following commands:

    cd console
    ls

The following files should appear: extraparams, mainparams, structure, structure_doc.pdf, testdata1 the structure file is the executable for the program; all the others are test files and the manual.

If you want to remove the test files (recommended), use the following command:

    rm extraparams mainparams testdata1

Move STRUCTURE files from your computer to the Katak server

I recommend creating the data files and param files with the STRUCTURE GUI to avoid formatting mistakes. To do so, just launch a job in the GUI as you normally would with the parameters and data file you want. The main files needed for the command line version of STRUCTURE will then appear in the folder from which you launched the job Those files are project_data extraparams and mainparams (the two param files are within another folder)

Open a new terminal window. Change directory to where your STRUCTURE files are; this will depend on where you put them; the following code assumes they are in a floder names structure on your Desktop on your Mac cd

    ~/Desktop/structure/

Secure copy (scp) the files to the structure folder in your Katak folder; will prompt for password

    scp project_data mainparams extraparams yourfolder@katak.ibis.ulaval.ca:/home/yourfolder/structure

Run STRUCTURE

To run STRUCTURE you need to be in the folder where the structure executable file is; the param and data files also need to be in that folder

run a single run of STRUCTURE with K = 2; this will write the results to the file output_file

    ./structure -i project_data -o output_file -K 2

You are more likely to run a full STRUCTURE job (i.e. multiple K and multiple iterations of K). To do so, you will need to submit your job properly to the server using qsub. The following steps will help you create two .sh script files that will take care of submitting a STRUCTURE batch job to Katak. In both of these files, you will need to modify a few parameters. You are strongly encouraged to learn how to use Joe (or another editing tool) to do so directly on the server.

Create a file containing the script for the structure run. Start by creating an empty file with Joe

    joe template_structure_job.sh

You will now be in Joe ready to edit the empty file template_structure_job.sh

Copy the script below in Joe, making the necessary changes:

change the name of your job after the flag -N (currently str_job); this will be used to name the log files
put your email address after the -M flag
change the amount of memory needed (currently 1G) and time necessary for the run (hh:mm:ss) after the -l flags
change the maximum number of K you want to run (MAX_K=), e.g. for K = 1 to 10, write MAX_K=10

For the last two variables, you will likely need to run some tests with your own data set. With the current workflow, each job will run one iteration of all Ks. So if you want to run big Ks on big datasets, it will still take a long time!!!

change the name of the folder in which the outfile will be placed by changing the text after OUTPUT_DIR

    #! /bin/bash

    #$ -N str_job
    #$ -M YOUREMAIL@ulaval.ca
    #$ -m beas
    #$ -pe smp 1
    #$ -l h_vmem=10G
    #$ -l h_rt=00:10:00
    #$ -cwd
    #$ -S /bin/bash

    # Global variables
    MAX_K=10
    OUTPUT_DIR=structure_output

    # Do not modify
    RUN_ID=__RUN_ID__

    # Create output folder
    mkdir $OUTPUT_DIR 2> /dev/null

    # Job
    for k in $(seq $MAX_K)
    do
        ./structure -m mainparams -K ${k} -i project_data -o $OUTPUT_DIR/outfile_k${k}
    done

press control-k then x to save and exit Joe; DO NOT press all three keys at a time: first ctrl-K, then x
you will now see a file named structure_batch.sh in your structure folder (if you type ls)
Create a file that will run multiple independent repicates of the file above.
start by creating an empty file with Joe
```
joe create_and_submit_jobs.sh
```

Copy the script below, making the necessary changes:

NUM_REP= specifies the number of replicates you want to run, currently 10

You can also change the names of the template and basename files

#!/bin/sh

# Global variables
NUM_REP=10
TEMPLATE=template_structure_job.sh
BASENAME=structure_job_

for i in $(seq $NUM_REP)
    do
        JOB_FILE=$BASENAME${i}.sh
        cp $TEMPLATE $JOB_FILE
        perl -i -sape **s/__RUN_ID__/$i/** -- -i=$i $JOB_FILE
        qsub $JOB_FILE
    done

press control-k then x to save and exit Joe. DO NOT press all three keys at a time: first control-k, then x.

You will now see a file named create_and_submit_jobs.sh in your structure folder (if you type ls)

Submit your job!

    qsub create_and_submit_jobs.sh

You will receive an e-mail when the job starts, and when it ends. The output files will appear in your structure folder.