Skip navigation and jump directly to page content

 IU Trident Indiana University

UITS Research Technologies

Cyberinfrastructure enabling research and creative activities
banner-image

Karst at Indiana University

On this page:


System overview

Karst (karst.uits.iu.edu) is Indiana University's newest high-throughput computing cluster. Designed to deliver large amounts of processing capacity over long periods of time, Karst's system architecture provides IU researchers the advanced performance needed to accommodate high-end, data-intensive applications critical to scientific discovery and innovation. Karst also serves as a "condominium cluster" environment for IU researchers, research labs, departments, and schools.

Karst is housed in the IU Bloomington Data Center and offers the following high-performance computing service models to the IU community:

  • High-throughput computing: Anyone with an account on Karst can use the system for standard cluster-based, high-throughput computing (i.e., submit batch jobs that are prioritized by fairshare and other scheduling policies, and executed as resources become available).
  • Note: Karst replaces the Quarry cluster, which was decommissioned January 30, 2015. IU's Big Red II and Mason systems will continue to provide high-performance and memory-intensive computation, respectively.

  • Condominium computing: The condominium computing service model provides a way for IU schools, departments, and researchers to fund computational nodes for their own research purposes without shouldering the cost, overhead, and management requirements of purchasing individual systems.
  • Condominium nodes are housed in the IU Bloomington Data Center, and are managed, backed up, and secured by UITS Research Technologies staff. Condominium nodes are available to "members" whenever they are needed, but when they are not in use, idle condominium nodes become available to other researchers and students on Karst. In this way, condominium computing promotes cost-effective expansion of IU's high-performance computing capabilities, enables efficient provisioning of computing resources to the entire IU research community, and helps conserve natural resources and energy.

  • Dedicated computing: The dedicated computing service model lets schools and departments host nodes that are dedicated solely to their use within Karst's physical, electrical, and network framework. This provides 24/7 access for school or departmental use, while leveraging the network and physical components of Karst, and the security and energy efficiency benefits provided by location within the IU Data Center.

If you are interested in the condominium or dedicated service models, or have questions, contact Research Technologies. If you have questions about Karst in general, contact High Performance Systems.

System information

Note:
The scheduled monthly maintenance window for Karst is the first Tuesday of each month, 7am-7pm.

Karst is equipped with 256 compute nodes, plus 16 dedicated data nodes for separate handling of data-intensive operations. All nodes are IBM NeXtScale nx360 M4 servers, each equipped with two Intel Xeon E5-2650 v2 8-core processors. Each compute node has 32 GB of RAM and 250 GB of local disk storage. Each data node has 64 GB of RAM and 24 TB of local storage. All nodes run Red Hat Enterprise Linux (RHEL) 6 and are connected via 10-gigabit Ethernet to the IU Science DMZ.

Karst provides batch processing and node-level co-location services that make it well suited for running high-throughput and data-intensive parallel computing jobs. Karst uses TORQUE integrated with Moab Workload Manager to coordinate resource management and job scheduling. The Data Capacitor II and Data Capacitor Wide Area Network (DC-WAN) parallel file systems are mounted for temporary storage of research data. The Modules environment management package on Karst allows users to dynamically customize their shell environments.

System summary
Machine type High-performance, high-throughput, data-intensive cluster computing
Operating system Red Hat Enterprise Linux 6.x
Memory model Distributed
Nodes IBM NeXtScale nx360 M4 servers
256 compute nodes
16 data nodes
Network 10-gigabit Ethernet per node
Compute node details Total Per node
CPUs 512 Intel Xeon E5-2650 v2 8-core processors 2 Intel Xeon E5-2650 v2 8-core processors
Processor cores 4,096 16
RAM 8 TB 32 GB
Local storage 64 TB 250 GB
Data node details Total Per node
CPUs 32 Intel Xeon E5-2650 v2 8-core processors 2 Intel Xeon E5-2650 v2 8-core processors
Processor cores 256 16
RAM 1 TB 64 GB
Local storage 384 TB 24 TB
Performance
Processing capability Rpeak = 85.1 teraFLOPS Rpeak = 332.8 gigaFLOPS
Power usage 0.89 gigaFLOPS per watt  

Working with data containing PHI

The Health Insurance Portability and Accountability Act of 1996 (HIPAA) established rules protecting the privacy and security of individually identifiable health information. The HIPAA Privacy Rule and Security Rule set national standards requiring organizations and individuals to implement certain administrative, physical, and technical safeguards to maintain the confidentiality, integrity, and availability of protected health information (PHI).

This system meets certain requirements established in the HIPAA Security Rule that enable its use for research involving data that contain protected health information (PHI). You may use this resource for research involving data that contain PHI only if you institute additional physical, administrative, and technical safeguards that complement those UITS already has in place. For more, see When using UITS Research Technologies systems and services, what are my legal responsibilities for protecting the privacy and security of data containing protected health information (PHI)? If you need help or have questions, contact UITS HIPAA Consulting.

Note:
Although PHI is a type of institutional data classified as Critical by the IU Committee of Data Stewards, other types of institutional data classified as Critical are not permitted on Research Technologies systems. Except for PHI, the most sensitive classification of institutional data allowed on Research Technologies resources is Restricted. For help determining which institutional data elements classified as Critical are considered PHI, see Which data elements in the classifications of institutional data are considered protected health information (PHI)?

Requesting an account

IU students, faculty, staff, and sponsored affiliates can request accounts on Karst.

Note:
Accounts remain valid only while the account holder is a registered IU student, or an IU faculty or staff member. On Big Red II, Karst, Mason, and the Research Database Complex, accounts are disabled during the semester following the account holder's departure from IU, and then are purged within six months. To exempt a research systems account from disabling, email a request to the Accounts Administration team. If the request is approved, the account will remain activated for one calendar year beyond the user's departure from IU. Then, at the end of the year, the account will be purged. Extensions beyond one year for research accounts are granted only for accounts involved in funded research and having an IU faculty sponsor, or with approval of the Dean or Director of Research and Academic Computing.

Accessing Karst

Once your account is created, you can access Karst via SSH or with the Karst Desktop Beta graphical remote desktop application:

  • SSH: Use an SSH2 client to connect to karst.uits.iu.edu); for example:
  •   ssh username@karst.uits.iu.edu
    

    Replace username with your IU Network ID username, and then authenticate using your Network ID passphrase. Alternatively, you can set up public-key authentication. SSH connections to karst.uits.iu.edu resolve to either of the following login nodes:

      h1.karst.uits.iu.edu
      h2.karst.uits.iu.edu
      h3.karst.uits.iu.edu
      h4.karst.uits.iu.edu
    
  • Karst Desktop Beta: Karst Desktop Beta is a remote desktop service that lets you work on Karst from a graphical desktop window running on your personal computer. Karst Desktop Beta provides numerous features that are especially helpful to users who are unaccustomed to working in Unix-like command-line environments. For more, see At IU, what is Karst Desktop Beta, and how do I connect to it from my personal computer?
Note:
In accordance with standards for access control mandated by the HIPAA Security Rule, you are not permitted to access data containing protected health information (PHI) using a group (or departmental) account. To ensure accountability and permit access to authorized users only, IU researchers must use their personal IU username and passphrase for all work involving PHI.

File systems

You can store files on your home directory or in scratch space:

  • Home directory: Your Karst home directory disk space is allocated on a network-attached storage (NAS) device. You have a 100 GB disk quota, which is shared (if applicable) with your accounts on Big Red II, Mason, and the Research Database Complex (RDC).
  • The path to your home directory is (replace username with your Network ID username):

      /N/u/username/Karst
    

    IU graduate students, faculty, and staff who need more than 100 GB of permanent storage can apply for accounts on the Scholarly Data Archive (SDA). See At IU, how do I apply for an account on the SDA?

    To check your quota, use the quota command. If the quota command is not already loaded by default, use the module load command to add it to your environment:

    module load quota
    

    To make permanent changes to your environment, edit your ~/.modules file. For more, see In Modules, how do I save my environment with a .modules file?

  • Local scratch: Karst provides 180 GB of local scratch disk space. Local scratch space is not intended for permanent storage of data, and is not backed up. Files in local scratch space are automatically deleted once they are 14 days old.
  • The path to the local scratch directory on Karst is:

      /tmp
    
  • Shared scratch: Once you have an account on one of the UITS research computing systems, you also have access to 3.5 PB of shared scratch space.
  • Shared scratch space is hosted on the Data Capacitor II (DC2) file system. The DC2 scratch directory is a temporary workspace. Scratch space is not allocated, and its total capacity fluctuates based on project space requirements. The DC2 file system is mounted on IU research systems as /N/dc2/scratch and behaves like any other disk device. If you have an account on an IU research system, you can access /N/dc2/scratch/username (replace username with your IU Network ID username). Access to /N/dc2/projects requires an allocation. For details, see The Data Capacitor II and DC-WAN high-speed file systems at Indiana University. Files in shared scratch space may be purged if they have not been accessed for more than 60 days.

For more, see On IU's research systems, how much allocated and short-term storage capacity is available to me?

Transferring your files to Karst

Karst supports SCP and SFTP for transferring files:

  • SCP: This command-line utility is included with OpenSSH. Basic use is:
  •   scp username@host1:~/file1 username@host2:~/file1_copy
    

    For example, to copy a file from your home directory on your local computer (e.g., ~/foo.txt) to your home directory on Karst, on the command line, enter (replace username with your IU Network ID username):

      scp ~/foo.txt username@karst.uits.iu.edu:~/foo.txt
    

    For more, see In Unix, how do I use SCP to securely transfer files between two computers?

  • SFTP: Use a command-line or graphical SFTP client to access, transfer, and manage your files.
  • For example, to transfer a file using command-line SFTP from your home directory on your local computer (e.g., ~/foo.txt) to your home directory on Karst, on the command line, enter (replace username with your IU Network ID username; at the password prompt, enter your Network ID passphrase):

      $ sftp username@karst.uits.iu.edu
      username@karst.uits.iu.edu's password:
      Connected to karst.uits.iu.edu.
      sftp> put foo.txt
      Uploading foo.txt to /N/hd00/username/Karst/foo.txt
      foo.txt 100% 43MB 76.9KB/s 09:39
      sftp> exit
    

    For more, see What is SFTP, and how do I use it to transfer files?

Computing environment

Supported shells

By default, new accounts on Karst are assigned the the Bourne-again shell (bash). Karst also supports the TENEX C shell (tcsh). To change your shell on Karst, send a request to the UITS High Performance Systems team indicating the shell you'd prefer to use.

For more about the Bash shell, see the Bash Reference Manual. For more about the TCSH shell, see the TCSH manual page.

Environment variables

To display the values of environment variables set for your user environment on Karst, on the command line, enter:

  printenv

To display the value of a particular environment variable (e.g., VARNAME), enter:

  echo $VARNAME

To change the value of an environment variable (e.g., VARNAME), enter:

  • In bash:
  •   export VARNAME=NEW_VALUE
    
  • In tcsh:
  •   setenv VARNAME NEW_VALUE
    

Startup files

When you log into Karst, your shell reads and executes commands from certain startup files:

  • The bash shell reads and executes commands from these files (in this order):
  •   /etc/profile 
      ~/.bash_profile 
      ~/.bashrc
    
    Note:
    The ~ (tilde) represents your home directory (e.g., ~/.bash_profile is the .bash_profile file in your home directory).
  • The tcsh shell reads and executes commands from these files (in this order):
  •   /etc/csh.cshrc
      /etc/csh.login
    

For more, see:

Setting up your user environment

On the research computing resources at Indiana University, the Modules environment management system provides a convenient method for dynamically customizing your software environment.

Modules is a command-line interface that provides commands for setting and modifying shell environment variables. These environment variables define values used by both the shell and the programs you execute on the shell command line.

The Modules environment management package simplifies the management of environment variables associated with various software packages, and lets you automatically modify environment variables as needed when switching between software packages.

Some common Modules commands include:

Command Action
module avail List all software packages available on the system.
module avail package List all versions of package available on the system; for example:

  module avail openmpi
module list List all packages currently added to your user environment.
module load package Add the default version of the package to your user environment; for example:

  module load intel
module load package/version Add the specified version of the package to your user environment; for example:

  module load intel/11.1
module unload package Remove the specified package from your user environment.
module swap package_A package_B Swap the loaded package (package_A) with another package (package_B). This is synonymous with:

  module switch package_A package_B
module show package Show the changes loading the specified package makes to your user environment (e.g., environment variables set, library paths added). This is synonymous with:

  module display package

To make permanent changes to your environment, edit your ~/.modules file. For more, see In Modules, how do I save my environment with a .modules file?

For more about the Modules package, see the module manual page and the modulefile manual page. Additionally, see On Big Red II, Karst, and Mason at IU, how do I use Modules to manage my software environment?

Available software

The Scientific Applications and Performance Tuning (SciAPT) team maintains the software on Karst and the other research computing systems at IU.

For a list of packages available on Karst, see Karst Modules in the IU Cyberinfrastructure Gateway. Alternatively, you can log into your Karst account and enter the module avail command.

Karst users can request software using the Software Request form.

Application development

Compilers

The GNU Compiler Collection (GCC) is added by default to your user environment on Karst. The Intel and Portland Group (PGI) compiler collections, and the Open MPI and MPICH wrapper compilers, are also available.

Recommended optimization options are -O3 and -xHost (the -xHost option will optimize based on the processor of the current host).

For the GCC compilers, the mtune=native and march=native options are recommend to generate instructions for the machine and CPU type.

Following are example commands for compiling serial and parallel programs on Karst:

  • Serial programs:
    • To compile the C program simple.c :
      • With the GCC compiler:
      •   gcc -O2 -o -mtune=native -march=native simple simple.c
        
      • With the Intel compiler:
      •   icc -o simple simple.c
        
    • To compile the Fortran program simple.f:
      • With the GCC compiler:
      •   g77 -o simple simple.f
        
      • With the Intel compiler:
      •   ifort -O2 -o simple -lm simple.f
        
  • Parallel programs:
    • To compile the C program simple.c with the MPI wrapper script:
    •   mpicc -o simple simple.c
      
    • To compile the Fortran program simple.f with the MPI wrapper script:
    •   mpif90 -o simple -O2 simple.f
      
    • To use the GCC C compiler to compile simple.c to run in parallel using OpenMP:
    •   gcc -O2 -fopenmp -o simple simple.c
      
    • To use the Intel Fortran compiler to compile simple.f to run in parallel using OpenMP:
    •   ifort -openmp -o simple -lm simple.f
      

Libraries

  • MKL: The Intel Math Kernel Library (MKL) is available on Karst for use with the Intel Compiler Suite. Loading the intel module automatically sets the environment variables needed to link your code with the MKL library.
  • ACML: The AMD Core Math Library (AMCL) is available on Karst for use with the Portland Group compilers. Loading the pgi module automatically sets the environment variables need to link your code with the ACML library.

Debugging

Both the Intel Debugger (IDB) and the GNU Project Debugger (GDB) are available on Karst.

For information about using the IDB, see the Intel IDB page.

For information about using the GDB, see the GNU GDB page. For an example, see Step-by-step example for using GDB within Emacs to debug a C or C++ program.

Queue information

Karst employs a default routing queue that funnels jobs, according to their resource requirements, into three execution queues configured to maximize job throughput and minimize wait times (i.e., the amount of time a job remains queued, waiting for required resources to become available). Depending on the resource requirements specified in either your batch job script or your qsub command, the routing queue (BATCH) automatically places your job into the SERIAL, NORMAL, or LONG queue.

Note:
The maximum wall time allowed for jobs running on Karst is 14 days. If your job requires more than 14 days of wall time, email the High Performance Systems group for assistance.

You do not have to specify a queue in your job script or in your qsub command to submit your job to one of the three batch execution queues; your job will run in the SERIAL, NORMAL, or LONG queue unless you specifically submit it to the DEBUG, PREEMPT, or INTERACTIVE queue, the properties of which are as follows:

  • DEBUG: The DEBUG queue is intended for short, quick-turnaround test jobs requiring less than 30 minutes of wall time.
  • Maximum wall time: 30 minutes
    Maximum nodes per job: None
    Maximum cores per job: 32
    Maximum number of jobs per queue: None
    Maximum number of jobs per user: 2
    Direct submission: Yes

    To submit a batch job to the DEBUG queue, either add the #PBS -q debug directive to your job script, or enter qsub -q debug on the command line.

    Note:
    For longer debugging or testing sessions, submit an interactive job to the INTERACTIVE queue instead.
  • INTERACTIVE: Interactive jobs submitted to the INTERACTIVE queue should experience less wait time (i.e., start sooner) than interactive jobs submitted to the batch execution queues.
  • Maximum wall time: 8 hours
    Maximum nodes per job: None
    Maximum cores per job: 1
    Maximum number of jobs per queue: 128
    Maximum number of jobs per user: 16
    Direct submission: Yes

    To submit an interactive job to the INTERACTIVE queue, on the command line, enter qsub with the -I and -q interactive options added; for example:

      qsub -I -q interactive -l nodes=1:ppn=1,walltime=4:00:00
    
    Note:
    If you enter qsub without the -q interactive option, your interactive job will be placed in the routing queue for submission to the SERIAL, NORMAL, or LONG batch execution queue, which most likely will entail a longer wait time for your job.
  • PREEMPT: Jobs submitted to the PREEMPT queue run on "condominium nodes" owned by members of the "condominium computing" service; however, when a job submitted by a "condominium node" owner is ready to dispatch (and no other nodes are available), the non-condominium job with the lowest accrued wall time will be preempted. Consequently, non-condominium jobs in the PREEMPT queue may dispatch multiple times before running to completion.
  • Maximum wall time: 14 days
    Maximum nodes per job: None
    Maximum cores per job: None
    Maximum number of jobs per queue: 1,800
    Maximum number of jobs per user: 200
    Direct submission: Yes

    To submit a job to the PREEMPT queue, add the #PBS -q preempt directive to your job script, or enter qsub -q preempt on the command line.

To see current status information for the work queues on Karst, on the command line, enter:

  qstat -q
Note:
To best meet the needs of all research projects affiliated with Indiana University, the High Performance Systems (HPS) team administers the batch job queues on UITS Research Technologies supercomputers using resource management and job scheduling policies that optimize the overall efficiency and performance of workloads on those systems. If the structure or configuration of the batch queues on any of IU's supercomputing systems does not meet the needs of your research project, fill out and submit the Research Technologies Ask RT for Help form (for "Select a group to contact", select High Performance Systems).

Running jobs on Karst

The default wall time for jobs running on Karst compute nodes is 30 minutes; the default virtual memory per job is 8 MB.

Note:
User processes on the login nodes are limited to 20 minutes of CPU time. Processes on the login nodes that run longer than 20 minutes are terminated automatically (without warning). If your application requires more than 20 minutes of CPU time, submit a batch job or an interactive session using the TORQUE qsub command. Because of this limit:
  • When running Java programs, add the -Xmx parameter (values must be multiples of 1,024 greater than 2 MB) on the command line to specify the Java Virtual Machine (JVM) maximum heap size. For example, to run a Java program (e.g., Hello_DeathStar) with a maximum heap size of 640 MB , on the command line, enter:
  •   java -Xmx640m Hello_DeathStar
    
  • You should perform any debugging or testing by running interactive jobs on the compute nodes. Specify the DEBUG queue for short, quick-turnaround test jobs (maximum wall time: 30 minutes) or the INTERACTIVE queue for longer jobs (maximum wall time: 8 hours).

Submitting jobs

IU's research computing clusters use the TORQUE resource manager (based on OpenPBS) and the Moab Workload Manager to manage and schedule batch jobs. For information about using TORQUE on IU research computing clusters, see What is TORQUE, and how do I use it to submit and manage jobs on high-performance computing systems? Moab uses fairshare scheduling to track usage and prioritize jobs. For information on fairshare scheduling and using Moab to check the status of batch jobs, see:

Use the TORQUE qsub command to submit non-interactive or interactive batch jobs for execution on Karst's compute nodes:

  • Non-interactive jobs: To run a job in batch mode on Karst, first prepare a TORQUE job script that specifies the application you want to run and the resources required to run it, and then submit it to TORQUE with the qsub command.
  • Do not specify a destination queue in your job script or your qsub command. TORQUE passes your job to the system's default routing queue (BATCH) for placement, based on its resource requirements, in the SERIAL, NORMAL, or LONG execution queue. From there, your job dispatches whenever the required resources become available.

    If your job has resource requirements that are different from the defaults (but not exceeding the maximums allowed), specify them either with TORQUE directives in your job script, or with the -l (a lower-case "L"; short for resource_list) option in your qsub command.

    Note:
    Command-line options override PBS directives in your job script.

    On the command line, you can specify multiple attributes with either one -l switch followed by multiple comma-separated attributes, or multiple -l switches, one for each attribute. For example, to submit a job (e.g., death_star.script) that requires 24 hours of wall time (instead of the default 30 minutes) and 100 GB of virtual memory (instead of the default 8 MB) to run on four cores on one node, you may enter either of the following commands (they are equivalent):

      qsub -l nodes=1:ppn=4,vmem=10gb,walltime=24:00:00 death_star.script
      qsub -l nodes=1:ppn=4 -l vmem=10gb -l walltime=24:00:00 death_star.script
    
    Note:
    Each Karst compute node has 16 CPU cores available, allowing you to "stack" up to 16 jobs on one compute node (provided none of the jobs has unique resource constraints, such as special memory requirements). To run up to 16 jobs "stacked" on a single compute node, specify -l nodes=1:ppn=1 in the script or qsub command for each job.
  • Interactive jobs: To submit an interactive job, use qsub with the -I (to specify an interactive job) and -q interactive (to specify submission to the INTERACTIVE batch queue) switches; for example:
  •   qsub -I -q interactive -l nodes=1:ppn=4,vmem=10gb,walltime=4:00:00
    

    Submitting your job to the INTERACTIVE queue directs it to a specific set of nodes that are configured for shared access (versus single-user in the general batch queues). Consequently, your interactive job most likely will dispatch faster in the INTERACTIVE queue than in the general execution queues.

Useful qsub options include:

Option Function
-a <date_time> Execute the job only after specified date and time (<date_time>).
-I Run the job interactively. (Interactive jobs are forced to be not re-runnable.)
-m e Mail a job summary report when the job terminates.
-q <queue_name> Specify the destination queue (<queue_name>) for the job. (On Karst, use this only for submitting jobs to the DEBUG, INTERACTIVE, or PREEMPT queues.)
-r [y|n] Declare whether the job is re-runnable. Use the argument n if the job is not re-runnable. The default value is y (re-runnable).
-V Export all environment variables in your current environment to the job.

For more, see the qsub manual page.

Monitoring jobs

To monitor the status of a queued or running job, use the TORQUE qstat command.

Useful qstat options include:

Option Action
-a Display all jobs.
-f Write a full status display to standard output.
-n List the nodes allocated to a job.
-r Display jobs that are running.
-u user1,user2 Display jobs owned by specified users.

For more, see the qstat manual page.

Deleting jobs

To delete a queued or running job, use the qdel command.

Note:
Occasionally, a node will become unresponsive and unable to respond to the TORQUE server's requests to kill a job. In such cases, try using qdel -W <delay> to override the delay between SIGTERM and SIGKILL signals; for <delay>, specify a value in seconds.

For more, see the qdel manual page.

Requesting single user time

Although UITS Research Technologies cannot provide dedicated access to an entire compute system during the course of normal operations, "single user time" is made available by request one day a month during each system's regularly scheduled maintenance window to accommodate IU researchers with tasks requiring dedicated access to an entire compute system. To request such single user time, complete and submit the Research Technologies Ask RT for Help form, requesting to run jobs in single user time on HPS systems. If you have questions, email the HPS team.

Acknowledging grant support

The Indiana University cyberinfrastructure, managed by the Research Technologies division of UITS, is supported by funding from several grants, each of which requires you to acknowledge its support in all presentations and published works stemming from research it has helped to fund. Conscientious acknowledgment of support from past grants also enhances the chances of IU's research community securing funding from grants in the future. For the acknowledgment statement(s) required for scholarly printed works, web pages, talks, online publications, and other presentations that make use of this and/or other grant-funded systems at IU, see If I use IU's research cyberinfrastructure, what sources of funding do I need to acknowledge in my published work?

Support

Support for research computing systems and software at Indiana University is provided by various UITS Research Technologies units:

For general inquiries about UITS Research Technologies systems and services, complete and submit the Research Technologies request for help form.