CyVerse_logo

Home_Icon Learning Center Home

CyVerse Frequently Asked Questions

CyVerse_logo

Home_Icon Learning Center Home

Account Portal FAQ

Your Profile

How do I update my account information?

CyVerse users should update their account information annually for continued access to services. To update, go to the CyVerse User Portal, log in, and click the account icon in the upper right corner.

In addition to updating your email, institution, occupation, preferences and other information that may have changed, please add your ORCID ID, a unique identifier which can help you receive credit for your work. Get an ORCID here https://orcid.org/register.

By keeping your account information current, our funders can see the value of CyVerse to our community and we learn which of our services and platforms are most helpful to you.

CyVerse_logo

Home_Icon Learning Center Home

Data FAQ

How do I access the CyVerse Data Store?

    • Method
    • Access Point
    • Upload & Download
    • Install & Setup
    • Account Required
    • Max File Size
    • More Information
    • Discovery Enivronment
    • Web
    • Both
    • No
    • Yes
    • Upload: 2 GB/file; Import: Up to max allocation and provider’s limit
      • Getting Started with the DE
      • Uploading Data
      • Downloading Files and Folders
    • Cyberduck
    • Desktop
    • Both
    • Yes
    • Login or Public
    • Very large > 10 GB
    • Installing and confirguring Cyberduck - with a CyVerse account - without a CyVerse account
    • iCommands
    • Command line
    • Both
    • Yes
    • Login or Public
    • Very large > 10 GB
      • Parallel bulk transfers
      • Good for command-line automation
      • Use with Linus and Mac OS
      • Installing and confirguring iCommands on CyVerse - with a CyVerse account - without a CyVerse account (download only)
    • FUSE Mount
    • Command line
    • Both
    • Yes
    • Yes
    • 1 GB/file
      • very, very slow for file transfers
      • Use with Unix/Linux only
      • Behaves like a Unix file system
      • Useful for viewing and navigating directories and their contents
      • Instructions on using FUSE

Fix or improve this documentation

Home_Icon Learning Center Home

CyVerse_logo

Home_Icon Learning Center Home

Analyses FAQ

What apps and workflows are in CyVerse?

CyVerse has hundreds of apps and workflows in the Discovery Environment (DE). For the list of applications available in the DE, see the List of Applications. Most apps in the DE have user manuals to give you a jump-start on using the app. You also can browse the list of tutorials to find help to learn a complicated workflow or how to use an app in the DE or VICE.

How can I make my app available for other people to use?

You can create a new app interface in the Discovery Environment and share it with other users and you also can install the app on VICE. In most cases, providing a Docker container (or a link to one) with the application of your choice is all you need to start. See the Developer’s section of the CyVerse Learning Center.

Containers FAQ

Does CyVerse have resources for GPU and containers (e.g. to stabilize R modules using the nvidia cuda)?

Yes, please email Tyson at tswetnam@cyverse.org for details.

While container performance is greater than a VM, how much is container performance below that of native mode, in general?

Please see the Conclusion section bullet #2 in Evaluation of Docker Containers for Scientific Workloads in the Cloud.

Are there tools for scanning publicly available containers in Atmosphere VMs for malware?

Docker Hub and Quay, two of the most popular public container image registries, provide security scanning for images that are uploaded to their sites. Details about how to enable or use these registries’ security scanning features can be found here: https://developers.redhat.com/blog/2019/06/26/using-quay-io-to-find-vulnerabilities-in-your-container-images/ and https://docs.docker.com/docker-hub/vulnerability-scanning/.

Other tools to scan your container images without using Docker Hub and Quay include Anchore, Clair, and Trivy, with new container-based security scanning software being developed all the time. Each solution seems to take a different approach to security scanning, so you might need to experiment to find the tool that works for your workflow. The easier tools to use are Anchore, which can be used as a container itself, and Trivy, which can be installed by a package manager.

When using Singularity, there is built-in integration with Clair using Singularity’s tools. Information about Singularity tools can be found here: https://github.com/singularityhub/stools

Fix or improve this documentation

Home_Icon Learning Center Home

CyVerse_logo

Home_Icon Learning Center Home

Collaboration FAQ

How can I get a letter of collaboration for my grant proposal that uses CyVerse?

To request a letter of collaboration, email informaticians@cyverse.org with the following information: - the CyVerse resources your project will use (e.g., storage, computing power, expertise for scaling, etc.) and indicate if any resulting datasets will be made publicly available in CyVerse - the name of the PI, proposal title, funding agency, and date you need the letter - if there is a template that must be used, please attach to your email

What is an external collaborative partnership and how do I apply?

External Collaborative Partnerships (ECP) pair member(s) of the CyVerse user community with expert CyVerse staff to address the computational needs of a scientific project. Requests are reviewed on an ongoing basis.

The criteria CyVerse uses to review ECP requests are available here: ECP criteria.

To help you complete the ECP application, the questions on the form are listed below.

External Collaborative Partnership Application Questions

See the full list of questions below:

Expand to read more

  • Project Principal Investigator (PI)
  • Institution
  • Collaborating personnel
    • Provide a detailed list of students, technicians, informaticians and/or developers who will be able to assist with project design and implementation, their respective computational science skill sets (e.g., web design, Python, GWAS, etc.), and their specific time commitments during the project (e.g., 1.5 hrs/day).
  • Previous interactions with CyVerse
  • Funding sources
  • Project title
  • Project description
  • Please summarize your proposal’s activities and desired outcomes (500 chars or less)
  • Scientific description
    • Provide a scientific description of your project. Describe how the proposed project is within the scope of CyVerse’s scientific Enablement Vision. Illustrate how any resulting deliverables have the potential to enable science for scientists beyond your immediate network of collaborators.
  • Technical description
    • Provide a technical description of your project. What is the computational need that Will be addressed with assistance from CyVerse? Identify specific potential deliverables to be implemented using CyVerse technologies, such as the Discovery Environment, Atmosphere, APIs, Data Store, Data Commons, etc.
  • Timeline and milestones for completing the project
    • Provide a timeline of specific monthly milestones (deliverables). Projects of short duration (~2 months) should provide weekly milestones.
  • Scientific and technical impact
    • Describe how the success of this collaboration will benefit your project and the broader community.
  • Communication and sharing plan
    • Will the data and/or workflows be made publicly available through CyVerse?
    • Will you be blogging and/or tweeting about the work?
    • Will you be giving a talk about your work at your institution or at a professional conference?
    • Will you be writing a news article for the CyVerse website or newsletter?
    • Will you be preparing a tutorial that uses the datasets or workflows?
    • Will you be leading a workshop(s) or webinar(s) to teach others to use the data or workflow(s)?

Fix or improve this documentation

Home_Icon Learning Center Home

CyVerse_logo

Home_Icon Learning Center Home

Training FAQ

How can I use CyVerse to teach my students?

You can use CyVerse’s Visual Interactive Computing Environment (VICE) to provide students with a controlled environment in which they can install tools, run analyses, or visual results in a consistent environment.

# Have your students sign up for CyVerse accounts. # In the CyVerse User Portal, navigate to the Requests section and complete the “Atmosphere: Reserving Cloud for Education”. If you do not have the CyVerse account information for all of your students when you complete the form, provide your user name and email to answer those questions. # You will be notified when your class is set up. Note that we cannot provide access to your class ahead of time until we receive the student usernames and emails.

I previously used Atmosphere to run instances for analysis and plotting. Where can I do this work now?

CyVerse’s Visual Interactive Computing Environment (VICE) is the place to run analyses that you previously ran in Atmosphere.

How can I use CyVerse in my course?

DNA Subway (See DNA Subway Guide) is especially useful for teaching gene concepts, phylogenetics, DNA barcoding and RNA-Seq. With a friendly user interface and based on the analogy of multiple subway stops and lines, student learn the basics of computational genomics workflows.

Using containers in DE/VICE is also a great teaching resource. By loading a container with the software tools, datasets, and analysis parameters necessary to run an analysis, containers help overcome many technological hurdles for both learning and teaching informatics.

Can CyVerse give a workshop at my institution?

Funding to support workshop requests is very limited, with priority for trainings at underserved institutions (rural, HBCU/Tribal/Hispanic-serving, etc.). Contact Education, Outreach, and Training Lead Jason Williams (williams@cshl.edu)

Fix or improve this documentation

Home_Icon Learning Center Home

CyVerse_logo

Home_Icon Learning Center Home

Bring Your Own (BYO) FAQ

How can I use CyVerse’s tools and resources from within my program/app?

See the information about our Science APIs; you can also contact Support using the blue chat icon Intercom at the bottom right of the platform. For projects requiring more extensive support, you can request community support or an external collaborative partnership (see Collaboration FAQ below).

Fix or improve this documentation

Home_Icon Learning Center Home

CyVerse_logo

Home_Icon Learning Center Home

Discovery Environment FAQ

Discovery Environment 2.0 Release

What is changing in the Discovery Environment?

The Discovery Environment 2.0 (DE 2.0) is the same great workbench for data, apps and analyses, but with a new look (interface) so that using CyVerse is even better! The new DE is faster and easier to navigate, has new and upgraded features, and provides a semi-customizable dashboard. You can use the DE 2.0 right now for:

  • Data: upload, create new files and folders, share with collaborators
  • Analyses: run and share analyses and workflows
  • Notifications: get status notifications of your uploads, jobs, results, etc.

Important Note: Some existing DE functions won’t be fully operational in the DE 2.0 until Summer 2021; please see the question “Can I still use the old DE?” for specific operations/tasks for which you must continue to use the old DE.

What will happen to my data, analyses, results, collaborators, public datasets, etc.?

No changes have been made that affect your data, analyses, results, etc. All your existing data in the Data Store and Data Commons and your apps and analyses can be accessed through the DE 2.0. Any changes you make going forward using either the DE 2.0 or the legacy DE will be reflected in both versions.

When will the DE 2.0 be available?

The DE 2.0 is available now! Poke around to see what’s new and take a tour: https://de.cyverse.org/help

Can I still use the old DE?

  • Yes, the old DE is still available to use until we turn it off on July 6, 2021.
  • Please use the legacy DE for some functions that are not yet fully operational in DE 2.0, such as:
    • Communities: adminstrating, managing, and browsing
    • Teams: request to join a team and notifications related to your request
    • Requesting access to Visual Interactive Computing Environment (VICE) apps
  • To access the old DE: https://legacy-de.cyverse.org

What do I have to do to access the DE 2.0?

For existing users, use your account credentials to: - Launch the DE 2.0 from its web address: https://de.cyverse.org - Launch the DE 2.0 from our website homepage (top navigation Launch menu)

For new users: - Sign up for an account at CyVerse; you get access to the DE and 2 GB of storage with your account - Next, login to CyVerse to validate your new account and start using the DE

Where can I find more information about the new DE?

  • Take the product tour in the DE 2.0
  • View a short DE 2.0 Webinar demonstration
  • Read the Learning Materials documentation (add public link)

What if I have problems finding my data, analyses, folders, etc.?

  • Take the DE 2.0 product tour which explains each of the icons on the DE 2.0 dashboard
  • Use the help icon in the upper right corner
  • Email us at support@cyverse.org

What will happen to my DE bookmark?

Your existing bookmark to the old DE has not changed, but will update to the DE 2.0 when the old DE is sunsetted July 6, 2021. Please bookmark the DE 2.0 web address: https://de.cyverse.org

Analyses Questions

Why has my job failed or been running forever?

The following recommendations can help you determine what went wrong and collect information for CyVerse staff in case you cannot resolve the problem yourself.

Expand to read more

Common things to check when troubleshooting an analysis

  • View the app’s parameters to make sure you used the correct input files and settings
  • Read through the app’s documentation page
  • It also may be helpful to read through documentation about the tool that was used to create the app. Check the app’s documentation page to see if a link was provided. If no link was provided, you can find specifics about the tool that was used and search for more information on the web.
  • Avoid the use of special characters and spaces in analysis names, file names, and folder names when submitting an analysis through the DE (e.g. ~ ` ! @ # $ % ^ & * ( ) + = { } [ ] | \ : ; ” ‘ < > , ? / and spaces)

Getting Help with an analysis

If you know that an analysis typically completes in 20 minutes but you have one that still shows Running status 24 hours after you submitted it, the app used for the analysis seems to have a problem, or you didn’t get any output files or the output files were not what you expected, you can submit a request for help directly in the Analyses window. The status of the analysis determines the Help information that is displayed.

  • In the Analyses window, find the analysis with the possible issue
  • Click the name of the failed analysis whose outputs you want to view
  • Review the suggestions for review
  • If you still need assistance, click I still need help and complete the form
  • Please go through all the troubleshooting steps yourself before requesting help. The problem is often something that you can diagnose yourself

Checking Log files for error

One of the main tools available for troubleshooting a failed analysis is the set of log files that are returned with each completed or failed analysis. These log files contain important information about the analysis, such as the settings that were used, files you used, and, in the case of a failed analysis, information to help explain why the analysis failed.

Because different apps are based on different tools, there is no standard method used for error reporting, so the same type of error may land in different log files. For example, one app may return errors to the stdout files (usually the screen, although it can be redirected and is generally captured in a log file here), while another saves its errors to the stderr files (which usually writes to a file, but can also be redirected). This means you may well have to look in more than one log file when troubleshooting a failed analysis.

The log files that most commonly contain error information are (numerals in the filename correspond to the step number that was logged in your analysis ):

  • condor-stderr and condor-input-stdout log files contain errors and details about Condor, the batch manager program that handles the execution of your analyses submission in the analyses queue.
  • condor-input-stderr and condor-input-stdout files contain details about outputs from the tool upon which the app is based.

How do I get help with a tool (app) or workflow?

The steps to get help depend on whether you’re a novice or an expert with the tool (executable or binary) on which the app or workflow is based.

If you are a novice:

  1. Learn more about the tools used:
  • Search the internet for the publication describing the tool and any related documentation.
  • Make sure you understand what the tool is designed to do, what inputs it can accept and in which format, and how to set any parameters.
  1. Search the internet for informative sites in your domain. For example, [SEQanswers](http://seqanswers.com) is the go-to online forum for the next-generation sequencing community.
  2. Talk with someone at your institution who is more experienced with the tool.
  3. Try to use the app in the Discovery Environment. Click ![infoicon](docs/images/InfoIcon.jpg) next to the app name to view the app manual and its sample test input files and expected outputs.

If you are experienced with the tool or workflow:

If you are experienced with the tool or workflow and need advice for how to work with very large-scale data or a complex workflow, you may request community support or [Extended Collaborative Support](http://www.cyverse.org/collaborate).

Why is my analysis sitting in the Submitted state for so long?

Analyses that use an app that runs on an HPC system can remain in the Submitted state for hours or even days. They may sit in in the queue in Submitted state waiting to run, along with other possibly long-running jobs that were in the queue first. Therefore, it may take several days for your analysis to get its turn to run. Once your analysis runs, its results will be returned to the Data Store and you will get a Notification that the analysis status has changed to Completed.

Is there a limit to how many analyses (jobs) I can run at the same time in the Discovery Environment?

Only 8 of your analyses will run at the same time. You can launch more, but they will not run until some of your analyses have completed.

I’m trying to run an analysis, but when I enter an input I can’t see my files. Why? I know they are there.

Because the app requires a folder as input, not a file, the files don’t show since they aren’t the appropriate inputs for the app. The files are indeed there but don’t show. Check the app’s input box; if it says “Select a folder”, then it requires a folder input. Put the file(s) you want to input into a folder and then use that folder as input. Note: You can use Drag and Drop to input the folder by finding it in the Data window, making sure the folder name is shown in the center panel, and then dragging the folder into the app’s input box.

How do I rerun a job I previously ran in the DE, but I want to change some parameters or use a different input? ================================================================================================++================

You can easily relaunch the same analysis with different settings:

#. In the Analyses window, click the app name in the App column for the analysis to rerun. This opens up an app window for that app, which is already configured with the inputs and settings you used for the previous analysis run. #. Change settings or inputs as needed. #. Click Launch Analysis to launch the new analysis.

`Learn more here.<https://wiki.cyverse.org/wiki/pages/viewpage.action?pageId=11446455#Analyses%20Menu:%20View%20Outputs,%20Parameters,%20and%20Info;%20Relaunch,%20Cancel,%20and%20Delete-Relaunch>`_

How do I run the same analysis on a number of files most efficiently?

You can create a file that contains a list of up to 16 files to use as input for high-throughput and batch file execution. Such a file is called an HT Analysis Path List file. Learn more here, and if you still have questions, read here.

I have a series of files that I want to analyze with the same app, but the output files all have the same name. How do I distinguish them so I can use them in a workflow?

You can avoid confusion by finding the output folder in your Data list and renaming each output file with a unique name. After renaming the output files, you can then use them together in a step of the analysis workflow.

Data Questions

How do I share data files and folders in the Discovery Environment?

To share a file or folder with a CyVerse user:

  1. Click the checkbox for the file or folder to share.
  2. Click ![shareicon](docs/images/ShareViaDEicon.jpg) in the row for the item.
  3. Begin entering the name in the search field and then select the user.

To create a public link to a file (not for folders) with someone who doesn’t have an account:

  1. Click ![linkicon](docs/images/shareviapubliclinkicon.jpg) in the row for the file.
  2. Copy the link, and click Done.
  3. Send the link.

Learn more at Sharing Files and Folders.

When sharing a file or folder, what permission should I give to my collaborator?

It depends on what you want to allow the collaborator to do with the file or folder. Options are: read, write (ability to edit the file or folder), and own (in addition to edit, can also delete and move; use this permission with caution). Learn more here.

Why can’t I rename or delete files in a folder that has been shared with me?

To rename a file or folder, you must have “write” permission, and to delete a file you must have “own” permission. To check the permission you have, click the checkbox for the item and look at the Permissions shown in the Details panel on the right. Contact the person who shared the file or folder with you if they did not give you the necessary level of permission. Learn more at Changing and Viewing Data Permission Levels in the DE.

How can I manage shared files and folders for my lab group or project?

See Setting Up a Shared Directory for a Lab or Project.

Why doesn’t anything happen when I move a folder I own to the trash?

If the folder has hundreds of files, it can take several hours for the deletion to complete in the DE. Please be patient and try refreshing your browser periodically. You cannot delete 1000 files or more in the DE. You must use iCommands instead. Note that deleted files may still show up in the search for awhile, but eventually will be fully purged from the system.

Can I have spaces in file and folder names?

No. Do not use spaces or special characters in file or folder names as they can cause analyses to fail. Learn more here.

Can I view my files in a genome browser?

You can view bam, vcf, and gff genome files you own in the genome browsers at Ensembl, UCSC, IGV, GBrowse, and jbrowse, and view Fasta genome files in CoGe. Learn more about viewing genome files in a genome browser or in CoGe.

App Questions

How do I share an app in the Discovery Environment?

You can share an app or workflow with the general public or just with specific users by following the steps described at Sharing Your App or Workflow and Editing the User Manual.

Why can’t I find a Discovery Environment app?

There are two common reasons why an app is not “visible” or doesn’t come up in search in the Apps list:

  • The app may be an HPC (high-performance computing) app, which is only displayed after you have logged in to Tapis, where the HPC apps are stored. To do so, click the HPC tab in the Apps window and enter your CyVerse username and password.
  • The app may not yet be public, or the app owner may not have shared the unpublished app with you. Check with the owner to see if it is indeed shared with you or is public.

If you still can’t find the app, it’s possible it has been deprecated. If an app is no longer returned in a search query, search for an app with a similar name or one that uses the same tool, topic, or operation. If you are the app integrator and need the app returned to the catalog, contact Support (support@cyverse.org) for assistance. Learn more about deprecated apps.

Fix or improve this documentation

Home_Icon Learning Center Home

CyVerse_logo

Home_Icon Learning Center Home

Data Store FAQ

Managing data

What if I need more space (storage) in the Data Store

Typically, every user has a 100GB allocation in the Data Store. You can request more space by completing an Allocation Increase Form.

Sharing data

How do I publish a large set of public data?

If you need assistance transferring a large dataset to CyVerse, please contact CyVerse Support (support@cyverse.org or use the blue Intercom icon Intercom at the bottom right). For more information on our policies, see CyVerse’s Collaboration Policy and Data Management Policy. For more information on using data at CyVerse, see the Data Store Guide.

What public datasets are in CyVerse?

CyVerse provides web access to its public datasets. Because the CyVerse Data Store underlies all CyVerse services, the public datasets may also be accessed through the Discovery Environment, Atmosphere, the Science APIs and iCommands. For more information on using data at CyVerse, see the Data Store Guide.

How do I request a Community Released Data Folder?

Community Released Data folders are available for evolving datasets that individuals or communities want to make available as quickly as possible for research and reuse, especially within CyVerse analysis platforms. Community Released Data folders are intended for datasets that are growing or changing frequently or that may not need long-term preservation.

Before you request a folder, please read this wiki article Publishing Data through the Data Commons, and this one Preparing Community Released Data Folders.

Then, if you meet the criteria, you can request a folder using this Request for Community Released Data Folders Form.

How do I connect to a shared or public folder with CyberDuck?

If you follow the default configuration for Cyberduck you will automatically be connected to your home directory (/iplant/home/$your_user_name). However, you can also open a new connection to any directory to which you have access permission. This includes folders that other users have shared with you, or public folder, such as those under /iplant/home/shared.

To do this read the instructions below:

Expand to read more

Open a connection to a public folder

  1. Open CyberDuck
  2. If the browser is not already open, select File - New Browser
  3. Create a new connection by clicking on the + in the lower right
    (next to the pencil and minus sign)
  4. In the top dropdown menu, select iPlant Data Store
  5. In the dialog box, name your connection something relevant, like the name
    of the folder you want to browse to
  6. Enter your user name in the appropriate field. If you are connecting to
    public folder, you can also enter anonymous in this field
  7. In the Path field, enter /iplant/home/shared, or some subdirectory.
  8. Close the dialog window. Now, in your list of connections, you should see
    a new connection with the name you chose. Click on that, and you should go
    directly to the public folder.

Once you are connected to /iplant/home/shared, you can browse down to any sub-directory.

Warning

DO NOT try to browse up to /iplant/home/ because it contains >50,000 folders
(one for each CyVerse user) and it will hang up.

Open a connection to a private folder that is shared with you

Follow the same steps as above, except in step 7, enter the path to the folder that was shared with you. For example, if user “janedoe” shares a folder called “data” with you, enter /iplant/home/janedoe/data in the Path field. You cannot connect as anonymous in this case, and must supply your own user name. See also: Cyberduck documentation.

How do I make a folder public with iCommands?

Although you can share files and folders in the DE and create public links, you must use iCommands make them visible to everyone. Permissions are set in iCommands by using ichmod (https://docs.irods.org/4.2.1/icommands/user/#ichmod).

To make a folder public, you give read permission to two users: ‘public’ (anyone signed in with a CyVerse account) and ‘anonymous’ (anyone on the web - no sign in required).

To recursively make a shared folder called ‘myfolder’ public, use the instructions below

Expand to read more

ichmod -r read public /iplant/home/shared/myfolder
ichmod -r read anonymous /iplant/home/shared/myfolder

To remove public access to the folder, use:

ichmod -r null public /iplant/home/shared/myfolder
ichmod -r null anonymous /iplant/home/shared/myfolder

See also our iCommands documentation.

Fix or improve this documentation

Home_Icon Learning Center Home

CyVerse_logo

Home_Icon Learning Home

Atmosphere FAQ

Refocusing CyVerse Atmosphere

CyVerse has refocused Atmosphere to better serve the growing need for container and cloud native development services. Read on for what this change means to you.

What has changed?

CyVerse has changed Atmosphere from a general purpose cloud computing environment to one that supports cloud-native development projects. In a cloud-native environment, you can more fully take advantage of the distributed, scalable, flexible nature of the public cloud, which means spending less time worrying about networks, servers, operating systems, etc. and more time on your research.

Why did Atmosphere change?

Because containers have become a powerful and preferred way for researchers to create, execute, and share analyses, largely replacing the capabilities of Atmosphere’s virtual machines, we are now directing resources towards the growing need for cloud-native services. In addition, the NSF recently extended CyVerse’s long-term partnership with Jetstream through 2025 to provide similar capabilities and interfaces with significantly larger CPU, GPU, and storage infrastructure, presenting an exciting option for our U.S. users.

What do I need to do?

If you are a US-based researcher using Atmosphere to develop VICE or other DE apps, you may continue to use Atmosphere for this purpose. Otherwise, users were asked to transfer their resources off Atmosphere volumes by the following dates:

Dates U.S. Users International Users
July 1, 2020 NA You will no longer be able to create an account.
August 1, 2020 NA You will no longer be able to log in. All resources should be moved off Atmosphere by this date.
September 1, 2020 You will no longer be able to log in. All resources should be moved off Atmosphere by this date. NA

What if I did not move my resources off Atmosphere by September 1, 2020?

After January 1, 2021, CyVerse will transfer any unclaimed Atmosphere volumes to the Data Store, when possible. We will archive any remaining unclaimed images with data and delete associated volumes.

How do I get help moving my resources off Atmosphere?

CyVerse staff can help with planning your move, migrating your data and tools from Atmosphere to CyVerse’s other services, and advising you on alternative offerings for moving your data and tools; please contact us using the chat service in Atmosphere or email support@cyverse.org.

What can I use instead of Atmosphere?

The CyVerse Discovery Environment is a simple web interface for managing data, running analyses, and visualizing results. See Getting Started with the Discovery Environment.

CyVerse staff will provide assistance for U.S.-based researchers to migrate their cloud images to Jetstream, which uses CyVerse Atmosphere as its primary interface. The CyVerse Atmosphere image must be owned by the user and must meet Jetstream’s requirements for importing images. CyVerse cannot guarantee 100% success when exporting a virtual disk image from CyVerse Atmosphere to Jetstream Atmosphere. Jetstream staff will not provide support for imported images, and instead recommends images be recreated in their cloud. For more information about Jetstream, see Getting Started with Jetstream.

I previously used Atmosphere to run instances for analysis and plotting. Where can I do this work now?

CyVerse’s Visual Interactive Computing Environment (VICE) is the place to run analyses that you previously ran in Atmosphere.

How do I use VICE?

The best place to start is by reviewing our guide. There are specific sections on using JupyterLab, RStudio, and Shiny. You can even share VICE apps with collaborators.

What has happened to my community-created image?

Existing community-created images with an associated DOI archive still visible on the CyVerse Atmosphere image catalog after September 1, 2020, but may only be launched by special permission. Other community-created images may be archived and eventually will be removed from the image catalog.

Can I get a copy of my image?

Yes. To obtain a copy of the virtual disk of any image you have created/authored,please contact support@cyverse.org by January 1, 2021. Although CyVerse cannot guarantee 100% success when importing a virtual disk image from Atmosphere, virtual disk images should be compatible with other OpenStack clouds.

What if I still want to use Atmosphere as a computing environment?

As stated above, Atmosphere has been refocused to support U.S.-based researchers’ cloud-native development projects.

Can I continue to use my Atmosphere images for a class or workshop?

You can use NSF’s Jetstream, which provides the same functionality as Atmosphere for launching images for classes and workshops. Contact support@cyverse.org for assistance with this process if needed. Jetstream also will meet your needs if you want to use something more high-powered, such as Jupyter Hub, for your teaching sessions.

What can I use Atmosphere for after September 1, 2020?

Atmosphere will remain available as a support service for cloud native development projects, such as U.S. users who are developing containers. Contact support@cyverse.org with a request.

Can I still use Atmosphere for single instances, such as launching a Jupyter Notebook?

CyVerse’s Visual Interactive Computing Environment (VICE) within the Discovery Environment will meet your needs for doing reproducible analyses using single instances. There is specific information on using JupyterLab.

Who can I contact if I have more questions?

Contact us using the chat service within Atmosphere or email support@cyverse.org.

Using Atmosphere

Where should I put save data on my Atmosphere Instance?

When doing work in Atmosphere, you will likely bring data into instances from the CyVerse Data Store, your local system, or another data source. However, within the instance, where should you put your data?

There are a several instance storage options within an instance, each with different considerations:

  • Instance volume (usually mounted as /vol_c )
  • Instance scratch directory, /scratch
  • FUSE-mount your Data Store directory
  • Home directory /home

Each solution will be briefly described below.

Instance Volume

Larger storage, good performance, durable, not accessible outside of Atmosphere

Expand to read more

An instance volume, often referred to as simply “volume”, is a durable, long-term storage option that can only be used within Atmosphere instances. This is a reasonable storage solution for larger data sets that will be used within Atmosphere for many weeks or months. We recommend backing up your data to your Data Store home directory.

Note

Note: A volume can be attached to one instance at a time, much like a virtual USB drive.

Instance Scratch Directory /scratch

Storage size according to instance size, high performance, ephemeral, not accessible outside of Atmosphere

Expand to read more

Every instance will have a ephemeral disk that is stored under /scratch, and its storage size is defined by the size of the instance. The ephemeral disk only exists while the instance is active, and any data stored within this directory may be deleted when the instances is inactive (i.e. suspended, stopped, or shelved). This is a reasonable place for intermediate files or interim “work” before moving it to a long term storage (1. CyVerse Data Store, 2. an attached volume).

FUSE-mount your Data Store directory

Large storage, low performance, extremely durable, accessible with CyVerse’s ecosystem

Expand to read more

You can mount your Data Store home directory within your instance through FUSE. This method of access is convenient within certain limits. Performance may degrade if there are too many concurrent file access (e.g. several dozen concurrent files accessed simultaneously) or the files are too large (e.g. 100 Gb+).

Home directory /home

Small storage, high performance, ephemeral, and not accessible outside of Atmosphere

Expand to read more

The /home directory, the directory which you see when you login to an instance, will have limited disk space, whatever is configured for the image. This directory should also be considered temporary, if the instance is deleted or imaged. Unlike the scratch directory, the home directory will persist if the instance becomes inactive.

Viewing attached volumes and scratch

A command that will help see what is mounted & available is df. The example below uses the -h option for storage reporting in more human readable form, see details Below

Expand to read more

$ df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        20G  2.6G   17G  14% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            7.9G   12K  7.9G   1% /dev
tmpfs           1.6G  412K  1.6G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            7.9G     0  7.9G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/vdb        158G   60M  150G   1% /scratch
/dev/vdc         64G   52M   61G   1% /vol_c

This instance has a 65 GB attached volume mounted at /vol_c and over 150GB available under /scratch.

One approach to deciding might be:

“Using /scratch” would fit if my analysis likely to “start” with the large dataset (30 fastq of 150 GB). Then, various tools are run that produce new data. Once those are complete, only a smaller set of outputs will be retained. In this case, we use /scratch for computing intermediate files and only keep outputs.

“Attaching a volume” would fit if my analysis likely to begin with a common input. Then, an analysis (or many) are performed. Once complete, the results are the input to “periodic” analysis or refined over time.

In the end, any of the storage options mentioned in this article can be used for performing analysis. It is a matter of evaluating what data needs to be retained or the performance requirements. The CyVerse Data Store, or a Volume, are the most durable options for keeping data.

How do I manage disk space on my Atmosphere Instance?

A recent change in Atmosphere “relocated” ephemeral storage mounted within the instance’s filesystem.

Previously, Atmosphere would mount the ephemeral storage under the home directory, /home. This allowed you to automatically have a larger temporary, working space when you established a remote connection via SSH or VNC. This temporary space is relative to the “instance size” you selected on launch.

Due to the nature of ephemeral storage, namely that this storage may be erased when the instance is not active, Atmosphere now mounts this storage onto /scratch. You are welcome to use /scratch if you need additional storage for active analysis or work.

See details below:

Expand to read more

Example:

        $ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             16G     0   16G   0% /dev
tmpfs           3.2G  8.7M  3.2G   1% /run
/dev/vda1        20G  4.9G   15G  26% /
tmpfs            16G     0   16G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/vdb        237G   15G  210G   7% /scratch
tmpfs           3.2G     0  3.2G   0% /run/user/0

Warning

It is important to note is that data under /scratch will not persist when shelving an instance. Also, instances which are in a suspended or stopped state may ultimately be shelved if the instance is inactive for too long. This consideration should be weighed when choosing where to put data. If a higher durability guarantee is required (or desired) within the instance, then store data in a volume. You can also backup and restore your data to the CyVerse Data Store.

Fix or improve this documentation

Home_Icon Learning Center Home

Getting help using CyVerse

The CyVerse team is small, but dedicated to helping you get what you need done. There are a number of ways to get help using CyVerse:

  • Search the CyVerse Learning Center.
  • Review videos by domain experts
  • Use in-app chat by clicking on Intercom. If you catch us outside of working hours (weekdays 9-5PM US Mountain Time Zone), we will follow up via email.
  • E-mail support@cyverse.org.

Don’t see your Question here?

Send us a note using in-app chat by clicking on Intercom. If you are familiar with GitHub, you can submit an issue, or send us a pull request. See Improving Existing Documentation in the CyVerse Learning Center.

Fix or improve this documentation

Home_Icon Learning Center Home