EMC Atmos - Fundamentals

1. What is Atmos

Traditional SAN and NAS storage solutions are much too expensive and cannot handle the demands and requirements of this new category of data. Features such as global distribution, policy-based data management, massive scale, and low management cost are ‘must have’ requirements. EMC Atmos cloud storage delivers these features and provides support for multi-tenant secure virtualized resource isolation and allocation. Atmos does this by leveraging off-the-shelf hardware, and combines it with a rich set of software features to provide an innovative and cost effective solution to store, secure, distribute, and manage traditional and new categories of data. Atmos is a new type of data storage and management framework that addresses a current void in the market. Atmos:

• Is based on a unique set of data services with no limits on namespace or location.

• Can be accessed using web or file-based services and provides automated protection and efficiency services to manage data.

• Can be implemented on an EMC-developed, exabyte-scale object store using physical or virtual appliances which address the challenges of storing and managing vast amounts of unstructured content for custom or packaged applications.

A multi-location Atmos system can be managed as a single autonomous resource and can be delivered as a self-service experience to consumers

The Atmos solution is comprised of commodity-based servers connected to standard 15 slot disk enclosures. Today the disk enclosures are being populated with one or multi terabyte SATA drives and up to four DAEs can be attached to each front-end node.
The number of nodes provisioned in an Atmos rack can vary depending on budget and performance requirements. Each rack includes a 24-port gigabit Ethernet switch to provide inter-node communication. Generation 3 hardware uses the same high level architecture but uses blade servers and high density DAE’s to improve capacity within the Atmos footprint.

The terminology used to describe a group of nodes installed in an Atmos rack connected by a gigabit Ethernet switch is an installation segment (IS). Very often large implementations have multiple ISs installed. They are defined as an RMG (Resource Management Group). The RMG most often refers to a site or location so it is possible to have a single IS as an RMG. This would be the case for a particular site with a single rack Atmos system.

The new Generation 3 Atmos hardware reflects a move to higher density storage capacity with lower power consumption within the same physical footprint. The new Gen 3 dense 480 is built around the Intel Phoenix blade server which provides four servers in a 2U chassis which essentially doubles the density of the previous server configuration.
The other major change is the 60-drive Voyager high density DAE which increases capacity within the Atmos footprint and simplifies the dual-stack 15-drive DAE configuration used in previous releases. It is expected that the straight forward cable configuration will improve reliability and decrease repair and upgrade time while moving toward a more standard hardware configuration.
Hardware configurations support fully populated 60-drive DAE’s or partially populated DAE’s of 15 or 30 drives with the ability to increase drives and expand capacity as needed.

Atmos Generation 3 uses the Intel Phoenix blade server. This model offers performance upgrades over Generation 2 servers within a smaller form factor. Internal disks can be accessed from the front while each server blade can be inserted or removed from the back. All video, LAN, and SAS connectivity is provided directly on the back of the equipment from each server card. Each chassis is equipped with redundant power supplies.

Allied Telesis has been the preferred switch vendor for many EMC products including Atmos. Since Atmos was introduced, Allied Telesis switches were used to provide internal inter-nodal connectivity and external connectivity to customer and access networks.
With the introduction of Generation 3 hardware, an optional 24-port switch from Arista Networks was made available for compatibility with 10 GbE customer networks. In later Generation 3 deployments, the Arista model 7048 switch will be the standard device used to provide connectivity between nodes displacing the popular Allied Telesis model

The 4U, 60-drive Voyager DAE includes up to 60, 3.5-inch disk drives. It supports 6 Gb/s data transfer speeds and provides redundant fan modules, two Link Control Cards (LCCs), two Inter Connect Modules (ICMs), and two power supplies. All of these functional modules are monitored by the Atmos system and all status, failures, and malfunctions are reported through the Atmos GUI. Recent updates allow for deployments of partially populated DAEs to accommodate different capacity configurations and the ability to expand as demand increases.

2. Atmos Features and Capabilities

The secret of creating an efficient content cloud is the underlying software. Atmos can be implemented on an EMC specific hardware platform or on industry standard VM servers. In either case, the Atmos solution provides massively scalable infrastructure supporting multiple petabytes of storage and billions of objects across multiple sites that can be globally distributed.
Storing, protecting, and distributing information are core functions of the Atmos system. The ability to efficiently manage the resources that support the environment is also important. It is conceivable that a traditional NAS solution could be implemented to store mass amounts of distributed unstructured data. What would the expense of such an implementation be? The cost of managing a NAS increases with the scale of the solution—quickly making it cost prohibitive.
Because Atmos is object based, stored data is abstracted from the physical storage which means traditional tasks of sizing, allocating, and managing the storage resource is greatly simplified. This is only half of the management story. In typical multi-user distributed environments, there is no provision for classifying stored information which means all data is treated equally. This scheme severely limits the solution provider from offering differentiated services.
Atmos uses a sophisticated policy management system that applies intelligence to every object stored. Policy definitions can easily be created to tell the system where to put the content and which actions to take. Additionally, object based metadata allows you to trigger policy for virtually any scenario including system, application, and user defined metadata.
Policies also support a comprehensive set of data services that allow you to perform operations such as replication, versioning, and retention. This means that distribution, protection, availability, performance, and disk efficiency can be considered for all objects and data types.

The philosophy and design of Atmos cloud storage is to provide the most comprehensive and innovative feature set in a single package without the need for additional licensing. The specific features and operational characteristics of Atmos are designed to meet the requirements of the newest demands and classifications of data produced today. Functionality is classified into six different groups:

• Scalability

• Policy-based Control

• Operational Efficiency

• Accessibility

• Security

• Self-management
  
  
  
  

Use Cases:

• Web Facing Applications – Atmos is the perfect global content store for digital images or rich content to Web 2.0 applications. It is low cost and provides distribution capabilities to locate data close to the consumer.

• Unstructured Content Storage (Cloud Drive) – Atmos provides a wide array of access options and has direct plug-in capability to many new sync and share as well as cloud drive applications providing ready access to data anytime, anywhere.

• Backup Target – The low cost and policy features of Atmos make it a strong candidate for backup-to-disk applications.

• Archive/Retention – Atmos provides a rich set of features and operational characteristics that make it a strong candidate for archive data. Low acquisition cost, low touch monitoring and maintenance, full replica, GeoParity, green drive, and auto delete features allow administrators to transition data to the precise amount of resource over the lifetime of the content.

• Content Distribution – You can leverage Atmos policy and your own data center locations to distribute content based on your needs, service levels, and consumption requirements.

• Service Provider (IaaS) – Atmos is the ideal infrastructure for storage as a service. Implemented on the Atmos Cloud Delivery Platform (ACDP), service providers can set up self service and resource monitoring to support auto provisioning and chargeback applications.

Flexible Storage Access

• Atmos provides flexible access to storage via its client service. The service runs on all Atmos nodes and has interfaces to accommodate access methods that fall into two basic categories: web-based access and file-based access.

• An object can be accessed via either web-based or file-based methods. Regardless of the access method, all objects are defined using a global, unique identifier – the object ID. Applications using the web service interface can interact with objects via this identifier directly. File-based access to objects uses traditional namespace paths and filenames with Atmos performing the translation to the object ID transparently.

• Cross platform web clients connect to Atmos web services over HTTP. The web service interface allows direct access to the object API over the Representational State Transfer (REST) protocol. The Atmos web services interface allows customers to quickly integrate Atmos with existing applications that, like Atmos, have a service-oriented architecture. Web service access must be explicitly enabled on each Atmos server where desired.

• For file-based access, the Common Internet File System (CIFS) and the Network File System (NFS) services are supported. However CIFS and NFS cannot coexist on any single Atmos server. The CIFS and NFS access methods need to be kept separate on a per server basis. The CIFS service must be explicitly enabled on each Atmos server where desired.

• Installable File System (IFS) access is provided through the network for clients running Red Hat Enterprise Linux release 5 (RHEL5). The client must load a file system userspace ‘Fuse’ package and install the EMC provided IFS bundle. The client will then have access to the Atmos clustered file system. By default, IFS is enabled on all Atmos servers.

Atmos Core Service

• Client Service (CS): Interface for clients to access storage. Communicates with SS, MDS, RMS, and MDLS services. 
• Metadata Location Service (MDLS): A distributed index that maps object IDs to the MDSs responsible for those objects. The purpose of MDLS is to determine an MDS that will handle a data request. 
• Metadata Service (MDS): Where metadata (as opposed to user data) is stored and managed. The MDS manages access to object metadata and the file system namespace. Policy Manager (PM): (part of the MDS) It stores policies and selects appropriate policies for objects and reconciles abstract policies into concrete layout descriptions. Resource Manager Service (RMS): Tracks the location and monitors the status and properties of service instances in the system in a distributed manner. It probes the network to find out what machines are on the network, what services are running on those machines, and what properties those services have. Properties vary according to the service. SS properties include storage available, unused storage, location, and features. 
• Storage Service (SS): Manages a set of disks on which user data is stored. The core service distribution is as follows: MDLS: Is located on the first two nodes installed (within the first install segment). 
• The first node installed in all other subsequent RMGs also runs MDLS. All other core services (CS, MDS,PM, RMS, and SS): On all nodes (install segments in all RMGs).

How Atmos work

• The application sends the file over an IP-based network to an Atmos which is mounted or mapped as any drive resource. Anytime you send a file to Atmos, it generates a unique identifier for this “object” using a hashing algorithm and other environment variables. The algorithm is calculated against the contents of the file to derive a value unique to that file.

• The result of this calculation is stored in Atmos as part of the object metadata and is used anytime you want to read or access the file. At some point, an acknowledgement will be sent back to the application signaling that the data is secured on Atmos.

• Note that the configuration of the Atmos policy plays a major role as to when the acknowledgment gets returned. If the policy dictates asynchronous replication, the acknowledgment will be sent after the first replica or copy of the data is written. If synchronous replicas are specified in the policy, all copies must be written before the acknowledgment is returned.

• During the write process, file metadata is processed by the policy system to determine how and where the data objects will be stored in the Atmos system. An object ID in Atmos looks like a random alphanumeric string but is the mechanism used by the system to track and locate stored data. If a user wants to recall the object, they simply use the original path and filename to do so. The client services front-end of Atmos acts as a proxy presenting standard web and file interfaces while obscuring the back-end object storage.

3. Atmos Managemtn Overview
Atmos provides various types of notifications and reports that help you maintain and if needed, troubleshoot your Atmos system. Atmos provides the following types of notifications:

• Alerts are created when situations result in errors and warnings, or to provide information. Sources may come from either hardware or software.

• Email notifications can be sent based on severity, which will trigger the email alerts. Each RMG has its own email alert configuration.

• SNMP – both SNMP trap generation and SNMP standard (MIB-II) MIB access are supported. For MIB access, the SNMP agent must be configured. 

Atmos also provides log collection and system reports, which are used by EMC Technical Support when troubleshooting:

• Log collection collects logs for debugging purposes. After the logs are collected, you can save the log package and send them to EMC Technical Support.

• System report collects data about your Atmos configuration and sends a report to the EMC System Reports database, through the System Report (SYR) mechanism. System reports are sent to the EMC System Reports Database to improve customer service, and allow EMC to provide timely support for Atmos issues.

 4. Docs