To start things off, a very esteemed engineer (John Walton) in our Symmetrix group got to read Part 1 of the FSS story. He provided the following feedback:

Hey Dave, in general most fault tolerant or highly available systems maintain a minimum of shared state space between members of separate fault domains. this reduces the work required to formally prove your system is resilient to faults. cache coherency protocols with distributed state are often difficult to do this with. in the past CC versions of HT have not been usable for this reason.

Regarding the coherency statement. thanks for bringing that up. as an overview, I was trying to get at the concept of coherency without getting too deep into the mechanics. Also to consider is the platform’s ability to facilitate node coherency. I know with the design of Socket L1-SP and dual HT links (one cHT and one nHT), AMD was promoting the ability to have coherency across a single 8 bit HT1.x channel while using the non-coherent link for sideband and/or random access. This actually IMPROVES somewhat with the HT3.x gen channels on Shanghai.  This would be due to increased bandwidth, low latency, and processing optimizations.

The next question comes from a very good friend of mine who has graciously taken the time out of his schedule to read this series.

What does (the FSS node architecture) do to the (AMD) 4P (83xx) offerings? Doesn’t it kind of negate the premium to AMD/Intel for the over 2P market?

There are a couple of different ways of looking at this question.  In most storage systems, node to node communication is made either over PCI(e or x) or using GigE. The upside to this is legacy bus usage (i.e. no real need to “think outside of the box”) and common platform accessibility. It also means that you would not need to use n-way processors for most applications, thus driving down platform and development costs.  The  big downside to this legacy bus based approach is scaling and bandwidth. Simple two node storage arrays using PCIe are limited to that connection which does nothing but handle simple I/O and bus chatter. Lose that link, you’re going to trespass, etc.  Re-establishing that link can be disruptive (in certain products) and generally can be a pain to handle.

The FSS uses AMD Opteron 8300 series processors for several reasons.  The first reason relates to using Hypertransport as a connectivity medium.  The 8300 series processors provide (in addition to the system HT links), HT links (8 bit) to other processors(and co-processors…remember, Torrenza is important!) within a given “system.”  The 2300 series does NOT provide this same level of HT capability.  Secondly, a trade-off had to be made with scaling vs. static nodes.  In a world where scaling storage processing nodes has become increasingly the best approach to handling accessory functions within the system OS, the FSS needs to be able to add significant processing and memory bandwidth capabilities that can only be provided by the 8300 series.  In an ironic twist, it’s more expensive for the processors (8300 vs 2300) but cheaper than designing a brand new interconnect schema for Intel Dunnington processors.

Closing Thoughts:

We’ll see if any other comments come in while this FSS article series is still in development.  As they do, I’ll keep updating.

Related articles by Zemanta

• EMC finally puts Symmetrix line on thin provisioning diet
• EMC does flash drives on big storage
• IBM’s XIV storage lacks high-end features, analyst says
• NetApp soups up its filers
• Battle of the SSD strategies: Sun vs. EMC
• EMC execs: We’ll drive SSD cost down ‘as fast as we can’
• IBM’s XIV storage lacks high-end features, analyst says

Note: This information is available within the VMware user community as well.  I am indebted to the person(s) in that community who provided this information, albeit in a slightly less “visual” way.

Just a quick note before we begin.  Just because you can INSTALL ESX 3.5 doesn’t mean that you’ll be able to USE ESX 3.5.  The Service Console will load separate from the hypervisor and, you’ll get a nasty little error stating “Mounting root failed” in the startup screens if you don’t follow these instructions.

Step 1:  Log in as root

Service Console - Root Logged In

You need to log into the ESX 3.5 service console as root in order to proceed.  If you’re like me and you are going to use the remote CLI (through PuTTY in my case), you’ll need to log in as your authenticated user (in my case “ssh”) and type “su – ” to change permissions to root.

Step 2: Use the “LSPCI” command to determine the PCIID of your SATA Controller

LSPCI Command - Determine the PCIID of your SATA Controller

The LSPCI command will give you a listing of all PCIIDs utilized in your system that will look similar to the list below.

List of all PCIIDs in current system

You’ll need to scan through the list and determine the SATA controller PCIID.  Once you find the PCIID, write it down on a piece of paper.  You’re going to need it in a few minutes.

Note: on my system, the nVidia MCP55 SATA controller PCIID was “037f“.  Your PCIID may be different.

Step 3: Find the “sata_nv.xml” file and update the PCIID

Now that you know your PCIID, you’re going to need to update the file that the hypervisor and service console reference for storage.  For nVidia SATA controllers, this file is called “sata_nv.xml” and can be found at the following path:

Location of the sata_nv.xml config file

The location of the sata_nv.xml file is /etc/vmware/pciid/ and is highlighted in the picture above.  Once you’ve found the file, you’ll need to update it with the appropriate PCIID you recorded in Step 2.

Editing the sata_nv.xml file with VI

The command line syntax is “vi sata_nv.xml”  Once you’ve entered that line, hit Enter and you’ll see the following:

VI display of the sata_nv.xml file

Move your cursor to the appropriate line (in the screen shot, that’d be “device id=00ee” or similar).  Hit “Insert” two times so that you see “Replace” in the lower left hand corner of your screen.  Anything that you type from this point on will overwrite the existing content of the file so, be careful.

Once you’ve updated the device id with the appropriate entry, hit the “Esc” key two times to clear the Replace function.  Type  “:” to access the main VI command complex, and then enter “w!” to overwrite the file contents with your updates.  Once that process finishes, type “:” again and hit “q!” to quit out of the VI editor.

Note:  The screenshots from this article are from a Dell PE1850 and consequently, it doesn’t show ALL the entries that would be found for current nVidia SATA solutions.  If you’re using any Tyan (or other) mainboard that has the NF3400/NF3600 chipset, you need to look for the device id line under the MCP55 line.  That is where you’ll need to update the device id.

Step 4: Reload the PCIID tables to overwrite current config and Reboot

Once you exit the VI editor, you’ll need to make sure that your changes are updated to the PCIID tables that the hypervisor and service console reference.

Updating the PCIID tables using ESXCFG

The command line syntax for doing this update is “esxcfg-pciid”.  Once you type this in, hit Enter and it will process the table updates.  The process should take around 15 seconds to complete and once done, it will put you back at the command line.  At this point you need to reboot to have the changes applied to your ESX installation.

Closing Thoughts:

Obviously, this is process can be a little more complicated for people who just want to “throw and go” within the virtualization space.  Part of the reason why this is required really has to do with the difference in implementation metrics from manufacturer to manufacturer as well as the generic driver model provided by VMware as part of the ESX package.  As future versions of the software are released, I’d expect that this amount of legwork would be reduced significantly.

Hope this helps get you up and running!

Cheers,

Dave Graham

Search Term #1: EMC NX4

Not really suprised here.  Honestly, take the Celerra NS-20, cut the price signficantly, allow blended SAS/SATA drive trays, and multi-protocol attachment (fibre AND iSCSI/NAS) and…you’ve got the NX4.  Perfectly blended for SMB/Commercial customers and without any additional “fluff” that you don’t need/want.  Did I mention you can get up to 60 disks on the thing?  All in all, it’s a big “win” for our customers who want CIFS/NFS, iSCSI, and 4Gb/s Fibre support but can’t break the bank doing so.  Click on the YouTube video displayed on the home page for a good overview of the box.

Search Term #2: EMC Navi Express

Second on the hit list is EMC Navi Express.  Since Navi Express is an integral part of the AX4 product line, it does merit some level of mention.  Take a look at the chart below.

Navi Express vs. Navisphere (Full)

As you can see from the chart, there are some minor limitations (depending on your definition of “minor”) but overall, it accomplishes a lot of the same flexibility and functionality that the full “Navisphere Manager” has.  Oh, and it’s HTML based, not Java.

Search Term #3: Is the Celerra any Good?

Seriously.  Someone, somewhere entered that phrase into Google or whatnot and landed here. Could have been NetApp for all I know.  Do I honestly need to answer this question?  If you want to know, take a look at EMC’s marketshare in NAS over the past 3 years.  That should provide a small measure of perspective.  Other than that, this question is a little vacuous, so, what part of Celerra do you want to know about?\

Search Term #4:  EMC CX4 Cache

I’m going to approach this as just looking for the cache sizes per Storage Processors on the CX4 models. Let’s begin.

• CX4-120:  3GB of cache per SP; 6GB total per complete system.
• CX4-240:  4GB of cache per SP; 8GB total per complete system.
• CX4-480:  8GB of cache per SP; 16GB total per complete system.
• CX4-960:  16GB of cache per SP; 32GB total per complete system.

Hopefully that will help your research as you look into my favourite EMC product.

cheers,

Dave

Related articles by Zemanta

• EMC targets SMBs, branch offices with new low-end storage
• EMC releases new multiprotocol NAS array aimed at SMBs
• EMC’s Celerra starts smaller and gets cheaper
• Low-end EMC NSX is on its way

Image via Wikipedia

Since I work in the SMB/Commercial space as a TC, I routintely am exposed to mixed fabric environments.  With the advent of iSCSI, we’ve seen a proportional shift towards iSCSI as a reduced-cost block storage fabric.  Legacy (2Gb/s) fibre still has presence in specific markets but the uptake of 4Gb/s fibre has been slowing down.  With FCoE being announced as the next logical evolution of converged fabrics and 8Gb/s FC and 10G iSCSI working their way to availability, does FCoE make sense for SMB/Commercial markets?

Personally, I do see FCoE (and DCE/DCF to a larger extent) penetrating and “working” in the Commercial/SMB markets…there are some big milestones that need to happen, though, before that penetration can really happen.

• Cost: the CNAs from Emulex and Qlogic are currently 5x’s the price of a solid 2 port FC HBA.  That’s way too pricey for SMB and is pushing it for Commercial.  Gen2 adapters (using converged ASICs) will show up in Q1/Q2 2009 and should drop the price point by about 20-30%.  That would make it more tenable.  Similarly, Cisco’s fully loaded pricing on the Nexus 5020 is $105,000.00 (give or take a few dollars)…considering that you’ll STILL need MDS-91xx series SAN switches and Catalyst IP switches in the environment, you’re looking at a decent investment somewhere north of $150,000.00 just to get it in the door.
• Availability: The sister “dog” to Cost.  If it’s not out there, you’ll pay more and you’ll have a harder time with support and acquisition.  People who are going to roll over to FCoE need to know that it’s commercially viable and that they can move their entire NOC or DC over to that platform.  Conversely, if you don’t have the early adopters, you won’t drive down cost or drive up availability.
• Support: So, similar to point A above, it’s good that the technology is here today, but who REALLY has FCoE solutions?  The Nexus, for now, is simply another appliance.  It’s neither a core switch nor an edge switch and still relies on fabric-specific devices to get stuff done.  From the CNA perspective, we’re still on Gen1…and those cards are physically HUGE (no low profile…yet), really HOT, and really pricey.

My Personal Take:

To dig even further on this… iSCSI doesn’t work as well as people think it does.  Most of my customers I have to warn about latency issues (again, astronomically higher PER LINK than FC) and bandwidth issues (80% of usable line speed, if that…typically, I rate it 60-80%).  You also need MORE ports to even get close to even performance with FC from a bandwidth perspective.  10G will assuage some of this by reducing latency and increasing bandwidth, but, even then, it’s not all rosy.  Further, most of the SAN guys don’t want to deal with networking topologies.  I can create a decent fabric for FC attached storage and hosts within 30 minutes (within ESX) and have it function extremely well.  I don’t have to grab IP addresses or iQDN names or deal with messy initiators (more of an Open Systems problem currently).  Drag, drop, commit. It’s that simple.

Check out the following Third I/O Report (commissioned by Emulex) for some level of validation as to what I just said: Third I/O Report: 10G iSCSI vs. 8Gb/s Fibre

Anyhow, those are my thoughts for this Tuesday.  Comments?

cheers,

Dave

PS> As a side note, Stuart Miniman has a quick blurb up @ his Tumblr site on FCoE.  Go here to check it out!

Related articles by Zemanta

• QLogic hypes ‘network consolidation’ with FCoE
• Brocade unfurls FCoE roadmap
• Vendors hype new Fibre Channel over Ethernet tech
• Witness the future of Brocade
• Hardware vendors primed for FCoE love-in
• CNBC hopes to fly Fibre Channel on its Ethernet super highway
• A consultant’s view of the SAN market
• The five biggest storage trends

Future Storage System - Operating System - Conceptual

I will review this diagram starting from the upper right hand corner and will drill down on each bullet point made.  Again, as these ideas are further developed, I’ll probably move to a more classic diagram via Visio or something similar, but, for now this will have to suffice. You should see my whiteboard right now.

Core:

When discussing the core of the operating system, I have to be somewhat ambiguous, namely, the underlying kernel technology will dictate the inherent flexibility of the design. I will be honest in that I believe a hypervisor-based model (a la VMWare, Xen, et al.) will enable more features and flexibility than a bare-metal OS with a direct kernel/HAL relationship.  Additionally, as we move further into the features and functionality of this FSS OS, this hypervisor building model will become extremely important.  I’ll reserve details on that for another post.

Getting back to the subject at hand, I’ve divide the “Core” section into 4 basic sections: Drivers, I/O Engine, Hypervisor (?), and Management (Base). Let’s discuss the roles of each of these.

• Drivers.  This particular aspect of the Core OS deals with how the hardware interfaces with the OS itself.  This can be either a generic driver model (a la VMWare) or can be manufacturer specific and developed/maintained (a la Hyper-V).  There are advantages to both approaches;  with generic driver models, you get away from havoc-causing revisions to driver stacks or resource models that can (and are) introduced on a regular basis.  By standardizing on a generic driver model, you can also set specific boundaries around drivers for QA purposes and keep your overall OS implentation “clean.”  With specific driver models, you reap the benefits of core optimizations made by the manufacturer. This adds to the overall performance envelope of the underlying hardware technology as it relates to the OS layer. It obviously introduces some level of risk to the OS in that driver refreshes could theoretically “break” existing connections or the OS itself, but this could be mitigated by extensive modeling and alpha/beta programs.
• I/O Engine. Again, talking in theoretics here, the I/O engine encompasses the basic storage functions for the OS.  This could be broken out into Data Protection Algorithms (i.e RAID) and Encode/Decode functions for passing data to/from cache and disk.
• Hypervisor.  As stated in the start of this section, the model I’m embracing is a hypervisor-based one that allows for functionality and features to be “plugged in” through software and hardware (don’t forget Torrenza!!!).
• Management (Base).  This management function relates to the capabilities of the FSS operator to accomplish basic tasks through a GUI or CLI interface.  It would involve RAID/Storage group creation, host assignment, registration & configuration, and other basic tasks that are typically accomplished.  As noted in the hardware overiew of Part 2, there would be a dedicated management port (GigE) designed to handle interaction.

Extended:

When discussing the extended section of the OS, I’m referring to features that extend the native capabilities of the OS as installed.  This section deals with building out the OS to cover such features as Thin (virtual) Provisioning, System Analytics, Quality of Services features, and DeDuplication.  I’ve kept this list to four major categories but I do assume that there could be other technologies that emerge that will add to this list.

• Thin (Virtual) Provisioning. Call it “thin,” call it “virtual,” it doesn’t matter.  Companies (and people) that get caught up in semantics really have better things to do with their time (like actually implementing hardware FC, not software emulation) .  Essentially, Thin Provisioning allows the Future Storage System to manage information growth by only allocating storage that is needed versus what is wanted.  By spoofing the storage size to the attached hosts, you’re able to better handle growth and storage utilization (important when you’re already dealing with usable capacities below 50%).
• System Analytics. Included in the Core OS is a base level of analytics, equivalent to the SysInfo command you could run on a typical Microsoft or VMWare OS.  By including this from the start, most customers would be able to report on their hardware/software frameworks and determine if there were any failures.  Beyond this basic level of reporting, there often arises the need to determine what is happening from a hardware standpoint, i.e. Processor load, RAID Group response time, etc.  The System Analytics extension would provide in-depth reporting facilities to “fine tune” particular aspects of the FSS.
• Quality of Service. Similar in approach to DRS-enabled clusters within the VMWare environment (where you can assign resource “shares” across several different slices of physical hardware), the QoS features of the FSS would allow RAID Group or LUN level performancing based on policies (covered in the “Specialized” Section below) in addition to potentially handing vNIC (IP, iSCSI, FCoE?) or vPORT (FC) splitting and throttling for bandwidth.  Current storage system implement similar features at the LUN level or RAID Group level only; the extension of QoS to the port level on the storage system itself is rather unique.
• Deduplication.  There’s no doubt about it; whether we’re talking about NetApp‘s ASIS or EMC’s Avamar, data deduplication is a feature that is here to stay.  The affects of deduplication may not be obvious at first but, over the lifespan of your data, it will enable quicker backups and replication in addition to increasing storage efficiency on your storage system.  Not much else to say about this particular technology for now, but, suffice it to say, as an extension to the Core OS, it can be a powerful tool.

Specialized:

Outside of the Extended features that could be “plugged in” to the Core framework, I determined that there might be a few more specialized tasks that would require extra integration from either 3rd party resources or just additional developmental work beyond the norm.  In this particular category, I’ve put In-Band/Out-of-Band (Data at Rest) Data Encryption and Policies (Spindown, Power Management).  Let’s dive into these for a moment.

• Data Encryption.  With compliance laws (PCI, HIPAA, DoD, et al) becoming more and more de rigueur in storage and communication, it almost seems that Data Encryption should be a Core component versus an add on.  Nevertheless, it’s important to recognize that you need to be able to accomplish Data Encryption (and Decryption) on resting data and on data that is being processed for storage.  If anything, this is a perfect use case for Torrenza pluggables.   Standardizing encrytion on the FSS also means that you’ll need to have some level of “hooks” to the Core engine.
• Policies.  Ah, Policy.  Everyone hates the word but doesn’t understand the importance of policy until they run into a policy-less situation.  In any case, the FSS policy add-on would be more akin to a scheduling engine for job processing, encrypt/decrypt, deduplication, tiering (storage tiering), etc.  It could assume responsibilities for power management as well, similar to the Dynamic Power Management (DPM) functions of VMWare. Whether this stays as an add on or moves to Core remains to be seen.

Business Continuity:

Business Continuity, like Data Deduplication, is a hot topic these days.  In my mind, Business Continuity encompasses not only Disaster Recovery metrics but also data management characteristics. To that end, I’ve broken this section into 3 distinct pieces: Remote replication, local replication, and tiering.

• Remote Replication. As the term states, this section is devoted to moving data from source FSS to target FSS.  The goal is ultimately data protection over any distance outside of the physical geography of the source array (i.e. New York City to New Jersey).  The mechanism for this replication would be either synchronous (acknowledged write commits from source/target arrays) or asynchronous (delayed ack from target array), depending on RPO/RTO SLAs.
• Local Replication. Local replication I view as being within the data center, NOC, or campus.  While the physical location might be removed somewhat, the communication would stay on dedicated high speed links (dark fibre, fibre connection, multi-fabric SAN, etc.).  Replication under these auspices would more than likely err on the side of synchronous but could be asynchronous as well.
• Tiering.  Before anyone objects, there is a method to my madness.  Typically, storage tiering would be slapped under the Extended or Specialized categories based on my diagram above.  However, think of it this way.  Moving low priority data from primary to secondary storage is a form of BC, isn’t it?  You’re moving, managing, and making available data under the auspices of a different SLA, potentially to different subsystems (i.e. Clariion to Centera), with different connectivity.  What if the Future Storage System could move data AND the communication channel from higher speed SSDs or mechanical disks on FCoE or IB to slower SATA drives using vNICs at GigE bandwidth levels?  That type of manipulation would obviously require some enormous computingcomputing capabilities (as well as potentially buffering to SSDs or high speed cache), but the end result could be completely managed data SLAs.

Closing Thoughts:

A lot to chew through on this article and I hope you’ll point out the flaws in my thinking.  For the next part, I probably will dive into the Core of the OS a bit more and see what technical underpinnings would work.

cheers,

Dave

Related articles by Zemanta

• SaaS Solutions for Remote Systems Management
• Business Continuity and Verizon Business Global Network Readiness
• Solid state, cloud storage on tap for a power-hungry 2008
• Open Source Hardware Management Software by Paragent
• eCos real-time OS makes short work of building a SAN appliance
• VMware chief says the OS is history
• Sun close to releasing xVM Server
• Venture-Funded Consultants at Security Innovation Show Companies How to Make Software Unbreakable
• The five biggest storage trends
• EMC targets SMBs, branch offices with new low-end storage
• Brocade unfurls FCoE roadmap
• Nexsan paints SAS green
• Witness the future of Brocade
• Vendors hype new Fibre Channel over Ethernet tech

Hypertransport I/O Expansion Topology

There are two different approaches that I’ll be taking a look at as part of this I/O expansion model: one approach is southbridge oriented with an additional southbridge providing additional PCIe lanes; the other is looking at an I/O expansion model based on integrated ASICs and fixed optical ports that can provide connectivity to Fibre Channel, FCoE, and IP technologies.  We’ll discuss both approaches separately and then dive into what each would look like practically (another diagram, I’m afraid).

Hypertransport I/O Node - details

To begin, we’ll cover the SouthBridge model of I/O expansion (left side of the diagram).  In this particular model, the expansion module includes another southbridge device to cover the additional PCIe lanes required for either x8 or x4 slots.  The SouthBridge would be connected to the primary southbridge on the base node over the HT link.  Currently, many commercial systems are using this type of topology (using either PCIe between discrete logic chips or another type of communication fabric like HT).  The southbridge(s), then, would provide discrete connectors to their own interfaces as well as maintaining system level consistency for I/O.  The advantage of this particular implementation would be the ability to utilize specific connectivity types (FCoE, FC, IB, IP, iSCSI) on the same type of pluggable cards as the base node (similar to Ultraflex on EMC Clariion CX4s).

The second variation on I/O expansion takes a different approach by integrating converged ASICs (cASIC) to handle I/O connectivity (right side of the diagram). This particular implementation has some obvious limitations from the start, namely, fixed port count, and limited fabric/topology support (IB is noticably lacking).  Additionally, the design would require more development and implementation work in order to support a passable physical implementation.  Limitations notwithstanding, there are some interesting integration points to look at.  First, the use of SFP+ or XFP pluggables to support optical or electrical physical connectivity types assuages most of the fabric connectivity issues in addition to providing some level of forward-looking interoperability.  Secondly, the use of converged ASICs allows for multi-protocol encapsulation, encode/decode support that would support the aforementioned pluggables without having to retool or replace a PCIe module.  I’m assuming that some sort of protocol bit would need to be included in the SFP/XFP pluggables to enable the cASICs to “flip” from one protocol to another.  I believe that Qlogic and Emulex (and certainly Brocade/LSI) are implementing similar types of logic into their converged ASICs on their FCoE HBAs, so the next logical step would be said integration into a larger system.

Closing Thoughts

As always, I’m interested in your feedback and understanding on this (or any) subject I’ve written about so far.  We learn together and this particular posting is trying to dig deeper, probably past the point of my intelligence.  We’ll see…

Part 4 is still up for grabs as to what it will cover, but, most assuredly, it’ll probably be something to do with the Operating System for the FSS.  Stay Tuned!!!!!

Related articles by Zemanta

• NextIO grabs $19m for virtualized PCI Express extravaganza
• The five biggest storage trends
• QLogic hypes ‘network consolidation’ with FCoE
• Hardware vendors primed for FCoE love-in
• SNWSpotlight: 8G FC and FCoE, Solid State Storage
• CNBC hopes to fly Fibre Channel on its Ethernet super highway
• Cisco’s Nexus forms core of data center drive
• New EMC Clariion line adds solid-state option
• Brocade releases second wave of HBAs
• Virtualized PCIe switch
• Vendors hype new Fibre Channel over Ethernet tech
• Brocade unfurls FCoE roadmap
• Brocade deal to help drive data center transition
• Cisco gives virtual servers virtual networking

Hypertransport Node Expansion (detailed)

We’ll approach this system overview the same way as previously done (starting from the lower left hand corner and working clockwise).

This diagram shows a layout of single node (Node A) and an expansion node (A prime).  In this diagram, we’re looking at a computational node add, not an I/O node (that’ll be covered in Part 3b ).  The basic architecture can be broken down into the following:

• Base Node with I/O
• Interconnect (physical/electrical)
• Expansion Node (computational or I/O)

Starting with the Base Node, we can see that the architecture is the exact same as covered in Parts 1 & 2.  It’s a simple dual physical processor system with dedicated memory banks per processor, a Southbridge controlling PCIe connectivity and system level I/O, and both coherent and non-coherent Hypertransport links between processors.  Although memory specifications are provided as part of the diagram, it’s inconsequential to the overall architecture.   The SouthBridge, as previously noted, would be utilized as the PCIe resource manager for system and storage I/O.  The coherent (cHT) and non-conherent (ncHT) Hypertransport links would be dedicated to intra-processor communication including facilities for NUMA (non-uniform memory access) and processor data requisitions.  This facility has already been implemented in current Bx series AMD Opteron processors as part of the Socket 1207 specification.

The interconnect portion of the diagram is both a mechanical and electrical “device.”  Essentially, it “glues” together processors into a larger “grid” doubling I/O bandwidth within the system and allowing for expansion beyond two physical processors (at least in this design).  Mechanically, it can be compared to the HTX physical interface originally specified by AMD.  Some HCA designs from Qlogic and others utilized this connector as a method of direct I/O injection into the processing path, lowering latency even further than the comparable PCI-X or PCIe models. (It was only about 1ns different, but, still, every bit helps).  Again, while HTX is the archetype for Hypertransport’s physical limitation, a slot based architecture wouldn’t be as effective as a more custom packaged model (thinking specifically about blade server-esque edge connectors).   Electrically, the HT 3.x protocol would be signaled to the expansion node as part of a larger mesh fabric to engage additional processing resources (CPUs or, as noted further down, Torrenza-compatible co-processors) or I/O resources.

In an ideal work, you’d be able to simply plug in the expansion node (especially the computation version) into the chassis while the system was running and have the OS dynamically add and allocate the resources to the rest of system.

Now the fun begins….

First, I’d like to touch on the reason for looking at expansion nodes at all.  In the storage world, there appear to be two basic approaches to adding processing power to head units:  clustering (a la Equallogic, NetApp, XIV, Permabit) or multiple storage processors (a la HDS USP-V, Symmetrix, etc.).  The advantage of clustering really trickles down into n-way ownership of data and multi-level hardware failures.  The disadvantage is having to maintain some level of heartbeat mechanism (CMI on EMC, for example) between nodes that can split-brain I/O and/or other system level processes.  Notice I said CAN not WILL.  The advantages of using multiple storage processors (directors) ties into complete hyper or LUN awareness and quick ownership failover in case of hardware meltdowns.  Additionally, tying multiple SPs together on a common bus can add to overall system performance (principles of Hypertransport, for example) via aggregated memory bandwidth and I/O.  The general disadvantages are that any sort of SP failover could ultimately impact other SPs in the system (thinking specifically of some sort of EMI burst or surge) and double-down hardware failures causing DU/DL (data unavailability or data loss).  In my mind, there’s obviously reasons to chose one over the other but, I think that ultimately, you could combine both (another article on that later).  Since we’re just evaluating a simple node add, we’re going to look at this as multiple SPs on a common backplane (HT).

Secondly, if you look at the diagram, you’ll see a note about Torrenza.  There’s a link elsewhere to the Torrenza wiki article, but to distill it down for consumption, here’s the quick n’ dirty.  Torrenza is an AMD initiative to allow dedicated co-processors access to the exact same HT and I/O stream as the CPU.  So, you want dedicated processing for x-type of application?  Install a co-processor into an available 1207 socket in the system.  Systems using Cell processors, for example, have been demonstrated behind doors (not commercially available to the best of my knowledge).  The ultimate goal here would be to allow specialized co-processors for applications (RSA disk encryption) that would be offloaded from the general storage I/O processors.  The application set is really endless.  Want to do data encryption inband or at rest?  Install an RSA encryption co-processor.  Want to do compression or de-dupe?  Install a compliant DSP or co-processor that performs that task.  When we look at the operating system for this Future Storage System, you’ll see even more applicability.

Closing Thoughts

Tomorrow we’ll take a look at what an I/O based expansion node would look like and see the implications there.

Again, comments and feedback are welcome!

Related articles by Zemanta

• AMD to make a splash in the server chipset space

Fleshing out the Hypertransport Storage System

Starting clockwise from the lower left hand corner, let’s take a deeper look at things.

First off, multi-core processors are here to stay.  It’s honestly not worth evaluating even dual core processors these days in that most server/workstation systems are designed around multiple processor cores. (AMD Shanghai/Istanbul, Intel Dunnington, for example) The obvious requirement from a software level is that the storage system OS is multi-threaded and contains a scheduler that understands that multiple logical cores exist and are usable.  Further, using processor affinity rules (a la VMWare’s ESX hypervisor layer), you could tie the OS to a specific core and leave the reast of the cores available for specific processes (i.e. RAID XOR or parity calculations, etc.).

Second on the list is inter-processor communication.  Historically, this would be called the “system bus” and any I/O traffic to processors and memory would be conducted over this link.  With the advent of direct memory access (see next point) and the subsequent removal of general memory traffic from the processor bus (outside of requests for memory pages not located directly in the processor’s “owned” bank), the traffic across these links would be limited to I/O and processing requests.  Pretty handy.  Using Hypertransport, you could have have (as is the case today) a coherent processor link and a non-coherent link (sideband traffic) that would be able to gang/un-gang as needed (feature of HT 3).

Thirdly, it’s important to realize the role that cache/memory plays in storage systems.  Not only is important for the array OS (FLARE, DART, OnTAP, etc. etc.) and it’s various operating needs, it’s also used for the read/write cache.  Obviously, the different OSs’ offer various methods of manipulation to tune reads vs. writes and allocate more to one bucket than the other.  The other necessary requirement for memory has to do with the amount of processors in a system.  Obviously, based on this design above, you can either tie your processors to a central MCH (Memory Controller Hub) like Intel has historically done (Nehalem changes this) or put the MCHs inside the physical processor die (for example, AMD Opteron processors).  An issue obviously will arise when you go to read memory from an adjoining processor, injecting latency and I/O hops, but overall the integrated MCH design reduces memory access and allows for additional bandwidth and reduced latency for direct processor accesses. A lot of this can be assuaged by careful system design, but, we’ll need to approach this in a later article more in depth.

Fourth on the hitlist is the Southbridge and PCIe.  PCIe is usually discussed from a platform perspective as having “lanes.” The higher the bandwidth, the more “lanes” required. In order to keep the system I/O per node within servicable spec, I’ve purposefully limited the PCIe expansion to 3 x x8 PCIe slots.  In my mind, to be truly multiprotocol, there needs to be support for the “standard” SAN connectivity types: Fibre Channel, iSCSI, and NAS.  In my mind, I can see where either Infiniband (doing something like IP over IB or IB-to-FC routing with Qlogic Silverstorm or Xsigo routers) or FCoE  (Cisco Nexus or Brocade DCX) would provide enough SAN-facing connectivity and bandwidth for customers without completely saturating the system bus.  At the same time, it’s important to allow for some level of I/O expandability to avoid issues of array downtime for port upgrades and the like.  This is currently an issue with NetApp, Lefthand, HP, HDS, Sun, 3Par, Pillar, Equallogic, et al and I’d point to the EMC Clariion CX4 line as the exemplar of how to handle the I/O expansion issue the correct way. Compellent has a pseudo-interface updating process documented somewhere, but honestly, it’s nowhere near as elegant as the CX4.

Since I didn’t touch on the SouthBridge portion of things, I’d just hastily point out that the AMD/ATI relationship has recently born significant fruit in the addition of new chipsets that will actually give the AMD platform a bit of a boost when it comes to server-class implementations.  Nothing against nVidia or Broadcom, per se, but I’m tired of MCP55/HT2000 based systems.

Fifth on the list is I/O interfaces to the SAN. As noted, you can see that I’ve included pretty much any interface on the market as an option.  Fibre Channel over Ethernet, Fibre Channel (discrete), Infiniband, iSCSI, and IP (GigE or greater) all have their relative merits on the market.  This type of flexibility is a must especially since some customers will never use certain interface types.  From a high level, I agree with the “multi protocol” box sentiment that everyone seems to espouse, but there needs to be a certain level of flexibility to remove those protocols that are not needed.  If you look at this platform as a “framework” for any protocol, you could easily add/remove interfaces as needed.  Customer wants only NAS/IP connectivity? The base framework won’t change, the I/O card(s) will.  I personally believe the trend is towards integrated fabrics/protocols like FCoE and IPoIB, but, we’ll see what the general acceptance is.

Finally, you need some way to manage the nodes and the OS.  There’s a GigE port per node to handle any management functions that could be needed.

In any case, that’s it for today!  Let me know your feedback and thoughts!

(special thanks to Stephen Todd and Ken Ferson @ EMC for getting me to THINK!)

10/14/08 : (EDITS) : made edits for clarity and appropriate product naming conventions.

Related articles by Zemanta

• AMD finally ships quad-core Opterons
• AMD announces 6- and 12-core Opterons
• Intel’s Six-Core Dunnington and Nehalem Microarchitecture Get Official [Intel]
• Intel’s Dunnington: Six cores on one chip
• AMD says Shanghai won’t be another Barcelona
• Hardware vendors primed for FCoE love-in
• EMC in fourth-gen CLARiiON call
• Vendors hype new Fibre Channel over Ethernet tech
• CNBC hopes to fly Fibre Channel on its Ethernet super highway
• Cisco switch consolidates functions in the data center
• HP jilts Intel SSDs

A case for Hypertransport connected nodes...

Reviewing the image, you can see that I’ve essentially “glued” the two nodes together using either Hypertransport 1.x or 3.x spec signaling. With this model, you COULD feasibly scale to n-way nodes but, for reasons of simplicity, I’ve kept it “pure.”  Preference would be for HT3 spec as it allows for ganging/un-ganging, bit width changes, and lower system power draw but HT1.x would be similarly effective (though not as integrated with future technology).  I’ve obviously not listed processor specs as such, but given that HT is really only available on AMD based platforms, you can take a guess as to where I’m heading.

Secondary to the actual computing side of things, I’ve looked for basic functionality for PCIe expansion.  Recognizing that PCIe (PCI Express in long form) is going to be the essential bus for the forseeable future, I made provision for three x8 slots for I/O.  There are some natural assumptions as part of the design.  First, x8 is going to handle the majority of bandwidth that can be thrown at it. (x8 Physical slot/Electrical signaling)  High bandwidth, low latency interconnects like Infiniband (SDR/DDR/QDR from Mellanox, Qlogic, et al.) and/or FCoE (Fibre Channel over Ethernet; from Brocade, Emulex, Qlogic, et al.) should be the necessary beneficiaries of this type of technology.  Secondly, since the topology is based on n-way nodes (>= 2), the PCIe lane count can be kept rather simple (currently, only 24 lanes are required for I/O; additional lanes would be required by southbridge/northbridge toplogies, etc.).  Keeping this expansion simplistic reduces the necessary power draw of component to a smaller scale and allows for some level of node portability.  Physically, it also reduces the size of the nodes as the 3 slots could be engineered in a smaller footprint.

If you see any significant flaws or just want to give feedback on the conceptual design, let me know.  I’m learning as I go along.

cheers,

Dave

Related articles by Zemanta

• AMD to make a splash in the server chipset space
• Hardware vendors primed for FCoE love-in
• Brocade unfurls FCoE roadmap
• QLogic hypes ‘network consolidation’ with FCoE
• Intel adds Fibre Channel over Ethernet implementation for Linux

Image via CrunchBase

I get asked a lot of questions regarding VMWare setup, etc. and I thought I’d take the time to push this out to the front page on the blog.

This is for Virtual Infrastructure 3: 3.5 Update 2/3:

Main Documentation Page:

http://www.vmware.com/support/pubs/vi_pages/vi_pubs_35u2.html

Quick Start Guide:

Quick Start Guide

Installation Guide:

Installation Guide

Basic Systems Admin Guide:

Systems Administration Guide

Virtual Center Installation Guide:

Virtual Center 2.5 Installation Guide

Virtual Machine Backup Guide:

Virtual Machine Backup Guide

Server Config Guide:

Server Config Guide

iSCSI and Fibre Config Guides:

iSCSI & Fibre Channel

Related articles by Zemanta

• VMware floats Cloud Edition
• VMware Security Advisory: ESX and ESXi Patches
• Cisco to Support VMware?
• Virtualization Analysis for VMware