Entries for Wenming Ye

Windows HPC Server 2008 Monitoring and Management with Rae Wang [Windows HPC Server 2008 Monitoring and Management with Rae Wang]

Wenming Ye — Tue, 17 Nov 2009 17:50:42 GMT

Monitoring and managing a large scale cluster often requires advanced tooling. System Administrators demand tools that help them to manage heterogeneous compute nodes, check cluster status at a glance, identify deviance, correlate node and job information, track changes, and the ability to integrate with existing IT infrastructure. Windows HPC Server 2008 admin console addresses all of the above problems with an integrated solution. The admin Console includes the following five main areas, charts and reports, configuration, node management, job management, and diagnostics. In addition, the console has a “pivoting” feature that allows the system administrator to navigate to different views by keeping the same context. Our Program manager Rae Wang, will go through each of the five areas with demonstrations and simple scenarios.

in reply to Windows HPC Server 2008 Monitoring and Management with Rae Wang

Performance and Management Improvements [Performance and Management Improvements]

Wenming Ye — Tue, 17 Nov 2009 17:50:42 GMT

The "Performance and Management Improvements" presentation looks at the improvements in Windows HPC Server 2008. The discussion begins with a look at Abe, the Windows HPC Server 2008-based NCSA cluster that ranked at #23 on the June 2008 TOP500 list. The presentation then reviews the history of computers running Windows HPC Server 2008 or Windows Compute Cluster 2003 that made it to the TOP500 list. The presentation then delves into some of the features that have led to performance improvements, such as NetworkDirect, the new RDMA networking interface. Next, learn about tuning the network (Infiniband and GigE), tuning the memory access, and other infrastructure tuning adjustments that can increase performance. The discussion continues with a look at tuning high-performance computing (HPC) applications, starting with the basics (such as perfmon) and then going into more detail. Then, see these performance tools and techniques demonstrated on a basic laptop. The presentation then concludes with a detailed look at event tracing (specifically, MS-MPI integrated with Event Tracing for Windows).

in reply to Performance and Management Improvements

New Job Scheduler Features [New Job Scheduler Features]

Wenming Ye — Tue, 17 Nov 2009 17:50:41 GMT

The "New Job Scheduler Features" presentation covers all aspects of the Job Scheduler in Windows HPC Server 2008. Beginning with a primer on Job Scheduler terminology and concepts, the presentation then dives into the Job Scheduler architecture, exploring the components and highlighting what has changed since the previous version. This is followed by a demonstration of the Job Scheduler features, including a look at the Job Submission Console and the use of Windows PowerShell in the command line. Learn about the enhancements to Job Scheduler, such as support for service-oriented architecture (SOA) and interoperability through features like the Open Grid Forum (OGF) HPC Basic Profile and third-party job schedulers. The presentation concludes with a discussion of the five new scheduling policies available, as well as a demonstration.

in reply to New Job Scheduler Features

High Performance and Productivity Computing with Windows HPC [High Performance and Productivity Computing with Windows HPC]

Wenming Ye — Tue, 17 Nov 2009 17:50:41 GMT

The "High Performance and Productivity Computing with Windows HPC" presentation begins with a history of high-performance computing (HPC) at Microsoft and a look at the goals of the HPC team. The discussion then delves into the various features of Windows HPC Server 2008, such as integrated monitoring and the comprehensive diagnostics suite, providing a comprehensive view of the characteristics and capabilities of the product.

in reply to High Performance and Productivity Computing with Windows HPC

Dual-Boot and Virtualization with Windows HPC Server 2008 and Linux Bull Advanced Server for Xeon [Dual-Boot and Virtualization with Windows HPC Server 2008 and Linux Bull Advanced Server for Xeon]

Wenming Ye — Tue, 17 Nov 2009 17:50:41 GMT

This white paper explores ways to provide optimal flexibility for running several operating systems on a high-performance computing (HPC) cluster—specifically dual-boot clusters and Hybrid Operating System Clusters (HOSCs). The choice of an operating system for an HPC cluster is a critical decision for IT departments. The goal of this paper is to show that simple techniques are available today to optimize the return on investment for an HPC infrastructure by making the choice of operating systems unnecessary and keeping the HPC infrastructure versatile and flexible. This paper introduces Hybrid Operating System Clusters. An HOSC is an HPC cluster that can run several operating systems simultaneously. This paper addresses the situation where two operating systems are running simultaneously on a cluster: Linux Bull Advanced Server for Xeon and Microsoft Windows HPC Server 2008. However, most of the information presented in this paper can apply to scenarios with three or more operating systems running simultaneously on a cluster, possibly other operating system distributions, with slight adaptations. This document gives general concepts and detailed setup information. Initially, the technologies necessary to design an HOSC are defined—dual-boot, virtualization, and Pre-boot Execution Environment (PXE). Next, different approaches of HOSC architectures are analyzed and technical recommendations are given with a focus on computing performance and management flexibility. The recommendations are implemented to determine the best technical choices for designing an HOSC prototype. The installation setup of the prototype and the configuration steps are explained. Finally, basic HOSC administrator operations are listed and ideas for more exhaustive HOSC testing are proposed.

in reply to Dual-Boot and Virtualization with Windows HPC Server 2008 and Linux Bull Advanced Server for Xeon

Integrating a Windows HPC Server 2008 Cluster into a Linux Environment [Integrating a Windows HPC Server 2008 Cluster into a Linux Environment]

Wenming Ye — Tue, 17 Nov 2009 17:50:41 GMT

This white paper provides an overview of a series of more-detailed documents that guide a Linux administrator through the process of introducing Windows HPC Server 2008 into a predominantly Linux-based cluster environment. The paper begins by describing the installation and configuration of mixed Windows-based and Linux-based clusters, with the goal of making access to the Windows cluster from a Linux environment as transparent as possible to the cluster users. There are intermediate testing steps provided to ensure that each component in the system is working correctly before moving on. The white paper also documents the test-bed configuration that was used for the development of the more-detailed guides. IT administrators can use this test-bed configuration to set up identity management, uniform file access, and job submission for a number of different selected scenarios. Next, the white paper gives general guidance for the installation and configuration of a Linux-based client system that can access the Windows-based cluster. Learn how to configure the workstation so that it can use single sign-on (SSO), the Active Directory Domain Services facility for authentication and authorization. The paper then describes how to configure Samba and network file system (NFS) file shares on a Linux server so they will be accessible to a user in the Active Directory environment. This is helpful because many HPC environments use large storage systems to contain all of the data they are processing, rather than storing the data on the HPC cluster itself. Finally, the whitepaper gives high-level guidance for integrating the Windows-based system into the scheduling environment of a Linux cluster. Using a Linux-based distributed resource management package such as the Sun Grid Engine, users can submit their workloads to a "manager" system, which can then transparently distribute individual jobs to either Linux-based clusters or the Windows-based cluster.

in reply to Integrating a Windows HPC Server 2008 Cluster into a Linux Environment

Performance Tuning a Windows HPC Cluster for Parallel Applications [Performance Tuning a Windows HPC Cluster for Parallel Applications]

Wenming Ye — Tue, 17 Nov 2009 17:50:40 GMT

This white paper discusses the factors that can affect the performance of your Windows HPC Server 2008 cluster with a focus on tuning and measuring tools and methods. The paper’s goal is to help you identify the interconnect hardware to choose for your cluster and to explain how to tune that interconnect and software stack for your application and needs. The paper focuses on the specifics of tuning and configuring Windows HPC Server 2008, and it should help you identify the kind of high-performance computing (HPC) cluster you have and help you determine the most effective areas to concentrate your performance-tuning efforts and resources. This paper does not cover specific models, processors, or brands of computers to buy. It does, however, make specific recommendations about what to consider in your purchasing decisions, the criteria for performance testing tools and methods, and specific configurations for general execution cases. Topics covered in the paper include general performance tuning tools and methods (NetworkDirect versus TCP/IP, GigE versus specialty networking, InfiniBand and the OpenFabrics alliance, memory and CPU constraints, and large clusters), tuning for messaging-intensive applications (heavy messaging, latency-sensitive messaging, setting the Message Passing Interface (MPI) network, performance tips for message-intensive applications, Microsoft Message Passing Interface (MS-MPI) shared memory, simple multipurpose daemon, and typical microbenchmark network performance), tuning for embarrassingly parallel applications, performance tuning and measurement tools (using built-in diagnostics in the administration console, using mpipingpong, mpipingpong examples, using NetworkDirect PingPong), and performance measurement and tuning procedures.

in reply to Performance Tuning a Windows HPC Cluster for Parallel Applications

Building and Measuring the Performance of Windows HPC Server 2008–Based Clusters for TOP 500 Runs [Building and Measuring the Performance of Windows HPC Server 2008–Based Clusters for TOP 500 Runs]

Wenming Ye — Tue, 17 Nov 2009 17:50:40 GMT

The goal of this white paper is to encourage customers to deploy and optimize Windows HPC Server 2008–based compute clusters, to attempt High-Performance LINPACK (HPL) benchmarking runs on these clusters, and to submit the benchmark results to the TOP500 List. This white paper begins by introducing the TOP500 List and emphasizing its importance. This paper also provides a brief overview of the HPL benchmark that is used by the TOP500 List to rank computer systems. Because a benchmarking run requires a commitment of resources and time, this paper intends to provide the motivation customers may need to help them decide to attempt TOP500 runs, even if their clusters are unlikely to actually make the TOP500 List. Next, this white paper provides guidance for deploying, verifying, and tuning a Windows HPC Server 2008–based cluster for best performance. The paper includes general, easy-to-follow steps with links for additional, more detailed information; considerations that are unique to a large-scale TOP500 run are discussed. The aim of this paper is to provide enough information so that an IT generalist is able to set up and optimize the performance of a large-scale, Windows-based cluster. This white paper then discusses how to build and prepare the HPL benchmark. Planning and tuning tools are examined and recommended, including a Microsoft Office Excel add-in for sizing parameters, submitting jobs, and managing and analyzing the results; these tools are essential for verifying and optimizing the performance of large, Windows-based clusters. The paper also highlights the LINPACK Tuning Wizard, a tool to automate the process of optimizing HPL for a large-scale cluster. Finally, this white paper describes how to refine the parameter values to optimize performance, and most importantly, how to submit the results to the TOP500 List. References are also given for further information.

in reply to Building and Measuring the Performance of Windows HPC Server 2008–Based Clusters for TOP 500 Runs

Windows HPC Server 2008 Job Templates [Windows HPC Server 2008 Job Templates]

Wenming Ye — Tue, 17 Nov 2009 17:50:40 GMT

Job templates serve as patterns for the creation and submission of jobs on a Windows HPC Server 2008-based cluster. They are defined by the IT administrator, and then employed by the end user; thus, job templates free the end user from having to learn the specialized Job Scheduler nomenclature before they can submit a job. In the "Introducing Job Templates" white paper, learn how to create job templates and find out how to use them to constrain resource allocation and prioritize access to shared resources. See how to control access to the job templates themselves, and how to make use of the default job template. This white paper is intended to help you become more comfortable with this useful Windows HPC Server 2008 feature.

in reply to Windows HPC Server 2008 Job Templates

Using Windows HPC Server 2008 Job Scheduler [Using Windows HPC Server 2008 Job Scheduler]

Wenming Ye — Tue, 17 Nov 2009 17:50:40 GMT

The "Using Windows HPC Server 2008 Job Scheduler" white paper describes the new Job Scheduler functions in Windows HPC Server 2008, such as job submission (including creating jobs, using the Job Scheduler API for C#, and using the HPC Basic Profile Web Service), job scheduling (including scheduling policies such as backfill and priority-based first-come first-served), and job execution (including the use of filters and security considerations). The paper also discusses the service-oriented architecture (SOA) programming model and how it can be used for effective job scheduling. The Windows HPC Server 2008 Job Scheduler provides greater scalability and supports advanced policies. It includes a new SOA mode that provides access to interactive applications through Windows Communication Foundation (WCF), ensuring a streamlined build/debug/deploy developer experience. The graphical interface for the Job Scheduler is now fully integrated into the Administration Console, and the command-line interface now uses Windows PowerShell for all Job Scheduler functions. In addition, core Job Scheduler functionality is exposed using the Open Grid Forum's HPC Basic Profile Web service. With the new Job Scheduler in Windows HPC Server 2008, administrators can more easily and effectively harness the power of high-performance computing.

in reply to Using Windows HPC Server 2008 Job Scheduler

Windows HPC Server 2008 Getting Started Guide [Windows HPC Server 2008 Getting Started Guide]

Wenming Ye — Tue, 17 Nov 2009 17:50:40 GMT

Windows HPC Server 2008 provides an integrated application platform for developing, deploying, running, and managing high-performance computing (HPC) applications. The "Windows HPC Server 2008 Getting Started Guide" provides basic conceptual information and general procedures for installing an HPC cluster using Windows HPC Server 2008. You can use this guide as a reference when you deploy and configure the head node in your HPC cluster, add compute nodes to the cluster, and verify that your cluster deployment has been successful. Checklists are provided for every task to guide you in the procedures. The guide begins with the first step in the deployment of your HPC cluster: making important decisions, such as deciding how you will be adding nodes and choosing a network topology. The next portion of the guide deals with deploying the head node; this involves installing Windows Server 2008 on the designated head node computer, joining that computer to a domain, and installing the Microsoft HPC Pack 2008 on the head node computer. After you have deployed the head node of your HPC cluster, you must configure it by following the configuration to-do list in HPC Cluster Manager. Once this is complete, the guide shows you how to add the compute nodes to the cluster. Next, the guide describes how Windows HPC Server 2008 simplifies the deploying process of compute nodes by providing automatic node imaging and naming, and other capabilities to streamline deployment tasks. Windows HPC Server 2008 also provides tools that you can use to monitor the progress of your deployment. Finally, after you have configured your head node and added all compute nodes to the cluster, the guide describes how to run diagnostic tests to validate cluster functionality and troubleshoot any configuration issues.

in reply to Windows HPC Server 2008 Getting Started Guide

Dramatic Acceleration of Excel-based Trading Simulations with Platform and Microsoft [Dramatic Acceleration of Excel-based Trading Simulations with Platform and Microsoft]

Wenming Ye — Tue, 17 Nov 2009 17:50:38 GMT

Tired of waiting hours for simulation results? In this solution overview, Platform will review the challenges of running Excel models in a distributed environment, and describe the benefits and details of how to deploy models without wholesale code/macro changes. Learn how to enhance application performance and empower quants and developers to deploy distributed applications quickly without modification, using Platform Symphony's Excel Connector with Windows HPC Server 2008 and Microsoft® Office 2007 Excel. You’ll discover how the world’s top financial firms can realize new revenue opportunities ahead of the competition by leveraging this joint solution.

in reply to Dramatic Acceleration of Excel-based Trading Simulations with Platform and Microsoft

Asian Options Pricing Video [Asian Options Pricing Video]

Wenming Ye — Tue, 17 Nov 2009 17:50:38 GMT

Windows HPC Server 2008 includes support for parallel applications that use the service-oriented architecture (SOA) programming model; these applications use compute clusters interactively to provide near real-time calculation of complex algorithms such as those used for Asian options pricing. This video demonstrates how to use Windows HPC Server 2008 for pricing Asian options using Monte Carlo pricing runs on a Microsoft Office Excel spreadsheet; with a single click, you can perform multiple iterations and get results in approximately four seconds. In a non-cluster environment (a single desktop), the same calculations would take about 45 seconds. This example uses three nodes of a cluster, each with four cores (for a total of 12 cores). The video then focuses on the code that performs the calculations. Beginning with an examination of the service code, the presenter identifies the six parameters from the spreadsheet. He then points out that there are two loops: the outer loop with the number of runs (1 million, in the example), and the inner loop with the Monte Carlo path period (in the example, the period is 20 this means there are 20 million calculations in this spreadsheet). Next, the video examines the client in this case, the client is the Excel spreadsheet. The example uses the technology "C# behind"" C# code behind the Excel spreadsheet. The presenter steps through the source code to show how to first establish the session and then attach to a running session. Finally, the video looks at the code behind the main Excel worksheet, and shows how and where the pricing results are actually calculated.

in reply to Asian Options Pricing Video

New Application Scenarios [New Application Scenarios]

Wenming Ye — Tue, 17 Nov 2009 17:50:37 GMT

The "New Application Scenarios" presentation examines the Windows Communication Foundation (WCF) programming model for high-performance computing (HPC) applications in Windows HPC Server 2008. The presentation begins by discussing the benefits in building, running, and managing a solution that combines WCF with HPC, and then delves into the architecture of such a solution. This is followed by a demonstration showing how to use an HPC and WCF solution for bond calculations in Microsoft Office Excel. The discussion then focuses on the details of WCF, with additional demonstrations that show how to create and deploy a service and how to create the client. The presentation concludes with a look at managing service-oriented architecture (SOA) applications through monitoring the WCF broker nodes, running diagnostic tests, and monitoring the service jobs. A final demonstration shows the admin console and the job console.

in reply to New Application Scenarios

WCF Broker Tutorial [WCF Broker Tutorial]

Wenming Ye — Tue, 17 Nov 2009 17:50:37 GMT

This tutorial guides you through the process of building and deploying a simple client and service based on Windows Communication Foundation (WCF). It then describes how to deploy and execute the client and service on a Windows HPC Server 2008-based cluster. After outlining the prerequisites, the tutorial begins by describing how to unpack the instructions for the example. The first step is then to configure the head node to be a broker (or router) node. The tutorial goes on to describe how to build and deploy the service using either xcopy or MSI (Windows Installer); steps are provided for both approaches. After deployment, the client will be ready to run. The next section of the tutorial guides you through the three procedures required to build a service-oriented architecture (SOA) application using the session API: creating the service, deploying the service DLL to the cluster (which involves registering the service DLL on each node in the cluster), and creating the client. The code for each procedure is also given and explained.

in reply to WCF Broker Tutorial

WCF Services Tutorial [WCF Services Tutorial]

Wenming Ye — Tue, 17 Nov 2009 17:50:37 GMT

This tutorial guides you through the main steps in using a Windows Communication Foundation (WCF) broker node for a service-oriented application: building the solution, deploying the service, and running the model. The application used in this tutorial is an Asian Options pricing service. The tutorial first describes how to customize the Microsoft Visual Studio 2008 project solution file so that it reflects the name of the cluster head node. Next, it shows how to build the solution with a single command. The tutorial then describes how to deploy the service by copying the binary files to the compute nodes, and concludes with running the Asian Options pricing model to show that it is functioning correctly.

in reply to WCF Services Tutorial

Windows HPC Server 2008 Datasheet [Windows HPC Server 2008 Datasheet]

Wenming Ye — Tue, 17 Nov 2009 17:50:37 GMT

Windows HPC Server 2008 provides a productive, cost-effective high performance computing (HPC) solution that runs on x64-bit hardware. Windows HPC Server 2008 can be deployed, managed and extended using familiar tools and technologies: Administrators can leverage existing Microsoft tools to centrally manage their entire Windows Server infrastructure without mastering the command-line interface. This datasheet describes the features and benefits of Windows HPC Server 2008 , which include enhanced productivity, scalable performance, improved monitoring and reporting, and the ability to quickly deploy a manageable infrastructure. The datasheet also includes an architectural overview and a summary of the system hardware and software requirements. Finally, because Windows HPC Server 2008 can provide a foundation for service-oriented applications (SOA), the datasheet also describes the steps required to create interactive sessions through Windows Communications Foundation (WCF).

in reply to Windows HPC Server 2008 Datasheet

Overview of SOA Programming Model and Runtime System for Windows HPC Server 2008 [Overview of SOA Programming Model and Runtime System for Windows HPC Server 2008]

Wenming Ye — Tue, 17 Nov 2009 17:50:36 GMT

Windows HPC Server 2008 includes support for parallel applications that use the service-oriented architecture (SOA) programming model; these applications use compute clusters interactively to provide near real-time calculation of complex algorithms. The "Overview of SOA Programming Model and Runtime System for Windows HPC Server 2008" white paper provides a technical overview of SOA applications and the Windows HPC Server 2008 functions that support the SOA model, including building and deploying SOA applications; architecture, runtime system, scaling, and performance considerations; and monitoring and troubleshooting. With the growing number and size of problems being tackled on ever-larger clusters, developers, users, and administrators face increasing challenges in meeting time-to-result goals. Applications must be developed quickly, run efficiently on the cluster, and be effectively managed so that application performance, reliability, and resource utilization are optimized. Building applications using a service-oriented architecture (SOA) approach with Windows HPC Server 2008 can help meet these challenges. Windows HPC Server 2008 provides a platform for SOA-based applications. The SOA programming model enables solution developers and architects to rapidly develop new high performance computing (HPC) cluster-enabled interactive applications, while also making it easy to modify existing distributed computing applications. With Windows HPC Server 2008, the developer build/debug/deploy experience is streamlined, the speed of processing is accelerated, and the management of the applications and systems is simplified.

in reply to Overview of SOA Programming Model and Runtime System for Windows HPC Server 2008

MS MPI and SDK Improvements [MS MPI and SDK Improvements]

Wenming Ye — Tue, 17 Nov 2009 17:50:35 GMT

This presentation begins with a bit of background on Windows HPC Server 2008 from a software developer's point of view, describing the various interfaces available and the Job Scheduler stack. Next, there is a demonstration on how to use the Scheduler API with C# in Microsoft Visual Studio. Next, the presentation examines the features of MPI.NET, as well as the new managed MPI library API, including performance improvements and Remote Direct Memory Access (RDMA) via NetworkDirect. Learn about the fundamentals of MPI, including some examples that illustrate key concepts. There is also a thorough discussion and demonstration of debugging; this is an important topic because parallel applications are notoriously hard to debug. Finally, the presentation focuses on event tracing, and offers guidance on running tracing activities in Windows HPC Server 2008. Two trace analysis tools are covered, the Jumpshot MPI trace viewer and the Intel Trace Analyzer MPI trace viewer (also known as Vampir). The presentation concludes by walking through the steps for tracing applications in detail.

in reply to MS MPI and SDK Improvements

The Windows Parallel Programming Ecosystem Datasheet [The Windows Parallel Programming Ecosystem Datasheet]

Wenming Ye — Tue, 17 Nov 2009 17:50:35 GMT

This datasheet explores the rich collection of tools that Microsoft and others provide to address the needs of multi-core and cluster programmers. First are the compilers. There are a number of optimizing compilers available for the Windows platform; Microsoft Visual Studio itself includes compilers for Visual C++, Visual C# and Visual Basic. New to the family is F#, a high-performance functional programming language. Others include Intel C++, PGI C++, and Fortran. The next category of tools the datasheet covers is profilers. Some perform system-wide profiling; others look for single Dcache misses in a three-level nested inner loop. For example, for Message Passing Interface (MPI) profiling, Windows HPC Server 2008 includes an Event Tracing for Windows (ETW)-based profiler that collects events and messages at the application, MPI library, and driver levels; performs clock corrections; and displays the results for analysis. Code analyzers, such as Marmot, Vampir, and the Intel Thread Checker, can be used with Windows HPC Server 2008. Also in the developer's toolbox are parallel programming models for both cluster and multi-core architectures. On the cluster side, there is Microsoft message passing interface (MS MPI), which is based on MPICH2 with some Windows-specific changes for security and performance. At the multi-core node level, Visual C++ provides support for the OpenMP standard in both native and managed (.NET Framework) modes. Partner compilers, such as those from Intel and PGI, also provide OpenMP support, as well automatic parallelization for certain loops. Additional tools covered include math libraries and visualization tools; testing features that are built into Visual Studio; UNIX coexistence and migration tools; and tools intended specifically for scientists, such as Mathematica and MATLAB. The datasheet concludes with a "Parallel Programming Tools at a Glance" chart that lists available tools grouped by their specific function.

in reply to The Windows Parallel Programming Ecosystem Datasheet

Detecting MPI Usage Errors with Marmot [Detecting MPI Usage Errors with Marmot]

Wenming Ye — Tue, 17 Nov 2009 17:50:35 GMT

This white paper describes how to install, deploy, and use Marmot, an automated tool that checks the accuracy of MPI application code during runtime.

in reply to Detecting MPI Usage Errors with Marmot

HPC Resource Kit: Tracing the Execution of MPI Applications with Windows HPC Server 2008 [HPC Resource Kit: Tracing the Execution of MPI Applications with Windows HPC Server 2008]

Wenming Ye — Tue, 17 Nov 2009 17:50:34 GMT

The version of the Microsoft Message Passing Interface (MS-MPI) included in Windows HPC Server 2008 has been paired with the Event Tracing for Windows (ETW) infrastructure in the Windows client and server operating systems, giving application developers the ability to create traces in production environments, create MPI event logs from all processes on all nodes running MPI applications, and tap into "live" event streams. This article describes the use of ETW in combination with Windows HPC Server 2008. It begins with the four basic steps: adding a "-trace" argument to the command, which launches your MS-MPI application; creating the CPU clock synchronization data for each process; formatting the binary .etl file for viewing and analysis; and copying the formatted event files to a single location and, optionally, merging them into a single, time-correlated log of MPI events on all processes and all nodes. A sample script to automate the MPI trace process is provided. The paper then explains how to interpret the trace output by reading the text event logs and by using the trace files view in Jumpshot. Finally, the paper covers some advanced tracing topics, such as troubleshooting, tracing applications that are integrated with the Windows HPC Server 2008 Job Scheduler, and tapping into the "live" ETW traces.

in reply to HPC Resource Kit: Tracing the Execution of MPI Applications with Windows HPC Server 2008

Tools for the Classic HPC Developer [Tools for the Classic HPC Developer]

Wenming Ye — Tue, 17 Nov 2009 17:50:34 GMT

There is a diverse range of high-performance computing (HPC) applications in use today, most of which were developed on UNIX-heritage cluster platforms like Fortran. This white paper, titled "Tools for the Classic HPC Developer," shows practical examples of porting legacy application code for use on Windows HPC Server 2008-based clusters, and provides an overview of the compiler, tools, and resources available to facilitate such migrations. The white paper begins by exploring the compiler technologies required to maximize performance of compute-intensive applications on Windows; available resources for parallel programming using OpenMP and message passing interface (MPI); tools available for debugging and profiling Windows cluster applications; and an introduction to Fortran programming within Microsoft Visual Studio 2008. The paper then discusses installing compilers and tools on a PGI workstation, porting the OpenMP and MPI network-attached storage (NAS) parallel benchmarks to Windows, debugging MPI and OpenMP programs, and building the Linpack HPC benchmark. Most of these concepts are presented through simple step-by-step Tutorials1 and companion example code.

in reply to Tools for the Classic HPC Developer

Using Microsoft Message Passing Interface (MS MPI) [Using Microsoft Message Passing Interface (MS MPI)]

Wenming Ye — Tue, 17 Nov 2009 17:50:34 GMT

Message Passing Interface (MPI) and MPI2 are widely accepted specifications for managing messaging in high performance computing (HPC) clusters. Among the most widely accepted implementations of MPI is the Argonne National Laboratory MPICH2 reference implementation, an open-source implementation. Windows HPC Server 2008 includes a version of this specification called Microsoft Message Passing Interface (MS MPI), which is not only based on and highly compatible with MPICH2, but also adds enhanced security and process management capabilities for enterprise environments and a new execution tracing feature for Windows HPC Server 2008. MS MPI uses an efficient shared-memory communication between cores on a compute node and NetworkDirect drivers to provide high-performance MPI network support for Gigabit Ethernet and InfiniBand adapters, in addition to adapters that have a NetworkDirect or Winsock Direct provider. The "Using Microsoft Message Passing Interface (MS MPI)" white paper begins with an introduction to MPI, a comparison of MS MPI and MPICH2, and a feature overview of MS MPI, including Mpiexec features and event tracing integration. The white paper also describes how to implement MPI, including a discussion of protocols, supported hardware, and topology. Finally, the white paper examines security in an MS-MPI implementation, including issues with Active Directory integration and credential and process management. The white paper also includes an appendix containing a sample MPI program.

in reply to Using Microsoft Message Passing Interface (MS MPI)

Windows HPC Server Development, the MPI Application Model [Windows HPC Server Development, the MPI Application Model]

Wenming Ye — Tue, 17 Nov 2009 17:50:33 GMT

Windows HPC Server 2008 introduces innovative features for traditional High Performance Computing solutions. Significant improvements have been implemented across the entire HPC stack. Learn about the new NetworkDirect RDMA technology and how it enables very fast, high-bandwidth, distributed shared memory computing. MS MPI is now integrated with Event Tracing for Windows easing the task of profiling, analysis, and visualization of HPC solution performance. A new Job Scheduler architecture delivers impressive performance while also enabling intelligent scheduling policies and resource management. Traditional HPC solution developers will appreciate the elegance of Microsoft’s latest platform technologies while those new to the parallel computing problem domain will want to consider Windows HPC based solutions

in reply to Windows HPC Server Development, the MPI Application Model