Table of contents

By almost any measure, the SciDAC community has come a long way since DOE launched the SciDAC program back in 2001. At the time, we were grappling with how to efficiently run applications on terascale systems (the November 2001 TOP500 list was led by DOE's ASCI White IBM system at Lawrence Livermore achieving 7.2 teraflop/s). And the results stemming from the first round of SciDAC projects were summed up in two-page reports. The scientific results were presented at annual meetings, which were by invitation only and typically were attended by about 75 researchers.

Fast forward to 2009 and we now have SciDAC Review, a quarterly magazine showcasing the scientific computing contributions of SciDAC projects and related programs, all focused on presenting a comprehensive look at Scientific Discovery through Advanced Computing. That is also the motivation behind the annual SciDAC conference that in 2009 was held from June 14–18 in San Diego.

The annual conference, which can also be described as a celebration of all things SciDAC, grew out those meetings organized in the early days of the program. In 2005, the meeting was held in San Francisco and attendance was opened up to all members of the SciDAC community. The schedule was also expanded to include a keynote address, plenary speakers and other features found in a conference format.

This year marks the fifth such SciDAC conference, which now comprises four days of computational science presentations, multiple poster sessions and, since last year, an evening event showcasing simulations and modeling runs resulting from SciDAC projects. The fifth annual SciDAC conference was remarkable on several levels. The primary purpose, of course, is to showcase the research accomplishments resulting from SciDAC programs in particular and computational science in general. It is these accomplishments, represented in 38 papers and 52 posters, that comprise this set of conference proceedings. These proceedings can stand alone as evidence of the success of DOE's innovative SciDAC efforts.

But from the outset, a critical driver for the program was to foster increased collaboration among researchers across disciplines and organizations. In particular, SciDAC wanted to engage scientists at universities in the projects, both to expand the community and to develop the next generation of computational scientists. At the meeting in San Diego, the fruits of this emphasis were clearly visible, from the special poster session highlighting the work of the DOE Computational Science Graduate Fellows, to the informal discussions in hotel hallways, to focused side meetings apart from the main presentations. A highlight of the meeting was the keynote address by Dr Ray Orbach, until recently the DOE Under Secretary for Science and head of the Office of Science. It was during his tenure that the first round of projects matured and the second set of SciDAC projects were launched. And complementing these research projects was Dr Orbach's vision for INCITE, DOE's Innovative and Novel Computational Impact on Theory and Experiment program, inaugurated in 2003. This program allocated significant HPC resources to scientists tackling high-impact problems, including some of those addressed by SciDAC teams. Together, SciDAC and INCITE are dramatically accelerating the field of computational science.

As has been noted before, the SciDAC conference celebrates progress in advancing science through large-scale modeling and simulation. Over 400 people registered to attend this year's talks, poster sessions and tutorials, all spanning the disciplines supported by DOE. While the principal focus was on SciDAC accomplishments, this year's conference also included invited presentations and posters from colleagues whose research is supported by other agencies. At the 2009 meeting we also formalized a developing synergy with the Department of Defense's HPC Users Group Meeting, which has occasionally met in parallel with the SciDAC meeting. But in San Diego, we took the additional steps of organizing a joint poster session and a joint plenary session, further advancing opportunities for broader networking. Throughout the four-day program, attendees at both meetings had the option of sitting in on sessions at either conference. We also included several of the NSF Petascale applications in the program, and have also extended invitations to our computational colleagues in other federal agencies, including the National Science Foundation, NASA, and the National Oceanographic and Atmospheric Administration, as well as international collaborators to join us in San Diego.

In 2009 we also reprised one of the more popular sessions from Seattle in 2008, the Electronic Visualization and Poster Night, during which 29 scientific visualizations were presented on high-resolution large-format displays. The best entries were awarded one of the coveted 'OASCR Awards.' The conference also featured a session about breakthroughs in computational science, based on the 'Breakthrough Report' that was published in 2008, led by Tony Mezzacappa (ORNL). Tony was also the chair of the SciDAC 2005 conference.

For the third consecutive year, the conference was followed by a day of tutorials organized by the SciDAC Outreach Center and aimed primarily at students interested in scientific computing. This year, nearly 100 participants attended the tutorials, hosted by the San Diego Supercomputer Center and General Atomics.

This outreach to the broader community is really what SciDAC is all about – Scientific Discovery through Advanced Computing. Such discoveries are not confined by organizational lines, but rather are often the result of researchers reaching out and collaborating with others, using their combined expertise to push our boundaries of knowledge. I am happy to see that this vision is shared by so many researchers in computational science, who all decided to join SciDAC 2009.

While credit for the excellent presentations and posters goes to the teams of researchers, the success of this year's conference is due to the strong efforts and support from members of the 2009 SciDAC Program Committee and Organizing Committee, and I would like to extend my heartfelt thanks to them for helping to make the 2009 meeting the largest and most successful to date.

Program Committee members were: David Bader, LLNL; Pete Beckman, ANL; John Bell, LBNL; John Boisseau, University of Texas; Paul Bonoli, MIT; Hank Childs, LBNL; Bill Collins, LBNL; Jim Davenport, BNL; David Dean, ORNL; Thom Dunning, NCSA; Peg Folta, LLNL; Glenn Hammond, PNNL; Maciej Haranczyk, LBNL; Robert Harrison, ORNL; Paul Hovland, ANL; Paul Kent, ORNL; Aram Kevorkian, SPAWAR; David Keyes, Columbia University; Kwok Ko, SLAC; Felice Lightstone, LLNL; Bob Lucas, ISI/USC; Paul Mackenzie, Fermilab; Tony Mezzacappa, ORNL; John Negele, MIT; Jeff Nichols, ORNL; Mike Norman, UCSD; Joe Oefelein, SNL; Jeanie Osburn, NRL; Peter Ostroumov, ANL; Valerio Pascucci, University of Utah; Ruth Pordes, Fermilab; Rob Ross, ANL; Nagiza Samatova, ORNL; Martin Savage, University of Washington; Tim Scheibe, PNNL; Ed Seidel, NSF; Arie Shoshani, LBNL; Rick Stevens, ANL; Bob Sugar, UCSB; Bill Tang, PPPL; Bob Wilhelmson, NCSA; Kathy Yelick, NERSC/LBNL; Dave Zachmann, Vista Computational Technology LLC.

Organizing Committee members were: Communications: Jon Bashor, LBNL. Contracts/Logistics: Mary Spada and Cheryl Zidel, ANL. Posters: David Bailey, LBNL. Proceedings: John Hules, LBNL. Proceedings Database Developer: Beth Cerny Patino, ANL. Program Committee Liaison/Conference Web Site: Yeen Mankin, LBNL. Tutorials: David Skinner, NERSC/LBNL. Visualization Night: Hank Childs, LBNL; Valerio Pascucci, Chems Touati, Nathan Galli, and Erik Jorgensen, University of Utah.

Again, my thanks to all.

Horst Simon San Diego, California June 18, 2009

Welcome to San Diego and the 2009 SciDAC conference. Over the next four days, I would like to present an assessment of the SciDAC program. We will look at where we've been, how we got to where we are and where we are going in the future.

Our vision is to be first in computational science, to be best in class in modeling and simulation. When Ray Orbach asked me what I would do, in my job interview for the SciDAC Director position, I said we would achieve that vision. And with our collective dedicated efforts, we have managed to achieve this vision.

In the last year, we have now the most powerful supercomputer for open science, Jaguar, the Cray XT system at the Oak Ridge Leadership Computing Facility (OLCF). We also have NERSC, probably the best-in-the-world program for productivity in science that the Office of Science so depends on. And the Argonne Leadership Computing Facility offers architectural diversity with its IBM Blue Gene/P system as a counterbalance to Oak Ridge.

There is also ESnet, which is often understated—the 40 gigabit per second dual backbone ring that connects all the labs and many DOE sites. In the President's Recovery Act funding, there is exciting news that ESnet is going to build out to a 100 gigabit per second network using new optical technologies. This is very exciting news for simulations and large-scale scientific facilities.

But as one noted SciDAC luminary said, it's not all about the computers—it's also about the science—and we are also achieving our vision in this area. Together with having the fastest supercomputer for science, at the SC08 conference, SciDAC researchers won two ACM Gordon Bell Prizes for the outstanding performance of their applications. The DCA++ code, which solves some very interesting problems in materials, achieved a sustained performance of 1.3 petaflops, an astounding result and a mark I suspect will last for some time. The LS3DF application for studying nanomaterials also required the development of a new and novel algorithm to produce results up to 400 times faster than a similar application, and was recognized with a prize for algorithm innovation—a remarkable achievement.

Day one of our conference will include examples of petascale science enabled at the OLCF. Although Jaguar has not been officially commissioned, it has gone through its acceptance tests, and during its shakedown phase there have been pioneer applications used for the acceptance tests, and they are running at scale. These include applications in the areas of astrophysics, biology, chemistry, combustion, fusion, geosciences, materials science, nuclear energy and nuclear physics.

We also have a whole compendium of science we do at our facilities; these have been documented and reviewed at our last SciDAC conference. Many of these were highlighted in our Breakthroughs Report. One session at this week's conference will feature a cross-section of these breakthroughs. In the area of scalable electromagnetic simulations, the Auxiliary-space Maxwell Solver (AMS) uses specialized finite element discretizations and multigrid-based techniques, which decompose the original problem into easier-to-solve subproblems. Congratulations to the mathematicians on this. Another application on the list of breakthroughs was the authentication of PETSc, which provides scalable solvers used in many DOE applications and has solved problems with over 3 billion unknowns and scaled to over 16,000 processors on DOE leadership-class computers. This is becoming a very versatile and useful toolkit to achieve performance at scale.

With the announcement of SIAM's first class of Fellows, we are remarkably well represented. Of the group of 191, more than 40 of these Fellows are in the 'DOE space.' We are so delighted that SIAM has recognized them for their many achievements.

In the coming months, we will illustrate our leadership in applied math and computer science by looking at our contributions in the areas of programming models, development and performance tools, math libraries, system software, collaboration, and visualization and data analytics. This is a large and diverse list of libraries. We have asked for two panels, one chaired by David Keyes and composed of many of the nation's leading mathematicians, to produce a report on the most significant accomplishments in applied mathematics over the last eight years, taking us back to the start of the SciDAC program. In addition, we have a similar panel in computer science to be chaired by Kathy Yelick. They are going to identify the computer science accomplishments of the past eight years. These accomplishments are difficult to get a handle on, and I'm looking forward to this report.

We will also have a follow-on to our report on breakthroughs in computational science and this will also go back eight years, looking at the many accomplishments under the SciDAC and INCITE programs. This will be chaired by Tony Mezzacappa.

So, where are we going in the SciDAC program? It might help to take a look at computational science and how it got started. I go back to Ken Wilson, who made the model and has written on computational science and computational science education. His model was thus: The computational scientist plays the role of the experimentalist, and the math and CS researchers play the role of theorists, and the computers themselves are the experimental apparatus. And that in simulation science, we are carrying out numerical experiments as to the nature of physical and biological sciences.

Peter Lax, in the same time frame, developed a report on large-scale computing in science and engineering. Peter remarked, 'Perhaps the most important applications of scientific computing come not in the solution of old problems, but in the discovery of new phenomena through numerical experimentation.'

And in the early years, I think the person who provided the most guidance, the most innovation and the most vision for where the future might lie was Ed Oliver. Ed Oliver died last year. Ed did a number of things in science. He had this personality where he knew exactly what to do, but he preferred to stay out of the limelight so that others could enjoy the fruits of his vision. We in the SciDAC program and ASCR Facilities are still enjoying the benefits of his vision. We will miss him.

Twenty years after Ken Wilson, Ray Orbach laid out the fundamental premise for SciDAC in an interview that appeared in SciDAC Review:

'SciDAC is unique in the world. There isn't any other program like it anywhere else, and it has the remarkable ability to do science by bringing together physical scientists, mathematicians, applied mathematicians, and computer scientists who recognize that computation is not something you do at the end, but rather it needs to be built into the solution of the very problem that one is addressing. '

As you look at the Lax report from 1982, it talks about how 'Future significant improvements may have to come from architectures embodying parallel processing elements—perhaps several thousands of processors.' And it continues, 'esearch in languages, algorithms and numerical analysis will be crucial in learning to exploit these new architectures fully.'

In the early '90s, Sterling, Messina and Smith developed a workshop report on petascale computing and concluded, 'A petaflops computer system will be feasible in two decades, or less, and rely in part on the continual advancement of the semiconductor industry both in speed enhancement and cost reduction through improved fabrication processes.' So they were not wrong, and today we are embarking on a forward look that is at a different scale, the exascale, going to 1018 flops.

In 2007, Stevens, Simon and Zacharia chaired a series of town hall meetings looking at exascale computing, and in their report wrote,

'Exascale computer systems are expected to be technologically feasible within the next 15 years, or perhaps sooner. These systems will push the envelope in a number of important technologies: processor architecture, scale of multicore integration, power management and packaging.'

The concept of computing on the Jaguar computer involves hundreds of thousands of cores, as do the IBM systems that are currently out there. So the scale of computing with systems with billions of processors is staggering to me, and I don't know how the software and math folks feel about it.

We have now embarked on a road toward extreme scale computing. We have created a series of town hall meetings and we are now in the process of holding workshops that address what I call within the DOE speak 'the mission need,' or what is the scientific justification for computing at that scale. We are going to have a total of 13 workshops. The workshops on climate, high energy physics, nuclear physics, fusion, and nuclear energy have been held. The report from the workshop on climate is actually out and available, and the other reports are being completed. The upcoming workshops are on biology, materials, and chemistry; and workshops that engage science for nuclear security are a partnership between NNSA and ASCR. There are additional workshops on applied math, computer science, and architecture that are needed for computing at the exascale. These extreme scale workshops will provide the foundation in our office, the Office of Science, the NNSA and DOE, and we will engage the National Science Foundation and the Department of Defense as partners.

We envision a 10-year program for an exascale initiative. It will be an integrated R&D program initially—you can think about five years for research and development—that would be in hardware, operating systems, file systems, networking and so on, as well as software for applications. Application software and the operating system and the hardware all need to be bundled in this period so that at the end the system will execute the science applications at scale. We also believe that this process will have to have considerable investment from the manufacturers and vendors to be successful.

We have formed laboratory, university and industry working groups to start this process and formed a panel to look at where SciDAC needs to go to compute at the extreme scale, and we have formed an executive committee within the Office of Science and the NNSA to focus on these activities. We will have outreach to DoD in the next few months. We are anticipating a solicitation within the next two years in which we will compete this bundled R&D process.

We don't know how we will incorporate SciDAC into extreme scale computing, but we do know there will be many challenges. And as we have shown over the years, we have the expertise and determination to surmount these challenges.

A brief history of the Leadership Computing Facility (LCF) initiative is presented, along with the importance of SciDAC to the initiative. The initiative led to the initiation of the Innovative and Novel Computational Impact on Theory and Experiment program (INCITE), open to all researchers in the US and abroad, and based solely on scientific merit through peer review, awarding sizeable allocations (typically millions of processor-hours per project). The development of the nation's LCFs has enabled available INCITE processor-hours to double roughly every eight months since its inception in 2004. The 'top ten' LCF accomplishments in 2009 illustrate the breadth of the scientific program, while the 75 million processor hours allocated to American business since 2006 highlight INCITE contributions to US competitiveness. The extrapolation of INCITE processor hours into the future brings new possibilities for many 'classic' scaling problems. Complex systems and atomic displacements to cracks are but two examples. However, even with increasing computational speeds, the development of theory, numerical representations, algorithms, and efficient implementation are required for substantial success, exhibiting the crucial role that SciDAC will play.

We present a shape optimization method for designing accelerator cavities with large scale computations. The objective is to find the best accelerator cavity shape with the desired spectral response, such as with the specified frequencies of resonant modes, field profiles, and external Q values. The forward problem is the large scale Maxwell equation in the frequency domain. The design parameters are the CAD parameters defining the cavity shape. We develop scalable algorithms with a discrete adjoint approach and use the quasi-Newton method to solve the nonlinear optimization problem. Two realistic accelerator cavity design examples are presented.

Current state-of-the-art beam dynamics simulations include multiple physical effects and multiple physical length and/or time scales. We present recent developments in Synergia2, an accelerator modeling framework designed for multi-physics, multi-scale simulations. We summarize recent several recent results in multi-physics beam dynamics, including simulations of three Fermilab accelerators: the Tevatron, the Main Injector and the Debuncher.

Recent developments have shown that one can get around the difficulties of finding the eigenvalues and eigenmodes of the large systems studied with high performance computation by using broadly filtered diagonalization [G. R. Werner and J. R. Cary, J. Comp. Phys. 227, 5200 (2008)]. This method can be used in conjunction with any time-domain computation, in particular those that scale very well up to 10000s of processors and beyond. Here we present results that show that this method accurately obtains both modes and frequencies of electromagnetic cavities, even when frequencies are nearly degenerate. The application was to a well-characterized Kaon separator cavity, the A15. The computations are shown to have a precision to a few parts in 10⁵. Because the computed frequency differed from the measured frequency by more than this amount, a careful validation study to determine all sources of difference was undertaken. Ultimately, more precise measurements of the cavity showed that the computations were correct, with remaining differences accounted for by uncertainties in cavity dimensions and atmospheric and thermal conditions. Thus, not only was the method validated, but it was shown to have the ability to predict differences in cavity dimensions from fabrication specifications.

SLAC's Advanced Computations Department (ACD) has developed the parallel 3D electromagnetic time-domain code T3P for simulations of wakefields and transients in complex accelerator structures. T3P is based on state-of-the-art Finite Element methods on unstructured grids and features unconditional stability, quadratic surface approximation and up to 6th-order vector basis functions for unprecedented simulation accuracy. Optimized for large-scale parallel processing on leadership supercomputing facilities, T3P allows simulations of realistic 3D structures with fast turn-around times, aiding the design of the next generation of accelerator facilities. Applications include simulations of the proposed two-beam accelerator structures for the Compact Linear Collider (CLIC) – wakefield damping in the Power Extraction and Transfer Structure (PETS) and power transfer to the main beam accelerating structures are investigated.

The concept and designs of plasma-based advanced accelerators for high energy physics and photon science are modelled in the SciDAC COMPASS project with a suite of Particle-In-Cell codes and simulation techniques including the full electromagnetic model, the envelope model, the boosted frame approach and the quasi-static model. In this paper, we report the progress of the development of these models and techniques and present recent results achieved with large-scale parallel PIC simulations. The simulation needs for modelling the plasma-based advanced accelerator at the energy frontier is discussed and a path towards this goal is outlined.

It was shown recently that it may be computationally advantageous to perform computer simulations in a Lorentz boosted frame for a certain class of systems. However, even if the computer model relies on a covariant set of equations, it was pointed out that algorithmic difficulties related to discretization errors may have to be overcome in order to take full advantage of the potential speedup. In this paper, we summarize the findings, the difficulties and their solutions, and review the applications of the technique that have been performed to date.

High-performance computations on Blue Gene/P at Argonne's Leadership Computing Facility have been used to determine phase shifts induced in injected RF diagnostics as a function of electron cloud density in the Main Injector. Inversion of the relationship between electron cloud parameters and induced phase shifts allows us to predict electron cloud density and evolution over many bunch periods. Long time-scale simulations using Blue Gene have allowed us to measure cloud evolution patterns under the influence of beam propagation with realistic physical parameterizations, such as elliptical beam pipe geometry, self-consistent electromagnetic fields, space charge, secondary electron emission, and the application of arbitrary external magnetic fields. Simultaneously, we are able to simulate the use of injected microwave diagnostic signals to measure electron cloud density, and the effectiveness of various mitigation techniques such as surface coating and the application of confining magnetic fields. These simulations provide a baseline for both RF electron cloud diagnostic design and accelerator fabrication in order to measure electron clouds and mitigate the adverse effects of such clouds on beam propagation.

We summarize recent advances in partitioning, load balancing, and matrix ordering for scientific computing performed by members of the CSCAPES SciDAC institute.

Adaptive mesh refinement and coarsening (AMR) is essential for the numerical solution of partial differential equations (PDEs) that exhibit behavior over a wide range of length and time scales. Because of the complex dynamic data structures and communication patterns and frequent data exchange and redistribution, scaling dynamic AMR to tens of thousands of processors has long been considered a challenge. We are developing ALPS, a library for dynamic mesh adaptation of PDEs that is designed to scale to hundreds of thousands of compute cores. Our approach uses parallel forest-of-octree-based hexahedral finite element meshes and dynamic load balancing based on space-filling curves. ALPS supports arbitrary-order accurate continuous and discontinuous finite element/spectral element discretizations on general geometries. We present scalability and performance results for two applications from geophysics: seismic wave propagation and mantle convection.

We are developing a new class of finite-volume methods on locally-refined and mapped grids, which are at least fourth-order accurate in regions where the solution is smooth. This paper discusses the implementation of such methods for time-dependent problems on both Cartesian and mapped grids with adaptive mesh refinement. We show 2D results with the Berger-Colella shock-ramp problem in Cartesian coordinates, and fourth-order accuracy of the solution of a Gaussian pulse problem in a polytropic gas in mapped coordinates.

SciDAC applications have a demonstrated need for advanced software tools to manage the complexities associated with sophisticated geometry, mesh, and field manipulation tasks, particularly as computer architectures move toward the petascale. In this paper, we describe a software component – an abstract data model and programming interface – designed to provide support for parallel unstructured mesh operations. We describe key issues that must be addressed to successfully provide high-performance, distributed-memory unstructured mesh services and highlight some recent research accomplishments in developing new load balancing and MPI-based communication libraries appropriate for leadership class computing. Finally, we give examples of the use of parallel adaptive mesh modification in two SciDAC applications.

There is a growing trend in nuclear reactor simulation to consider multiphysics problems. This can be seen in reactor analysis where analysts are interested in coupled flow, heat transfer and neutronics, and in fuel performance simulation where analysts are interested in thermomechanics with contact coupled to species transport and chemistry. These more ambitious simulations usually motivate some level of parallel computing. Many of the coupling efforts to date utilize simple code coupling or first-order operator splitting, often referred to as loose coupling. While these approaches can produce answers, they usually leave questions of accuracy and stability unanswered. Additionally, the different physics often reside on separate grids which are coupled via simple interpolation, again leaving open questions of stability and accuracy. Utilizing state of the art mathematics and software development techniques we are deploying next generation tools for nuclear engineering applications. The Jacobian-free Newton-Krylov (JFNK) method combined with physics-based preconditioning provide the underlying mathematical structure for our tools. JFNK is understood to be a modern multiphysics algorithm, but we are also utilizing its unique properties as a scale bridging algorithm. To facilitate rapid development of multiphysics applications we have developed the Multiphysics Object-Oriented Simulation Environment (MOOSE). Examples from two MOOSE-based applications: PRONGHORN, our multiphysics gas cooled reactor simulation tool and BISON, our multiphysics, multiscale fuel performance simulation tool will be presented.

In the past two decades, computational methods have emerged as an essential component of the scientific and engineering enterprise. A diverse assortment of scientific applications has been simulated and explored via advanced computational techniques. Computer vendors have built enormous parallel machines to support these activities, and the research community has developed new algorithms and codes, and agreed on standards to facilitate ever more ambitious computations. However, this track record of success will be increasingly hard to sustain in coming years. Power limitations constrain processor clock speeds, so further performance improvements will need to come from ever more parallelism. This higher degree of parallelism will require new thinking about algorithms, programming models, and architectural resilience. Simultaneously, cutting edge science increasingly requires more complex simulations with unstructured and adaptive grids, and multi-scale and multi-physics phenomena. These new codes will push existing parallelization strategies to their limits and beyond. Emerging data-rich scientific applications are also in need of high performance computing, but their complex spatial and temporal data access patterns do not perform well on existing machines. These interacting forces will reshape high performance computing in the coming years.

The mathematical modeling of systems often requires the use of both nonlinear and discrete components. Discrete decision variables model dichotomies, discontinuities, and general logical relationships. Nonlinear functions are required to accurately represent physical properties such as pressure, stress, temperature, and equilibrium. Problems involving both discrete variables and nonlinear constraint functions are known as mixed-integer nonlinear programs (MINLPs) and are among the most challenging computational optimization problems faced by researchers and practitioners. In this paper, we describe relevant scientific applications that are naturally modeled as MINLPs, we provide an overview of available algorithms and software, and we describe ongoing methodological advances for solving MINLPs. These algorithmic advances are making increasingly larger instances of this important family of problems tractable.

Efficient solution of large-scale, ill-conditioned and highly-indefinite algebraic equations often relies on high quality preconditioners together with iterative solvers. Because of their robustness, the factorization-based algorithms could play a significant role when they are combined with iterative methods, particularly in the development of scalable solvers. We present our recent work in using the direct solver SuperLU code base to develop a new supernode-based ILU preconditioner and a domain-decomposition hybrid solver. Our ILU preconditioner is a modification of the classic ILUTP approach, incorporating a number of techniques to improve robustness and performance, which include new dropping strategies that accommodate the use of supernodal structure in the factored matrix. Our hybrid solver is based on the Schur complement method. We use parallel graph partitioning to obtain hierarchical interface/domain decomposition, and multiple parallel direct solvers to solve the subdomain problems simultaneously, and parallel preconditioned iterative solvers to solve the interface problem. We will demonstrate the effectiveness of our new techniques by applying them to two SciDAC applications, modeling next-generation particle accelerators and fusion devices.

This paper presents large-scale computations and theoretical or computational aspects of the spectral element methods for solving Maxwell's equations that have potential applications in nanoscience for surface-enhanced Raman scattering (SERS) and solar cell devices. We study the surface-enhanced electromagnetic fields near the surface of metallic nanoparticles using spectral element discontinuous Galerkin method. We solve Maxwell's equations in time-domain and provide accuracy and efficiency of our method compared to the conventional finite difference method. We demonstrate light transmission properties for nanoslab and nanoslits, and time-averaged electric fields over the cross sections of nanoholes in a hexagonal array.

Reactor simulation depends on the coupled solution of various physics types, including neutronics, thermal/hydraulics, and structural mechanics. This paper describes the formulation and implementation of a parallel solution coupling capability being developed for reactor simulation. The coupling process consists of mesh and coupler initialization, point location, field interpolation, and field normalization. We report here our test of this capability on an example problem, namely, a reflector assembly from an advanced burner test reactor. Performance of this coupler in parallel is reasonable for the chosen problem size and range of processor counts. The runtime is dominated by startup costs, which amortize over the entire coupled simulation. Future efforts will include adding more sophisticated interpolation and normalization methods, to accommodate different numerical solvers used in various physics modules and to obtain better conservation properties for certain field types.

Much progress in realistic modeling of core-collapse supernovae has occurred recently through the availability of multi-teraflop machines and the increasing sophistication of supernova codes. These improvements are enabling simulations with enough realism that the explosion mechanism, long a mystery, may soon be delineated. We briefly describe the CHIMERA code, a supernova code we have developed to simulate core-collapse supernovae in 1, 2, and 3 spatial dimensions. We then describe the results of an ongoing suite of 2D simulations initiated from a 12, 15, 20, and 25 M_⊙ progenitor. These have all exhibited explosions and are currently in the expanding phase with the shock at between 5,000 and 20,000 km. We also briefly describe an ongoing simulation in 3 spatial dimensions initiated from the 15 M_⊙ progenitor.

The target of the Roadrunner Universe project at Los Alamos National Laboratory is a set of very large cosmological N-body simulation runs on the hybrid supercomputer Roadrunner, the world's first petaflop platform. Roadrunner's architecture presents opportunities and difficulties characteristic of next-generation supercomputing. We describe a new code designed to optimize performance and scalability by explicitly matching the underlying algorithms to the machine architecture, and by using the physics of the problem as an essential aid in this process. While applications will differ in specific exploits, we believe that such a design process will become increasingly important in the future. The Roadrunner Universe project code, MC³ (Mesh-based Cosmology Code on the Cell), uses grid and direct particle methods to balance the capabilities of Roadrunner's conventional (Opteron) and accelerator (Cell BE) layers. Mirrored particle caches and spectral techniques are used to overcome communication bandwidth limitations and possible difficulties with complicated particle-grid interaction templates.

We present results of large-scale three-dimensional weakly magnetized supersonic turbulence simulations with an isothermal equation of state at grid resolutions up to 1024³ cells with the Piecewise Parabolic Method on a Local Stencil. The turbulence is driven by a large-scale isotropic solenoidal force in a periodic computational domain and fully develops in a few flow crossing times. We then evolve the flow for a number of flow crossing times and analyze various statistical properties of the saturated turbulent state. We show that the energy transfer rate in the inertial range of scales is surprisingly close to a constant, indicating that Kolmogorov's phenomenology for incompressible turbulence can be extended to magnetized supersonic flows. We also discuss numerical dissipation effects and convergence of different turbulence diagnostics as grid resolution refines from 256³ to 1024³ cells.

We use hydrodynamic cosmological simulations in a (600 Mpc)³ volume to study the observability of baryon acoustic oscillations (BAO) in the intergalactic medium as probed by Lyman alpha forest (LAF) absorption. The large scale separation between the wavelength of the BAO mode (∼150 Mpc) and the size of LAF absorbers (∼100 kpc) makes this a numerically challenging problem. We report on several 2048³ simulations of the LAF using the ENZO code. We adopt WMAP5 concordance cosmological parameters and power spectrum including BAO perturbations. 5000 synthetic HI absorption line spectra are generated randomly piercing the box face. We calculate the cross-correlation function between widely separated pairs. We detect the BAO signal at z=3 where theory predicts to moderate statistical significance.

Core-collapse supernovae are among Nature's most energetic events. They mark the end of massive star evolution and pollute the interstellar medium with the life-enabling ashes of thermonuclear burning. Despite their importance for the evolution of galaxies and life in the universe, the details of the core-collapse supernova explosion mechanism remain in the dark and pose a daunting computational challenge. We outline the multi-dimensional, multi-scale, and multi-physics nature of the core-collapse supernova problem and discuss computational strategies and requirements for its solution. Specifically, we highlight the axisymmetric (2D) radiation-MHD code VULCAN/2D and present results obtained from the first full-2D angle-dependent neutrino radiation-hydrodynamics simulations of the post-core-bounce supernova evolution. We then go on to discuss the new code Zelmani which is based on the open-source HPC Cactus framework and provides a scalable AMR approach for 3D fully general-relativistic modeling of stellar collapse, core-collapse supernovae and black hole formation on current and future massively-parallel HPC systems. We show Zelmani's scaling properties to more than 16,000 compute cores and discuss first 3D general-relativistic core-collapse results.

There are two principal scientific objectives in the study of Type Ia supernovae – first, a better understanding of these complex explosions from as near first principles as possible, and second, enabling the more accurate utilization of their emission to measure distances in cosmology. Both tasks lend themselves to large scale numerical simulation, yet take us beyond the current frontiers in astrophysics, combustion science, and radiation transport. Their study requires novel approaches and the creation of new, highly scalable codes.

It is now easier to discover thousands of protein sequences in a new microbial genome than it is to biochemically characterize the specific activity of a single protein of unknown function. The molecular functions of protein sequences have typically been predicted using homology-based computational methods, which rely on the principle that homologous proteins share a similar function. However, some protein families include groups of proteins with different molecular functions. A phylogenetic approach for predicting molecular function (sometimes called "phylogenomics") is an effective means to predict protein molecular function. These methods incorporate functional evidence from all members of a family that have functional characterizations using the evolutionary history of the protein family to make robust predictions for the uncharacterized proteins. However, they are often difficult to apply on a genome-wide scale because of the time-consuming step of reconstructing the phylogenies of each protein to be annotated. Our automated approach for function annotation using phylogeny, the SIFTER (Statistical Inference of Function Through Evolutionary Relationships) methodology, uses a statistical graphical model to compute the probabilities of molecular functions for unannotated proteins. Our benchmark tests showed that SIFTER provides accurate functional predictions on various protein families, outperforming other available methods.

Over the past decade, genome-scale metabolic models have gained widespread acceptance in biology and bioengineering as a means of quantitatively predicting organism behavior based on the stoichiometry of the biochemical reactions constituting the organism metabolism. The list of applications for these models is rapidly growing; they have been used to identify essential genes, determine growth conditions, predict phenotypes, predict response to mutation, and study the impact of transcriptional regulation on organism phenotypes. This growing field of applications, combined with the rapidly growing number of available genome-scale models, is producing a significant demand for computation to analyze these models. Here we discuss how high-performance computing may be applied with various algorithms for the reconstruction, analysis, and optimization of genome-scale metabolic models. We also performed a case study to demonstrate how the algorithm for simulating gene knockouts scales when run on up to 65,536 processors on Blue Gene/P. In this case study, the knockout of every possible combination of one, two, three, and four genes was simulated in the iBsu1103 genome-scale model of B. subtilis. In 162 minutes, 18,243,776,054 knockouts were simulated on 65,536 processors, revealing 288 essential single knockouts, 78 essential double knockouts, 99 essential triple knockouts, and only 28 essential quadruple knockouts.

This study explores the mechanism of pore creation in cellular membranes by MccE92 bacterial proteins. The results of this study are then compared with the mechanism of alpha-synuclein (aS)-based pore formation in mammalian cells, and its role in Parkinson's disease.

We have run computational chemistry calculations approaching the Petascale level of performance (∼ 0.5 PFlops). We used the Coupled Cluster CCSD(T) module of the computational chemistry code NWChem to evaluate accurate energetics of water clusters on a 1.4 PFlops Cray XT5 computer.

An overview of the parallel algorithms for ab initio molecular dynamics (AIMD) used in the NWChem program package is presented, including recent developments for computing exact exchange. These algorithms make use of a two-dimensional processor geometry proposed by Gygi et al. for use in AIMD algorithms. Using this strategy, a highly scalable algorithm for exact exchange has been developed and incorporated into AIMD. This new algorithm for exact exchange employs an incomplete butterfly to overcome the bottleneck associated with exact exchange term, and it makes judicious use of data replication. Initial testing has shown that this algorithm can scale to over 20,000 CPUs even for a modest size simulation.

Bayesian inference provides a means incorporate activities usually associated with post-model development evaluation into the process of model development. Presented is a review of the factors that make this approach challenging, strategies for making the process practical for model development of complex multi-physics phenomena, and suggestions on areas requiring additional research. An analysis is presented of the strategy that was used to determine the joint probability for six non-linearly related parameters important to clouds and convection within the NCAR Community Atmosphere Model version 3.1.

The General Curvilinear Ocean Model (GCOM) differs significantly from the traditional approach, where the use of Cartesian coordinates forces the model to simulate terrain as a series of steps. GCOM utilizes a full three-dimensional curvilinear transformation, which has been shown to have greater accuracy than similar models and to achieve results more efficiently. The GCOM model has been validated for several types of water bodies, different coastlines and bottom shapes, including the Alarcon Seamount, Southern California Coastal Region, the Valencia Lake in Venezuela, and more recently the Monterey Bay. In this paper, enhancements to the GCOM model and an overview of the computational environment (GCOM-CE) are presented. Model improvements include migration from F77 to F90; approach to a component design; and initial steps towards parallelization of the model. Through the use of the component design, new models are being incorporated including biogeochemical, pollution, and sediment transport. The computational environment is designed to allow various client interactions via secure Web applications (portal, Web services, and Web 2.0 gadgets). Features include building jobs, managing and interacting with long running jobs; managing input and output files; quick visualization of results; publishing of Web services to be used by other systems such as larger climate models. The CE is based mainly on Python tools including a grid-enabled Pylons Web application Framework for Web services, pyWSRF (python-Web Services-Resource Framework), pyGlobus based web services, SciPy, and Google code tools.

One strategy for reducing US dependence on petroleum is to develop new combustion technologies for burning the fuel-lean mixtures of hydrogen or hydrogen-rich syngas fuels obtained from the gasification of coal and biomass. Fuel-flexible combustion systems based on lean premixed combustion have the potential for dramatically reducing pollutant emissions in transportation systems, heat and stationary power generation. However, lean premixed flames are highly susceptible to fluid-dynamical combustion instabilities making robust and reliable systems difficult to design. Low swirl burners are emerging as an important technology for meeting design requirements in terms of both reliability and emissions for next generation combustion devices. In this paper, we present simulations of a lean, premixed hydrogen flame stabilized on a laboratory-scale low swirl burner. The simulations use detailed chemistry and transport without incorporating explicit models for turbulence or turbulence/chemistry interaction. Here we discuss the overall structure of the flame and compare with experimental data. We also use the simulation data to elucidate the characteristics of the turbulent flame interaction and how this impacts the analysis of experimental measurements.

Many applications in engineering and physical sciences involve turbulent flows interacting with shock waves. High-speed flows around aerodynamic bodies and through propulsion systems for high-speed flight abound with interactions of shear driven turbulence with complex shock waves. Supernova explosions and implosion of a cryogenic fuel pellet for inertial confinement fusion also involve the interaction of shockwaves with turbulence and strong density variations. Numerical simulations of such physical phenomena impose conflicting demands on the numerical algorithms. Capturing broadband spatial and temporal variations in a turbulent flow suggests the use of high-bandwidth schemes with minimal dissipation and dispersion, while capturing the flow discontinuity at a shock wave requires numerical dissipation.

Results from three promising shock-capturing schemes a) high order WENO, b) nonlinear artificial diffusivity with compact finite differences, and c) a hybrid approach combining high-order central differencing with WENO near the shocks are compared using the Taylor-Green problem and compressible isotropic turbulence with eddy-shocklets. The performance of each scheme is characterized in terms of an effective bandwidth. The comparison highlights the damaging effect of numerical dissipation when the WENO scheme is applied everywhere. The hybrid approach is found to be best suited for studying shock-turbulence interactions.

Results from previous DNS and LES studies of the canonical shock-turbulence interaction problem, i.e. the interaction of isotropic turbulence with a (nominally) normal shock and comparison with available theory and experimental data are recalled. The principal physical effects include the amplification of turbulent kinetic energy across the shock and its anisotropy, change in turbulence length scales across the shock, departure from the common assumption of strong Reynolds analogy (used in modeling turbulence in high-speed flows), and the distortion of the shock due to its interaction with the incident turbulence. In this context new results from our SciDAC sponsored project are shown and open issues are mentioned. New DNS results achieve a significantly larger turbulence Reynolds number and allow an exploration of the nonlinear effects. While the linear interaction theory of Ribner provides useful estimates of the amplification of vorticity fluctuations across the shock, it misses the strong nonlinear dynamics of the energized and highly anisotropic vorticity downstream of the shock. It is found that previous DNS studies also underestimated this effect. The simulations show that turbulent self-stretching and tilting mechanisms bring about a relatively rapid return to isotropy in the turbulent vorticity field. The turbulent velocity field, however, does not show any appreciable tendency towards isotropy. It is further observed that when the turbulence interacting with the shock is sufficiently energetic the instantaneous shock structure is significantly modified; local regions of significant over-compression are found as well as regions where the mean shock compression is nearly isentropic. Estimates for the computational resources necessary for studying this fundamental shock-turbulence interaction problem at higher Reynolds number on peta-scale computing systems are given.

There is an urgent and growing demand for high-fidelity simulations that capture complex turbulence-chemistry interactions in propulsion and power systems, and in particular, that capture and discriminate the effects of fuel variability. This project addresses this demand using the Large Eddy Simulation (LES) technique (led by Oefelein) and the Direct Numerical Simulation (DNS) technique (led by Chen). In particular, we are conducting research under the INCITE program that is tightly coupled with funded projects established under the DOE Basic Energy Sciences and Energy Efficiency and Renewable Energy programs that will provide the foundational science required to develop a predictive modeling capability for design of advanced engines for transportation. Application of LES provides the formal ability to treat the full range of multidimensional time and length scales that exist in turbulent reacting flows in a computationally feasible manner and thus provides a way to simulate reacting flow phenomena in complex internal-combustion engine geometries at device relevant conditions. Application of DNS provides a way to study fundamental issues related to small-scale combustion processes in canonical configurations to understand dynamics that occur over a range of reactive-diffusive scales. Here we describe the challenges and present representative examples of the types of simulations each respective tool has been used for as part of the INCITE program. We focus on recent experiences on the Oak Ridge National Laboratory (ORNL) National Center for Computational Sciences (NCCS) Cray-XT Platform (i.e., Jaguar).

Porting of the legacy code MFIX to a high performance computer (HPC) and the use of high resolution simulations for the design of a coal gasifier are described here. MFIX is based on a continuum multiphase flow model that considers gas and solids to form interpenetrating continua. Low resolution simulations of a commercial scale gasifier with a validated MFIX model revealed interesting physical phenomena with implications on the gasifier design, which prompted the study reported here. To be predictive, the simulations need to model the spatiotemporal variations in gas and solids volume fractions, velocities, temperatures with any associated phase change and chemical reactions. These processes occur at various time- and length-scales requiring very high spatial resolution and large number of iterations with small time-steps. We were able to perform perhaps the largest known simulations of gas-solids reacting flows, providing detailed information about the gas-solids flow structure and the pressure, temperature and species distribution in the gasifier. One key finding is the new features of the coal jet trajectory revealed with the high spatial resolution, which provides information on the accuracy of the lower resolution simulations. Methodologies for effectively combining high and low resolution simulations for design studies must be developed. From a computational science perspective, we found that global communication has to be reduced to achieve scalability to 1000s of cores, hybrid parallelization is required to effectively utilize the multicore chips, and the wait time in the batch queue significantly increases the actual time-to-solution. From our experience, development is required in the following areas: efficient solvers for heterogeneous, massively parallel systems; data analysis tools to extract information from large data sets; and programming environments for easily porting legacy codes to HPC.

Most of the energy produced worldwide comes from the combustion of fossil fuels. In the context of global climate changes and dramatically decreasing resources, there is a critical need for optimizing the process of burning, especially in the field of gas turbines. Unfortunately, new designs for efficient combustion are prone to destructive thermo-acoustic instabilities. Large Eddy Simulation (LES) is a promising tool to predict turbulent reacting flows in complex industrial configurations and explore the mechanisms triggering the coupling between acoustics and combustion. In the particular field of annular combustion chambers, these instabilities usually take the form of azimuthal modes. To predict these modes, one must compute the full combustion chamber comprising all sectors, which remained out of reach until very recently and the development of massively parallel computers. A fully compressible, multi-species reactive Navier-Stokes solver is used on up to 4096 BlueGene/P CPUs for two designs of a full annular helicopter chamber. Results show evidence of self-established azimuthal modes for the two cases but with different energy containing limit-cycles. Mesh dependency is checked with grids comprising 38 and 93 million tetrahedra. The fact that the two grid predictions yield similar flow topologies and limit-cycles enforces the ability of LES to discriminate design changes.

Particle-in-cell (PIC) methods have proven to be effective in discretizing the Vlasov-Maxwell system of equations describing the core of toroidal burning plasmas for many decades. Recent physical understanding of the importance of edge physics for stability and transport in tokamaks has lead to development of the first fully toroidal edge PIC code – XGC1. The edge region poses special problems in meshing for PIC methods due to the lack of closed flux surfaces, which makes field-line following meshes and coordinate systems problematic. We present a solution to this problem with a semi-field line following mesh method in a cylindrical coordinate system. Additionally, modern supercomputers require highly concurrent algorithms and implementations, with all levels of the memory hierarchy being efficiently utilized to realize optimal code performance. This paper presents a mesh and particle partitioning method, suitable to our meshing strategy, for use on highly concurrent cache-based computing platforms.

The emergence and continuing use of multi-core architectures and graphics processing units require changes in the existing software and sometimes even a redesign of the established algorithms in order to take advantage of now prevailing parallelism. Parallel Linear Algebra for Scalable Multi-core Architectures (PLASMA) and Matrix Algebra on GPU and Multics Architectures (MAGMA) are two projects that aims to achieve high performance and portability across a wide range of multi-core architectures and hybrid systems respectively. We present in this document a comparative study of PLASMA's performance against established linear algebra packages and some preliminary results of MAGMA on hybrid multi-core and GPU systems.

Coordinated by the Partnership for Advanced Computing in Europe (PRACE) Europe is restructuring and strengthening its high-performance computing infrastructure with the aim to create a model HPC ecosystem. At the tip of the pyramid, up to six centres are envisaged that will operate systems of the highest performance class. The HPC Research Infrastructure (HPC-RI) will comprise European, national and regional centres. Science communities are integral partners, strong links will include Grid and Cloud users. The HPC-RI strives at providing scientists all over Europe, on the one hand, with unlimited and independent access to state-of-the-art computer resources in all performance classes and, on the other hand, with a world-class pan-European competence and support network.

While the hardware-oriented buildup of the infrastructure is making progress, high-quality user support and software development in the upcoming era of unprecedented parallelism and exascale on the horizon have become the imminent challenges. This has been clearly recognized by the European Commission, who will issue calls for proposals to fund petascale software development in summer 2009. Although traditional support structures are well established in Europe's major supercomputing centres, it is questionable if these structures are able to meet the challenges of the future: in general, support structures are based on cross-disciplinary computer science and mathematics teams; disciplinary computational science support usually is given in an ad-hoc, project-oriented manner.

In this paper, we describe our approach to establish a suitable support structure–Simulation Laboratories (SL). SLs are currently being established at the Jülich Supercomputing Centre of the Forschungszentrum Jülich (FZJ) and at the Steinbuch Centre for Computing (SCC) of the Karlsruhe Institute for Technology (KIT) in Germany. While SLs are community-oriented, i.e. each SL focusses on a specific community, they are structured in a strictly interdisciplinary manner, comprising mathematicians, computer scientists and technicians along with disciplinary scientists. SLs are led by a disciplinary scientist, and representatives of the respective disciplines give guidance to its operation. This concept is proposed as a model for and might become an integral element of a future pan-European HPC support and software research structure.

The Performance Engineering Institute (PERI) originally proposed a tiger team activity as a mechanism to target significant effort optimizing key Office of Science applications, a model that was successfully realized with the assistance of two JOULE metric teams. However, the Office of Science requested a new focus beginning in 2008: assistance in forming its ten year facilities plan. To meet this request, PERI formed the Architecture Tiger Team, which is modeling the performance of key science applications on future architectures, with S3D, FLASH and GTC chosen as the first application targets. In this activity, we have measured the performance of these applications on current systems in order to understand their baseline performance and to ensure that our modeling activity focuses on the right versions and inputs of the applications. We have applied a variety of modeling techniques to anticipate the performance of these applications on a range of anticipated systems. While our initial findings predict that Office of Science applications will continue to perform well on future machines from major hardware vendors, we have also encountered several areas in which we must extend our modeling techniques in order to fulfill our mission accurately and completely. In addition, we anticipate that models of a wider range of applications will reveal critical differences between expected future systems, thus providing guidance for future Office of Science procurement decisions, and will enable DOE applications to exploit machines in future facilities fully.

Data movement, both within the memory system of a single processor node and between multiple nodes in a system, limits the performance of many Krylov subspace methods that solve sparse linear systems and eigenvalue problems. Here, s iterations of algorithms such as CG, GMRES, Lanczos, and Arnoldi perform s sparse matrix-vector multiplications and Ω(s) vector reductions, resulting in a growth of Ω(s) in both single-node and network communication. By reorganizing the sparse matrix kernel to compute a set of matrix-vector products at once and reorganizing the rest of the algorithm accordingly, we can perform s iterations by sending O(log P) messages instead of Ω(s·log P) messages on a parallel machine, and reading the on-node components of the matrix A from DRAM to cache just once on a single node instead of s times. This reduces communication to the minimum possible. We discuss both algorithms and an implementation of GMRES on a single node of an 8-core Intel Clovertown. Our implementations achieve significant speedups over the conventional algorithms.

The number of cores in high-end systems for scientific computing are employingis increasing rapidly. As a result, there is an pressing need for tools that can measure, model, and diagnose performance problems in highly-parallel runs. We describe two tools that employ complementary approaches for analysis at scale and we illustrate their use on DOE leadership-class systems.

We present LAPACKrc, a family of FPGA-based linear algebra solvers able to achieve more than 100x speedup per commodity processor on certain problems. LAPACKrc subsumes some of the LAPACK and ScaLAPACK functionalities, and it also incorporates sparse direct and iterative matrix solvers. Current LAPACKrc prototypes demonstrate between 40x-150x speedup compared against top-of-the-line hardware/software systems. A technology roadmap is in place to validate current performance of LAPACKrc in HPC applications, and to increase the computational throughput by factors of hundreds within the next few years.

Acceleration technologies, in particular GPUs and Cell, are receiving considerable attention in modern-day HPC. Compared to classic accelerators and traditional CPUs, these devices not only exhibit higher compute density, but also sport significant memory bandwidth and vector-like capabilities to stream data at bandwidth of 100 GB/s or more. The latter qualifies such accelerators as a rebirth of vector computing. With large-scale deployments of GPUs such as Tokyo Tech's TSUBAME 1.2 supercomputer facilitating 680 GPUs in a 100-Teraflops scale supercomputer, we can demonstrate that, even under a massively parallel setting, GPUs can scale both in dense linear algebra codes as well as vector-oriented CFD codes. In both cases, however, careful algorithmic developments, especially latency hiding, are important to maximize their performance.

This paper describes an application driven methodology for understanding the impact of future architecture decisions on the end of the MPP era. Fundamental transistor device limitations combined with application performance characteristics have created the switch to multicore/multithreaded architectures. Designing large-scale supercomputers to match application demands is particularly challenging since performance characteristics are highly counter-intuitive. In fact, data movement more than FLOPS dominates. This work discusses some basic performance analysis for a set of DOE applications, the limits of CMOS technology, and the impact of both on future architectures.

Computer systems anticipated in the 2015 – 2020 timeframe are referred to as Extreme Scale because they will be built using massive multi-core processors with 100's of cores per chip. The largest capability Extreme Scale system is expected to deliver Exascale performance of the order of 10¹⁸ operations per second. These systems pose new critical challenges for software in the areas of concurrency, energy efficiency and resiliency. In this paper, we discuss the implications of the concurrency and energy efficiency challenges on future software for Extreme Scale Systems. From an application viewpoint, the concurrency and energy challenges boil down to the ability to express and manage parallelism and locality by exploring a range of strong scaling and new-era weak scaling techniques. For expressing parallelism and locality, the key challenges are the ability to expose all of the intrinsic parallelism and locality in a programming model, while ensuring that this expression of parallelism and locality is portable across a range of systems. For managing parallelism and locality, the OS-related challenges include parallel scalability, spatial partitioning of OS and application functionality, direct hardware access for inter-processor communication, and asynchronous rather than interrupt-driven events, which are accompanied by runtime system challenges for scheduling, synchronization, memory management, communication, performance monitoring, and power management. We conclude by discussing the importance of software-hardware co-design in addressing the fundamental challenges for application enablement on Extreme Scale systems.

Parallel scripting is a loosely-coupled programming model in which applications are composed of highly parallel scripts of program invocations that process and exchange data via files. We characterize here the applications that can benefit from parallel scripting on petascale-class machines, describe the mechanisms that make this feasible on such systems, and present results achieved with parallel scripts on currently available petascale computers.

Hadoop is an open-source data processing framework that includes a scalable, fault-tolerant distributed file system, HDFS. Although HDFS was designed to work in conjunction with Hadoop's job scheduler, we have re-purposed it to serve as a grid storage element by adding GridFTP and SRM servers. We have tested the system thoroughly in order to understand its scalability and fault tolerance. The turn-on of the Large Hadron Collider (LHC) in 2009 poses a significant data management and storage challenge; we have been working to introduce HDFS as a solution for data storage for one LHC experiment, the Compact Muon Solenoid (CMS).

High-performance computing systems have already approached peta-scale with hundreds of thousands of processors/cores in many deployments. These systems promise a new level of predictive and knowledge discovery ability as researchers gain the capability to model dependencies between phenomena at scales not seen earlier. These applications are highly I/O and data intensive, leading scientists to observe that performing I/O and subsequent analyses are major bottlenecks in effectively utilizing peta-scale systems and a major hurdle in accelerating discoveries. Although significant progress has been made in performance, interfaces, and middleware runtime systems for I/O in the recent past, significantly more research and development needs to be carried out to scale the performance to the desired levels for systems containing tens to hundreds of thousands of cores. In this work we outline our recent achievements and current research for designing scalable I/O software and enabling data analytics in storage systems. We also enumerate key challenges for the I/O systems and discuss ongoing efforts that address these challenges.

The primary mission of user facilities operated by Basic Energy Sciences under the Department of Energy is to produce data for users in support of open science and basic research [1]. We trace back almost 30 years of history across selected user facilities illustrating the evolution of facility data management practices and how these practices have related to performing scientific research. The facilities cover multiple techniques such as X-ray and neutron scattering, imaging and tomography sciences. Over time, detector and data acquisition technologies have dramatically increased the ability to produce prolific volumes of data challenging the traditional paradigm of users taking data home upon completion of their experiments to process and publish their results. During this time, computing capacity has also increased dramatically, though the size of the data has grown significantly faster than the capacity of one's laptop to manage and process this new facility produced data. Trends indicate that this will continue to be the case for yet some time. Thus users face a quandary for how to manage today's data complexity and size as these may exceed the computing resources users have available to themselves. This same quandary can also stifle collaboration and sharing. Realizing this, some facilities are already providing web portal access to data and computing thereby providing users access to resources they need [2]. Portal based computing is now driving researchers to think about how to use the data collected at multiple facilities in an integrated way to perform their research, and also how to collaborate and share data. In the future, inter-facility data management systems will enable next tier cross-instrument-cross facility scientific research fuelled by smart applications residing upon user computer resources. We can learn from the medical imaging community that has been working since the early 1990's to integrate data from across multiple modalities to achieve better diagnoses [3] – similarly, data fusion across BES facilities will lead to new scientific discoveries.

Parallel file systems are gaining in popularity in high-end computing centers as well as commercial data centers. High-end computing systems are expected to scale exponentially and to pose new challenges to their storage scalability in terms of cost and power. To address these challenges scientists and file system designers will need a thorough understanding of the design space of parallel file systems. Yet there exist few systematic studies of parallel file system behavior at petabyte- and exabyte scale. An important reason is the significant cost of getting access to large-scale hardware to test parallel file systems. To contribute to this understanding we are building a parallel file system simulator that can simulate parallel file systems at very large scale. Our goal is to simulate petabyte-scale parallel file systems on a small cluster or even a single machine in reasonable time and fidelity. With this simulator, file system experts will be able to tune existing file systems for specific workloads, scientists and file system deployment engineers will be able to better communicate workload requirements, file system designers and researchers will be able to try out design alternatives and innovations at scale, and instructors will be able to study very large-scale parallel file system behavior in the class room. In this paper we describe our approach and provide preliminary results that are encouraging both in terms of fidelity and simulation scalability.

Utility computing, elastic computing, and cloud computing are all terms that refer to the concept of dynamically provisioning processing time and storage space from a ubiquitous "cloud" of computational resources. Such systems allow users to acquire and release the resources on demand and provide ready access to data from processing elements, while relegating the physical location and exact parameters of the resources. Over the past few years, such systems have become increasingly popular, but nearly all current cloud computing offerings are either proprietary or depend upon software infrastructure that is invisible to the research community. In this work, we present Eucalyptus, an open-source software implementation of cloud computing that utilizes compute resources that are typically available to researchers, such as clusters and workstation farms. In order to foster community research exploration of cloud computing systems, the design of Eucalyptus emphasizes modularity, allowing researchers to experiment with their own security, scalability, scheduling, and interface implementations. In this paper, we outline the design of Eucalyptus, describe our own implementations of the modular system components, and provide results from experiments that measure performance and scalability of a Eucalyptus installation currently deployed for public use. The main contribution of our work is the presentation of the first research-oriented open-source cloud computing system focused on enabling methodical investigations into the programming, administration, and deployment of systems exploring this novel distributed computing model.

We describe how we integrate, build, and test the Open Science Grid (OSG) software stack, which is used to provide a production quality infrastructure for grid sites and users across OSG to run their grid jobs. The software stack is sufficiently complex that building and testing the software stack is non-trivial and requires a variety of types of testing including internal integration testing as well as end-user testing.

As scientific instruments and computer simulations produce more and more data, the task of locating the essential information to gain insight becomes increasingly difficult. FastBit is an efficient software tool to address this challenge. In this article, we present a summary of the key underlying technologies, namely bitmap compression, encoding, and binning. Together these techniques enable FastBit to answer structured (SQL) queries orders of magnitude faster than popular database systems. To illustrate how FastBit is used in applications, we present three examples involving a high-energy physics experiment, a combustion simulation, and an accelerator simulation. In each case, FastBit significantly reduces the response time and enables interactive exploration on terabytes of data.

The Integrated Plasma Simulator (IPS) provides a framework within which some of the most advanced, massively-parallel fusion modeling codes can be interoperated to provide a detailed picture of the multi-physics processes involved in fusion experiments. The presentation will cover four topics: 1) recent improvements to the IPS, 2) application of the IPS for very high resolution simulations of ITER scenarios, 3) studies of resistive and ideal MHD stability in tokamk discharges using IPS facilities, and 4) the application of RF power in the electron cyclotron range of frequencies to control slowly growing MHD modes in tokamaks and initial evaluations of optimized location for RF power deposition.

VPIC [1], a first-principles 3d electromagnetic charge-conserving relativistic kinetic particle-in-cell code, was recently adapted to run on Los Alamos's Roadrunner [2], the first supercomputer to break a petaflop (10¹⁵ floating point operations per second) in the TOP500 supercomputer performance rankings. [3] We summarize VPIC's modeling capabilities, VPIC's optimization techniques and Roadrunner's computational characteristics. We then discuss three applications enabled by VPIC's unprecedented performance on Roadrunner: modeling laser plasma interaction in upcoming inertial confinement fusion experiments at the National Ignition Facility, modeling short-pulse laser GeV ion acceleration and modeling reconnection in space and laboratory plasmas.

FACETS (Framework Application for Core-Edge Transport Simulations), is now in its third year. The FACETS team has developed a framework for concurrent coupling of parallel computational physics for use on Leadership Class Facilities (LCFs). In the course of the last year, FACETS has tackled many of the difficult problems of moving to parallel, integrated modeling by developing algorithms for coupled systems, extracting legacy applications as components, modifying them to run on LCFs, and improving the performance of all components. The development of FACETS abides by rigorous engineering standards, including cross platform build and test systems, with the latter covering regression, performance, and visualization. In addition, FACETS has demonstrated the ability to incorporate full turbulence computations for the highest fidelity transport computations. Early indications are that the framework, using such computations, scales to multiple tens of thousands of processors. These accomplishments were a result of an interdisciplinary collaboration among computational physics, computer scientists and applied mathematicians on the team.

Performance prediction for ITER is based upon the ubiquitous experimental observation that the plasma energy confinement in the device core is strongly coupled to the edge confinement for an unknown reason. The coupling time-scale is much shorter than the plasma transport time-scale. In order to understand this critical observation, a multi-scale turbulence-neoclassical simulation of integrated edge-core plasma in a realistic diverted geometry is a necessity, but has been a formidable task. Thanks to the recent development in high performance computing, we have succeeded in the integrated multiscale gyrokinetic simulation of the ion-temperature-gradient driven turbulence in realistic diverted tokamak geometry for the first time. It is found that modification of the self-organized criticality in the core plasma by nonlocal core-edge coupling of ITG turbulence can be responsible for the core-edge confinement coupling.

The rf-SciDAC collaboration is developing computer simulations to predict the damping of radio frequency (rf) waves in fusion plasmas. Here we extend self-consistent quasi-linear calculations of ion cyclotron resonant heating to include the finite drift of ions from magnetic flux surfaces and rf induced spatial transport. The all-orders spectral wave solver AORSA is iteratively coupled with a particle based update of the plasma distribution function.

Electron transport in burning plasmas is more important since fusion products first heat electrons. First-principles simulations of electron turbulence are much more challenging due to the multi-scale dynamics of the electron turbulence, and have been made possible by close collaborations between plasma physicists and computational scientists. The GTC simulations of collisionless trapped electron mode (CTEM) turbulence show that the electron heat transport exhibits a gradual transition from Bohm to gyroBohm scaling when the device size is increased. The deviation from the gyroBohm scaling can be induced by large turbulence eddies, turbulence spreading, and non-diffusive transport processes. Analysis of radial correlation function shows that CTEM turbulence eddies are predominantly microscopic but with a significant tail in the mesoscale. A comprehensive analysis of kinetic and fluid time scales shows that zonal flow shearing is the dominant decorrelation mechanism. The mesoscale eddies result from a dynamical process of linear streamers breaking by zonal flows and merging of microscopic eddies. The radial profile of the electron heat conductivity only follows the profile of fluctuation intensity on a global scale, whereas the ion transport tracks more sensitively the local fluctuation intensity. This suggests the existence of a nondiffusive component in the electron heat flux, which arises from the ballistic radial E X B drift of trapped electrons due to a combination of the presence of mesoscale eddies and the weak de-tuning of the toroidal precessional resonance that drives the CTEM instability. On the other hand, the ion radial excursion is not affected by the mesoscale eddies due to a parallel decorrelation, which is not operational for the trapped electrons because of a bounce averaging process associated with the electron fast motion along magnetic field lines. The presence of the nondiffusive component raises question on the applicability of the usual quasilinear theory for the CTEM electron transport. This is in contrast to the good agreement between the quasilinear transport theory and simulation results of the electron heat transport in electron temperature gradient (ETG) turbulence, which is regulated by a wave-particle decorrelation. Therefore, the transport in the CTEM turbulence is a fluid-like eddy mixing process even though the linear CTEM instability is driven by a kinetic resonance. In contrast, a kinetic process dominates the transport in the ETG turbulence, which is characterized by macroscopic streamers.

The confinement and stability properties of the edge of a magnetically confined fusion plasma have long resisted theoretical explanation. Numerical simulation with the M3D extended MHD code shows that large periodic Edge Localized Modes (ELMs), associated with otherwise good confinement, represent a new type of nonlinear MHD plasma instability, in which a coherent plasma instability couples to the favorable part of a chaotic magnetic field. The magnetic surface bounding the plasma typically contains at least one X-point. Under small perturbations, such a Hamiltonian surface splits into two different asymptotic limits. A ballooning-type plasma instability can couple to the "unstable" field manifold that forms bulges around the original magnetic surface and then grow to penetrate deep into the plasma. The magnetic field becomes chaotic over the affected region, allowing plasma loss from the core to the outside. Eventually the instability saturates and the plasma relaxes back towards its original axisymmetric shape. The interaction between the plasma and the homoclinic field perturbation drives a multi-stage process that reproduces many experimental observations. This type of instability may help to explain the wide range of ELM and ELM-free behavior observed in fusion plasmas with X-points, as well as properties of plasma confinement.

Adaptive mesh refinement and coarsening (AMR) is essential for the numerical solution of partial differential equations (PDEs) that exhibit behavior over a wide range of length and time scales. Because of the complex dynamic data structures and communication patterns and frequent data exchange and redistribution, scaling dynamic AMR to tens of thousands of processors has long been considered a challenge. We are developing ALPS, a library for dynamic mesh adaptation of PDEs that is designed to scale to hundreds of thousands of compute cores. Our approach uses parallel forest-of-octree-based hexahedral finite element meshes and dynamic load balancing based on space-filling curves. ALPS supports arbitrary-order accurate continuous and discontinuous finite element/spectral element discretizations on general geometries. We present scalability and performance results for two applications from geophysics: seismic wave propagation and mantle convection.

Earthquakes occurring around the world are responsible for extensive loss of life and infrastructure damage. On average, 1100 earthquakes with significant damage potential occur world-wide per year, and a major societal challenge is to design a human environment that contains appropriate earthquake resistance. Design of critical infrastructure such as large buildings, bridges, industrial facilities and nuclear power plants in seismically active regions is a significant scientific and engineering challenge that encompasses the multiple disciplines of geophysics, geotechnical and structural engineering. Because of the great complexities in earthquake physical processes, traditional approaches to seismic hazard assessment have relied heavily on historical earthquake observations. In this approach, observational data from many locations is homogenized into an empirical assessment of earthquake hazard at any specific site of interest. With major advancements in high performance computing platforms and algorithms, it is now possible to utilize physics-based predictive models to gain enhanced insight about site-specific earthquake ground motions and infrastructure response. This paper discusses recent advancements in geophysics and infrastructure simulations and future challenges in implementing advanced simulations for both earthquake hazard (future ground motions) and earthquake risk (infrastructure response and damage) assessments.

We describe our experiences running PFLOTRAN–a code for simulation of coupled hydro-thermal-chemical processes in variably saturated, non-isothermal, porous media– on leadership-class supercomputers, including initial experiences running on the petaflop incarnation of Jaguar, the Cray XT5 at the National Center for Computational Sciences at Oak Ridge National Laboratory. PFLOTRAN utilizes fully implicit time-stepping and is built on top of the Portable, Extensible Toolkit for Scientific Computation (PETSc). We discuss some of the hurdles to "at scale" performance with PFLOTRAN and the progress we have made in overcoming them on leadership-class computer architectures.

Three-dimensional (3D) geophysical imaging is now receiving considerable attention for electrical conductivity mapping of potential offshore oil and gas reservoirs. The imaging technology employs controlled source electromagnetic (CSEM) and magnetotelluric (MT) fields and treats geological media exhibiting transverse anisotropy. Moreover when combined with established seismic methods, direct imaging of reservoir fluids is possible. Because of the size of the 3D conductivity imaging problem, strategies are required exploiting computational parallelism and optimal meshing. The algorithm thus developed has been shown to scale to tens of thousands of processors. In one imaging experiment, 32,768 tasks/processors on the IBM Watson Research Blue Gene/L supercomputer were successfully utilized. Over a 24 hour period we were able to image a large scale field data set that previously required over four months of processing time on distributed clusters based on Intel or AMD processors utilizing 1024 tasks on an InfiniBand fabric. Electrical conductivity imaging using massively parallel computational resources produces results that cannot be obtained otherwise and are consistent with timeframes required for practical exploration problems.

This paper will provide a brief overview of two frameworks being developed for subsurface simulations. The first is based on the Smoothed Particle Hydrodynamics algorithm and is designed to model flow and transport at the pore scale. The second is based on the Subsurface Transport Over Multiple Phases subsurface continuum code (STOMP) and is designed to simulate flow at the Darcy scale. Both frameworks have been built using the Common Component Architecture toolkit. These frameworks will eventually be combined into a single framework for performing hybrid multiscale simulations that seamlessly integrate both the particle and continuum simulations together.

The Support Architecture for Large-Scale Subsurface Analysis (SALSSA) provides an extensible framework, sophisticated graphical user interface (GUI), and underlying data management system that simplifies the process of running subsurface models, tracking provenance information, and analyzing the model results. The SALSSA software framework is currently being applied to validating the Smoothed Particle Hydrodynamics (SPH) model. SPH is a three-dimensional model of flow and transport in porous media at the pore scale. Because fluid flow in porous media at velocities common in natural porous media occur at low Reynolds numbers, it is important to verify that the SPH model is producing accurate flow solutions in this regime. Validating SPH requires performing a series of simulations and comparing these simulation flow solutions to analytical results or numerical results using other methods. This validation study has been greatly aided by the application of the SALSSA framework, which provides capabilities to setup, execute, analyze, and administer these SPH simulations.

We discuss Quantum Chromodynamics calculations using the lattice regulator. The theory of the strong force is a cornerstone of the Standard Model of particle physics. We present USQCD collaboration results obtained on Argonne National Lab's Intrepid supercomputer that deepen our understanding of these fundamental theories of Nature and provide critical support to frontier particle physics experiments and phenomenology.

The calculation of the spectrum of QCD is key to an understanding of the strong interactions, and vital if we are to capitalize on the experimental study of the spectrum. In this paper, we describe progress towards understanding the spectrum of resonances of both mesons and baryons from lattice QCD, focusing in particular on the resonances of the I = 1/2 nucleon states, and of charmonium mesons composed of the heavy charmed quarks.

In this paper we present a project of visualizing topological charge in Lattice QCD in order to understand its evolution under the Markov Chain Monte Carlo algorithms and how to maximize its equilibration. Lattice QCD is a very computationally expensive technology for computing properties of composite particles from a few fundamental parameters such as quark masses. Our research plays an important role in validating these computations by showing that the autocorrelation length is small and it can eventually lead to even more efficient algorithms.

Lattice QCD currently provides our only means of solving QCD (Quantum Chromo Dynamics) – the theory of the strong nuclear force – in the low-energy regime, and thus of crucial importance for theoretical and experimental research programs in High Energy and Nuclear Physics. Under the SciDAC program, a software infrastructure has been developed for lattice QCD that effectively utilize the capabilities of the INCITE facilities. These developments have enabled a new generation of Nuclear Physics calculations investigating the spectrum and structure of matter, such as the origin of mass and spin. This software infrastructure is described and recent results are reviewed.

We present details of lattice QCD computations we are performing on the Cray XT series of computers, from BigBen – an XT3 hosted at the Pittsburgh Supercomputing Center (PSC) – through Jaguar (XT4) and Kraken (XT5) – which are hosted at the National Center for Computational Science (NCCS) and the National Institute of Computational Science (NICS), respectively, at Oak Ridge National Laboratory (ORNL). We discuss algorithmic tuning to make the computation more efficient and present some recent results.

We present results for the equation of state of quark-gluon plasma in 2+1 flavor QCD with physical strange quark mass and close to the physical light quark masses. We also study the chiral condensate and susceptibility which are quantities sensitive to the chiral symmetry restoration aspects of the transition between ordinary matter and the quark-gluon plasma.

A tool for writing platform-independent optimized computational routines, QA₀, is described. Application to optimizing SciDAC Lattice QCD codes is presented.

Most calculations in lattice Quantum Chromodynamics (QCD) involve the solution of a series of linear systems of equations with exceedingly large matrices and a large number of right hand sides. Iterative methods for these problems can be sped up significantly if we deflate approximations of appropriate invariant spaces from the initial guesses. Recently we have developed eigCG, a modification of the Conjugate Gradient (CG) method, which while solving a linear system can reuse a window of the CG vectors to compute eigenvectors almost as accurately as the Lanczos method. The number of approximate eigenvectors can increase as more systems are solved. In this paper we review some of the characteristics of eigCG and show how it helps remove the critical slowdown in QCD calculations. Moreover, we study scaling with lattice volume and an extension of the technique to nonsymmetric problems.

Running First-Principles Molecular Dynamics (FPMD) simulations on large parallel platforms presents a number of practical challenges related to the large size of the datasets generated during simulations and to the lack of flexibility of parallel FPMD codes for the implementation of new algorithms. In this paper, we present two approaches implemented in the Qbox code, that alleviate these problems and facilitate large-scale FPMD simulations. We first describe a parallel I/O strategy based on MPI-IO that fully exploits the performance of parallel file systems. We also describe the implementation of a client-server interface that enables coupled quantum-classical computations in which the quantum simulation run by Qbox is "driven" by another application. This feature can be used to couple FPMD simulations with other simulation methods such as e.g. Path Integral Monte Carlo, thermodynamic integration or replica exchange dynamics.

Recent improvements to existing HPC codes NEMO 3-D and OMEN, combined with access to peta-scale computing resources, have enabled realistic device engineering simulations that were previously infeasible. NEMO 3-D can now simulate 1 billion atom systems, and, using 3D spatial decomposition, scale to 32768 cores. Simulation time for the band structure of an experimental P doped Si quantum computing device fell from 40 minutes to 1 minute. OMEN can perform fully quantum mechanical transport calculations for real-word UTB FETs on 147,456 cores in roughly 5 minutes. Both of these tools power simulation engines on the nanoHUB, giving the community access to previously unavailable research capabilities.

Hydrogen-fuelled vehicles require a cost-effective, lightweight material with precisely targeted thermodynamics and fast kinetics of hydrogen release. Since none of the conventional metal hydrides satisfy the multitude of requirements for a practical H₂ storage system, recent research efforts have turned to advanced multicomponent systems based on complex hydrides. We show that first-principles density-functional theory (DFT) calculations have become a valuable tool for understanding and predicting novel hydrogen storage materials and understanding the atomic-scale kinetics of hydrogen release. Recent studies have used DFT calculations to (i) predict crystal structures of new solid-state hydrides, (ii) determine phase diagrams and thermodynamically favoured reaction pathways in multinary hydrides, and (iii) study microscopic kinetics of diffusion, phase transformations, and hydrogen release.

The DCA++ code was one of the early science applications that ran on jaguar at the National Center for Computational Sciences, and the first application code to sustain a petaflop/s under production conditions on a general-purpose supercomputer. The code implements a quantum cluster method with a Quantum Monte Carlo kernel to solve the 2D Hubbard model for high-temperature superconductivity. It is implemented in C++, making heavy use of the generic programming model. In this paper, we discuss how this code was developed, reaching scalability and high efficiency on the world's fastest supercomputer in only a few years. We show how the use of generic concepts combined with systematic refactoring of codes is a better strategy for computational sciences than a comprehensive upfront design.

Though the thermal conductivity of solid UO₂ is well characterized, its value is sensitive to microstructure changes. In this study, we propose a two-way coupling of a mesoscale phase field irradiation model to an engineering scale, finite element calculation to capture the microstructure dependence of the conductivity. To achieve this, the engineering scale thermomechanics system is solved in a parallel, fully-coupled, fully-implicit manner using the preconditioned Jacobian-free Newton Krylov (JFNK) method. Within the JFNK function evaluation phase of the calculation, the microstructure-influenced thermal conductivity is calculated by the mesoscale model and passed back to the engineering scale calculation. Initial results illustrate quadratic nonlinear convergence and good parallel scalability.

The linearly scaling three-dimensional fragment (LS3DF) method is an O(N) ab initio electronic structure method for large-scale nano material simulations. It is a divide-and-conquer approach with a novel patching scheme that effectively cancels out the artificial boundary effects, which exist in all divide-and-conquer schemes. This method has made ab initio simulations of thousand-atom nanosystems feasible in a couple of hours, while retaining essentially the same accuracy as the direct calculation methods. The LS3DF method won the 2008 ACM Gordon Bell Prize for algorithm innovation. Our code has reached 442 Tflop/s running on 147,456 processors on the Cray XT5 (Jaguar) at OLCF, and has been run on 163,840 processors on the Blue Gene/P (Intrepid) at ALCF, and has been applied to a system containing 36,000 atoms. In this paper, we will present the recent parallel performance results of this code, and will apply the method to asymmetric CdSe/CdS core/shell nanorods, which have potential applications in electronic devices and solar cells.

The calculation of the spectrum of QCD is key to an understanding of the strong interactions, and vital if we are to capitalize on the experimental study of the spectrum. In this paper, we describe progress towards understanding the spectrum of resonances of both mesons and baryons from lattice QCD, focusing in particular on the resonances of the I = 1/2 nucleon states, and of charmonium mesons composed of the heavy charmed quarks.

We describe a fast real-analysis based O(N) algorithm based on multiresolution analysis and low separation rank approximation of functions and operators for solving the Schrödinger and Lippman-Schwinger equations in 3-D with spin-orbit potential to high precision for bound states. Each of the operators and wavefunctions has its own structure of refinement to achieve and guarantee the desired finite precision. To our knowledge, this is the first time such adaptive methods have been used in computational physics, even in 1-D. Accurate solutions for each of the wavefunctions are obtained for a sample test problem. Spin orbit potentials commonly occur in the simulations of semiconductors, quantum chemistry, molecular electronics and nuclear physics. We compare our results with those obtained by direct diagonalization using the Hermite basis and the spline basis with an example from nuclear structure theory.

We present details of lattice QCD computations we are performing on the Cray XT series of computers, from BigBen – an XT3 hosted at the Pittsburgh Supercomputing Center (PSC) – through Jaguar (XT4) and Kraken (XT5) – which are hosted at the National Center for Computational Science (NCCS) and the National Institute of Computational Science (NICS), respectively, at Oak Ridge National Laboratory (ORNL). We discuss algorithmic tuning to make the computation more efficient and present some recent results.

High-performance computing is now enabling the calculation of certain hadronic interaction parameters directly from Quantum Chromodynamics, the quantum field theory that governs the behavior of quarks and gluons and is ultimately responsible for the nuclear strong force. In this paper we briefly describe the state of the field and show how other aspects of hadronic interactions will be ascertained in the near future. We give estimates of computational requirements needed to obtain these goals, and outline a procedure for incorporating these results into the broader nuclear physics community.

The UNEDF SciDAC project to develop and optimize the energy density functional for atomic nuclei using state-of-the art computational infrastructure is briefly described. The ultimate goal is to replace current phenomenological models of the nucleus with a well-founded microscopic theory with minimal uncertainties, capable of describing nuclear data and extrapolating to unknown regions.

The structure and reactions of light nuclei represent fundamental and formidable challenges for microscopic theory based on realistic strong interaction potentials. Several ab initio methods have now emerged that provide nearly exact solutions for some nuclear properties. The ab initio no core shell model (NCSM) and the no core full configuration (NCFC) method, frame this quantum many-particle problem as a large sparse matrix eigenvalue problem where one evaluates the Hamiltonian matrix in a basis space consisting of many-fermion Slater determinants and then solves for a set of the lowest eigenvalues and their associated eigenvectors. The resulting eigenvectors are employed to evaluate a set of experimental quantities to test the underlying potential. For fundamental problems of interest, the matrix dimension often exceeds 10¹⁰ and the number of nonzero matrix elements may saturate available storage on present-day leadership class facilities. We survey recent results and advances in solving this large sparse matrix eigenvalue problem. We also outline the challenges that lie ahead for achieving further breakthroughs in fundamental nuclear theory using these ab initio approaches.

In this paper we present a project of visualizing topological charge in Lattice QCD in order to understand its evolution under the Markov Chain Monte Carlo algorithms and how to maximize its equilibration. Lattice QCD is a very computationally expensive technology for computing properties of composite particles from a few fundamental parameters such as quark masses. Our research plays an important role in validating these computations by showing that the autocorrelation length is small and it can eventually lead to even more efficient algorithms.

One of the central challenges facing visualization research is how to effectively enable knowledge discovery. An effective approach will likely combine application architectures that are capable of running on today's largest platforms to address the challenges posed by large data with visual data analysis techniques that help find, represent, and effectively convey scientifically interesting features and phenomena.

The Computer Supported Collaborative Work research community has identified that the technology used to support distributed teams of researchers, such as email, instant messaging, and conferencing environments, are not enough. Building from a list of areas where it is believed technology can help support distributed teams, we have divided our efforts into support of asynchronous and synchronous activities. This paper will describe two of our recent efforts to improve the productivity of distributed science teams. One effort focused on supporting the management and tracking of milestones and results, with the hope of helping manage information overload. The second effort focused on providing an environment that supports real-time analysis of data. Both of these efforts are seen as add-ons to the existing collaborative infrastructure, developed to enhance the experience of teams working at a distance by removing barriers to effective communication.

We segment the stabilization region in a simulation of a lifted jet flame based on its topology induced by the Y_OH field. Our segmentation method yields regions that correspond to the flame base and to potential auto-ignition kernels. We apply a region overlap based tracking method to follow the flame-base and the kernels over time, to study the evolution of kernels, and to detect when the kernels merge with the flame. The combination of our segmentation and tracking methods allow us observe flame stabilization via merging between the flame base and kernels; we also obtain Y_CH2O histories inside the kernels and detect a distinct decrease in radical concentration during transition to a developed flame.

We present several methods for visualizing motion, vector fields, and flows, including polygonal surface advection, visibility driven transfer functions, feature extraction and tracking, and motion frequency analysis and enhancement. They are applied to chaotic attractors, turbulent vortices, supernovae, and seismic data.

Changes are needed in the way that visualization is performed, if we expect the analysis of scientific data to be effective at the petascale and beyond. By using similar techniques as those used to parallelize simulations, such as parallel I/O, load balancing, and effective use of interprocess communication, the supercomputers that compute these datasets can also serve as analysis and visualization engines for them. Our team is assessing the feasibility of performing parallel scientific visualization on some of the most powerful computational resources of the U.S. Department of Energy's National Laboratories in order to pave the way for analyzing the next generation of computational results. This paper highlights some of the conclusions of that research.

Climate scientists and meteorologists are working towards a better understanding of atmospheric conditions and global climate change. To explore the relationships present in numerical predictions of the atmosphere, ensemble datasets are produced that combine time- and spatially-varying simulations generated using multiple numeric models, sampled input conditions, and perturbed parameters. These data sets mitigate as well as describe the uncertainty present in the data by providing insight into the effects of parameter perturbation, sensitivity to initial conditions, and inconsistencies in model outcomes. As such, massive amounts of data are produced, creating challenges both in data analysis and in visualization. This work presents an approach to understanding ensembles by using a collection of statistical descriptors to summarize the data, and displaying these descriptors using variety of visualization techniques which are familiar to domain experts. The resulting techniques are integrated into the ViSUS/Climate Data and Analysis Tools (CDAT) system designed to provide a directly accessible, complex visualization framework to atmospheric researchers.

The PDF gives full details of the electronic visualization and poster night.