Inside the secret city

In October 1942, the farmers working in the quiet valley west of Knoxville, US, came back from tending their fields to find notices on their doors. The US Army had visited every homestead along a 17-mile stretch of the Clinch river, passing along the winding dirt roads that cut through the woodland at the edge of the Great Smoky Mountains. The notices told the residents they had six weeks to leave their land forever. The second world war had come to Tennessee.

The site had been handpicked for a military base by Brigadier General Leslie Groves, the officer in charge of the secret Manhattan Project. He chose the valley for several reasons: it was remote, but had good road and rail links, and during the Great Depression massive hydroelectric dams had been built in the area, providing a secure source of electricity. Just as importantly, the valley had several natural ridge lines, shielding each part of the operation from the other. That kind of natural barrier mattered when you were building an atomic bomb.

By autumn 1943, the army had taken full control of the valley, transforming it on a scale never seen before or since. Within two years, a thriving community had emerged behind the fence, home to 75,000 people – scientists, soldiers and their families. They even had future celebrities scattered among them; one of the assistant cafeteria managers, Harland Sanders, would go on to found Kentucky Fried Chicken. Save for a regular convoy of army trucks, nobody went in and nobody went out.

Most of the workers did not understand their secret task, but the scale of the operation told them how much it mattered. In one valley sat the largest building in the world, the K-25 gaseous diffusion plant, its 152,000m² of floor space dedicated to enriching uranium. In another was Y-12, home to the Manhattan Project’s calutrons, large magnets on a racetrack for separating uranium isotopes. Finally, in a humble building of corrugated iron that looked like an old steel mill, there was X-10. Inside, Italian-American physicist and Enrico Fermi was present when the world’s first permanent nuclear reactor went critical, bathing uranium rods with slow neutrons to convert it first into neptunium, then into the first gram quantities of plutonium that would inform the design of the first atomic bomb.

The secret city needed a name, something that wouldn’t raise any eyebrows if overheard by foreign agents. In the end, the Army settled on Oak Ridge. 75 years later, Chemistry World paid them a visit.

From war to peace

Today, Oak Ridge National Laboratory (ORNL) is the largest science and energy lab in the Department of Energy’s national lab system. In the 75 years since its inception, ORNL has remained a powerhouse of scientific achievement in the US. Y-12, now a separate facility, remains just across the ridge, still a national security complex; the X-10 reactor remains too, powered down and preserved as a monument to the birth of nuclear power.

‘At the end of the war we had the best neutron source in the world,’ explains James Roberto, who retired as associate laboratory director in December 2017. ‘Eugene Wigner [the Nobel prize-winning physicist and the lab’s director of R&D from 1946] had a vision. Why not build a national laboratory around it? The US was in the process of trying to build a commercial nuclear power industry, and needed a reactor that could be used for fuels and separation chemistry.’ Nobody outside the Manhattan Project knew about nuclear reactors – so Oak Ridge helped develop nuclear engineering, teaching thousands of new nuclear engineers and even helping the US Navy develop its nuclear fleet.

’We also developed neutron scattering soon after the Manhattan Project, that was our first Nobel prize –it was awarded in 1994 – and we started the production and distribution of radioisotopes for medicine. That was our first major tech transfer event. Now it’s a billion dollar industry. Back then we made iodine-131, phosphorus-32, and carbon-14 because nobody else could make them. Today we still make isotopes nobody else can make.’ That includes targets used to discover new elements, including one that honours Oak Ridge’s contribution to this science: element 117, tennessine, discovered using a berkelium target in 2010.

ORNL is no longer solely focused on atomic power. It has the largest materials science portfolio and most diverse energy portfolio in the US. In one lab, a team is 3D printing an entire submarine (previously, they made a 1960s-era Shelby Cobra muscle car that was shown at the Detroit Motor Show – it was so perfect they had to put up a sign saying it had been 3D printed); in another they have built the world’s smallest neutrino detector. The latest major addition is the Spallation Neutron Source, an accelerator that provides the most intense pulsed neutron beams in the world, used to probe materials and conduct experiments across all scientific disciplines.

Once past the gatehouse, you’d never know you were in a secure facility. The central hub feels more like a university campus. Driving around the sprawling grounds, flocks of wild turkeys milling about the treeline make it feel more rural idyll than a scientific facility with 4700 staff, 4000 visiting researchers a year and an annual budget of $1.5 billion (£1.07 billion). ‘This budget comes in a thousand different projects, and we compete for all of them,’ Roberto says. ‘All operations support as well as salaries come from these projects, there is no separate source of funding.’

ORNL is a place, Roberto says, that comes up with a billion-dollar innovation at least once a decade. ‘And this is Tennessee, not California,’ he grins. Time for a tour.

High flux and hot cell labs

I’m standing in a control room that looks like something out of a science fiction movie. Below, seen through the viewing gallery, is what looks like an inviting pool. But it’s not somewhere you’d want to swim – at the bottom, 17 feet (5m) below the surface, is the top of a nuclear reactor. The High Flux Isotope Reactor (HFIR) was the 13th reactor built at Oak Ridge, in 1965, and is the only one still in use.

In the pool, you can see the top of the reactor vessel from the control room and the spent fuel cylinders from previous reactor cycles. One of them has an electric blue glow emanating from it, casting a strange light into the water. This is Cherenkov radiation, betraying the beta particles emitted by fission that travel faster than the speed of light in water. In the reactor, a single fuel cylinder is surrounded by a one-foot-thick layer of beryllium to reflect neutrons back toward the core, allowing the reactor to go critical. Safety plates, held up by electromagnets, are ready to come crashing down as a safety feature should anything go wrong.

Kevin Smith, deputy division director for the reactor research division and my guide for this stop, takes me over to a dummy cylinder to look inside. Two feet long, its interior has three smaller cylinders – from the top it looks like a bullseye. The outer and middle sections are filled with 540 metal plates, curved to provide a constant width for the coolant regardless of where it enters. At the centre, a five-inch (12cm) hole allows the loading of target rods, ready to be bathed in neutrons to undergo transmutation into heavier elements.

‘It’s a flux trap,’ Smith explains. ‘The neutrons leak into the centre and reach peak flux.’ He pulls a two-foot long rod, roughly as thick as a pencil, from the cylinder’s centre. ‘You screw off the top and then you can put seven of these rabbits in there. Rough numbers: if you were designing a full-size experiment to go in here, it’s at least hundreds of thousands of dollars. These little rabbits alone are worth $10,000.’

A reactor cycle is about 25 days, and the products range from trace amounts of fermium to the plutonium-238 used by NASA to power the Mars Rover or Voyager probe. Currently the reactor is producing californium-252, which takes four cycles. The HFIR is one of only two facilities in the world capable of producing measurable quantities of trans-curium elements: the other is in Dmitrovgrad, Russia.

Once created, the material has to be separated. Up the hill, a short distance away, sit the hot labs. These are the beating heart of nuclear and radiochemistry at ORNL. ‘The walls are 54 inch concrete, each window has three panes of leaded glass that range between 3–8 inches, and in between each of those is a mineral oil,’ explains Julie Ezold, the Cf-252 programme manager. The operators work 12-hour shifts, 24 hours a day. Acid–base chemistry is used to separate target materials from the aluminium matrix it’s trapped within. Alpha-hydroxyisobutyric acid is then used in a column to separate out each of the target elements. ‘Because of the distance and the pH, it separates by element in reverse order: fermium, einsteinium and so on,’ Ezold says. ‘It’s pure chemistry, just with a bit of radiation added to it. I still think of it as experimentation, not production. The complete process is never the same twice.’

Operating the hot cell cubicle – as each of the labs’ containment units are called – is an art unto itself. Mechanical arms extend through the glass into the cell’s narrow confines, operated so expertly the metal claws can operate a micropipette inside. Watching the team at work is mesmerising: balletic twists and flicks combined with an expert eye. Just knowing where all the lines and pipes go is hard enough – in the cell’s ceiling everything hangs down like spaghetti. Sometimes the team uses mirrors to see what’s going on; sometimes the puppeteer needs a spotter. The extraction is so precise that nanogram qualities can be obtained. ‘They do magic,’ Ezold says. ‘It’s both science and art.’

The Cf-252 material is then cleaned up and loaded on a resin to make it stable. It’s then fired through an underground pipe to an adjacent hot cell facility. ‘One microgram of Cf-252 emits approximately 2 million neutrons a second. We transfer 20–30 milligrams per package. It only takes 15 seconds but everybody is evacuated and we only do it in the evening,’ says Ezold. From there it can be processed into wires for commercial suppliers.

The next stop has wires of a different kind.

Remember the Titan

Deep in the main campus, behind a ridge line from HFIR and the hot labs, is a vast, white room, like something out of a science-fiction movie. White floors, white walls, white ceiling. You need ear protectors for the constant hum, a thick shirt for the chill of refrigerant in the air. At the front, blazed across along row of cabinets, is a single word: Titan. This is only the first of eight such rows, all part of one of the world’s fastest supercomputers. ‘It basically fills a basketball court,’ comments Buddy Bland, project director of the leadership computing facility. ’That’s without all the disk drives and so forth. It’s something close to 5000 square feet (460m²) when you put all of it together.’

The measure of supercomputer speed is in floating point operations a second (flops). Titan’s predecessor, Jaguar, ran at around 2.6 petaflops (2.6×10¹⁵ flops). In 2009, it was fastest computer in the world, but even then it was too slow. ‘Our users were telling us they could not solve some of the problems in a reasonable amount of time,’ Bland explains. The answer was Titan. Its maximum processing capability is 27 petaflops, and when it became operational in 2012, it took the fastest supercomputer crown.

Titan is, in many respects, a normal computer. It runs on the Linux operating system, is connected to the Internet (as one of the major hubs of the research web it has a direct fibre connection to Cern, Japan and beyond) and can turn on and off just like a normal PC. But of course, Oak Ridge doesn’t do anything on a normal scale. Each of Titan’s nodes has a 6GB graphics processing unit and 32GB central processing unit – more powerful than any games console on the market. Titan has 18,688 such nodes working together simultaneously – 560,640 cores – recording to around 20,000 drives. ‘The drives can hold about 32 petabytes of data and can stream it in and out of Titan at about a terabyte a second.’

Such processing power requires a lot of energy. The power cables to Titan are inch-thick copper wires, needed to feed its hungry demands. ‘The machine uses 9MW of power,’ Bland says. ‘The entire city of Oak Ridge only uses 100MW.’ The cabinets heat the air blowing through them to almost 50°C, but it is then cooled back to 20°C by a constant stream of 1,1,1,2-tetrafluoroethane. There are occasional power outages but nothing that affects the system. ‘Lightning strikes happen, and sometimes small animals decide to electrocute themselves in the transformers,’ Bland concedes. ‘We don’t lose power often, but when we do we just rerun the program.’

Unlike most research computers, Titan focuses on the most ambitious research projects – projects that require and can make effective use of one of the world’s largest supercomputers. ‘We offer high-performance computing and data resources that are 10 to 100 times more powerful than systems typically available for open scientific research elsewhere,’ Bland says. ‘Anyone in the world can apply to use it. Typically we say, “If you cannot do it someplace else because it’s too big or takes too much time, we’re the place to come to.” That’s what it means to be a leadership facility.’

Unsurprisingly, the raw power of Titan is a big draw for computational chemists. At any point in the year, the behemoth could be running electronic structure calculations, molecular dynamic simulations, density functional theory or Monte Carlo simulations. Bland explains that the machine is being used by a computational chemistry group in Denmark; in the past it’s been used to model entire nuclear power reactors, allowing contractors to compare how a reactor performs against the simulation.

Despite its staggering capabilities, Titan’s fate is sealed. Currently it’s only the fifth fastest in the world: for Bland’s users, that’s not fast enough. Already ORNL is building a new machine, Summit, next door. Expected to reach up to 200 petaflops, it will probably be the world’s next top supercomputer. From the moment Summit switches on, Titan will be obsolete. ‘We’ll have an overlap of maybe six months to a year,’ he estimates. ‘Once we do that, it’ll be turned off and it’ll get scrapped.’

Scrap a $100 million machine? Bland nods. ‘In the same way that your six-year-old laptop wouldn’t get re-used, that even a school wouldn’t take it for free … this morning we were already having a meeting planning for the machine that will replace Summit.’

You have to move fast when you’re in the lead.

At the nanoscale

Computing isn’t the only area where things move fast and ORNL wants to stay in the lead. ‘ORNL has cutting-edge microscopes that do things you can’t do anywhere else right now,’ says Hans Christen, the director of the Center for Nanophase Materials Sciences at ORNL. ‘The newest ORNL addition is a Nion instrument that is a monochromated scanning transmission electron microscope. That gives us an energy resolution of about 10meV – we’re in the same ranges a neutron can probe.’ This makes it possible to measure heat variations in a circuit at the nanoscale, ‘the lowest point at which the concept of temperature makes sense’.

The centre led by Christen is another leadership facility: users come here when they can’t do something on their own. In another example of using electron microscopes, he shows how shining an electron beam at the interface between crystalline and amorphous regions in silicon allows the amorphous material to crystallise. The beam can be used to write your name. ‘That’s just the boring part,’ Christen shrugs. Changing the structure allows you to encapsulate at the nanoscale, or drill a hole through graphene. ‘We’re turning microscopy around and using it as a tool to manipulate materials.’

The tour is yet another example of how ORNL is science’s answer to Willy Wonka’s chocolate factory. There’s a helium ion microscope – without using electrons, this interacts with a much smaller area of the sample surface, leading to even higher resolutions than scanning electron microscopes. ‘There are only about 10 of them in the world,’ Christen explains. ‘It’s the highest spatial resolution chemical imaging that you can do.’ In the next room they are building 3D nanostructures out of pure gold, which could one day be used for molecular machines, electrode channels, and even 3D cages to trap individual cells for biology. In the next, copper nanoparticles are being deposited on carbon spikes, used in the electrochemical catalysis of carbon dioxide to produce ethanol directly. With yields of 63%, it’s potentially a major waste-to-fuel breakthrough.

The nanoscale world moves quickly. And, like everything else at ORNL in the past 75 years, the focus is on leading, not keeping pace. ‘It’s a constant race, right? We have small things that become obsolete just as soon as they’re here,’ Christen says. ‘If we do nothing, we’d still have something that’s better than most universities today but we’d not long remain the leaders in the field.’

Christen smiles, summing up the laboratory’s mantra. ‘We create the instruments that do something no other instrument can. That’s what works. So that’s what we do.’