AT&T 5G: Behind the scenes in Texas as the new mobile network goes live

AT&T has begun lighting up its 5G mobile network across the United States, with the carrier finally seeing the fruits of the research it kicked off over half a decade ago.

The 5G network went live in mid-December in parts of Dallas, Houston, San Antonio, Waco, Atlanta, Charlotte, Indianapolis, Jacksonville, Louisville, Oklahoma City, New Orleans, and Raleigh, using millimetre-wave (mmWave) spectrum.

In the first half of 2019, it will also be switched on across Los Angeles, San Diego, San Francisco, San Jose, Las Vegas, Orlando, and Nashville.

To see how AT&T's 5G rollout came together in time to launch by late 2018, ZDNet visited the carrier's Dallas HQ, Plano Foundry, Waco trial site, and Austin innovation labs.

Early 5G deployment in Waco

The carrier's rollout began with a trial in 2017 in Waco, Texas, at the Magnolia Market at the Silos.

The test network enabled AT&T to perform its first official 5G data call, with the purpose of the trial run to prove that mmWave would work for 5G, AT&T Labs director of Advanced Wireless Technology Group Arun Ghosh told ZDNet.

SEE: How 5G will transform business (ZDNet special report) | Download the report as a PDF (TechRepublic)

"We came into a commercial setting, and we still work here with a lot of early prototypes," Ghosh said during a tour of the Silos in Waco.

"We learned a couple of things -- I think one of the things that we learned is that usually, people have a very knee-jerk reaction when you say millimetre-wave because of previous experience with satellites, how susceptible they were to the signals getting faded and blocked, and how large the devices had to be; you had to have this big parabolic dish.

"So one of the things that technology has come a long way in is taking that to a much more robust and mobile platform."

Read also: AT&T to go live in 12 cities in December

AT&T kicked off its 5G trial network at the Magnolia Market lifestyle and homewares centre a year ago, in December 2017, using mmWave spectrum and 5G radio and antenna prototypes, as well as a series of Wi-Fi access points scattered throughout the complex.

The 5G radio was attached to an existing cell tower located around 180 metres from the receiver sitting in a window at the Silos.

Around 5,000 people visit the Silos daily, and a part from customers who are able to use the high-speed, high-capacity 5G Wi-Fi connection, the food trucks operating inside the grounds also use it for their point-of-sales systems and back-office management devices.

The 5G trial network in Waco was able to provide connectivity during Magnolia's Spring Festival earlier this year, which saw a 70,000-strong crowd descend on the centre -- an event for which AT&T would normally have had to deploy a cell on wheels.

"This is the only customer that we have left that's on our early pilot," Ghosh said, after the owners requested that AT&T keep the pilot up and running until the live network was ready to go.

"Once we have the commercial network, we will most likely swap it out with the commercial 5G equipment and turn the experimental system down at that point. The experimental system is big, it's clunky, it's power hungry, it's an earlier generation."

AT&T launches 5G across 12 cities

AT&T launched its live commercial 5G network in Waco on December 21, as well as across 11 other US cities. Much like the test network, the carrier is providing a 5G-compatible mobile hotspot that current smartphones can connect to via Wi-Fi.

CTO Andre Fuetsch told ZDNet that the launch involved working alongside partners Ericsson, Qualcomm, and Netgear at a "dramatically accelerated" timeline.

"We were pushing ourselves at speeds we've never done before, and the whole supply chain ecosystem that we're working with, we've never worked this fast with them before," Fuetsch said during an interview in Dallas this week.

Some of the towers AT&T went live with were built just days prior, he said. For such a fast launch, everything had to be developed in parallel, which is why AT&T chose to work with just one vendor, one chipset manufacturer, and one device maker initially.

"When we sat down a year ago and said that we were going to get this launch by the end of 2018, we had to focus on one particular RAN vendor to make this happen, one particular device vendor, and one chip vendor. Complicating it more, given the aggressiveness of the timeline, was too much. It was already complex enough," Fuetsch said.

The next seven launches at the start of 2018 will use Samsung and Nokia, alongside Ericsson.

"You'll see in 2019 a multi-prong 5G strategy with those three vendors," he added.

Fuetsch said there were several factors that went into choosing each early launch location, including spectrum availability; the receptiveness of the local municipality, with some more open to new technologies than others; where AT&T had Ericsson markets; proximity to its core datacentres and mobility packet core; and a selection across both urban and regional areas.

Unlike Verizon's fixed 5G offering, Fuetsch added AT&T's hotspot will work throughout the country across LTE and 5G with seamless handover, making the deployment both forwards and backwards compatible as more areas are switched on.

Read also: AT&T hits new 5G milestone with standards-based mobile device

AT&T will take learnings from this initial deployment and apply them to its future 5G expansion, Fuetsch said, including how mmWave signals interact with the real world.

Some of the issues around working with mmWave spectrum include short propagation distances and problems with building penetration, so AT&T is using reflections off existing objects to help project the signals to user devices -- even taking into account the changing seasons.

"Just based on reflections, you can get a pretty amazing service with millimetre-wave, so we expect as we deploy more, as we learn more about the particular areas that we're serving -- the building materials, the foliage, trees, all of those aspects play into how the service is going to perform," Fuetsch told ZDNet.

"And then also as you go through seasonality, trees get lots of leaves, leaves fall off, things like that, we're going to learn a lot more. But we're very confident that it will be a very positive experience, and we're pretty excited about some of the early test results we're finding in the cities that we've just turned on recently."

The physical challenges of deploying the actual radio equipment are also being overcome one location at a time. The carrier is using city-owned property for its attachments, so it must adhere to rules and regulations as well as guidelines around aesthetics, said Fuetsch.

One city particularly concerned with aesthetics is San Francisco, which has insisted that AT&T shroud its 5G equipment in decorative pieces from local artists.

"One of these compromises we did: There were three districts within San Francisco that we worked with the local community, where they selected local artists from those neighbourhoods to design, take their art, which we would then put on these vinyl wraps, and then wrap the equipment, and that seemed to have a very positive reaction," Fuetsch said.

As well as gaining experience across physical deployment and spectrum concerns next year, AT&T will use 2019 as a "big learning year" for 5G pricing and marketing to see what actually encourages customers to make the shift over to the new mobile network, Fuetsch said.

AT&T will initially be offering the Netgear Nighthawk 5G Mobile Hotspot and 5G data usage for free to selected enterprise and consumer customers for the first 90 days; as of spring 2019, customers will then be able to order the device and service for $499 upfront and $70 per month for 15GB of data.

The most interest in early 5G has come from enterprise customers, with local businesses in those 12 areas being selected, but consumer customers will also be "brought on in a limited fashion", Fuetsch said.

AT&T is also expecting to launch two 5G smartphones next year: One in the first half using mmWave spectrum, and the second using sub-6GHz in the second half of the year.

While the CTO would not comment on form factor, he said the smartphones are coming along well, and AT&T is "involved in every device that attaches to our network ... very involved in 5G smartphones".

But while mobile broadband is a big use case for 5G, Fuetsch did say that the new networks should not only be thought of as serving endpoint customers and machines; they are also a solution for providing backhaul instead of fiber.

"Small cells will become more intelligent, so as we deploy more of these to serve at let's say a city park, they can be deployed in a manner -- not today, but in the near future -- where they can actually form a mesh network in terms of how they connect to each other and backhaul the traffic," he explained.

"So you would only essentially need one particular node that would have to have a fiber connection. And this is important because today, you're basically having to make sure that there's a fiber facility to serve it; in future, we have the option to take advantage of millimetre-wave as a backhaul mechanism, and then over time as demand grows, it buys you time to eventually fiber them up."

The carrier is planning to have nationwide 5G coverage by early 2020, combining both its sub-6GHz and mmWave bands -- but what applications will actually require such a step-change in speed and capacity?

5G and the Internet of Things

One of the biggest areas of 5G work for AT&T is across the Internet of Things (IoT).

Thanks to the ultra low latency (ULL) enabled by 5G, AT&T IoT head Chris Penrose told ZDNet that the new mobile network opens up many new IoT use cases, such as remotely controlled mining, construction, and surgery.

The world of autonomy becomes increasingly possible when bringing ULL into the market, he said.

Due to this overlap between IoT and 5G, Penrose said his team has been "heavily involved" in 5G preparations and testing with the network arm of the company.

"At the end of the day, so many of the initial proof-of-concept solutions looking to be built are IoT in nature, and my team is set up as a vertically structured organisation so we've got expertise in automotive, we've got expertise in healthcare, we've got expertise in manufacturing, we've got expertise in retail," he explained when meeting with ZDNet in Dallas.

"And so we will pull in the right people on the application or solution we're looking to bring, we'll bring in my vertical experts as well as my technology experts, and we're heavily involved with everything going on.

"At the end of the day, it's really a massive team effort across this business to have everybody sitting around the table looking at what we can go create together."

One of the capabilities enabled by 5G networks is using multiple real-time video streams in smart cities deployments rather than just low-power sensors across infrastructure, Penrose said.

"They say a picture's worth 1,000 words; video is worth 100 sensors," he told ZDNet.

"You can get a lot more rich information off an image or a video stream, so how does that open up a whole bunch of different opportunities? Certain things can be processed locally and at the edge, but then if you wanted to be able to turn on all the video cameras in a certain city area because of a threat situation -- being able to take all that information on in real time to then help first responders be able to address an emergency situation becomes a very good use case."

Read also: AT&T and the push for smarter cities

Vehicle manufacturers have also come to AT&T for advice on how they can utilise 5G for better connected cars, he said.

In relation to the physical challenges of deploying 5G, Penrose said the IoT arm's smart cities partnerships are also helping with this. Its partnership with the City of Los Angeles, for example, is "accelerating zoning of putting up 5G infrastructure" in a faster and more economic manner.

Up in AT&T's Plano Foundry, the company is working on solving existing IoT-centric problems its customers have, and tapping into the 5G networked throughout the innovation lab to do so.

"What we try to do is to bring innovation from the outside and tap into whether it's innovation happening in startups, or innovation that we should be bringing to our customers," Vishy Gopalakrishnan, AT&T VP of Ecosystem and Innovation, told ZDNet in Plano.

"This is one of six Foundry locations and they all do similar kind of work, but this one focuses specifically on co-creation with our enterprise customers."

Some of the solutions AT&T has worked on in Plano include a grocery store using sensors to keep track of how many potato chip packets are left on a shelf; using a connected cooler to calculate when to restock cans of Red Bull; heat mapping customers moving around stores; and even helping construction companies calculate when they should clean a Porta Potty.

5G enables network slicing to work on separate enterprise use cases more effectively, director of AT&T Foundry Craig Lee said.

The Foundry's ideation station, used by customers when they first come to AT&T with a problem that needs solving, involves using Lego and Play Dough to create a mockup of the kind of solution they want.

In a process that Gopalakrishnan called "rapid prototyping", AT&T and its vendors then use these mockups to work out whether the solution is financially deployable and technically capable. The carrier interacts with its customers on at least a weekly basis, with the initial prototype usually produced within two weeks, Lee added.

Working with AT&T in the Foundry are vendors including Ericsson, Cisco, Tech Mahindra, and Nokia.

"5G depends on the applications and the solutions," Penrose argued.

"There are going to be massive amounts of sensors we need to put on in a certain area, we wouldn't be able to do it as it exists now on the network ... as we're sensorising everything and as we're putting more and more connections and sensors out there, the infrastructure needs to be able to support that."

Developing 5G technology in the labs

Developing all of these use cases together, while combining internally as well as with vendor partners on specific customer solutions, has resulted in AT&T opening a series of labs across the United States.

One such facility, the Arboretum Lab in Austin, is run by Ghosh, whose team works across testing and tweaking the sending and receiving of 5G radio signals, developing and building their own equipment to do so, and taking these findings back to AT&T's vendors.

Ghosh used the Waco trial as a testbed to determine how to build out commercial 5G.

"For Waco, I know we used a lot of the measurement from the pilot to actually plan the network," Ghosh told ZDNet.

Read also: AT&T unveils 5G Innovation Zone with Samsung

His team's PANDA (path-loss and network data acquisition) project dealt with collecting data from the network and feeding it back into the carrier's planning tools to make adjustments in the modelling.

"Planning tools, especially in these frequency bands ... everything is very new, many of them don't really model the things properly; sometimes there's a big gap between what the planning tool actually observed in Waco, what the planning tool actually predicted, and what we saw on the ground," Ghosh explained in Austin.

"Planning tools in our view still need to mature, and [this is] what PANDA is supposed to do."

Ghosh said PANDA took just three months to develop with a team of four people. He has now moved on to PANDA 2.0, dubbed ROACH (real-time omni-array channel) -- essentially PANDA with four phased arrays to get omni-directional coverage -- which will use VR headsets to "see" the network beams and make adjustments.

"What you would see on the VR is first of all, there is the camera that sits on top, and you'd have a 360-degree view of what it looks like from the viewer's point of view, then on top of that you would get a lot of information related to which cell you are receiving and which direction," Ghosh explained.

"You can click [on the visualised information] and what it's going to tell you is which base station it's coming from, which beam, and what signal strength you are getting, and maybe a few other parameters."

While proof-of-concepts are designed, developed, and tested at Arboretum Lab, the Spectrum Lab, also located in Austin, is used for support testing on the carrier's live mobile networks.

Spectrum Lab already has access to a Nokia 5G radio head, supporting both LTE and 5G at the sub-6GHz band. Lab workers have tested the 5G equipment so far with LTE, but as the Nokia software is not yet ready, they are expecting to begin running tests across the 5G equipment in May to June 2019.

Read also: A peek behind AT&T's 5G debut and $700 million spent in Kentucky (TechRepublic)

For now, the lab is responsible for call processing, data storage, and software-defined networking, with the facility housing routers, switches, virtual network functions, and call equipment across its 2G, 3G, and 4G LTE networks.

So with legacy networks still live and being supported, and 5G well and truly on its way to being a commercial reality, what are the labs turning to next?

As 5G deploys, 6G research begins

Ghosh's team began working on millimetre-wave way back in 2011, when AT&T was only just beginning to deploy 4G LTE.

Likewise, he said that as 5G is starting to launch, he is already turning to 6G.

"We have started working on bit by bit, little pieces of 6G, only theoretical at this point. You will probably not see any equipment on that for another four or five years," Ghosh told ZDNet.

"For 6G, we are leveraging a lot of university relationships. So part of that work gets funded at universities, and there's some very specific directions."

For Ghosh, 6G represents a series of opportunities for things he wants to do with mmWave that are "clearly out of the scope of 5G at this point".

"There are things that are slowly happening even in millimetre-wave that will make millimetre-wave at 6G everything that you can do with mobile," he said.

"Everything -- high mobility, high speed, all of that. All digital beam forming, but it requires a lot of new science."

Related Coverage

AT&T 5G to go live across 12 cities this week (TechRepublic)

AT&T will be launching 5G across 12 cities on December 21, offering the Netgear Nighthawk 5G Mobile Hotspot to select customers.

AT&T hits new 5G milestone with standards-based mobile device

AT&T completed the world's first mmWave mobile 5G browsing session with a standards-based commercial device

AT&T Fiber 1000: The fastest broadband internet money can buy

Fiber to the Home (FTTH) is being rolled out in limited markets. But if you have access to it, you need to consider it above other home and small business broadband options.

AT&T unveils 5G Innovation Zone with Samsung

Samsung and AT&T are looking to trial smart factory technologies in their new innovation zone in Austin.

AT&T to launch streaming service after $85b Time Warner purchase

Despite the Department of Justice's appeal against its Time Warner acquisition, AT&T has published an SEC filing outlining its plans to launch a streaming service by Q4 next year using WarnerMedia films and TV series.

How Louisville is poised to be one of AT&T's first 5G cities (TechRepublic)

AT&T has spent $700 million in the past three years in Kentucky to improve wireless and wired networks and support the debut of 5G in Louisville. Louisville is one of 12 cities slated for 5G from AT&T.

Mobile device security: A guide for business leaders (Tech Pro Research)

Attacks against mobile devices are growing more widespread and more sophisticated, requiring companies to adopt new tools, strategies, and best practices to safeguard their data assets.