Researchers at the company are trying to understand their A.I. system’s mind—examining its neurons, running it through psychology experiments, and putting it on the therapy couch.

For more than a century, scientists have treated chemistry as the master key to building new materials, tuning bonds and charges to coax atoms into useful patterns. A new wave of research is quietly ...