In the DMLab-30 suite the developers have turned to use the Weighted Actor-Learner Architecture. This relies on an efficient distributed architecture with TensorFlow, which itself uses multiple distributed actors to learn the agent behaviour. Periodically actors stop the exploration to send the data to a center parameter server.
Links
https://www.deepmind.com/blog/scalable-agent-architecture-for-distributed-training