{"id":698455,"date":"2020-10-16T00:44:48","date_gmt":"2020-10-16T07:44:48","guid":{"rendered":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/?post_type=msr-blog-post&p=698455"},"modified":"2020-10-26T21:32:22","modified_gmt":"2020-10-27T04:32:22","slug":"enabling-linear-acceleration-and-lossless-performance-for-large-scale-deep-learning-training-a-bmuf-based-adam-optimizer-parallelization-practice","status":"publish","type":"msr-blog-post","link":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/articles\/enabling-linear-acceleration-and-lossless-performance-for-large-scale-deep-learning-training-a-bmuf-based-adam-optimizer-parallelization-practice\/","title":{"rendered":"Enabling linear acceleration and lossless performance for large-scale deep learning training, a BMUF-based Adam optimizer parallelization practice"},"content":{"rendered":"

As an adaptive learning rate stochastic optimization method, Adam has been widely used in the field of deep learning since it was first proposed in 2014. In order to improve its training efficiency when applied to large-scale tasks, Adam is often combined with the synchronous stochastic gradient (SSG) technique to run in parallel on multiple workers. In this article, we refer to this method as \u201cSync-Adam.\u201d<\/p>\n

Essentially, Sync-Adam speeds up training by distributing the gradient computation of samples within a minibatch to multiple workers, and so communication occurs very frequently. With the increase of parallel workers, the number of samples within a minibatch also increases proportionally, which often hurts the final model performance. To solve the poor scalability issue of SSG-based Adam, we plugged Adam into a Blockwise Model-Update Filtering (BMUF) framework.<\/p>\n

BMUF is a communication-efficient distributed optimization framework proposed by researchers from MSRA\u2019s speech group in 2016. This framework periodically synchronizes model-update information among parallel workers and combines it with historically updated information to improve the performance of the global model. Compared with the SSG-based method, BMUF realizes low communication frequency, near linear speed up, and very little model performance degradation. Given all these advantages, BMUF has been widely used in the industry to train large-scale deep learning based models.<\/p>\n

We used BMUF to parallelize Adam and evaluated it on Microsoft\u2019s large-scale OCR and speech product tasks. Experimental results show that BMUF-Adam delivers almost linear speed-up without performance degradation with up to 64 workers on large-scale OCR tasks, and similar effects are achieved on large vocabulary continuous speech recognition (LVCSR) task with up to 32 workers.<\/p>\n

Next, we will investigate how we can empower Adam with BMUF to achieve further positive results.<\/p>\n

Review of BMUF<\/h2>\n

In the BMUF-based training framework, we have N parallel workers. A worker can be a GPU card, a compute node, etc. Given a block of training data, it will be partitioned into N splits, and each split will contain \u03c4 mini-batches. Starting from a common initial model parameter \u03b8_(t-\u03c4)^((init)), all workers will update their local models with respective data splits for \u03c4 steps in parallel to obtain N local models{\u03b8_(t,1),\u03b8_(t,2),\u2026,\u03b8_(t,N)}. This procedure is called \u201cintra-block parallel optimization\u201d (IBPO). Instead of treating the averaged local models \u03b8 \u0305_t as an updated global model, BMUF introduces a block momentum to leverage historical updated information to obtain a better global model as follows:<\/p>\n

\"\"<\/p>\n

\"\"<\/p>\n

where \u03b7 refers to block momentum and \u03b6 refers to block learning rate. We set \u03b6=1 in this paper, which is common practice for BMUF. As discussed in [2], block momentum can compensate per mini-batch’s inadequate contribution to the final model-update caused by averaged operation to improve model performance. We defined a new variable \u03c1_n to represent the number of equivalent mini-batches required to get \u0394_n. Since<\/p>\n

\"\"<\/p>\n

therefore\uff1a<\/p>\n

\"\"<\/p>\n

If \u03b7=1-1\/N according to [2], lim\u252c(n\u2192\u221e)\u2061(\u03c1_n)=N\u03c4, which is equal to the number of mini-batches in a data block. This deduction shows that lim\u252c(n\u2192\u221e)(\u0394_n)can simulate model-update resulting from processing N\u03c4mini-batch data blocks in serial. This conclusion requires an assumption that \u03b8 \u0305_t-\u03b8_(t-\u03c4)^((init)) is stationary.<\/p>\n

In this article, we use the Nesterov block momentum technique to obtain the initial model for the next IBPO operation.<\/p>\n

\"\"<\/p>\n

Since<\/p>\n

\"\"<\/p>\n

therefore,<\/p>\n

\"\"<\/p>\n

Review of Adam<\/h2>\n

Adam is an adaptive learning rate based stochastic optimizer, whose learning rates are determined by the first and second moment of the stochastic gradient as follows:<\/p>\n

\"\"<\/p>\n

where \u2a00 represents element-wise multiplication and g_t is the stochastic gradient of the t-th mini-batch. Based on the assumption that $ E[g_t]and E[g_t\u2299g_t]are stationary, Adam uses an exponential moving average to estimate these two moments as follows:<\/p>\n

\"a<\/p>\n

where<\/p>\n

\"\"<\/p>\n

BMUF-Adam<\/h2>\n

Since BMUF is a general distributed optimization framework, we can plug Adam into it as a local optimizer. According to our revision of BMUF and Adam, if we directly plug Adam into BMUF without any change, the 1st and 2nd order moments of Adam will be mismatched with the model parameter. Assume we set:<\/p>\n

\"\"<\/p>\n

Obviously, due to the existence of \u03b7\u0394_n, the averaged 1st and 2nd moments will become stale for the initial model parameter of the next IBPO operation. Since the number of equivalent minibatches of \u03b7\u0394_n is \u03b7\u03c1_n, we believe the m_t^((init)),v_t^((init)) that is compatible with \u03b8_t^((init)) can be obtained by processing \u03b7\u03c1_n minibatches starting from m \u0305_t,v \u0305_t.<\/p>\n

According to the 1st moment update formulation, we can have:<\/p>\n

\"\"<\/p>\n

Combined with the stationary consumption of E[g_t], we have:<\/p>\n

\"\"<\/p>\n

Where E[g^((n))] represents the expectation of the minibatch stochastic gradient of n-th IBPO. Therefore,<\/p>\n

\"\"<\/p>\n

because<\/p>\n

\"\"<\/p>\n

Finally, we obtain:<\/p>\n

\"\"<\/p>\n

According to the formulation of m \u0305_t and m_t^((init)), it\u2019s not difficult to find that when \u03c4 is small and N is large, m_(t-\u03c4)^((init)) overly contributes to m \u0305_t, which results in more serious inconsistency between \u03b8_t^((init)) and m \u0305_t, and this phenomenon is verified in our experiments. With the introduction of an equivalent minibatch, the contribution of stale m_(t-\u03c4)^((init)) to m_t^((init)) can be reduced significantly, which will improve the model performance. Meanwhile, using small \u03b2_1 can also reduce the contribution of m_(t-\u03c4)^((init)). The above analysis can also be applied to the 2nd moment. Finally, we can obtain BMUF-Adam as follows:<\/p>\n

\"\"<\/p>\n

Experimental results<\/h2>\n

We first verified our method on Microsoft\u2019s large scale English OCR task. First, different moments estimation strategies are compared when Adam is plugged into BMUF. The experimental results are listed in Table 1 (\u03c4=8 in these experiments):<\/p>\n

\"table\"

Table 1: experimental results<\/p><\/div>\n

We find that when an averaging strategy is used, model performance degradation becomes increasingly serious with an increase in parallel workers. The proposed strategy can relieve the degradation significantly, and combined with a small \u03b2_1, the model performance is further improved. Next, we compared Sync-Adam with BMUF-Adam on this task. Experimental results are listed in Table 2:<\/p>\n

\"table\"

Table 2: a comparison of Sync-Adam and BMUF-Adam<\/p><\/div>\n

With an increase in parallel workers, Sync-Adam suffers increasingly serious performance degradation and the speedup ratio worsens. As a comparison, BMUF-Adam can achieve almost linear speedup with little performance degradation.<\/p>\n

We also verified BMUF-Adam on the LVCSR task with 6000 hours of Microsoft product data, and the results are listed in Table 3:<\/p>\n

\"table\"

Table 3: experimental results on the speech recognition task<\/p><\/div>\n

Again, BMUF-Adam can achieve almost linear speedup with little performance degradation and shows excellent scalability.<\/p>\n

Conclusion<\/h2>\n

In this article, we used BMUF to parallelize the widely used Adam algorithm, and experimental results show that compared with the traditional SSG based method, BMUF-Adam can achieve faster training speed, better model performance, and better scalability. This algorithm has been applied to several Microsoft products, and we welcome everyone to give it a try. ????<\/p>\n

More details:
\nhttps:\/\/ieeexplore.ieee.org\/stamp\/stamp.jsp?tp=&arnumber=9052983<\/p>\n

References:
\n[1] D. P. Kinagma, J. Ba, \u201cAdam: a method for stochastic optimization,\u201d Proc. ICLR-2015, arXiv:1412.6980.
\n[2] K. Chen, Q. Huo, \u201cScalable training of deep learning machines by incremental block training with intra-block parallel optimization and blockwise model-update filtering,\u201d Proc. ICASSP-2016, pp.5880-5884.
\n[3] K. Chen, H. Ding, Q. Huo, \u201cParallelizing Adam optimizer with blockwise model-update filtering,\u201d Proc. ICASSP-2020, pp.3027-3031.<\/p>\n","protected":false},"excerpt":{"rendered":"

As an adaptive learning rate stochastic optimization method, Adam has been widely used in the field of deep learning since it was first proposed in 2014. In order to improve its training efficiency when applied to large-scale tasks, Adam is often combined with the synchronous stochastic gradient (SSG) technique to run in parallel on multiple […]<\/p>\n","protected":false},"author":34512,"featured_media":701359,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-content-parent":199560,"msr_hide_image_in_river":0,"footnotes":""},"research-area":[],"msr-locale":[268875],"msr-post-option":[],"class_list":["post-698455","msr-blog-post","type-msr-blog-post","status-publish","has-post-thumbnail","hentry","msr-locale-en_us"],"msr_assoc_parent":{"id":199560,"type":"lab"},"_links":{"self":[{"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post\/698455","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post"}],"about":[{"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-blog-post"}],"author":[{"embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/users\/34512"}],"version-history":[{"count":1,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post\/698455\/revisions"}],"predecessor-version":[{"id":698521,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post\/698455\/revisions\/698521"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/media\/701359"}],"wp:attachment":[{"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/media?parent=698455"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=698455"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=698455"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=698455"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}